Skip to content

Field Guide / Governing adaptively

ConceptConceptual framing

Self-limiting vs self-sustaining error

In any system with propagation loops there is a threshold. Below it, errors are self-limiting: corrections outpace spread, and mistakes fade. Above it, errors are self-sustaining: errors seed new errors through people and records faster than they are corrected, and contamination persists no matter how good any single day looks.

The most useful question in AI governance is a simple one: when an error occurs, does correction outpace its spread, or does its spread outpace correction? Correction wins, and the system cleans itself — mistakes happen, then fade. Spread wins, and errors compound: people adopt them, records store them, retrieval repeats them, and the system settles into a contaminated equilibrium that no individual correction fixes.

The threshold reframes what governance is for. The goal is not zero errors — no ceiling allows that — but a self-limiting regime: correction capacity that outpaces spread. And the threshold is a property of the whole arrangement, which is why system-side levers (verification before writes, reviewer capacity, provenance labels, loop cuts) can move a deployment across it when model improvements alone cannot.

The threshold also explains why deployments surprise their owners. Growth — more users, more automation, more record coupling — adds propagation paths. A system that was comfortably self-limiting at pilot scale can cross into self-sustaining territory simply by succeeding, with no change to the model at all.

The PAN Lab's central gauge is exactly this: watch which side of the threshold a scenario sits on, and which combinations of governance choices move it — in both directions.

This page is conceptual framing, a way of seeing, not an empirical claim. Documented real-world events appear in the Domain Atlas with citations; testable versions of these ideas live in the PAN Lab.

Self-limiting vs self-sustaining error is one lens among twelve in the Field Guide. Seeing what it means for a live deployment is what engagement is for.

Work With Paramerge