Domain Atlas / Software engineering AI (coding assistants)
Randomized coding-assistant field experiments
Explore this deployment in the PAN Lab ↗
Company-run, pre-registered, peer-reviewed randomized rollouts of a commercial code-completion assistant across 4,867 developers at three enterprises found a pooled 26.08 percent increase in completed tasks, with gains concentrated among less-experienced developers. An independent randomized study of 16 experienced open-source maintainers on 246 tasks in familiar repositories bounded the expert tail from the other direction: those developers were about 19 percent slower with the AI while believing themselves about 20 percent faster — a measured perception-reality gap that means a uniform productivity number overstates the effect for senior engineers.[2]
What happened
The software-engineering domain's causal anchor is a set of company-run randomized rollouts of a commercial code-completion assistant across 4,867 developers at two named enterprises and an anonymized third. The experiments were pre-registered and peer-reviewed, and they found a pooled 26.08 percent increase in completed tasks. The central fact is not the average but its shape: the gains were concentrated among less-experienced developers, and near zero for experts.
The expert tail is bounded from the other direction by an independent randomized study of 16 experienced open-source maintainers working 246 tasks in repositories they already knew well. Those developers were about 19 percent slower with the AI than without it — while believing themselves about 20 percent faster. That perception-reality gap is the domain's most important honesty datum: the measure most deployments track (how the tool feels, via acceptance rates) is exactly the measure that is miscalibrated for the engineers the tool helps least. A uniform productivity number, quoted without the seniority distribution, reports the novice result and lets it stand for everyone.
Two structural features define the deployment. First, the operator and the verifier collapse into one person: the engineer who accepts a suggestion is also its reviewer of record, so there is no independent check between an accepted suggestion and the shared repository unless the organization builds one. Second, the shared repository the assistant writes into is the same corpus later engineers and later assistants read as fact, so an accepted error does not sit in one place — it propagates as inherited code. The organization-level literature closes the loop: a cross-industry program measured a 7.2 percent decrease in delivery stability for every 25 percent increase in AI adoption, evidence that the individual speed-up does not compose to the organization's delivery outcome unless code-review and testing gates absorb the churn the assistant adds. Honesty boundaries: several authors are affiliated with the assistant's vendor and the experiments were run by the firms themselves — the peer-reviewed venue and the pre-registration are the checks that make the numbers credible despite that.
The sociotechnical reading
This case moves the Atlas out of social services and clinical care and into an organization's own workforce, but the governance grammar is the same — and it exposes two traps that are specific to generation into a shared codebase. The first is that the benefit is a distribution masquerading as an average. A pre-registered randomized experiment is strong evidence, and it says the tool delivers a real, large lift — to novices. For experts, an independent randomized study measured a slowdown paired with a confident feeling of speed-up. So the honest service term here is not a single percentage; it is a curve over seniority, and a deployment that reports the pooled number is reporting the group it helps most. The governable move is to measure who the tool actually helps rather than how fast it makes people feel.
The second trap is the collapse of the operator into the verifier. In the clinical documentation domain the clinician reviews an AI draft before it enters the record; here the engineer who accepts the suggestion is the review, so there is no second read between the machine's output and the shared repository unless the organization inserts one. That matters because the repository is not a private notebook — it is the corpus later engineers copy from and later assistants train and contextualize on, so an accepted error becomes inherited code with a long half-life, exactly as an unreviewed clinical note becomes inherited fact. And the organization-level evidence says the individual speed-up actively degrades delivery stability as adoption climbs unless the code-review and testing gates are resourced to absorb the extra volume. The map's instruction is that a coding assistant's real governance surfaces are downstream of the keystroke: an independent review between acceptance and merge, a reconciliation of machine-written code against a standard before it becomes the corpus, and gates sized to the churn. The honest boundary throughout: no product or software outcome is modeled on the Lab diagram. The downstream users of the software are boundary-only; suggestions, acceptances, and merges are institutional signals, and the productivity numbers, the expert slowdown, and the delivery-stability bound live in the case file, never on any network.
The concepts used in this reading are defined in the Field Guide; the governance responses live in the Practice Library. The model organization for this case can be stress-tested in the PAN Lab.