Software engineering AI (coding assistants)

The software-engineering domain's causal anchor: pre-registered, peer-reviewed randomized rollouts of a commercial code assistant across 4,867 developers found a pooled 26% increase in completed tasks — concentrated among less-experienced developers. An independent study of 16 experienced maintainers bounds the other tail: about 19% slower with the AI while believing themselves 20% faster. The engineer who accepts a suggestion is also its reviewer, and individual gains do not compose to org delivery unless gates absorb the added churn.

An in-house machine-learning code-completion system built, deployed, and measured by Google's own platform organization for 10,000+ developers, with a control group: 25–34% acceptance, a 6% reduction in coding iteration time versus control, 3% of new code characters from the model. The structural point is unification — one organization owns the model, the monorepo, the review gates, and the telemetry, so it can tune the whole loop; the corresponding risk is that the measuring, building, and deploying party are the same, so no external check exists at all.

A regulated bank ran a structured six-week experiment with ~100 of its 5,000 engineers before scaling a commercial coding assistant to ~1,000, and published its own measurement. Engineers reported productivity and code-quality gains — and recorded the security impact as explicitly inconclusive: a real gating decision taken under uncertainty, with the unknown honestly carried forward rather than resolved by assertion. That recorded unknown is the case's most valuable datum, because the class-level security literature says the unknown is not hypothetical.

ZoomInfo's systematic four-phase evaluation-to-rollout of GitHub Copilot across 400+ developers, well documented: 33% acceptance, 20% of suggested lines accepted, 72% satisfaction, per-language variation, stated limitations. Its evaluation instrument is acceptance-rate telemetry — the measure most correlated with perceived, not real, productivity. And it reported no security evaluation at all: an unrecorded unknown, one step less honest than a recorded inconclusive one. The value here is the documentation quality of an ordinary, competent adoption.