Domain Atlas / Caseworker documentation & copilots
Amsterdam Smart Check
The City of Amsterdam spent roughly five years and an estimated EUR 535,000 building a deliberately fair, explainable welfare-fraud screening model with nearly every recommended pre-deployment safeguard in place - a bias audit, training-data reweighting that approximately equalized wrongful-flag rates on retrospective data, a data-protection assessment and a human-rights assessment, external and academic review, a citizen panel, and dual algorithm-register transparency - and discontinued it after a 2023 live pilot on nearly 1,600 applications. In the investigating journalists' analysis of aggregate data the city provided, the group disparities re-emerged inverted on the live pilot, now more likely to wrongly flag Dutch nationals, women, and applicants with children, with the tool flagging more applications than the analog process and no better than caseworkers at finding genuine cases. The Dutch national algorithm register records the deployment ending September 2023 and lists it out of use, and the responsible alderman announced the halt in November 2023.[3]
What happened
Smart Check (Slimme Check) was the City of Amsterdam's investigation-worthiness screener for new social-assistance (bijstand) applications, and it is unusual in this atlas because the city that built it is the one that shut it down. The city began designing it in 2019; the effort ran roughly five years at an estimated cost of about EUR 535,000 — about EUR 500,000 in in-house development plus a EUR 35,000 review by Deloitte — the city's own estimate rather than an audited figure. The model was an Explainable Boosting Machine, an interpretable gradient-boosted model, over 15 behavioral and administrative features drawn from the municipal personal-records database (citizen ID, address, housing situation) and Income Services systems (prior applications, assets, income). It was trained on about 3,400 historical enforcement investigations, and sensitive attributes — gender, nationality, age — along with known proxies such as postal code were deliberately excluded from the features. Per the investigation's published methodology, a risk score above 0.56 produced an advisory "investigation-worthy" label. The output was advisory only and fed a multi-stage human pipeline: an enforcement employee validated each label, a separate investigator conducted any detailed review (with powers including bank-record access and home visits), and a different Income Services official made the final grant decision, with staff training and written instructions against over-reliance. The national algorithm register frames this as "meaningful human intervention," not automated decision-making.
The governance stack was as complete as the period offered. The city performed a data-protection impact assessment (DPIA), an IAMA human-rights-and-algorithms assessment, an AI impact assessment, and a BIO security quickscan; it ran the design past the city data protection officer and the Personal Data Commission; it commissioned the Deloitte review and consulted the University of Amsterdam's Civic AI Lab; it published entries in both the Amsterdam and Dutch national algorithm registers; and it convened a citizen Participation Council of up to 15 benefit recipients and interest-group representatives. That council, briefed in depth in late 2021 (MIT Technology Review dates the first briefing to the fall of 2021; the city's own research platform says the end of 2021), concluded in March 2022 that the project should be stopped, citing discriminatory variables such as year of birth, disproportionate data processing, and use of an algorithm without transparency or democratic control. The city removed some variables and strengthened data safeguards, but overrode the core stop recommendation and proceeded to a pilot. Internal testing in May 2022 had already found the model wrongly flagged non-Dutch applicants about 30% more often, non-Western applicants nearly twice as often, and men about 14% more often; the team then applied training-data reweighting, which approximately equalized wrongful-flag rates between Western and non-Western applicants on retrospective data.
Then it went live, and the fairness did not transfer. In a pilot on nearly 1,600 real applications in 2023 (the city's platform gives the window as April to August 2023; the investigation analyzes a pilot dataset described as June to August 2023), the bias re-emerged inverted: Dutch nationals and women were now more likely to be wrongly flagged — women about 22% more, in the investigation's analysis of the pilot data — an additional, unpublicized bias against applicants with children appeared, the system flagged more applications than the analog paper process rather than fewer, and it was no better than caseworkers at identifying genuine cases. An internal pre-pilot claim of a roughly 20% accuracy edge over caseworkers did not hold live. The city halted the project. The Dutch national algorithm register records the deployment ending in September 2023 and lists the system as "out of use"; the responsible alderman publicly announced termination in late November 2023, saying he could not have justified continuing a pilot that showed the algorithm contained substantial bias. All of the load-bearing figures here — the group disparities, the flag-volume increase, the precision of flags — are the investigating journalists' computations on aggregate confusion matrices the city itself provided under a GDPR-compliant remote-analysis regime, in which the city ran the journalists' code on real applicant data and returned aggregate results; they are not city publications, and the sign of the disparity inverted between the retrospective and live datasets.
Two things complicate the tidy reading of a fair-AI failure. First, the human baseline the system aimed to improve on was itself measurably flawed: caseworkers were more likely to wrongly flag Dutch nationals and women, and more than 50% of human-initiated investigations found no wrongdoing, against a fraud prevalence of roughly 3% of applications among about 35,000 welfare recipients (a city figure as of the end of 2024). Some project staff argued that reverting to the analog process was itself "a decision to go back to" a biased system, so whether the halt was a governance success or the abandonment of a fixable tool is a live disagreement, not a settled verdict. Second, the transparency was real and consequential: the investigation — by MIT Technology Review with Lighthouse Reports and Trouw, with Pulitzer Center support — obtained cooperative access to multiple model versions and the city's codebase, and in June 2025 released the full methodology, both model files (original and bias-reweighted), the bias-evaluation pipeline, and the city-provided confusion matrices on GitHub. A civil-society commentary from the Racism and Technology Center (labeled advocacy) reads the same events as the overriding of a citizen panel's clear negative advice and argues that responsible-AI tweaking missed the deeper question of whether such a system should be built at all. This is the transparent twin of the atlas's Rotterdam welfare-fraud case: same country and problem class, but Rotterdam is the covert, non-compliant deployment exposed from outside, while Amsterdam is the best-practice-compliant build that halted itself and published its own artifacts.
The sociotechnical reading
Almost every other case in this atlas is a story about a control that was missing, ignored, captured, or severed. Amsterdam is the opposite problem, and that is what makes it the atlas's cleanest counterexample to the idea that assurance is something you finish before launch. Nearly every recommended pre-deployment node was present and actuated: a bias audit that found real disparities, debiasing that measurably equalized wrongful-flag rates on retrospective data, a DPIA and a human-rights assessment, an external review, an academic consult, dual transparency registers, and a citizen panel. The debiasing was not theater — it genuinely worked on the data the team had. And the tool still had to be shut down. So the distinctive lesson is not the familiar "governance maturity is not evidence of accuracy," because here there WAS evidence of accuracy; it is sharper than that. The evidence was about the wrong distribution. Fairness is not a property you install into a model and then own; it is a relationship between a model and the live population it scores, and a pre-deployment audit can only verify that relationship against yesterday's sample. When the population the model actually met in the pilot differed from the retrospective one, the fairness did not transfer, and the disparity re-emerged wearing the opposite face — onto the majority group, women, and parents.
Read on the system map, the failure sits exactly where the map says it must. One screener scored every application, so its tilt was not case-by-case noise but a systematic property of the whole run — which is precisely why a three-humans-deep review pipeline, each human reading an individual file, could not see it. Per-case review is structurally blind to a population-level error geometry; that is not a lapse by any reviewer, it is a category mismatch between what the humans could inspect and where the harm lived. And the one node absent from an otherwise maximal stack is the one that watches that geometry: a standing live-bias monitor on a schedule, with the authority to halt. The pilot's bias evaluation was post-hoc and one-shot, and the inverted disparity was found not by the deployment but by outside investigators, months later. The map's other quiet finding is about the two governance edges that were present but did not bind. The citizen panel raised a reasoned objection — stop — and the deployment simply overrode it; a check a system can absorb without reckoning is a consultation, not a control. This is the atlas's evidence for the revoke lever, and it earns the label honestly, because the city did in the end pull it: the halt is a real, exercisable governance action, and Amsterdam exercised it in public and released its artifacts. But the last honest note is that the exit was governed, not free. Halting reverted to an analog human process that was itself biased and cleared more than half its investigations as fruitless, so "stop" bought time and accountability, not safety. The boundary the whole reading rests on: none of this measures the applicants themselves. What moved here was the fairness of an institution's own screening geometry against a shifting population — and whether any individual was wrongly denied assistance is documented in the record, not computed from anything on this map.
The concepts used in this reading are defined in the Field Guide; the governance responses live in the Practice Library. The model organization for this case can be stress-tested in the PAN Lab.