Behavioral-health & crisis triage

A national VA model flags the 0.1% of patients at highest suicide risk each month — a working, coordinator-mediated program that improved appointments and safety plans without reducing suicide deaths in two evaluations, and that a 2024 investigation found under-flagged women by leaving their risk factors out of the model.

A suicide-risk model that scores routine hospital records made clinicians far more likely to screen a flagged patient — but only when its alert was made interruptive rather than passive, in a trial too small to show a single prevented suicide.

An EHR-embedded machine-learning score that flags high suicide-risk patients within about 30 minutes of a virtual mental-health intake and routes them into the same assessment-and-outreach workflow as a positive self-report screen — a carefully co-designed integration whose main effect is to add a very-low-PPV sensor to an existing human screen, published as a feasibility design with no outcome-effectiveness evaluation.

A national crisis line built an in-house model to rank its most-at-risk texters and reorder who gets a counselor first — while the same anonymized conversations became a corpus that was routed to a for-profit it owned a stake in, to train commercial software. A 2022 exposé ended the data-sharing in three days; the deeper failure was that the review body that should have gated it was reportedly never asked, and that consent was gathered at the moment of crisis via a Terms of Service.

A proprietary overdose-risk score embedded in most US prescription-monitoring programs assigns patients secret 000-999 numbers that can gate their access to pain medication — never independently validated, not regulated by the FDA, uncontestable by the patient, and, when independent researchers rebuilt a version of it in 2026, far less accurate than the vendor claims.

A self-referral triage chatbot that genuinely widened access to NHS talking therapies — with the largest gains among under-served groups — while nearly all the evidence that it works was measured, published, and acted on by the company that built it.

A rule-based CBT chatbot used by roughly 1.5 million people was deliberately retired on a published schedule — with a transcript-access window and scheduled data anonymization — because its maker judged the FDA marketing-authorization economics unsustainable, not because the tool failed clinically. The Atlas's reference point for what a responsible wind-down looks like.

An AI quality-assurance tool that scores 988 crisis counselors on their own call practice — not the callers — expanding measured review from under 3% of calls toward nearly all of them. The domain's positive, governed counterexample: the AI sits on the supervision link over the operators, with strong peer-reviewed reliability evidence but a vendor-authored, still-unpublished trial of whether its feedback actually helps.

Gaggle Safety Management, an AI service, scans everything students write on school-issued accounts — email, homework, art, the student newspaper — around the clock, flagging anything that might signal self-harm. Used by roughly 1,500 US districts covering about 6 million students, it works as a multi-hop chain: a machine flags, an off-site vendor reviewer triages, a counselor gets an alert, and an after-hours emergency flag can send police to a home. In Vancouver, Washington nearly 1 in 10 students triggered alerts in a year; in Lawrence, Kansas about two-thirds of 1,200-plus flags were deemed nonissues — and the archive of flagged suicide notes and coming-out essays was accidentally released, nearly 3,500 unredacted documents at once. The atlas's discontinuation-that-wasn't case: when nine students sued, Lawrence quit the vendor mid-lawsuit and quietly swapped in an equivalent one with no board vote, and a federal judge found the district broke open-records law trying to withhold the swap's paperwork from the student journalists it had been flagging.

Camera-based, contact-free vital-signs monitoring installed in NHS psychiatric bedrooms. In March 2026 the health ombudsman partly upheld a complaint that a trust never sought a patient's consent, gave her no information, and would not switch the camera off when she asked — and found even the revised procedure still lets staff film a refusing, capacitous patient. The defining defect is vendor capture of the evaluation link: the company selling the tool authored the trust's business case and shaped the evidence meant to vouch for it, then rebranded mid-inquiry. Contested throughout; an ICO investigation is open.

ODMAP, a nationwide rule-based early-warning network, fires overdose spike alerts from aggregate, unnamed event counts rather than scoring any individual. Its algorithm is a plain threshold, not machine learning; the hard questions live off the model and on the links - the shared store sits inside a federal drug-enforcement program where public-health and law-enforcement read access coexist, the threshold learns from the store's own reporting, and every figure is program-self-published with no independent evaluation.

In November 2023 New York City launched NYC Teenspace, buying population access to a commercial teletherapy platform for every resident aged 13 to 17 — a $26M three-year contract, free at the point of use, with parental consent and a New York-licensed therapist. Inside the therapy chat runs a proprietary NLP model that scans teen-authored messages for suicide and self-harm language and fires an urgent alert to the teen's own treating clinician; the algorithm never acts on its own, and in the first six months it flagged roughly 50 teens (under 1% of users) while therapists handled 36 high-risk events. Two things about the oversight record define this case. The model has never been examined by anyone outside the vendor: its one published accuracy figure (83% agreement with a human expert) is a vendor claim from a 2020 study of ADULT platform data, and the independent evaluation the health department said in September 2024 it was planning had still not appeared by mid-2026. Meanwhile the surface anyone COULD inspect — the public sign-up pages — was inspected, by a parent-privacy coalition running a free tracker audit tool: 15 ad trackers and 34 cookies sending teen visitors' identifiers to Meta, Amazon, Google, Microsoft and others, several of them companies the city was simultaneously suing over teen mental-health harm. The vendor stripped the trackers on December 11, 2024 and cut the sign-up form to age and zip code, and a week after that the city said the contract's Data Security Rider would be amended to bar marketing and advertising use of teen data outright — and within four weeks advocates documented trackers again on pages beyond the landing page, including the page hosting the revised privacy policy. The audited surface and the algorithmic surface never met.

Between October 2024 and November 2025, Character.AI built its entire crisis-safety stack in four dated increments, each within days to weeks of a specific lawsuit, court ruling, state investigation, federal compulsory order, or published investigation: a self-harm phrase screen raising a 988 referral (blog dated the same day the first wrongful-death complaint was filed), a separate under-18 model, a weekly parental usage summary, and — one week after an investigation found dozens of harmful personas live and seven weeks after FTC 6(b) orders reached seven companies — the removal of open-ended chat for its roughly two million under-18 users. The one independent measurement of the screen found it firing three times across sixteen adversarial conversations, on two exact phrasings, dismissable and non-blocking, with firing frequency rising after the testers contacted the company. Rather than establish that a detector worked, the operator deleted the function; on 7 January 2026 it, both founders, and Google agreed to settle five suits, terms undisclosed and causation never adjudicated.

A national nonprofit told its entire paid helpline staff they were terminated four days after their union election was certified, and said a chatbot would take over a service that had fielded nearly 70,000 contacts in a year. The chatbot was suspended two days before the handover after testers published screenshots of it giving dieting advice — advice the nonprofit and the research team attribute to a generative feature the operating vendor had added and the nonprofit says it never approved, while the vendor said only that it was still trying to determine how a closed system had delivered such content. The helpline closed on schedule anyway.

A national health system scores every patient with an opioid prescription for one-year overdose and suicide risk, refreshes the score nightly, and since April 2018 has required interdisciplinary case review of whoever lands in the top percentile at all 140 of its medical centers - and then did what almost no deployment does: it randomized its own governance lever. Two experiments ran at once across those centers, one on when the mandated tier widened from the top 1% to the top 5%, one on whether the policy notice carried an accountability paragraph. The mandate made flagged patients 5.1 times more likely to be reviewed. The accountability paragraph produced fewer reviews than the plain mandate, not more.