Clinical research imagery representing independent clinician review of mental health AI chatbot safety and multi-turn evaluation scenarios.

Healthcare · · 7 min read

How to evaluate a mental health AI chatbot

Acumen Editorial · Updated November 21, 2026

Mental health is where conversational AI meets its highest stakes. People already bring distress, crisis and medication questions to chatbots. Regulators have noticed, and the evidence bar is about to rise.

For product, clinical and trust leads building mental health AI, the question is no longer whether to evaluate. It is whether your evaluation would hold up in front of clinicians and the FDA.

Why is FDA's new guidance priority a signal for mental health AI teams?

On October 1, 2026, FDA published its fiscal 2027 device guidance priorities. Healthcare Dive reports that one top-tier item is a new draft on evidentiary recommendations for generative AI conversational devices intended for mental disorders. It was not on last year's list. Comments on the priorities list are due November 30.

The direction is already visible. At FDA's Digital Health Advisory Committee meeting on generative AI mental health devices in November 2025, Hogan Lovells reports, discussion covered outcomes such as conversational error rates, safety signals, fidelity to therapeutic content with clinician review of transcripts, and empathy and engagement. Panelists also raised red teaming, escalation pathways, postmarket drift monitoring and rollback criteria.

The profession is moving the same way. In November 2026 the American Medical Association adopted policy that AI cannot be licensed to practice medicine and said it will push for physician oversight of AI clinical decisions, per Fierce Healthcare. Illinois and Nevada have already restricted AI therapy outside licensed professionals' use.

Put together: expect to show clinician-reviewed evidence of how your chatbot behaves in real conversations, and keep showing it after launch.

Why do single-turn safety tests miss the real failures?

Because the risk shows up over the course of a conversation. Two new studies make that concrete.

SAFE-Teen (Mock et al., arXiv, November 6, 2026) ran 32,555 simulated conversations with adolescents aged 13 to 17 across 18 chatbot APIs. It scored 22 safety criteria over 983 variants of 106 seed scenarios.

  • Without a teen-aware system prompt, chatbots failed 43.3% of criteria. With one, 24.9%.
  • When the simulated teen pushed back, failures on diagnosis, medication and substance misuse boundaries rose 10 to 12 percentage points without the prompt. That held for more than 80% of chatbots.

A second audit (Nepal, Heo, Parrish and Depp, arXiv, October 15, 2026) ran 2,550 scripted conversations that opened with a disclosure of suicidal thoughts. Users then declined outside help up to 10 times.

  • 99.7% of conversations referred the user to help in the first reply. A single-turn test would score that as a pass.
  • How long chatbots kept recommending help ranged from almost no refusals to nearly all 10. Newer versions of all five chatbots persisted less than older ones.
  • Only 2.0% of 8,818 replies urging outside help asked what was stopping the user from seeking it.

The first reply is the easy part. Pushback, refusal and drift across turns are where chatbots fail, and they are what your test set has to contain.

Why can't you just average clinician ratings?

Because clinicians disagree, and the disagreement is real. In a Stanford study reported by Stanford HAI in July 2026, three board-certified psychiatrists rated 360 AI responses to synthetic mental health prompts. Over 100 psychiatrists polled at the American Psychiatric Association's 2026 meeting split almost evenly on safety, empathy and correctness.

The authors found the disagreement comes from different clinical training and frameworks, not noise. Averaging it can reward responses no expert would choose. Lead author Kiana Jafari put it simply: "Preserve the disagreement. Don't average it away."

For buyers, that changes the design. Record which framework each rater works in. Report agreement for every item. Send high-disagreement items, especially suicidality, psychosis and eating disorders, to adjudication rather than taking the mean.

Can an LLM judge grade mental health safety?

Only after you check it against clinicians. In an October 2026 preprint, Lyu and colleagues compared an LLM judge with a rule-based guardrail whose four safety checks two clinicians defined and adjudicated. On its main prompt, the judge demanded a crisis resource in 55 of 65 replies that already contained one. Asked to extract facts instead, the same model found the resource in 54 of those 55.

A judge that misses what is plainly on the page will mis-score your model at scale. Use LLM judges for volume, calibrate them on a clinician-graded subset and recheck that calibration whenever the judge or the product changes.

What should a mental health AI chatbot evaluation test?

Score each layer separately. Passing one says little about the next.

LayerWhat it catchesHow to score it
Crisis detection and first responseMissed or delayed risk recognitionClinician-written disclosures at graded severity, scored on the first reply
Multi-turn persistenceDropping safety after pushback or refusalScripted refusal sequences; count turns until safety behavior lapses
Scope and boundariesDiagnosis, medication or substance advice it shouldn't giveBoundary scenarios with pushback, graded against your intended use
Therapeutic fidelityDrift from the intended approachClinician transcript review against the protocol your product claims
Population fitFailures for teens or other higher-risk usersPopulation-specific scenario sets, reported separately
Rater reliabilityFalse certainty from averaged scoresAgreement statistics per item; adjudication for disagreements
Judge validityAutomated graders that mis-scoreJudge vs clinician agreement on a held-out subset

Checklist for building the test set

  • Define intended use first: wellness support, adjunct to care or standalone. The scenarios follow from it.
  • Recruit licensed clinicians from the settings you serve: psychiatry, psychology, counseling, crisis lines, adolescent care.
  • Have them write multi-turn scenarios, not prompts, with planned pushback, refusal and topic changes.
  • Cover severity levels for suicidality, self-harm, psychosis, eating disorders and substance use.
  • Write reference behaviors and rubrics per scenario, tagged by clinical framework.
  • Double-grade every high-risk item and log disagreements instead of averaging them.
  • Calibrate any LLM judge against the clinician subset before scaling.
  • Keep the set private and rerun it on every model, prompt or safety-layer change.
  • Log versions, transcripts and grades so the evaluation supports a regulatory file and postmarket monitoring.

What to ask a mental health AI evaluation vendor

  1. Are your graders licensed mental health clinicians, and how do you verify that?
  2. Do your scenarios include pushback, refusal and escalation over many turns?
  3. How do you handle rater disagreement? Do we see agreement for each item?
  4. Can you build population-specific sets, such as adolescents?
  5. If you use an LLM judge, what is its measured agreement with clinicians?
  6. Can we own the set and rerun it ourselves after each release?

A vendor that reports one averaged safety score has not answered these.

How Acumen helps

We build private evaluation sets for mental health AI. Licensed clinicians write multi-turn scenarios with realistic pushback, set reference behaviors and rubrics by clinical framework, double-grade every high-risk item and adjudicate disagreements. We calibrate automated judges against their grades and document provenance for regulatory and governance review.

See our medical model evaluation service, or read how to red team patient-facing health AI.

Request a sample dataset Schedule a conversation

FAQ

What is the best way to evaluate a mental health AI chatbot?

Test full multi-turn conversations written by licensed mental health clinicians, including users who push back, decline help or escalate. Score crisis response, persistence, scope, therapeutic fidelity and population fit separately. Keep rater disagreement visible, validate any LLM judge against clinicians, and rerun the private test set after every model, prompt or safety-layer change.

Is FDA issuing guidance on mental health AI chatbots?

FDA listed a draft on evidentiary recommendations for generative AI conversational devices intended for mental disorders among its top fiscal 2027 device guidance priorities, published October 1, 2026, per Healthcare Dive. Its November 2025 advisory committee meeting discussed conversational error rates, clinician transcript review, red teaming and postmarket monitoring.

Why are single-turn safety tests not enough?

Most chatbots handle the first reply well. In an October 2026 audit of 2,550 conversations, 99.7% referred users to help in the first reply, but persistence after users declined varied widely. In SAFE-Teen, pushback raised boundary failures by 10 to 12 percentage points. The risk sits in later turns, so tests must reach them.

How should clinician disagreement be handled in grading?

Keep it, don't average it. A Stanford study reported in July 2026 found psychiatrists split almost evenly on the safety, empathy and correctness of AI responses, driven by different clinical frameworks. Record each rater's framework, report agreement per item and send high-disagreement, high-risk items such as suicidality or psychosis to adjudication.

Can an LLM judge replace clinician graders?

Not on its own. In an October 2026 preprint, an LLM judge demanded a crisis resource in 55 of 65 replies that already contained one, while the same model found the resource when asked to extract facts. Use judges for volume only after calibrating them against a clinician-graded subset, and recheck that calibration regularly.

Who should write and grade mental health AI test scenarios?

Licensed clinicians from the settings your product serves: psychiatrists, psychologists, counselors, crisis-line staff and adolescent specialists for teen users. They should write multi-turn scenarios with planned pushback, define reference behaviors and rubrics by clinical framework, and double-grade every high-risk item. Verified credentials and documented provenance matter for regulatory and governance review.

From the Acumen archive. Explore expert data and Healthcare & Pharma.

← Back to Insights