Clinical AI safety · · 8 min read
How to red team patient-facing health AI: a clinician-led checklist
Acumen Editorial · Updated November 20, 2026
Patient-facing AI now does intake, triage and symptom guidance before a clinician ever sees the case. That is where the stakes are highest and the testing is often thinnest. A benchmark score tells you how a model answers exam questions. It does not tell you what happens when a worried patient leaves out the one detail that matters.
On November 16, 2026, JMIR AI published the clearest answer yet to how a health system should test these tools. An academic medical center red-teamed a conversational intake and triage platform with practicing clinicians, fixed what they found and tested again. The method is reproducible. The numbers are useful for any buyer writing a pre-deployment plan.
If you own clinical AI safety, governance or product at a health system or a health AI company, here is how we would run it.
What is red teaming for patient-facing health AI?
Red teaming is structured adversarial testing. Clinicians play patients and try to make the system fail: miss an emergency, give unsafe advice, overstep its role or reassure when it should escalate.
It differs from evaluation in one way. Evaluation asks how often a model is right on a representative set. Red teaming asks where it breaks and how badly. For a patient-facing tool, the worst case matters more than the average.
What did the new JMIR AI study find?
The study ran in two phases, with the evaluation team firewalled from the software developers.
| Measure | Phase 1 | Phase 2 (after fixes) |
|---|---|---|
| Red teamers | 48 practicing healthcare professionals | 27 new clinicians |
| Scenarios | 6 per person: 3 medium-risk, 3 high-risk | 8 high-risk |
| High-risk failure rate | 30.2% (116 transcripts) | 8.5% (234 transcripts) |
| Medium-risk failure rate | 5.6% (126 transcripts) | Not retested |
| Rater agreement (Fleiss κ) | 0.70 | 0.70 |
Between phases, independent raters scored transcripts on a five-domain safety rubric. Findings went to engineers as a de-identified remediation brief. They added behavioral guardrails, revised clinical prompts and adjusted context handling. High-risk failures fell 21.7 percentage points, a 72% relative reduction.
Three details matter for buyers. Gains held on both scripted vignettes and prompts clinicians improvised. Remaining failures clustered around triage calibration and boundary violations. And the authors still call for continuous monitoring and clinician oversight after launch.
Why don't benchmarks catch these failures?
Because patient-facing failures live in conversations, not in single answers.
A Glass Box Medicine-led study in npj Digital Medicine had physicians review 888 chatbot answers to 222 patient-style primary care questions. Problematic responses ranged from 21.6% to 43.2% by model, and unsafe responses from 5.0% to 13.5%. The authors warned that "millions of patients could be receiving unsafe medical advice from publicly available chatbots."
A John Snow Labs red teaming framework found the same pattern from another angle. Across 11 models and 690 scenarios, mean scores looked high, yet several models scored zero on at least one scenario. Changing patient demographics amplified errors on equity tasks by 10 to 20%. The automated judge also missed failures clinicians caught, such as empathetic wording that masked a missing safety action.
Averages hide the cases that hurt patients. Red teaming is built to find them.
How should you red team a patient-facing health AI tool?
Use this checklist before launch and after every material change.
- Define the intended use and the harm. Write down what the tool may do (intake, triage, education) and what it must never do (diagnose, prescribe, delay emergency care).
- Build a scenario taxonomy by risk tier. Cover emergencies, ambiguous symptoms, missing context, conflicting details, vulnerable groups and out-of-scope requests. Weight the set toward high-risk cases.
- Recruit practicing clinicians, matched by specialty. Emergency medicine and nursing for triage, pediatrics for pediatric flows, behavioral health for mental health content.
- Mix scripted and improvised prompts. Scripted vignettes make results comparable. Improvised ones find what your script missed.
- Vary the patient, not just the question. Swap age, sex, language and health literacy to probe bias.
- Score on a fixed rubric with independent raters. Use at least two raters per transcript and report chance-corrected agreement, not raw agreement.
- Grade severity, not just pass or fail. A missed emergency and an awkward phrase are not the same finding.
- Firewall the testers from the builders. Engineers get a de-identified remediation brief, not live access to testers.
- Retest with new clinicians. Fresh testers stop the team fixing only the cases it has seen.
- Validate the exact configuration you ship. Model version, system prompt and reasoning settings all change behavior.
- Keep the findings. Every failure becomes a regression case, a training example or a monitoring check.
The configuration point is easy to skip. K-Bench, a University of Roehampton-led benchmark for high-risk mental health conversations, tested 125 model configurations. A therapeutic system prompt moved scores by just +0.11 points on average across 61 matched pairs. Higher reasoning settings lowered them by 0.58 points across 30 pairs. The authors concluded that prompt and reasoning effects must be validated for the exact model in use.
How many clinicians and scenarios do you need?
The JMIR AI study is a useful reference point: 48 clinicians in the first round, 27 new clinicians in the second, and 476 scored transcripts in total. Raters reached a Fleiss κ of 0.70 in both rounds.
Your numbers depend on scope. A narrow intake bot needs fewer scenarios than a general symptom assistant. We would size the set by three things: the number of risk tiers, the specialties the tool touches and the patient groups it serves. Then add enough transcripts per cell to compare before and after with confidence.
What should you ask a red teaming vendor?
1. Are your red teamers practicing clinicians, and how are credentials verified?
2. How do you match clinicians to the specialties our tool touches?
3. Which safety rubric do you use, and can we see it before we sign?
4. How many raters score each transcript, and what agreement do you report?
5. Do testers improvise, or only run scripts?
6. How do you separate testers from our engineers?
7. Do findings come back as data we can train on and rerun, or only as a report?
For the wider evaluation plan, see our guide on how to evaluate a clinical LLM.
How Acumen helps
We run the clinician side of this loop. Our healthcare AI red teaming service supplies credential-verified, specialty-matched clinicians, a scenario taxonomy built around your intended use, severity-graded findings and a remediation brief your engineers can act on.
Every finding also comes back as data: adversarial cases, reference responses and regression sets you can rerun on each release. Teams use it for pre-deployment audits and post-launch monitoring. Each engagement covers specialties and scored transcripts, documented for your governance file.
FAQ
What is red teaming for patient-facing health AI?
It is structured adversarial testing in which clinicians play patients and try to make the AI fail. They probe for missed emergencies, unsafe advice, boundary violations and bias. Unlike standard evaluation, which measures average accuracy, red teaming looks for the worst failures and grades them by severity so engineers can fix them before patients see them.
How effective is clinician red teaming?
A two-phase study published in JMIR AI on November 16, 2026 is the best current evidence. Clinician red teaming plus targeted engineering fixes cut high-risk failures in a patient-facing intake and triage platform from 30.2% to 8.5%, a 72% relative reduction. Gains held on improvised prompts, though some triage and boundary failures remained.
Who should do the red teaming?
Practicing clinicians matched to the tool's scope. The JMIR AI study used 48 healthcare professionals in its first round and 27 new clinicians in its second. Emergency medicine and nursing fit triage tools, pediatrics fits pediatric flows and behavioral health fits mental health content. Keep the testers separate from the engineers who build the fixes.
Can automated judges replace clinician red teamers?
Not on their own. A John Snow Labs framework used an automated judge with human review and found the judge missed failures clinicians caught, including empathetic replies that skipped a needed safety action. Automated scoring is useful for scale and regression checks. High-risk items and disagreements should still go to clinicians for final grading.
How often should you red team a patient-facing AI tool?
Before launch and after every material change: a new model version, a new system prompt, a new reasoning setting or a new use case. K-Bench found prompt and reasoning effects differ by model configuration, so a change that helps one setup can hurt another. Between rounds, rerun past findings as a regression set.
What should a red teaming report include?
The scenario taxonomy and risk tiers, the rubric, who tested and their specialties, failure rates by tier, rater agreement, severity-graded findings with transcripts, and a remediation brief. It should also hand back the failing cases as reusable data, so each fix can be verified and the same failure caught automatically on every later release.