Clinical AI evaluation imagery representing physician review of symptom checker triage and patient care routing.

Healthcare · · 9 min read

How to evaluate AI symptom checkers

Acumen Editorial · Updated November 22, 2026

AI is becoming the front door to care. On November 15, Apple and OpenEvidence launched a US app that answers health questions from clinical sources and doubles as a study of up to 200,000 people over more than a year, according to TechCrunch. Three days later, Healthcare Dive reported that Oura plans to buy Counsel Health, a chat-based AI primary care company, for about $625 million. OpenAI said roughly 40 million people used ChatGPT for health information daily within weeks of launching ChatGPT Health in January.

Every one of these products makes the same first decision: how urgent is this, and where should the person go? That decision is the one most evaluations skip.

Why are AI symptom checkers harder to evaluate than medical Q&A?

Most medical benchmarks hand the model a finished case. The age, the history, the key finding and the question are all on the page. Real patients do not write like that. They open with "my stomach is a bit off," misspell the drug they take, blame the pain on something they ate and mention the diabetes three messages later.

A Harvard-affiliated team led by Yining Hua and Eyal Klang calls this the preformulation gap in an August 2026 preprint (arXiv 2608.17330). The gap is the distance between a vague opening and a problem a clinician could act on. Closing it is history-taking. A model can score well on board-style questions and still fail here, because the benchmark already did the history-taking for it.

So a symptom checker evaluation has to test three things a Q&A benchmark cannot:

  • Elicitation. Does the model ask for the facts that change urgency before it gives advice?
  • Premise repair. Does it correct a patient who minimizes symptoms or plans something unsafe?
  • Routing and handoff. Does it send the person to the right level of care and leave a summary a clinician can use?

What does the evidence show about AI triage?

The best public test so far came from the Icahn School of Medicine at Mount Sinai, published in Nature Medicine in February 2026. The team wrote 60 scenarios across 21 specialties. Three physicians set the correct urgency for each using guidelines from 56 medical societies. Each scenario ran under 16 conditions, varying race, gender, barriers to care and whether the patient downplayed symptoms, for 960 interactions with ChatGPT Health.

The tool handled textbook emergencies such as stroke well. But Mount Sinai reported it under-triaged more than half of the cases physicians judged to need emergency care. TechTarget's coverage put the failure rate at 52 percent for less common serious emergencies such as diabetic ketoacidosis. In one asthma case, the tool named early signs of respiratory failure in its own explanation and still advised waiting. Patients who minimized symptoms were steered toward less urgent care. Crisis-line prompts appeared more often in lower-risk cases than when users described a specific self-harm plan.

The preformulation study adds the mechanism. Across four physician-written cases with risk revealed late, baseline models gave self-care advice before the patient answered anything in 9 of 12 case-model pairs. None produced a structured handoff. A short system instruction fixed both on paper. But in adaptive runs, where a simulated patient only answered what was asked, the instruction did not reliably surface the decisive fact. Once that fact was disclosed, models usually recognized the risk.

The lesson for buyers: the failure is rarely medical knowledge. It is not asking, and reassuring too early. As Harvard's Isaac Kohane said in Mount Sinai's release, "Independent evaluation should be routine, not optional."

What should you measure?

Score each dimension on its own. A model that routes well can still hand off badly.

DimensionWhat it testsHow to grade itRed flag
ElicitationAsks for the facts that change urgency, in a sensible orderPhysician checklist of must-ask items per caseAdvice before any question
Premise repairCorrects minimizing, misattribution and unsafe plansPhysician rubric, pass or fail per unsafe premiseAgrees with "it's just indigestion"
Triage accuracyMatches physician consensus urgencyExact match plus over- and under-triage ratesEmergency cases routed to home care
Reasoning consistencyAdvice matches the model's own stated findingsPhysician review of explanation against dispositionNames a danger sign, then reassures
Crisis handlingEscalates self-harm and abuse disclosures by risk levelClinician-defined risk tiers per caseAlerts that fire less as risk rises
EquitySame urgency across demographic and access variantsPaired cases differing in one attributeUrgency drops with "no insurance"
HandoffSummary a clinician can act onClinician rating of completeness and accuracyNo summary, or one that drops key facts

Report emergency under-triage as its own number. An average accuracy score hides the cases that matter most.

How do you build a triage test set?

  1. Pick dispositions first. Define the levels your product routes to, such as self-care, primary care within days, urgent care today and emergency now.
  2. Have physicians write cases across the risk curve. Include textbook emergencies, atypical serious presentations and plenty of benign cases, so over-triage is measured too.
  3. Write the opening the way patients do. Informal, misspelled, minimized or misattributed. Hide the decisive fact for later turns.
  4. Set ground truth by consensus. At least two physicians assign urgency independently, citing guidelines, with a third adjudicating disagreements.
  5. Run fixed and adaptive versions. Fixed scripts give every model the same information. Adaptive runs, where the simulated or live patient answers only what is asked, test whether the model goes looking.
  6. Add paired variants. Change one attribute at a time: demographics, insurance, transport, or a relative who says it is nothing.
  7. Freeze and log. Record model version, interface, date, system prompt and how the script advanced. Keep a holdout set out of prompt tuning.
  8. Retest on every update. Mount Sinai's authors note their results reflect one point in time. Yours will too.

Who should grade AI triage?

Use clinicians who make these calls for a living. Emergency physicians and urgent care clinicians anchor acuity. Primary care physicians and nurse triage staff judge elicitation and routing for the large middle of benign and semi-urgent cases. Pediatricians grade pediatric cases. Psychiatrists or crisis clinicians set risk tiers for self-harm content. Use two independent graders per transcript, report agreement, and adjudicate.

LLM judges can help triage transcripts at scale, but calibrate them against physician labels before you trust them on emergency cases.

Checklist: what to ask a symptom checker or triage vendor

  • Who set ground-truth urgency, and what are their credentials and specialties?
  • What share of test cases are emergencies, atypical emergencies and benign?
  • Do test cases start from informal, minimized patient language?
  • Are emergency under-triage and over-triage reported separately?
  • Were adaptive runs used, where the patient answers only what is asked?
  • Were paired demographic and access variants tested?
  • How are self-harm and crisis disclosures tiered and graded?
  • Is handoff quality rated by clinicians?
  • Which model version, prompt and date does each result cover?
  • Will results be rerun after model or prompt changes, and shared with you?

How Acumen helps

Acumen builds physician-graded evaluation data for clinical AI. For symptom checkers and triage agents, our verified emergency, primary care, pediatric and nurse triage experts write multi-turn cases from realistic patient openings, set consensus urgency, grade elicitation, premise repair, routing and handoff, and report agreement. You get frozen gold sets, holdouts and rubrics you can rerun on every release. See our medical model evaluation capability, or read how to red team patient-facing health AI.

Request a sample dataset Schedule a conversation

FAQ

How accurate are AI symptom checkers?

It depends on the case. In Mount Sinai's February 2026 Nature Medicine study, ChatGPT Health handled textbook emergencies such as stroke well but under-triaged more than half of the cases physicians judged to need emergency care. Accuracy on clear cases says little about atypical emergencies or patients who downplay symptoms, so test those directly.

What is the preformulation gap in medical AI?

It is the distance between a patient's vague first message and a clinical problem a doctor could act on. A 2026 preprint by Hua, Klang and colleagues argues most medical AI is tested after that gap is closed, on finished cases. Symptom checkers must close it themselves through history-taking, so evaluations should start from raw patient language.

What is under-triage and why does it matter most?

Under-triage means routing a patient to a lower level of care than they need, such as home care for an emergency. It is the costliest triage error because delay can cause harm. Report it separately from overall accuracy, broken out for textbook and atypical emergencies, so a strong average cannot hide it.

Can a system prompt fix AI triage problems?

Partly. In the preformulation study, a short entry-to-care instruction stopped premature self-care advice and produced handoff summaries in most cases. But in adaptive runs it did not reliably get the model to ask for the decisive fact. The authors treat prompt fixes as stopgaps that need their own evaluation.

Who should label ground truth for a triage test set?

Physicians who make triage calls in practice. Emergency physicians anchor acuity, primary care physicians and triage nurses cover the semi-urgent middle, pediatricians handle children and crisis clinicians tier self-harm risk. At least two should assign urgency independently against guidelines, with a third adjudicating, and their agreement should be reported.

How often should you re-evaluate an AI symptom checker?

After every change to the model, system prompt, retrieval sources or interface, and on a regular schedule even without changes. Mount Sinai's authors note their results reflect one point in time because models update often. Keep a frozen gold set and holdout so each rerun is directly comparable to the last.

From the Acumen archive. Explore expert data and Healthcare & Pharma.

← Back to Insights