Clinical evaluation imagery representing physician review of ambient AI scribe notes.

Clinical AI evaluation · · 7 min read

How to evaluate an ambient AI scribe

Acumen Editorial · Updated November 21, 2026

Health systems keep buying ambient scribes. This week brought two more deployments. On November 19, 2026, Health IT Answers reported a multiyear deal to deploy Concurrence's clinical documentation copilot across Thriveworks' behavioral health network, and general availability of Greenway Health's Novare agents for documentation and coding in ambulatory practices.

The evidence behind these tools is better than it was a year ago. It is also contradictory. Randomized trials report less time in notes and less exhaustion. A controlled comparison of 11 scribe tools found human notes scored higher in every case. Both can be true, which is why a vendor demo or a published study cannot stand in for your own evaluation.

If you own clinical AI, informatics or procurement at a health system, here is how we would test a scribe before scaling it.

What does the evidence say about ambient AI scribes?

The strongest studies point in different directions because they measure different things.

StudyDesignWhat it found
UCLA trial, NEJM AI (Nov 2025)Randomized, 238 outpatient physicians, 14 specialties, two scribes vs. usual careOne scribe cut time-in-note 9.5% vs. control; the other's 1.7% drop was not significant. Clinically significant inaccuracies were reported "occasionally."
Afshar et al., NEJM AI (Dec 2025)Stepped-wedge randomized, 66 practitioners, 71,487 notes0.36 fewer hours per day on notes; work exhaustion fell; documentation quality scores stayed high (3.97–4.99 of 5)
VA-led study, Annals of Internal Medicine (Apr 2026)5 standardized primary care visits, 11 AI scribes vs. 18 human clinicians, 30 blinded ratersHuman notes scored higher in every case, especially on thoroughness, organization and usefulness
Pilot, JMIR Medical Informatics (Apr 2026)31 physicians, 7,545 AI-drafted notes, 356 physician-reviewedOmissions in 18% of reviewed notes, hallucinations in 11.5%, accidental inclusions in 9.3%

The real-world trials measure what happens when a physician reviews and edits the draft. The controlled study measures the draft itself. Your evaluation needs both views: what the tool produces, and what your clinicians do with it.

The financial case is still open. The Peterson Health Technology Institute reviewed eight health systems and 10 vendors in March 2025. It found burnout gains but no demonstrated financial return, and it called for standard industry metrics. Those metrics still do not exist. You have to set your own.

What errors do ambient AI scribes make?

Three kinds, and they need to be counted separately.

  • Omissions. Something said in the visit never reaches the note. In the JMIR Medical Informatics pilot, accidental omissions were the most frequent error type.
  • Hallucinations. The note states something that was never said or done.
  • Distortions. The fact is there, but wrong: a dose, a side, a duration, who said it.

Omissions are the easiest to miss in review. A wrong statement stands out on the page. A missing one does not. A physician skimming a clean, well-organized draft has no prompt to notice the allergy, the red-flag symptom or the chronic condition that was discussed and dropped.

Omissions also have a source worth tracing. In a Cape Town neurosurgery preprint, transcription failures explained 10 of the 15 omissions in AI notes. The preprint is not peer reviewed and a co-author works for the scribe developer, so we treat it as a direction, not a benchmark. The lesson holds anyway: test the audio pipeline, not just the language model.

Why can't you rely on vendor studies?

Because the conditions that matter most are local.

Specialty changes the note. A neurosurgery consult, a behavioral health session and a pediatric well visit need different content, structure and level of detail. A score from primary care does not transfer to psychiatry.

Setting changes the audio. Exam room noise, interpreters, family members, accents and telehealth connections all affect transcription, and transcription errors flow into the note.

Configuration changes the output. Templates, the underlying model and EHR integration differ by deployment. The Afshar trial reported no drift in software performance during its 24 weeks. That is a finding about one tool in one study, not a guarantee for yours.

How should you evaluate an ambient AI scribe?

Use this checklist before scaling and after every material change.

  • Define the note you expect. Write a reference standard per specialty: required sections, required elements and what must never appear.
  • Sample real visits across your mix. Cover each specialty, visit type, language and setting you will deploy to, including telehealth.
  • Grade against the audio, not the draft. Reviewers need the recording or a verified transcript. Grading the note alone hides omissions.
  • Use physicians from the same specialty. A behavioral health note should be graded by behavioral health clinicians.
  • Count error types separately. Report omissions, hallucinations and distortions as distinct rates.
  • Grade severity. A missed medication change and a misplaced social history line are not the same finding. Use a fixed scale.
  • Use at least two independent graders. Report chance-corrected agreement and adjudicate disagreements with a third senior reviewer.
  • Use a validated quality instrument. The VA-led study used a modified PDQI-9. The Afshar trial used the PDSQI-9. Either gives you a comparable baseline.
  • Measure the edited note too. Track what clinicians change before signing. Heavy edits signal hidden cost.
  • Run a controlled comparison for time. Use EHR log data against a control group, not self-report alone.
  • Check coding output. If the scribe suggests codes, grade them against certified coders.
  • Keep the graded set. Every scored note becomes a regression case for the next release.

For the wider method, see our guide on how to evaluate a clinical LLM.

How many notes do you need to review?

Enough to see the errors that matter in each specialty you deploy to.

The JMIR Medical Informatics pilot reviewed 356 of 7,545 notes, about 4.7%. The Cape Town preprint scored 49 encounters with two consultant raters and a third adjudicator, and reached a weighted kappa of 0.814. Its authors still called it exploratory and too small for rare safety outcomes.

We size a review by three things: the number of specialties, the visit types within each and the severity you most need to catch. Rare, severe errors need larger samples. Add notes per cell until before-and-after comparisons stop moving.

What should you ask an ambient AI scribe vendor?

1. Which specialties and visit types were in your evaluation, and do they match ours?

2. Were notes graded against the audio or against the draft alone?

3. Who graded them, how were their credentials verified, and what agreement did they reach?

4. Do you report omission, hallucination and distortion rates separately?

5. How do you grade severity?

6. What changes when you update the model, and how do you tell us?

7. Can we run our own graded evaluation on our visits before we sign?

How Acumen helps

We run the physician side of scribe evaluation. Our medical model evaluation service supplies credential-verified, specialty-matched clinicians who grade draft notes against the visit audio, classify omissions, hallucinations and distortions, grade severity and adjudicate disagreements.

You get the scores and the data: reference notes, graded error sets and regression cases you can rerun on every vendor release. Teams use it for vendor selection, pre-scale validation and post-launch monitoring. Each engagement is documented for your governance file.

Request a sample dataset Schedule a conversation

FAQ

How do you evaluate an ambient AI scribe?

Have physicians from the specialties you deploy to grade real draft notes against the visit audio. Count omissions, hallucinations and distortions separately, grade each by clinical severity and use two independent graders with adjudication. Measure time-in-note with EHR log data against a control group, and repeat the evaluation after every model or template change.

Do ambient AI scribes save physicians time?

Randomized evidence says yes, modestly and not for every tool. In a UCLA trial of 238 physicians published in NEJM AI, one scribe cut time-in-note 9.5% versus control, while the other's 1.7% drop was not significant. A stepped-wedge trial of 66 practitioners reported 0.36 fewer hours per day on notes.

How accurate are AI scribe notes?

It depends on the tool and the test. In a JMIR Medical Informatics pilot, physicians found omissions in 18% of 356 reviewed notes and hallucinations in 11.5%. A VA-led Annals of Internal Medicine study found human-written notes outscored 11 AI scribes in every case on standardized primary care visits. Accuracy must be checked locally.

What is the most common AI scribe error?

Omissions. In the JMIR Medical Informatics pilot, accidental omissions were the most frequent error type. They are also the hardest to catch, because a clean, well-organized draft gives the reviewing physician no signal that something said in the visit is missing. Grading against the audio, not the draft alone, is the only reliable check.

Who should grade AI scribe notes?

Practicing physicians from the same specialty as the notes, working from the visit audio or a verified transcript. Use at least two independent graders, report chance-corrected agreement and send disagreements to a senior adjudicator. A validated instrument such as a modified PDQI-9 gives a comparable baseline across vendors and releases.

How often should you re-evaluate an ambient AI scribe?

Before scaling, then after every material change: a new underlying model, new templates, a new specialty or a new EHR integration. Between those points, rerun your graded notes as a regression set. One NEJM AI trial saw no performance drift over 24 weeks, but that finding covers one tool in one study, not every deployment.

From the Acumen archive. Explore expert data and Healthcare & Pharma.

← Back to Insights