Healthcare · · 8 min read
How to evaluate AI radiology report generation
Acumen Editorial · Updated November 21, 2026
Report generation is the most visible use of foundation models in radiology. Draft reports are already in prospective testing at some institutions. The question for imaging AI leads, health-system clinical AI teams and model builders is no longer whether a model can write a fluent report. It is whether your evaluation can find the errors that fluency hides.
Where does AI radiology report generation stand today?
It is useful as a drafting assistant and not ready to work alone. That is the conclusion of a narrative analysis by Yee Seng Ng and Elise Dunning of the University of Washington, published November 6, 2026 in Current Problems in Diagnostic Radiology and reported by AuntMinnie. The authors write that the models' "strongest near-term role is as assistants that draft and accelerate work."
Maturity varies by modality. Chest radiography is furthest along, helped by public datasets such as MIMIC-CXR, which has more than 370,000 image-report pairs. CT report models have had limited clinical validation. MRI work centers on brain imaging and is mostly not built for report generation. Ultrasound and mammography lag.
The authors cite an AI challenge at the 2025 American College of Radiology annual meeting. Attending radiologists, blinded to authorship, judged chest radiograph reports. They rated 65.4% of AI-generated reports clinically acceptable, against 79.6% of radiologist-written reports. That gap is the space your evaluation has to measure.
What errors does an AI radiology report evaluation need to catch?
The dangerous errors read well. The UW authors list hallucinated findings, omitted findings, wrong laterality and fabricated comparisons to prior studies, all delivered in confident clinical language. They also flag training-data bias and automation bias as barriers to safe use.
Each of these needs its own test. A model that never invents a nodule can still drop a small effusion. A model with perfect findings can still say "unchanged from prior" when no prior exists. A single overall score hides which failure you have.
The draft also changes the reader. A radiologist editing a fluent report reads differently from one dictating from scratch. That is where automation bias enters. Your evaluation should test the pair, not just the model: give radiologists drafts with seeded errors and measure how many they catch before sign-off. A model with a low error rate can still be unsafe if its errors are the ones readers stop looking for.
Why do text-overlap metrics fall short?
Because a report can match the reference closely and still be clinically wrong. Swap "left" for "right" and most of the words stay the same. Drop "no" before "pneumothorax" and the overlap barely moves.
The reference report is not ground truth either. FDA is funding research on exactly this problem. In September 2026, AuntMinnie and Radiology Business reported a $1.29 million FDA contract, under its Broad Agency Announcement program for regulatory science, to Cognita Imaging, a subsidiary of Mosaic Clinical Technologies, owned by Radiology Partners. The project, "Virtual Subject Matter Expert Agents for Radiology Report Evaluation," builds on GREEN, an open-source evaluator designed to identify and explain clinically significant errors.
Its design is instructive. Several LLMs act as a jury and their judgments are compared rather than trusting one evaluator. Radiologists adjudicate clinically important disagreements. The team treats disagreement as a question, not a verdict: the error could sit in the generated report, the AI evaluator or the original radiologist's report. It also rebuilds smaller validation cohorts from about one million examinations to see what smaller studies miss. The award funds evaluation methods only. It does not clear a product or set FDA standards.
How should you measure workflow value?
Pair accuracy with time and edits, in the real workflow. The strongest prospective evidence so far comes from Northwestern Medicine, published in JAMA Network Open in June 2025. Across a 12-hospital system, a generative model drafted reports for plain radiographs inside the live reporting workflow.
- On 23,960 radiographs, mean documentation time fell from 189.2 to 159.8 seconds, a 15.5% gain.
- In peer review of 800 studies, clinical accuracy and textual quality showed no significant difference between model-assisted and unassisted reports.
- Addenda rates were flat: 0.13% before and 0.14% with the model.
- A shadow pneumothorax flag reached 72.7% sensitivity and 99.9% specificity on retrospective review.
The authors note limits: a nonrandomized design, one health system and plain radiographs only. That is the point for buyers. A result on chest X-rays in one system tells you little about CT at yours.
What should an AI radiology report evaluation test?
Score each layer separately.
| Layer | What it catches | How to score it |
|---|---|---|
| Finding accuracy | Missed and invented findings | Radiologists grade drafts against the images, by finding |
| Laterality and location | Wrong side, level or segment | Targeted cases with lateralized and multifocal findings |
| Comparison claims | Fabricated or wrong prior comparisons | Cases with and without priors; verify every comparison statement |
| Negation and certainty | Dropped "no," overstated or hedged findings | Paired cases that differ only by presence of a finding |
| Critical findings | Delayed or missed urgent results | Enriched set of urgent findings with time-to-flag |
| Edit burden and time | Drafts that cost more to fix than to write | Edit distance, time per study and addenda in live or simulated reading |
| Grader validity | Automated scores that disagree with radiologists | Judge or jury vs adjudicated radiologist grades on a held-out subset |
| Modality and population fit | Results that don't transfer | Separate sets per modality, scanner, site and patient group |
Checklist for building the test set
- Define intended use: draft for radiologist sign-off, preliminary read or triage. The pass bar follows from it.
- Recruit board-certified radiologists with subspecialty coverage that matches your case mix.
- Grade against images, with the original report as one input, not the answer key.
- Use a written error taxonomy with severity levels for missed, invented, lateralized, comparison and negation errors.
- Enrich for critical and rare findings. Random sampling under-tests them.
- Double-read high-severity items and adjudicate disagreements.
- Calibrate any automated grader on the adjudicated subset before scaling it.
- Build separate sets for each modality and site. Do not carry chest X-ray results to CT.
- Keep the set private and rerun it on every model, prompt or template change.
- Log versions, grades and adjudications so the evaluation supports governance and regulatory review.
What to ask a radiology AI evaluation vendor
- Are your graders board-certified radiologists, and in which subspecialties?
- Do they read the images, or only compare text to the original report?
- Which error taxonomy do you use, and how is severity assigned?
- How do you adjudicate disagreement between radiologists, or between radiologists and an automated grader?
- What is your automated grader's measured agreement with radiologists?
- Can you build modality-specific and site-specific sets, and can we rerun them ourselves?
A vendor that reports one overall similarity score has not answered these.
How Acumen helps
We build private evaluation sets for AI radiology report generation. Verified radiologists grade drafts against the images using a written error taxonomy, double-read high-severity cases and adjudicate disagreements. We enrich for critical findings, split sets by modality and site, calibrate automated graders against radiologist grades and document provenance for governance review.
See our medical model evaluation service, or read our guide to LLM-as-a-judge for medical AI.
FAQ
How do you evaluate AI-generated radiology reports?
Have radiologists grade AI drafts against the images, not only against the original report. Score clinically significant errors by type, including missed findings, invented findings, wrong laterality and fabricated prior comparisons. Measure time and edit burden alongside accuracy, calibrate any automated grader on adjudicated cases, and build separate test sets for each modality and site.
How accurate are AI-generated radiology reports today?
It depends on modality. At a 2025 American College of Radiology challenge cited by University of Washington authors, blinded attending radiologists rated 65.4% of AI chest radiograph reports clinically acceptable versus 79.6% of radiologist-written ones. Chest radiography is the most mature area. CT, MRI, ultrasound and mammography report generation have had limited clinical validation.
What errors are most common in AI radiology reports?
University of Washington authors writing in November 2026 list hallucinated findings, omitted findings, wrong laterality and fabricated comparisons to prior studies. These errors are dangerous because the surrounding language is fluent and confident. An evaluation should test each error type with targeted cases and score them separately rather than folding them into one overall number.
Are text-similarity metrics enough to grade report generation?
No. A report can share most of its words with the reference and still be clinically wrong, for example by flipping laterality or dropping a negation. The reference report can also contain errors. Use radiologist grading against images with an error taxonomy, and use automated graders only after checking their agreement with adjudicated radiologist grades.
What is FDA doing about AI radiology report evaluation?
In September 2026, FDA awarded a $1.29 million contract under its Broad Agency Announcement program to Cognita Imaging to research report evaluation methods. The project uses several LLMs as a jury, with radiologists adjudicating clinically important disagreements, across about one million examinations. It funds research on methods only and does not clear products or set standards.
Does AI report drafting save radiologists time?
Prospective evidence suggests it can. In a 2025 JAMA Network Open study at Northwestern Medicine, AI drafts for plain radiographs cut mean documentation time by 15.5% across 23,960 studies, with no significant difference in peer-reviewed clinical accuracy. The study was nonrandomized, at one system and on radiographs only, so test at your own sites.