Clinical AI evaluation imagery representing specialist review of oncology treatment recommendations.

Healthcare · · 8 min read

How to evaluate oncology AI decision support

Acumen Editorial · Updated November 22, 2026

Oncology is where clinical AI meets its hardest test. Treatment depends on cancer type, stage, biomarkers, prior lines and drug approvals that change every few months. Decision support tools are moving into this space fast. The question for health-system clinical AI leads, oncology informatics teams and model builders is not whether a model sounds like an oncologist. It is whether your evaluation can tell when it is wrong.

Why is oncology AI decision support moving now?

Because the tools are reaching the point of care. Fierce Healthcare reported that OpenEvidence will integrate OncoKB, Memorial Sloan Kettering's precision oncology knowledge base, into its platform for clinicians outside MSK. OncoKB annotates genetic alterations by oncogenic effect, level of evidence and associated therapies, organized by cancer type. MSK will also bring OpenEvidence into its Epic workflow. As of September, the company reported 1.12 million licensed-verified U.S. clinicians on its platform.

Trial matching is moving too. TrialGPT 2.0, described in an arXiv paper revised October 30, 2026 and backed by an NIH-TrialBench consortium, was evaluated on 288 retrospective cases across four care settings and six months inside a live precision oncology tumor board. The authors report at least one clinician-recommended trial in the top 10 for about 91% of cases and 55.0% less clinician screening time.

Knowledge integration and strong benchmark scores are not the same as a validated recommendation for your patients. That gap is what your evaluation has to close.

How well do LLMs match tumor boards today?

Partly, and inconsistently. Three recent studies show the pattern.

  • Head and neck tumor board, Mainz. A prospective study in JMIR AI (August 20, 2026) ran two locally hosted open-weight models on 17 patients after the board had decided. One model matched the curative-versus-palliative intent in 94% of cases, the other in 59%. Neither matched the board's first-line regimen in more than 41% of cases. Board members flagged potentially hazardous information in 27% and 17% of ratings. The recommendation changed in one case.
  • Breast cancer, agentic pipelines. An arXiv study (July 13, 2026) compared seven pipelines on 72 real cases from stage I to IV, scored against 1,147 case-specific rubrics. The best configuration reached a global score of 0.594. Oncologist error analysis found missing or incorrect recommendations, flawed justifications, citation errors, outdated claims and overconfidence. The authors call these systems "insufficient for unsupervised clinical use."
  • Multiple myeloma, blinded raters. A Frontiers in Artificial Intelligence study (December 2025) had three myeloma specialists rate five systems on 50 scenarios. Clinical use readiness stayed low for all of them, from 2.7% to 25.3%. The authors note that the top system's developers have a financial interest in it.

The lesson is not that one model wins. It is that performance moves with disease, stage, prompt and pipeline, and only disease-specific experts can see it.

What errors should an oncology AI evaluation catch?

The errors that matter most are about time and sequence, not grammar. In the myeloma study, specialists found systems that omitted newly approved therapies, recommended drugs earlier than their approved line, suggested withdrawn or unapproved agents and sequenced treatments poorly. Each of those reads as a plausible plan.

Oncology adds three failure types that general clinical tests miss:

  1. Stale evidence. A recommendation can be correct for last year's label and wrong today. Every case needs an evidence date.
  2. Biomarker misreads. An alteration's actionability depends on the cancer type and the level of evidence behind it. A model that applies a therapy across tumor types without support is making an error, not a judgment call.
  3. Missing context. In the Mainz study, the one change to a decision came from a model flagging a pending follow-up exam. Models also misread cases. Test what the model notices, not only what it recommends.

Can automated graders replace oncologist review?

Not on their own. Rubric-based scoring makes oncology evaluation scale, and the breast cancer study shows how: the rubric generator saw the real clinical decisions that the evaluated systems did not. But someone has to write and check those rubrics, and the error analysis in that study still came from oncologists.

Use automated graders the way you would use a junior reviewer. Calibrate them on a set that oncologists have graded and adjudicated. Measure their agreement by cancer type and error type, not only overall. Send every high-severity disagreement back to a specialist. Reported inter-rater reliability matters too: in the myeloma study, specialists disagreed substantially on 32% of the top system's accuracy scenarios. If experts split, an automated grader trained on one of them will not settle the question.

What should an oncology AI evaluation test?

Score each layer separately. One overall score hides which failure you have.

LayerWhat it catchesHow to score it
Treatment intentWrong curative vs. palliative framingOncologist grade against board or chart decision
Regimen and line of therapyWrong regimen, wrong line, off-label use without flagCase-specific rubric written by disease specialists
Approval and evidence dateWithdrawn, pending or superseded drugsEach case stamped with an evidence cutoff; verify every agent
Biomarker interpretationActionability misread across tumor typesAlteration-specific cases with expected evidence level
CitationsWrong, outdated or invented sourcesCheck every cited source against the claim it supports
Context and safetyMissed pending exams, contraindications, hazardsSeeded cases; hazard rating by multiple oncologists
ConsistencyAnswers that shift between runsRepeat runs on the same case; report variance
Fit to populationResults that do not transferSeparate sets per cancer type, stage, setting and site

Checklist for building an oncology test set

  • Define intended use: tumor-board prep, point-of-care question answering, trial matching or biomarker interpretation. The pass bar follows from it.
  • Recruit oncologists by disease area, plus pathology, radiation oncology and pharmacy where the cases need them.
  • Use real or tumor-board cases where you can, de-identified, with the final decision recorded.
  • Stamp every case with an evidence cutoff date and re-grade when guidelines or approvals change.
  • Write case-specific rubrics with required, acceptable and harmful elements.
  • Enrich for rare alterations, later lines of therapy and recent approvals. Random samples under-test them.
  • Double-grade high-severity items and adjudicate disagreements. Report inter-rater agreement.
  • Run each case several times and report consistency, not just a best score.
  • Keep the set private and rerun it on every model, prompt, retrieval or knowledge-base change.
  • Log versions, grades and adjudications for governance review.

What should you ask an oncology AI evaluation vendor?

  1. Which oncology subspecialties do your graders cover, and how are their credentials verified?
  2. Do you grade on real cases with recorded decisions, or only on synthetic vignettes?
  3. How do you handle evidence dates and new approvals?
  4. Do you score biomarker interpretation separately, by cancer type?
  5. How do you measure and report inter-rater agreement?
  6. Can we keep the set private and rerun it ourselves?

A vendor that reports one accuracy number across all cancers has not answered these.

How Acumen helps

We build private evaluation sets for oncology AI. Verified oncologists in the relevant disease areas write case-specific rubrics, grade recommendations, rate hazards and adjudicate disagreements. We date-stamp evidence, enrich for rare alterations and later lines, split sets by cancer type and stage, and document provenance for governance and regulatory review.

See our medical model evaluation service, or read our guide to evaluating AI agents for clinical trials.

Request a sample dataset Schedule a conversation

FAQ

How do you evaluate oncology AI decision support?

Have oncologists in the relevant disease area grade recommendations on real or tumor-board cases using case-specific rubrics. Score treatment intent, regimen and line of therapy, approval status, biomarker interpretation, citations and hazards separately. Stamp each case with an evidence date, run cases more than once, and build separate sets for each cancer type, stage and care setting.

How accurate are LLMs at oncology treatment recommendations?

Results vary widely. In a 2026 head and neck tumor board study in JMIR AI, two local models matched the board's first-line regimen in 35% and 41% of 17 cases. In a 2026 breast cancer study, the best agentic pipeline scored 0.594 on case-specific rubrics. Both author groups say outputs need clinician supervision.

What errors are most common in oncology AI recommendations?

Studies report missing or incorrect recommendations, flawed justifications, citation errors, outdated claims and overconfidence. Myeloma specialists in a 2025 Frontiers in AI study also found omitted new approvals, drugs suggested earlier than their approved line, withdrawn or unapproved agents and poor treatment sequencing. These errors read as plausible plans, so disease-specific experts must grade them.

Why does evidence dating matter in oncology AI evaluation?

Oncology approvals and guidelines change often, so a recommendation can be right for an old label and wrong today. Stamp every test case with an evidence cutoff, record which approvals the answer key assumes, and re-grade affected cases when guidelines or labels change. Otherwise a model can lose points for being current or gain points for being stale.

Can automated graders evaluate oncology AI on their own?

Not reliably. Rubric-based automated grading helps evaluation scale, but oncologists still need to write the rubrics, calibrate the grader on adjudicated cases and review high-severity disagreements. Specialists themselves can disagree: in one 2025 myeloma study they disagreed substantially on 32% of the top system's accuracy scenarios. Measure grader agreement by cancer type and error type.

Who should grade oncology AI evaluation sets?

Board-certified oncologists in the disease area the tool covers, such as breast, thoracic, hematologic or head and neck oncology. Add pathologists for biomarker cases, radiation oncologists for combined-modality plans and oncology pharmacists for dosing and sequencing. Verify credentials, double-grade high-severity items, adjudicate disagreements and report inter-rater agreement with the results.

From the Acumen archive. Explore expert data and Healthcare & Pharma.

← Back to Insights