Healthcare AI evaluation imagery representing expert review of clinical evidence for health technology assessment dossiers.

Pharma · · 8 min read

How to evaluate AI for HTA dossiers

Acumen Editorial · Updated November 22, 2026

Market access teams are using AI to search literature, screen studies, extract data and draft dossier sections. Assessors have noticed. In July 2026 the EU HTA Coordination Group adopted principles for AI use in joint clinical assessment dossiers, and NICE already expects AI use to be declared and justified. At ISPOR Europe 2026 in Vienna this month, NICE described how it validates its own AI tools. The question for HEOR, market access and medical affairs leaders is no longer whether AI belongs in the dossier. It is whether you can prove it did the work correctly.

Why do HTA dossiers need an AI evaluation now?

Three pressures meet in the same document.

  • More products in scope. EU joint clinical assessments have applied since January 12, 2025 to new cancer active substances and all advanced therapy medicinal products. EMA says orphan medicines join in January 2028 and all centrally authorised medicines in 2030.
  • Tighter timelines. A ZS review presented at ISPOR Europe 2026 notes that JCA literature searches must be cut off within three months of dossier submission, against six months under frameworks such as NICE. Multiple PICOs, each needing its own evidence synthesis, add volume and error risk.
  • Clear disclosure rules. Both the EU and NICE now say how AI use must be reported. A dossier that cannot show what the AI did, and who checked it, invites questions at the worst possible time.

AI promises real speed. A Pharmacoevidence poster at ISPOR 2026 reported an estimated 80–85% reduction in development time for a NICE briefing book drafted with retrieval-augmented generation and expert review. Speed without a validation record, though, moves the risk from the timeline to the assessment.

What do the EU and NICE rules actually require?

The two frameworks differ in emphasis but agree on the core: the submitter stays accountable, a human oversees each step, and AI use is documented.

RequirementEU HTA Coordination Group (JCA dossiers)NICE position statement
AccountabilityDeveloper stays fully responsible for evidence, methods and conclusionsSubmitting organisation remains accountable
Human oversightNo step fully automated without a human responsible for quality and accuracyKeep "a capable and informed human in the loop"
DisclosureDocument AI use in retrieval, screening, extraction, risk of bias, analysis and reportingDeclare AI use and explain the choice of method
RecordsTool, version, date, developer and purpose; prompts available on requestReport risks identified, such as bias, and steps taken
Higher-risk usesNot addressed by applicationComparative effect estimates need sensitivity analysis and triangulation
Other lawEU AI Act, copyright and data protectionEvidence of security against data poisoning and prompt injection

Sources: Cytel and Becaris Publishing summaries of the HTA CG principles adopted July 15, 2026; NICE ECD11.

As Cytel put it, "AI does not change accountability." Spelling and grammar tools are exempt from EU disclosure, according to Becaris. Everything that shapes evidence is not.

What should you measure at each dossier step?

Evaluate the steps where an AI error changes what assessors see. We use the dimensions below.

Dossier stepWhat to testWho gradesFailure it catches
Literature searchRecall against an expert-built reference set of relevant studiesInformation specialists, HEOR reviewersMissed trials that make the dossier incomplete
ScreeningInclude and exclude decisions against dual human screeningHEOR reviewers, cliniciansWrong exclusions, PICO drift
Data extractionField-level accuracy for outcomes, arms, populations and time pointsClinicians, biostatisticiansSwapped arms, wrong denominators, missed subgroups
Risk of biasAgreement with expert judgments by domainMethodologistsOverly generous quality ratings
PICO mappingCorrect assignment of evidence to each requested PICOClinicians, market access leadsEvidence reported against the wrong comparator
Drafted textEvery claim traceable to a cited sourceMedical writers, cliniciansUnsupported statements, invented citations

Report each step separately. An extraction error and a missed study fail in different ways, and a single blended accuracy score hides both.

How do you build a gold set for HTA evidence work?

  1. Start from closed reviews. Use completed systematic reviews in your therapy areas where the included studies and extracted data are already adjudicated.
  2. Add expert dual coding. Have two qualified reviewers independently screen and extract a sample, then adjudicate disagreements with a senior methodologist. Keep the record.
  3. Seed the hard cases. Include near-miss studies, multi-arm trials, subgroup-only results and abstracts later updated by full papers. These test the judgment PICO scoping depends on.
  4. Hold some back. Keep a holdout set the team never uses to tune prompts. NICE added holdout data to its own prompt test suite after an external review.
  5. Freeze and log. Record tool versions, prompts and settings for each run, so you can reproduce results and supply prompts if assessors ask.
  6. Rerun on change. Model, prompt and retrieval updates all require a rerun on the frozen set before the tool touches a live dossier.

How does NICE test its own AI tools?

NICE's ISPOR Europe 2026 abstract offers a useful template. Its internal guide moves tools from proof of concept to shadow testing against real NICE workflows. Development and evaluation datasets stay separate, success metrics are set in advance, and experiments run in reproducible, documented environments. Prompt design and settings are recorded in a structured format. An external partner reviewed the guide from a sociotechnical angle, which led NICE to add detail on accountability.

The lesson for submitters is direct. If the assessor validates its own tools this way, a dossier built with AI should show at least the same discipline: predefined metrics, a separate test set and a reproducible record.

Checklist: what to ask an AI vendor for HEOR or market access work

  • Which dossier steps does the tool touch, and how is each one validated?
  • What is search recall against an expert reference set, not just precision?
  • Who built your gold sets, and what are their credentials and therapy areas?
  • How do you measure agreement between human reviewers, and how are disagreements adjudicated?
  • Can every extracted value and drafted claim be traced to a source page?
  • Do you log tool versions, prompts and settings per run, and can we export them for a JCA?
  • How do you handle PICO changes late in scoping?
  • What security controls address prompt injection and data poisoning?
  • Where does a named human sign off, and is any step fully automated?

A vendor that cannot answer the first, fifth and sixth questions has a productivity tool, not a dossier-ready one.

How Acumen helps

Acumen builds expert-graded evaluation and reference data for pharma AI. For HTA work, that means verified HEOR specialists, clinicians, biostatisticians and methodologists who screen and extract in duplicate, adjudicate disagreements, judge risk of bias and map evidence to PICOs. We deliver frozen gold sets, holdout splits, rubrics and agreement statistics your team can rerun on every tool change and cite in your dossier. See our medical affairs and regulatory data capability, or read how to evaluate AI for literature review.

Request a sample dataset Schedule a conversation

FAQ

Can you use AI to prepare an EU joint clinical assessment dossier?

Yes. The HTA Coordination Group principles adopted in July 2026 allow AI use but keep the developer fully responsible for the dossier. No step should be fully automated without a human responsible for quality and accuracy, and meaningful AI use in retrieval, screening, extraction, risk of bias, analysis and reporting must be documented.

What must you disclose about AI in a JCA dossier?

According to Becaris Publishing's summary of the HTA CG principles, dossiers should name each AI tool, its version, date, developer and intended purpose, and identify AI-generated or AI-informed outputs. Prompts should be recorded and made available if requested. AI used only for spelling and grammar does not need to be disclosed.

What does NICE require when submissions use AI?

NICE's position statement asks submitters to declare AI use, justify it, prefer simpler methods where they work and keep a capable, informed human in the loop. Submitters should report risks such as bias and the steps taken against them. AI used to estimate comparative treatment effects needs sensitivity analysis and triangulation with clinical evidence.

Which products fall under EU joint clinical assessment?

According to EMA, JCAs have applied since January 12, 2025 to new cancer active substances and all advanced therapy medicinal products. Orphan medicines join in January 2028, and all centrally authorised medicines in 2030. Selected high-risk medical devices also came under the regulation from 2026.

How do you validate an AI tool for systematic literature reviews in HTA?

Compare it with expert dual screening and extraction on a frozen gold set. Measure search recall, screening agreement, field-level extraction accuracy and risk-of-bias agreement separately. Keep a holdout set out of prompt tuning, log versions and prompts, and rerun the test whenever the model, prompt or retrieval setup changes.

Who should grade AI outputs for an HTA dossier?

Use the experts who would do the work by hand. Information specialists judge search recall, HEOR reviewers and clinicians judge screening and PICO mapping, biostatisticians check extracted data, and methodologists grade risk of bias. Use two independent graders, report their agreement and have a senior reviewer adjudicate so the gold set can be defended.

From the Acumen archive. Explore expert data and Healthcare & Pharma.

← Back to Insights