Life Sciences · · 7 min read
How to evaluate AI for literature review
Acumen Editorial · Updated November 21, 2026
Evidence work is where AI entered life sciences quietly. Teams now use it to screen abstracts for a systematic review, pull study characteristics into a table, assemble a background section, or answer a medical information query from published literature.
The published evidence on whether it works splits in a way that matters for procurement. On screening, the numbers look good. On finding and citing the right paper, they look bad. A single vendor demo will show you the first and hide the second.
If you own evidence synthesis, medical information or research informatics, here is how we would test a tool before it touches work that others cite.
How accurate is AI at literature screening?
Better than most buyers expect. A systematic review and meta-analysis posted to medRxiv in March 2026 pooled 18 studies of large language models screening medical literature. For title and abstract screening, pooled sensitivity was 0.92 (95% CI 0.81–0.96) and specificity 0.94 (95% CI 0.90–0.97). For full-text screening, across four studies, both reached 0.99.
Three caveats sit inside those numbers, and the authors name them.
Heterogeneity was high: I² was 95.82% for sensitivity and 99.90% for specificity at the title and abstract stage. Results varied by study, and the review could not explain why. Most included studies were methodological evaluations rather than real-world reviews. And the reference standard was human screening, which has its own error rate.
Prompting mattered. Prompts that included examples or chain-of-thought reasoning reached pooled sensitivity of 0.95, versus 0.86 without them (p < 0.01). The authors position these models as high-sensitivity assistive tools, not replacements for human reviewers.
For a review, sensitivity is the number to negotiate on. A missed eligible study is an error you cannot find later. A wrongly included study gets caught at full text.
Why do AI tools still invent references?
Because retrieval and generation are different jobs, and most tools blur them.
A comparative study of AI-assisted retrieval (Gao et al., arXiv:2603.22344) took 40 original articles from major medical journals, prompted five free consumer AI tools for ten references each, and checked all 2,000 against DOI, PubMed and Google Scholar. Across tools, 47.8% of references failed every applicable check. The best tool missed completely on 11.2% of references; the worst on 78.5%. The mean score ratio across tools was 0.29 out of 1.
Two findings are worth carrying into a vendor conversation. Prompts telling the model to double-check its references did not appear to improve accuracy. And the models approximated topical relevance reasonably well while failing on bibliographic metadata, which is the opposite of what a reader checking a reference list assumes. The authors call these tools "assistive, not authoritative."
That study tested free consumer tools, not enterprise literature platforms with grounded retrieval. The point is not the specific percentages. It is that screening accuracy tells you nothing about citation accuracy, so you have to measure both.
Detection tooling will not rescue you either. Researchers at GESIS tested five citation-checking tools against 104 manually verified references, 33 of them problematic. The two strongest catchers also produced 34 and 36 false positives; the cleanest tool missed 15 of 33 problems. Flagged references still need a human to check DOI, title, authors and venue.
What should you measure?
Split the evaluation by task. Each one fails differently.
| Task | Primary metric | Secondary | Reference standard | Who grades |
|---|---|---|---|---|
| Title and abstract screening | Recall on eligible studies | Workload saved, run-to-run consistency | Dual human screening with adjudication | Trained reviewers in the field |
| Full-text eligibility | Recall, then precision | Reasons given for exclusion | Human full-text decisions | Subject-matter reviewer |
| Data extraction | Field-level exact match | Error type: wrong unit, wrong arm, missing | Human-extracted table | Methodologist or clinician |
| Reference retrieval and citation | Share of references resolving to the correct record | Relevance to the claim | DOI, PubMed, publisher record | Information specialist |
| Synthesis and drafting | Claim-to-source traceability | Unsupported statements per 1,000 words | Source documents | Subject-matter expert |
Two measurement rules apply across all of them. Report recall separately from accuracy, because a tool that excludes most records scores well on accuracy while losing the studies you needed. And run the same test twice. Run-to-run variation is a property of these systems, not a fluke.
How do you run the evaluation?
- Build a labelled reference set from a completed review of your own: the full search output, the human include and exclude decisions, and the reasons.
- Hold it back. Do not send it to a vendor for tuning. Keep a private copy for re-testing.
- Include the hard records: conference abstracts, non-English studies, preprints, wrong-population near-misses, and studies you argued about.
- Measure recall first, at the threshold the vendor recommends, and ask what happens to recall at their default settings.
- Test extraction field by field, and classify errors by type rather than counting a row wrong.
- Verify every reference the tool returns against DOI, PubMed and the publisher record.
- Check claim-to-source traceability in any drafted text: can each statement be traced to a source that supports it?
- Repeat the run. Compare the two sets of decisions.
- Define the trust tier for each step, from low to high consequence, and require a human decision above your threshold.
- Log the AI use so the method can be reported and reproduced.
- Re-test on every model or prompt change, including silent vendor-side updates.
A reporting standard now exists for this. ARISMA, published in August 2026 by researchers at the University of Southern Denmark, sets out 22 reporting items for AI-assisted reviews, each requiring disclosure of the AI-supported steps, the human oversight applied and the validation evidence behind it. It treats AI as "a bounded assistant whose use must be benchmarked, logged, and reversible." It complements PRISMA rather than replacing it. If you are building an internal policy, it is a reasonable spine.
For how to read vendor-published accuracy claims before any of this, see how to read medical AI benchmark claims.
Can AI run a systematic review on its own?
No, on current evidence, and the gap is not only about accuracy.
A systematic review carries accountability. Someone signs the method, defends the eligibility decisions and answers questions about why a study was excluded. Pooled sensitivity of 0.92 across 18 heterogeneous methodological studies does not transfer to your review, your question or your corpus. It tells you the approach is worth testing.
The sensible division of labour is narrow. Let the tool do first-pass screening with a human second pass, surface candidate extractions for human confirmation, and draft nothing that is not traced to a verified source.
How Acumen helps
We build the graded evaluation sets that make this measurable. Our scientific AI model evaluation service supplies private, task-specific tests written and graded by practising scientists and information specialists, and rerun on every model release.
For evidence work, that means labelled screening sets with documented eligibility decisions, extraction tasks with field-level reference answers, citation-verification sets built on records with confirmed identifiers, and rubric-graded review of drafted text for unsupported claims.
FAQ
How accurate is AI at screening medical literature?
A March 2026 meta-analysis of 18 studies found pooled sensitivity of 0.92 and specificity 0.94 for title and abstract screening, and 0.99 for both at full text across four studies. Heterogeneity was very high, most studies were methodological rather than real-world, and the authors describe these models as assistive tools rather than replacements for human reviewers.
How often does AI invent citations?
It depends heavily on the tool. A comparative study prompting five free consumer AI tools for 2,000 references on medical articles found 47.8% failed every applicable check against DOI, PubMed and Google Scholar. The best tool failed on 11.2%, the worst on 78.5%. Enterprise tools with grounded retrieval may do better, which is why you test your own.
Can citation-checking tools catch hallucinated references?
Partly. Researchers at GESIS tested five tools against 104 verified references, 33 of them problematic. The strongest catchers flagged most problems but produced over 30 false positives each, while the cleanest tool missed 15 of 33. Treat their output as a triage signal, and confirm DOI, title, authors and venue by hand.
What metric should we use for AI literature screening?
Recall on eligible studies, reported separately from accuracy. A tool that excludes most records looks accurate while losing the studies your review needed. Track workload saved and run-to-run consistency alongside it, and run the same test twice: variation between identical runs is a property of these systems.
Do better prompts fix the problem?
They help with screening and not with citations. In the 2026 meta-analysis, prompts with examples or chain-of-thought reasoning reached pooled sensitivity of 0.95 versus 0.86 without, a significant difference. In the retrieval study, instructing models to double-check their references did not appear to improve accuracy. Prompting is not a substitute for verification.
How should we document AI use in a review?
Follow a reporting standard. ARISMA, published in August 2026, defines 22 reporting items for AI-assisted systematic, scoping and mapping reviews, each requiring the AI-supported steps, the human oversight applied and the validation evidence. It complements PRISMA 2020 and assumes consequential decisions stay with named humans.