Drug safety · · 8 min read
How to evaluate AI for pharmacovigilance
Acumen Editorial · Updated November 20, 2026
AI is moving into drug safety departments. Veeva has been building agents for its Vault Safety product, and IntuitionLabs reported in October 2026 that initial go-lives of Veeva's Falcon platform for clinical, regulatory and safety work were still planned for later this year. Other vendors are pitching the same thing: faster intake, automatic coding and draft narratives.
The research is catching up at the same time. On November 20, 2026, a narrative review in Frontiers in Pharmacology set out a human-supervised design for AI in medication safety. It warns against inferring causality, incidence or patient-level risk from text alone. In August, four FDA scientists published a perspective in Frontiers in Drug Safety and Regulation. Their conclusion: generative AI points toward hybrid human-AI systems, not full automation.
So the question for safety leaders is not whether to use AI. It is which tasks to hand it, and how to prove it works on your cases. Here is how we would test it.
Which pharmacovigilance tasks can AI handle today?
The FDA authors propose a "computable PV" framework that rates each task by how suitable it is for automation. Routine, well-defined tasks are the best candidates: completeness checks, deduplication and structured extraction. Causality assessment still relies on expert judgment.
The results they cite support that split. An LLM-based system called ALIGN reached 87 to 90 percent accuracy in MedDRA coding. Probabilistic deduplication tools cut manual workload by up to 60 percent. Those are real gains on narrow tasks.
Here is how we group the work:
- Intake and triage. Spotting valid cases, the four minimum criteria and seriousness in emails, call notes, literature and social listening.
- Extraction. Pulling drugs, events, dates, doses and outcomes into structured fields.
- Coding. Mapping events to MedDRA and products to the drug dictionary.
- Deduplication. Finding follow-ups and duplicate reports across sources.
- Narratives. Drafting case narratives that match the source and your conventions.
- Causality and signal support. Organizing evidence for a safety physician who makes the call.
Each task fails differently, so each needs its own test.
Why aren't exact-match scores enough?
Because MedDRA is a hierarchy, and a strict string match hides how often the code is clinically fine. A TU Dresden-led preprint posted in August 2026 tested a retrieval-augmented LLM pipeline on adverse events from three acute myeloid leukemia trials. Exact match at the Lowest Level Term was 50 to 58 percent. At the Preferred Term level it was 78 to 85 percent, and at System Organ Class 90 to 93 percent. Clinical experts judged 91 to 97 percent of the codes clinically correct.
One model, four very different scores. The authors recommend pairing hierarchical metrics with expert review. We agree. A vendor quoting one accuracy number has not told you which level it measured or who judged it.
Model choice is not a shortcut either. In a small pilot, the FDA authors found a fine-tuned BioBERT model beat a zero-shot Llama-4 model on named entity recognition: F1 of 0.54 against 0.42 on FAERS reports, and 0.53 against 0.31 on VAERS. The general model produced more hallucinated boundaries and label confusion. Bigger and newer is not the same as better on your data.
Where must humans stay in the loop?
Causality, first. The FDA perspective notes that human adjudicators fully agreed on binary causality classifications in only about 61 percent of cases. In the authors' informal pilot, the LLM was overly optimistic on ambiguous cases. It called more of them causality-assessable than human reviewers did.
That matters for evaluation design. If experts disagree nearly four times in ten, a single reviewer's label is not ground truth. You need multiple graders, adjudication and reported agreement.
The industry framework says the same. The CIOMS Working Group XIV report, published December 4, 2025, proposes treating AI tools like medicinal products: defined approved uses, operating parameters, known side effects and warnings. CIOMS also warns that misguided automation could "risk eliminating the value added by humans in the loop."
How should you build a pharmacovigilance evaluation set?
Build it from your own cases, privately, and score each task on its own. Here is the checklist we use:
- Define the approved use for each tool. Which task, which sources, which products, and what it must never decide.
- Use real, messy inputs. De-identified source documents: call center notes, literature abstracts, partner reports, foreign-language cases.
- Seed known cases. Plant duplicates, missing minimum criteria, serious cases and special situations you already have answers for.
- Write reference answers. Gold extractions, codes at every MedDRA level, reference narratives and adjudicated causality calls.
- Match graders to the task. Case processors for extraction, coding specialists for MedDRA, drug safety physicians for seriousness and causality.
- Grade blind and adjudicate. At least two graders per item, disagreements resolved, agreement reported.
- Measure the misses. Track missed serious cases separately. A false alarm costs minutes. A missed serious case can miss a reporting clock.
- Document everything. Your plan, results and deviations belong in your quality system and inspection file.
- Rerun on every change. New model version, MedDRA release, new source type or new product means a new run.
Which metrics matter for each pharmacovigilance task?
Pick the metric that matches the failure that hurts most.
| Task | Failure that matters | Primary metric | Who grades |
|---|---|---|---|
| Intake and triage | Missed valid or serious case | Recall on seeded serious cases | Case processors, safety physicians |
| Extraction | Wrong drug, event or date | Field-level precision and recall | Case processors |
| MedDRA coding | Clinically wrong code | Accuracy at LLT, PT and SOC, plus expert-judged correctness | Coding specialists |
| Deduplication | Missed follow-up or false merge | Pairwise precision and recall | Case processors |
| Narratives | Unsupported or missing fact | Source-support error rate | Safety writers, safety physicians |
| Causality support | Overconfident call on thin evidence | Agreement with adjudicated reference, by case difficulty | Drug safety physicians |
Report each metric by case difficulty. Averages hide the ambiguous cases, and those are where the FDA pilot saw the model go wrong.
What should you ask a pharmacovigilance AI vendor?
Ask for evidence, not a demo:
1. Which tasks was the tool tested on, and on whose cases?
2. Who wrote the reference answers, and what are their roles and credentials?
3. What is recall on serious cases, reported separately from overall accuracy?
4. Which MedDRA level does the coding accuracy refer to, and was clinical correctness judged by experts?
5. How does the tool flag uncertainty, and what happens to low-confidence cases?
6. Can we run our own private evaluation set before go-live and after every update?
7. What audit trail exists for each suggestion, edit and approval?
How Acumen helps
We build the evaluation and training data behind regulated pharma AI. Our medical affairs and regulatory data service supplies private evaluation sets written and graded by drug safety physicians, pharmacovigilance case processors, coding specialists and regulatory experts. That includes gold extractions, codes at every MedDRA level, reference narratives, adjudicated causality calls and regression runs on every release. Every item is reviewed by experts and adjudicated.
Use it to test a vendor's tool, to tune your own model, or both. For how European regulators frame oversight of AI in medicines work, see our guide to EMA AI guidance.
FAQ
What is AI in pharmacovigilance used for?
Mostly routine case work. Teams use AI to triage incoming reports, extract drugs, events and dates, suggest MedDRA codes, find duplicates and draft narratives. FDA scientists writing in 2026 rated completeness checks, deduplication and structured extraction as the best candidates for automation, while causality assessment still depends on expert judgment.
How accurate are LLMs at MedDRA coding?
It depends on the level you measure. A 2026 TU Dresden-led preprint found 50 to 58 percent exact matches at the Lowest Level Term, 78 to 85 percent at Preferred Term and 90 to 93 percent at System Organ Class. Clinical experts judged 91 to 97 percent of codes clinically correct. Always ask which level a claim refers to.
Can AI assess causality for adverse events?
It can organize the evidence, but a safety physician should make the call. FDA authors note that human adjudicators fully agreed on binary causality in only about 61 percent of cases, and an LLM in their pilot was overly optimistic on ambiguous cases. Treat AI as support for causality review, and test it against adjudicated references.
What does CIOMS say about AI in pharmacovigilance?
The CIOMS Working Group XIV report, published December 4, 2025, gives industry, regulators and academics shared terms and concepts for AI in drug safety. It proposes that AI tools have clearly defined approved uses, operating parameters, known side effects and warnings, much like medicines. It also warns that misguided automation can remove the value humans add.
How do you build a test set for pharmacovigilance AI?
Start from your own de-identified cases across every source type. Seed known duplicates, serious cases and incomplete reports. Write gold extractions, codes at each MedDRA level, reference narratives and adjudicated causality calls. Have at least two qualified graders per item, resolve disagreements, and rerun the set after every model, dictionary or source change.
Who should grade pharmacovigilance AI outputs?
Match the grader to the task. Case processors judge triage, extraction and deduplication. Coding specialists judge MedDRA and drug dictionary codes. Safety writers and drug safety physicians judge narratives. Drug safety physicians judge seriousness and causality support. Report agreement between graders alongside scores, because disagreement on hard cases is common and informative.