Patient experience data · · 7 min read
How to validate AI-simulated patients
Acumen Editorial · Updated November 21, 2026
Simulated patients are moving into pharma work fast. Teams use them to pressure-test patient support chatbots, rehearse concept elicitation interviews, draft patient journeys and stand in for hard-to-reach populations.
A preprint posted on November 10, 2026 puts a number on the risk. Researchers at the University of Notre Dame and Simon Fraser University tested 17 language models playing patients against item-level data from eight real human samples. Their hypothesis has a plain name: the average patient. The model answers like a textbook case, not like a person.
That matters because patient experience data now carries regulatory weight. An FDA-commissioned assessment published in July 2026 found that 85% of new molecular entity reviews in 2025 mentioned patient experience data, up from 69% in 2021. If simulated patients feed any of that work, they need to be validated like any other instrument.
If you own patient insights, patient-facing AI or patient-centered outcomes, here is how we would test them.
What are AI-simulated patients?
AI-simulated patients are language models prompted to answer as a patient with a stated condition, severity or background. They fill out questionnaires, take part in mock interviews or talk to a chatbot under test.
They are different from synthetic HCP personas, which stand in for prescribers in market research. Simulated patients speak for people whose experience is often private, variable and poorly captured in published text. That is exactly where an average answer does the most damage.
Do simulated patients answer like real people?
Not yet, on current evidence. The Notre Dame and Simon Fraser preprint compared model answers with real respondents at matched total scores. Real samples included NHANES and NHIS population data, a clinical sample and a study of students told to feign symptoms.
| Measure | Real respondents | AI-simulated patients |
|---|---|---|
| Spread between respondents at the same total | Baseline | 1.5 to 11 times less variation |
| PHQ-9 answers of "not at all" | 10–15% | 0–4.5% |
| PHQ-9 responses endorsing all items 1–8 | 47–62% | 99.1–100% |
| Simulated interviews rated as endorsing all eight symptoms | 49–85% | 97.5–100% |
| Assessment agent placing the patient in the stated severity band | 42–61% with a real symptom profile | 20–26% with a severity-only persona |
Real people skip symptoms. Simulated patients rarely do. Even students told to fake symptoms answered "not at all" about as often as genuine patients, while every model did so at most 0.37 times as often.
The authors tested the obvious fixes. Better prompts, published personas, larger models and reasoning all reduced the pattern. None removed it. One step worked best: having the model voice a real symptom profile restored half to two thirds of the missing variation.
The paper is a preprint focused on psychiatric questionnaires. The mechanism is general. Ask a model to be "a patient with moderate disease" and you get the textbook version of that disease.
Why does this matter for pharma teams?
Three uses are exposed.
Testing patient-facing AI. If you test a patient support chatbot or symptom checker against simulated patients, every test patient presents with a full set of typical symptoms. Your system never meets the patient who has three of eight, or the one who downplays. The preprint shows this directly: an assessment agent tested on severity-only personas placed patients correctly far less often.
Patient insights and journey work. Averages erase subgroups. The FDA assessment names representativeness of sub-populations as a core challenge for patient experience data, alongside data quality and methodological rigor. A simulated panel makes that problem harder to see.
Analysis of real patient data. AI also enters on the analysis side. A Columbia-led study in JAMIA (August 2026) ran GPT-5.2 over 122 heart failure transcripts. Of the quotations it pulled, 68% were verbatim, 20% paraphrased, 6% partially hallucinated and 3% fully hallucinated. The authors call LLMs "analytic partners rather than autonomous coders."
The stakes keep rising. In the FDA assessment, 91.5% of orphan-designated NME reviews mentioned patient experience data, and about 40% of NME labels did. Patient-reported outcomes were the most cited type, at 80% of NME reviews that mentioned it.
How do you validate AI-simulated patients?
Use this checklist before simulated patients inform any decision, test or deliverable.
- Name the use. Write down what the simulated patients will be used for: chatbot testing, interview rehearsal, hypothesis generation or analysis support. Validate for that use only.
- Get a real reference sample. Collect item-level answers from consented patients and caregivers who match the condition, severity and setting.
- Compare at matched severity. Match simulated and real respondents on total score, then compare item by item. Totals alone will look fine.
- Measure spread. Check variation between respondents at the same total. Low spread is the first sign of an average patient.
- Count absent symptoms. Track how often each symptom is reported as absent. Real patients skip symptoms.
- Check structure. Compare item correlations. Real symptoms cluster in specific ways.
- Test separability. See whether a reviewer or classifier can tell simulated from real answers. If it can, so can reality.
- Ground personas in real profiles. Build each persona from a real, de-identified symptom profile, not a label and a severity word.
- Include the hard cases. Add patients with atypical presentations, low health literacy, rare disease and caregiver proxies.
- Verify every quote. When AI codes real transcripts, check each quotation against the source before it reaches a report.
- Have clinicians review safety. Ask clinicians to review patient-facing test cases for missed red flags and adverse event handling.
- Re-validate on change. Rerun the comparison when the model, prompt, persona set or target population changes.
For the vendor questions that sit on top of this, see what to ask a synthetic persona vendor.
Where do simulated patients still earn their place?
Used inside these limits, they are useful. They let a team generate many test conversations quickly, rehearse an interview guide before fieldwork and stress-test a chatbot's tone and escalation logic.
| Task | Simulated patients alone | Simulated patients grounded in real profiles | Real patients and caregivers |
|---|---|---|---|
| Drafting interview guides | Reasonable | Better | Final check |
| Load-testing chatbot flows | Reasonable | Better | Final check |
| Testing symptom recognition | Weak: symptoms rarely absent | Partial | Required |
| Subgroup and rare disease insight | Weak | Partial | Required |
| Evidence that may inform regulatory work | Not suitable | Not suitable | Required |
The rule is simple. The closer the output gets to a decision about real patients, the more real patients it needs.
How Acumen helps
We supply the real patients behind a validation study. Our patient experience data service provides consented, de-identified patient and caregiver input with clinician review, for pharma and digital health teams.
You get a matched reference sample with item-level answers, real symptom profiles to ground simulated personas, rated test conversations for patient-facing AI and clinician safety review. Teams use it to validate simulated panels, test chatbots before launch and check AI-coded transcripts.
FAQ
What are AI-simulated patients?
AI-simulated patients are language models prompted to answer as a patient with a stated condition, severity or background. Pharma and digital health teams use them to fill out questionnaires, rehearse interviews and test patient-facing chatbots. They differ from synthetic HCP personas, which stand in for prescribers, because they speak for private and highly variable patient experience.
Do simulated patients answer like real people?
Not yet. A November 2026 preprint testing 17 models found simulated patients varied 1.5 to 11 times less than real respondents with the same totals. On the PHQ-9 they answered "not at all" in 0–4.5% of answers, versus 10–15% for real people. Prompts, personas, larger models and reasoning reduced the gap but did not close it.
How do you validate AI-simulated patients?
Compare them with a real reference sample at matched severity, item by item. Measure spread between respondents, how often symptoms are absent, item correlations and whether reviewers can tell simulated from real answers. Ground each persona in a real symptom profile, include hard cases and re-validate whenever the model, prompt or population changes.
Can simulated patients replace real patient research?
No. They can speed up drafting, rehearsal and load-testing, but they compress the variation that makes patient experience data useful. The FDA's July 2026 assessment names data quality, methodological rigor and sub-population representativeness as core challenges for patient experience data. Anything that may inform decisions about real patients needs real patients and caregivers.
How can you make simulated patients more realistic?
Ground them in real data. In the November 2026 preprint, having the model voice a real symptom profile restored half to two thirds of the missing variation, more than prompt changes or published personas. Build personas from consented, de-identified patient profiles, and include atypical presentations, low health literacy and caregiver perspectives.
Is it safe to use LLMs to analyze patient interview transcripts?
With human oversight, yes. In a JAMIA study of 122 heart failure transcripts, GPT-5.2 found useful themes, but 6% of quotations were partially hallucinated and 3% fully hallucinated. The authors describe LLMs as analytic partners, not autonomous coders. Check every quotation against the source and keep a human-led analysis as the reference.