Life Sciences · · 7 min read
How to evaluate AI for statistical programming
Acumen Editorial · Updated November 21, 2026
AI coding agents can now take raw trial data, a statistical analysis plan and a set of specs, and hand back a full package of analysis datasets. That is real progress. It is also where the risk starts, because a package can look finished and still be wrong.
A new preprint makes the point with numbers. For biostatistics and statistical programming leads, it changes what a credible evaluation of AI looks like.
What did ClinDataBench find?
ClinDataBench, posted to medRxiv on November 19, 2026 by researchers at Georgia Tech, the University of Hong Kong, Nanyang Technological University and University College London, tests whether AI agents can build a submission-style analysis package.
Each agent got 22 SDTM domains, the SAP and dataset specifications from the public CDISC SDTM/ADaM pilot study. It had to deliver 13 ADaM datasets, Define-XML metadata and a program that rebuilds the datasets offline. Agents ran in sealed containers, so they could not look up the published answers.
What the authors report from this exploratory pilot of six models:
- Three closed models and DeepSeek delivered all 13 datasets in every setting. Two other open-weight models delivered no complete package.
- Pinnacle 21 Community findings ranged from 15 to 64 dataset-rule pairs.
- Letting agents revise against their own validator report cut findings in every usable case. It did not cut failed derivations in any of them.
- The package with the fewest findings had none of the 234 efficacy summary records in the form the specifications define.
The authors' conclusion: "lower validator finding counts do not necessarily indicate more correct derivations." It is a small pilot on one public study, and the authors say so. The lesson still holds for any team piloting AI on real studies.
Why isn't a clean validator report enough?
Because conformance checks structure, not values. A validator confirms that variables, controlled terminology and metadata follow the standard. It cannot tell you whether a baseline flag, an analysis window or an imputation was derived the way your SAP says.
That gap matters more with AI. An agent that learns to satisfy the validator can drive findings down while leaving derivations untouched, which is exactly what ClinDataBench saw. If your pilot scores AI on validator output, you are measuring the part of the job it already does well.
The same goes for code review. A program can run cleanly, rebuild every dataset and still apply the wrong population or the wrong baseline visit. Reviewers who read AI code line by line tend to check that it runs and looks familiar. The derivation errors that delay a submission hide one level down, in the values. That is why the reference has to be programmed independently, by someone who never saw the agent's output.
Can AI choose the right analysis, not just run it?
Not reliably yet. TrialAgentBench, a NeurIPS 2026 paper from Segal, Bolton, Torr and colleagues, tests whether agents can pick defensible estimands and methods and turn results into development decisions, using simulated patient data.
GPT-5.4 completed all 400 analysis tasks. It matched the required estimand or method class on 215 of them and produced an acceptable sample-size plan for 161. In multi-phase programmes, GPT-5.4 and Claude Opus 4.7 often nominated the wrong starting asset under confounding, and timed stop or advance decisions in line with the evidence in fewer than a third of programmes.
The pattern matches what we see across scientific AI. Agents finish the workflow. Judgment is where they fail. In statistical programming, judgment is the derivation rule in a footnote of the SAP and the call a statistician makes when the data don't fit it.
What should an AI statistical programming evaluation test?
Score each layer of the work separately. A pass on one layer says nothing about the next.
| Layer | What it catches | How to score it |
|---|---|---|
| Conformance | Structural and terminology errors | Validator findings, tracked but never used alone |
| Value-level derivations | Wrong flags, windows, imputations, populations | Compare every derived value with an independently programmed reference |
| Traceability | Undocumented or unreproducible logic | Rebuild from code in a clean environment; check Define-XML against what the code does |
| Specification fidelity | Logic that drifts from the SAP and specs | Statisticians map each derivation back to its spec line |
| Analysis choice | Wrong estimand, method or sample-size reasoning | Blind review by biostatisticians against a written rationale |
| Effort and rework | Hidden cost of fixing AI output | Time spent by programmers to reach submission quality |
The middle rows are where the money is. They need people who have programmed and reviewed submission datasets, not a script.
How do you build the test set?
Use your own studies and your own conventions. A public pilot study is a fine start. It is not your therapeutic area, your SAP template or your sponsor standards.
- Pick closed studies across therapeutic areas and designs you run often, including at least one with messy data
- Freeze the SDTM, SAP, specs and final ADaM so the reference cannot drift
- Plant known traps: protocol deviations, partial dates, visit windows, intercurrent events, changed analysis populations
- Have senior programmers independently program the reference, and record who did what
- Compare AI output value by value, not dataset by dataset
- Have biostatisticians review analysis choices blind to which output came from the agent
- Keep the set out of vendor hands and rerun it after every model, prompt or tool change
- Log every run with inputs, versions and outputs so the evaluation itself is audit-ready
Where does AI fit in double programming?
Treat it as one more programmer, not a substitute for independent QC. Cytel's Manoj Kumar Maripally, writing in May 2026, argues quality control should scale with risk under ICH Q9: full double programming for high-risk analyses, peer review with automated tests for medium risk, automation for low risk. He is blunt: "AI-generated outputs should not be treated as final deliverables."
Regulators point the same way. The guiding principles of good AI practice in drug development that FDA and EMA released on January 14, 2026 include risk-based assessment and data quality and lifecycle management. Your evaluation is the evidence that you applied both.
What should you ask an AI vendor for statistical programming?
- Have you measured derivation accuracy value by value, not only validator findings?
- Which studies and standards versions did you test on, and were any public?
- Who programmed the reference outputs, and what is their submission experience?
- Can the agent rebuild every dataset from code in a clean environment?
- How does it document derivations in Define-XML and in the program itself?
- Can we run our own private test set against every release before it touches a live study?
A vendor that answers with a Pinnacle 21 screenshot has not answered the question.
How Acumen helps
We build private evaluation sets for AI in clinical statistical programming. Senior statistical programmers and biostatisticians write the tasks from studies that look like yours, program independent references, plant the traps that matter and grade every derived value. We document provenance for audit and rerun the set with each model release.
See our life sciences model evaluation service, or read how to evaluate AI agents for clinical trials.
FAQ
Can AI build ADaM datasets from SDTM?
Often, in structure. In ClinDataBench, a medRxiv preprint posted November 19, 2026, three closed models and DeepSeek delivered all 13 ADaM datasets from 22 SDTM domains in every setting. But delivering datasets is not the same as deriving them correctly. The authors found validator-clean packages that still had wrong values, so every derivation needs checking.
Is a clean Pinnacle 21 report enough to trust AI-generated datasets?
No. Pinnacle 21 checks conformance: structure, controlled terminology and metadata. It cannot confirm that a flag, window or imputation follows your SAP. In ClinDataBench, the package with the fewest validator findings had none of 234 efficacy summary records in the specified form. Track validator findings, but score derivations value by value against a reference.
Can AI choose the right estimand or statistical method?
Not reliably yet. In TrialAgentBench, a NeurIPS 2026 paper using simulated trial data, GPT-5.4 completed all 400 analysis tasks but matched the required estimand or method class on 215 and gave an acceptable sample-size plan on 161. Analysis choice should be reviewed blind by biostatisticians against a written rationale.
Should AI replace double programming?
No. Treat AI as one more programmer inside a risk-based quality process. Cytel's Manoj Kumar Maripally recommends scaling QC to risk under ICH Q9, with full double programming kept for high-risk analyses. AI can draft code and datasets, but independent human programming and review should still sign off anything that reaches a submission.
Who should grade an AI statistical programming evaluation?
Senior statistical programmers with submission experience should program the independent references and grade derived values. Biostatisticians should review estimands, methods and sample-size reasoning, blind to which output came from the agent. Matching graders to the therapeutic area and your sponsor standards matters, since conventions for windows, imputation and populations vary between studies.
What do FDA and EMA expect when AI is used in drug development?
FDA and EMA released 10 guiding principles of good AI practice in drug development on January 14, 2026. They include risk-based assessment and data quality and lifecycle management. For statistical programming, that means scaling validation to the risk of each output and keeping documented, reproducible evidence of how the AI was tested and monitored.