Data Governance · · 7 min read
HIPAA compliant AI training data: what it takes
Acumen Editorial · Updated November 20, 2026
Every clinical AI program hits the same wall. The model needs realistic medical data. The privacy office, the security review and, for regulated products, FDA all want to know where that data came from.
The rules are not getting simpler. OCR Director Paula Stannard said at the April 2026 HIPAA Summit that the Security Rule applies to AI "in the same manner as any other technology," according to Morgan Lewis. HHS has said the Security Rule governs ePHI used in AI training data. The proposed Security Rule overhaul, first published in January 2025, would require written inventories of AI tools that touch ePHI. In July 2026, Fierce Healthcare and Alston & Bird reported that the final rule had slipped to July 2027. Delay is not relief. It is a window to get your data sourcing in order.
Here is how we think about it.
Can you use PHI to train an AI model?
Sometimes, but it is the hardest path. HIPAA does not have an AI-specific provision. It treats model training like any other use of PHI. You need a permitted purpose, the right agreements and the minimum necessary data.
Contracts are tightening around this. Morgan Lewis reported from the 2026 HIPAA Summit that organizations are writing business associate agreements that bar vendors from using PHI to train or improve AI models without appropriate authorization. HIPAA Journal notes that some state laws require explicit consent before sensitive data is used for automated processing, and restrict secondary uses such as model training.
For most training and evaluation work, the cleaner answer is data that never contained PHI, or data that has been properly de-identified.
What are the options for medical AI training data?
There are four common sources. Each carries different obligations. HIPAA Summit panelists made the same point: PHI, limited data sets and de-identified data "have distinct regulatory implications."
| Data source | HIPAA status | What you need | Main risk |
|---|---|---|---|
| Identified patient records (PHI) | Fully regulated | Permitted purpose or authorization, BAA, minimum necessary, Security Rule controls | Contract and consent limits on training use |
| Limited data set | Regulated, with some direct identifiers removed | Data use agreement, permitted purpose | Scope creep beyond the agreed use |
| De-identified data | Not PHI once properly de-identified | Safe Harbor or Expert Determination, documented | Re-identification, especially in rare conditions |
| Clinician-authored cases | No PHI when written without patient identifiers | Authoring rules, identifier screening, provenance records | Real details slipping into "composite" cases |
The last row is where expert data fits. A physician who writes a case from practice, with a presentation, workup and reference answer, creates new content. It is not a patient record. That only holds if the authoring rules forbid real identifiers and someone checks.
Most programs mix sources. A team might fine-tune on de-identified notes from a health-system partner, then use clinician-authored cases for reasoning traces, preference data and evaluation. That works, as long as every item carries a record of where it came from and which rules apply to it. Mixed data without labels is where audits get slow and expensive.
What does HIPAA de-identification require?
HHS recognizes two methods.
Safe Harbor. Remove 18 categories of identifiers, including names, geographic units smaller than a state and dates more specific than the year. The last category catches "any other unique identifying number, characteristic, or code." Removal is not enough on its own. The covered entity must also have no actual knowledge that the remaining data could identify someone.
Expert Determination. A qualified expert applies statistical and scientific methods and concludes that the risk of identification is "very small." The expert documents the methods and results. HHS says there is no fixed numerical threshold that meets the standard everywhere.
De-identification is not a one-time stamp. HIPAA Journal warns that if a vendor's AI system re-identifies de-identified information, that can become a notifiable breach. Rare diseases, small populations and rich free text raise the risk. Ask how a vendor tests for it.
Why does FDA care about your training data?
If your model is, or will become, part of a medical device, the data is part of the evidence. FDA's January 2025 draft guidance on AI-enabled device software asks sponsors to describe how data was collected, cleaned and processed, and how test data stays independent of training data.
FDA is also rethinking how it evaluates generative AI. In an August 18, 2026 discussion paper, CDRH floated a "competency-based" model, closer to how clinicians are credentialed. Cooley's summary describes nonclinical benchmarking across competencies such as safety and clinical proficiency, plus postmarket review of real-world outputs by independent clinician adjudicators. The paper is not guidance. But the direction is clear: expert-built benchmarks and clinician review are becoming part of the regulatory record.
So compliance is not only privacy. It is provenance. Who wrote this data, when, under what instructions, reviewed by whom, and was any of it ever used in the test set?
Checklist: what to ask a medical AI data vendor
Use this in procurement, privacy and security review.
- Does any of the data contain PHI? If yes, under what permitted purpose and which agreement?
- If data is de-identified, which method was used: Safe Harbor or Expert Determination? Can we see the documentation?
- How do you test for re-identification risk in rare conditions and free text?
- For clinician-authored cases, what authoring rules prohibit real patient identifiers, and who screens for them?
- Are the clinicians credentialed and matched to the specialty? How is that verified?
- Do you keep a provenance record for every item: author, date, instructions, reviewers, revisions?
- Will you sign a BAA where needed, and do your contracts restrict training use of any PHI?
- How are evaluation sets kept separate from training data, and can you document it?
- Can your documentation support an FDA data description and our security risk analysis?
- What state-law restrictions on secondary use or automated processing have you assessed?
A vendor that cannot answer these clearly is handing the compliance risk back to you.
How Acumen helps
We build clinical AI training data that starts clean. Practicing physicians, matched by specialty, author cases, reference answers and rubrics from their own expertise, under rules that prohibit real patient identifiers. We screen every item, adjudicate with a second reviewer, and keep a provenance record your privacy, security and regulatory teams can read. Evaluation sets stay held out and documented.
We are healthcare, life sciences and pharma only. Our documentation is built for the reviews your product has to pass, not added afterward.
See how it works on our Clinical AI Training Data page. For testing what you train, read How to evaluate a clinical LLM.
FAQ
What makes AI training data HIPAA compliant?
Data is HIPAA compliant for AI training when it contains no protected health information, or when any PHI is used under a permitted purpose with the right agreements, minimum necessary limits and Security Rule controls. Most teams rely on properly de-identified data or clinician-authored cases written without patient identifiers, backed by documentation of how every item was created.
Can PHI be used to train an AI model?
In some cases, but it is the hardest path. HIPAA has no AI-specific rule, so training is treated like any other use of PHI. You need a permitted purpose or authorization and the right agreements. Business associate agreements increasingly bar vendors from training on PHI without authorization, and some state laws restrict secondary uses such as model training.
What is the difference between Safe Harbor and Expert Determination?
Safe Harbor removes 18 categories of identifiers, such as names, small geographic units and specific dates, and requires no actual knowledge that the rest could identify someone. Expert Determination uses a qualified expert who applies statistical methods, concludes the identification risk is very small and documents the analysis. HHS sets no single numerical threshold for that standard.
Is clinician-authored data considered PHI?
Not when a clinician writes a new case from expertise without real patient identifiers. It is new content, not a patient record. That depends on enforced authoring rules and screening, because real details can slip into a case presented as a composite. Ask any vendor how they prevent and detect identifiers in clinician-written material.
Does FDA review AI training data?
For AI-enabled medical devices, yes. FDA's January 2025 draft guidance asks sponsors to describe how training data was collected, cleaned and processed, and how test data stays independent. FDA's August 2026 discussion paper on generative AI devices also points toward expert benchmarking and independent clinician review, which makes documented, expert-built data part of the evidence.
When will the new HIPAA Security Rule take effect?
It is not final. HHS proposed the overhaul in January 2025, including written inventories of AI tools that touch ePHI. In July 2026, Fierce Healthcare and Alston & Bird reported that the final rule had been pushed to July 2027. OCR has said the current Security Rule already applies to AI like any other technology.