The FDA cleared 950 AI-enabled medical devices by the end of 2023. That number has roughly tripled since 2019, and the regulatory frameworks governing those devices have not kept pace in a straight line. What has emerged instead is a layered, sometimes overlapping set of requirements spanning the FDA, HIPAA, the Joint Commission, and newer frameworks like the HAIRA model that healthcare AI teams need to navigate simultaneously.
This is not a theoretical compliance problem. An AI diagnostic tool that performs brilliantly in a clinical trial can still expose a health system to enforcement risk if its governance structure fails to address Protected Health Information (PHI) handling, post-market performance monitoring, or accreditation standards. Getting one piece right while ignoring the others is increasingly not an option.
What follows is a practical breakdown of each framework, how they interact, and what responsible AI governance in healthcare actually looks like in 2025.
Why AI Governance Is Different in Healthcare
Most industries deploying AI face questions about accountability and bias. Healthcare adds three additional stakes: patient safety, privacy protections with criminal penalties, and accreditation standards tied to reimbursement. Miss a governance requirement in retail and you might face a fine. Miss one in healthcare and the consequences can include patient harm, HIPAA enforcement, FDA warning letters, and loss of Joint Commission accreditation.
AI governance in healthcare sits at the intersection of four distinct regulatory and standards-setting bodies, each with its own authority, enforcement mechanism, and scope:
- The FDA governs AI as a medical device or component of a medical device.
- HIPAA governs how AI systems handle patient data.
- The Joint Commission sets accreditation standards that incorporate AI safety and quality.
- The HAIRA model provides a risk assessment framework specifically designed for healthcare AI.
Understanding what each body requires, and where their requirements overlap, is the foundation of any defensible healthcare AI governance program.
FDA Regulation of AI in Healthcare: Software as a Medical Device
The FDA’s approach to AI in healthcare runs through a single foundational concept: Software as a Medical Device (SaMD). If an AI system meets the definition of a medical device, it falls under FDA oversight regardless of whether it is embedded in physical hardware. A standalone algorithm that reads chest X-rays and flags pulmonary nodules is a medical device. An AI that analyzes ECG data and predicts arrhythmia risk is a medical device. An administrative scheduling tool is not.
The distinction matters because FDA medical device classification determines the compliance pathway:
Device Class | Risk Level | Examples | Pathway |
Class I | Low risk | Administrative AI, general wellness | General controls, often exempt |
Class II | Moderate risk | AI-assisted radiology, triage support | 510(k) premarket notification |
Class III | High risk | AI for life-sustaining decisions | Premarket Approval (PMA) |
The Predetermined Change Control Plan (PCCP)
The most practically significant recent development in FDA AI policy is the Predetermined Change Control Plan. Traditional device regulation assumes a fixed product: you submit a device, it gets cleared, it stays that way. Machine learning models are not fixed products. They can drift, retrain, and update continuously, which creates a fundamental tension with static approval frameworks.
The PCCP addresses this by allowing manufacturers to submit, upfront, a description of the modifications they anticipate making and the validation protocols they will follow for each change. An approved PCCP means those modifications can be implemented without triggering a new 510(k) submission, provided the changes stay within the approved boundaries. This is not a blank check for continuous modification; it requires rigorous specification of modification scope and performance evaluation criteria.
For healthcare AI governance teams, the PCCP represents both a compliance mechanism and a governance discipline. Building a robust PCCP requires mapping anticipated model changes, defining performance metrics, and establishing monitoring protocols before deployment, which are exactly the governance activities that responsible AI programs should be doing anyway.
Post-Market Surveillance Requirements
FDA’s 2021 AI Action Plan signaled a clear intent to strengthen post-market surveillance for AI-enabled devices. The concern is not just initial safety but ongoing safety as models encounter real-world data distributions that differ from training data.
Current guidance requires SaMD manufacturers to maintain performance monitoring systems capable of detecting clinically meaningful degradation. For adaptive AI systems, this means tracking model drift, demographic performance disparities, and edge case failures. The FDA has specifically flagged algorithmic bias as a post-market surveillance concern, pointing to documented cases where commercially deployed AI tools performed significantly worse on underrepresented patient populations.
HIPAA and AI: Where Privacy Law Meets Machine Learning
HIPAA does not mention artificial intelligence. It was enacted in 1996 and amended primarily in 2013 through the HIPAA Omnibus Rule, both well before machine learning in healthcare became a meaningful compliance consideration. What HIPAA does contain is a framework for Protected Health Information that applies to AI systems in ways that create specific governance obligations.
When AI Systems Become a HIPAA Risk
Three scenarios create the most significant HIPAA exposure in healthcare AI deployment:
Training data governance. AI models trained on patient records are training on PHI. The use of PHI for purposes beyond the original treatment, payment, or healthcare operations purpose requires authorization or a formal research framework. Many health systems have deployed AI using data that was collected for clinical operations and repurposed for model training without adequate authorization structures. The HHS Office for Civil Rights has not issued AI-specific HIPAA guidance as of 2025, but general PHI use principles apply.
Vendor management and Business Associate Agreements. When a health system uses an external AI vendor to process patient data, that vendor is a Business Associate under HIPAA. The Business Associate Agreement (BAA) must specifically address AI processing, data retention, model training use of PHI, and breach notification. Generic BAAs that predate AI deployment frequently fail to cover these specifics.
Inferential data creation. This is the most legally uncertain area. When an AI system generates a new prediction or risk score from patient data, that derived output may itself constitute PHI if it can be linked to an identifiable individual. A model that generates a readmission risk score for a specific patient has created what most privacy lawyers would treat as PHI, even though the raw score did not exist in the source data.
The Minimum Necessary Standard and AI
HIPAA’s minimum necessary standard requires covered entities to limit PHI access to the minimum necessary to accomplish the intended purpose. For AI governance, this translates directly into data minimization requirements during model development and deployment. A training dataset that includes more patient identifiers than the model needs to perform its function may violate this standard.
Governance programs should document the data elements used in AI training, the justification for each element’s inclusion, and the de-identification or anonymization approach applied. The Safe Harbor method and the Expert Determination method under HIPAA offer two compliance pathways for de-identifying training data, each with different governance implications.
The HAIRA Model: A Framework Built for Healthcare AI Risk
The Health AI Risk Assessment (HAIRA) model was developed specifically to address the gap between general AI risk frameworks and the specific context of healthcare deployment. While the NIST AI Risk Management Framework (AI RMF) and ISO/IEC 42001 provide useful general structures, HAIRA addresses risk dimensions that are unique to clinical settings.
HAIRA organizes healthcare AI risk across five domains:
HAIRA Domain | Risk Focus | Governance Action |
Clinical Safety | Direct patient harm from AI error | Clinical validation, monitoring protocols |
Data Integrity | Training data quality, PHI handling | Data governance, lineage tracking |
Algorithmic Fairness | Performance disparities across populations | Bias audits, demographic performance analysis |
Operational Risk | Workflow integration, clinician dependency | Change management, override protocols |
Regulatory Compliance | FDA, HIPAA, accreditation requirements | Compliance mapping, documentation |
HAIRA and the Clinical Validation Imperative
One of HAIRA’s most important contributions is its emphasis on clinical validation as distinct from technical validation. A model can achieve high AUC scores on a held-out test set and still perform unpredictably when deployed in a clinical workflow. Clinical validation asks whether the AI produces reliable outputs when integrated into how clinicians actually work, including the time pressures, documentation requirements, and handoff processes of real care environments.
HAIRA’s clinical safety domain specifically requires prospective validation in the target deployment setting before go-live, not just retrospective validation on historical data. For health systems operating under Joint Commission oversight, this aligns directly with accreditation expectations around clinical decision support governance.
Algorithmic Fairness Requirements in HAIRA
Healthcare AI has a documented bias problem. A landmark 2019 study published in Science found that a widely used health system algorithm systematically underestimated the illness severity of Black patients, directing fewer resources to them than to equally sick white patients. The bias was not intentional; it emerged from using healthcare costs as a proxy for health needs, which reflected historical disparities in healthcare access.
HAIRA requires formal algorithmic fairness assessment as part of pre-deployment risk evaluation. This means stratified performance analysis across race, ethnicity, age, sex, socioeconomic status, and geographic factors relevant to the use case. Health systems that skip this step are not just missing a governance best practice; they are taking on both patient safety risk and regulatory exposure as the FDA increasingly treats demographic performance disparities as a post-market surveillance concern.
Joint Commission Standards and Healthcare AI Accountability
The Joint Commission accredits more than 22,000 healthcare organizations in the United States, and its accreditation standards have direct financial consequences: Medicare and Medicaid reimbursement eligibility depends on accreditation status for most health systems. When the Joint Commission updates its standards to address AI, the entire industry responds.
The Joint Commission’s approach to AI governance operates through three primary channels: clinical decision support standards, leadership accountability requirements, and patient safety event reporting.
Clinical Decision Support (CDS) Governance Standards
The Joint Commission’s clinical decision support standards apply to AI tools that influence clinical decision-making. The standards require that CDS tools, including AI-based tools, be reviewed and approved through a formal clinical governance process before deployment. This means a multidisciplinary review that includes clinical informatics, physician representation, nursing leadership, compliance, and legal.
Post-deployment, the standards require ongoing monitoring of CDS performance and a formal mechanism for clinicians to report concerns, override recommendations, and trigger review of a tool that appears to be generating inappropriate guidance. The ability to override and the process for escalating override patterns are not optional features; they are governance requirements.
Leadership and Accountability Structures
Joint Commission leadership standards require that executive leadership take accountability for AI governance, not just delegate it to IT or informatics departments. This has practical implications for governance program design. An AI governance committee that sits exclusively in a technology department and lacks C-suite representation and clinical leadership does not meet accreditation expectations.
The accountability chain the Joint Commission expects looks something like this: the board sets AI governance policy direction, executive leadership owns the AI risk management program, clinical and operational leadership governs specific deployment decisions, and frontline staff have clear channels for reporting concerns.
Patient Safety Event Reporting and AI
When an AI system contributes to a patient safety event, the Joint Commission’s sentinel event and adverse event reporting requirements apply. The governance challenge is attribution: in a complex clinical workflow where multiple decision-support tools, clinical judgment, and process factors interact, determining the AI’s contribution to an adverse outcome requires retrospective analysis capability that many current AI deployments lack.
Health systems with mature AI governance programs build this into deployment architecture: logging AI recommendations alongside clinical actions, maintaining audit trails that can reconstruct the decision environment at the time of care, and establishing clear protocols for AI-involved incident review.
How FDA, HIPAA, HAIRA, and the Joint Commission Interact
These four frameworks do not cover the same ground, and they do not conflict with each other, but they do require governance programs to address different aspects of the same AI deployment. A practical way to think about the relationship:
- The FDA governs what the AI can do and whether it is safe to deploy as a medical device.
- HIPAA governs how the AI handles patient data throughout its lifecycle.
- HAIRA provides the risk assessment methodology for evaluating the AI before and after deployment.
- The Joint Commission governs how the AI is integrated into clinical operations and organizational accountability.
The governance implication is that a compliant healthcare AI program requires all four frameworks to be addressed. An AI system can be FDA-cleared, HIPAA-compliant in its data handling, and still expose a health system to Joint Commission deficiencies if clinical oversight structures are inadequate. HAIRA provides the connective tissue, offering a risk assessment process that maps to all three regulatory frameworks simultaneously.
Health systems that have successfully integrated these requirements typically do so through a single AI governance framework that uses HAIRA as the risk assessment backbone, maps each HAIRA domain to the applicable FDA, HIPAA, and Joint Commission requirements, and assigns clear ownership for each compliance obligation.
Building a Healthcare AI Governance Program: Practical Requirements
Theoretical compliance frameworks become operational through specific governance structures. Based on the requirements across all four frameworks, a healthcare AI governance program needs to address six concrete areas:
1. Pre-Deployment Clinical and Regulatory Review
Every AI system that touches patient care, patient data, or clinical decision-making requires a structured pre-deployment review. The review committee should assess FDA classification and compliance pathway, PHI handling and HIPAA compliance, HAIRA risk score and mitigation plan, and clinical validation evidence. The review outcome should be documented with specific approval conditions, monitoring requirements, and a defined review trigger that would require re-evaluation.
2. Data Governance for AI Training and Operation
A standalone data governance program that covers AI training data needs to address the source, authorization basis, de-identification method, retention policy, and audit trail for every dataset used in model development. For AI systems using real-time patient data in operation, the program needs to address how PHI flows through the model, where outputs are stored, and how those outputs are classified under HIPAA.
3. Performance Monitoring and Drift Detection
Deployed AI systems require ongoing performance monitoring that goes beyond uptime metrics. Clinical performance monitoring should track accuracy metrics stratified by patient demographic groups, flag statistically significant performance changes, and generate alerts when model outputs diverge from expected ranges. The monitoring program should be designed to satisfy both FDA post-market surveillance expectations and Joint Commission CDS monitoring standards.
4. Clinician Training and Override Protocols
Joint Commission accreditation expects that clinical staff who interact with AI tools understand how those tools work at a functional level, what the tools’ known limitations are, and how to report concerns or override recommendations. Training documentation and override tracking are both governance artifacts that surveyors will review.
5. Incident Investigation Protocols
When AI is involved in or adjacent to a patient safety event, the investigation protocol needs to capture AI-specific elements: what recommendation the AI made, what information was available to the AI at the time, whether the recommendation was overridden, and whether the AI’s output was a contributing factor. Retroactively building this investigation capability after an incident is significantly harder than designing it into deployment architecture.
6. Vendor Due Diligence and Contract Requirements
Health systems increasingly deploy AI through external vendors, making vendor governance a core program component. Vendor due diligence should assess FDA clearance status, HIPAA Business Associate Agreement adequacy, post-market surveillance practices, bias testing documentation, and contractual rights to audit model performance. The BAA should specifically address AI training data use, model retraining practices, and data breach responsibilities associated with AI processing.
Frequently Asked Questions
Does every AI tool used in a hospital require FDA clearance?
Not every AI tool qualifies as a medical device under FDA’s definition. Administrative AI tools such as scheduling, billing, and operational analytics generally do not require clearance. AI tools that influence clinical decision-making, diagnose conditions, or process clinical data to inform treatment typically do fall under FDA SaMD regulation. When in doubt, the FDA’s Decision Support Software guidance and the IMDRF SaMD framework provide the primary classification criteria.
What is the difference between de-identification and anonymization under HIPAA?
HIPAA provides two formal de-identification methods: Safe Harbor, which requires removal of 18 specified identifiers, and Expert Determination, which requires a qualified statistical expert to certify that re-identification risk is very small. Anonymization is not a HIPAA term; de-identification is. For AI training data governance, health systems should document which method was applied and maintain that documentation as part of the AI governance record.
How does the HAIRA model relate to the NIST AI Risk Management Framework?
HAIRA is a healthcare-specific implementation of broader AI risk management principles, including those in the NIST AI RMF. Where NIST AI RMF provides a general-purpose risk management structure applicable across industries, HAIRA applies that structure to clinical and regulatory contexts specific to healthcare. Organizations that have already implemented NIST AI RMF can treat HAIRA as a sector-specific overlay that adds clinical safety, algorithmic fairness, and HIPAA compliance dimensions.
What does Joint Commission accreditation require for AI governance documentation?
Joint Commission surveyors reviewing AI governance will typically look for evidence of formal CDS approval processes, ongoing monitoring documentation, clinician training records, override tracking, and leadership accountability structures. Specific documentation requirements vary by accreditation program (hospital, ambulatory, home care), but the underlying expectation is that AI deployment follows the same governance rigor applied to other clinical decision support tools.
Can AI-generated clinical outputs be considered PHI under HIPAA?
Most privacy counsel and the general HHS interpretation suggest that AI-generated outputs linked to an identifiable individual, such as a risk score or a diagnosis suggestion, constitute PHI. This means those outputs are subject to the same access, use, and disclosure rules as other PHI. Health systems should build this assumption into their AI governance programs and ensure that generated outputs are stored, accessed, and audited accordingly.
What is a Predetermined Change Control Plan and when is it required?
A PCCP is an FDA mechanism that allows AI medical device manufacturers to pre-specify planned modifications and the validation methods for those modifications. It is not currently required for all AI devices but is strongly encouraged for adaptive machine learning models. An approved PCCP allows approved modifications to proceed without a new premarket submission, making it both a regulatory compliance tool and a practical enabler of model improvement.
The Governance Foundation Healthcare AI Cannot Skip
The frameworks covered here: FDA SaMD regulation, HIPAA’s PHI requirements, the HAIRA risk model, and Joint Commission accreditation standards, were each developed for different purposes by different bodies. What they share is a common concern about patient safety and accountability that healthcare AI governance programs must reflect in their design.
The practical starting point for most health systems is a gap analysis that maps current AI deployments against each framework’s requirements. That analysis will surface the specific documentation, monitoring, and oversight gaps that need to be addressed. From there, building a unified governance program that addresses all four frameworks simultaneously is more efficient than trying to maintain separate compliance programs for each.
GAICC’s ISO/IEC 42001 training programs are designed to build the AI governance expertise that healthcare organizations need to meet these requirements. Explore our certification pathway to develop the skills to lead compliant, responsible AI programs in healthcare settings.
