Hospitals Are Deploying AI Faster Than Regulators Can Keep Up, Landmark Review Warns

America’s hospitals have quietly become testing grounds for one of the most consequential technological transformations in modern medicine. By the end of 2025, the U.S. Food and Drug Administration had cumulatively authorized more than 1,451 artificial intelligence and machine learning-enabled medical devices, and deep learning models now read radiological images with accuracy comparable to expert radiologists, predict sepsis onset hours before clinicians can detect it, and inform oncology decisions through precision genomic analysis. Beyond the bedside, AI systems automate clinical documentation, optimize administrative workflows, and stratify patient risk across entire health systems. The global healthcare AI market was valued at roughly $14.4 billion in 2023, with hospitals investing billions on the promise of sharper diagnostics, lighter clinician workloads, and better outcomes. Yet according to a new systematic review and regulatory gap analysis published in Frontiers in Digital Health, the governance frameworks meant to keep patients safe have not kept pace with this explosive deployment—and the consequences are already measurable.

The study, led by Joseph Conteh of the University of New Mexico together with Naim Kapucu, Eunice F. Ebevuhe, and Oluwapelumi Wumi Adeniji, was conducted and reported in accordance with the PRISMA 2020 reporting guideline, with study selection governed by predefined PICOS eligibility criteria. The team searched four databases—PubMed, Web of Science, Scopus, and Google Scholar—for peer-reviewed articles published from January 2017 through January 2026, with final searches completed between January 15 and 18, 2026. Of 1,065 records identified, just 22 studies met the eligibility criteria. Methodological quality was appraised with design-appropriate tools: the Critical Appraisal Skills Programme checklist for qualitative and certain observational designs, the Newcastle-Ottawa Scale for cohort and case-control studies, and QUADAS-2 for diagnostic accuracy studies. A narrative synthesis was followed by a structured regulatory gap analysis of federal statutes, regulations, and agency guidance, in which a gap was recorded only when at least two independent sources converged. Inter-rater agreement at the title and abstract screening stage was substantial, with Cohen’s kappa of 0.84.

The review’s central diagnosis is that the federal regulatory architecture was built for a fundamentally different technological paradigm. Authority over hospital AI is dispersed across agencies that each hold a partial slice of oversight: the FDA evaluates safety and effectiveness of Software as a Medical Device through pre-market review; the Office of the National Coordinator for Health Information Technology certifies health IT and mandates transparency for predictive decision-support interventions through the HTI-1 rule; the Centers for Medicare and Medicaid Services shapes adoption through conditions of participation, quality reporting, and payment policy; HIPAA governs the privacy and security of health data; and the Federal Trade Commission polices deceptive or unfair AI claims. In a decentralized federal environment, state legislation has emerged as a supplementary force, producing a patchwork landscape—California, for example, has established some of the nation’s strictest privacy and transparency standards, requiring healthcare institutions using AI-assisted diagnostic tools to inform patients about the system’s functionality, diagnostic basis, and potential risks.

The most technically striking finding concerns the chasm between pre-market validation and real-world performance. A COVID-19 chest radiograph AI diagnostic tool that achieved an area under the receiver operating characteristic curve greater than 0.95 on open datasets dropped to approximately 0.70 after prospective deployment across multiple hospital settings, varying from 0.66 to 0.74 across time quartiles over 19 post-deployment weeks. Predictive models delivering AUC values of 0.96 for readmission prediction and 0.83 from roughly 280,000 cases were tested only on retrospective data without external validation. The review is careful to distinguish deficiencies in the evidence submitted to the FDA from deficiencies in FDA requirements themselves—many validation gaps originate in the pre-submission research pipeline rather than in statutory standards, a distinction that matters because the remedies differ. Even stringent validation systems such as APPRAISE-AI, which rates methodology, results, and reproducibility, do not fully account for temporal and situational variation in clinical settings.

When regulatory-grade evidence does exist, the results can be dramatic. The KATE triage system outperformed nurses by 26.9% in general acuity assignment across 729 reviewed patient records, decreased undertriage from 19.8% to 9.7% and overtriage from 20.4% to 14.4%, and captured 1,880,841 features at 12.79 per encounter with 98.47% accuracy. An FDA-authorized AI tool for pulmonary embolism detection achieved 99.9% negative predictive value across 11,736 real-world imaging tests and slashed worklist detection times from 7,714 minutes to 87 minutes while reducing missed cases from 44.8% to 2.6%. A machine learning sepsis prediction algorithm enabled antibiotic delivery within 2.76 hours, cutting in-hospital mortality from 21.3% to 9.0%—a 58% relative reduction. But these successes remain exceptions rather than the norm, and the review found that AI’s relationship with CMS quality metrics is mixed: in a longitudinal observational study spanning 70 CMS quality indicators from 2022 to 2025, hospitals using AI showed positive trends for pneumonia mortality and hospital-acquired conditions but negative trends for readmission rates and sepsis treatment compliance. Because these are observational associations, they do not establish causal effects.

The structured gap analysis identified four critical, structurally interconnected failures. The first is the post-market monitoring gap: only 58.1% of hospitals in a national sample performed post-implementation monitoring in 2024, and the majority of those lacked continuous drift detection. In related survey data, 47.4% of hospitals did not evaluate post-deployment accuracy, 51.4% had no bias assessment protocols, and in 61% of hospitals post-deployment accuracy was assessed but bias was not, with all monitoring remaining sporadic. The second is the equity assurance gap. The training dataset of a readmission prediction model included less than 2% Hispanic and Asian participants while Caucasian patients made up more than 75%, with no subgroups evaluated for differential performance. The KATE system showed an 11.4 percentage point accuracy disparity between adults (78.5%) and children (67.1%) without subgroup analyses for race, ethnicity, or socioeconomic status. The review also distinguishes algorithmic bias from structural adoption disparities: larger hospitals had a 15.2% higher probability of adopting machine learning than smaller ones, health system-owned hospitals had a 26.8% higher probability than independently owned ones, and regions of high healthcare need in low-adoption areas had persistently lower adoption probabilities—including ratios of 0.75 for primary care Health Professional Shortage Areas and 0.74 for high Area Deprivation Index areas.

The third failure is the accountability clarification gap, which the review dissects into four frequently conflated dimensions: legal liability, clinical responsibility, institutional governance, and vendor obligations. Radiologists bore responsibility for diagnosing pulmonary embolism, yet responsibility allocation for 12 cases of missed diagnosis remained unclear. An AI-clinician hybrid framework for COVID-19 diagnosis did not determine responsibility distribution among developers, informaticians, and treating physicians, and a randomized clinical trial of sepsis prediction specified no clear distribution of responsibility in case of patient harm. The problem is compounded by market concentration: 79% of hospitals use predictive models created by electronic health record vendors, yet no frameworks exist for determining accountability between hospitals, developers, vendors, and clinicians. The fourth is the governance capacity gap, which operates as a cross-cutting constraint across the others. Only 65% of hospitals used predictive models and only 44% assessed biases, with higher implementation rates concentrated among larger and more profitable institutions. Even best-practice tools illustrate the barrier: DEPLOYR required Azure Functions, specialized Python libraries, data warehouse access with institutional review board approval, and registration at the Epic App Orchard, while the MedPerf federated benchmarking initiative demanded collaboration between more than 20 firms, 20 universities, and 9 hospitals across 13 countries, encountering regulatory, legal, and reputational barriers throughout.

The documented harms lend urgency to these technical findings. The widely cited study by Obermeyer and colleagues demonstrated that a commonly used healthcare algorithm systematically underestimated illness severity among Black patients, resulting in only 17.7% being identified as eligible for advanced care programs when a corrected, unbiased algorithm would have identified 46.5%. Insufficiently externally validated sepsis prediction technologies have generated alert fatigue that paradoxically diminishes clinician responsiveness to genuine patient deterioration signals. Survey data show that 73.1% of clinicians regarded lack of trust, transparency, or bias as critical barriers to implementation, and evaluations of large language models found that GPT-4 provided consistent responses in only 67 of 80 clinical scenarios (83.8%), with both GPT-3.5 and GPT-4 demonstrating weakness in recognizing diversity, equity, and inclusion principles. The review notes that the HTI-1 rule’s transparency mandates, while a positive step, suffer from underdeveloped enforcement mechanisms and unstandardized disclosure content.

International comparisons sharpen the picture. The European Union Artificial Intelligence Act imposes binding pre-market conformity assessment for high-risk AI systems, including most medical AI, and centralizes enforcement in designated market-surveillance authorities while assigning explicit obligations to both providers and deployers. U.S. requirements, by contrast, remain dispersed across FDA, ONC, and CMS authorities, with enforcement divided among federal and state actors holding partial jurisdiction, and post-market mechanisms such as Predetermined Change Control Plans still in early implementation. Drawing on these models, the authors propose a lifecycle governance framework with six evidence-linked components: rigorous pre-market clinical validation with standardized benchmarking, mandatory post-deployment monitoring with automated drift detection and performance alerts, standardized transparency and explainability aligned with the HTI-1 rule, equity-centered assessment protocols requiring subgroup analysis and bias reporting, clear accountability frameworks delineating responsibility among developers, vendors, institutions, and clinicians, and scalable governance infrastructure with targeted support for under-resourced settings.

The authors acknowledge key limitations: the retrospective registration of the review protocol on the Open Science Framework, the restriction to English-language publications, database coverage limited to four sources while Embase, CINAHL, and ClinicalTrials.gov were excluded, and the heterogeneity of the included evidence, much of it retrospective and single-center. International studies from Canada, Saudi Arabia, and the Netherlands were used to contextualize rather than as direct evidence of U.S. regulatory performance. Still, the core conclusion stands on firm ground: industry projections indicate AI-enabled technologies will influence decision-making in most major U.S. health systems within the coming decade, and if the identified gaps persist as deployment scales, the risks to patient safety, equity, and clinical trust will grow commensurately. The review closes by grounding its call in medicine’s oldest ethical principle—primum non nocere, first, do no harm—arguing that only a national, lifecycle-focused, systems-based governance approach can ensure that hospital AI remains safe, ethical, and trustworthy as it reshapes clinical care.

Subject of Research: A systematic review and regulatory gap analysis of AI governance in U.S. hospitals, identifying four critical regulatory gaps and proposing a lifecycle governance framework.

Article Title: A framework for safe, ethical, and trustworthy AI deployment and integration in hospitals: a systematic review and regulatory gap analysis with emphasis on the U.S. regulatory context

Article References: A framework for safe, ethical, and trustworthy AI deployment and integration in hospitals: a systematic review and regulatory gap analysis with emphasis on the U.S. regulatory context. (n.d.). https://doi.org/10.3389/fdgth.2026.1955958

Image Credits: AI Generated

DOI: 10.3389/fdgth.2026.1955958

Keywords: AI governance, hospital AI, FDA regulation, health equity, patient safety, systematic review, regulatory gaps, digital health, framework, safe, ethical, trustworthy