The evidence gap is striking. A 2025 meta-analysis in npj Digital Medicine examined 83 studies of generative AI diagnostic models, specifically large language and multimodal models rather than medical AI as a whole, and found a pooled diagnostic accuracy of just 52.1 percent, with a 95 percent confidence interval running from 47.0 to 57.1 percent. That figure sits 15.8 percentage points below expert physicians, a statistically significant shortfall. The authors of the meta-analysis caution that the included studies were highly heterogeneous and most carried a high risk of bias, so the pooled number should be read as an indicator of how weak the evidence base in this subfield is, rather than as a precise estimate of how these systems actually perform. Even so, it is a sobering baseline for a technology being deployed at scale in hospitals.

Two real-world sepsis prediction systems illustrate how dramatically outcomes can diverge. The Epic Sepsis Model, deployed across health systems worldwide, was reported to identify high-risk patients in 87 percent of cases. But when researchers restricted the analysis to data available before clinicians themselves suspected sepsis, that figure fell to 63 percent, and to 53 percent before a blood culture was ordered. In other words, the model had largely learned to detect clinician suspicion rather than the underlying physiology of the disease. In practice it missed roughly two-thirds of cases and generated so many false alarms that staff at one major health system acknowledged only 13 percent of its alerts in 2023 — a figure that reflects alert fatigue and workflow burden as much as raw model failure, but which renders the system nearly useless at the bedside.

Contrast that with TREWS, the Targeted Real-time Early Warning System developed at Johns Hopkins with sustained input from nurses and intensivists. Because it was built around clinical workflow from the start, a prospective multisite trial was able to demonstrate genuine patient benefit: reduced sepsis mortality, shorter duration of organ failure, and more timely administration of antibiotics. Mirugwe is careful to note that the two systems differ on several dimensions, including training data, the definition of the prediction target, deployment context, and institutional support, so the contrast cannot be reduced to a single cause. The clearest methodological difference, however, is that TREWS treated clinical co-design as a technical requirement from the outset rather than as an optional consultation, and this appears to be a major contributing factor in its more favourable real-world performance.

Beneath these individual failures lies a structural one: the datasets on which diagnostic algorithms are trained frequently do not reflect the populations they serve. Representation bias has become a dominant failure mode in medical AI. In a decade-long review of authorised devices, fewer than 30 percent disclosed the demographic composition of their training data, although disclosure norms have been shifting over the period examined. For much of that cohort, clinicians had no way of knowing whose biology the algorithm had actually learned from. The consequences of this opacity are not hypothetical. MIT researchers showed in 2024 that popular diagnostic models analysing chest radiographs had acquired a superhuman ability to predict a patient’s race, sex, and age from images alone. The models were exploiting demographic shortcuts that correlated with pathology in the training cohort — correlations that reliably break down when the tool is applied elsewhere.

Geography compounds the problem. A diabetic retinopathy algorithm validated at 87 percent sensitivity delivered only 60 to 80 percent sensitivity in public-health screening settings in northern India, with specificity falling as low as 14 percent. Part of the gap is attributable to the lower image quality of the non-mydriatic cameras used in routine screening compared with the pristine images used during validation. Same disease, same imaging modality, different population — and effectively a different machine. Tools trained at tertiary academic centres carry case-mix assumptions that simply do not hold in district hospitals in sub-Saharan Africa or rural Southeast Asia, precisely the settings where AI is most aggressively promoted as a solution to chronic specialist shortages.

A second structural failure concerns who builds these systems. AI diagnostic tools are typically constructed by engineers, evaluated by statisticians, and regulated by agencies whose frameworks were originally designed for drugs and static devices. The clinician who must integrate the model’s output with a patient’s history, physical examination, and ward context is routinely absent from the development process. The predictable results are alert fatigue, automation bias, and the silent override, in which clinicians quietly learn to ignore recommendations they have found unreliable. Evidence suggests that teams combining clinicians and data scientists produce more generalisable models, because clinicians understand which features are genuinely predictive and which are confounded by patterns of care. Excluding that knowledge, Mirugwe argues, is not a resourcing choice but an epistemic error.

The third failure is regulatory. Frameworks vary across jurisdictions but share one weakness: they use pre-market accuracy as the gateway to deployment without requiring prospective validation in the populations who will actually use the tool. In the United States, 96.4 percent of AI medical devices were cleared through the 510(k) pathway without any clinical trial evidence. Of the 950 AI-enabled devices the Food and Drug Administration had authorised through November 2024, 60 were subject to recall, accounting for 182 recall events in total, and 43 percent of those recalls occurred within a year of authorisation, mostly for diagnostic errors. The European Union’s AI Act of 2024 imposes the most demanding pre-market regime but does not yet specify outcome-level trial requirements. China requires localised trials for Class III devices, India’s 2025 draft guidance classifies AI imaging as Class C requiring clinical validation, and Japan uses an adaptive framework. Across every jurisdiction, demographic disclosure remains the exception rather than the rule, and lower-income settings remain exposed to tools validated entirely elsewhere.

Mirugwe proposes three achievable reforms. First, prospective external validation on demographically and geographically diverse populations should be a non-negotiable prerequisite for authorisation, with the existing TRIPOD+AI and STARD-AI frameworks already providing the methodological scaffolding. Second, performance must be disaggregated by clinically relevant subgroup: a tool with 92 percent overall accuracy but 74 percent accuracy in one subgroup is selectively unreliable, and the patients it fails are predictable. Demographic disclosure should become universal. Third, clinicians must become structural partners in development from problem definition onwards, not consultees reviewing a finished product. Achieving this at scale would require international coordination, and the International Medical Device Regulators Forum offers a ready mechanism through which harmonised standards could be adopted across jurisdictions. The obstacles are real — much training data is proprietary or retrospective, and privacy-preserving techniques such as federated learning can make full disclosure and prospective validation harder to mandate — but these are arguments for phasing requirements in and developing validation methods suited to distributed data, not for treating clinical readiness as optional.

The gap between algorithmic performance and clinical benefit, the article concludes, is not an engineering problem awaiting a better model. It is a governance problem created by norms that reward accuracy on curated datasets, regulatory pathways that mistake on-paper accuracy for clinical readiness, and a publication culture that treats external validation as optional. For a patient in a Lagos clinic whose retinal image is read by an algorithm trained on a different continent, the difference between 87 percent sensitivity in a trial and 60 percent in the field is a substantially higher risk of a missed diagnosis. An AI diagnostic tool that passes a benchmark test has not passed a clinical one. TREWS demonstrates that the gap can be closed where co-design and rigorous validation are taken seriously, but until regulation, publication standards, and funding criteria embed that distinction, the gap will remain the predictable default of the current system rather than an inevitable one.

Subject of Research: The gap between benchmark accuracy and clinical benefit in authorised AI diagnostic tools

Article Title: When the algorithm passes the test but fails the patient

Article References: Mirugwe, A. (2026). When the algorithm passes the test but fails the patient. PLOS Digital Health, 5(10), e0001774. https://doi.org/10.1371/journal.pdig.0001774

Image Credits: AI Generated

DOI: 10.1371/journal.pdig.0001774

Keywords: medical AI, diagnostic algorithms, benchmark accuracy, clinical validation, regulation, representation bias, sepsis prediction, Epic Sepsis Model, TREWS, FDA, EU AI Act, health equity

The research team, led by Miad Islam of Saint Leo University together with collaborators from institutions in Bangladesh, the United Kingdom, and the United States, built a framework that combines three complementary neural network architectures into a single sepsis prediction engine. The first component is a one-dimensional convolutional neural network, or 1D-CNN, which excels at scanning sequences of clinical measurements and picking out local patterns, much as a radiologist might scan an image for telling features. Layered on top of that is a bidirectional long short-term memory network, or BiLSTM, a recurrent architecture that reads the patient’s clinical timeline in both directions, capturing how early vital-sign changes foreshadow later laboratory abnormalities and how recent developments reframe earlier ambiguity. The final piece is a Transformer attention block, the same family of architecture that powers modern language models, which allows the network to weigh the relative importance of different clinical variables and time points dynamically rather than treating every input as equally significant. Together, these components form a model capable of learning the subtle, multi-scale signatures that precede sepsis onset.

The privacy mechanism at the heart of the study is federated learning, a training paradigm in which the data never moves. Instead of shipping records to a central server, each participating hospital trains the model locally on its own patients. Only the resulting model parameters, essentially long lists of numerical weights, are transmitted to a coordinating server, which averages them using an algorithm known as Federated Averaging, or FedAvg. The updated global model is then sent back to each site for another round of local training. Over many such rounds, the shared model gradually absorbs the statistical diversity of all participating institutions without any single record, diagnosis, or lab value ever being exchanged. For the purposes of this study, the researchers simulated this arrangement by partitioning a MIMIC-IV-style ICU dataset across three virtual hospital clients, allowing them to evaluate how the federated approach behaves under realistic distributed conditions while working with data that posed no privacy risk.

The results are striking for a model trained under such constraints. The federated hybrid framework achieved an accuracy of 93.6 percent and an area under the receiver operating characteristic curve, or AUROC, of 0.959, a standard measure of a classifier’s ability to distinguish septic from non-septic patients across all possible thresholds. Precision came in at 0.871, meaning that when the model raised an alarm it was usually justified, and specificity reached 98.24 percent, indicating it rarely flagged healthy patients falsely. The F1-score, which balances precision against recall, was 0.759, a figure that reflects the inherent difficulty of detecting a condition that is rare relative to the ICU population. For comparison, the researchers also trained a centralized baseline model on all the data pooled together, and it reached an AUROC of 0.997. That gap is real, but the federated model’s performance remains firmly in the range clinicians would consider useful, and it was achieved without the data centralization that privacy law makes impractical.

Perhaps even more consequential for real-world deployment is the framework’s communication efficiency. In federated systems, the cost of repeatedly shipping model weights between hospitals and the coordinating server can become prohibitive, particularly for institutions with limited bandwidth. Across all federated training rounds in this study, the total communication cost was just 36.09 megabytes, a figure small enough to travel over ordinary hospital networks in seconds. That efficiency matters because it suggests the approach could scale to many more participating sites without the coordination overhead becoming a bottleneck. In distributed healthcare environments, where connectivity is often uneven and IT infrastructure varies widely between a major academic medical center and a community hospital, keeping the communication footprint light is not a luxury but a prerequisite.

A model that clinicians cannot understand is a model they will not trust, so the researchers turned to SHAP, or SHapley Additive exPlanations, a technique borrowed from cooperative game theory that assigns each input variable a quantified contribution to every individual prediction. The SHAP analysis identified the Sequential Organ Failure Assessment score, a composite measure of dysfunction across six organ systems, as the most influential predictor of sepsis risk, followed by lactate level, white blood cell count, creatinine, and procalcitonin. Each of these is a familiar marker to any intensivist: lactate rises when tissues are starved of oxygen, white blood cells surge or crash during systemic infection, creatinine signals kidney injury, and procalcitonin is a well-established biomarker of bacterial sepsis. The fact that the model’s attention converged on variables that already carry clinical weight is reassuring, because it suggests the network learned genuine physiology rather than exploiting statistical artifacts in the data.

The authors are candid about the limits of their work, and their honesty is itself instructive. The study used a MIMIC-IV-style dataset rather than live records from multiple hospitals, and the federated setup was a simulation rather than a deployment across real institutions with genuinely heterogeneous patient populations, equipment, and documentation practices. The researchers also note that they did not formally define an adversary or threat model. They did not assess whether a malicious server or a colluding client could infer patient-level information from the exchanged model updates, a known vulnerability in federated systems where gradient information can sometimes leak details about training data. Techniques such as differential privacy, secure aggregation, and homomorphic encryption exist to close these gaps, and the authors flag rigorous validation on real multi-institutional ICU records as a necessary next step before any claim of scalable, secure deployment can be made.

Those caveats notwithstanding, the study lands at a moment when the tension between data hunger and data privacy has become the defining challenge of clinical artificial intelligence. The most accurate models are typically those trained on the largest and most diverse datasets, yet the most diverse datasets are precisely the ones that privacy law keeps fragmented across institutions. Federated learning offers a mathematically principled compromise, and pairing it with architectures that capture both local temporal patterns and long-range dependencies, as this hybrid CNN-BiLSTM-Transformer design does, may prove to be a template for predictive medicine well beyond sepsis. Acute kidney injury, respiratory failure, and cardiac deterioration are all conditions where early, distributed, privacy-preserving prediction could change outcomes.

What makes this work compelling as a piece of the larger puzzle is its demonstration that privacy and performance need not be framed as a zero-sum trade. A model trained without ever seeing another hospital’s records came within a few percentage points of the centralized ideal, and it did so with a communication budget measured in tens of megabytes and with explanations that map onto established clinical intuition. The road from a three-client simulation to a network of hospitals sharing a living model is long, and it will require threat modeling, regulatory negotiation, and prospective clinical validation. But the direction of travel is clear. If the next generation of ICU decision support can be trained collectively while respecting the sovereignty of every patient record, the hours that matter most in sepsis may finally be spent on treatment rather than detection.

Subject of Research: Privacy-preserving federated deep learning for sepsis prediction in intensive care units

Article Title: A hybrid deep learning framework for privacy-preserving sepsis prediction in distributed ICU environments using federated learning simulation

Article References: Islam, M., Mohiuddin, T., Rahman, M. A., Sharfuddin, M., Islam, M. S., Sunny, S. R., & Lokesh, E. (2026). A hybrid deep learning framework for privacy-preserving sepsis prediction in distributed ICU environments using federated learning simulation. Discover Social Science and Health. https://doi.org/10.1007/s44155-026-00489-1

Image Credits: AI Generated

DOI: 10.1007/s44155-026-00489-1

Keywords: federated learning, sepsis prediction, deep learning, ICU, electronic health records, privacy-preserving AI, Transformer, BiLSTM, SHAP explainability, clinical decision support, health informatics, FedAvg