AI Mortality Models Collapse When Moved Between Hospitals, Study Warns

Machine-learning models that promise to forecast which intensive-care patients will die may look impressive on paper and fail completely the moment they are moved to a different hospital. That is the stark conclusion of a new study published in Frontiers in Digital Health, in which researchers systematically transferred three clinical prediction models between two large intensive care databases and watched their decision-making fall apart in ways that conventional performance scores never revealed. The work offers one of the clearest demonstrations yet that the numbers hospitals rely on to certify an algorithm may say far less about real-world safety than anyone would like to believe.

The research team, led by Jorge Robalino-Díaz and William Villegas-Ch of Universidad de Las Américas in Quito, Ecuador, together with colleagues at Universidad UTE, focused on a problem that has quietly plagued clinical artificial intelligence for years: dataset shift. When a model is trained on data from one institution and deployed at another, differences in equipment, protocols, patient populations, and measurement practices subtly change the statistical landscape the algorithm must navigate. Prior studies have suggested that deep-learning models can lose between 10 and 25 percent of their performance when applied across populations, and that domain-adaptation techniques claw back only a modest 5 to 15 percent. The new study goes a step further by asking not just how much performance degrades, but how the failure actually looks inside the model’s decision machinery.

To do this, the team built a rigorous multi-environment evaluation using two well-known critical-care resources: the MIMIC-III clinical database, drawn from a single-center intensive care unit, and the eICU Collaborative Research Database, which pools data from many hospitals. Their primary cohort consisted of 110 MIMIC intensive care episodes, with 30 percent in-hospital mortality, and 1,324 eICU episodes, with a much lower mortality rate of 8.8 percent. Crucially, the researchers restricted every prediction to information available in the first 24 hours after admission, a so-called landmark design that prevents the model from cheating by peeking at measurements taken later in the stay. They also enforced a strict shared-feature rule: only variables that were sufficiently observed in both databases, such as heart rate, blood pressure, glucose, creatinine, and sodium, made it into the models.

Three representative algorithms were put to the test: Logistic Regression, the transparent linear workhorse of clinical statistics; Random Forest, an ensemble of decision trees; and Gradient Boosting, a sequential method that iteratively corrects its own errors. Each model was trained in one domain and evaluated in the other, a design that explicitly breaks the usual assumption that training and test data come from the same distribution. The headline numbers were sobering. In the transfer from eICU to MIMIC, the area under the ROC curve, a standard measure of a model’s ability to rank sick patients above healthy ones, ranged from just 0.598 for Logistic Regression to 0.642 for Random Forest and 0.613 for Gradient Boosting, barely above the 0.5 that corresponds to coin-flipping.

But the truly alarming findings emerged only when the researchers examined what happened at the decision threshold. Clinical models typically convert a predicted probability into a yes-or-no call by comparing it against a fixed cutoff, conventionally 0.5. When the team inspected the distribution of predicted probabilities, they found that Random Forest, despite its respectable AUC, produced no positive predictions at all after transfer: every single MIMIC patient was classified as a survivor, including all 33 who died. Even more unsettling, this failure was already visible during internal validation on eICU data, meaning the flaw was baked into the model from the start rather than being a new symptom of the hospital switch. Gradient Boosting detected only two of the 33 deaths, while Logistic Regression caught 18 but at the cost of mislabeling 33 of 77 surviving patients as likely to die.

The reverse transfer, training on MIMIC and testing on eICU, told a similar story with different flavors of failure. Logistic Regression detected 44 of 117 deaths but generated a flood of 422 false positives, a burden that would overwhelm any clinical workflow. Random Forest preserved high specificity, correctly clearing 97 percent of survivors, yet it found only 4 of the 117 patients who actually died. Gradient Boosting landed somewhere in between, detecting 31 deaths with a precision of just 0.109. In other words, the same transfer direction could fail as alarmism, as silence, or as near-total blindness, depending on the algorithm.

The researchers’ decision-level diagnostic analysis explains why. By plotting the predicted probabilities for survivors and non-survivors separately, they showed that a model can retain genuine ranking information, the ability to place sicker patients higher on its score, while the entire probability distribution slides below the fixed 0.5 threshold. When that happens, the model defaults to a constant negative prediction regardless of the true outcome, producing perfect-looking specificity alongside zero sensitivity. This is precisely the kind of structural failure that aggregate metrics such as accuracy and AUC-ROC cannot detect, because those measures summarize performance across all possible thresholds rather than at the single operating point where clinical decisions are actually made.

Calibration told a parallel story of misalignment. Using Brier scores, calibration curves, and expected calibration error computed over ten probability bins, the team documented substantial gaps between predicted probabilities and observed event rates, particularly after transfer. Logistic Regression’s calibration error reached 0.235 in the eICU-to-MIMIC direction, and post hoc recalibration revealed slopes far below the ideal value of one, indicating systematically distorted confidence. A sensitivity analysis using F1-maximizing thresholds found that the tree-based models would have needed cutoffs as low as 0.067 and 0.027 to recover meaningful sensitivity, confirming that the models were not devoid of signal but that their probability scales were badly out of sync with the conventional decision rule. Notably, the researchers caution that these optimized thresholds used test-set information and serve only as a diagnostic, not as deployable clinical cutoffs.

The study is careful about its own limits. The MIMIC demo cohort is small, the MIMIC-to-eICU direction rested on a weak and unstable source model, and the analysis is retrospective, so the authors frame their results as exploratory evidence of threshold-specific instability under dataset shift rather than a definitive estimate of deployment performance. They also excluded source-specific variables such as the eICU’s APACHE-predicted mortality score, which would have leaked the answer into the inputs, and they verified that class-weighting choices changed operating points without rescuing cross-domain performance. These methodological guardrails strengthen the central message rather than weaken it.

The implications reach well beyond the two databases studied here. As hospitals increasingly buy and share clinical AI tools, regulators and clinicians need to demand evidence of calibration, threshold behavior, feature harmonization, and operating-point stability across institutions, not just a flattering AUC from the developer’s own data. The authors call for future work using the full MIMIC and eICU resources, site-aware validation designs, thresholds locked from source-domain validation data or prespecified clinical utility criteria, and reporting aligned with emerging standards such as TRIPOD-AI and PROBAST-AI. Until then, the study stands as a warning: a model that never says yes cannot save lives, no matter how beautifully it ranks them.

Subject of Research: Cross-domain transportability and fixed-threshold decision behavior of ICU mortality prediction models under dataset shift

Article Title: Multi-environment evaluation and decision-level analysis of clinical prediction models under dataset shift

Article References: Robalino-Díaz, J., Cabrera-Andrade, A., Zambrano, A. K., & Villegas-Ch, W. (2026). Multi-environment evaluation and decision-level analysis of clinical prediction models under dataset shift. Frontiers in Digital Health, 8, Article 1874296. https://doi.org/10.3389/fdgth.2026.1874296

Image Credits: AI Generated

DOI: 10.3389/fdgth.2026.1874296

Keywords: clinical prediction models, dataset shift, ICU mortality, MIMIC-III, eICU database, machine learning, decision threshold, calibration, AUC-ROC, model transferability, health informatics, false positives