How to build clinical prediction models that survive new populations: a scoping review maps the methods

A clinical prediction model that performs brilliantly in the hospital where it was built can quietly fall apart the moment it is applied elsewhere. A new scoping review, published on 1 October 2026 in the open-access journal Diagnostic and Prognostic Research, has systematically mapped the methods researchers have proposed for building such models so that they hold up when they meet new populations — and it found that only a small, scattered literature exists, spanning both medical statistics and machine learning.

The review, led by Kritchavat Ploddi with Glen P. Martin, Maurice O’Connell and Matthew Sperrin, defines the problem precisely. Generalisability refers to a clinical prediction model maintaining its predictive performance in a target population that overlaps with the data it was developed on — for example, a model built on hospitalised patients in Manchester being evaluated across all UK hospitals. Transportability refers to maintaining performance in a target population that is entirely separate from the development data, such as taking that same hospital model to patients in primary care. Both are typically assessed through external validation, but the authors argue that external validation alone captures performance on a single dataset at a single moment and says nothing about how a model will behave in changing environments.

The core difficulty is data distribution shift. Prediction models are usually developed to minimise predictive error in the training data, often using penalised likelihood to avoid overfitting, but that does not guarantee the error is minimised in the target population. The shift can arise from changes in technology, such as different coding definitions across electronic health record platforms used by different hospitals; from changes in population or setting, such as moving a model developed in hospitalised patients to a community population; or from behaviour, since adopting an effective prediction model can improve treatment outcomes and thereby gradually erode the model’s own performance. Rather than patching models after the fact with updating techniques, the review focused on methods applied at the moment of initial development, so that a model would work in new settings without adjustment.

To find this literature, the team searched MEDLINE, Embase, medRxiv and arXiv on 4 August 2023, using title search terms combining generalisability, transportability and dataset shift with established filters for prediction modelling studies. The searches identified 1,761 articles, of which 172 went to full-text screening; 13 were ultimately eligible. Citation searching added four more, and one paper from a validation set that the search strategy failed to retrieve was also included, giving 18 papers in total. The authors acknowledge limitations in this process: data extraction was performed only by the primary author without independent verification, databases such as Scopus and Web of Science were not searched, and the inconsistent terminology used across the medical statistics and machine learning communities raises the possibility that relevant papers were missed — indeed, one of five known key papers was not captured by the search terms at all.

The review’s central contribution is a two-way classification. Methods are either data-driven, relying on empirical data to maximise model robustness, or knowledge-driven, incorporating external knowledge of causal relationships before any data is used. Independently, methods either aim for generalisability across a range of possible target populations, or for transportability to one specific target population — the latter often requiring data from that target population during development.

Among the data-driven approaches, the simplest idea is heterogeneity: developing models on data collected from multiple centres or geographical regions dilutes location-specific patterns and can yield a more generalisable model. De Jong and colleagues extended internal-external cross-validation so that training data is non-randomly clustered, with a generalisability penalty built into the model selection criteria; in case studies, the predictors selected led to more stable models, though calibration and discrimination still varied across populations, and the method’s reliance on meta-analytic aggregation acts as a set of hyperparameters that directly influence performance. Reps took a related route with ensemble techniques, training L1-regularised logistic regression models on five distinct healthcare databases and combining them through weighted and stacked ensembles, validated with a leave-one-database-out scheme. The ensemble improved discriminative performance on a new, unseen database, but calibration — the calibration-in-the-large and calibration slope — remained poor, and the original findings came from US populations, which may not apply elsewhere.

For transportability to a specific target population, the review highlights density ratio weighting, in which individuals in the development data are assigned weights based on how similar they are to the target population, typically the ratio of the probability of being sampled from the target to that of being sampled from the development population. Gao and colleagues estimated these weights indirectly by fitting penalised logistic regression and random forest classifiers to combined development and target data, predicting each individual’s probability of coming from the target population; applied to sepsis detection in hospitalised patients, the weighting improved discrimination consistently, but calibration as measured by Brier scores did not improve uniformly, particularly for complex models with many hyperparameters. Steingrimsson and colleagues showed that inverse-odds weighting provides minimal benefit for correctly specified models but significantly reduces mean squared error for incorrectly specified ones, tailoring them to the target distribution. A key limitation applies across these methods: they require covariate data from the target population, and they do not work under concept shift, where the relationship between predictors and outcome itself differs between populations. The review also notes that in completely non-overlapping populations, the required sampling probabilities cannot be correctly identified, so these methods probably offer no additional benefit there.

Two further data-driven entries round out that category. Qian and colleagues introduced adversarial validation, originally for credit score prediction, in which a classifier predicts the probability that an observation comes from the target population; those probabilities are then used during internal validation — either to select the subset of validation data most resembling the target or as sample weights — so that the final model is tailored to transport there, though the approach has seen limited application in clinical prediction. Separately, Pajouheshnia and colleagues demonstrated that differences in measurement error patterns between populations can degrade performance even without dataset shift, and suggested that ensuring consistent measurement processes for predictors across populations could mitigate that risk.

The knowledge-driven approaches lean on causal reasoning, and all seven identified studies target transportability to a specific population. Building on Pearl and Bareinboim’s transportability formulas, these methods require researchers to specify in advance which predictors are stable across settings and which vary — where stability means the conditional distribution of the outcome given the predictors is invariant, not that the predictors’ prevalence is the same. Selection diagrams, directed acyclic graphs containing a selection variable marking membership of the development or target population, formalise this: predictors are admissible if the outcome is independent of the selection variables given them. Subbaswamy and colleagues extended this into the graph surgery estimator, which uses causal algorithms to find predictor sets for which operations on the selection variables are identifiable from observational data, effectively severing population-specific influences; in simulated linear Gaussian experiments it maintained a near-constant mean squared error where ordinary least squares and graph pruning failed, and on real-world UCI Bike Sharing data it achieved the lowest error in several test cases. Bellamy and colleagues contributed the related notion of shortcut predictors — site-specific features, such as a watermark in an electrocardiogram, that must be identified and removed with expert causal knowledge before training.

Other knowledge-driven strategies include minimising worst-case loss over subpopulations defined by mutable and immutable variables, proposed by Subbaswamy and colleagues and extended with minimax optimisation by Liu in 2023, though both leave open how best to build the final prediction model on the resulting stable subset. Causal predictor selection restricts predictors to those in the Markov blanket of the outcome or its direct causes, since the probability of a consequence given its cause is invariant across populations while the reverse is not — an approach suited to prognostic prediction but not to diagnostic prediction, where symptoms are used to predict disease. Neto and colleagues proposed a counterfactual, causality-aware approach for anticausal prediction problems, finding that models adjusted for confounding using only development data were not robust when dataset shift from selection bias was present, and that data from the target population is needed to adjust confounding there.

The authors stress that these methods are rarely used in practice, possibly because of limited awareness, and that no study has yet proposed a causal methodology resistant to effect modifiers for building generalisable prediction models with minimal extra assumptions. They also note a trade-off: a model developed on highly heterogeneous data may fail in a new population if the development data were too diverse. Looking forward, the review recommends integrating data-driven and causal approaches, developing methods that reduce the need for target population data, extending the work to unstructured data such as images and text, and — most importantly — evaluating all these methodologies on both simulated and real-world datasets presenting various types of dataset shift, to understand their strengths and weaknesses in broader applications.

Subject of Research: Methodological approaches for building clinical prediction models that generalise or transport to new populations

Article Title: Scoping review of methodology for aiding generalisability and transportability of clinical prediction models

Article References: Ploddi, K., Martin, G. P., O’Connell, M., & Sperrin, M. (2026). Scoping review of methodology for aiding generalisability and transportability of clinical prediction models. Diagnostic and Prognostic Research, 10(1), Article 25. https://doi.org/10.1186/s41512-026-00237-8

Image Credits: AI Generated

DOI: 10.1186/s41512-026-00237-8

Keywords: clinical prediction models, generalisability, transportability, scoping review, dataset shift, causal inference, density ratio weighting, ensemble methods, external validation, machine learning, selection diagrams, methodology