Counterfactual Models Reveal How Treatment Shifts Distort Prognostic Predictions

Prognostic models sit at the heart of modern clinical decision-making. They take measurable patient characteristics—age, biomarkers, tumor grade, comorbidities—and convert them into an individual estimate of the risk of developing a specific health outcome. Yet a subtle and persistent threat to their accuracy has long troubled epidemiologists: treatment strategies evolve. When the regimens patients actually receive change over time or differ across populations, the relationship between predictors and outcomes shifts, and the model’s performance can quietly degrade. A new study published in Frontiers in Epidemiology offers a rigorous methodological answer, using counterfactual risk modeling to ask what a well-known prostate cancer prediction model would look like in a world where no patient received surgery.

The research, conducted by Daniela Zugna, Nicolas Destefanis, Renata Zelic, Andreas Pettersson, and Lorenzo Richiardi, draws on the Turin Prostate Cancer Prognostication (TPCP) cohort, which enrolled consecutive patients diagnosed with non-metastatic prostate cancer between January 2008 and December 2013 at the University Hospital Città della Salute e della Scienza di Torino in Italy. Patients under 85 years of age who were residents of the Province of Turin and free of systemic metastases at diagnosis were followed longitudinally from their diagnostic biopsy until death, emigration, or the end of 2021. The primary outcome was metastatic progression, defined as the first record of metastatic disease after diagnosis, with death from other causes acting as a competing event that precludes the outcome from ever occurring.

The team focused on the Memorial Sloan Kettering Cancer Center (MSKCC) prediction model, a pre-treatment risk stratification tool that estimates the risk of metastatic progression using clinical and pathological parameters collected at diagnosis: age, prostate-specific antigen (PSA), the International Society of Urological Pathology (ISUP) histological grade, clinical tumor stage, and the numbers of positive and negative biopsy cores. The model had recently been updated with TPCP data under the observed treatment regimen. The central question of the new study was whether this model, developed in a cohort where 42.5 percent of patients underwent radical prostatectomy within six months of diagnosis, remains valid for predicting metastatic progression in a hypothetical treatment-naive population where no one receives surgery.

Why does this matter? In clinical practice, patients receive treatment based on their baseline characteristics, their perceived risk, and their life expectancy. When a prognostic model is developed in such a partially treated cohort while ignoring treatment, the efficacy of the therapy can mask the true underlying risk. The predicted risk in treated patients may underestimate their actual risk in the absence of treatment, potentially leading clinicians to undertreat precisely those patients who would have benefited most from therapy. Conventional workarounds each carry their own flaws: restricting analyses to untreated patients introduces selection bias because treatment decisions are themselves based on prognostic factors; stratifying or adjusting for treatment becomes difficult when regimens evolve or when treatment information is unavailable at the moment the model is used for pre-treatment decision-making.

The methodological innovation of the study lies in the parametric g-formula, a causal inference technique that allows consistent estimation of outcome probabilities under hypothetical treatment assignments, provided all relevant predictors of both treatment and outcome are measured. The researchers divided the five-year follow-up into monthly intervals, estimated the probability of metastatic progression and death from other causes in each interval using pooled logistic regression, and modeled the probability of initiating prostatectomy as a time-dependent absorbing state. Because treatment decisions were assumed to be made at diagnosis, no time-dependent confounders affected the treatment-outcome association. Monte Carlo simulations then generated a counterfactual dataset—ten times the size of the observed cohort—in which no patient underwent prostatectomy, with outcomes assigned according to each patient’s baseline covariates and the hypothetical treatment assignment.

The observed analysis included 710 of 891 patients after a complete-case analysis excluded 181 patients with missing values. During five years of follow-up, 48 patients developed metastatic prostate cancer and 83 died from other causes. In the counterfactual cohort of 7,090 simulated observations, 523 patients developed metastatic disease while 1,059 died from other causes. These seemingly paradoxical numbers reflect the competing risks structure: when prostatectomy was included in regression models fitted to the observed data, it showed a strongly inverse association with death from other causes (hazard ratio 0.39, 95 percent confidence interval 0.20 to 0.75), likely reflecting the selection of patients eligible for curative surgery. Removing prostatectomy therefore increased the risk of competing mortality, which in turn limited the occurrence of metastatic cancer—the competing event censored the outcome of interest more often.

Performance comparisons at five years post-diagnosis revealed that the model updated in the counterfactual cohort achieved an area under the curve (AUC) of 0.84 (95 percent CI: 0.82 to 0.85), while the original model applied to the counterfactual data achieved an AUC of 0.83 (95 percent CI: 0.81 to 0.85). Calibration was similarly close: the updated model had a calibration slope of 1.02 (95 percent CI: 0.99 to 1.12) and intercept of 0.00 (95 percent CI: −0.10 to 0.10), compared with 1.07 (95 percent CI: 0.97 to 1.17) and 0.05 (95 percent CI: −0.04 to 0.15) for the applied model. The differences were marginal, and the authors expect larger gaps when treatment is more strongly associated with the event of interest. Notably, the updated model reclassified the predicted five-year metastatic risk in 11.6 percent of patients—7.1 percent received a higher predicted risk and 4.5 percent a lower one—and predicted a higher risk of death from other causes in 53.2 percent of patients, underscoring how counterfactual predictions change not only through the treatment’s direct effect on the outcome but also through its impact on competing events.

The g-formula rests on strong, untestable assumptions: consistency, exchangeability between treatment groups, positivity of treatment assignment, no unmeasured confounding, no measurement error, and no model misspecification. Because the method involves specifying multiple regression models, violations can substantially distort results. The authors checked for misspecification by comparing parametric g-formula estimates with nonparametric estimates of the outcome risk under the natural course, finding close agreement that supported the adequacy of their models. Still, residual confounding by indication—surgery being offered to healthier patients—may have contributed to the predicted higher mortality in the no-surgery scenario. The authors point out that alternative approaches such as ignoring treatment, stratifying, or regressing on treatment are similarly susceptible to competing-risks bias and model misspecification, and that assessing and comparing the extent of bias across methods remains a subject for future work.

The study also situates itself within a growing literature. Previous reviews have catalogued causal methods enabling predictions under hypothetical interventions, comparing strategies such as excluding or censoring treated patients, inverse probability treatment weighting, modeling treatment as a time-dependent covariate, and marginal structural models. Dickerman and colleagues applied the g-formula to predict mortality among HIV patients under static and dynamic counterfactual treatment scenarios, finding that prediction models were systematically mis-calibrated in populations with different treatment strategies. Keogh and van Geloven extended validation approaches to survival data using artificial censoring and weighting. The Turin study adds a competing-risks dimension to this methodological conversation, showing that even a treatment with no direct effect on the primary outcome can alter counterfactual predictions through its effect on the competing event.

The authors acknowledge limitations: the TPCP cohort reflects treatment approaches from roughly fifteen years ago, so predictions may not apply to current patients; radiotherapy was excluded because its clinical indications are too heterogeneous to classify reliably; and the analysis focused on static interventions, though the methodology can be extended to time-varying confounding and dynamic treatment strategies. Their conclusion is measured but consequential. When a prediction model cannot be validated in a new population because treatment strategies have shifted, counterfactual risks under hypothetical regimens can still be cautiously estimated—provided the causal inference assumptions hold. Counterfactual modeling, they argue, should be considered a useful complementary assessment of model transportability and calibration under clinically relevant treatment scenarios, a lens through which the quiet distortions that evolving therapies impose on prognostic models can finally be seen and corrected.

Subject of Research: Predicting disease progression under a “treatment-naive” counterfactual scenario in a competing risks framework

Article Title: Predicting disease progression under a “treatment-naive” counterfactual scenario in a competing risks framework

Article References: Zugna, D., Destefanis, N., Zelic, R., Pettersson, A., & Richiardi, L. (2026). Predicting disease progression under a “treatment-naive” counterfactual scenario in a competing risks framework. Frontiers in Epidemiology, 6, Article 1850482. https://doi.org/10.3389/fepid.2026.1850482

Image Credits: AI Generated

DOI: 10.3389/fepid.2026.1850482

Keywords: Predicting, disease, progression, under, treatment-naive, counterfactual, scenario, competing, risks, framework, scientific research