AI Model for Predicting IVIG Resistance in Kawasaki Disease Fades Over Time, Study Shows

A machine learning model designed to predict which children with Kawasaki disease will resist first-line treatment loses much of its accuracy within just a few years, according to a new multicenter study published in BMC Medicine. The research, which tracked nearly 6,500 children treated between 2006 and 2024, provides some of the clearest real-world evidence yet that clinical prediction models can silently degrade over time, and it demonstrates that carefully chosen updating strategies can restore much of that lost performance.

Kawasaki disease is an inflammatory condition of childhood that can damage the coronary arteries if left untreated. Standard therapy is intravenous immunoglobulin, or IVIG, but roughly one in ten children fails to respond to this treatment and faces a higher risk of serious cardiac complications. Identifying these children early matters enormously, because clinicians can escalate therapy before irreversible damage occurs. Over the past decade, research groups have built statistical and machine learning models to flag IVIG-resistant cases using routinely collected clinical and laboratory data, and several such models have reported promising accuracy in their original datasets.

The new study, led by Hongbiao Huang and colleagues across hospitals in Fujian and Jiangsu provinces in China, set out to answer a question that has received surprisingly little attention in pediatric medicine: what happens to a prediction model after it is deployed? Clinical prediction models are typically developed on historical data and then assumed to remain valid indefinitely. But the patients, the laboratory methods, the treatment protocols, and even the circulating pathogens that shape disease presentation all change over time. This phenomenon, known as temporal drift, can erode a model’s performance in ways that go unnoticed unless performance is actively monitored.

To investigate, the team assembled a retrospective cohort of 6,443 children with Kawasaki disease treated between 2006 and 2024 at multiple centers. They divided the cohort into three eras: a pre-COVID period from 2006 to 2019 with 3,788 children, a during-COVID period from 2020 to 2022 with 1,837 children, and an after-COVID period from 2023 to 2024 with 818 children. The proportion of children who developed IVIG resistance varied dramatically across these eras, at 7.0 percent, 13.9 percent, and 10.1 percent respectively. This shift in the underlying rate of resistance, known as case-mix change, is precisely the kind of drift that can destabilize a prediction model.

The researchers compared five machine learning algorithms, including logistic regression with LASSO regularization, support vector machines, and random forests, and selected the random forest model for deeper analysis. When the model developed on pre-COVID data was tested on later cohorts, its performance deteriorated markedly. The area under the receiver operating characteristic curve, or AUC, a standard measure of how well a model separates resistant from responsive patients, fell from 0.776 during the COVID era to just 0.629 after COVID. A value of 0.5 would represent no discrimination at all, so a drop of this magnitude means the model was barely better than chance in the most recent period.

Just as concerning was the collapse in calibration, which reflects whether a model’s predicted probabilities actually match observed event rates. The calibration slope fell from 0.894 to 0.431 across the same transition. A perfectly calibrated model has a slope of 1.0, so a slope of 0.431 indicates that the model was systematically overestimating risk in the newest cohort. Discrimination and calibration are distinct properties: a model can rank patients correctly while assigning probabilities that are wildly off. The Brier score, which combines both aspects into a single penalty, and decision curve analysis, which quantifies clinical net benefit at various risk thresholds, confirmed the deterioration from multiple angles.

Having documented the drift, the team then tested five updating strategies designed to keep the model current. These included recalibrating on temporally separated data, refitting on all accumulated history, weighting recent cases more heavily with a half-life scheme, using rolling windows that discard older data, and adjusting for calendar year. All analyses used ten imputed datasets to handle missing values, and uncertainty was quantified with 500 bootstrap resamples, a level of methodological rigor that lends weight to the comparisons. Refitting on all history achieved a mean AUC of 0.787 and a mean Brier score of 0.070, while the calendar year adjusted random forest and a rolling window random forest with a three-year window both achieved a mean AUC of 0.804.

The calendar year adjusted random forest emerged as one of the strongest performers overall. It achieved the lowest Brier score among the evaluated strategies, an observed-to-expected ratio closest to 1, indicating that its predicted event counts matched reality most closely, and slightly higher net benefit across most decision thresholds. In practical terms, this means the model not only ranked patients well but also produced risk estimates that clinicians could act on without systematically over- or under-treating. The finding suggests that explicitly accounting for the passage of calendar time, rather than treating all historical data as equally relevant, is an effective way to combat temporal drift.

The COVID-19 pandemic adds a fascinating natural experiment dimension to these results. The pandemic disrupted healthcare systems worldwide, altered infection patterns, changed healthcare-seeking behavior, and may have modified the immunological landscape of childhood inflammatory diseases. The sharp rise in IVIG resistance during the COVID era, from 7.0 percent to 13.9 percent, followed by a partial decline afterward, illustrates how dramatically the clinical environment can shift in a short window. A model trained on pre-pandemic data simply had no way of anticipating this change, which is exactly why static models fail when the world moves beneath them.

The implications extend well beyond Kawasaki disease. Prediction models are being embedded in electronic health records and clinical decision support systems at a rapid pace, yet few institutions have systematic processes for monitoring whether these tools still perform as advertised. This study demonstrates concretely that a model can lose accuracy within a few years and that time-aware updating strategies can recover much of the lost performance. The authors argue that ongoing performance monitoring and periodic model updating should be considered a routine part of long-term clinical implementation, not an afterthought. As artificial intelligence spreads through pediatric and adult medicine alike, this multicenter cohort offers a sobering and constructive lesson: a prediction model is not a finished product but a living tool that must be maintained, recalibrated, and refreshed as the patients and the world it serves continue to change.

Subject of Research: Temporal drift, calibration monitoring, and adaptive updating of an IVIG resistance prediction model in Kawasaki disease: a multicenter pediatric cohort study

Article Title: Temporal drift, calibration monitoring, and adaptive updating of an IVIG resistance prediction model in Kawasaki disease: a multicenter pediatric cohort study

Article References: Huang, H., He, Y., Zhang, J., Dong, J., Lin, F., Wang, C., Xu, Z., Chen, Q., Li, L., Zheng, X., Li, X., Hou, M., Liu, P., Tang, Q., & Lv, H. (2026). Temporal drift, calibration monitoring, and adaptive updating of an IVIG resistance prediction model in Kawasaki disease: a multicenter pediatric cohort study. BMC Medicine. https://doi.org/10.1186/s12916-026-05303-y

Image Credits: AI Generated

DOI: 10.1186/s12916-026-05303-y

Keywords: Temporal, drift, calibration, monitoring, adaptive, updating, IVIG, resistance, prediction, model, Kawasaki, disease