Most COVID-19 AI models never made it past internal validation, review finds

A systematic review of artificial intelligence research on COVID-19 has found that most machine learning and deep learning models built for diagnosis, prognosis, and outbreak forecasting were never tested beyond internal validation, leaving a persistent gap between reported technical performance and real-world clinical readiness.

The review, published in Applied Network Science, followed PRISMA 2020 guidelines and searched six databases—IEEE Xplore, ACM Digital Library, SpringerLink, ScienceDirect, Wiley Online Library, and arXiv—for studies published between December 2019 and 31 January 2024. Of 110 records identified, 57 met the eligibility criteria and were assessed using a six-domain CASP-informed methodological appraisal together with a validation-maturity assessment.

The quantitative synthesis covered application domain, publication timing, dataset geographic provenance and scale, principal model family, imaging modality, and validation strategy. Diagnostic imaging and clinical diagnosis formed the largest evidence domain, followed by prognosis, forecasting, remote monitoring and Internet of Things applications, online social network and natural language processing surveillance, and multimodal or hybrid systems.

Validation maturity emerged as the central weakness. Forty-six of the 57 studies, or 80.7 percent, relied primarily on internal splitting or cross-validation, while only 11 studies, or 19.3 percent, reached external, cross-dataset, or multicenter validation. This means the large majority of models were evaluated on data drawn from the same sources used to train them, a practice that can overstate how well a model will perform in new hospitals, countries, or patient populations.

Many models did report strong internal performance, particularly for chest X-ray and CT-based diagnosis and for clinical-risk prediction. However, the review found that generalizability was constrained by several recurring problems: retrospective or geographically restricted datasets, source heterogeneity, data leakage, shortcut learning, hidden confounding, temporal drift, and subgroup bias.

Approaches based on online social networks and Internet of Things devices offered scalable surveillance and patient monitoring, but the review noted they remained limited by noisy signals, demographic bias, privacy risks, and implementation variability.

The authors concluded that the evidence reveals a persistent gap between technical performance and real-world readiness. To close it, they recommended that future pandemic AI research prioritize independent and multicenter validation, transparent dataset provenance, fairness-aware and explainable evaluation, multimodal integration, calibration under distribution shift, and privacy-preserving approaches such as federated learning.

As a systematic review, the study summarizes the published literature rather than testing any model itself, and its findings reflect studies indexed in the six databases through January 2024. The article is published open access, and the version currently available is being shared early as accepted peer-reviewed research, subject to further editorial edits before the final version of record appears.

Subject of Research: A systematic review of machine learning and deep learning methods for COVID-19 diagnosis, prognosis, and outbreak forecasting

Article Title: A systematic review of machine learning and deep learning methods for COVID-19 diagnosis, prognosis, and outbreak forecasting

Article References: Jodayree, M., Ghafi, A. K., Khodadadi, I., Jahanshahi, J. A., Tofangdarzade, A., & Pirkhedri, A. (2026). A systematic review of machine learning and deep learning methods for COVID-19 diagnosis, prognosis, and outbreak forecasting. Applied Network Science. https://doi.org/10.1007/s41109-026-00836-w

Image Credits: AI Generated

DOI: 10.1007/s41109-026-00836-w

Keywords: COVID-19, machine learning, deep learning, systematic review, diagnosis, prognosis, outbreak forecasting, validation, generalizability, federated learning, chest X-ray, public health surveillance