AI Turns Cognitive Test Data Into Images to Forecast Multiple Sclerosis Disability

Multiple sclerosis is an unpredictable disease, and few of its unpredictabilities matter more to patients than the question of when their disability will begin to accumulate. For people living with the relapsing-remitting form of the condition, periods of neurological stability can lull both patient and clinician into a false sense of security, while the underlying disease process quietly advances. A new study published in PLOS Digital Health suggests that a familiar and low-burden clinical tool, the computerised cognitive test, may hold far more predictive power than anyone realised, provided it is analysed in a radically different way. By transforming thousands of split-second reaction time measurements into images and feeding those images into a deep learning model, an international research team has built a system that can estimate how soon a patient’s disability will progress.

The research, led by Chao Zhu and Daniel Merlo together with colleagues from institutions across Australia, Europe and beyond, drew on two rich data sources. Clinical information came from MSBase, a large international registry that tracks outcomes in people with multiple sclerosis over many years, while cognitive measurements came from MSReactor, a computerised battery of cognitive tests that patients can complete remotely. The study population consisted of 746 patients with relapsing-remitting multiple sclerosis, whose records spanned the period from February 2016 to September 2022, with a median follow-up of 3.2 years. During that window, the researchers tracked which patients experienced confirmed disability progression, the clinical milestone that marks a turning point in the disease course.

The central challenge the team faced was one of dimensionality. A single session of computerised cognitive testing produces a torrent of reaction time data, with each trial of each task yielding a measurement precise to the millisecond. When those sessions are repeated serially over months and years, the resulting dataset becomes a sprawling, irregularly sampled time series that resists conventional statistical analysis. Traditional approaches collapse this richness into summary statistics, such as the mean reaction time per test, and in doing so discard much of the temporal texture that may carry the earliest signals of neurological change. The researchers suspected that this discarded texture was precisely where the predictive value lay.

Their solution was as elegant as it was unconventional. Rather than treating the reaction time data as numbers to be crunched, they treated it as something to be seen. Serial reaction time measurements from three distinct cognitive tasks were encoded into multicolour images using the RGB colour scheme familiar from digital photography. Psychomotor function, measured by a task labelled R, was mapped to one colour channel; attention, labelled G, to another; and working memory, labelled B, to the third. Each patient’s cognitive history thus became a visual artefact, a kind of portrait of their brain’s processing speed over time, in which patterns of variability, slowing and fluctuation could be rendered visible to an algorithm designed to detect them.

Once the images were constructed, the team deployed a convolutional neural network, the same class of algorithm that powers facial recognition and medical image analysis, to extract the salient features from each patient’s cognitive portrait. These learned features were then combined with standard clinical variables, such as demographic information and disease history, and fed into a Transformer-based survival model the researchers named MS-TranSurv. Survival models are a staple of clinical statistics, designed to predict the time until an event occurs, but the Transformer architecture, originally developed for processing sequential data in natural language, allowed the model to weigh the relative importance of different pieces of information flexibly and in context. The output was a personalised estimate of the time to confirmed disability progression.

Evaluating such a model requires rigorous benchmarks, and the team compared MS-TranSurv against several formidable competitors. These included Dynamic DeepHit and Recurrent Deep Survival Machines, both state-of-the-art deep learning approaches to survival prediction, as well as a classical Cox proportional hazards model built on clinical variables alone, and a stripped-down version of MS-TranSurv that used only mean test values rather than the full image-encoded data. Performance was assessed with three complementary metrics: the concordance index, which measures how well the model ranks patients by risk; the integrated Brier score, which captures calibration, or how closely predicted probabilities match observed outcomes; and the time-dependent area under the receiver operating characteristic curve, which gauges classification accuracy at specific time points.

The results were encouraging, if measured. MS-TranSurv achieved a concordance index of 0.61, with a 95 percent confidence interval of 0.54 to 0.68, indicating slightly better discrimination than the benchmark models. Its time-dependent area under the curve reached 0.74, with a confidence interval of 0.63 to 0.85, while its integrated Brier score of 0.24, spanning 0.16 to 0.32, demonstrated calibration comparable to the alternatives. Notably, the version of the model that consumed individual test-level data, preserving the full granularity of the reaction time recordings, outperformed the variant relying on summary measures. This finding vindicates the team’s core intuition: the fine-grained temporal structure of cognitive performance carries genuine predictive information that averages destroy.

It is worth being clear about what these numbers do and do not mean. A concordance index of 0.61 reflects modest, not spectacular, discrimination, and the wide confidence intervals reveal the uncertainty inherent in a cohort of this size. The authors are candid that the model’s advantage over existing approaches is incremental rather than transformative. Yet the significance of the work lies less in the headline metrics than in the demonstration of feasibility. Serial cognitive assessment is inexpensive, non-invasive and can be administered remotely, making it one of the few monitoring tools that patients could realistically deploy from home at high frequency. Proving that this data stream can be converted into survival-based risk predictions opens a door that conventional analysis had kept shut.

The clinical implications extend in several directions. For neurologists managing relapsing-remitting multiple sclerosis, a validated risk prediction tool could sharpen decisions about when to escalate disease-modifying therapy, moving beyond reactive treatment changes after progression becomes evident toward proactive intervention while the window for preventing irreversible damage remains open. For clinical trial designers, the model offers a route to enrichment, the practice of recruiting participants at higher risk of the outcome being studied, which reduces trial size and duration and accelerates the evaluation of new therapies. The framework’s generality is perhaps its most exciting feature: the image-encoding strategy could be applied to other high-dimensional digital biomarkers, from wearable sensor streams to speech recordings, wherever longitudinal data outstrips the reach of traditional statistics.

The study also underscores a broader shift in neurology toward remote, continuous monitoring. Cognitive decline often precedes measurable physical disability in multiple sclerosis, and subtle changes in processing speed may flag disease activity that standard clinical examinations miss. A patient completing a brief computerised battery at home every few months generates a longitudinal record of brain function that no clinic visit schedule could match. Combined with machine learning methods capable of reading that record in full resolution, such monitoring could transform risk stratification from an art informed by occasional snapshots into a science built on dense, personalised data. The MSBase and MSReactor cohorts that made this study possible hint at the scale of what such data ecosystems could ultimately deliver, and the research team’s framework offers a template for turning the digital exhaust of routine care into clinically actionable foresight.

Subject of Research: Deep learning prediction of disability progression in relapsing-remitting multiple sclerosis using image-encoded cognitive reaction time data

Article Title: Strategies for integrating artificial intelligence and cognitive assessment to predict disability progression in relapsing-remitting multiple sclerosis: A model development study

Article References: Zhu, C., Zhang, X., van der Walt, A., Taylor, B., Monif, M., Kalincik, T., Lechner-Scott, J., Buzzard, K., Kilpatrick, T., Barnett, M., Zhou, Z., Jokubaitis, V., Gresle, M., Darby, D., Mehta, D., Ge, Z., Butzkueven, H., & Merlo, D. (2026). Strategies for integrating artificial intelligence and cognitive assessment to predict disability progression in relapsing-remitting multiple sclerosis: A model development study. PLOS Digital Health, 5(9), e0001722. https://doi.org/10.1371/journal.pdig.0001722

Image Credits: AI Generated

DOI: 10.1371/journal.pdig.0001722

Keywords: multiple sclerosis, artificial intelligence, deep learning, cognitive assessment, disability progression, survival analysis, digital biomarkers, reaction time, Transformer model, remote monitoring, clinical trial enrichment, MSBase registry