Handwriting Data Flags Alzheimer’s in Rigorous Machine Learning Stress Test

Online handwriting—the digital record of how a person draws and writes on a tablet—offers a low-burden window into the cognitive and motor changes associated with Alzheimer’s disease. But small datasets can flatter machine learning models when preprocessing, task selection, or feature explanation leaks into the evaluation. A new study set out to see how well handwriting-based classification survives a deliberately strict, leakage-controlled test.

The researchers evaluated eight classifiers on the DARWIN dataset, which covers 174 participants performing 25 writing tasks described by 450 predictors. Instead of a single train-test split, they ran ten repetitions of nested stratified cross-validation with 5 outer and 3 inner folds, confining all preprocessing, hyperparameter tuning, and calibration to the outer training data. Random Forest achieved the highest descriptive ROC-AUC, 0.938 ± 0.033 across 50 held-out folds. A Friedman omnibus test found overall differences among models (p < 0.001), yet none of the 28 paired comparisons remained significant after Holm correction—suggesting the apparent superiority of any single classifier is fragile.

The team also probed how practical design choices affect performance. Training-derived decision thresholds, with a median of 0.461, produced held-out sensitivity of 0.869 and specificity of 0.831, though the authors stress these are exploratory operating characteristics, not clinical cutoffs. A fully nested selection procedure that reduced the task battery to seven tasks reached an AUC of 0.916 ± 0.037, below the 0.956 ± 0.030 obtained with all 25 tasks, and the selected subsets were unstable, with a mean Jaccard similarity of just 0.195.

Explainability analysis added a further caution. Cross-validated SHAP values emphasized temporal variables in the handwriting data, but a sensitivity analysis using all 450 features showed that individual feature rankings depend partly on an ANOVA preselection step. The authors describe the work as an integrated, leakage-controlled evaluation rather than a new classifier or a validated diagnostic instrument, and note that external prospective validation remains necessary before any real-world use.

Subject of Research: Machine learning classification of Alzheimer's disease from online handwriting on the DARWIN dataset

Article Title: Handwriting-based Alzheimer’s disease classification on the DARWIN dataset: Repeated nested validation, task-level reduction, and cross-validated explainability

Article References: Fatima, T., Lashari, T. A., Moetesum, M., & Lashari, S. A. (2026). Handwriting-based Alzheimer’s disease classification on the DARWIN dataset: Repeated nested validation, task-level reduction, and cross-validated explainability. PLOS One, 21(10), e0361117. https://doi.org/10.1371/journal.pone.0361117

Image Credits: AI Generated

DOI: 10.1371/journal.pone.0361117

Keywords: Alzheimer's disease, online handwriting, machine learning, DARWIN dataset, nested cross-validation, Random Forest, ROC-AUC, SHAP, feature selection, data leakage, cognitive-motor assessment, Handwriting-based