How a Hidden Flaw in EEG Machine Learning Inflates Accuracy by 35 Percentage Points

A single, easily overlooked decision in how brain-signal data are divided can turn a mediocre diagnostic classifier into one that appears almost perfect. A new study published in Frontiers in Neuroinformatics quantifies this effect with unusual rigor, showing that a common evaluation mistake in clinical electroencephalography (EEG) machine learning inflates reported accuracy by roughly 35 percentage points across two completely different patient populations. The finding strikes at the heart of a growing literature that promises EEG-based diagnosis of attention-deficit/hyperactivity disorder (ADHD) and Parkinson’s disease, and it delivers a blunt methodological warning: many published performance figures may reflect the model memorizing individual people rather than detecting disease.

The study, conducted by Burak Aggul of Istanbul Topkapi University, tackles a problem the author calls window leakage. Clinical machine-learning pipelines rarely feed entire recordings into a model. Instead, a continuous EEG trace is chopped into short segments, or windows, each of which becomes a separate training example. This practice multiplies the apparent size of a dataset from a few hundred recordings into tens of thousands of segments. The trouble arises when those segments are randomly divided between training and test sets. If windows from the same participant appear on both sides of the split, the classifier can latch onto stable person-specific signatures, such as electrode impedance patterns, skull geometry, session artifacts, or device characteristics, rather than the disease-related signal the researchers intended to measure.

To demonstrate that this is not a niche concern, the study deliberately chose two clinically and technically dissimilar public cohorts. The first comprised 121 children aged 7 to 12, including 61 diagnosed with ADHD and 60 controls, whose EEG was recorded at 128 Hz from 19 standard scalp positions while they viewed cartoon images and counted characters. The second comprised 149 adults, 100 with Parkinson’s disease and 49 controls, whose eyes-open resting-state EEG was captured at 500 Hz from 63 channels. Before analysis, both datasets were harmonized to a common montage of 18 channels and a uniform sampling rate of 128 Hz, with a fifth-order zero-phase Butterworth band-pass filter restricting the signals to 0.5 to 40 Hz. Each channel was standardized within each participant, and no statistics were pooled across people, ensuring that the only manipulated factor in the audit was the unit of the train-test split.

The audit itself was elegantly simple. A Random Forest classifier received 252 hand-engineered features per two-second window: for each of the 18 channels, Welch spectra yielded absolute and relative power in the delta, theta, alpha, beta, and gamma bands, alongside four time-domain statistics. When windows were randomly assigned to an 80/20 split, repeated ten times, balanced accuracy reached 95.59 percent in the ADHD cohort and 92.02 percent in the Parkinson’s cohort, with ROC AUC values near 0.99. When the same classifier was evaluated with participants fully held out, using stratified eight-fold participant-disjoint cross-validation, balanced accuracy collapsed to 60.98 percent for ADHD and 56.17 percent for Parkinson’s. The inflation, roughly 35 points in both cohorts, replicated across disorders, age groups, recording tasks, channel densities, and acquisition systems, pointing to the segmented structure of the data itself as the common culprit.

The statistical implications run deeper than the headline numbers. Adjacent windows with 50 percent overlap share half their samples, and every window inherits its participant’s label, so window counts cannot be treated as independent observations for confidence intervals or significance testing. The study illustrates this with participant-level bootstrap intervals: despite 15,014 ADHD windows and 17,056 Parkinson’s windows, the effective inferential sample sizes remained 121 and 149 people. For the best ADHD model, pooled balanced accuracy was 0.734 with a 95 percent confidence interval spanning 0.656 to 0.804; for the best Parkinson’s model, 0.635 with an interval of 0.553 to 0.716. Those are wide bands, and they signal moderate research-grade discrimination rather than clinical readiness.

Under the correct, leakage-free evaluation, the study compared four neural architectures across both cohorts using a matched protocol with fixed hyperparameters, balanced training weights, and validation participants drawn exclusively from each outer-training fold. No single architecture dominated. In the ADHD cohort, a convolutional-recurrent CNN-LSTM achieved the highest subject-level balanced accuracy at 73.21 percent, while the compact EEGNet, with only 1,730 trainable parameters, posted the highest ROC AUC at 0.755. In the Parkinson’s cohort, EEGNet led with 63.75 percent balanced accuracy and 0.730 ROC AUC, while the 1D-CNN achieved the highest conventional accuracy at 70.58 percent, a divergence driven by the 100-to-49 class imbalance that makes balanced accuracy the more clinically honest metric. Crucially, none of the paired architecture comparisons survived Bonferroni correction, and the author is explicit that the tests are descriptive rather than confirmatory.

Window duration proved equally dataset-dependent. Testing 1-, 2-, 4-, and 8-second windows with 50 percent overlap, the study found no consistent cross-cohort ranking. One-second windows maximized ADHD balanced accuracy at 70.98 percent, while 8-second windows produced the highest ADHD ROC AUC at 0.788. In Parkinson’s, the 2-second reference performed best on balanced accuracy, yet longer windows improved AUC descriptively without yielding a usable fixed-threshold balance. The trade-off is structural: short windows generate more training examples but greater redundancy, while long windows carry more temporal context but sharply fewer instances, dropping from 31,716 to 2,768 ADHD windows between 1 and 8 seconds. The practical lesson is not that one duration is universally superior, but that the choice must be prespecified and reported.

The study also subjected its own custom architecture, ATCNN-SA, which combines multi-scale convolution, squeeze-and-excitation feature recalibration, temporal attention, and bidirectional recurrence, to four component-removal ablations. The results were humbling. Removing the bidirectional LSTMs raised ADHD balanced accuracy by 5.13 points, while removing temporal attention improved Parkinson’s balanced accuracy from 60.78 to 67.22 percent. Yet no ablation difference was significant after correction, and the direction of effects flipped between cohorts. Because the reduced variants were not independently retuned, the experiments test component removal under a controlled protocol rather than the best achievable version of each architecture, and they demonstrate that the contribution of attention and recurrence cannot be reliably isolated in cohorts of this size.

The broader context makes these findings urgent. Previous work, including a 2024 survey of translational deep-learning EEG studies, found that participant-level holdout is frequently unclear or absent, and recent analyses of the same pediatric ADHD cohort showed pronounced performance drops when moving from window-wise to subject-wise validation. The present study extends that warning with a matched cross-cohort design and makes its entire evaluation auditable: fold assignments, out-of-fold probabilities, training histories, timing records, and executable code are released alongside the paper. This transparency matters because the field’s reproducibility crisis in machine-learning-based science has been traced repeatedly to exactly this kind of leakage.

The procedural prescription that emerges is unambiguous. Participants must be separated before any model fitting, feature selection, or hyperparameter tuning; results must be reported at the subject level with class-balanced metrics such as balanced accuracy, sensitivity, specificity, ROC AUC, and precision-recall AUC; uncertainty must be resampled at the participant level rather than the window level; and the exact order of filtering, windowing, splitting, aggregation, and model selection must be disclosed. Until these practices become standard, claims that an EEG classifier can diagnose ADHD or Parkinson’s disease in new patients should be read with considerable skepticism, because the impressive numbers behind them may be measuring the model’s memory for familiar people rather than its insight into disordered brains.

Subject of Research: Window leakage in clinical EEG machine learning evaluation across ADHD and Parkinson's disease cohorts

Article Title: Window leakage in clinical EEG machine learning: matched subject-independent evaluation across ADHD and Parkinson cohorts

Article References: Aggul, B. (2026). Window leakage in clinical EEG machine learning: matched subject-independent evaluation across ADHD and Parkinson cohorts. Frontiers in Neuroinformatics, 20, Article 1930795. https://doi.org/10.3389/fninf.2026.1930795

Image Credits: AI Generated

DOI: 10.3389/fninf.2026.1930795

Keywords: EEG, window leakage, ADHD, Parkinson's disease, machine learning, data leakage, subject-independent validation, cross-validation, deep learning, clinical neuroscience, reproducibility, balanced accuracy