Deep Learning’s Anomaly Detection Supremacy May Be an Illusion, Study Finds

Time series anomaly detection—the task of spotting unexpected events in data ordered over time, from cyber attacks on water treatment plants to faulty spacecraft sensors to irregular heartbeats—has become one of machine learning’s most fashionable battlegrounds. For the past decade, papers in this field have overwhelmingly concluded that deep learning models, with their vaunted representational power, decisively outperform older classical techniques such as support vector machines, k-nearest neighbors, and principal component analysis. But a new open-access study published in the journal Machine Learning by Nancy Xu and Henrik Boström of KTH Royal Institute of Technology in Stockholm suggests that much of this apparent progress may rest on shaky methodological ground. After dissecting 18 of the field’s most influential papers, the researchers found recurring problems—data leakage, undisclosed dataset modifications, and underspecified experiments—that render many published results questionable or impossible to reproduce.

The stakes are far from academic. Practitioners in industry, healthcare, and infrastructure security rely on published benchmark comparisons to choose which detection systems to deploy. If the evidence that sophisticated neural networks beat simpler methods is distorted by flawed evaluation practices, organizations may be investing in expensive, complex models that offer no real advantage. The KTH team’s investigation reads less like a routine review and more like a forensic audit of a field’s collective assumptions, and its findings echo a broader reproducibility crisis that has been gathering force across machine learning research in recent years.

The first and most pervasive problem the researchers identified is data leakage, which occurs when information from the test set—data that should be invisible during evaluation—contaminates the assessment process. In time series anomaly detection, models produce continuous anomaly scores that must be converted into binary predictions via a threshold. The most damaging practice, found in seven of the 18 papers examined, involves selecting that threshold by trying many values on the test set and keeping whichever yields the best score. In any real deployment, no such oracle exists; the threshold must be chosen without knowing the answers. A second variant tunes the threshold so that a fixed percentage of test predictions are flagged as anomalous, often matching the known anomaly rate of the test set—still a form of test-set knowledge that would be unavailable in practice.

Even more striking is the widespread use of the point-adjust algorithm, a post-hoc evaluation trick in which if a model detects at least one anomalous point within a continuous segment of anomalies, the entire segment is credited as correctly detected. Six of the examined papers used this method directly. The authors illustrate its distorting power with a simple example: original predictions scoring 0.5 precision, 0.2 recall, and 0.286 F1 score jump to 0.8 across all three metrics after point adjustment. Previous work has shown that under point adjustment, random guessing can compete with state-of-the-art methods. Because the adjustment compares predictions to ground-truth labels before scoring, it is, in the researchers’ view, a clear case of test-set leakage. In total, eleven of the 18 papers suffered from leakage through thresholding or point adjustment.

The second category of problems involves silent modifications to benchmark datasets. Many popular datasets, including NASA’s Soil Moisture Active Passive (SMAP) and Mars Science Laboratory (MSL) satellite collections and the Server Machine Dataset (SMD) from a large internet company, contain multiple independent entities—separate sensors or servers whose recordings should be treated as distinct time series. Yet when the researchers inspected official code repositories, they discovered that six of the ten papers using such datasets had concatenated entities into single long series, or otherwise altered them, without disclosure. Concatenation fabricates a temporal order that does not exist in reality, violating the basic assumption that time series data is genuinely ordered in time. Because papers report a single score for multi-entity datasets without explaining how it was aggregated, these alterations would be undetectable without close examination of the code.

The third issue is underspecification. Hyperparameters and architectural details dramatically affect model performance, yet of the 18 papers examined, only three reported the hyperparameters of the baseline methods they compared against, and none of those had official repositories. Window sizes—a parameter that directly changes how much evaluation data can be extracted—ranged from three to three hundred across papers, making direct comparison against previously reported scores unreliable. Official implementations of cited baselines can be missing, broken, or outdated; the repository for the MADGAN method, for instance, still runs on TensorFlow 1 and had not been updated in eight years, while three separate public reimplementations diverge in how they handle one of its core modules. Readers simply cannot judge whether baseline comparisons were fair.

To test whether these methodological flaws had distorted the field’s central claim—that deep learning decisively beats classical methods—Xu and Boström ran their own large-scale empirical investigation. They evaluated two classical methods, k-nearest neighbors and PCA, alongside six deep learning approaches: a basic autoencoder, a variational autoencoder (VAE), LSTM-VAE, MADGAN, Omnianomaly, and USAD. Crucially, they deliberately used suboptimal, untuned settings for the classical methods, on the assumption that even handicapped traditional approaches should still fall clearly behind properly configured deep models if the literature were correct. Experiments spanned six popular benchmarks—SWaT, WaDi, SMAP, MSL, SMD, and the UCR Time Series Anomaly Archive—using sliding window sizes of 12, 48, and 128, with evaluation based on threshold-free metrics, AUC-ROC and AUC-PRC, and statistical hypothesis testing via the Friedman and Nemenyi tests.

The results upended expectations. Across the multi-entity datasets, the groups of statistically indistinguishable methods were large and overlapping, mixing deep learning and classical models with no clear winner. On the UCR archive, the humble k-nearest neighbors method frequently appeared in the top-performing group and was the best performer overall. On the single-entity SWaT and WaDi datasets, performance was similarly tight, with a basic autoencoder and, on WaDi, a trio of VAE-family models leading by slim margins. The researchers could not reject the assumption that optimized deep learning models, unoptimized deep learning models, and unoptimized classical methods perform similarly on standard benchmarks. Even with purposefully poor settings, the older methods frequently outperformed the baseline results reported in published papers—results the team found they could not replicate.

Why, then, do published comparisons show such dramatic gaps in favor of deep learning? The authors argue the methodological problems make the question unanswerable in the current literature: leakage inflates deep learning scores by optimizing against test-set information, modified datasets scramble comparisons, and underspecification hides whether any true difference lies in hyperparameters, data pipelines, or implementation details. Earlier critiques of the field have focused on flawed benchmark datasets and misleading metrics, prompting the creation of curated alternatives such as UCR and TSB-UAD and more informative measures like VUS. But the KTH study contends that problems in experimental design and reporting run deeper, and that better datasets alone cannot fix evaluations that leak or experiments that cannot be reconstructed.

Rather than proposing yet another competing standard, Xu and Boström call for transparency: full disclosure of the entire experimental setup in the paper text, and release of complete code including data preprocessing pipelines and baseline experiments, so that readers can judge the feasibility of the setup for themselves. They argue that since comparison against baselines is the yardstick of progress in the field, reproducing baseline results is just as critical as reproducing the headline method’s performance. Future directions they outline include metrics robust to noisy labels and detection timing, deeper investigation into why classical methods perform so surprisingly well, and a shared framework for fairly comparing deep and classical approaches. For a field racing to deploy anomaly detectors in hospitals, power grids, and spacecraft, the message is sobering but constructive: before trusting the next headline result, check whether the baseline was ever really beaten.

Subject of Research: Methodological flaws and reproducibility in deep learning-based time series anomaly detection

Article Title: Addressing Methodological Challenges in Time Series Anomaly Detection

Article References: Addressing Methodological Challenges in Time Series Anomaly Detection. (n.d.). https://doi.org/10.1007/s10994-026-07183-y

Image Credits: AI Generated

DOI: 10.1007/s10994-026-07183-y

Keywords: time series, anomaly detection, deep learning, reproducibility, data leakage, benchmarks, machine learning, evaluation metrics, point-adjust, baselines, autoencoders, methodology