DQS offers a budget-friendly query strategy to improve unsupervised anomaly detection

Anomaly detection systems promise to flag the unusual before it becomes the catastrophic—a spike in sensor readings that warns of an impending industrial failure, an irregular heartbeat buried in a physiological signal, a subtle drift in the telemetry of a spacecraft. Yet behind the clean marketing language of “unsupervised” machine learning lies an inconvenient truth that researchers have long quietly acknowledged: many methods sold as unsupervised are not truly unsupervised at all. They sneak in a small quantity of labelled data, often for the critical step of deciding where to draw the line between normal and anomalous behaviour. A new study published in the Journal of Big Data confronts this contradiction head-on and proposes an elegant, budget-conscious way to make that borrowed supervision count for as much as possible.

The research, conducted by Lucas Correia, Thomas Bäck and Anna V. Kononova of the Leiden Institute of Advanced Computer Science at Leiden University in the Netherlands, together with Jan-Christoph Goos of Mercedes-Benz AG in Stuttgart, introduces a novel technique called the dissimilarity-based query strategy, or DQS. Rather than labelling data blindly and exhaustively—an approach whose cost balloons prohibitively in big data settings—the method borrows a page from the active learning playbook: it asks a human or automated oracle to label only a small, carefully chosen subset of samples, selected to be maximally informative. The result, according to the authors, is a threshold-refinement procedure that improves detection performance even when the labelling budget is tiny, and even when some of the labels handed over by the oracle are simply wrong.

To appreciate why threshold selection matters so much, it helps to understand how modern anomaly detection actually works. Most data-driven approaches to monitoring multivariate time series—streams of synchronized measurements from many sensors—train a model to capture the statistical signature of normal behaviour. A temporal variational autoencoder, or TeVAE, of the kind used in this study, learns a compressed representation of healthy operating patterns and then assigns each new observation an anomaly score, a number that quantifies how badly the observation deviates from what the model considers routine. But a score alone does not an alarm make. The system still needs a cutoff: above the threshold, declare an anomaly; below it, stay silent. Choose the threshold too low and the system drowns its operators in false alarms, a failure mode so corrosive that many industrial anomaly detection deployments have been abandoned for exactly this reason. Choose it too high and genuine faults slip through unnoticed. Setting that cutoff well is arguably the single most consequential decision in the entire pipeline, and it is precisely the step where purportedly unsupervised methods most often reach for labelled data.

The traditional unsupervised alternative is to set the threshold using statistical properties of the anomaly scores themselves—typically by assuming that the vast majority of points are normal and picking a high quantile of the score distribution. This works tolerably well when the underlying assumptions hold, but real-world data rarely cooperates. Contamination of the training data with anomalies, skewed score distributions, and distributional shifts between training and deployment conditions all conspire to make the quantile heuristic unreliable. What practitioners really want is a way to calibrate the threshold against ground truth, but ground truth labels are expensive. Labelling every recorded data point by hand, especially in a big data environment where millions of multivariate observations accumulate, is simply not feasible. This is the gap that active learning is designed to fill: instead of labelling everything, label strategically.

Active learning frames the labelling problem as a dialogue between a learner and an oracle. The learner nominates a handful of samples; the oracle—a human expert, a maintenance log, a simulation—returns their labels. The art lies in the nomination. Correia and colleagues compared three families of query strategies for selecting which time series segments to send to the oracle. The random-based query strategy, RQS, is the blunt instrument: pick samples at random, without any intelligence about what might be informative. The top-based query strategy, TQS, goes to the opposite extreme, selecting the samples with the highest anomaly scores on the theory that these are the points whose true nature will most constrain the threshold. The uncertainty-based query strategy, UQS, hedges its bets by picking samples whose labels the system is least sure about. And then there is the newcomer: DQS, the dissimilarity-based query strategy, which takes a different philosophical tack entirely.

The insight behind DQS is that diversity, not extremity, may be the key to efficient learning. If a query strategy picks a batch of samples that all look alike—say, a cluster of high-scoring segments that all stem from the same type of operational state—the oracle’s answers, however accurate, convey redundant information. DQS combats this redundancy by explicitly measuring how similar candidate samples are to one another and selecting a batch that is as heterogeneous as possible. The technical tool it uses for measuring similarity is dynamic time warping, or DTW, a classic algorithm from the time series analysis literature. DTW computes an alignment between two sequences that may be stretched or compressed in time relative to one another, allowing the algorithm to recognize that two sensor traces describing the same underlying phenomenon may unfold at different speeds. Applied here to anomaly score sequences rather than raw signals, DTW gives DQS a principled distance metric: two segments whose anomaly score curves can be aligned with low warping cost are considered similar, and the strategy avoids querying both. The aim, in the authors’ words, is to maximise the diversity of queried samples, so that every precious label carries new information.

The experimental evaluation probed two questions that matter enormously in practice but are unevenly treated in the literature. The first is the straightforward one: does querying the oracle actually improve detection performance compared with a purely unsupervised threshold? The answer was an emphatic yes, and—crucially—the advantage held across all four query strategies, not just DQS. Even a small budget of queried labels, used to refine the threshold rather than to retrain the detector, delivered better anomaly detection than the unsupervised baseline. The second question is thornier and has been conspicuously underexplored: what happens when the oracle makes mistakes? Human labellers are fallible, and even automated oracles such as maintenance databases can misattribute causes. The researchers therefore deliberately injected mislabelling into their experiments, corrupting a fraction of the oracle’s answers to see how fragile each strategy would prove.

The results paint a nuanced picture. DQS, it turns out, performs best in small-budget scenarios—exactly the regime where active learning is most needed and where every label matters most. When the number of queries is severely limited, maximising diversity appears to squeeze the most value out of each interaction with the oracle. However, the other strategies showed greater robustness when mislabelling entered the picture. There is a plausible intuition here: a strategy that aggressively seeks out diverse, unusual samples may also be more exposed to the consequences of any single wrong label, whereas random sampling dilutes the damage of individual errors across a broad, statistically representative selection. The practical lesson is not that DQS is universally superior, but that the choice of query strategy should be conditioned on two budgetary realities: how many labels can be afforded, and how trustworthy the oracle is likely to be.

Perhaps the most important conclusion of the study is one of framing. Once labels are queried, the system is no longer truly unsupervised—the authors are refreshingly candid about this. But their findings show that the strict unsupervised ideal, in which no labels touch the threshold at any point, is a standard that most published methods fail to meet anyway, and one that costs real performance to maintain. The pragmatic position emerging from this work is that whenever it is feasible to consult an oracle at all, an active learning-based threshold should be preferred to the unsupervised alternative, because the performance gains survive even imperfect labelling. For industries drowning in unlabelled sensor data—automotive engineering, where co-author Goos’s affiliation with Mercedes-Benz AG is suggestive, along with manufacturing, energy and process control—this amounts to an actionable recipe: train your detector unsupervised, then spend a tiny, intelligently allocated labelling budget calibrating the alarm.

The study also makes a quieter but valuable contribution to scientific candour in the field. By systematically exploring mislabelling—a topic the authors describe as underexplored—they highlight how fragile the benchmark culture of anomaly detection research can be. Papers frequently report results under the assumption of perfect labels, an assumption that dissolves on contact with real deployment. Demonstrating that active thresholding degrades gracefully rather than catastrophically under label noise is the kind of unglamorous robustness evidence that separates methods that work in the laboratory from methods that work in the factory.

The work, which received no dedicated funding, is published open access, and the authors note that the shared version is a citable, peer-reviewed accepted manuscript carrying a permanent DOI, subject to final editorial formatting. As anomaly detection systems continue their migration from academic benchmarks into safety-critical infrastructure, the questions this team has asked—how few labels are enough, which samples deserve them, and how much error can be tolerated—will only grow in importance. DQS offers one compelling answer to the second of those questions: when the budget is tight, spend it on difference, not on more of the same.Subject of Research: Active learning-based threshold selection for unsupervised anomaly detection in multivariate time series, introducing the dissimilarity-based query strategy (DQS) using dynamic time warping.