The work, carried out by Simone Amirato, Fabrizio Angiulli, and Fabio Fassetti of the University of Calabria in Italy, addresses what the authors call the Anomaly Explanation problem. Their idea is deceptively simple: an explanation of an anomaly is a compact subset of features, a subspace, in which the anomalous examples deviate most sharply from the normal population. If a fraud-detection system flags a transaction, the explanation might boil down to three or four attributes, such as transaction amount, merchant category, and time of day, that together make the record look nothing like ordinary behavior. The difficulty is that finding the best such subset is a combinatorial nightmare. With d features, the number of possible subsets grows as the power set of the feature set, a search space so vast that exhaustive enumeration is hopeless even for moderately sized problems.

RLAX, short for Reinforcement Learning-based Anomaly eXplanation, recasts this search as a Markov Decision Process. Each state in the process is a candidate subspace, and each action corresponds to adding a single feature to the current selection. The agent starts from the empty set and progressively builds up an explanation, one feature at a time. What guides it is a reward signal derived from a quantity the authors call the Subspace Margin. This margin measures the difference between the anomaly scores of known outlier examples and known inlier examples within the current subspace, normalized by the square root of the subspace size. The anomaly score itself is a classic distance-based measure: the average distance from a point to its K nearest neighbors, which serves as a proxy for local density. When the margin grows after adding a feature, the agent earns a positive reward; when the margin shrinks, the reward is negative.

To learn a policy over this enormous action space, the team turned to Proximal Policy Optimization, or PPO, a policy-gradient algorithm that has become a standard tool in the reinforcement learning repertoire. The choice is deliberate. Value-based methods such as Deep Q-Networks struggle when the number of actions is very large, and in RLAX the number of actions equals the number of features, which can easily reach into the hundreds or thousands in real applications. Policy gradient methods, by contrast, handle large action spaces gracefully. The authors used the Stable-Baselines3 implementation of PPO with a custom actor-critic policy that masks inadmissible actions, keeping the default architecture of two 64-unit tanh layers for both the policy and value networks. Once training is complete, the learned policy is distilled into a ranking of features, either through the empirical mean of discounted rewards each action collected during training or directly from the policy’s action probabilities, and candidate subspaces are read off from the top of that ranking.

A key design element is the Explanation Gain, a measure that trades off separation accuracy against explanation size. Formally, the gain of a subspace is its separation quality minus a penalty proportional to the fraction of features it uses, with a tunable coefficient alpha that defaults to 0.5. This penalty is what pushes the agent toward compact explanations. The experimental analysis shows why it matters: the subspace that maximizes raw separation, measured by the Area Under the ROC Curve, may require more than 30 percent of the available features, whereas the best-gain solution often involves only a small fraction of them while preserving comparable discriminative performance. In other words, the framework resists the temptation to throw every feature at the problem and instead delivers explanations an analyst can actually read.

The journal version of the work extends an earlier conference paper with a Subspace Refinement Strategy that makes the exploration genuinely non-monotone. In addition to add actions, the agent can now remove features it previously selected or issue a stop action to terminate the episode early. This flexibility matters for two reasons. First, remove actions let the agent backtrack out of locally suboptimal regions of the search space, revising earlier selections that looked promising in isolation but hurt in combination. Second, the stop action prevents the agent from wandering through suboptimal territory merely to exhaust its step budget. The authors impose a budget on add operations to prevent loops, but leave remove operations unconstrained since they are inherently bounded by the number of prior additions. Episodes can end through the stop action, budget exhaustion, or a patience criterion that triggers after several consecutive negative rewards.

The empirical evaluation spans benchmark datasets from the ODDS repository, including arrhythmia with 274 features, mnist with 100 features, and several others, alongside two substantially larger real-world datasets: Internet Advertisements with 1,558 features and Swarm Behaviour with 2,400 features. Against competitors including V-GAN, SOD, HiCS, a genetic subspace-selection method, and random feature selection, RLAX achieved the highest Explanation Gain on every benchmark dataset, indicating the most favorable balance between discriminative power and compactness. Statistical significance against random selection was confirmed with one-sided paired t-tests adjusted using Holm’s correction. The refinement strategy also proved its worth in training dynamics: on representative datasets it produced markedly shorter episodes, saving roughly 30 percent of execution time while achieving improved AUC at the same feature subset sizes. On synthetic scalability tests, RLAX’s runtime grew only moderately, by factors of roughly 1.5 to 1.9 as the number of instances increased twentyfold, a growth rate substantially lower than that of the genetic and SOD baselines.

Perhaps the most striking evidence for the framework’s practical value comes from a qualitative case study on the cardio dataset, which contains cardiotocography measurements of fetal heart rate with well-established clinical semantics. For a representative anomaly, RLAX selected four top features: the number of prolonged decelerations per second, the histogram variance of the fetal heart rate, the percentage of time with abnormal long-term variability, and the number of fetal movements per second. Each of these maps directly onto recognized indicators of fetal distress in the obstetric literature. Prolonged decelerations are documented as strong markers of potential fetal hypoxia and acidosis, abnormal long-term variability is one of the four fundamental characteristics used to interpret a cardiotocography trace, reduced fetal movements are a classic clinical warning sign, and heart rate variance captures deviations in oscillation amplitude in either direction. An algorithm with no medical knowledge, in other words, recovered a clinically coherent explanation.

The authors are careful about the limits of their approach. Because RLAX explores the search space through single-feature modifications, strongly non-additive interactions between features, where an informative combination only pays off after several individually unrewarding additions, can be harder to discover, though the remove actions partially mitigate this. They also note that the method assumes access to a small set of labeled anomalies and inlier examples, which in practice may come from a detector’s output or from a domain expert. Future work, they write, will explore domain-informed reward shaping, alternative separation measures, and settings with heterogeneous features and weaker supervision. The source code has been released publicly, and all datasets used in the study are openly available.

What makes RLAX more than a technical curiosity is the broader shift it represents in how machine learning systems might communicate their judgments. As automated decisions increasingly touch medicine, security, and environmental monitoring, the demand is growing not just for accurate detectors but for systems that can articulate the basis of a flag in terms a human can verify. By demonstrating that a reinforcement learning agent can navigate a combinatorially explosive search space and return explanations that align with clinical knowledge, the Calabria team offers a concrete template for that kind of accountability. The anomalies that matter most are often the ones we cannot explain away, and a tool that points precisely at the handful of features responsible turns a cryptic alarm into the beginning of an investigation rather than the end of one.

Subject of Research: A reinforcement learning framework for explaining anomalies through compact feature subspaces

Article Title: RLAX: A Reinforcement Learning-Based Framework for Explaining Anomalous Data

Article References: RLAX: A Reinforcement Learning-Based Framework for Explaining Anomalous Data. (n.d.). https://doi.org/10.1007/s10994-026-07182-z

Image Credits: AI Generated

DOI: 10.1007/s10994-026-07182-z

Keywords: reinforcement learning, anomaly explanation, explainable AI, feature selection, PPO, subspace margin, anomaly detection, machine learning, interpretable AI, outlier detection, cardiotocography, high-dimensional data

In a study published in Cluster Computing, Yonghwan Kim, Sungwoo Kim, and Kichun Lee introduce osESD, an open-source package for Python and R that transforms the extreme studentized deviate test, a classical outlier-detection procedure dating back to Frank Grubbs’ work in 1969, into an online algorithm capable of learning on the fly. The result is a detector that keeps pace with streaming data whose statistical patterns shift over time, a scenario that has long crippled conventional approaches.

The foundation of the new method is the generalized extreme studentized deviate test, or GESD, a procedure that iteratively flags the most extreme observation in a sample, tests whether that observation is statistically inconsistent with the rest, and repeats until no further outliers emerge. GESD is prized for its simplicity and its ability to detect multiple outliers at once without requiring the user to know in advance how many anomalies are present. But it was designed for static batches of data, not for the endless, ever-changing rivers that define modern streaming systems. Applied naively to a live feed, GESD either floods the user with false alarms or goes deaf to genuine problems, because its internal assumptions about the data’s distribution break down the moment the underlying pattern changes.

The Hanyang team’s insight was to make the test adaptive. Their osESD algorithm, short for online sequential extreme studentized deviate tests, maintains two running summaries of the stream: change-rate trend vectors and residual trend vectors. The change-rate vectors track how quickly the data’s level and spread are evolving, allowing the method to distinguish a genuine anomaly from an ordinary shift in the data’s baseline. The residual vectors capture the discrepancy between observed values and the model’s current expectations, providing a normalized measure of surprise that remains meaningful even as the stream’s overall character drifts. Together, these components implement a form of online learning, updating the test’s internal state with each new observation rather than requiring a fixed training window.

This adaptability matters because real-world streams rarely hold still. A server’s traffic pattern changes with the workday. A factory sensor’s readings shift as equipment wears. A stock’s volatility swings with market sentiment. The researchers designed their synthetic benchmarks specifically to probe these regimes: six scenarios built from autoregressive integrated moving average models, including a stationary sanity check, a series with one abrupt parameter change at the halfway point, one with two abrupt changes, one with parameters that drift gradually and linearly, one with correlated noise, and one combining correlated noise with an abrupt change. They also generated state-space datasets with local linear trends and sinusoidal seasonality, varying the noise level across conditions to stress-test robustness.

Against these challenges, osESD delivered striking results. On the Yahoo benchmark datasets, a standard proving ground for time-series anomaly detection that includes real production traffic alongside labeled synthetic series, the method achieved perfect precision, recall, and F1 scores on example runs, identifying every labeled anomaly while flagging nothing that was not there. On one benchmark series, the Python implementation completed its analysis in roughly 0.7 seconds; the R version did the same job in under 0.3 seconds. The authors report that model learning is fast and that performance compares favorably with numerous competing models across both synthetic and real datasets, including streaming methods such as robust random cut forests and online isolation forests, which demand considerably more computational machinery.

Perhaps the most practical innovation is automation. Tuning an anomaly detector typically requires an expert to balance detection sensitivity against false-alarm rates, a trade-off that varies with every dataset. The package includes an auto_osESD function that performs a grid search over the algorithm’s parameters, including window sizes, significance levels, and learning lengths, and selects the configuration that best matches the user’s priorities. Users can weight precision, recall, F1 score, and computation time according to their needs: a fraud team might prioritize recall above all, while a latency-sensitive monitoring system might weight speed heavily. In demonstration runs, the automated tuner recovered near-perfect detection scores, with one configuration reaching an F1 score of about 0.985 on a challenging seasonal dataset. The function even handles unlabeled data, producing anomaly indices and visualizations without performance metrics when ground truth is unavailable.

The researchers validated their approach on real financial data as well, drawing on historical stock prices for Apple and Tesla from Yahoo Finance, where trading volumes exhibit exactly the kind of pattern changes that defeat static detectors. The breadth of the evaluation, spanning the four Yahoo benchmark suites, the six ARIMA scenarios, the seasonal state-space series, and real market data, gives the results unusual credibility for a methods paper, and the authors have released all their datasets, apart from the license-restricted Yahoo benchmarks, through a public GitHub repository.

The significance of the work extends beyond any single algorithm. It demonstrates that classical statistical tests, often dismissed as relics next to deep learning approaches such as autoencoders and long short-term memory networks, can be re-engineered for the streaming era with minimal computational cost. Neural methods for anomaly detection can be powerful but are notoriously hungry for training data, sensitive to hyperparameters, and opaque in their decisions. A sequential hypothesis test, by contrast, offers interpretable thresholds, principled significance levels, and a mathematical guarantee about false-alarm rates, qualities that matter enormously in industrial settings where a wrong alarm carries real consequences.

For organizations drowning in telemetry, the arrival of a lightweight, automated, statistically grounded anomaly detector available through a simple pip install is welcome news. The work, supported by the National Research Foundation of Korea, suggests a broader lesson for the field: the future of streaming analytics may depend less on inventing entirely new machinery than on teaching trusted old tools to keep learning. As data streams grow faster and less predictable, the tests that survive will be the ones that never stop adapting.

Subject of Research: Online statistical anomaly detection in pattern-changing streaming data using sequential extreme studentized deviate tests

Article Title: Anomaly detection of automatic online extreme studentized deviate tests for pattern-changing streaming data

Article References: Kim, Y., Kim, S., & Lee, K. (2026). Anomaly detection of automatic online extreme studentized deviate tests for pattern-changing streaming data. Cluster Computing, 29(12), Article 713. https://doi.org/10.1007/s10586-026-06514-x

Image Credits: AI Generated

DOI: 10.1007/s10586-026-06514-x

Keywords: anomaly detection, streaming data, GESD test, online learning, time series, statistical hypothesis testing, outlier detection, Python, machine learning, Cluster Computing, real-time monitoring, Anomaly

A team of statisticians at the University of KwaZulu-Natal in Durban, South Africa, has now proposed an elegant solution. In a study published in the journal Advances in Statistical Climatology, Meteorology and Oceanography, Natalie D. Benschop and her colleagues Temesgen Zewotir, Rajen N. Naidoo, and Delia North introduce a new data-standardization procedure designed to detect multiple forms of error simultaneously in strongly correlated meteorological sensor data. Their method, tested through extensive computer simulations and applied to a real network of monitoring stations, proved more comprehensive than comparable techniques at flagging everything from momentary spikes to prolonged instrument malfunctions.

The challenge the researchers set out to solve stems from the peculiar nature of high-frequency weather data. Because the Earth rotates on a predictable 24-hour cycle, hourly air temperature readings at one location tend to move in near lockstep with readings at neighbouring stations, provided they lie within roughly 200 kilometres of each other and experience the same weather systems. Hourly temperatures also depend on the hour of the day and the month of the year, giving the data multiple overlapping seasonal patterns. Any validation technique must respect this intricate correlation structure, yet most existing methods were never designed for it.

Existing approaches fall short in several ways, the authors argue. Many techniques reduce a set of correlated series to a single variable before hunting for anomalies, which means they only detect errors that strike several stations at once and overlook problems confined to a single series. Others assume that some series in a network are clean and can serve as trustworthy benchmarks, an assumption the researchers reject as unrealistic when every station runs unattended on hardware that may be cheap, aging, or misconfigured. Still others are tailored to catching just one type of outlier, forcing analysts to stack several methods together to achieve anything resembling complete coverage.

The new procedure builds on an existing double-standardization approach, originally developed for daily temperature data, but modifies it in two crucial ways. The first modification replaces conventional statistics with robust ones. Traditional z-scores, which measure how many standard deviations a value lies from the mean, are notoriously vulnerable to contamination: a handful of extreme readings can inflate the mean and standard deviation, masking the very outliers an analyst is trying to find. The team instead uses the median and the median absolute deviation, or MAD, statistics that remain stable as long as fewer than half the data points are erroneous. This idea echoes the Hampel identifier, a robust anomaly-detection technique popularized in the 1960s.

The second modification is more subtle and concerns how the initial standardization is performed. In the original double-standardization scheme, each series is first standardized with seasonally controlled parameters, meaning February temperatures are compared only with other February temperatures, and so on. That works well for daily data with a single seasonal cycle. But for hourly data, which carry both a daily and an annual cycle, every consecutive reading would be transformed with a different set of parameters, and this fragments the correlations between stations. The researchers instead standardize each series globally, using a single mean and standard deviation for the entire series. This preserves the strong temporal correlations between locations, allowing the second round of standardization, which compares all stations at each point in time, to exploit those correlations fully.

To test the method, the team ran a Monte Carlo simulation study. They generated clean hourly temperature series for 25 locations in South Africa’s Mpumalanga province, using observed daily minimum and maximum temperatures from the South African Air Quality Information System, and then deliberately corrupted the data with twelve different types of aberration: solitary spikes and dips, level shifts of varying duration, and irregular diurnal patterns caused by timestamp errors, some severe enough to invert the day-night temperature cycle entirely. The adapted procedure, combining robust statistics with global standardization, matched or beat every alternative in detecting severe and nearly all moderate perturbations, and it detected large portions of the irregular diurnal patterns that rival methods largely missed. Notably, it even caught a level shift lasting more than half the study period, a scenario the researchers expected to defeat all four techniques tested.

The real-world case study reinforced these findings. Applying the method to 28 series of hourly air temperatures recorded across Mpumalanga in 2018, the team flagged 1,399 suspicious values exceeding a threshold of four standardized deviations. After detailed interrogation, 763 of these were confirmed as genuine errors, and a further 103 values belonging to partially flagged sequences were also invalidated, for a total of 886 nullified readings. Among the confirmed errors were simultaneous temperature spikes at three stations that also reported physically impossible humidity values above 130 percent, a 22-hour cold snap at one station that saw readings plummet below freezing in April, and weeks of instrument malfunction at a station logging temperatures below minus 48 degrees Celsius. The method even uncovered a two-week inversion of the diurnal pattern at one station, where maximum temperatures were being logged at night and minimums during the day, a classic symptom of timestamp error.

The researchers are candid about the method’s limitations. Robust detection tends to produce somewhat higher rates of false positives, so a slightly higher deviation threshold than the conventional value of three is advisable; in retrospect, a threshold of five would have flagged only 0.6 percent of the case-study data while still catching all confirmed errors, pushing efficiency above 70 percent. The technique also requires a sufficiently large spatial set of strongly correlated series, ideally with typical correlations somewhat above 0.8, and it assumes the data are roughly symmetrically distributed, though the authors suggest trimmed statistics as a fallback for skewed data.

What makes the work broadly significant is its generality. The procedure applies to any collection of time series that are strongly correlated through time, whether they measure humidity, solar radiation, or ambient pressure, and the authors suggest potential uses even beyond meteorology. As automated sensor networks proliferate and low-cost devices are deployed in ever greater density, the volume of error-prone environmental data will only grow. A single, simple procedure that can simultaneously expose spikes, level shifts, and warped daily cycles, without assuming any series is clean, offers researchers a practical safeguard for the datasets on which much of modern environmental science depends.

Subject of Research: A data-standardization procedure for detecting multiple types of outliers in correlated high-frequency meteorological sensor data

Article Title: A new data-standardization procedure for comprehensive outlier detection in correlated meteorological sensor data

Article References: Benschop, N. D., Zewotir, T., Naidoo, R. N., & North, D. (2025). A new data-standardization procedure for comprehensive outlier detection in correlated meteorological sensor data. Advances in Statistical Climatology, Meteorology and Oceanography, 11(2), 133-158. https://doi.org/10.5194/ascmo-11-133-2025

Image Credits: AI Generated

DOI: 10.5194/ascmo-11-133-2025

Keywords: outlier detection, meteorological sensors, data standardization, time series, robust statistics, median absolute deviation, z-scores, air temperature, Monte Carlo simulation, South Africa, data quality, automatic weather stations

The study, published in Natural Hazards and Earth System Sciences by Veronica Gironelli and colleagues at the Istituto Nazionale di Geofisica e Vulcanologia (INGV) in Milan, addresses a stubborn gap in seismic hazard assessment. Italy possesses one of the richest macroseismic archives in the world, with intensity data points — IDPs — stretching back nearly a millennium. Yet the association between large historical earthquakes and the specific geological structures that generated them has traditionally rested on qualitative interpretation. The new approach replaces that subjectivity with a standardised pipeline that can, in principle, be applied systematically to every strong Italian earthquake since 1117.

The workflow begins with a deceptively simple but crucial step: cleaning the data. Macroseismic intensity is an ordinal measure, graded on scales such as the Mercalli–Cancani–Sieberg (MCS) or the European Macroseismic Scale 1998 (EMS-98), each with twelve degrees of increasing shaking severity. Because intensities are assigned from documentary descriptions of effects on people, buildings and the environment, individual data points can be anomalously high or low relative to the expected pattern of attenuation with distance. Such outliers can poison downstream analyses. The team therefore compares each observed intensity with the value predicted by a regionally calibrated intensity prediction equation (IPE), and flags as outliers any observation deviating by more than three standard deviations — roughly 2.25 intensity units — from the model prediction. This conservative threshold respects the inherent uncertainty of about one intensity degree in the data while removing only the most egregious discrepancies.

The importance of this pre-processing is vividly illustrated by the 5 May 1990 Potenza earthquake in the Southern Apennines. The macroseismic epicentre computed from the raw intensity field sat tens of kilometres from any instrumental location, a mismatch serious enough that the solution was excluded from Italy’s parametric earthquake catalogue. The residual analysis revealed 116 outliers, including suspiciously high intensities clustered around Naples — far from the true source — that turned out to reflect cumulative damage from the catastrophic 1980 Irpinia earthquake a decade earlier. Once these contaminated points were removed, the recalculated macroseismic epicentre moved to within 16.7 kilometres of the instrumental location, down from a 37.3-kilometre misfit. A similar exercise on the well-documented 2016 Amatrice earthquake showed that removing a single inflated intensity value at Pescara del Tronto, where damage had been amplified by unstable landslide deposits, brought the macroseismic magnitude down from 6.46 to 6.26 — a difference with major consequences, since the larger value would imply a rupture area of roughly 270 square kilometres, far exceeding the approximately 180 square kilometres constrained by satellite radar and geological data.

With a cleaned macroseismic field in hand, the workflow moves into its constructive phase. Earthquake parameters — epicentral location and magnitude — are recomputed using the Boxer code, the same tool underpinning the Italian Parametric Earthquake Catalogue CPTI15. Boxer locates the epicentre as the barycentre of the sites recording the highest intensities and estimates magnitude from the areas enclosed by isoseismal contours. Fault dimensions are then derived from magnitude using the widely adopted empirical scaling relationships of Wells and Coppersmith, and a rectangular three-dimensional fault plane is constructed around the hypocentre, with its orientation, dip and slip direction drawn from authoritative repositories including the Italian Seismic Bulletin, the CLASS catalogue of absolute locations, and the Database of Individual Seismogenic Sources (DISS).

The critical innovation lies in how candidate sources are tested. For each proposed fault geometry, the team simulates ground shaking using the ITA18 ground motion model, calibrated for shallow crustal earthquakes in Italy, computing predicted peak ground acceleration and velocity across the affected region. These ground motion parameters are converted into expected intensities using two independent ground-motion-to-intensity conversion equations, allowing the simulated shaking scenario to be compared directly, point by point, with the observed macroseismic field. The best-fitting source is the one that minimises two statistical metrics: the between-event residual, which captures systematic bias, and the root-mean-square error, which captures scatter. Thresholds of half an intensity degree for the former and one intensity degree for the latter define when a source can be considered reliable.

The method was validated on three instrumentally recorded earthquakes of different character: the 1984 Monti della Meta event (moment magnitude 5.86), the 1989 Prealpi Vicentine earthquake (magnitude 4.85), and the 2008 Parmense earthquake (magnitude 5.36). For Monti della Meta, nine alternative fault configurations were tested, and the winning model — built on the CLASS hypocentral location and the global centroid moment tensor magnitude — not only minimised the intensity residuals but also aligned its surface projection with the Barrea–Castelnuovo fault system mapped in the ITHACA database of capable faults. Crucially, where accelerometric recordings existed, the residuals between predicted and recorded ground motions matched the intrinsic standard deviation of the ground motion model itself, providing an independent stamp of validation.

The true test, however, comes from the deep past. The team applied the workflow to the 3 June 1781 Cagli earthquake, one of the most damaging events to strike the interior of the Marche region, and one with no clear seismotectonic interpretation. Two alternative, equally credible macroseismic fields exist for this event, and no capable faults are mapped at the surface near the epicentral area. Four source hypotheses were simulated: two consistent with the Montiego thrust proposed in the DISS database, and two aligned with a system of shallow normal faults in the Monte Nerone area documented by field geologists. The residual analysis favoured the sources built from the alternative intensity dataset, and suggested that a slightly lower magnitude — 6.33 rather than 6.51 — fits the observations better. In effect, the workflow allowed the researchers to adjudicate between competing historical datasets and competing tectonic hypotheses using quantitative evidence rather than expert judgement alone.

The implications extend well beyond Italian borders. Seismic hazard models in many regions of the world depend on catalogues of historical earthquakes whose magnitudes and locations are derived from exactly the kind of documentary evidence used here — and whose assigned faults are frequently matters of conjecture. By providing standardised acceptance thresholds, a transparent chain of processing, and validation against multiple independent benchmarks including instrumental recordings and mapped geological structures, the workflow offers a template that other countries with long documentary records could adapt. The authors note that the approach can be systematically applied to reconstruct the seismogenic sources of all Italian historical earthquakes of moment magnitude 5.5 and above for which consistent intensity distributions are available.

There is also a forward-looking payoff. Once a historical earthquake’s source is credibly identified, realistic shaking scenarios can be generated for the event — showing modern planners precisely which towns and infrastructure would be shaken if the same fault ruptured again. In a country where centuries-old faults still lie beneath densely populated cities, turning the written testimony of past survivors into a map of future risk may prove one of the most consequential applications of historical seismology in decades.

Subject of Research: A methodological workflow for identifying earthquake seismogenic sources from macroseismic intensity data

Article Title: A methodological workflow for the identification of earthquake sources from macroseismic data

Article References: A methodological workflow for the identification of earthquake sources from macroseismic data. (n.d.). https://doi.org/10.5194/nhess-26-4569-2026

Image Credits: AI Generated

DOI: 10.5194/nhess-26-4569-2026

Keywords: macroseismic data, seismogenic sources, earthquake intensity, historical seismology, seismic hazard assessment, intensity prediction equation, ground motion simulation, Boxer method, Italian earthquakes, fault geometry, outlier detection, shaking scenarios

The intellectual foundation of ADPOD is density-peak clustering, a technique introduced by Alex Rodriguez and Alessandro Laio in a landmark 2014 paper in Science. The core insight of that method was elegant: real cluster centers are points that sit in regions of high density while being far away from any other point of even higher density. Plotting density against distance to the nearest denser neighbor produces a characteristic pattern in which true cluster centers leap out as outliers of the decision graph itself. The approach became popular because it can identify clusters of arbitrary shape without requiring analysts to specify how many clusters exist in advance, and because it needs only a single pass over pairwise distances rather than the iterative refinement that k-means demands.

Yet for all its appeal, density-peak clustering has long carried well-known baggage. The original formulation estimates density by counting how many points fall within a fixed cutoff distance, a crude measure that ignores the fine-grained structure of the local neighborhood. Two points can have identical counts of nearby neighbors while occupying radically different positions in the geometry of the data. The method also traditionally requires a human to eyeball the decision graph and choose cluster centers by hand, which makes it difficult to deploy in automated pipelines. And when researchers tried to repurpose density-peak clustering for outlier detection, they discovered a deeper conceptual gap: the framework offered no effective relative-distance mechanism for deciding, point by point, whether an instance is normal or anomalous relative to the clusters around it.

ADPOD attacks each of these weaknesses in turn. The first and most fundamental innovation is a new way of measuring density, which the authors call affinity density. Instead of simply counting neighbors within a radius, the algorithm looks at the intersection between the k-nearest-neighbor sets of two samples, in other words, how many neighbors the two points share, and combines that overlap with the actual distances between them. This dual criterion captures something that raw counts cannot: the degree to which two points belong to the same local structure. Points that share many of the same closest companions are strongly affiliated, even if they are not literally adjacent, while points that share few or none are structurally isolated. The result is a density estimate that is sensitive to the shape and connectivity of the data rather than just its coarse packing.

The second step automates a task that previously demanded human judgment. ADPOD determines cluster centers automatically by computing the product of each point’s affinity density and its decision distance, the distance to the nearest point of higher density. Genuine cluster centers score high on both factors simultaneously: they sit in dense regions and are well separated from other dense regions. Taking the product means that a point must excel on both counts to be selected, which filters out false candidates that are merely isolated or merely dense. This removes the manual parameter tuning that has long been a bottleneck for practitioners who want to run density-peak methods at scale or embed them in systems where no expert is available to inspect the graphs.

Third, the algorithm replaces the raw Euclidean distance between data objects with a cluster relative distance. Rather than asking how far a point is from another point in absolute terms, ADPOD asks how far it is relative to the cluster structure it is being judged against. This matters because datasets rarely have uniform density. In a mixture of a tight, compact cluster and a loose, diffuse one, absolute distances systematically misjudge the diffuse cluster’s members, flagging legitimate points as suspicious simply because their neighborhood is spread out. A relative metric levels the playing field, evaluating each point against the local standards of its own cluster rather than against a global yardstick that favors compact regions.

The final component is a novel outlier factor that quantifies the degree of outlierness for every object in the dataset. Drawing together the affinity density, the automated cluster assignments, and the relative distance metric, this factor produces a score that ranks points by how anomalous they are, allowing analysts to set thresholds or inspect the most extreme cases first. The design reflects a broader trend in the field, visible in recent work on relative density ratios, local entropy, and granular-ball detectors: the recognition that a single global statistic is rarely sufficient, and that reliable detection requires combining local structural evidence with cluster-level context.

The experimental case for ADPOD rests on extensive testing across synthetic datasets, where the ground truth is known by construction, and real-world datasets, where it must be inferred. The authors evaluated performance using the area under the receiver operating characteristic curve, the standard measure introduced by Hanley and McNeil that captures how well a detector trades off true positives against false positives across all possible thresholds. Against a competitive field that includes LOF, the classic local-outlier-factor method from 2000; HBOS, a fast histogram-based scorer; ECOD, a recent approach based on empirical cumulative distribution functions; and a series of newer density-peak-derived detectors, ADPOD reportedly achieved significantly better results. The comparison set also spans graph-based methods, autoencoder approaches, and k-means variants with outlier removal, giving a broad picture of the current state of the art.

The practical stakes of this line of research are considerable. Outlier detection underpins fraud detection in finance, fault diagnosis in industrial robots and wireless sensor networks, cleaning of abnormal power-curve data from wind turbines, bot detection on social platforms, and anomaly monitoring over streaming text. In each of these settings, the cost of a missed anomaly can be severe, and the cost of a false alarm erodes trust in the system. Methods that require manual tuning are particularly ill-suited to these domains, where data arrives continuously and distributions drift over time. An algorithm that selects its own cluster centers and adapts its density estimates to local structure is a step toward detectors that can run unattended. The authors’ own prior work, including a relative skewness density ratio outlier factor published in the same journal, shows a sustained research program aimed at exactly this kind of automation.

ADPOD also illustrates how a decade-old idea can keep yielding returns when its assumptions are reexamined. Rodriguez and Laio’s density peaks gave the community a powerful mental model of clustering as the search for high-density islands separated by low-density moats. What ADPOD demonstrates is that the model’s two central ingredients, density and distance, were both defined too narrowly. Density becomes richer when it incorporates neighborhood overlap, and distance becomes fairer when it is measured relative to cluster context. With those refinements in place, the same decision-graph logic that once required a trained eye to interpret can be computed, scored, and turned directly into an outlier ranking. For a field increasingly asked to make trustworthy automatic judgments about which data can be believed, that combination of conceptual clarity and hands-off operation may prove to be the algorithm’s most important feature.

Subject of Research: Outlier detection using affinity density peak clustering

Article Title: ADPOD: An outlier detection algorithm based on affinity density peak clustering

Article References: ADPOD: An outlier detection algorithm based on affinity density peak clustering. (n.d.). https://doi.org/10.1007/s10489-026-07430-8

Image Credits: AI Generated

DOI: 10.1007/s10489-026-07430-8

Keywords: outlier detection, density-peak clustering, affinity density, anomaly detection, data mining, machine learning, clustering algorithms, k-nearest neighbors, unsupervised learning, Applied Intelligence, ADPOD, outlier

The motivation is straightforward. Hydrologists building machine learning models for precipitation prediction face a cascade of choices before a single forecast is produced. Should suspicious values in the raw record be flagged with classical statistical tests or with modern density-based algorithms? How should gaps in the data be filled? Which learning algorithm deserves the effort of hyperparameter tuning? And does adding another decade of historical data genuinely help? Most studies compare these ingredients one or two at a time within a single catchment, leaving open the possibility that conclusions drawn in one climate simply do not transfer to another. The new research tackles this gap head-on by running a full factorial experiment across five Iranian basins that span markedly different precipitation regimes.

The data underpinning the study come from ERA5, the European Centre for Medium-Range Weather Forecasts’ flagship global reanalysis product, which provides gridded monthly precipitation estimates from 1990 through 2023. Reanalysis products like ERA5 blend observations with numerical weather model output, offering a consistent long-term record even in regions with sparse ground instrumentation. The researchers deliberately framed the forecasting task in a simple, reproducible way: one-step-ahead prediction of monthly precipitation using the three preceding monthly values and the month number as predictors. This minimalist setup ensures that differences in performance can be attributed to the modeling components under investigation rather than to an elaborate feature-engineering pipeline.

The experimental design is where the study earns its statistical teeth. Eight different outlier-detection methods were applied to each basin’s precipitation series, ranging from classical procedures such as Grubbs’ test and Tukey’s fences to robust approaches like the Hampel filter and density-based techniques including the Local Outlier Factor and Isolation Forest. Six imputation methods were then used to fill any missing values. On top of each resulting dataset, the team trained four machine learning models representing distinct families of algorithms: Random Forest, an ensemble of decision trees; XGBoost, a gradient-boosting system known for its efficiency and accuracy on tabular data; Gaussian Process Regression, a Bayesian non-parametric method; and the Multi-Layer Perceptron, a classic feedforward neural network. Each model was trained on three different record lengths — 12, 22, and 32 years of monthly data — producing a large grid of modeling configurations for every basin.

Crucially, the evaluation protocol respected the temporal structure of the data. The final three years of each series were held out as an independent test period, untouched during model development. Hyperparameter optimization was performed exclusively within the training data using TimeSeriesSplit, a cross-validation scheme that preserves chronology, and the tuning was carried out separately for two objective functions: one maximizing the Nash–Sutcliffe Efficiency (NSE) and one maximizing the Kling–Gupta Efficiency (KGE). These two metrics capture different aspects of forecast quality — NSE emphasizes the match of variances around the observed mean, while KGE decomposes performance into correlation, bias, and variability — so optimizing for each separately guards against conclusions that hinge on the choice of score.

The headline result is a striking dependence of component importance on climate. In the humid Haraz basin, the choice of machine learning model was overwhelmingly the dominant factor, accounting for 82.0 percent of the variance in NSE and 59.9 percent of the variance in KGE. In other words, in a wet basin where precipitation is abundant and relatively well-behaved statistically, picking the right algorithm matters far more than any preprocessing decision. As mean precipitation decreases, however, the picture inverts. In the drier basins, outlier-removal methods rose to prominence, contributing up to 31.5 percent of the NSE variance and as much as 54.5 percent of the KGE variance — meaning that in arid settings, how you clean the data can matter as much as or more than which model you choose.

This pattern has a plausible physical explanation. In dry regions, monthly precipitation is low, episodic, and heavily skewed, so a handful of extreme or anomalous values exert an outsized influence on the training signal. An outlier-detection method that aggressively removes or downweights such points can reshape the entire learning problem, whereas in a humid basin the signal is strong enough that individual anomalies barely register against the background of frequent, substantial rainfall. The finding suggests that data-cleaning pipelines should be treated as climate-sensitive design choices rather than one-size-fits-all preprocessing steps.

Two other results will surprise practitioners. First, the choice of imputation method — how missing values are filled — generally made a negligible contribution to performance variance across the basins. The lone exception was the Aras basin under the KGE metric, where imputation accounted for 16.2 percent of the variance, a reminder that even broadly negligible factors can matter in specific climatic and metric contexts. Second, and perhaps most counterintuitively, the length of the training record showed no consistent direct relationship with improved predictive performance. Adding ten or twenty extra years of data did not reliably produce better forecasts, challenging the common assumption that more history is always better in hydrological machine learning.

The methodological lesson is that factorial ANOVA offers a powerful, underused lens for machine learning research in the geosciences. By treating each modeling decision as a factor in a designed experiment and quantifying the share of performance variance each factor explains, researchers can move beyond leaderboard-style comparisons of individual algorithms and toward a principled understanding of where modeling effort is best spent. The approach also naturally exposes interactions between components — for instance, whether the benefit of a particular outlier filter depends on which algorithm is downstream of it — which pairwise comparisons tend to obscure.

For water managers and forecasters, the practical takeaway is that there is no universally optimal recipe for precipitation prediction. A workflow tuned in a temperate, humid catchment may misallocate effort entirely if transplanted to an arid one, where data cleaning deserves the lion’s share of attention. The authors argue for basin-specific, climate-aware modeling strategies: diagnose the precipitation regime first, then decide whether algorithm selection or data preprocessing is the higher-leverage investment. As machine learning spreads deeper into hydrology and water resources management, studies like this one suggest that the most valuable question is not which model is best, but which decision, in this place and this climate, actually moves the needle.

Subject of Research: Relative importance of machine learning modeling components for precipitation prediction across contrasting climatic basins

Article Title: Assessing the relative importance of machine learning modeling components for precipitation prediction using ANOVA

Article References: Dalir Gabrabad, A., Hosseinpour, M., & Ashrafzadeh, A. (2026). Assessing the relative importance of machine learning modeling components for precipitation prediction using ANOVA. Earth Science Informatics, 19(11), Article 204. https://doi.org/10.1007/s12145-026-02260-1

Image Credits: AI Generated

DOI: 10.1007/s12145-026-02260-1

Keywords: machine learning, precipitation forecasting, ANOVA, outlier detection, missing data imputation, Random Forest, XGBoost, Gaussian Process Regression, Multi-Layer Perceptron, ERA5 reanalysis, Nash-Sutcliffe Efficiency, Kling-Gupta Efficiency

To understand why this matters, it helps to recall how geometric model fitting has evolved. Since Martin Fischler and Robert Bolles introduced the Random Sample Consensus paradigm, or RANSAC, in 1981, the standard approach has been to repeatedly sample small subsets of points, hypothesize a model such as a line, a plane, or a homography, and count how many points agree with it. When a scene contains only one structure and a moderate number of outliers, this strategy works remarkably well. But real scenes are rarely so cooperative. An image pair of a building contains dozens of planes at different depths; a point cloud of a street contains ground surfaces, vehicle bodies, and facades simultaneously. Multi-structure fitting requires the algorithm not only to find models but to segment the data, deciding which points belong to which model instance and which are simply noise.

In recent years, researchers have increasingly turned to density-based clustering algorithms to perform this segmentation. Methods such as DBSCAN, introduced by Ester and colleagues in 1996, group points based on how densely they pack together, and their robustness to noise made them attractive for fitting pipelines. However, as the authors of the new study point out in their abstract, these algorithms carry a critical limitation: they depend on a fixed threshold parameter, conventionally called MinPts, which specifies the minimum number of neighbors required for a point to be considered part of a dense region. This threshold often lacks adaptability to varying data characteristics. When input data is severely contaminated with outliers, determining suitable threshold values becomes genuinely difficult, and a poorly chosen value leads directly to suboptimal fitting results, with inlier structures fragmented or outliers absorbed into spurious clusters.

The core innovation of IACF is to let information theory decide the threshold automatically. Entropy, in the sense introduced by Claude Shannon, quantifies the disorder or uncertainty of a distribution. The researchers propose an entropy-based adaptive strategy that measures the disorder of local neighborhoods in the data and uses this measurement to dynamically determine an optimal clustering threshold, denoted MinPts*, for each specific dataset without any prior knowledge. In other words, instead of an engineer guessing how many neighbors should define a dense region, the algorithm inspects the statistical structure of the data itself and derives the answer. This is a meaningful shift: the parameter that most strongly controls density-based clustering behavior is no longer a fixed constant but a data-driven quantity that adapts to the contamination level and geometric complexity of each problem instance.

The method unfolds in three coordinated stages. First, the authors construct a reliable neighborhood graph by integrating motion and preference information. Preference analysis, a technique well established in the multi-model fitting literature through works such as T-linkage by Magri and Fusiello and accelerated hypothesis generation by Chin, Yu, and Suter, represents each data point by its pattern of agreement with a large set of sampled hypotheses. Points belonging to the same structure share similar preference vectors, so combining this preference signature with motion consistency information allows the graph-building step to prune invalid edges while preserving the potential local inlier structures that matter for fitting. This graph pruning is the first line of defense against outliers, removing connections that would otherwise let mismatched points contaminate each other’s neighborhoods.

Second, the entropy-based adaptive strategy operates on this graph, quantifying the disorder of local neighborhoods to set the optimal MinPts* threshold for the dataset at hand. Third, building on both components, the authors develop an Entropy-Adaptive Density-based Dominant Set Clustering algorithm, or EADSC. Dominant set clustering, formalized by Pavan and Pelillo in their influential work on pairwise clustering, is a graph-theoretic framework in which clusters emerge as dominant sets, subsets of vertices that are internally coherent with respect to their external environment. By making the dominant set search entropy-adaptive, EADSC can autonomously detect model-related subgraphs corresponding to individual model instances and accurately partition inliers from outliers, without the analyst supplying the number of structures in advance.

The experimental evaluation reported in the paper covers several challenging datasets, and the authors state that the proposed IACF method outperforms several state-of-the-art fitting methods in terms of both fitting accuracy and computational efficiency. That dual improvement is notable because robustness and speed are usually in tension in this field. Methods that achieve high accuracy on heavily contaminated data, such as hypergraph optimization approaches, energy minimization and mode-seeking frameworks, and residual-based consensus techniques, often pay a steep computational price, while faster methods sacrifice accuracy when outlier ratios climb. An adaptive threshold that correctly identifies inlier structure on the first pass reduces wasted hypothesis sampling, which is where much of the computational cost of fitting pipelines accumulates.

The context in which this work appears is a field under growing pressure from real-world deployment. The reference list of the paper traces the trajectory of the problem: from RANSAC and its many guided-sampling descendants, through preference-analysis methods and hypergraph formulations, to recent deterministic fitting approaches such as latent semantic consensus and spatial clustering guided two-view fitting, and graph-based techniques for mismatch removal. Applications cited in the surrounding literature range from multimodal image matching and feature matching with large numbers of outliers, to instantaneous frequency estimation of multi-component signals with multi-sensor consensus, to 3D point cloud map merging for large-scale environments and defect detection in printed-circuit boards. Each of these tasks shares the same mathematical skeleton: data generated by multiple geometric models plus noise, and an algorithm that must recover the models without knowing in advance how many there are or how badly the data is corrupted.

What makes the entropy angle particularly compelling is its generality. The authors connect their contribution to a broader line of research on autonomous clustering, including recent work by Yang and Lin on clustering by fast find of mass and distance peaks and toward autonomous distributed clustering. The shared ambition across these efforts is to remove human tuning from the clustering loop, letting statistical measures of structure determine algorithmic parameters. In the IACF framework, entropy serves as a universal yardstick of neighborhood disorder: a well-organized inlier neighborhood has low entropy, while a neighborhood polluted by outliers exhibits high disorder. By searching for the threshold that best separates these regimes, the algorithm effectively asks the data how much contamination it contains and adjusts its density criteria accordingly. This is precisely the kind of self-configuration that matters for systems operating in uncontrolled environments, where the outlier ratio can swing dramatically between frames or between scenes.

The practical implications extend across the computer vision stack. Robust multi-structure fitting sits upstream of fundamental estimation tasks: fundamental matrix and homography estimation in two-view geometry, motion segmentation in video, plane detection in point clouds, and registration of 3D scans. Improving the accuracy and speed of the fitting stage propagates improvements into every downstream application, from augmented reality anchoring to industrial inspection. The authors have made the datasets used in the study publicly available through the Adelaide RMF benchmark repository and a project page, and state that the custom code is available from the corresponding author upon reasonable request, which should make the method straightforward for other groups to evaluate against their own pipelines.

Funded by the Natural Science Fund of Fujian Province, the Natural Science Foundation of Xiamen, the National Natural Science Foundation of China, and several Xiamen municipal science and technology programs, the research reflects a sustained investment in the geometric foundations of machine perception. Whether entropy-adaptive thresholds will become a standard component of future fitting pipelines remains to be seen, but the study makes a clear case that the era of fixed MinPts values in density-based fitting may be drawing to a close. As vision systems are asked to operate in ever messier environments, algorithms that measure the disorder of their own inputs and adapt on the fly offer a persuasive path toward fitting that is both more accurate and more efficient, no matter how many structures the world throws at them.

Subject of Research: Information-theoretic adaptive density clustering for robust multi-structure geometric model fitting in computer vision

Article Title: Information-theoretic adaptive clustering for robust multi-structure model fitting

Article References: Information-theoretic adaptive clustering for robust multi-structure model fitting. (n.d.). https://doi.org/10.1007/s10586-026-06563-2

Image Credits: AI Generated

DOI: 10.1007/s10586-026-06563-2

Keywords: model fitting, density-based clustering, information entropy, adaptive threshold, dominant sets, RANSAC, computer vision, outlier removal, preference analysis, multi-model fitting, neighborhood graph, Cluster Computing

The problem is more consequential than it might first appear. Most outlier detection algorithms were designed with numerical data in mind. They measure distances, densities, and deviations in spaces where every attribute is a number, and where concepts like proximity are mathematically straightforward. But real datasets are rarely so cooperative. Medical records contain diagnoses and blood types. Customer databases contain product categories, subscription tiers, and regions. Fraud datasets mix transaction amounts with merchant categories and card types. In these settings, the numerical machinery of classical anomaly detection either breaks down or silently distorts the data, and practitioners have been left to improvise without solid evidence about which improvisation works.

Ferreira and Cordeiro frame the field’s improvisations as three competing strategies. The first is to apply algorithms that can process categorical data directly, respecting the discrete nature of attributes such as color, brand, or diagnosis. The second is to convert categorical attributes into numerical ones before detection, transforming the data into a form that standard numerical detectors can consume. The third is the bluntest instrument: simply remove the categorical attributes and run detection on the numerical columns alone. Each strategy has intuitive appeal and intuitive drawbacks, but until now there has been no rigorous, large-scale comparison of how they actually perform across diverse datasets and detectors.

To settle the question, the researchers mounted an unusually comprehensive experimental campaign. They assembled 47 datasets and ran 14 different outlier detection algorithms across them, evaluating all three strategic approaches under controlled conditions. This scale matters. Many published comparisons in machine learning rest on a handful of benchmark datasets, which makes it dangerously easy to overfit conclusions to particular data quirks. By sweeping across dozens of datasets with varied characteristics, the study could distinguish between strategies that win consistently and strategies that win only under specific conditions, a distinction that turns out to be central to the paper’s findings.

The headline result is that the first strategy, applying algorithms that handle categorical data natively, is usually the preferred choice. And within that strategy, one detector stood out from the pack: CBRW, an algorithm that approaches categorical outliers through a weighted-graph perspective on attribute values. When datasets contain categorical attributes, the evidence indicates that reaching for a purpose-built categorical detector, particularly CBRW, is the most reliable default. This finding carries real practical weight, because it suggests that the common habit of forcing categorical data through numerical pipelines may often be an unnecessary detour that costs detection quality.

But the study resists the temptation of a one-size-fits-all verdict. The second strategy, converting categorical attributes into numbers before detection, proved superior in certain contexts, and the researchers found that which contexts depends on the characteristics of the data itself. Notably, when this conversion route was taken, detectors such as Isolation Forest, known as iForest, and KNN-outlier achieved better results in those favorable settings. Isolation Forest works by randomly partitioning the data and isolating records that are easy to separate from the rest, while KNN-outlier flags points that sit far from their nearest neighbors. Both are staples of the numerical anomaly detection toolkit, and the study shows they can still earn their keep on mixed data, provided the categorical-to-numerical conversion is done well.

That proviso led the authors to a second, finer-grained comparison: among the many techniques for converting categorical attributes into numerical representations, which one should practitioners choose? Here again the study produced a clear signal. Correspondence Analysis, a classical statistical method that maps categorical variables into a low-dimensional numerical space while preserving the associations between attribute values, often yielded the best detection results among the conversion approaches evaluated. For teams committed to running numerical detectors on mixed data, this finding offers a concrete, evidence-backed recommendation in place of an arbitrary choice of encoding technique.

Perhaps the most forward-looking contribution goes beyond ranking the strategies. Because the best approach depends on data characteristics, the researchers built a predictive model, a meta-level classifier trained on their experimental findings, that examines a new dataset and predicts which of the three strategies is most likely to perform best. The model achieves 80 percent accuracy in identifying the winning strategy. In a field where practitioners often have no principled basis for choosing an approach, an 80 percent reliable recommendation engine represents a meaningful step toward automated, data-driven method selection, analogous to the meta-learning systems that have begun to guide algorithm choice in supervised learning.

The study also delivers a quieter but equally important negative result: the third strategy, discarding categorical attributes altogether, does not emerge as a recommended path. Throwing away information is the kind of shortcut that survives in practice because it is easy, and part of the value of a systematic comparison is that it can retire such shortcuts with evidence. The findings collectively suggest that the information carried by categorical attributes is genuinely useful for anomaly detection, whether consumed directly by a categorical-aware algorithm or preserved through a careful numerical conversion such as Correspondence Analysis.

For the broader machine learning community, the significance of this work lies in its methodological honesty. Rather than crowning a single universal champion, it maps the conditions under which each strategy excels and then builds a bridge from those conditions to practical recommendations for new data. The authors have also committed fully to reproducibility: all code, detailed results, parameter values tested, and datasets used in the study are freely available for download on GitHub, allowing other researchers to scrutinize, extend, or challenge the conclusions. Supported in part by the Brazilian agency CAPES and published open access, the survey arrives as both a practical handbook for anyone deploying anomaly detection on messy real-world data and a benchmark against which future categorical outlier detection methods will now have to be measured. As fraud, cybersecurity, and data-quality applications continue to multiply across industries dominated by categorical information, the question this study answers is no longer an academic curiosity. It is an operational decision worth getting right, and for the first time, practitioners have a large-scale evidence base to guide it.

Subject of Research: Comparative evaluation of outlier detection methods for categorical and mixed data

Article Title: A comparative evaluation of outlier detection in categorical and mixed data

Article References: Ferreira, F. P., & Cordeiro, R. L. F. (2026). A comparative evaluation of outlier detection in categorical and mixed data. Artificial Intelligence Review. https://doi.org/10.1007/s10462-026-11687-3

Image Credits: AI Generated

DOI: 10.1007/s10462-026-11687-3

Keywords: outlier detection, categorical data, mixed data, anomaly detection, CBRW, Isolation Forest, KNN-outlier, Correspondence Analysis, data conversion, machine learning, data mining, benchmark evaluation

The paper’s most striking number concerns how these pipelines have been measured, not how they perform. When feature selection, scaling, outlier removal and the popular SMOTE-ENN resampling technique are all fitted on the full dataset before cross-validation — a practice the authors trace through much of the published literature — reported F1 scores are inflated by an average of 28.9 percentage points, with a range from 0.4 to 61.2 points depending on the cohort. The dominant culprit is resampling before splitting: SMOTE synthesises new minority-class points by interpolating between existing ones, so a test-fold patient can become a near-duplicate of training data, while the accompanying Edited Nearest Neighbours rule deletes borderline majority records using their labels. The evaluation, in effect, grades the model on questions it has already seen.

The inflation is largest precisely where the underlying task is hardest. On a small hypertension cohort of 175 patients, where blood pressure is essentially unpredictable from anthropometric measurements alone, the flawed protocol reports an F1 of 93.75 percent; the corrected protocol yields 35.5 percent, with the area under the ROC curve hovering near chance. On an easy, well-posed chronic kidney disease diagnosis task, the difference between protocols is a mere 0.4 points. Leakage, the authors conclude, does not add a constant bonus — it manufactures performance in proportion to how little genuine signal exists, which is why mid-to-high-nineties accuracy figures on small clinical cohorts deserve deep suspicion.

Under the corrected protocol, with every fitted step confined inside the training fold, the benchmark’s verdict on outlier removal is unambiguous. Twelve detectors — including tuned DBSCAN, Local Outlier Factor, Isolation Forest, One-Class SVM, robust covariance estimation, HDBSCAN, and the recent distribution-based methods ECOD and COPOD — were compared against a no-cleaning control over 50 folds per cell, using the Nadeau–Bengio corrected resampled t-test with Holm adjustment to account for overlapping training sets. Across all 130 method-by-cohort cells, no detector significantly beat the control. The single statistically significant result in either direction was a loss: HDBSCAN on the diabetic retinopathy cohort, dropping F1 by 9.37 points.

The authors did not merely tune their own detectors carefully and declare victory. To test the objection that poor configuration explains the null result, they granted an oracle: for each cohort, the best of 44 parameter settings of their proposed method was selected with hindsight on the very folds being reported. Even this impossible advantage yielded a mean gain of only 1.24 points — less than the 3.20 points that picking the best of 44 genuinely equivalent settings would produce from fold-to-fold noise alone. On two cohorts, the oracle’s chosen configuration removed nothing whatsoever. Across all 440 evaluated configurations, F1 correlated negatively with the fraction of data removed on seven of nine informative cohorts, as strongly as r = −0.99. Within this family of methods, removing more is monotonically worse.

If cleaning were merely useless, it would be a wasted step. The study’s forensic analysis shows it is actively hazardous. Every detector examined removes minority-class records preferentially — the proposed YadroSeg method deleted 32.8 percent of minority-class patients against 18.5 percent of majority-class records, a pattern significant on 8 of 10 cohorts, while Isolation Forest showed the most extreme ratio at 4.4 to 1. Triaging the deleted rows against physiological ranges, only about 15 percent contained impossible values that could be called genuine data errors; roughly 63 percent were rare but clinically plausible patients. The mechanism is not a defect in any particular algorithm: removal correlates strongly with local sparsity in feature space (r between 0.43 and 0.69) and barely at all with label impurity. Density-based detectors find sparse regions, and in clinical data sparse regions are populated by uncommon patients, not corrupted records.

The study is also a remarkable act of scientific self-correction. An earlier version of this work reported that its graph-based detector, YadroSeg, achieved the best F1 on three of five datasets, reaching 98.87 percent — a figure produced by the very leakage the revised paper dissects. The same configuration scores 100.00 under the flawed protocol and 58.51 under the corrected one. The authors pre-registered a series of tests to understand why their density-variation analysis, adapted from image segmentation, failed to transfer to tabular data, and report that two of their own registered explanations were refuted by the experiments designed to check them. The final finding is structural: driving the coring threshold until no core vertex survives leaves the set of removed records bit-identical on all ten cohorts, because the density sequence over a constructed clinical feature graph lacks the small set of dominant boundaries that thresholding presupposes.

What survives the transfer is the density ordering itself. When exposed as a continuous score rather than collapsed into a threshold, the peeling sequence recovered injected corruption better than the published algorithm on ten of ten cohorts, ranking second of nine scorers behind LOF. Measuring this honestly required the authors to correct their own evaluation a second time: their initial comparison scored detectors on the arbitrary masks they emitted, at flag rates differing by more than an order of magnitude. At a matched flag budget, every detector tested ranked corrupted records two to three times above chance — though none exceeded a detection F1 of 0.275, confirming that none of them is a reliable corruption detector, particularly for flipped labels, which leave a record exactly where it was in feature space.

The corrected implementation also brings a scalability lesson. The original formulation required a dense distance matrix that scaled as N to the power 2.67 and became infeasible beyond 32,000 records; the revised version, using radius queries on a spatial index, measures at N to the power 1.48 — competitive with DBSCAN and LOF, though an order of magnitude behind Isolation Forest’s effectively flat profile. The authors withdraw their earlier claim of O(N log N) scaling as unsupported by measurement, and confirm the exponents on two real clinical registries of over 100,000 and 250,000 records. A large-cohort confirmation arm on those registries, reaching nearly nine times the sample size of the largest primary cohort, reproduced the central null: cleaning never helped, and where differences reached significance they ran against it.

The practical recommendations are pointed. Every fitted preprocessing step should live inside the cross-validation loop, ideally enforced structurally through pipeline tools rather than manual sequencing. Any study adopting outlier removal should include a no-cleaning control and report the class composition of what it deletes. Calibration should be reported alongside discrimination, since resampling is known to damage probability calibration — a property the study observed across all configurations. And for reviewers, the single most informative check on a paper using the cleaning-plus-resampling-plus-ensemble template is whether the resampler is fitted inside or outside the validation loop. On the balance of evidence, the authors recommend against routine density-based outlier removal in clinical tabular prediction: a measurable risk of discarding rare patients, with no measurable benefit in return.

Subject of Research: Evaluation leakage and outlier removal in machine-learning prediction of chronic diseases from clinical tabular data

Article Title: Graph-based density-variation outlier removal for chronic disease prediction: A leakage-free benchmark and component-wise ablation

Article References: Davronov, R. R., & Adilova, F. T. (2026). Graph-based density-variation outlier removal for chronic disease prediction: A leakage-free benchmark and component-wise ablation. Results in Engineering, 32, Article 112842. https://doi.org/10.1016/j.rineng.2026.112842

Image Credits: AI Generated

DOI: 10.1016/j.rineng.2026.112842

Keywords: machine learning, outlier detection, data leakage, chronic disease prediction, SMOTE, class imbalance, DBSCAN, clinical data, cross-validation, YadroSeg, reproducibility, density-based clustering

The core problem the authors tackle is that count matrices behave differently from typical tabular data. In a standard spreadsheet, each column describes a different trait — a birth year, a height, a gender — and the columns do not even share a common data type. In a count matrix, by contrast, every entry counts occurrences of the same kind of event, and both rows and columns share a common domain. Generic clustering algorithms, which usually assume continuous features or require pre-processing such as normalization, often produce unreliable results on raw counts. Earlier work, notably a widely cited 2010 paper by O’Hara and Kotze, showed that log-transforming count data is frequently unsuitable, which limits the usefulness of traditional pipelines that depend on such transformations.

3CPO — short for Clustering and Column selection of Count Data using a Poisson-based Optimization — builds on a simple but powerful modeling idea: the expected value of each entry in the matrix can be written as a product of a row-specific factor and a column-specific factor. The row factor captures the overall scale of a row, for example the length of a document, while the column factors act like cluster centroids, describing the proportions that characterize each group of rows. This formulation, which echoes earlier Poisson clustering methods such as PoissonL and PoissonC, assumes that every row is scaled by its size and that every cluster follows its own characteristic column proportions.

Those assumptions are strict, and real data routinely violate them. The authors illustrate the point with word counts in letters: a greeting phrase appears roughly once regardless of letter length, contradicting the scaling assumption, while common words like ‘the’ scale with document length but carry little information about the topic. To handle this, 3CPO divides the columns of the matrix into three disjoint subsets. Columns in the first subset, C1, are genuinely relevant for clustering and follow the full Poisson model with cluster-specific parameters. Columns in C0 scale with the rows but behave similarly across all clusters, so they are modeled with a single shared parameter. Columns in C− are pure noise, modeled by a column-specific average that ignores both row scale and cluster membership.

The algorithm then solves an optimization problem: find the parameters, the clustering of rows, and the column partition that minimize a log-loss derived from the Poisson likelihood. It does so with an Expectation Maximization-style iterative procedure that alternates between updating the expected values, reassigning rows to clusters using a Poisson-based score, and reassigning columns to the three subsets. A penalty term motivated by the Minimum Description Length principle keeps the set of relevant columns small — intuitively, describing K cluster-specific values for a column costs about (K−1) times the logarithm of the column sum in bits, so a column only earns its place in C1 if the improvement in likelihood justifies that cost. Because each step never increases the loss and there are finitely many possible clusterings, the procedure is guaranteed to converge to a local optimum, with a worst-case runtime that grows linearly in the number of rows, columns, clusters, and iterations.

The experiments are extensive. The authors compared 3CPO against Poisson-based baselines, Spherical k-Means, k-Means combined with several normalizations (z-scores, min-max scaling, relative frequencies, and the revealed comparative advantage measure used in economics), and a suite of co-clustering algorithms including CROINFO, CoclustMod, CoclustSpecMod, ELBM, SELBM, and TauCC. They evaluated one synthetic and eleven real-world data sets spanning gene expression, single-cell RNA sequencing, text corpora such as BBCSports, BBCNews, Reuters21578 and 20Newsgroups, handwritten digits, and economic wholesale data, using Unsupervised Clustering Accuracy, Normalized Mutual Information, and the Adjusted Rand Index.

The results are striking. 3CPO was the top performer in eight of twelve comparisons against traditional clustering algorithms, beating all competitors by more than 38 percent on the synthetic data and by more than 6 percent on WebKB, while remaining within one standard deviation of the best method in most cases where it did not win. Against co-clustering methods it ranked among the top three on every data set. On text data, it outperformed k-Means and Spherical k-Means combined with TF-IDF and BM25 weighting in four out of five scenarios, despite working directly on raw word counts. Because it operates on interpretable counts rather than opaque embeddings, domain experts can inspect the selected columns and see exactly which terms drive each cluster — the analysis of BBCSports and BBCNews showed words like ‘party’ and ‘govern’ characterizing a politics cluster and ‘athlete’ and ‘olymp’ characterizing an athletics cluster, while stop words were correctly relegated to the uninformative subsets.

The column selection itself proved remarkably effective. On high-dimensional data sets such as BBCSports, Reuters, 20Newsgroups and a gene expression data set, 3CPO kept only a fraction of the original columns — roughly 32 percent for the gene expression data and about 40 percent for BBCSports — while still outperforming methods that used everything. Histogram analyses showed the algorithm was not simply discarding sparse, zero-heavy columns; the distribution of zeros was similar across all three column subsets, indicating that 3CPO responds to genuine structural patterns in the counts. Robustness experiments reinforced the point: when noise columns with uniformly distributed values were added to the synthetic data, 3CPO was the only algorithm whose clustering quality remained perfect, and it tolerated noise values up to around 32 before degrading, far beyond the limits of every competitor.

The authors also built in an optional outlier detection mechanism. Rows are compared against a data set-wide background model, and any row that fits no cluster better than the background is flagged as an outlier rather than forced into one. This improved clustering scores on nearly all data sets, and on the gene expression data set 3CPO achieved a perfect clustering result while identifying only about fourteen outliers. An additional MDL-based penalty even allows the algorithm to estimate the number of clusters itself: it recovered the correct number for the synthetic and BBCSports data sets and was off by only one for the gene expression data, outperforming standard heuristics such as elbow detection, silhouette scores, and BIC-based estimates in the high-dimensional text settings.

The authors are candid about limitations. The Poisson model ties the variance to the mean, so data with strong overdispersion or zero inflation might be better served by a negative binomial formulation; the penalty terms involve heuristic choices; and the number of clusters must either be supplied or estimated with the proposed heuristic, which struggled on some tabular data sets. Still, the overall message is compelling: by taking the statistical nature of counts seriously and letting the data itself reveal which features matter, 3CPO delivers clusters that are both more accurate and far easier to interpret. The code is publicly available on GitHub, and the authors suggest that refining outlier handling and integrating cluster-number estimation more tightly are promising directions for future work. For anyone analyzing contingency tables, text counts, or sequencing data, the study makes a strong case that the essentials are best found by modeling the counts as they are — not by transforming them into something they are not.

Subject of Research: A Poisson-based subspace clustering algorithm for count data with integrated column selection

Article Title: Poisson subspace clustering: focusing on the essentials in count data

Article References: Leiber, C., Puolamäki, K., & Mannila, H. (2026). Poisson subspace clustering: focusing on the essentials in count data. Data Mining and Knowledge Discovery, 40(6), Article 98. https://doi.org/10.1007/s10618-026-01230-x

Image Credits: AI Generated

DOI: 10.1007/s10618-026-01230-x

Keywords: count data, Poisson distribution, clustering, subspace clustering, column selection, expectation maximization, minimum description length, data mining, machine learning, gene expression, text clustering, outlier detection