New Standardization Method Catches Every Kind of Error in Weather Sensor Data

Automatic weather stations have quietly become the backbone of modern environmental science. Scattered across cities, farmland, and remote wilderness, these programmable devices measure air temperature, humidity, solar radiation, and atmospheric pressure every few minutes, day and night, without a human anywhere in sight. The resulting torrents of data feed climate models, health studies, and renewable-energy forecasts. But there is a problem that researchers have long preferred not to dwell on: the data are often riddled with errors, and no single existing technique can reliably catch them all.

A team of statisticians at the University of KwaZulu-Natal in Durban, South Africa, has now proposed an elegant solution. In a study published in the journal Advances in Statistical Climatology, Meteorology and Oceanography, Natalie D. Benschop and her colleagues Temesgen Zewotir, Rajen N. Naidoo, and Delia North introduce a new data-standardization procedure designed to detect multiple forms of error simultaneously in strongly correlated meteorological sensor data. Their method, tested through extensive computer simulations and applied to a real network of monitoring stations, proved more comprehensive than comparable techniques at flagging everything from momentary spikes to prolonged instrument malfunctions.

The challenge the researchers set out to solve stems from the peculiar nature of high-frequency weather data. Because the Earth rotates on a predictable 24-hour cycle, hourly air temperature readings at one location tend to move in near lockstep with readings at neighbouring stations, provided they lie within roughly 200 kilometres of each other and experience the same weather systems. Hourly temperatures also depend on the hour of the day and the month of the year, giving the data multiple overlapping seasonal patterns. Any validation technique must respect this intricate correlation structure, yet most existing methods were never designed for it.

Existing approaches fall short in several ways, the authors argue. Many techniques reduce a set of correlated series to a single variable before hunting for anomalies, which means they only detect errors that strike several stations at once and overlook problems confined to a single series. Others assume that some series in a network are clean and can serve as trustworthy benchmarks, an assumption the researchers reject as unrealistic when every station runs unattended on hardware that may be cheap, aging, or misconfigured. Still others are tailored to catching just one type of outlier, forcing analysts to stack several methods together to achieve anything resembling complete coverage.

The new procedure builds on an existing double-standardization approach, originally developed for daily temperature data, but modifies it in two crucial ways. The first modification replaces conventional statistics with robust ones. Traditional z-scores, which measure how many standard deviations a value lies from the mean, are notoriously vulnerable to contamination: a handful of extreme readings can inflate the mean and standard deviation, masking the very outliers an analyst is trying to find. The team instead uses the median and the median absolute deviation, or MAD, statistics that remain stable as long as fewer than half the data points are erroneous. This idea echoes the Hampel identifier, a robust anomaly-detection technique popularized in the 1960s.

The second modification is more subtle and concerns how the initial standardization is performed. In the original double-standardization scheme, each series is first standardized with seasonally controlled parameters, meaning February temperatures are compared only with other February temperatures, and so on. That works well for daily data with a single seasonal cycle. But for hourly data, which carry both a daily and an annual cycle, every consecutive reading would be transformed with a different set of parameters, and this fragments the correlations between stations. The researchers instead standardize each series globally, using a single mean and standard deviation for the entire series. This preserves the strong temporal correlations between locations, allowing the second round of standardization, which compares all stations at each point in time, to exploit those correlations fully.

To test the method, the team ran a Monte Carlo simulation study. They generated clean hourly temperature series for 25 locations in South Africa’s Mpumalanga province, using observed daily minimum and maximum temperatures from the South African Air Quality Information System, and then deliberately corrupted the data with twelve different types of aberration: solitary spikes and dips, level shifts of varying duration, and irregular diurnal patterns caused by timestamp errors, some severe enough to invert the day-night temperature cycle entirely. The adapted procedure, combining robust statistics with global standardization, matched or beat every alternative in detecting severe and nearly all moderate perturbations, and it detected large portions of the irregular diurnal patterns that rival methods largely missed. Notably, it even caught a level shift lasting more than half the study period, a scenario the researchers expected to defeat all four techniques tested.

The real-world case study reinforced these findings. Applying the method to 28 series of hourly air temperatures recorded across Mpumalanga in 2018, the team flagged 1,399 suspicious values exceeding a threshold of four standardized deviations. After detailed interrogation, 763 of these were confirmed as genuine errors, and a further 103 values belonging to partially flagged sequences were also invalidated, for a total of 886 nullified readings. Among the confirmed errors were simultaneous temperature spikes at three stations that also reported physically impossible humidity values above 130 percent, a 22-hour cold snap at one station that saw readings plummet below freezing in April, and weeks of instrument malfunction at a station logging temperatures below minus 48 degrees Celsius. The method even uncovered a two-week inversion of the diurnal pattern at one station, where maximum temperatures were being logged at night and minimums during the day, a classic symptom of timestamp error.

The researchers are candid about the method’s limitations. Robust detection tends to produce somewhat higher rates of false positives, so a slightly higher deviation threshold than the conventional value of three is advisable; in retrospect, a threshold of five would have flagged only 0.6 percent of the case-study data while still catching all confirmed errors, pushing efficiency above 70 percent. The technique also requires a sufficiently large spatial set of strongly correlated series, ideally with typical correlations somewhat above 0.8, and it assumes the data are roughly symmetrically distributed, though the authors suggest trimmed statistics as a fallback for skewed data.

What makes the work broadly significant is its generality. The procedure applies to any collection of time series that are strongly correlated through time, whether they measure humidity, solar radiation, or ambient pressure, and the authors suggest potential uses even beyond meteorology. As automated sensor networks proliferate and low-cost devices are deployed in ever greater density, the volume of error-prone environmental data will only grow. A single, simple procedure that can simultaneously expose spikes, level shifts, and warped daily cycles, without assuming any series is clean, offers researchers a practical safeguard for the datasets on which much of modern environmental science depends.

Subject of Research: A data-standardization procedure for detecting multiple types of outliers in correlated high-frequency meteorological sensor data

Article Title: A new data-standardization procedure for comprehensive outlier detection in correlated meteorological sensor data

Article References: Benschop, N. D., Zewotir, T., Naidoo, R. N., & North, D. (2025). A new data-standardization procedure for comprehensive outlier detection in correlated meteorological sensor data. Advances in Statistical Climatology, Meteorology and Oceanography, 11(2), 133-158. https://doi.org/10.5194/ascmo-11-133-2025

Image Credits: AI Generated

DOI: 10.5194/ascmo-11-133-2025

Keywords: outlier detection, meteorological sensors, data standardization, time series, robust statistics, median absolute deviation, z-scores, air temperature, Monte Carlo simulation, South Africa, data quality, automatic weather stations