New Method Filters Highway Sensor Data in Real Time

Published in the journal Data Science and Engineering, the study addresses a critical challenge in pavement engineering: the massive volume of data generated by high-frequency sensors. By filtering out meaningless data segments, the proposed method saves approximately 91.63% of storage space while achieving an average recall value of 0.91 for dynamic response identification.

Highway structural health monitoring systems rely on strain sensors embedded in pavement to track long-term performance for maintenance decision-making. To capture transient information from high-speed vehicle loads, these systems often operate at sampling frequencies up to 2000 Hz. This high frequency results in data sizes reaching 5 to 6 terabytes per month, creating a heavy burden on data storage infrastructure. The core problem identified by the authors is that existing dynamic response identification approaches typically guarantee either reduced data volume or high accuracy, but rarely both simultaneously, and often lack real-time processing capabilities.

The study, led by Zhixin Qi and colleagues, proposes a method that selects effective features from monitoring data and designs a training data generation strategy for machine learning models. The framework is designed to filter meaningless data in real time, addressing the cold start problem associated with unlabeled and sparse data. In highway monitoring, the ratio of dynamic responses to total data is typically around 1 to 200, making it difficult to establish well-trained models without specific strategies for handling this imbalance.

To overcome the lack of labels in historical data, the researchers employed a crowdsourced labeling approach. They divided historical structural health monitoring data into equal-length segments and engaged nine domain experts to judge whether each segment contained dynamic responses. Based on these expert judgments, the team constructed a training dataset where the proportion of positive examples, segments containing dynamic responses, was adjusted to a 1:1 ratio with negative examples. This proportion adjustment was crucial for training a high-performance model, as the original 1:200 ratio would have hindered effective learning.

Feature selection was another key component of the methodology. The team evaluated candidate statistical features including mean value, variance, skewness, peakness, maximum value, and minimum value. Using mutual information, a concept that measures the correlation between a feature and its corresponding label, they identified which features were most useful for identification. Experimental results indicated that range and peak to average ratio had the highest mutual information values, leading the authors to select these as the effective features for their model. This data-driven approach ensures that the model focuses on the most relevant statistical characteristics of the sensor data.

The experimental evaluation utilized real structural health monitoring data collected by eight single-point fiber grating sensors from the Jilin Tonghua Highway. The sensors operated at a sampling frequency of 250 Hz, and the data covered a seven-day period. The team compared their proposed framework against three commonly used identification methods: the threshold method, the entropy method, and the energy method. Additionally, they tested six classical machine learning models within their framework, including decision tree, k-nearest neighbors, naive Bayes, support vector machine, logistic regression, and random forest.

Performance metrics focused on the sum of true positives and false positives, which represents the data segments preserved in storage, and recall, which measures the ratio of dynamic responses correctly identified and preserved. The results showed that the three traditional methods identified fewer dynamic responses and exhibited considerable variation across different sensors. In contrast, the machine learning methods within the proposed framework achieved recalls higher than 0.8 in most cases while preserving a reasonable number of data segments. The support vector machine model achieved the highest average recall, but it preserved 29.34% of the data. The k-nearest neighbors model, which the authors recommend for practical use, achieved the second-highest average recall while preserving only 8.37% of the data, striking a better balance between accuracy and storage efficiency.

Efficiency was a primary design goal, and the study demonstrated that the method meets real-time requirements. The total time costs were directly proportional to the number of data segments processed. The traditional methods processed each segment in 0.02 milliseconds, while the machine learning methods in the framework processed each segment in an average of 0.037 milliseconds. Specifically, the k-nearest neighbors method took 0.0357 milliseconds to process a single data segment. This speed allows the system to perform real-time identification of dynamic responses even at a sampling frequency of 1000 Hz, ensuring that the monitoring system can keep up with the incoming data stream without significant lag.

The authors also conducted parameter tuning to optimize the performance of the machine learning models. For the k-nearest neighbors model, they varied the number of neighbors from 1 to 15 and found that a value of 9 provided the highest recall with a relatively small number of preserved segments. They noted an odd-even effect where even numbers of neighbors led to tied votes, forcing boundary samples to be classified as negative. For other models, such as decision trees and random forests, they adjusted parameters like maximum depth and the number of estimators to balance model complexity and stability. These tuning efforts ensured that the recommended configuration was robust and effective for the specific characteristics of the highway monitoring data.

The study highlights the importance of adapting machine learning techniques to the specific constraints of structural health monitoring. By combining crowdsourced labeling, mutual information-based feature selection, and efficient machine learning models, the researchers created a system that is both accurate and resource-efficient. The approach addresses the practical need for real-time processing in large-scale monitoring networks, where storage costs and computational resources are significant concerns. The use of real-world data from the Jilin Tonghua Highway validates the method’s applicability to actual highway infrastructure.

Limitations of the study include the reliance on expert labeling for the initial training data generation, which may introduce subjectivity, although the use of multiple experts helps mitigate this. The feature selection process is data-dependent, meaning that different structural health monitoring datasets may require recomputation of mutual information values to identify the most effective features. The authors acknowledge that the characteristics of highway monitoring data are continuously changing due to environmental and traffic variations. Therefore, they suggest that future work should focus on designing self-adaptive dynamic response identification models that can adjust to these changes without manual retraining.

This research contributes to the broader field of data science in engineering by demonstrating how machine learning can be applied to solve specific industrial problems involving high-frequency sensor data. The significant reduction in storage space, combined with high recall and real-time processing capabilities, makes the proposed method a viable solution for modern highway monitoring systems. As infrastructure monitoring becomes more widespread, the need for efficient data management tools will continue to grow, and this study provides a robust framework for addressing these challenges.

The proposed framework operates by processing incoming structural health monitoring data in discrete segments of 100 data points each. This specific segment length was determined by the physical characteristics of vehicle traffic and sensor capabilities. For a vehicle with a 2.5-meter axle spacing traveling at 100 kilometers per hour, the time interval to pass a single sensor is approximately 0.090 seconds. At a sampling frequency of 1000 Hz, this duration corresponds to roughly 90 data points. By setting the segment length to 100 points, the system ensures that the complete passing information of a vehicle is captured within a single unit for analysis. The authors note that even if a dynamic response event is split across two adjacent segments, the framework is designed to identify both segments accurately, preventing the loss of critical transient data.

The algorithm’s computational efficiency is characterized by a linear time complexity relative to the number of data segments processed. Each individual segment undergoes identification operations that execute in constant time, resulting in an overall time complexity of O(n). The space complexity is determined by the storage of identified positive segments, which is proportional to the number of segments containing dynamic responses. Because the proportion of segments with dynamic responses is significantly less than one, the storage requirement remains low. This mathematical structure supports the method’s ability to handle high-frequency data streams without accumulating excessive computational or storage overhead over time.

The experimental setup utilized a computer equipped with an AMD Ryzen 7 5800HS CPU operating at 3.20 GHz and 16 GB of memory. All algorithms were implemented in Python. To ensure the reliability of the reported efficiency metrics, the researchers tested each result five times and reported the average time costs. This repeated testing protocol helps account for minor variations in processing speed, providing a more stable estimate of the method’s real-time performance. The hardware specifications are relevant to understanding the baseline performance, as the method’s speed is contingent upon the processing power of the underlying system.

The study’s data availability is limited, with the authors stating that the data supporting the findings are partially available from the corresponding author upon reasonable requests. This restriction may limit the immediate reproducibility of the exact experimental results by other researchers. However, the detailed description of the methodology, including the specific feature selection process and parameter tuning steps, provides a clear roadmap for implementation. The reliance on real-world data from the Jilin Tonghua Highway, collected over a seven-day period, grounds the study in practical application scenarios rather than purely synthetic datasets.

The authors identify the instability of machine-learning-based identification models as a key challenge for future development. They propose designing an integrated identification framework that combines different base models to enhance robustness. Additionally, they aim to develop self-adaptive models that can adjust to the continuously changing characteristics of highway structural health monitoring data. These future directions suggest that while the current method is effective for the specific conditions tested, ongoing adaptation will be necessary to maintain performance as environmental and traffic conditions evolve.

Subject of Research: Real-time machine learning method for filtering highway structural health monitoring data

Article Title: Real-Time Dynamic Response Identification for Highway Structural Health Monitoring Data

Article References: Qi, Z., Su, X., Wang, Y., Chao, Z., Dong, Z., & Wang, H. (2026). Real-Time Dynamic Response Identification for Highway Structural Health Monitoring Data. Data Science and Engineering, 11(1), 248-259. https://doi.org/10.1007/s41019-025-00336-4

Image Credits: AI Generated

DOI: 10.1007/s41019-025-00336-4

Keywords: structural health monitoring, highway engineering, machine learning, data storage, real-time processing, feature selection, pavement monitoring, sensor data, Real-Time, Dynamic, Response, Identification