DeepSIP Framework Improves Air Quality Forecasting by Filling Sensor Gaps

Researchers have developed a unified deep learning framework called DeepSIP, designed to address the persistent challenge of missing data in environmental sensor networks. The study, published in Aerosol and Air Quality Research, introduces a method that simultaneously imputes missing sensor readings and forecasts particulate matter concentrations. By integrating these two tasks into a single pipeline, the framework aims to enhance the reliability of air quality monitoring systems that often suffer from data gaps due to hardware failures or communication losses.

The primary motivation for this research stems from the critical role of continuous sensor data in environmental monitoring. In domains such as industrial IoT, healthcare, and air quality assessment, missing observations can introduce bias and reduce the performance of predictive models. Traditional statistical imputation techniques, while computationally efficient, often fail to capture the nonlinear and dynamic characteristics of environmental time series. More advanced methods like K-Nearest Neighbors (KNN) and Multiple Imputation by Chained Equations (MICE) have shown some improvement but still struggle with high-dimensional and heterogeneous sensor data.

To address these limitations, the authors proposed DeepSIP, which employs a two-stage architecture. The first stage utilizes an autoencoder-based imputation module to reconstruct missing values. This module learns latent representations from correlated sensor variables and contextual information, such as time and date. The second stage involves a deep predictive network, specifically a seven-layer dense neural network, trained on the imputed datasets to model complex temporal and cross-sensor dependencies for accurate forecasting.

The study evaluated the framework using seven real-world particulate matter (PM) datasets collected in Amman, Jordan. The data were gathered from four low-cost sensors deployed as mobile platforms and seven fixed air quality monitoring stations operated by the Jordanian Ministry of Environment. The monitoring campaign covered the year 2024, resulting in a continuous dataset of 8,783 hourly observations. The low-cost sensors were calibrated against reference-grade instruments to ensure accuracy, with validation processes including laboratory testing and outdoor co-location.

The dataset presented significant challenges due to high levels of missingness. For instance, one low-cost sensor recorded only 8,297 values, representing approximately 94.47% missing data, while another had about 61.43% missing entries. This extreme sparsity transformed the problem from a straightforward supervised prediction task into a complex unsupervised data imputation problem. The researchers analyzed the missing data patterns and determined that the missingness was largely Missing at Random (MAR), as it showed weak correlation with temporal features.

During the imputation phase, the authors compared four methods: the proposed autoencoder-based approach, KNN, MICE, and Linear Interpolation. The autoencoder method consistently achieved the lowest reconstruction errors, measured by Mean Squared Error (MSE), Mean Absolute Error (MAE), and Root Mean Squared Error (RMSE). In contrast, KNN produced an RMSE of 0.8583, and Linear Interpolation yielded an RMSE of 2.284. MICE demonstrated the poorest performance, with high error metrics suggesting difficulty in modeling the complex nonlinear relationships present in the data.

The autoencoder architecture was designed as a two-stage process to handle sensors with varying levels of missingness. The first stage focused on sensors with high missingness, using auxiliary features to infer missing values based on inter-sensor correlations. The second stage used the imputed high-missing sensors as additional inputs to train a separate autoencoder for low-missing sensors. This tailored strategy minimized reconstruction errors for predominantly complete sensors while effectively handling the more sparse data.

Following imputation, the deep learning model was trained to predict PM2.5 concentrations. The model utilized a lag-based feature engineering approach, incorporating PM2.5 observations from the previous 168 hours to predict future hourly levels. The neural network consisted of an input layer with seven features, followed by hidden layers with 256, 128, 64, 32, 16, and 8 neurons, using ReLU activation functions. The output layer consisted of a single neuron with linear activation for regression. The model was trained using the Adam optimizer with a learning rate of 0.001 over 1,000 epochs.

The results demonstrated that the DeepSIP framework achieved high forecasting accuracy across all seven monitoring stations. The model produced consistently low MAE, MSE, and RMSE values, along with high R-squared scores. These findings indicate that the latent representation learning effectively preserved nonlinear temporal and cross-sensor relationships, leading to improved downstream predictive performance. The study also examined whether imputation methods artificially inflate forecasting performance, a rarely investigated aspect in prior work, and found that high-quality imputation was crucial for reliable predictions.

Validation of the imputation process involved masking randomly selected complete observations and reconstructing them using the autoencoder. The average MSE for imputation ranged from 0.029 to 0.054 across the seven stations, indicating very low imputation errors. Visual analysis of predicted versus actual sensor values showed a strong positive correlation, with points closely aligning with the ideal y=x relationship. This confirmed the model’s ability to accurately reconstruct missing values even for sensors with substantial data gaps.

The study acknowledges several limitations. The framework’s performance may degrade under conditions of extreme data missingness where inter-sensor correlations are weak. Additionally, the current implementation does not explicitly incorporate spatial dependencies or exogenous environmental variables such as traffic intensity or wind speed. The deep neural architecture, while accurate, has limited interpretability, which may constrain its applicability in decision-support settings requiring model transparency.

Future work will focus on incorporating spatial features and local meteorological variables to improve predictive robustness. The authors plan to investigate advanced modeling approaches, including transformer-based architectures and recurrent neural networks, to better capture complex temporal dependencies. Integrating uncertainty quantification into both imputation and prediction stages is also a priority, aiming to provide more transparent assessments of model confidence. These enhancements are expected to improve the practical applicability of the framework for environmental monitoring and public health decision-making.

The DeepSIP framework was specifically engineered to handle the heterogeneity of missing data patterns found in real-world sensor networks, where different devices often experience varying rates of data loss. In the Amman, Jordan study, the four low-cost sensors exhibited distinct levels of sparsity, with missing data rates ranging from approximately 38.57% to 94.47%. This variability necessitated a specialized approach that could distinguish between sensors with minor gaps and those with extensive data voids. The researchers categorized the sensors into two groups based on a 50% missingness threshold. Sensors with missingness rates exceeding 50% were treated as high-missing, while those at or below this threshold were classified as low-missing. This distinction was critical because low-missing sensors require precise reconstruction to preserve subtle variations in the data, whereas high-missing sensors benefit more from correlation-based inference using auxiliary features.

To address this imbalance, the authors implemented a two-stage autoencoder architecture that tailors the reconstruction strategy for each sensor group. In the first stage, the model trains exclusively on auxiliary features and high-missing sensors, leveraging correlations to impute missing values. The second stage then uses the imputed high-missing sensors as additional inputs to train a separate autoencoder for the low-missing sensors. This second model is trained only on rows where all low-missing sensors are fully observed, allowing it to capture subtle variations without interference from highly incomplete data. By splitting the reconstruction into two distinct objectives, the framework avoids the learning bias that occurs when a single autoencoder optimizes a uniform reconstruction loss across sensors with vastly different data completeness levels.

The experimental evaluation was conducted on a system equipped with a 13th Gen Intel Core i7-1360P processor and 16 GB of RAM, utilizing Python libraries such as Scikit-learn and TensorFlow for model implementation. The researchers employed a chronological train-test split strategy, allocating the early 80% of samples to training and the late 20% to validation. This approach ensured temporal consistency and prevented data leakage, guaranteeing that both imputation and forecasting processes relied solely on past data. For the imputation phase, the performance of each method was assessed using Root Mean Squared Error (RMSE), Mean Absolute Error (MAE), and Mean Squared Error (MSE) across the seven location-based datasets. The results were averaged to provide a level playing field for comparison, with the autoencoder-based method consistently demonstrating the lowest error metrics.

Advanced deep learning imputation models such as BRITS and SAITS were not considered for this study because they require sufficiently rich temporal information and end-to-end training on partially observed sequences, conditions that were not satisfied by the highly sparse dataset. Given the approximately 95% sparsity in some sensors, the cluster-based methodology using fully observed data samples was adopted to clearly separate the imputation and prediction phases. The autoencoder was trained for 1,000 epochs with a batch size of 32, using an 80–20 train-validation split. To evaluate performance, approximately 20% of each sensor’s values were artificially masked, and the Normalized Mean Squared Error (NMSE) was computed for each sensor to ensure fair comparison across sensors with varying scales and variances.

The study also highlights the importance of data preprocessing in the success of the imputation pipeline. Categorical variables such as day of the week, month, and hour were transformed into numerical formats, and all numerical features were scaled using Min-Max scaling to fall within a uniform range. This normalization is crucial for neural and distance-based models like autoencoders, which are sensitive to feature magnitude. Additionally, a missing data mask was created using binary flags to indicate the presence or absence of each sensor reading, facilitating further analysis of the missing data patterns. The researchers used t-tests to compare average PM values between groups where sensor readings were missing and where they were present, helping to determine whether the missingness was Missing Completely at Random (MCAR) or related to other factors.

While the DeepSIP framework demonstrated strong performance in the Amman study, the authors note that its applicability extends to other sensor-driven domains with incomplete multivariate time series data. The unified pipeline is relevant for smart city infrastructure, precision agriculture, healthcare systems, and industrial Internet of Things applications. However, the current implementation primarily leverages temporal information from individual sensors and does not explicitly incorporate spatial dependencies or exogenous environmental variables. Future enhancements aim to integrate local meteorological features, such as temperature inversions or wind events, which can drive sudden fluctuations in PM2.5 levels. The authors also plan to investigate transformer-based architectures and recurrent neural networks to better capture complex temporal dependencies, alongside integrating uncertainty quantification to provide more transparent assessments of model confidence in both imputation and prediction stages.

Subject of Research: Deep learning framework for imputing missing sensor data and forecasting particulate matter concentrations in air quality monitoring.

Article Title: DeepSIP: A Deep Learning Approach for Sensor Data Imputation and Time Series Forecasting

Article References: Surakhi, O., Hossein Motlagh, N., Iturrate-Garcia, M., Abu Tabbaneh, A., Saleh, S., Almashni, A., AL-Refaie, M., LovriĆ, M., Huang, K., Zhao, Y., Zhao, T., Wang, L., Matsumi, Y., & Hussein, T. (2026). DeepSIP: A Deep Learning Approach for Sensor Data Imputation and Time Series Forecasting. Aerosol and Air Quality Research, 26(8), Article 62. https://doi.org/10.1007/s44408-026-00145-y

Image Credits: AI Generated

DOI: 10.1007/s44408-026-00145-y

Keywords: Deep Learning, Air Quality, Data Imputation, Time Series Forecasting, Particulate Matter, Sensor Networks, Environmental Monitoring, Autoencoder, Missing Data, Machine Learning, DeepSIP, Deep