AI Learns to Spot Clouds in Lidar Data Without Site-Specific Retraining

Researchers in China have developed an artificial intelligence framework that identifies cloud layers in multi-wavelength Raman lidar observations and, crucially, transfers to lidar stations it has never seen before without any site-specific retuning. The method, developed for the China Aerosol Raman Lidar Network (CARLNET) operated by the China Meteorological Administration, is described in a study published in Atmospheric Measurement Techniques, and its developers plan to deploy it for automated cloud and aerosol processing across the network.

Cloud-layer identification is a prerequisite step in nearly every automated ground-based lidar processing chain. Cloud masks and cloud boundaries constrain quality control, reference-layer selection, absolute calibration, attenuation correction, and the optical retrievals that follow. When low clouds are missed in the near-overlap region, the valid retrieval range may be misinterpreted; when thin high clouds or subtle multilayer structures go undetected, the reference region used for calibration can be contaminated, introducing systematic biases into backscatter and extinction products. Conversely, aerosol layers or noise fluctuations misclassified as clouds can unnecessarily interrupt retrievals and reduce data availability.

These tasks become markedly harder across a network. CARLNET included 49 deployed multi-wavelength Raman-polarization lidar systems during the study period, of which 47 sites met the data-quality and completeness requirements, spanning arid inland, monsoon-influenced, high-elevation, and humid coastal regions of China. Instrument configuration, laser energy, receiver response, maintenance status, background light, and aerosol environment all vary from site to site, altering signal-to-noise ratio and the apparent contrast between clouds, aerosols, and clear air. Methods tuned at one site may simply fail at another.

The new framework, developed by Weijie Zou, Zhenping Yin, and colleagues at Wuhan University and partner institutions, operates directly on observation-level inputs: parallel-polarized elastic signals at 355, 532, and 1064 nanometers, together with volume depolarization ratios at 355 and 532 nm. Retrieved optical products such as backscatter and extinction coefficients are deliberately excluded from the model inputs, because those quantities themselves depend on calibration assumptions and reference-layer selection. Using them as inputs would couple cloud detection to the very inversion steps it is meant to support.

Architecturally, the model is an anisotropic encoder-decoder built on a U-Net design. The authors emphasize that lidar time-height observations are physically asymmetric: the vertical axis reflects true atmospheric structure, while the time axis reflects the evolution of weather processes. The network therefore applies vertical downsampling while never pooling along the temporal dimension, and it uses direction-specific convolutions—3×1 kernels for temporal continuity and 1×3 or 1×5 kernels for vertical structure. Each input sample consists of 10 consecutive 5-minute profiles, a 50-minute window, and the vertical dimension is resampled from 2048 range gates at 15-meter resolution to 1056 bins, retaining fine sampling below roughly 12 kilometers where cloud boundaries and aerosol-cloud transitions are most frequent.

Training proceeds in three stages. First, self-supervised reconstruction pretraining teaches the model spatiotemporally coherent structures and cross-channel consistency from unlabeled 2024 observations by masking channels and time frames and asking the network to reconstruct them. Second, a distillation stage injects physics-informed guidance from the conventional Value Distribution Equalization (VDE) algorithm, a rule-based cloud-aerosol discrimination method that applies a histogram-equalization-like transformation to elastic echoes. Discretized VDE-derived cloud probabilities, averaged across the three wavelengths, serve as soft supervisory targets rather than absolute truth, with additional penalty terms suppressing aerosol-induced false positives and non-physical detections in the near-range blind zone below roughly 500 meters where overlap is incomplete. Third, the model is fine-tuned on a limited set of expert-refined cloud masks, annotated by multiple lidar experts whose pixel-wise agreement on a 15 site-day subset reached an IoU of 95.8 percent.

Evaluation was deliberately strict. Three environmentally contrasting sites—Aksu (51628) in a temperate continental arid region, Yan’an (53845) in a temperate monsoon region, and Fuzhou (58847) in a subtropical humid coastal region—were held out from both pretraining and fine-tuning. Testing used continuous January 2025 observations, about 88 site-days, temporally independent of the 2024 training data. Under this setting, with no test-site observations used at any training stage, the framework achieved an F1 score of 0.937, compared with 0.928 for direct supervised training from scratch—a 12.5 percent relative reduction in inconsistency. The authors note this difference should be read as a modest deterministic gain under a fixed data split, not formal evidence of statistical significance.

The advantages become clearer in geometric products. Converting masks to cloud-base and cloud-top heights, the pretrained model achieved a cloud-base mean absolute error of 59.1 meters, comparable to 55.3 meters for direct supervised training and far better than the VDE baseline’s 123.6 meters. For detectable cloud tops, the pretrained model reduced the error to 82.2 meters—about 12.6 percent better than direct supervision (94.0 meters) and roughly 67 percent better than VDE (250.7 meters). Detection rates of the two deep learning models were nearly identical, 93.9 versus 94.1 percent, indicating the benefit lies in more accurate boundary delineation and multilayer preservation rather than simply detecting more clouds. The authors caution that reported cloud tops refer to the highest boundary supported by a detectable lidar return; when optically thick clouds fully attenuate the signal, the physical top cannot be recovered.

An independent check against radiosondes added a second observing system. Radiosondes launched daily at 00:00 and 12:00 UTC from stations within 100 meters of the lidar sites provided humidity profiles, from which moist layers were identified using altitude-dependent relative-humidity thresholds following Zhang et al. (2016). Across 47 valid lidar-radiosonde comparison cases aggregated over the three test sites, the proposed model showed a structural consistency rate of 38.3 percent with radiosonde-derived moist layers, versus 34.0 percent for VDE and 27.7 percent for expert annotation. Because balloons drift up to roughly 30 kilometers during ascent and humidity thresholds carry their own uncertainties, the authors stress these figures are consistency indicators between two observing systems, not absolute accuracy measurements.

Several limitations remain. The framework does not explicitly separate precipitation from cloud layers, and its inputs remain affected by incomplete near-range overlap, daytime background radiation, signal saturation, and severe attenuation above optically thick low clouds. Its behavior under highly irregular sampling and prolonged data gaps requires further evaluation, and broader tests on external lidar networks are needed to establish cross-instrument generalizability. The results also stem from a single training run under a fixed configuration. Even so, the study demonstrates that a cloud-layer identification technique can learn transferable structural representations from heterogeneous network data, reduce dependence on both absolute calibration and dense manual annotation, and hold up at unseen sites—an encouraging step toward reliable, network-scale automated lidar processing.

Subject of Research: A machine learning framework for cloud-layer identification in multi-channel Raman lidar networks, validated across held-out sites of the China Aerosol Raman Lidar Network.

Article Title: A self-supervised and distillation-guided framework for robust cloud-layer identification in multi-channel Raman lidar networks

Article References: Zou, W., Yin, Z., Sun, T., Bu, Z., Dai, Y., Wang, H., Wang, L., He, Y., Li, S., Chen, J., Wang, X., Li, S., & Müller, D. (2026). A self-supervised and distillation-guided framework for robust cloud-layer identification in multi-channel Raman lidar networks. Atmospheric Measurement Techniques, 19(20), 6757-6779. https://doi.org/10.5194/amt-19-6757-2026

Image Credits: AI Generated

DOI: 10.5194/amt-19-6757-2026

Keywords: Raman lidar, cloud detection, CARLNET, self-supervised learning, knowledge distillation, deep learning, atmospheric remote sensing, radiosonde validation, cloud boundaries, U-Net, lidar networks, China Meteorological Administration