Operational estimates of surface solar irradiance from the Copernicus Atmosphere Monitoring Service radiation service exhibit systematic, scene-dependent prediction errors that are driven primarily by cloud properties, according to research published in Atmospheric Measurement Techniques. Atmospheric scientists Antti Lipponen, Jorge Lezaca, Yves-Marie Saint-Drenan, Marion Schroedter-Homscheidt, and Antti Arola applied explainable machine-learning architectures to diagnose and correct deviations between satellite-derived model outputs and ground-based radiometer measurements across Europe and Africa. By combining extreme gradient boosting residual models with cooperative game theory attributions, the investigation determined that cloud optical depth is the dominant variable associated with discrepancies in global, diffuse, and direct beam solar radiation data products.
Accurate quantification of surface solar irradiance is critical for atmospheric energy budget assessments, climate monitoring networks, and the planning and operation of commercial solar power utilities. While surface radiometers provide high-precision benchmark observations, their sparse geographical distribution necessitates reliance on continuous satellite-derived products such as the Copernicus Atmosphere Monitoring Service radiation service, designated as CRS. The operational CRS pipeline combines geostationary observations from the Meteosat Second Generation Spinning Enhanced Visible and Infrared Imager using the APOLLO-NG cloud retrieval algorithm alongside atmospheric composition forecasts and radiative transfer parameterizations. Despite its widespread deployment, the system exhibits localized errors arising from idealized plane-parallel cloud assumptions, unresolved sub-grid geometry, aerosol optical depth approximations, and discrepancies between point-scale surface stations and satellite pixel footprints.
Rather than substituting the underlying physical radiative transfer scheme with a fully learned direct prediction network, the research team constructed a residual-learning framework. The authors trained extreme gradient boosting models, implemented via XGBoost, exclusively on nineteen publicly available operational CRS input variables to predict the mathematical difference between collocated ground observations and operational model estimates. The study separated its workflow into two distinct model families: diagnostic residual models designed to explore internal error structures within collocated training distributions, and predictive correction models independently tuned and tested to establish post-processing bias reduction across unseen dates and separate geographic coordinates.
The experimental dataset integrated ground-based radiometer records collected throughout 2021 from twenty-one high-quality measurement stations within the field of view of the Meteosat Second Generation satellite. These sites were drawn from three primary monitoring networks: the Baseline Surface Radiation Network, the enerMENA network in Northern Africa, and the Southern African Universities Radiometric Network. Ground observations of global horizontal irradiance, diffuse horizontal irradiance, and beam normal irradiance were quality controlled and averaged over centered fifteen-minute intervals surrounding exact satellite pixel scan times to mitigate high-frequency cloud passage noise. To ensure radiometric measurement integrity, analyses were restricted to solar zenith angles under eighty degrees, where pyranometer cosine response uncertainties remain constrained.
To interpret the diagnostic residual models, the researchers implemented TreeSHAP, an algorithm computing Shapley Additive exPlanations based on cooperative game theory. In this diagnostic implementation, ten XGBoost models with deep tree limits were fitted across one hundred thousand sampled data rows per irradiance component, and the resulting attributions were pooled. The SHAP values decomposed instantaneous residual predictions into additive marginal contributions from individual inputs, establishing whether specific atmospheric parameters systematically induced model underestimation or overestimation across global, diffuse, and direct irradiance streams.
Across all three solar radiation components, cloud optical depth emerged as the primary feature driving residual structure within the diagnostic models, yielding a mean absolute SHAP attribution of 24.5 watts per square meter for global horizontal irradiance. Cloud optical depth was also the only evaluated variable exhibiting a strictly consistent, monotonic relationship with the operational residuals. In both global and diffuse horizontal irradiance regimes, elevated cloud optical depths systematically generated positive SHAP attributions, indicating that operational radiative transfer simulations tend to underestimate surface irradiance under optically thick clouds, whereas low optical depths corresponded to operational overestimation. Secondary cloud parameters, including cloud fractional cover and cloud type classifications, demonstrated substantial overall importance but lacked monotonic trends, exhibiting dispersed impacts across the entire residual spectrum.
Atmospheric aerosol constituents and surface reflectance variables acted as secondary drivers of operational discrepancies, displaying more complex, component-dependent behaviors. Inorganic aerosol optical depths for ammonium and nitrate, which were recently incorporated into the operational CAMS service, exhibited high diagnostic importance rankings, particularly for diffuse and beam normal components. However, because ammonium and nitrate optical depths share a high mutual correlation coefficient of 0.95 in the input dataset, the authors emphasized that their identical ranking represents an exploratory association rather than proof of erroneous optical specifications.
The diagnostic framework also identified systematic dependencies between model residuals and solar illumination geometry. Group-wise aggregations of absolute SHAP values binned across solar zenith angles revealed that cloud parameter attributions expand sharply at angles exceeding seventy degrees, coinciding with long atmospheric slant paths where direct beam attenuation intensifies. While aerosols exhibited elevated relative normalized contributions under small solar zenith angles at low-latitude sites, cloud representation remained the prevailing factor across most solar elevations. Single-case waterfall analyses further illustrated this operational contrast: an overcast case from Palaiseau, France, showed residuals overwhelmingly governed by cloud optical depth underestimation, whereas a cloud-free, aerosol-loaded scene from Cabauw, the Netherlands, exhibited residual decompositions dominated by contrasting positive and negative species-specific aerosol optical depth components.
In the predictive post-processing phase, independently optimized XGBoost correction models demonstrated substantial skill when evaluated on untouched station-day test partitions comprising fifteen percent of the annual data. For global horizontal irradiance, the machine-learning correction reduced root mean squared error from 106.2 to 87.5 watts per square meter, representing an aggregate improvement of 17.6 percent, while increasing the coefficient of determination from 0.866 to 0.909. Diffuse horizontal irradiance root mean squared error decreased by 20.5 percent, dropping from 76.9 to 61.1 watts per square meter, while beam normal irradiance error dropped by 23.9 percent, from 204.9 to 155.9 watts per square meter. These reductions were observed consistently across all twenty-one individual ground monitoring stations, outperforming a simple station-mean residual baseline that yielded error reductions of only 0.8 to 2.5 percent.
Spatial transferability of the post-processing models was assessed through a rigorous leave-one-station-out cross-validation experiment, in which models were iteratively trained on twenty stations and tested against an entirely omitted location. In this spatial test, pooled root mean squared error reductions were notably more modest, recording gains of 1.7 percent for global horizontal irradiance, 8.4 percent for diffuse horizontal irradiance, and 0.5 percent for beam normal irradiance, accompanied by increases in mean absolute error for global and direct beam components. Stations situated in coastal, island, or mountainous terrains, such as Izaña, experienced localized performance deteriorations linked to acute point-to-pixel representativeness mismatches. Sensitivity trials revealed that omitting slowly varying surface bidirectional reflectance distribution function parameters markedly enhanced spatial transferability, boosting pooled leave-one-station-out improvements to 9.2 percent for global and 10.9 percent for direct irradiance.
The study authors concluded that while machine-learning post-processing can effectively mitigate systematic biases on unseen days at known monitoring sites, current models require expanded geographical coverage and multi-year calibration before operational deployment. More fundamentally, the explainable machine-learning framework successfully isolates the specific input variables and retrieval modules that contribute most heavily to operational solar radiation errors. By confirming that cloud optical depth retrieval uncertainties under high solar zenith angles dominate the error budget, the findings provide a transparent, physics-guided roadmap for refining satellite cloud algorithms and radiative transfer parameterizations across future Copernicus Atmosphere Monitoring Service iterations.
Subject of Research: Solar irradiance model error attribution and correction using explainable machine learning
Article Title: Machine-learning-based approach for solar radiation model prediction error identification, attribution and correction
Article References: Lipponen, A., Lezaca, J., Saint-Drenan, Y.-M., Schroedter-Homscheidt, M., & Arola, A. (2026). Machine-learning-based approach for solar radiation model prediction error identification, attribution and correction. Atmospheric Measurement Techniques, 19(22), 7627-7647. https://doi.org/10.5194/amt-19-7627-2026
Image Credits: AI Generated
Keywords: solar irradiance, Copernicus Atmosphere Monitoring Service, CAMS Radiation Service, extreme gradient boosting, SHAP analysis, cloud optical depth, global horizontal irradiance, diffuse horizontal irradiance, beam normal irradiance, radiative transfer, aerosol optical depth, surface albedo, solar zenith angle, TreeSHAP, atmospheric monitoring, error attribution
Cite Scienmag News
Russell Cooper. (November 19, 2026). Machine Learning Pinpoints Cloud Optical Depth as Main Operational Solar Irradiance Error. Scienmag. https://scienmag.com/machine-learning-pinpoints-cloud-optical-depth-as-main-operational-solar-irradiance-error/
Russell Cooper. "Machine Learning Pinpoints Cloud Optical Depth as Main Operational Solar Irradiance Error." Scienmag, 19 November 2026, https://scienmag.com/machine-learning-pinpoints-cloud-optical-depth-as-main-operational-solar-irradiance-error/. Accessed 19 November 2026.
Russell Cooper. "Machine Learning Pinpoints Cloud Optical Depth as Main Operational Solar Irradiance Error." Scienmag. November 19, 2026. https://scienmag.com/machine-learning-pinpoints-cloud-optical-depth-as-main-operational-solar-irradiance-error/








