AI Weather Forecasting Gets a Ruthless Audit: New Survey Exposes Hidden Flaws in How We Judge Machine-Learning Models

Weather forecasting has quietly become one of the most visible success stories of modern artificial intelligence. In just a few years, machine-learning models have gone from experimental curiosities to systems that can rival, and in some metrics outperform, the world’s best numerical weather prediction models run on supercomputers. But according to a comprehensive new survey published in the journal Artificial Intelligence Review, the fast-moving field of AI-driven weather and climate science has a measurement problem — and the way the community currently trains and evaluates its models may be systematically rewarding the wrong kind of forecasts. The paper, authored by Andreas Holzinger of BOKU University and Graz University of Technology, together with Sandro Fiore of the University of Trento, Tullio Degiacomi of Hypermeteo, Fabrizio Antonio of the CMCC Foundation and Heimo Müller of Medical University Graz, offers both a panoramic map of the field and a pointed critique of its evaluation culture.

The survey’s central argument is that weather and climate represent an unusually demanding, and unusually revealing, testbed for artificial intelligence. Unlike image recognition or language modelling, atmospheric science confronts machine-learning systems with petabyte-scale spatiotemporal data spread across a rotating sphere, governed by partial differential equations and constrained by global observational and reanalysis archives. On top of that sits a challenge that most mainstream AI benchmarks simply do not have: climate non-stationarity. The statistical properties of the atmosphere are themselves shifting as the planet warms, which means models trained on the past may be silently invalidated by the very future they are asked to predict. The authors argue that this combination of scale, physics and drift makes meteorology a proving ground whose lessons generalize far beyond forecasting.

To organize an enormous and sometimes chaotic literature, the survey classifies the major AI architectures by their physical inductive biases — the built-in assumptions each design makes about the structure of the world. Convolutional neural networks, the workhorses of early deep-learning weather prediction, assume local spatial structure and translation invariance. Graph neural networks treat the atmosphere as an irregular mesh, naturally handling the geometry of the sphere and unstructured computational grids. Transformers bring global attention mechanisms that can capture long-range teleconnections such as the El Niño–Southern Oscillation and the Madden–Julian oscillation, which link weather patterns across entire hemispheres. Generative models, including generative adversarial networks and diffusion-based approaches, address a different problem entirely: producing realistic ensembles and downscaling coarse global fields to fine local detail.

Two unifying mathematical themes run through this taxonomy. The first is geometric deep learning, the program of designing networks whose internal operations respect the symmetries of the underlying space — in this case, rotation and translation on a sphere rather than on a flat plane. The second is operator learning, exemplified by neural operators such as the Fourier neural operator and its spherical variant, which learn mappings between entire function spaces rather than between individual data points. This distinction matters because weather and climate models are fundamentally functions of functions: they map one continuous field of temperature, pressure and wind onto another. Architectures that respect the spherical geometry of the planet and learn operators rather than fixed-resolution maps are, the authors argue, better positioned to generalize across resolutions and physical regimes.

Beyond architecture, the survey gives systematic treatment to three areas it considers underappreciated. Representation learning and foundation models — large networks pre-trained on vast atmospheric archives and then adapted to many downstream tasks — are examined as the emerging backbone of the field. Uncertainty quantification receives extensive attention, covering techniques from Bayesian approaches to ensemble generation, all aimed at answering the question operational forecasters care most about: not just what will happen, but how confident we should be. And causal discovery is framed as a complement to pure prediction, a way of using machine learning not merely to reproduce correlations in reanalysis data but to probe the physical mechanisms connecting them — a distinction that becomes critical when the climate itself is changing.

The paper’s most provocative contribution, however, is its naming and dissection of what the authors call evaluation pathologies in scientific machine learning. The first is the RMSE smoothness bias. Because root mean square error and mean-squared-error training objectives penalize sharp, spatially displaced features more harshly than blurry, averaged ones, models optimized on these metrics are systematically pushed toward smooth, blurred forecasts. A prediction that gets the shape of a storm exactly right but places it a few dozen kilometers off can score worse than a smeared, featureless field that is wrong everywhere but mildly. The practical consequence is that the metrics used to declare AI models superior to numerical weather prediction may be quietly selecting for aesthetically smooth mediocrity while penalizing the crisp, high-impact detail that matters most to forecasters and the public.

The second pathology the authors identify is ERA5 training–evaluation circularity. ERA5, the European Centre for Medium-Range Weather Forecasts’ flagship reanalysis, is the de facto training ground for most AI weather models — but it is also the reference against which those models are scored. A model trained to reproduce ERA5 and then evaluated against ERA5 is, in a meaningful sense, being graded on its own homework. The circularity inflates apparent skill, obscures the reanalysis’s own biases, and makes it difficult to know how models would perform against genuinely independent observations. The third pathology, benchmark overfitting, compounds the problem: as the community iterates on a small set of standard test cases, models increasingly specialize to those cases, and leaderboard gains stop translating into real-world forecasting skill.

The survey does not stop at diagnosis. It frames the field’s open problems as scientific machine learning challenges that extend well beyond meteorology. Distribution shift under a non-stationary climate is the paradigm case: any AI system deployed over years must cope with input statistics that drift, potentially violating the stationarity assumptions baked into training. Physical consistency of learned operators — whether a neural network’s predictions obey conservation laws and dynamical constraints even far from its training distribution — remains unsolved. Sample efficiency in data-sparse regimes, such as the ocean interior, polar regions and the developing world’s observation networks, tests whether foundation-model approaches can transfer knowledge to places with few measurements. And the authors argue for intrinsic interpretability: not post-hoc explanations bolted onto a black box, but models whose internal reasoning is transparent enough for scientists to trust and interrogate, a theme connected to the explainable AI research program the work was partly funded to advance.

Why does this matter now? Because the operational stakes are rising fast. Deep-learning weather prediction systems are already being trialed by major forecasting centers, and skill on benchmarks is being cited as evidence they can replace or supplement physics-based simulation. If the benchmarks reward blur and circularity, the field risks institutionalizing models that look excellent on paper while underperforming on the rare, extreme events — hurricanes, heat waves, flash floods — where forecasts save lives. The survey’s argument is that verification against rare extremes, using metrics such as the fractions skill score and the continuous ranked probability score alongside traditional correlation measures, must become central rather than peripheral to how AI forecasters are judged.

The broader lesson, the authors contend, is that weather and climate offer scientific machine learning a uniquely honest mirror. The domain combines massive data, hard physics, distribution drift and unforgiving operational verification — a combination that strips away the comfortable assumptions of mainstream AI benchmarking. The survey, published open access with funding support from the Austrian Science Fund and the European Union’s Horizon Europe RI-SCALE project, is intended as both a map and a challenge: a structured account of where AI methods for the atmosphere stand today, and a warning that the path forward runs through better evaluation, not just bigger models. If the field heeds it, the same rigor that makes forecasting trustworthy in a changing climate could reshape how machine learning is validated across the sciences.

Subject of Research: Artificial intelligence methods, benchmarking, and scientific machine learning challenges for weather and climate prediction

Article Title: Artificial intelligence for weather and climate: a survey of methods, benchmarks, and scientific machine learning challenges

Article References: Holzinger, A., Fiore, S., Degiacomi, T., Antonio, F., & Müller, H. (2026). Artificial intelligence for weather and climate: a survey of methods, benchmarks, and scientific machine learning challenges. Artificial Intelligence Review. https://doi.org/10.1007/s10462-026-11690-8

Image Credits: AI Generated

DOI: 10.1007/s10462-026-11690-8

Keywords: artificial intelligence, weather forecasting, climate modeling, machine learning, neural operators, geometric deep learning, uncertainty quantification, benchmark design, distribution shift, spatiotemporal learning, foundation models, scientific machine learning