How Few Ensemble Members Does a Weather-Style Forecast Filter Really Need? Chaos Sets the Limit

Every day, weather centers around the world run not one forecast but dozens of them in parallel, each nudged slightly differently, to estimate how uncertain the future atmosphere might be. The mathematical engine behind this strategy, the ensemble Kalman filter, has an awkward open secret: nobody knows exactly how many ensemble members are truly required for it to work well. Run too few, and the filter can catastrophically diverge from reality; run too many, and the computational bill becomes crushing. A new study by Kota Takeda of Nagoya University and the RIKEN Center for Computational Science and Takemasa Miyoshi of RIKEN, published in Nonlinear Processes in Geophysics, now offers a strikingly clean answer rooted in the geometry of chaos itself: the minimum ensemble size is set by the number of unstable directions in the dynamics, plus one.

The question is far from academic. Ensemble members are expensive. In operational geophysical data assimilation, each member requires integrating a full numerical model of the atmosphere or ocean forward in time, so the ensemble size directly multiplies the cost of every forecast cycle. Theoretical results had suggested that guaranteeing long-term accuracy might require an ensemble larger than the entire state dimension of the system, a condition laughably out of reach for models with millions of variables. Takeda and Miyoshi set out to close the gap between such idealized mathematical requirements and what is actually feasible on supercomputers, and in doing so they uncovered a rule that connects filter performance to one of the most fundamental quantities in chaos theory.

The key conceptual move in the study is a redefinition of what it means for a filter to be accurate. Standard practice judges an ensemble Kalman filter by its time-averaged root-mean-square error at a fixed level of observation noise. The authors instead adopt a stronger, mathematically sharper criterion they call noise-scaled, or r-asymptotic, filter accuracy. Imagine making the observations ever more precise, shrinking the noise level r toward zero. A filter is deemed accurate in this asymptotic sense if its long-term squared analysis error shrinks in proportion to r squared, meaning the error stays on the order of the observation noise itself. This formulation, borrowed from rigorous mathematical analyses of filtering algorithms, does more than repackage the usual error metric. By Jensen’s inequality, it provides a stronger guarantee than the conventional RMSE criterion, and crucially, it treats the noise level as an asymptotic parameter, allowing a qualitative classification of filter behavior: either the error tracks the noise downward, or it does not, and there is no ambiguous middle ground.

With this criterion in hand, the authors connect the minimum ensemble size to the instability of the underlying dynamical system, quantified through Lyapunov exponents. These exponents measure the asymptotic exponential rates at which infinitesimally small perturbations to a trajectory grow or decay. A chaotic system has at least one positive exponent, meaning tiny errors balloon exponentially, which is precisely why long-range weather prediction is fundamentally limited. The number of positive exponents, denoted N+, counts the independent unstable directions along which perturbations grow. For autonomous continuous-time systems such as atmospheric models, at least one exponent is always zero, corresponding to perturbations along the flow itself, and the remaining directions are stable, with decaying perturbations. The tangent space at any point on the system’s attractor thus splits into unstable, neutral, and stable subspaces, and the authors argue that spanning the unstable subspace is the non-negotiable requirement for long-term filter accuracy.

The reasoning is intuitive once stated. The ensemble Kalman filter works by representing forecast uncertainty through the spread of its members, and it corrects the state estimate most strongly in the directions where that uncertainty is largest. If the ensemble is too small to represent uncertainty in every unstable direction, errors along the unrepresented directions escape correction and grow exponentially until they reach the size of the entire attractor, a failure mode known as filter divergence. Covariance inflation, a standard remedy that artificially enlarges the ensemble spread, can compensate for modest underestimation of uncertainty, but it cannot conjure directions that the ensemble simply does not span. Localization, another common technique that damps spurious long-range correlations, was deliberately excluded from the study so that the pure relationship between ensemble size and dynamical instability could be isolated.

Earlier theoretical work pointed toward the conjecture the authors set out to test. González-Tokman and Hunt proved in 2013 that for discrete-time hyperbolic systems, an ensemble of at least N+ plus one members bounds the analysis error by the order of the observation noise, but under idealized assumptions, including an initial ensemble concentrated on the unstable subspace, that practitioners cannot verify. Separately, Bocquet and colleagues showed rigorously for linear systems that the Kalman filter’s error covariance collapses onto the unstable-neutral subspace, requiring at least N0 plus one members, where N0 counts both positive and zero exponents. Takeda and Miyoshi sharpened the prediction: in the joint limit of long time and small noise, the influence of neutral directions should become negligible, so the true minimum should be m* equals N+ plus one, one member fewer than the N0-based bound.

To test this, the authors turned to the Lorenz 96 model, a forty-variable chaotic system on a periodic domain that has long served as the standard proving ground for data assimilation methods. By tuning the external forcing parameter, they generated systems with different degrees of instability. With forcing set to 8, Lyapunov analysis using a QR-decomposition-based algorithm revealed thirteen positive exponents and a largest exponent of about 1.67, predicting a minimum ensemble size of fourteen. The numerical experiments with the ensemble transform Kalman filter, a deterministic variant of the EnKF, confirmed the prediction with remarkable crispness. When the ensemble size was fourteen or larger, the squared error scaled with r squared as the noise level shrank, the hallmark of filter accuracy. With thirteen members, the error stubbornly remained of order one regardless of how precise the observations became. Repeating the experiment with forcing set to 16, which yields fifteen positive exponents and a largest exponent of about 3.82, again confirmed the threshold, this time at sixteen members.

The study also delivers a practical innovation: an ensemble spin-up and downsizing method designed to realize the idealized initial conditions of the theory. The procedure begins with a large ensemble, runs the filter through a spin-up period, and then uses singular value decomposition to compress the ensemble perturbations down to the target smaller size, retaining the leading modes. The idea is that after spin-up the ensemble mean sits close to the true state and its perturbations have naturally aligned with the unstable subspace, so the trimmed ensemble inherits that alignment. The experiments showed this matters enormously. Even when the initial ensemble had an accurate mean but perturbations pointing in the wrong directions, the spin-up and downsizing route sustained filter accuracy across a range of inflation factors, while skipping the spin-up led to immediate divergence or painfully slow error decay. In the borderline case where the ensemble size exactly equals the predicted minimum, downsizing dramatically accelerated the convergence of the error to the noise level.

The implications reach beyond the toy model. If the rule m* equals N+ plus one carries over to realistic geophysical systems, forecast centers could estimate the required ensemble size a priori from the Lyapunov spectrum of their model, rather than discovering the threshold by expensive trial and error. The authors note caveats: the estimate was verified only for systems with a single zero Lyapunov exponent, and systems with multiple zero exponents, such as coupled ocean-atmosphere models, remain an open test case. They also recommend pairing the method with adaptive inflation schemes to avoid manual tuning, and they flag the analysis of localization, which would require defining a local notion of instability, as future work. Still, the result stands as a rare instance where an abstract mathematical criterion, the scaling of error with vanishing noise, translates into a concrete, testable, and potentially cost-saving design rule for the algorithms that keep weather forecasts honest.

Subject of Research: Minimum ensemble size for accurate ensemble Kalman filtering of chaotic dynamical systems, linked to the number of positive Lyapunov exponents

Article Title: Noise-scaled accuracy of the ensemble Kalman filter with an instability-based minimum ensemble size

Article References: Takeda, K., & Miyoshi, T. (2026). Noise-scaled accuracy of the ensemble Kalman filter with an instability-based minimum ensemble size. Nonlinear Processes in Geophysics, 33(3), 335-346. https://doi.org/10.5194/npg-33-335-2026

Image Credits: AI Generated

DOI: 10.5194/npg-33-335-2026

Keywords: ensemble Kalman filter, data assimilation, Lyapunov exponents, chaos, Lorenz 96 model, filter divergence, ensemble size, unstable subspace, covariance inflation, geophysical forecasting, Nonlinear Processes in Geophysics, Noise-scaled