Extended Kalman filtering on Stiefel manifolds

Every Kalman filter ever built rests on a quiet assumption: that the world it observes is flat. The classical filter, invented by Rudolf Kálmán in 1960, blends a prediction with a measurement by averaging them in an ordinary vector space, where straight lines, distances and variances behave exactly as linear algebra promises. But many modern sensors do not deliver numbers in a vector space at all. They deliver directions, orientations, antenna configurations — points that live on curved surfaces where the very notion of an average must be redefined. A team of mathematicians at Uppsala University, Umeå University and Aalto University has now built a generalization of the extended Kalman filter that works directly on one of the most important of these curved spaces, the Stiefel manifold, and their simulations suggest it can match the accuracy of far more expensive second-order filters without ever computing a second derivative.

The Stiefel manifold is the space of matrices with pairwise orthonormal columns — collections of mutually perpendicular unit vectors. It appears whenever only relative strengths and phases matter, not absolute magnitudes. In MIMO radio systems for cellular networks, for example, a configuration of antennas interferes constructively to produce a directed signal with high energy efficiency. The optimal antenna configuration for each receiver can be represented as a vector, but because the total strength of the currents is irrelevant, only their relative strength and phase, the meaningful object is an orthogonal matrix. These configurations can be computed by retaining the orthogonal part in the polar decomposition of the underlying matrix — a heavily non-linear projection onto the Stiefel manifold, not to be confused with the linear projections of elementary linear algebra.

The problem the researchers tackle is a familiar one dressed in unfamiliar geometry. An object moves around with a drift in some vector space, described by a stochastic differential equation whose solution has a well-defined mean and covariance. But the measurements are taken on a non-linear space: a radioactive dust particle whose incoming radiation direction is measured on a sphere, a quantum particle in a circular accelerator whose position is sensed on a torus, or a spacecraft whose attitude is determined from the directions of a star and Earth’s magnetic field. If the state lives in a vector space but the measurements happen on a manifold, the classical filter’s averaging machinery breaks down. How does one weight a prediction against a measurement when both are points on a curved surface?

The answer, in a nutshell, is to use curves. Manifolds have enough geometrical structure to allow measurement of distances between points, and through the Riemannian exponential map — which takes a starting point and a velocity in the tangent space and traces the constant-speed curve that stays on the manifold — one can connect any two nearby points. The researchers filter the predicted point and the measurement by using a curve that connects them, then obtain the filtered mean by taking a weighted average along that curve. The tangent space at each point contains all possible velocities that keep curves infinitesimally on the manifold, and the logarithm map, the inverse of the exponential map, translates a measurement back into a velocity vector where ordinary linear filtering can operate.

The genuinely novel ingredient is a mapping between scalar true variance on the manifold and the variance of the noise in the ambient space. When a normal random variable in the ambient space is projected onto the manifold, the resulting projected normal distribution has an inherent average and an intrinsic scalar variance that respects the curvature of the manifold. For the 2-sphere, this mapping is known to be injective and has an explicit formula. For general Stiefel manifolds, no explicit formula exists, so the researchers conjecture that an injective mapping always exists — a conjecture they verified numerically for several Stiefel manifolds using Monte Carlo estimation, drawing 100 pseudo-random samples for each variance level and each manifold. The mapping inherently takes into account the curvature of the manifold, corresponding to higher-order derivatives, and gives a virtually cost-free way of folding that curvature into the moment-matching of the covariance.

With this variance translation in hand, the filter itself follows the classical predict-update cycle. The prediction propagates the previous estimate through the noiseless dynamics and inflates the covariance to account for process noise, approximated by the function that maps ambient isotropic variance to intrinsic manifold variance. The update assimilates the new measurement in the tangent space of the predicted mean: the measurement is mapped to a tangent vector via the logarithm map, the Kalman gain is computed from the predicted and measurement variances, and the gain-rescaled tangent vector is retracted back onto the manifold to form the new state estimate. The update step explicitly minimizes the mean squared error of the linearized error in the tangent space, finding the optimal trade-off between prior information and new information consistent with the local Riemannian metric. The whole procedure can be interpreted as a recursive maximum a posteriori estimator acting on the tangent space, where the optimal correction is found where the gradient of the log-posterior vanishes.

The simulations are striking. On the 2-sphere, with a stochastic process driven by an anti-symmetric drift matrix, the researchers compared their intrinsic extended Kalman filter against a naive extrinsic filter and a second-order extended Kalman filter across signal-to-noise ratios from 0 to 10 decibels. At the harshest noise level tested, the raw measurements carried an average error of 0.3495, while the intrinsic filter reduced this to 0.1844 — essentially identical to the second-order filter’s 0.1846. The pattern held across all noise levels and all signal-to-noise ratios: the intrinsic filter matched or beat the second-order filter while the naive extrinsic filter lagged behind. On the space of orthogonal 4-by-2 matrices, the results repeated almost exactly, with the intrinsic filter again tracking the second-order filter’s performance to three decimal places. Depending on the situation, one can expect about a 60 percent filtering gain under optimal conditions.

The researchers also grounded the method in a physically informed application: spacecraft attitude determination. Following classical work on two-vector attitude determination, a spacecraft in orbit measures the directions of a star and Earth’s magnetic field, which together span the two orthogonal directions needed to determine its orientation relative to a fixed coordinate system on Earth. Because the spacecraft orbits Earth roughly every 90 minutes while measurements arrive every 10 seconds, the observed point is continuously rotated by the orbital motion — exactly the one-parameter Lie group generated by the drift in the underlying stochastic dynamics. Running the filter over 100 repetitions of this scenario gave a mean error of 0.0040, matching the second-order filter’s 0.0040 and improving tenfold on the raw measurements’ 0.040.

The payoff is computational. Second-order filters require the costly computation of second-order derivatives, which becomes expensive as dimensions grow, and particle filters — though accurate — suffer from the curse of dimensionality, with the estimation space growing exponentially and requiring a very large number of particles for proper estimation. The intrinsic filter achieves second-order accuracy using only first-order machinery: the exponential and logarithm maps, the polar decomposition, and the variance translation function. The researchers note that for large noise there seems to be little gain over the simpler classical filter, which is expected — with large noise the observations no longer capture the curvature of the underlying manifold and instead resemble observations from the ambient space.

Limitations remain. The filter requires isotropic noise — a restrictive assumption, since on general Stiefel manifolds allowing arbitrary covariance matrices breaks the one-to-one correspondence between ambient and projected variance on which the whole approach rests. The logarithm map is only defined for points within the injectivity radius, and while recent work has established a lower bound for this radius on the Stiefel manifold under the canonical metric, the exact value remains an open question. The update steps are also only accurate as long as the actual process does not drift significantly in the radial direction, though choosing an anti-symmetric drift guarantees this on average. Future directions include adapting the variance function to the true projected distribution rather than the projected normal approximation, and extending the approach to other homogeneous spaces such as Grassmann manifolds or higher-dimensional tori, which appear as models for collections of angles in protein chains and robotic motion. For now, the message is clear: the sixty-year-old Kalman filter has learned to navigate curvature, and it did so without paying the second-derivative toll.

Subject of Research: Mathematics and Signal Processing

Article Title: Extended Kalman filtering on Stiefel manifolds

Article References: Extended Kalman filtering on Stiefel manifolds. (n.d.). https://doi.org/10.3389/frsip.2026.1829279

Image Credits: AI Generated

DOI: 10.3389/frsip.2026.1829279

Keywords: Kalman filter, Stiefel manifolds, nonlinear filtering, signal processing, directional statistics, spacecraft attitude, Riemannian geometry, projected normal distribution, Extended, Kalman, filtering, Stiefel