Learning enhanced ensemble filters

Bach, E. ORCID: https://orcid.org/0000-0002-9725-0203, Baptista, R., Calvello, E., Chen, B. and Stuart, A. (2026) Learning enhanced ensemble filters. Journal of Computational Physics, 547. 114550. ISSN 0021-9991 doi: 10.1016/j.jcp.2025.114550

Abstract/Summary

The filtering distribution in hidden Markov models evolves according to the law of a mean-field model in state–observation space. The ensemble Kalman filter (EnKF) approximates this mean-field model with an ensemble of interacting particles, employing a Gaussian ansatz for the joint distribution of the state and observation at each observation time. These methods are robust, but the Gaussian ansatz limits accuracy. Here this shortcoming is addressed by using machine learning to map the joint predicted state and observation to the updated state estimate. The derivation of methods from a mean field formulation of the true filtering distribution suggests a single parametrization of the algorithm that can be deployed at different ensemble sizes. And we use a mean field formulation of the ensemble Kalman filter as an inductive bias for our architecture. To develop this perspective, in which the mean-field limit of the algorithm and finite interacting ensemble particle approximations share a common set of parameters, a novel form of neural operator is introduced, taking probability distributions as input: a measure neural mapping (MNM). A MNM is used to design a novel approach to filtering, the MNM-enhanced ensemble filter (MNMEF), which is defined in both the mean-field limit and for interacting ensemble particle approximations. The ensemble approach uses empirical measures as input to the MNM and is implemented using the set transformer, which is invariant to ensemble permutation and allows for different ensemble sizes. In practice fine-tuning of a small number of parameters, for specific ensemble sizes, further enhances the accuracy of the scheme. The promise of the approach is demonstrated by its superior root-mean-square-error performance relative to leading methods in filtering the Lorenz ‘96 and Kuramoto-Sivashinsky models.

Item Type	Article
URI	https://centaur.reading.ac.uk/id/eprint/127478
Identification Number/DOI	10.1016/j.jcp.2025.114550
Refereed	Yes
Divisions	Science > School of Mathematical, Physical and Computational Sciences > National Centre for Earth Observation (NCEO) Interdisciplinary Research Centres (IDRCs) > Centre for the Mathematics of Planet Earth (CMPE) Science > School of Mathematical, Physical and Computational Sciences > Department of Mathematics and Statistics Science > School of Mathematical, Physical and Computational Sciences > Department of Meteorology
Publisher	Elsevier
Download/View statistics	View download statistics for this item