In supervised data assimilation machine learning emulation, the training data contain targets produced by an existing data assimilation scheme, such as analysis increments. By contrast, data assimilation networks were recently proposed to learn the analysis operator while they are embedded in the forecast–analysis cycle: their only targets are the true trajectory and the observations thereof. They are therefore trained to produce a stable and accurate sequential estimator, rather than to reproduce the output of a prescribed data assimilation algorithm. Conceptually more fundamental, yet computationally more challenging, such learned data assimilation scheme was shown to achieve accuracy comparable to that of the ensemble Kalman filter when applied to low-order chaotic dynamics. Strikingly, the same accuracy can be reached with a single state forecast instead of an ensemble, hence bypassing the need to explicitly represent forecast uncertainty.
In this study, we extend the investigation of such learned analysis operators beyond the preliminary experiments reported so far. First, we analyse the emergence of local patterns encoded in the operator, which accounts for the remarkable scalability of the approach to high-dimensional state spaces. Second, we assess the performance of the learned operators in stronger nonlinear regimes of the chaotic dynamics. We show that they can match the efficiency of the iterative ensemble Kalman filter, the baseline in this context, while avoiding the need for nonlinear iterative optimisation. Throughout the paper, we seek underlying reasons for the efficiency of the approach, drawing on insights from both machine learning and nonlinear data assimilation.
Agence Nationale de la RechercheANR-24-EXMA-0002ANR-24-EXMA-0001ANR-22-CPJ2-0143-01Introduction
Accurate prediction of geophysical flows relies on the continual correction of model trajectories using observations. This sequential data assimilation (DA) process is essential in high-dimensional chaotic systems, where errors amplify rapidly and model imperfections accumulate over time . In operational meteorology and oceanography, ensemble-based Kalman filters and ensemble variational methods provide reliable and well-understood frameworks for these updates, but they remain computationally demanding and rely on explicit representations of flow-dependent forecast error covariances.
Recent developments have suggested that part of this complexity can be replaced by learned analysis operators. In supervised DA machine learning emulation, the training dataset contains targets produced by an existing DA scheme, such as analysis increments e.g.,. By contrast, one could learn the analysis operator while it is embedded in the forecast–analysis cycle: its only targets are the true trajectory and the observations thereof . Such analysis operator is therefore trained to produce a stable and accurate sequential estimator, rather than to reproduce the output of a prescribed DA algorithm. Within this latter approach, termed data assimilation networksDANs,, a surprising result has emerged: as demonstrated by , a learned analysis operator can match the accuracy of a well-tuned ensemble Kalman filter (EnKF) even when it uses only single forecast trajectories, without any ensemble. This result challenges the long-standing assumption that explicit ensemble representations are indispensable to estimate flow-dependent uncertainties in chaotic systems.
Explaining this phenomenon is a central question for DA methodology. Initial investigations suggest that the learned operator implicitly reconstructs aspects of the analysis error covariances normally diagnosed from an ensemble, effectively uncovering key uncertainty directions from the forecast state alone . This behaviour is consistent with viewing the full DA cycle as a random dynamical system, for which generalised forms of the multiplicative ergodic theorem offer a state-dependent structure linking model trajectories to dominant error-growth directions. At the same time, numerical experiments show that the neural network is likely to identify local patterns rather than memorising global states, which explains why its performance scales to larger systems and remains robust across different model dimensions.
This paper deepens the investigation of these mechanisms. In Sect. , we dive into the methodology of DANs, and recall the main results and questions raised in . In Sect. , building on a more thorough analysis of the performance dependence on the batch size, dataset length, and the number of assimilation cycles used during backpropagation, we study how the operator behaves when interpreted as a diagnostic of uncertainty, and we propose methods to expose and interpret the local structures it extracts. In Sect. , we test the limits of the approach in more strongly nonlinear regimes, where the iterative ensemble Kalman filter and its generalisations often serve as the most accurate baselines. We show that the learned operator can reach comparable performance without requiring an ensemble or a nonlinear optimisation, and we offer a data assimilation-based interpretation for this ability. Section presents our conclusions. Supporting numerical and mathematical results are collated in the appendices of this paper.
Throughout the paper, we will illustrate our results with the Lorenz-96 model L96,. More broadly, our goal is not only to document the performance of the learned analysis operators but also to clarify the mechanisms that underlie them, thereby contributing to the growing theoretical understanding of how deep learning and sequential data assimilation interact in chaotic geophysical systems .
Theory and methods
In this section, we provide a deeper description of the problem, its context, and its mathematical formulation.
Sequential data assimilation for chaotic dynamics
Mathematically, data assimilation (DA), and in particular filtering algorithms, are meant to accurately estimate the state vector xkt∈Ex=ΔRNx of a physical system, where “t” stands for truth, at times tk for k=1,…,K along a trajectory of the dynamical system. These states are evolved according to
xk+1t=Mxkt,
where M is the integrated model over tk+1-tk=Δt. The dynamical system M is assumed to be chaotic, such as for most geofluids, which is a prime incentive for frequently updating our knowledge of the system. It is furthermore assumed ergodic and autonomous, i.e. does not explicitly depend on time. Furthermore, the physical system is observed through
yk=Hk(xkt)+εk,εk∼N(0,Rk),
where yk∈Eyk=ΔRNy,k is the observation vector at tk obtained from the hidden state xkt via an observation operator Hk, and perturbed by a white-in-time Gaussian noise εk of mean 0 and covariance matrix Rk. This very common but simplified formulation of the filtering problem with additive Gaussian noise is sufficient for the goals of this paper.
A filtering DA scheme estimates the hidden state xkt at tk from the observations available up to that time. The analysis xka is a point estimator, such as the posterior mean or maximum a posteriori estimate, of the conditional probability density function p(xkt|yk,yk-1,…,y1). A sequential filtering DA scheme infers xka from yk and from background information about the state at tk-1 (possibly brought forward to tk using M).
Learning the data assimilation analysisTraining scheme
The approach developed in , subsequently referred to as Boc24, in the wake of and is summarised in the following since it is the foundation of the present paper. The analysis step of the DA scheme is assumed to be given by the (incremental) analysis operator aθ, typically a (deep) neural network, which depends on a set of weights and biases, stacked in the θ vector, and which is defined, at time tk, through
2axka=xkf+aθ(xkf,Hk⊺Rk-1δk),2bδk=Δyk-Hk(xkf),
where δk is the innovation, xka is the analysis state mentioned previously, and xkf is the forecast state, to be defined shortly. Hk is the tangent linear operator of Hk. Should aθ be a function of δk rather than Hk⊺Rk-1δk, it could only handle static observation configurations, and would require to be retrained whenever that configuration changes. Although not the focus of this paper, this important issue is circumvented here by using the mapping from observation to state space Hk⊺Rk-1, such that both inputs of aθ are in the same static state space Ex. For compactness, we define the projected innovation
ζk=ΔHk⊺Rk-1δk=Hk⊺Rk-1(yk-Hk(xkf)).
The DAN analysis operator thus receives two fields in state space, xkf and ζk.
In the DA forecast step, the analysis state is propagated to the subsequent date through
xk+1f=Mxka.
The operator aθ is trained by comparing the analysis states to the true states, through the loss
L(θ)=∑r=1Nr∑k=1Ncxkt,r-xka,r(θ)2,
where xkt,r are the state vectors of the true trajectory, indexed by the time index k and a trajectory index r as Nr of them are processed concurrently in the training of aθ.
The Nc parameter counts the number of cycles of each DA run. It can potentially be infinite since trajectories can be generated online as the training progresses. The implementation of similar losses is detailed in , , and Sect. II of Boc24. This notably requires to truncate backpropagation in time which restricts the dependence on θ over Niter≪Nc cycles only so as to reduce the computational cost and the excessive GPU memory (VRAM) requirement. Alternative losses based on probability density functions (pdf) diagnostics are possible and discussed in, e.g., . We have shown that a semi-supervised loss where the training dataset reduces to the sparse and noisy observations is also possible by generalising a proposal of , but is out of the scope of this paper.
Accuracy of the discovered data assimilation scheme
Note that an ensemble variant of the update Eq. (2) was first considered in , , and later by Boc24. Experimenting with the same low-order chaotic model, they found an accuracy of the learned DA scheme close to that of a well-tuned EnKF, which is per se very promising. In those experiments, DAN is built on an ensemble of analyses and forecasts. However, Boc24 demonstrated that this accuracy remains unchanged when these ensembles are reduced to a single state. This is a very surprising result since it is expected (and verified in the L96 context) that the EnKF-like methods critically relying on an ensemble representing the errors of the day, have a significant edge over other sequential DA methods that leverage a single forecast state such as 3D-Var and 4D-Var seefor a quantitative comparison with the same model. This explains why numerical weather prediction (NWP) centres operating 4D-Var actually rely on an ensemble of such 4D-Var, a technique called EDA see, e.g., Chap. 7 inand references within, or use information from a concurrent EnKF . That is why achieving the accuracy of a well-tuned EnKF with a single forecast state should have far-reaching implications on the mechanisms and designs of DA algorithms for chaotic dynamics, and warrants investigating the reason for such feat.
Investigating the reason for this efficiency
To unveil the mechanisms leveraged by the learned analysis operator to achieve this accuracy, Boc24 performed a linearisation of the operator aθ in the projected innovation ζ:
aθ(x,ζ)≈Pa(x)ζ,
where Pa(x) is the resulting linear operator that acts on ζ. Comparing with the linearisation of the classical Kalman update, it must coincide with the analysis error covariance matrix which could be associated by formal analogy to the learned analysis operator aθ, hence the notation. This covariance matrix was numerically obtained in Boc24 through a linear regression in between a large ensemble of ζ samples (corresponding to the projected innovations H⊺R-1δ) and aθ(x,ζ) outputs. It was found that this error covariance matrix is remarkably close to that of a well-tuned EnKF, especially for its dominant eigenvectors (which carry most of the uncertainty in the analysis).
Hence, the performance of DAN, as reported in Boc24, must be to a large extent due to its ability to infer flow-dependent effective analysis error covariances from the forecast state, as revealed by the local dependence of aθ(xf,ζ) on the projected innovation ζ. Such an ability is pivotal for maintaining the accuracy of sequential DA over time. In the present paper, we will sometimes reason in terms of the forecast error covariance matrix Pf in place of Pa, since the flow dependence is more naturally encoded in Pf, while Pa is obtained after conditioning on the observations. Moreover, in the linear/Gaussian case, these two covariance matrices are straightforwardly related by
Pa-1=(Pf)-1+H⊺R-1H,
assuming invertibility of the error covariance matrices. Beyond the linear/Gaussian case, perturbing the innovations and applying DAN can generate an empirical analysis ensemble; propagating this ensemble with the model M then provides a way to diagnose the corresponding forecast error covariances.
Other alternatives to ensemble forecasting to estimate the flow-dependent error statistics, such as deriving the dynamics of the statistical moments of the errors , or estimating these dynamics through machine learning are either numerically very costly or in their infancy.
Boc24 conjectured that DAN's ability to assess the flow-dependent error statistics can be fundamentally explained by the existence of an ad-hoc multiplicative ergodic theorem (MET). From the seminal MET result by , we know that, for an autonomous ergodic dynamical system such as M, there exists, for almost every state on the attractor, a measurable mapping from the state to the corresponding Oseledets subspaces. Generalising, one can consider the whole sequential DA process as a dynamical system on its own . Such DA process is not autonomous because it indirectly depends on the truth trajectory and the time-dependent observation operators. Moreover, it is a random process, since stochasticity is injected via the noisy observations. It turns out that generalised variants of the MET for non-autonomous random dynamics are possible and are potentially applicable to such sequential DA process. Hence, Boc24 conjectured that aθ can exploit such state-dependent information to infer an effective analysis error covariance matrix, together with how to process this information and combine it with the innovation.
To explain the efficiency of aθ, one may suggest that the neural network memorises global configurations of the forecast state, with very limited ability to generalise. On the contrary, Boc24 showed via indirect scaling experiments that aθ learns to identify local patterns (i.e. with a limited range in space), which was made easier by the architecture of aθ being a residual convolutional neural network. Indeed, when the dimension Nx of the L96 state space is increased, and new aθ operators are learned but with a fixed number of weights and biases of the backbone architecture, the accuracy remains that of a well tuned EnKF of matching dimension. Yet, in the large Nx limit, aθ with the same number of degrees of freedom should not be able to memorise increasingly numerous global patterns. Hence, aθ must extract local patterns, and a limited number of them. This is further supported by learning aθ with the original L96 dimension, Nx=40, but applying it to L96 with significantly different Nx in successful DA runs (this is allowed by the convolutional architecture which does not explicitly depend on Nx), performing on par with a well tuned EnKF. Hence, any local pattern learned in the case Nx=40, must still be spotted by aθ where Nx≠40. This outcome is consistent with the existence of such local patterns, since L96 is an extensive model when Nx is increased, with the number of nonlinear interacting waves in the model being proportional to Nx. This neat scalability has limitations. If increasing the dimension changed the local statistics or local dynamical balances of the model, for example through a substantial change of spectral slope or local instability mechanisms, a DAN trained at the smaller dimension should not be expected to generalise without fine-tuning, additional training, or architectural adaptation.
Exploration of data assimilation networks
With the previously established context in mind, we now explore what DANs (aθ) learn in mildly nonlinear regimes. The neural network associated to aθ implements Eq. (2). The architecture for aθ we choose in the present paper is the same as the one reported in Boc24, i.e. a simple residual convolutional neural network that accounts well for the spatial homogeneity of L96 (and hence its statistical stationarity). For the sake of self-sufficiency, Appendix describes this architecture. All the training tasks are carried out over a training dataset with a minimisation of the loss controlled by computing the loss over a validation dataset.
Numerical setup
The numerical experiments of the present paper are performed on L96, a chaotic model abundantly used for benchmarking new sequential data assimilation algorithms. As a reminder, L96 represents a mid-latitude zonal circle of the global atmosphere. It is governed by Nx=40 ordinary differential equations:
dxndt=(xn+1-xn-2)xn-1-xn+F,
where F=8, and with cyclic boundary conditions. The resulting model is chaotic with 13 positive and 1 neutral Lyapunov exponents. Its Lyapunov time, defined as the inverse of the first Lyapunov exponent, is about 0.60, which corresponds to 3 d of a typical weather forecasting model .
Although the results are generalisable to sparse observations, the observation operator will mostly be chosen to be the identity Hk=ΔIx. The observations are read off the true state and perturbed with an unbiased white-in-time Gaussian additive noise of covariance matrix Rk=ΔIx following Eq. (). The time-step between observation batches and updates is set to Δt=tk+1-tk=0.05 for all experiments with the exception of Sect. . This will be our reference DA setup. Examples of alternative sparseness and noise levels are given in Boc24, but here as well when relevant.
All the aθ operators learned in this configuration are subsequently evaluated with a time-averaged analysis root mean square error (RMSE) whose mean square is averaged over the Nx variables. This RMSE is assessed over a dataset of independent test trajectories, similarly to traditional DA twin experiments. For brevity, this score computed for each trained DAN scheme will simply be called test RMSE of the DAN scheme in the rest of this paper.
In this configuration, running a well-tuned EnKF in a twin experiment and comparing its analysis to the truth yield a test RMSE between 0.18 and 0.20 depending on the ensemble size Ne and whether localisation is used or not. By contrast, a basic but well-tuned 3D-Var or a reasonably short window basic but well tuned 4D-Var yields a test RMSE of about 0.40. They largely underperform the EnKF because they fail to capture the errors of day .
In order to be able to perform a large number of training experiments on a limited number of GPUs and limited VRAM, we carried out a sensitivity study on the batch size, the size of the datasets, and the backpropagation truncation, beyond the restricted set of experiments reported in Boc24. Since the results are technical and mostly of practical interest, they are reported in Appendix . They were nonetheless instrumental in obtaining the main numerical results of this paper.
Linearisation in the innovations
In this section, we discuss the relevance of expanding aθ linearly in the innovations, as in Eq. (). We recall that the focus is on the analysis step of the DAN process, where Ne=1, i.e. a single forecast state is propagated in between updates. Under the additional Gaussian and quasi-linear assumptions used in this subsection, a close to optimal analysis associated with the Kalman update is the posterior mean which coincides with the maximum a posteriori of the conditional pdf, hence by the minimum of the cost function associated to the analysis. In the following, the observation operator is assumed linear (or linearised) for simplicity, i.e. H=ΔH.
A Gaussian standpoint on the analysis
Here, we further assume that the background errors are Gaussian. Nonetheless, as opposed to a basic 3D-Var, the background error covariance matrix depends on the forecast state. Hence, the typical analysis cost function associated to the analysis at any given time step has the form:
J(x)=12y-HxR-12+12‖x-xf‖Pf(xf)-12.
As a consequence, J is quadratic in x, strictly convex, and its minimum argument is xa=xf+[H⊺R-1H+(Pf(xf))-1]-1H⊺R-1(y-Hxf)9=xf+Pa(xf)ζ,
where the projected innovation ζ has been defined by Eq. (). Equation () matches the linearisation Eq. (). The purpose of this derivation is to show that, under Gaussian/quasi-linear assumptions, DAN is likely to solve the optimisation problem Eq. (), that its update follows Eq. (), and that its Jacobian with respect to ζ plays the role of an effective analysis error covariance matrix Pa(xf). Whether this local interpretation is sufficient is then tested numerically.
Numerical evidence
To numerically test whether such a linearisation is a good approximation for aθ, we created a modified DAN which explicitly learns an effective map xf↦Pa(xf) and linearly combines it with the projected innovations, with the goal to strictly follow the update equation Eq. (). Details on our scalable implementation of Eq. (), for such a DAN linear in the projected innovations (from now on called linear-in-ζ DAN), can be found in Appendix . One must keep in mind that this linear-in-ζ DAN is numerically inefficient since it requires to build a representation of the covariance matrix Pa before being applied to the projected innovations, an operation which is likely to be achieved with our standard DAN without ever computing Pa explicitly.
This deconstruction of aθ directly connects with the heuristic developed by and , where the mapping xf↦Pf(xf) is learned and then successfully used within a classical EnKF. Numerically, the linear-in-ζ DAN scheme turns out to be as accurate as a well-tuned EnKF with Ne=40 in the reference setup (Δt=0.05 in particular), with a test RMSE of ∼0.19, which is in line with the results by and . Hence, we can claim that the good performance of DAN in this mildly nonlinear regime importantly relies on the (implicit) estimation of the mapping xf↦Pf(xf), and subsequently xf↦Pa(xf).
On the importance of the <inline-formula><mml:math id="M133" display="inline"><mml:mrow><mml:msup><mml:mi mathvariant="bold">x</mml:mi><mml:mi mathvariant="normal">f</mml:mi></mml:msup><mml:mo>↦</mml:mo><mml:msup><mml:mi mathvariant="bold">P</mml:mi><mml:mi mathvariant="normal">f</mml:mi></mml:msup><mml:mo>(</mml:mo><mml:msup><mml:mi mathvariant="bold">x</mml:mi><mml:mi mathvariant="normal">f</mml:mi></mml:msup><mml:mo>)</mml:mo></mml:mrow></mml:math></inline-formula> map
Let us focus on , whose algorithmic constructions targeted at estimating Pf(xf) are especially transparent and whose numerical tests were carried out in the reference setup described in Sect. . Their Algorithm A1 proceeds as follows: (i) the state xf is backtracked by T time steps; (ii) the tangent linear model, evaluated along the resulting state trajectory from t-T to t0, is applied to a matrix of state perturbations εIx; and (iii) the resulting perturbations at time t0 are used to estimate Pf(xf). This procedure is closely related to the Assimilation in the Unstable Space (AUS) methods , since for sufficiently large T the output of the tangent linear model at t0 provides a square-root factor of the backward Lyapunov vectors. But it is already sufficient to yield a competitive analysis relying on the errors of the day estimated through Pf(xf) which notably relies on a single state.
Algorithm A2 of also backtracks the state xf by T time steps but differs in the subsequent step: instead of propagating perturbations, it applies a (square-root) Kalman filter to an initial covariance matrix ε2Ix from t-T to t0, yielding a more refined estimate of Pf(xf) at t0. This yields an even more accurate DA algorithm on par with a well tuned EnKF, again relying on Pf(xf) assessed from a single state.
and have laid the theoretical grounds to understand why these principled demonstrations are successful. focused on a degenerate Kalman filter, equivalent to an EnKF in Gaussian and quasi-linear conditions. In this setting, the covariance evolution is formally decoupled from the state update, with the important caveat that the forecast error covariance Pf depends on the state xf at time t0. Indeed, showed that, asymptotically (i.e. for large T), Pf depends on the system dynamics only through the Lyapunov vectors. By the MET, the corresponding Oseledets subspaces are measurable functions of the state xf at t0 (for almost every state on the attractor). Hence, following , Algorithm A2 by can be directly interpreted as the single state-dependent covariance propagation of a degenerate Kalman filter. In the case of nonlinear dynamics, enforcing the dependence of the error covariance Pf on xf is even more beneficial since the covariance evolution now explicitly depends on xf.
These theoretical results and proof experiments provide a rationale for the existence of a map xf↦Pf(xf) and its importance in estimating the errors-of-the-day in DA schemes based on a single forecast state Ne=1.
Patterns
In this section, we visualise a footprint of the local patterns leveraged by DAN to make its inferences. To that end, we study the dependence of ∇ζaθ(x,ζ)|ζ=0≈Pa(x) on x, i.e. the forecast state. This can be seen as a linear-in-ζ sensitivity analysis of aθ with respect to its inputs.
Mean marginal gain tensor
Specifically, the sensitivity analysis will focus on Jacobians of the analysis operator with respect to x:
10aΓ(x,ζ)=Δ∇x∇ζaθ(x,ζ),10bΓ(x)=ΔΓ(x,0)=∇x∇ζaθ(x,ζ)|ζ=0.
Since Γ(x) is a proxy to ∇xPa(x) as per Eq. (), Γ(x) can be interpreted as the marginal variation of Pa(x) when the forecast state x is perturbed. Under Gaussianity and linearity and assuming H⊺R-1=Ix, Pa(x) coincides with the Kalman gain. That is why we will call Γ(x) the marginal gain tensor, or simply marginal gain. Mathematically, this is a 3-tensor field, i.e. a map from the state space Ex to Ex3. Component-wise, using a conventional placement of indices, its definition reads
[Γ(x)]ijl=Δ∂xl∂ζjaθ,i(x,ζ)|ζ=0.
To mitigate the complexity in studying the map x↦Γ(x), we introduce the mean marginal gain, Γ‾. This 3-tensor is defined as the average of Γ(x) over the K states of a long trajectory Tx=xktk=1,…,K∈ExK of the M–based ergodic chaotic dynamics:
Γ‾=Δ〈Γ(x)〉x∈Tx=K→∞∫dxπ(x)Γ(x)=Ex∼πΓ(x),
where π is the invariant distribution of the ergodic chaotic dynamics.
As recalled in Sect. , ∇ζaθ(x,ζ)|ζ=0≈Pa(x) was heuristically estimated in Boc24 using samples of ζ followed by a linear regression. This estimation through a regression can be formally justified using the following argument. We wish to average ∇ζaθ(x,ζ) over the pdf ρ(ζ) of the projected innovations ζ, which is assumed to be a Gaussian with a positive-definite covariance matrix Σρ. Then it can be shown that
13aEζ∼ρ∇ζaθ(x,ζ)=∫dζρ(ζ)∇ζaθ(x,ζ)13b=Σρ-1Covζ∼ρζ,aθ(x,ζ),
where Cov⋅,⋅ is the covariance operator, and Σρ-1 operates on the first tensor factor. A proof is given in Appendix , along with a generalisation to the case where Σρ is only semi positive-definite, which is made necessary because the set {ζk=Hk⊺Rk-1δk}k may only span a subspace of Ex if Hk is not injective. This result is none other than Stein's lemma applied to the Gaussian approximation of ρ and aθ(x,⋅). It aligns with its original use in ensemble DA and with the connection established by Lemma 2 of between Σρ-1Covζ∼ρζ,aθ(x,ζ), which is a perturbations-based estimator , and a SmoothGrad estimator . In the limit where ζ, as a random vector, is concentrated around 0 and ρ can be approximated as Gaussian, which corresponds to the most common weak assimilation regime where the information content of the innovation is small compared to that of the background, we have
14aΣρ-1Covζ∼ρζ,aθ(x,ζ)=∫dζρ(ζ)∇ζaθ(x,ζ)14b≈∇ζaθ(x,ζ)|ζ=0,
which, with Eq. (), relates the integral form Eζ∼ρ[∇ζaθ(x,ζ)] to the gradient form ∇ζaθ(x,ζ)|ζ=0 of the sensitivity of aθ(x,ζ) with respect to ζ.
Building on this correspondance, we can connect the following mean marginal gain
Γ̃=ΔEx∼π,ζ∼ρ[Γ(x,ζ)]=∫dxdζπ(x)ρ(ζ)Γ(x,ζ),
defined from Eq. () and more closely related to the DA process, to the mean marginal gain Γ‾ as defined in Eq. ():
16aΓ̃=Ex∼π,ζ∼ρΓ(x,ζ)16b=Ex∼π∇xEζ∼ρ∇ζaθ(x,ζ)16c≈Ex∼π∇x∇ζaθ(x,ζ)|ζ=016d=Ex∼πΓ(x)16e=Γ‾.
However, the empirical mean of Γ(x,ζ) from which to evaluate Γ̃, and denoted Γ^, and which is the numerical estimation of the sensitivity out of a long enough DA run, may differ from both theoretical means Γ̃ and Γ‾. That is why we show in Appendix how Γ^ approximates Γ̃.
Note that Eq. () is a simple point estimator, which formalises that the averaged Jacobian Eζ[∇ζaθ(x,ζ)] is well approximated by ∇ζaθ(x,ζ)|ζ=0 when the distribution of projected innovations is concentrated near zero. The Stein lemma, which provides a regression-based estimator of this average, is not required per se to ascertain Eq. (), but offers an alternative expression which we leverage in Appendix .
Invariance and equivariance
An important way to reduce the computational cost and complexity of interpreting the mean marginal gain Γ̃ is to exploit the symmetries of the DA problem, when applicable. If the model variables are observed homogeneously and homoscedastically, then the probability laws defining the DA problem are invariant under these symmetries, and the corresponding analysis operator is equivariant. As a result, the components of Γ̃ related by symmetry are redundant, so that its interpretation can be restricted to a reduced set of representative components.
For instance, in the one-dimensional periodic L96 model with homogeneous and homoscedastic observations, a cyclic translation of the state variables induces the same cyclic translation of the observations, innovations, and analysis increments. The probability laws defining the DA problem are therefore invariant under the translation group, while the corresponding analysis operator is equivariant. Consequently, if the trained DAN preserves this symmetry, any trajectory-averaged sensitivity tensor must have a circulant structure: the information associated with different grid points is identical up to a cyclic shift. The group-theoretical argument below provides a formal justification for this circulant averaging.
Let us generically denote G such group of symmetries for the DA process, which are assumed to be isometries. It is chosen to be the maximal group for which both the dynamics and the observation process are equivariant. Appendix gives the formal definitions for this group of symmetries.
The action of g∈G on Γ(x,ζ) is denoted g⊙Γ(x,ζ). It is a tensor field for G, in the sense that it is equivariant under the action of this symmetry group following the transformation rule, ∀g∈G,∀x∈Ex,∀ζ∈Ex:
g⊙Γ(x,ζ)=ΔΓ(g∘x,g∘ζ)=g⊗g⊺⊗g⊺∘Γ(x,ζ),
where the ordering of the three tensorial factors follows the convention of Eq. (). A proof of the equivariance Eq. () is proposed in Appendix .
Invoking the ergodicity of the dynamics, any symmetry of G leaves the invariant distribution of the dynamics π, and the projected innovation distribution ρ, unchanged, ∀g∈G,∀x∈Ex,∀ζ∈Ex:
π(g∘x)=π(x),ρ(g∘ζ)=ρ(ζ),
where g∘(⋅) denotes the action of g on a field.
Leveraging the equivariance of the marginal gain and the symmetries of the invariant distribution, we now demonstrate the invariance of the mean marginal gain Γ̃; for g∈G, we have
19ag⊙Γ̃=∫dxdζπ(x)ρ(ζ)g⊙Γ(x,ζ)19b=∫dxdζπ(x)ρ(ζ)Γ(g∘x,g∘ζ)=∫d(g-1∘x)d(g-1∘ζ)19c×π(g-1∘x)ρ(g-1∘ζ)Γ(x,ζ)19d=∫dxdζπ(x)ρ(ζ)Γ(x,ζ)=Γ̃,
where a change of variables was carried out from Eq. () to Eq. (). The invariance of π and ρ under G, and the fact that the determinant of the Jacobian of g-1 is 1 since g-1 is an isometry were utilised from Eq. () to Eq. (). We finally conclude:
∀g∈G:g⊙Γ̃=Γ̃.
The same result can be obtained for Γ‾, with a simpler derivation only involving the invariant distribution π of the underlying dynamics. However, the symmetry group Γ‾ must be the same as that of Γ̃, and not the potentially larger group associated to π:
∀g∈G:g⊙Γ‾=Γ‾.
As a consequence, in the rest of this section, we assume that the results apply equally to either Γ̃ or Γ‾.
Plot of the normalised mean marginal gain Ω‾ij(r)/max|Ω‾ij(r)| in the fully observed DA configuration of the L96 model, obtained with a model size of Nx=40(panel a) and Nx=80(panel b).
Numerical illustrations
Leveraging the equivariance induced by the translational invariance of the L96 model, for a fixed reference site r, we define the slice matrix Ω‾(r)∈RNx×Nx component-wise by
Ω‾ij(r)=ΔΓ‾ijr.
The index r is not an additional free index of Ω‾(r); it labels the chosen slice of the averaged 3-tensor. For visual convenience, r is chosen to be in the middle of the domain r=⌊Nx/2⌋ in the L96 case. The simple but tedious details justifying Eq. () are given in Appendix . How to numerically compute this Ω‾(r) matrix is then discussed in Appendix .
To start with, we choose the fully observed DA setup H=R=Ix, which applies to both the training of aθ and the subsequent DA tests. The computation of Ω‾(r) is carried out through the composite approach (see Appendix ). The results are shown in Fig. .
The dominant values of |Ω‾(r)| form a pattern. They are concentrated in the vicinity of the perturbed variable of the forecast state, which makes them local. To further qualify the local patterns, we also trained aθ on a L96 model but with Nx=80, i.e. twice the size of the standard L96 model, while all other parameters either related to the dynamics or the DA experiments, remain the same. Then, Ω‾(r) is similarly computed and plotted in Fig. . As expected, the same pattern emerges with the same spatial extension. This further supports the EnKF-like performance of aθ trained with Nx=40 when tested with Nx=80 (as recalled in Sect. ).
Patterns with non-trivial observation operators
We carried out the exact same experiments described above, but now with three different observation configurations instead of the full observation setup. The corresponding Ω‾(r) are plotted in Fig. . Panel (a) corresponds to the fully observed configuration for reference. The first configuration (panel b) corresponds to the observation of every other site: [H]ji=δi,2j for 1≤i≤Nx and 1≤j≤⌊Nx/2⌋. The second (panel c) corresponds, at each time step tk, to an observation at Nyk random but distinct sites, where Nyk is uniformly drawn at each time step in between 0 and Nx (bounds included). The third configuration (panel d) is the same as the second but with Nyk uniformly drawn at each time step in between ⌊Nx/2⌋ and Nx. It is remarkable that the same local pattern emerges in all configurations, even when every other site is never observed. However, the magnitude of the sensitivities (values of the patterns) changes depending on the information balance in the analysis and hence in the gain magnitude. Finally, note that Ω‾(r) only represents an average pattern. It is possible to exhibit a family of modal patterns but it would go beyond the aim of the present paper.
Plot of the mean marginal gain Ω‾ij(r) normalised by the maximum of its absolute value over all entries and over the four panels, in several sparse observation DA configuration of the L96 model.
Data assimilation networks in stronger nonlinear regimes
In Sect. , we made several contributions to the understanding of DAN. The numerical experiments were set in a mildly nonlinear regime of L96 where Δt=0.05 in between analyses, known to correspond to 6 h of a synoptic meteorological model , and a forcing F set to 8. We now examine the ability of the method to learn efficient analysis schemes under stronger nonlinear conditions. While we exclude cases in which the nonlinearity is so severe that it induces implicit or explicit recurrent multi-modal priors, we do consider regimes that exhibit substantial departures from mild nonlinearity.
Probing such mildly to strongly nonlinear regimes can be achieved by less frequent observation, typically increasing the update time-step Δt. The dimensional analysis of Appendix 1 in shows that varying Δt is indeed relevant to achieve such objective for the L96 model versus, e.g., increasing the observation error amplitude. The forcing F can also be varied to that end. However, it rather stands as a signature of the magnitude of the instability of the dynamics (e.g., a covariant function of the Kaplan-Yorke dimension, a measure of the fractal dimension of the dynamics attractor) rather than the signature of the deviation from Gaussianity.
Performance as a function of the update time-step
Increasing Δt to multiples of 0.05 is the experimental design chosen in and to evaluate the performance of the iterative Ensemble Kalman Filter (IEnKF). As the ensemble variant of the iterative Kalman filter , the IEnKF still stands, to our best knowledge, as the most accurate scalable DA method in mildly to strongly nonlinear conditions . It is hence a hard-to-beat baseline for learning an advanced DA analysis scheme in such conditions. The remarkable performance of the IEnKF stems from its ensemble-variational formulation obtained from Bayesian first principles: an ensemble is used to construct a time-dependent prior, and the analysis is performed through a nonlinear iterative optimisation .
The IEnKF can be made even more accurate (both for smoothing and filtering) by choosing longer DA windows, yielding the iterative ensemble Kalman smoother IEnKS,. However, it works over longer DA windows, as opposed to the DAN implemented here, which would bias the comparison. Actually, we have developed (recurrent) variants of DAN that work on extended DA windows similarly to the IEnKS, but reporting on them is beyond the goals of this paper.
We choose a simple common observational configuration for the many DA methods we intend to compare. All sites are observed through Hk=ΔIx and Rk=ΔIx, a well documented setup in the literature. In this configuration, any useful, but not necessarily accurate, DA method must exhibit a test RMSE slightly below 1. We consider the following DA methods:
A well tuned EnKF with an ensemble of size Ne=Nx=40. Optimal multiplicative inflation is addressed through the finite-size EnKF . This first contender is meant to illustrate the progressive failure of the EnKF in stronger nonlinear conditions.
A well tuned IEnKF with an ensemble of size Ne=Nx=40, whose optimal multiplicative inflation is addressed through the finite-size IEnKF . This is our hard baseline.
A baseline learned DAN scheme using the neural network and the training parameters values set in Appendix .
A boosted learned DAN scheme using Nf=80, Nr=219, Niter=32, Sb=512, which is parameter and data intensive, and hence much more time-consuming to train on a single GPU, while its inference remains cheap.
A learned DAN scheme where the explicit dependence on the forecast state is discarded, while the dependence on the projected innovations is maintained. We expect the resulting DA method to perform similarly to a well tuned 3D-Var, see Boc24.
A learned DAN scheme where the activation functions are all linear, such that aθ is linear in its inputs. Like the previous approach, we expect the resulting DA method to perform similarly to a well tuned 3D-Var, see Boc24.
Test RMSEs of DA methods as a function of the update time-step Δt in between analyses. See text for details.
To avoid early divergences in the training when Δt≫0.05, all the DAN schemes benefited from a modified Eq. (2):
xka=xkf+αHk⊺Rk-1δk+aθ(xkf,Hk⊺Rk-1δk),
where α is a trainable scalar. Note that this formulation is mathematically equivalent to the original Eq. (2). This is only meant to explicitly enforce the solution xka=yk when Hk=Ix, which guides the training in its first few epochs.
The test RMSEs of those DA methods, over a long DA run, are displayed in Fig. as a function of Δt. Note that Δt=0.60 is already very significantly nonlinear as it corresponds to the Lyapunov time of L96, i.e. the time horizon beyond which the DA system becomes significantly non-Gaussian.
As expected, the two degraded DAN operators severely underperform the other DA schemes, which leverage non-static priors, with a test RMSE that ranges from 0.38 for Δt=0.05 to an asymptotic RMSE below 1 for much larger Δt. This is consistent with the findings of Boc24 and mirrors the performance of a 3D-Var with static background covariance matrix.
The EnKF offers a very good performance in the mildly nonlinear regime Δt≈0.05 but gradually degrades as Δt is increased. Moreover, beyond Δt=0.80, the EnKF becomes uninformative and its test RMSE is not reported. As already shown by , the IEnKF offers significantly better performance, from a marginal improvement over the EnKF at Δt=0.05 that gets more and more significant as Δt is increased. Note that, beyond Δt=0.60, the IEnKF is trickier to stabilise and hence its performance is not reported.
Remarkably, the boosted DAN operator achieves performance very similar to the IEnKF, but can still be learned for much larger Δt, and remains informative with still a very significant edge over the static prior methods. Again, this is achieved without the use of an ensemble but of a single forecast state. Moreover, the learned DAN does not explicitly resort to a nonlinear (Gauss-Newton) iterative minimisation, in contrast to the IEnKF. This aspect of such DAN is reminiscent of approaches meant to learn a solver for a variational DA problem . The baseline DAN operator is slightly less performing but follows the same trend. We could have evaluated the methods for even larger Δt>1.40; however, Δt=1.20 already stands for twice the Lyapunov time, which is equivalent to 6 d in the L96 correspondence.
We suspect that the test RMSEs should primarily be a function of NiterΔt rather than just Δt, accounting for the forgetful effect associated to the chaotic dynamics. Hence, to reach the same test RMSEs, Niter could be roughly chosen inversely proportional to Δt. To test this hypothesis, we compare the test RMSE on a large number of trained DANs, varying Δt (including for Δt≤0.05) and Niter with results shown in Fig. . Leveraging the findings of Appendix and the use of smaller batches to mitigate the numerical cost, we have chosen for their training, Nr=216=65536 and Sb=27=128. The test RMSEs are shown in Fig. . The results corroborate the intuition with the weaker and weaker dependence on Niter of the performance when Δt is increased. Conversely, a much larger Niter is required when Δt gets very small (Δt≈0.01).
Test RMSEs of DANs as a function of both Δt and Niter.
Hence, DAN is numerically more difficult to train when Δt<0.05 requiring larger Niter, which corresponds to quasi-linear regimes where DAN is nonetheless of limited interest.
Performance as a function of the energy forcing
The forcing F injects and removes energy from the L96 dynamics and feeds their instability. Besides Δt, this is another tunable parameter of the nonlinearity of the dynamics. There is a wealth of dynamical phenomenology of the L96 dynamics for a wide range of Fsee, e.g.,and references therein. For the range of F we focus on here, instabilities progressively develop in between 1≲F≲4, while chaos fully sets in beyond F≳4. The number of Lyapunov exponents, and similarly the Kaplan-Yorke dimension, increases monotonically from 4 to about 30 where it saturates .
We choose the same setup as in the previous experiments and consider the following DA methods:
A well tuned EnKF with an ensemble of size Ne=Nx=40. Optimal multiplicative inflation is addressed through the finite-size EnKF , in its Dirac-Jeffreys variant required to handle the weakly nonlinear regime (4≲F≲8).
A well tuned IEnKF with an ensemble of size Ne=Nx=40, whose optimal multiplicative inflation is addressed through the finite-size IEnKF in its Dirac-Jeffreys variant. This is our hard baseline.
A (baseline) learned DAN scheme using the neural network and training parameters as defined in Appendix .
The test RMSEs are displayed in Fig. . The IEnKF has a slight edge over the EnKF with larger F with the increasing Kaplan-Yorke dimension. Hence, the DA system does not deviate much from Gaussianity as F increases, rather, it is mainly the magnitude of the instabilities that changes. The DAN achieves a performance in between that of the EnKF and that of the IEnKF, which is patent for large F. Note that it turns out trickier to train DANs for F getting close to 4. There, the dynamics become more and more laminar and exhibit almost periodic waves, with patterns that, although simpler, are very different from those learned in the regimes explored so far.
Test RMSEs of DA methods as a function of the forcing F. See text for details.
Reasons for the efficiency of data assimilation networks
In the light of the previous numerical results, we discuss the reasons why DAN can be as accurate as the IEnKF, without an ensemble, without any experimental tuning, and even in mildly to strongly nonlinear conditions.
We have already shown in Sect. that in the mildly nonlinear regime, i.e. Δt≈0.05, the success of DAN mainly resides in its implicit estimation of the map xf↦Pf(xf), in line with the results by and , and closely related to the map xf↦Pa(xf). As shown in Sect. , this amounts to assume a non-static but Gaussian prior in the analysis: this is equivalent to having ‖x-xf‖Pf(xf)-12 as the background term in the analysis cost function. Whether this mechanism is sufficient to ensure the performance of DAN when Δt≥0.05 is doubtful.
The linear-in-<inline-formula><mml:math id="M376" display="inline"><mml:mi mathvariant="bold-italic">ζ</mml:mi></mml:math></inline-formula> data assimilation network beyond mild nonlinearity
To address this question, let us assess the linear-in-ζ DAN, see Sect. , in stronger nonlinear regimes Δt≥0.05. We could anticipate that it accounts well for the errors of the day, but that it may not be able to handle stronger nonlinearity/non-Gaussianity, e.g., if Δt≫0.05, because of the implied Gaussian prior. It should hence match the EnKF, rather than the IEnKF.
Let us check that hypothesis numerically. The test RMSEs of this specific DAN scheme, which mirrors Eq. (), are reported in Fig. , as the linear-in-ζ DAN. Let us first remark that, in accordance with the claims of Sect. , it performs as well as the EnKF for Δt=0.05. However, as Δt increases, its performance significantly degrades compared to the EnKF, not to mention the IEnKF. Even though it relies on an estimate of the xf↦Pa(xf) map, its underlying Gaussian assumption penalises it beyond the mildly nonlinear regime, as expected.
Deviation of the data assimilation network prior from Gaussianity
We conclude that the full DAN scheme not only exploits the flow-dependent information represented by the xf↦Pf(xf) map, but also learns an effective nonlinear analysis response consistent with a non-Gaussian prior, which is more informative than the Gaussian prior associated to the analysis cost function background term ‖x-xf‖Pf(xf)-12. As shown, both abilities are required for DAN to perform so well in the extended range of mildly to strongly nonlinear regimes.
Scaling of the standardised analysis increments of aθ in the projected innovations, for a large range of Δt values.
We investigate the deviation of aθ from the presumed quasi-linearity in ζ by defining the scalar function
r(λ)=Ex,ζPa(x)-1aθ(x,λζ)-aθ(x,0)Pa(x)-1aθ(x,ζ)-aθ(x,0),
where ⋅ is the Euclidean norm. It depends on a dimensionless scale parameter λ>0 and measures deviations from the linearity in ζ. In Eq. (), Pa(x)-1 is meant to standardise the deviations from linearity. Indeed, in the quasi-linear regime, we have
Pa(x)-1aθ(x,λζ)-aθ(x,0)≈λζ,
such that r(λ)≃λ should hold. The ratio r(λ) is estimated using perturbations as in Boc24: x and ζ are sampled from the forecast state xf and the projected innovations ζ=H⊺R-1(y-Hxf) that are obtained from a long trajectory.
The deviation from r(λ)≃λ is demonstrated in Fig. with λ↦r(λ) plotted for several values of Δt in the range [0.05,1.5]. These curves should be appreciated knowing the range of values taken by the projected innovations in a long DA run for the selected values of Δt which, for each Δt, points to the relevant range of λ values to consider and which mainly contributes to the computation of r(λ). This is shown in Fig. in the form of histograms of those values. Given the range of innovation values and the logarithmic scale of Fig. , the most relevant part of the scaling behaviour sits in the range λ∈[1,10]. In this range of the scaling, as seen in Fig. , the more nonlinear the DA run, the steeper λ↦r(λ): the innovation impact is stronger than the one expected in the linear regime, especially for large magnitude innovations.
Histograms of the projected innovations' norm, for a large range of Δt values.
Discussion and conclusions
In this paper, we have continued to explore the potential of learning sequential DA operators with neural networks for tracking chaotic dynamical systems, following on the initial results and conclusions of Boc24 who built on the concepts of data assimilation networks (DANs) introduced by and . Assuming the dynamics to be known, the focus is on learning the analysis. Compared to learning the analysis from a dataset of inputs and outputs of DA runs, the training is numerically challenging since the analysis operator is learned through several DA cycles and from trajectories of the dynamics and its observations only.
The resulting DANs are robust, in the sense that they do not require inflation or any other correction and regularisation. They can operate without an ensemble and only use the forecast state as prior information. And yet, they are as accurate as a well tuned EnKF in mildly nonlinear regime, and as accurate as a well tuned IEnKF from mildly to strongly nonlinear conditions.
Abilities of data assimilation networks
We have previously shown that, to achieve this level of performance, DAN must implicitly learn a map xf↦Pa(xf), at least in the mildly nonlinear regime. Its existence is supported by a multiplicative ergodic theorem applied to the entire DA process viewed as a dynamical system. The network must implicitly identify spatial local patterns in xf in order to internally represent components of Pf(xf) or Pa(xf), a property that makes the learned DA method scalable.
In this paper, we further examined the reliance of DANs on the map xf↦Pa(xf) by constructing an ad hoc learnable DAN that explicitly incorporates this mapping. For the L96 dynamics, we identified an average characteristic local pattern by leveraging both the invariant distribution of the dynamics and the translational symmetry of the model. We also showed that DAN can learn an effective nonlinear analysis response consistent with non-Gaussian prior information inferred from xf, and that this nonlinear analysis is necessary for DAN to match the performance of the IEnKF under more stringent nonlinear conditions. In this regime, we demonstrated that implicit or explicit knowledge of the mapping alone is insufficient. Thus, both mechanisms must operate within DAN, paralleling IEnKF's critical reliance on its ensemble for flow-dependent error estimation and on its Gauss-Newton iterative solver to probe departures of the dynamics from linearity.
Implication for the end-to-end processors in numerical weather prediction
Taken in the context of past DA literature, these results are somewhat surprising. For a long time, it was believed that an ensemble was essential for estimating flow-dependent errors. Moreover, the proper construction of non-Gaussian priors has remained a long-standing challenge, whereas even a simple DAN can rapidly learn an effective nonlinear analysis response consistent with such non-Gaussian information. These findings should inform the (re-)design of future DA algorithms.
These considerations have implications for what DAN-like operators are capable of achieving. For example, recently developed end-to-end atmospheric processors that ingest observations and predict future observations, implicitly construct their own internal representation of the system and repeatedly compare this latent state to newly assimilated observations. In doing so, they implicitly learn an analysis operator in their latent space. concluded that their processor, GraphDOP, must be able to learn not only a climatological background but also dynamical priors, a result that may seem surprising given that GraphDOP does not rely on any explicit background information. However, in light of our results, GraphDOP can be interpreted as learning a mapping from the 12 h observational snapshot used as input to the processor to a latent representation that contains information analogous to underlying error covariances. Consequently, it must be able to construct its own background, incorporating advanced error statistics that go beyond a mere climatology. This interpretation is consistent with the diagnostic study of , who applied classical DA tools, including forecast-sensitivity observation-impact diagnostics, to GraphDOP. Their results suggest that GraphDOP learns physically meaningful spatial relationships and latent Earth-system representations from observations. While this does not show that GraphDOP explicitly estimates a forecast-error covariance, it supports the broader view that end-to-end observation-driven processors can develop DA-like internal representations without being supplied with an explicit background state.
This challenges the view that the accuracy of such end-to-end processors is fundamentally limited by the absence of a background (such as a forecast ensemble or climatological information) in their inputs. While an explicit background representation would certainly provide additional information, the extent of the achievable performance gains remains a subtler question.
Ensembles, uncertainty quantification, and multiplicative ergodic theorem
Our results show that, in the perfect-model filtering setting considered here, an explicit forecast ensemble is not strictly required for a learned analysis operator to extract the flow-dependent information needed to achieve EnKF-like point-estimation accuracy. This conclusion should not be interpreted as a dismissal of ensemble representations in general. Ensembles remain essential for probabilistic forecasting, uncertainty quantification, the diagnosis of model and observation errors, smoothing, risk-sensitive applications, and regimes in which the posterior distribution is strongly non-Gaussian or multi-modal.
Moreover, the deterministic analysis operator aθ could itself be used as a building block for ensemble generation. For instance, applying aθ to perturbed innovations would produce a set of analysis states, in a way that is consistent with the local, innovation-dependent interpretation used in, e.g., Appendix to obtain the linear regression approximation of aθ with respect to ζ. Such an analysis ensemble could then be propagated by the forecast model to estimate forecast uncertainty. We have not pursued this probabilistic use of DAN in the present study, since our focus was on point-estimation accuracy, but it represents a natural direction for future work.
At a more theoretical level, the success of DAN may be facilitated by the existence, in the present setting, of a sufficiently regular and local dependence of the relevant flow-dependent analysis correction on the forecast state. This should not be taken for granted in general. For example, the multiplicative ergodic theorem ensures that covariant Lyapunov subspaces are defined as measurable functions of the state, for almost every state on the attractor, but it does not guarantee that this state-to-Oseledets-splitting map is continuous, local, or easily learnable from finite data. Other applications beyond DA, such as learning covariant Lyapunov vectors directly from the state, may therefore involve maps with poorer regularity or locality, making them substantially more difficult to learn and possibly less scalable. This question is left for future work.
Neural network architecture
As described in Boc24, we choose for aθ a basic residual convolutional neural network (CNN) architecture. A schematic of the CNN architecture is displayed in Fig. . It begins with an initial convolution that takes Ne+1 channels as inputs and, with Nf filters, outputs Nf channels. This initial layer is followed by Nb residual blocks. Each one of these blocks is a succession of Nsb sub-blocks. Each subblock is made of: (i) a convolutional layer with Nf channels as inputs, which has Nf filters and a kernel size fv for each of its filter, (ii) a batch normalisation layer, and (iii) an activation function chosen to be Mish . The CNN ends with a final convolutional layer that takes Nf channels as inputs and, with Ne filters, outputs Ne channels. The kernel size of the initial and final channels is fv. Hence, the internal state of the CNN consists of Nf copies of the latent space which we simply choose to be isomorphic to the state space RNx. In this paper, only Ne=1 is used. For all experiments, the internal architecture parameter values are Nb=5, Nsb=5, fv=5.
Architecture of the residual convolutional network where, for this specific illustration, Nb=2, Nsb=3. convN1,N2,f is a generic one-dimensional convolutional layer of dimension N1, with N2 filters of kernel size f. See text for more details.
Sensitivity to the batch size, the size of the datasets, and the backpropagation truncation
The key parameters in the design of aθ and especially in its training are:
the number of channels/filters Nf processed by the convolutional neural network (see Appendix ). This essentially gives away the complexity of the neural network, how many features it can identify and process.
the number of trajectory Nr processed in parallel. The larger Nr, the more DA runs the neural network can learn from and be made robust against. 10% of these are reserved for validation.
the number of cycles Niter through which the gradient is computed. This corresponds to the limit imposed by the truncated backpropagation through time. The larger Niter≪Nc, the better the information transmission from one cycle to the next should be learned, but the more costly and less accurate the gradients.
the size of the batch Sb. Hence, the number of steps in each epoch is about Nr/Sb (training and validation).
The training parameters Nr, Niter, and to a lesser extend Nc are the key dimensions specifying how the data are fed to the training schedule as schematised in Fig. .
This schematic describes how data are fed to the training scheme; it must be read from left to right (time arrow). For instance, in epoch 2, Niter-long segments of Nr trajectories feed the training algorithm. The segments are organised and provided in batches of size Sb.
The dependence of the performance of aθ on those parameters was studied in Boc24. The values Niter=16, Nr=218, Sb=2048 were chosen for the aθ hyperparameters of the reference setup, as a compromise between training speed and accuracy of the resulting aθ. However, the dependence on Niter and Sb was barely reported and discussed, so that we focus on them in what follows.
Using the reference DA setup, we compute the test RMSE of the DAN schemes as a function of the truncation number Niter. The results are shown in Fig. . As expected, Niter≤5 prevents DAN from learning an efficient prior that relies on the errors of the day. It is however remarkable that the improvement in the test RMSE as Niter increases is noticeable up to about Niter≃40, which corresponds to about 6 times the doubling time of the L96 model in the reference setup.
Test RMSE of DAN as a function of the truncation cycles number Niter=1,…,40 in the truncated backpropagation. For each value of Nc, an ensemble of 5aθ operators is learned. Plus and minus one standard deviation of the RMSE are displayed as shades around the RMSE curve.
We have also experimented with the batch size Sb much more thoroughly than in Boc24. We found that using smaller batches is beneficial to the performance of the trained aθ. This may also enable reducing the number of trajectories Nr in the dataset. Nonetheless, we empirically found that the number of steps, i.e. ∼Nr/Sb in each epoch still needs to remain large to achieve high accuracy. Experimenting, we learned a large set of aθ operators with Nr in the range 210–218, while Sb is chosen in the range 23–28. The corresponding test RMSEs are plotted as a function of either Sb, as shown in Fig. a, or Nr as shown in Fig. b.
Test RMSEs of DAN as a function of the batch size Sb for a selection of Nr values (panel a) and as a function of the number of dataset trajectories Nr for a selection of Sb values (panel b). For each pair (Sb,Nr), an ensemble of 3aθ operators is learned, from which error bars (plus or minus the standard deviation) are estimated and added to the curve plots.
For this specific configuration a scaling law can be numerically estimated. The following Ansatz is assumed:
RMSE1/γ=fNrαSbβ,
where f is a polynomial of order 2. This Ansatz (3 polynomial coefficients and 3 exponents) is fitted to the test RMSE results. The fit relevance can be visualised through the plot of fγ in Fig. . The fitted exponents are γ=1.435, α=0.103, and β=0.060. What really matters is the ratio ρ=β/α=0.58, since the test RMSE turns out to show a strong dependence on Nr/Sbρ.
Checking the relevance of the functional regression by plotting the fit function fγ with respect to Nrα/Sbβ (curve), and of the test RMSE results (dots).
In practice, the interest of using smaller batches Sb and hence smaller Nr must be weighted against the overheads created by the dataset pipeline but also by the transfer of the batches from RAM to VRAM. It is however possible to run several training experiments in parallel over the same GPU with smaller batches. Hence, the efficiency of using mini-batches is very dependent on the accelerator device(s) and the type of experiment to conduct. Note that, when training aθ on much higher dimensional systems, relying on mini-batches may become mandatory so as to fit into VRAM.
Finally, the scaling law Eq. () can be leveraged to reduce both Nr and Sb and still be able to extrapolate to values yielding better test RMSEs, using the dependence on Nr/Sbρ. This scaling could nonetheless change with a different DA setup.
Neural network implementation of the effective <inline-formula><mml:math id="M520" display="inline"><mml:mrow><mml:msup><mml:mi mathvariant="bold">x</mml:mi><mml:mi mathvariant="normal">f</mml:mi></mml:msup><mml:mo>↦</mml:mo><mml:msup><mml:mi mathvariant="bold">P</mml:mi><mml:mi mathvariant="normal">a</mml:mi></mml:msup><mml:mo>(</mml:mo><mml:msup><mml:mi mathvariant="bold">x</mml:mi><mml:mi mathvariant="normal">f</mml:mi></mml:msup><mml:mo>)</mml:mo></mml:mrow></mml:math></inline-formula> map
Here, we detail how to build the linear-in-ζ DAN meant to enforce Eq. () in Sect. . Implementing the map (xf,ζ)↦Pa(xf)⋅ζ as a neural network is non-trivial if we wish to make it scalable. First, Pa(xf) is symmetric positive definite; to enforce such constraint, the usual most efficient approach is to write: Pa(xf)=Xa(xf)Xa(xf)⊺, where Xa(xf) is a matrix of anomalies which may be easier to interpret than Pa(xf). The first difficulty towards scalability is the fact that Pa(xf), or Xa(xf), is of size Nx×Nx, which is considered non-scalable. However, if Pa(xf) can be approximated as low-rank, then Pa(xf)≈Xa(xf)Xa(xf)⊺ with Xa(xf) of size Nx×Nr, with Nr remaining small enough when Nx is increased. Unfortunately, Pa(xf) should not realistically be considered low-rank. However, one can exploit the locality of the covariances and write Pa(xf)≈ρ∘Xa(xf)Xa(xf)⊺, where ρ is the localisation correlation matrix and ∘ is the Schur/Hadamard product. In addition to making Pa(xf) full rank in spite of a manageable number of parameters in Xa(xf), it also tapers spurious correlations that could be learned in the course of the training. Indeed, the implementation of the localisation significantly accelerates the training. This is reminiscent of the proposal by to estimate Xa through a loss involving a Schur product with ρ.
Furthermore, given Pa=ρ∘XaXa⊺ and the projected innovation ζ, the implementation of such mapping can be efficiently coded using see, e.g.,Pa⋅ζ=ρ∘XaXa⊺⋅ζ=∑l=1NrXla∘ρ⋅Xla∘ζ,
where the Xla are the columns of Xa, and ⋅ denotes the usual matrix/vector multiplication. The matrix multiplication by the localisation matrix ρ is scalable since ρ is assumed to be a banded matrix. With L96 in mind, i.e. in a one-dimensional context, with a localisation matrix support of (band-)width Nl, the numerical complexity of Pa⋅ζ is NrNx(2+Nl).
If localisation is not useful and Pa(xf) is low-rank, then it is easy to implement Pa⋅ζ=Xa⋅Xa⊺⋅ζ, which is reminiscent of the very popular machine learning attention mechanism. The numerical complexity is then 2NrNx.
An alternative linear-in-ζ DAN is to implement an hypernetwork which would map xf to a set of weights and biases of a linear neural network that would then be applied to ζ. Naively, the number of weights and biases could scale like Nx2. However, we can instead map xf to weights and biases of a sequence of convolutional neural networks that we later apply to ζ. Yet, we did not test this more sophisticated construction since the former approach is scalable and successful.
Here, we offer a proof of the Stein lemma in the degenerate case where the Gaussian density is singular within the embedding space. This is useful with sparse observations resulting in ζ confined within a subspace of Ex. This subsumes the full-rank case. Let us assume that ρ is the pdf defined over Ex of the Gaussian random vector ζ, whose mean is ζ‾ and whose covariance matrix is Σρ which is assumed positive semi-definite. Hence, the support of ζ may be singular in Ex. That is why we resort to the singular value decomposition Σρ=UΣλU⊺, where U is an orthonormal (though not necessarily orthogonal) matrix such that U⊺U=Ix, and Σλ is a positive definite diagonal matrix of rank lower or equal to Nx. We can parametrise the random vector ζ by
ζ(ω)=ζ‾+Uω,
where the random vector ω has λ(ω)=n(0,Σλ) for Gaussian pdf. Then, denoting f(ζ)=Δaθ(x,ζ) for brevity, we have
Covζ∼ρζ,f(ζ)D2a=Covω∼λζ(ω),f(ζ(ω))D2b=∫dωλ(ω)(ζ(ω)-ζ‾)⊗f(ζ(ω))-f(ζ‾)D2c=∫dωλ(ω)(Uω)⊗f(ζ(ω))D2d=UΣλ∫dωλ(ω)(Σλ-1ω)⊗f(ζ(ω))D2e=-UΣλ∫dωλ(ω)∇ωlnλ(ω)⊗f(ζ(ω))D2f=-UΣλ∫dω∇ωλ(ω)⊗f(ζ(ω))D2g=UΣλ∫dωλ(ω)∇ωf(ζ(ω))D2h=UΣλU⊺∫dωλ(ω)(∇ζf)(ζ(ω))D2i=ΣρEω∼λ(∇ζf)(ζ(ω))D2j=ΣρEζ∼ρ∇ζf.
Hence, we conclude:
Eζ∼ρ∇ζf=Σρ†Covζ∼ρζ,f(ζ),
where (⋅)† is the Moore-Penrose inverse operator, which comes with the regularisation choice to taper Eζ∼ρ∇ζf outside of the range of ζ. Applied to f(ζ)=aθ(x,ζ), this yields:
Eζ∼ρ∇ζaθ(x,ζ)=Σρ†Covζ∼ρζ,aθ(x,ζ),
which, if Σρ is full rank, can be written (usual Stein lemma)
Eζ∼ρ∇ζaθ(x,ζ)=Σρ-1Covζ∼ρζ,aθ(x,ζ).
Comparing the empirical and theoretical mean marginal gains
In Sect. , we showed that the theoretical mean marginal gain Γ‾ is expected to be a good approximation of the less simple theoretical mean
Γ̃=∫dxdζπ(x)ρ(ζ)∇x∇ζaθ(x,ζ).
The latter can now be related to the empirical mean of Γ over a long DA run, denoted Γ^, and which is obtained from numerical experiments. This is meant to ensure that a pattern emerging from our approximation of Γ‾, is nonetheless consistent with those learned through aθ over the training DA dataset. Hence, given a long trajectory of true states and projected innovations Tx,ζ={(xkt,ζk)}k=1,…,K⊂(Ex×Ex)K, the empirical sensitivity associated to Γ̃, and hence Γ‾, should be
E2aΓ^=Δ〈Γ(xt,ζ)〉(xt,ζ)∈Tx,ζE2b=K→∞∫dxtdζp(xt,ζ)Γ(xt,ζ),
where p(xt,ζ) is the joint distribution of xt and ζ. However, ζk not only depends on xkt but also on the forecast xkf, so that:
Γ^=∫dxtdxfdζp(xt,xf,ζ)Γ(xt,ζ),
or, introducing the forecast error ef=Δxf-xt,
Γ^=∫dxtdefdζp(xt,ef,ζ)Γ(xt+ef,ζ).
By marginalising over xt, we have
E5aΓ^=∫dxtπ(xt)∫defdζp(ef,ζ|xt)Γ(xt+ef,ζ)E5b=∫dxtπ(xt)∫defdζp(ζ|ef)p(ef|xt)Γ(xt+ef,ζ)E5c≈∫dxtπ(xt)∫dζ[p(ζ|ef)Γ(xt+ef,ζ)]ef=0E5d=∫dxtπ(xt)∫dζρ(ζ)Γ(xt,ζ)=Γ̃.
From Eq. () to Eq. (), we use p(ef,ζ|xt)=p(ζ|ef)p(ef|xt) since the full dependence of ζ on xt is in ef. From Eq. () to Eq. (), we assume that the forecast error norm is small compared to the norm of xt, which should indeed be the case in the weak assimilation regime. From Eq. () to Eq. (), we approximate p(ζ|ef=0) by the marginal ρ(ζ). This points to the (reasonable) approximations made when identifying Γ̃, Eq. (), with the empirical marginal gain Γ^, 〈Γ(x,ζ)〉(x,ζ)∈Tx,ζ, emerging from a long DA run.
Moreover, a formal expression for ρ(ζ|ef) which appeared in the previous derivation is
p(ζ|ef)=∫dεdHdRp(ε,H,R|ef)E6×δ(ζ-H⊺R-1(ε-Hef)),
where p(ε,H,R|ef) is the pdf of the joint distribution for the observation error ε, the observation operator H, and the observation error covariance matrix R, given the forecast error ef. Marginalising over ef, we obtain
E7aρ(ζ)=∫defp(ζ|ef)p(ef)=∫defdεdHdRp(ef,ε,H,R)E7b×δ(ζ-H⊺R-1(ε-Hef)).
This expression is helpful to formally investigate the symmetries of the distribution of ζ, for which Eq. () applies.
Proof of the equivariance of the marginal gain tensor
We wish to prove the equivariance of the marginal gain tensor, i.e. that for all x and ζ, Γ(g∘x,g∘ζ)=g⊗g⊺⊗g⊺∘Γ(x,ζ), under the action of an isometry g∈G. The action of g on either state vector x or ζ is represented by an orthogonal matrix G: g∘x=Gx, g∘ζ=Gζ.
We say that g preserves the fibres (preimages) of Hk if, for all x, x′ such that Hk(x)=Hk(x′) one has Hk(g∘x)=Hk(g∘x′). An isometry gy defined over the observation space of Hk can then be associated to such g through gy∘Hk(g⊺∘(⋅))=Hk(⋅). The action of such induced isometry gy is represented by the orthogonal matrix Gy: gy∘y=Gyy. Because G and Gy are orthogonal, one has G-1=G⊺ and Gy-1=Gy⊺.
With these definitions in hand, we consider a maximal group of isometries G defined over the state space such that
G preserves the fibres of all Hk, inducing a group of isometries Gy (possibly for each k). Gy is isomorphic to G,
the associated Gy satisfies GyRkGy⊺=Rk, which accounts for any heteroscedasticity of the observation error statistics,
G commutes with the autonomous dynamics, i.e. GM(G⊺(⋅))=M(⋅).
A consequence of the equivariance with respect to Hk, i.e. GyHkG⊺(⋅)=Hk(⋅), is GyHkG⊺=Hk, readily obtained by differentiation.
A simplifying classical trick would be to consider that the physical system is fully observed at any time, with observation error variances that can take infinite values in the absence of observations, which would shift the observation equivariance constraint onto that of its statistics.
Coming back to the proof of the equivariance we first consider the optimisation problem that defines (x,ζ)↦aθ(g∘x,g∘ζ) for any g in such G:
F1aL(θ|{xkt,yk})=∑k=1Kxkt-xka(θ)2,F1bxka=xkf+aθ(Gxkf,Gζk),F1cζk=Hk⊺Rk-1(yk-Hk(xkf)),F1dxk+1f=M(xka),
which is equivalent to
F2aL(θ|{xkt,yk})=∑k=1KGxkt-Gxka(θ)2,F2bGxka=Gxkf+Gaθ(Gxkf,Gζk),Gζk=GHk⊺Gy⊺GyRk-1Gy⊺GyF2c×(yk-Hk(G⊺Gxkf)),F2dGxk+1f=GM(G⊺Gxka),
where Eq. () is obtained from Eq. () because G is orthogonal, and Eqs. (, , ) are obtained from a multiplication on the left by G and insertion of G⊺G=Ix and Gy⊺Gy=Iy in Eqs. (, , ). Hence, denoting x̃ka=Gxka, x̃kf=Gxkf, ζ̃k=Gζk, the problem is reformulated as
F3aL(θ|{xkt,yk})=∑k=1KGxkt-x̃ka(θ)2,F3bx̃ka=x̃kf+Gaθ(x̃kf,ζ̃k),ζ̃k=GyHkG⊺⊺(GyRk-1Gy⊺)F3c×(Gyyk-GyHk(G⊺x̃kf)),F3dx̃k+1f=GM(G-1x̃ka).
Leveraging the assumptions on G, we finally obtain
F4aL(θ|xkt,yk)=∑k=1KGxkt-x̃ka(θ)2,F4bx̃ka=x̃kf+Gaθ(x̃kf,ζ̃k),F4cζ̃k=Hk⊺Rk-1(Gyyk-Hk(x̃kf)),F4dx̃k+1f=M(x̃ka).
This shows that aθ(G⋅,G⋅) is the solution of Eq. (F1) whose input is the dataset xkt,yk, while Gaθ(⋅,⋅) is the solution of Eq. (F4) whose input is the dataset Gxkt,Gyyk=GyHk(xkt)+Gyεk=Hk(Gxkt)+Gyεk. Since both the invariant distribution of the dynamics and the distribution of the observations errors are invariant under G, the datasets xkt,yk and Gxkt,Gyyk must asymptotically yield the same solution for a large enough number of samples K. This proves the equivariance, ∀g∈G,∀x∈Ex,∀ζ∈Ex:
aθ(g∘x,g∘ζ)=g∘aθ(x,ζ).
Then, taking the gradient with respect to x and ζ of aθ(g∘x,g∘ζ) yields a covariant action g⊗g onto the tensor factors for x and ζ, alternatively a contravariant action g⊺⊗g⊺ on to the right-hand side of Eq. (), which proves the equivariance of Γ, Eq. (). The equivariance assumption on the observation operator is rather stringent. A weaker assumption is to assume that GyHk(G⊺xkf)k=1,…,K almost spans the same set as Hk(xkf)k=1,…,K for K→∞. Then the optimisation problems Eq. (F1) and Eq. (F4) should almost coincide. This is for instance useful when one considers random observation operators for which operator instances have no specific symmetry, while their distribution does exhibit the symmetry, a case occurring in Sect. .
Sleek representation of the mean marginal gain
Assume that the states and projected innovations are defined as fields over a physical manifold D, and further discretised at Nx collocation space points of D indexed by i∈⟦1,…,Nx⟧. Hence, G is a discrete group of isometries. As a consequence, g∈G can be seen as a bijection of ⟦1,…,Nx⟧ and the action of g∈G on the fields x and ζ reads
[g∘x]i=xg(i),[g∘ζ]j=[ζ]g(j),
respectively. Likewise, the action of g∈G on Γ‾ is, for all i,j,l:
g⊙Γ‾ijl=g⊗g⊺⊗g⊺∘Γ‾ijl=Γ‾g(i)g⊺(j)g⊺(l),
so that Eq. () reads, for all i,j,l:
Γ‾ijl=Γ‾g(i)g⊺(j)g⊺(l).
Let us choose one of the collocation points in D with index r∈⟦1,…,Nx⟧. With the above assumptions, the orbit of the site indexed by r under the action of G is ⟦1,…,Nx⟧. We can then define a 2-tensor Ω‾(r) componentwise by, for all i,j:
Ω‾ij(r)=Γ‾ijr.
For all l∈⟦1,…,Nx⟧, we can pick at least one glr∈G such that glr(r)=l and let us denote its inverse by grl which coincides with its adjoint g⊺ and satisfies grl(l)=r in particular. Hence, we have from Eq. () and for all i,j,l:
Γ‾ijl=Γ‾glr(i)grl(j)grl(l)=Γ‾glr(i)grl(j)r=Ω‾glr(i)grl(j)(r).
As a consequence of the symmetry and Eq. (), the 3-tensor Γ‾ can be entirely specified by the 2-tensor Ω‾(r), where r, a reference site index, is arbitrarily chosen. In the case where D is one-dimensional (as for L96), Ω‾(r) is a matrix, hence depictable and more easily interpretable.
Numerical computation of the mean marginal gain
The mean marginal gain can be computed from states of a trajectory Tx of the ergodic dynamics, and the ability to evaluate x↦Γ(x) as defined by Eq. (). The trajectory should be long enough so that its states adequately sample the invariant distribution π. From Eq. (), we hence have the empirical estimator:
Γ‾=〈Γ(x)〉x∈Tx=1K∑k=1KΓ(xk).
As a result, the computational complexity of the mean marginal gain is proportional to K, but may be significantly alleviated by the presence of symmetries as discussed before. Such symmetries must make both π and ρ invariant even though the definition of Γ‾ only implicitly depends on ρ.
We now turn to the estimation of the marginal gain Γ(x). Its computation can be achieved through several routes with distinct numerical complexities which, as approximations, may not be equivalent and may lead to mildly differing results.
The first way to compute Γ(x) is through automatic differentiation. As a second-order sensitivity of aθ with respect to x and ζ, Γ(x)=∇x∇ζaθ(x,ζ)|ζ=0 requires taking the Jacobian of aθ twice. Hence, such computation through either JAX, PyTorch or TensorFlow, can be prohibitive, with a substantial need for GPU memory. On an NVIDIA RTX5000 Ada GPU with 32 GB of memory, using the JAX-inspired PyTorch torch.func module (https://pytorch.org/docs/stable/func.html, last access: 7 August 2026), we found it to be achievable with the L96 model, difficult with a Kuramoto-Sivashinsky model , but prohibitive with a single-layer QG model on the sphere. Hence, it is likely to be impractical with high-dimensional models.
Note that the mean Eq. () can be computed through updates whenever a new Γ(xk) is computed, preventing the need to store them. Moreover, when exploiting symmetries of G, the intermediate tensor
Ωij(r)(xk)=1Nx∑l=1NxΓgrl(i)glr(j)l(xk),
can be computed, and will contribute to the computation of Ω‾(r) through the update of
Ω‾(r)=〈Ω(r)(x)〉x∈Tx=1K∑k=1KΩ(r)(xk).
In Boc24, either the gain K(x) or Pa(x) were obtained by generating an ensemble of perturbations to feed a regression. The same idea can be used for Γ(x), once again assuming a quasi-linear behaviour of aθ(x,ζ) and ∇xaθ(x,ζ) as functions of ζ. From the results in Sect. and Appendix , we infer that
H3aΓ(x)=∇x∇ζaθ(x,ζ)|ζ=0H3b≈Eζ∼ρ∇x∇ζaθ(x,ζ)H3c≈Σρ†Covζ∼ρζ,∇xaθ(x,ζ),
which tells that a sampling approach can be applied to ∇xaθ(x,⋅). Assuming aθ is regular enough, the gradients with respect to x and ζ commute and we also have Γ(x)≈∇xEζ∼ρ∇ζaθ(x,ζ), although this could considerably complexify backpropagation if automatic differentiation is used to handle ∇x.
Automatic differentiation is hence used only once for the computation of the Jacobian ∇xaθ(x,⋅), as opposed to the full automatic differentiation approach. Hence, Γ(x) can be computed using a composite Monte Carlo/differentiation approach. The details of the subsequent regression are reported in Appendix .
Regression for the composite mean marginal gain
An ensemble of Np perturbations ∂ap generated from Np samples ζp∼N(0,Ξ) for p=1,…,Np should first be computed,
∂apil=∂xlaθ(x,ζp)i,
via an ensemble of first-order Jacobians. In the best linear unbiased estimator framework, the covariance matrix Ξ should roughly match H⊺R-1(R+HPfH⊺)R-1H where Pf and R+HPfH⊺ are the forecast error and innovation covariance matrices, respectively. Hence, in the weak assimilation regime, we can use the approximation Ξ≈H⊺R-1H, which should be regarded as a scale for the perturbations anyway, and generate samples with ζ=H⊺R-12ξ, where ξ∼N(0,Iy). Introducing for p=1,…,Np, the recentred samples
∂ap′=∂ap-Np-1∑p=1Np∂ap,ζp′=ζp-Np-1∑p=1Npζp,
we have from the definitions C=ΔCovζ∼ρζ,∇xaθ(x,ζ) and D=ΔΣρ:
Cijl≈∑p=1Np[ζp′]j[∂ap′]il,Djl≈∑p=1Np[ζp′]j[ζp′]l,
which, from Eq. (), yields
Γijl=∑m=1NxCiml[D-1]mj.
Obviously, in a high-dimensional context, the approach would necessitate reduction methods such as Lanczos vectors or (randomised) singular value decompositions, and the generation of the ensemble would require a massive vectorisation on GPUs. Ωr can then be computed by averaging Γ using, e.g., Eqs. (H2). Moreover, it is not difficult to show that Ωr, as defined by Eq. (), can alternatively be obtained by first averaging over C and D before performing the inversion of the regression, that is:
I5aΩr=C‾D‾-1,I5b[C‾]ijl=1Nx∑s=1NxCgsl(i)gls(j)s,I5c[D‾]ij=1Nx∑s=1NxDgsj(i)s.
Either way, the composite approach may turn out numerically cheaper than the full differentiation approach.
Code and data availability
The core source code for training and testing one-dimensional DANs is publicly available under https://github.com/cerea-daml/Dan1D and Zenodo (10.5281/zenodo.21427793, ).
Author contributions
MB performed the mathematical, algorithmic, and numerical analysis. MB and TSF discussed the implications of the results. All the authors worked on the structure of the manuscript. All the authors reviewed and edited the manuscript.
Competing interests
The contact author has declared that none of the authors has any competing interests.
Disclaimer
Publisher's note: Copernicus Publications remains neutral with regard to jurisdictional claims made in the text, published maps, institutional affiliations, or any other geographical representation in this paper. The authors bear the ultimate responsibility for providing appropriate place names. Views expressed in the text are those of the authors and do not necessarily reflect the views of the publisher.
Acknowledgements
The authors thank an anonymous Reviewer and Patrick N. Raanes for their sharp and inspiring comments and suggestions which benefited the paper. AI was used to polish the English of a limited number of paragraphs of the manuscript, and for a thorough check of notation consistency. CEREA is a member of Institut Pierre-Simon Laplace (IPSL).
Financial support
This project – Learning efficient Data Assimilation from Artificial Intelligence (DAbyAI) – has been supported by NVIDIA and their Academic Grant Program through the grant of two RTX 6000 Ada GPUs that were intensively used in the numerical experiments of this work. This paper is also a contribution to the DRUIDS project, supported by France 2030 PEPR Maths-Vives, grant ANR-24-EXMA-0002. Tobias S. Finn acknowledges the support of the project SASIP (grant no. G-24-66154) funded by Schmidt Sciences – a philanthropic initiative that seeks to improve societal outcomes through the development of emerging science and technologies, and the support of France 2030 PEPR Maths-Vives, grant no. ANR-24-EXMA-0001, project Climaths/GenClimEx. Sibo Cheng acknowledges the support of the French Agence Nationale de la Recherche (ANR) under reference ANR-22-CPJ2-0143-01.
Review statement
This paper was edited by Natale Alberto Carrassi and reviewed by Patrick N. Raanes and one anonymous referee.
ReferencesAgarwal, S., Jabbari, S., Agarwal, C., Upadhyay, S., Wu, S., and Lakkaraju, H.: Towards the Unification and Robustness of Perturbation and Gradient Based Explanations, in: Proceedings of the 38th International Conference on Machine Learning, vol. 139, edited by Meila, M. and Zhang, T., Proceedings of Machine Learning Research, PMLR, 110–119, https://proceedings.mlr.press/v139/agarwal21c.html (last access: 7 August 2026), 2021.Allen, A., Markou, S., Tebbutt, W., Requeima, J., Bruinsma, W. P., Andersson, T. R., Herzog, M., Lane, N. D., Chantry, M., Hosking, J. S., and Turner, R. E.: End-to-end data-driven weather prediction, Nature, 10.1038/s41586-025-08897-0, 2025.Arnold, L.: Random Dynamical Systems, Springer Berlin, Heidelberg, 10.1007/978-3-662-12878-7, 1998.Asch, M., Bocquet, M., and Nodet, M.: Data Assimilation: Methods, Algorithms, and Applications, Fundamentals of Algorithms, SIAM, Philadelphia, 10.1137/1.9781611974546, 2016.Bannister, R. N.: A review of operational methods of variational and ensemble-variational data assimilation, Q. J. Roy. Meteor. Soc., 143, 607–633, 10.1002/qj.2982, 2017.Barone, A., Carrassi, A., Savary, T., Demaeyer, J., and Vannitsem, S.: Structural origins and real-time predictors of intermittency, Chaos, 35, 103119, 10.1063/5.0287572, 2025.Bocquet, M.: Ensemble Kalman filtering without the intrinsic need for inflation, Nonlin. Processes Geophys., 18, 735–750, 10.5194/npg-18-735-2011, 2011.Bocquet, M.: Dan1D: One-Dimensional Data Assimilation Network [Computer software], Zenodo [code], 10.5281/zenodo.21427793, 2026 (code also available at: https://github.com/cerea-daml/Dan1D, last access: 7 August 2026).Bocquet, M. and Carrassi, A.: Four-dimensional ensemble variational data assimilation and the unstable subspace, Tellus A, 69, 1304504, 10.1080/16000870.2017.1304504, 2017.Bocquet, M. and Farchi, A.: On the consistency of the perturbation update of local ensemble square root Kalman filters, Tellus A, 71, 1–21, 10.1080/16000870.2019.1613142, 2019.Bocquet, M. and Sakov, P.: Combining inflation-free and iterative ensemble Kalman filters for strongly nonlinear systems, Nonlin. Processes Geophys., 19, 383–399, 10.5194/npg-19-383-2012, 2012.Bocquet, M. and Sakov, P.: Joint state and parameter estimation with an iterative ensemble Kalman smoother, Nonlin. Processes Geophys., 20, 803–818, 10.5194/npg-20-803-2013, 2013.Bocquet, M. and Sakov, P.: An iterative ensemble Kalman smoother, Q. J. Roy. Meteor. Soc., 140, 1521–1535, 10.1002/qj.2236, 2014.Bocquet, M., Raanes, P. N., and Hannart, A.: Expanding the validity of the ensemble Kalman filter without the intrinsic need for inflation, Nonlin. Processes Geophys., 22, 645–662, 10.5194/npg-22-645-2015, 2015.Bocquet, M., Gurumoorthy, K. S., Apte, A., Carrassi, A., Grudzien, C., and Jones, C. K. R. T.: Degenerate Kalman filter error covariances and their convergence onto the unstable subspace, SIAM/ASA J. Uncertain. Quantif., 5, 304–333, 10.1137/16M1068712, 2017.Bocquet, M., Farchi, A., Finn, T. S., Durand, C., Cheng, S., Chen, Y., Pasmans, I., and Carrassi, A.: Accurate deep learning-based filtering for chaotic dynamics by identifying instabilities without an ensemble, Chaos, 34, 091104, 10.1063/5.0230837, 2024.Boucher, E., Alexe, M., Lean, P., Pinnington, E., Lang, S., Laloyaux, P., Zampieri, L., de Rosnay, P., Bormann, N., and McNally, A.: Learning Coupled Earth System Dynamics with GraphDOP, arXiv [preprint], 10.48550/arXiv.2510.20416, 2025.Boudier, P., Fillion, A., Gratton, S., Gürol, S., and Zhang, S.: Data Assimilation Networks, J. Adv. Model. Earth Sy., 15, e2022MS003353, 10.1029/2022MS003353, 2023.Buehner, M., McTaggart-Cowan, R., Beaulne, A., Charette, C., Garand, L., Heilliette, S., Lapalme, E., Laroche, S., Macpherson, S. R., Morneau, J., and Zadra, A.: Implementation of Deterministic Weather Forecasting Systems based on Ensemble-Variational Data Assimilation at Environment Canada. Part I: The Global System, Mon. Weather Rev., 143, 2532–2559, 10.1175/MWR-D-14-00354.1, 2015.Carrassi, A., Ghil, M., Trevisan, A., and Uboldi, F.: Data assimilation as a nonlinear dynamical systems problem: Stability and convergence of the prediction-assimilation system, Chaos, 18, 023112, 10.1063/1.2909862, 2008.Carrassi, A., Bocquet, M., Bertino, L., and Evensen, G.: Data Assimilation in the Geosciences: An overview on methods, issues, and perspectives, WIREs Clim. Change, 9, e535, 10.1002/wcc.535, 2018.Carrassi, A., Bocquet, M., Demaeyer, J., Gruzien, C., Raanes, P. N., and Vannitsem, S.: Data assimilation for chaotic dynamics, in: Data Assimilation for Atmospheric, Oceanic and Hydrologic Applications, vol. IV, edited by: Seon, K. P. and Liang, X., Springer International Publishing, Cham, 1–42, 10.1007/978-3-030-77722-7_1, 2022.Chekroun, M. D., Simonnet, E., and Ghil, M.: Stochastic climate dynamics: Random attractors and time-dependent invariant measures, Physica D, 240, 1685–1700, 10.1016/j.physd.2011.06.005, 2011.Cheng, S., Quilodran-Casas, C., Ouala, S., Farchi, A., Liu, C., Tandeo, P., Fablet, R., Lucor, D., Iooss, B., Brajard, J., Xiao, D., Janjic, T., Ding, W., Guo, Y., Carrassi, A., Bocquet, M., and Arcucci, R.: Machine learning with data assimilation and uncertainty quantification for dynamical systems: a review, IEEE/CAA J. Autom. Sin., 10, 1361–1387, 10.1109/JAS.2023.123537, 2023.Cintra, R. S. and de Campos Velho, H. F.: Data assimilation by artificial neural networks for an atmospheric general circulation model, Chap. 17, IntechOpen, 265–286, 10.5772/intechopen.70791, 2018. Daley, R.: Atmospheric Data Analysis, Cambridge University Press, New York, ISBN 978-0-521-38215-1, 1991.Desroziers, G., Camino, J.-T., and Berre, L.: 4DEnVar: link with 4D state formulation of variational assimilation and different possible implementations, Q. J. Roy. Meteor. Soc., 140, 2097–2110, 10.1002/qj.2325, 2014.Fablet, R., Chapron, B., Drumetz, L., Mémin, E., Pannekoucke, O., and Rousseau, F.: Learning Variational Data Assimilation Models and Solvers, J. Adv. Model. Earth Sy., 13, e2021MS002572, 10.1029/2021MS002572, 2021.Fillion, A., Bocquet, M., and Gratton, S.: Quasi-static ensemble variational data assimilation: a theoretical and numerical study with the iterative ensemble Kalman smoother, Nonlin. Processes Geophys., 25, 315–334, 10.5194/npg-25-315-2018, 2018.Fillion, A., Bocquet, M., Gratton, S., Gürol, S., and Sakov, P.: An iterative ensemble Kalman smoother in presence of additive model error, SIAM/ASA J. Uncertain. Quantif., 8, 198–228, 10.1137/19M1244147, 2020.Filoche, A., Brajard, J., Charantonis, A., and Béréziat, D.: Learning 4DVAR Inversion Directly from Observations, in: Computational Science – ICCS 2023, edited by: Mikyška, J., de Mulatier, C., Paszynski, M., Krzhizhanovskaya, V. V., Dongarra, J. J., and Sloot, P. M., Springer Nature Switzerland, Cham, 414–421, 10.1007/978-3-031-36027-5_32, 2023.Flandoli, F. and Tonello, E.: An introduction to random dynamical systems for climate, https://pagine.dm.unipi.it/flandoli/Part1bis.pdf (last access: 7 August 2026), 2021.Frerix, T., Kochkov, D., Smith, J., Cremers, D., Brenner, M., and Hoyer, S.: Variational Data Assimilation with a Learned Inverse Observation Operator, in: Proceedings of the 38th International Conference on Machine Learning, edited by Meila, M. and Zhang, T., vol. 139 of Proceedings of Machine Learning Research, PMLR, 3449–3458, https://proceedings.mlr.press/v139/frerix21a.html (last access: 7 August 2026), 2021.Ghil, M. and Sciamarella, D.: Review article: Dynamical systems, algebraic topology and the climate sciences, Nonlin. Processes Geophys., 30, 399–434, 10.5194/npg-30-399-2023, 2023.Härter, T. P. and de Campos Velho, H. F.: Data Assimilation Procedure by Recurrent Neural Network, Eng. Appl. Comp. Fluid, 6, 224–233, 10.1080/19942060.2012.11015417, 2012.Jazwinski, A. H.: Stochastic Processes and Filtering Theory, Academic Press, New York, ISBN 978-0-12-381550-7, 10.1016/S0076-5392(09)60368-4, 1970.Kalnay, E.: Atmospheric Modeling, Data Assimilation and Predictability, Cambridge University Press, Cambridge, ISBN 978-0-521-79629-3, 10.1017/CBO9780511802270, 2003.Karimi, A. and Paul, M. R.: Extensive chaos in the Lorenz-96 model, Chaos, 20, 043105, 10.1063/1.3496397, 2010.Keller, J. D. and Potthast, R.: AI-based data assimilation: Learning the functional of analysis estimation, arXiv [preprint], 10.48550/arXiv.2406.00390, 2024.Kuramoto, Y. and Tsuzuki, T.: Persistent propagation of concentration waves in dissipative media far from thermal equilibrium, Prog. Theor. Phys., 55, 356–369, 10.1143/PTP.55.356, 1976.Lafon, N., Fablet, R., and Naveau, P.: Uncertainty Quantification When Learning Dynamical Models and Solvers With Variational Methods, J. Adv. Model. Earth Sy., 15, e2022MS003446, 10.1029/2022MS003446, 2023.Laloyaux, P., Alexe, M., Boucher, E., Lean, P., Pinnington, E., Lang, S., Necker, T., and McNally, A.: Using data assimilation tools to dissect GraphDOP, arXiv [preprint], 10.48550/arXiv.2510.27388, 2025.Lean, P., Alexe, M., Boucher, E., Pinnington, E., Lang, S., Laloyaux, P., Bormann, N., and McNally, A.: Learning from nature: insights into GraphDOP's representations of the Earth System, arXiv [preprint], 10.48550/arXiv.2508.18018, 2025.Liu, J. S.: Siegel's formula via Stein's identities, Stat. Probabil. Lett., 21, 247–251, 10.1016/0167-7152(94)90121-X, 1994.Lorenz, E. N. and Emanuel, K. A.: Optimal sites for supplementary weather observations: simulation with a small model, J. Atmos. Sci., 55, 399–414, 10.1175/1520-0469(1998)055<0399:OSFSWO>2.0.CO;2, 1998.Lu, F.: U-Net Kalman Filter (UNetKF): An Example of Machine Learning-Assisted Data Assimilation, J. Adv. Model. Earth Sy., 17, e2023MS003979, 10.1029/2023MS003979, 2025.Maddy, E. S., Boukabara, S. A., and Iturbide-Sanchez, F.: Assessing the Feasibility of an NWP Satellite Data Assimilation System Entirely Based on AI Techniques, IEEE J. Sel. Top. Appl., 17, 9828–9845, 10.1109/JSTARS.2024.3397078, 2024.McCabe, M. and Brown, J.: Learning to Assimilate in Chaotic Dynamical Systems, in: Advances in Neural Information Processing Systems, vol. 34, edited by: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W., Curran Associates, Inc., 12237–12250, https://proceedings.neurips.cc/paper_files/paper/2021/file/65cc2c8205a05d7379fa3a6386f710e1-Paper.pdf (last access: 7 August 2026), 2021.Misra, D.: Mish: A Self Regularized Non-Monotonic Neural Activation Function, arXiv [preprint], 10.48550/arXiv.1908.08681, 2019. Oseledec, V. I.: A multiplicative ergodic theorem. Ljapunov characteristic numbers for dynamical systems, Trans. Moscow Math Soc., 19, 197–231, 1968.Palatella, L., Carrassi, A., and Trevisan, A.: Lyapunov vectors and assimilation in the unstable subspace: theory and applications, J. Phys. A-Math. Theor., 46, 254020, 10.1088/1751-8113/46/25/254020, 2013.Pannekoucke, O. and Fablet, R.: PDE-NetGen 1.0: from symbolic partial differential equation (PDE) representations of physical processes to trainable neural network representations, Geosci. Model Dev., 13, 3373–3382, 10.5194/gmd-13-3373-2020, 2020. Pannekoucke, O., Ricci, S., Barthelemy, S., Ménard, R., and Thual, O.: Parametric Kalman filter for chemical transport model, Tellus A, 68, 31457, 10.3402/tellusa.v68.31547, 2016.Pannekoucke, O., Bocquet, M., and Ménard, R.: Parametric covariance dynamics for the nonlinear diffusive Burgers equation, Nonlin. Processes Geophys., 25, 481–495, 10.5194/npg-25-481-2018, 2018.Raanes, P. N., Stordal, A. S., and Evensen, G.: Revising the stochastic iterative ensemble smoother, Nonlin. Processes Geophys., 26, 325–338, 10.5194/npg-26-325-2019, 2019.Ribeiro, M. T., Singh, S., and Guestrin, C.: Why Should I Trust You?: Explaining the Predictions of Any Classifier, in: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '16, Association for Computing Machinery, New York, NY, USA, 1135–1144, 10.1145/2939672.2939778, 2016.Sacco, M. A., Pulido, M., Ruiz, J. J., and Tandeo, P.: On-line machine-learning forecast uncertainty estimation for sequential data assimilation, Q. J. Roy. Meteor. Soc., 150, 2937–2954, 10.1002/qj.4743, 2024.Sakov, P.: On building the state error covariance from a state estimate, arXiv [preprint], 10.48550/arXiv.2411.14809, 2025.Sakov, P., Oliver, D. S., and Bertino, L.: An iterative EnKF for strongly nonlinear systems, Mon. Weather Rev., 140, 1988–2004, 10.1175/MWR-D-11-00176.1, 2012.Sivashinsky, G. I.: Nonlinear analysis of hydrodynamic instability in laminar flames-I. Derivation of basic equations, Acta Astronaut., 4, 1177–1206, 10.1016/0094-5765(77)90096-0, 1977.Smilkov, D., Thorat, N., Kim, B., Viégas, F. B., and Wattenberg, M.: SmoothGrad: removing noise by adding noise, CoRR, arXiv [preprint], 10.48550/arXiv.1706.03825, 2017.Stordal, A. S., Szklarz, S. P., and Leeuwenburgh, O.: A Theoretical look at Ensemble-Based Optimization in Reservoir Management, Math. Geosci., 48, 399–417, 10.1007/s11004-015-9598-6, 2016.Tang, H. and Glass, J.: On Training Recurrent Networks with Truncated Backpropagation Through time in Speech Recognition, in: 2018 IEEE Spoken Language Technology Workshop (SLT), 48–55, 10.1109/SLT.2018.8639517, 2018.van Kekem, D. L. and Sterk, A. E.: Travelling waves and their bifurcations in the Lorenz-96 model, Physica D, 367, 38–60, 10.1016/j.physd.2017.11.008, 2018.Wishner, R. P., Tabaczynski, J. A., and Athans, M.: A Comparison of Three Non-Linear Filters, Automatica, 5, 487–496, 10.1016/0005-1098(69)90110-1, 1969.