The ensemble Kalman filter and its variants have shown to be robust for data assimilation in high dimensional geophysical models, with localization, using ensembles of extremely small size relative to the model dimension. However, a reduced rank representation of the estimated covariance leaves a large dimensional complementary subspace unfiltered. Utilizing the dynamical properties of the filtration for the backward Lyapunov vectors, this paper explores a previously unexplained mechanism, providing a novel theoretical interpretation for the role of covariance inflation in ensemble-based Kalman filters. Our derivation of the forecast error evolution describes the dynamic upwelling of the unfiltered error from outside of the span of the anomalies into the filtered subspace. Analytical results for linear systems explicitly describe the mechanism for the upwelling, and the associated recursive Riccati equation for the forecast error, while nonlinear approximations are explored numerically.

It is well understood that in chaotic physical systems, dynamical instability
is among the leading drivers of forecast uncertainty

Inspired by the “assimilation in the unstable subspace” (AUS) methodology of
Anna Trevisan and her collaborators

The above mathematical studies demonstrate how the stable dynamics in perfect
models dissipate forecast errors, in sequential filters, such that a reduced
rank representation of the error covariance matrix in the unstable–neutral
subspace alone suffices to control error growth. This behavior, similarly
understood in the smoothing problem

The sources of model error are varied and a common simplifying assumption in
data assimilation is that it takes the form of additive, Gaussian noise that
is white in time. The work of

However, the uniform bound may be impractically large due to the excitation of model errors by the transient instabilities in stable directions. While uncertainty is asymptotically dissipated by the stable dynamics, the reintroduction of uncertainty from model error significantly differentiates models with additive errors. Newly injected errors are subject to the growth rates of the local (in time) Lyapunov exponents, and stable Lyapunov exponents of sufficiently high variance may experience transient periods of growth. Therefore, strategies for representing the forecast error with a low rank ensemble must be adapted for imperfect models to account for a residual error in the span of the stable, backward Lyapunov vectors which never vanishes and, moreover, may go through transient periods of growth. As a consequence, confining the error description within a reduced rank Kalman filter to only the unstable–neutral subspace does not suffice when model error is present and suggests that one must include additional, asymptotically stable, modes (Grudzien et al., 2018).

Furthermore, in this current work we show that such an increase of the ensemble span does not automatically render the filter optimal: one may also need to account for the injection of error from unfiltered directions into the ensemble span. In particular, when an ensemble-based Kalman gain is used to correct the forecast errors, the dynamics induce error propagation which transmits uncertainty from the uncorrected, complementary subspace into the ensemble span. In this study, the propagation of error in the linear Kalman filter, written in a basis of backward Lyapunov vectors, will reveal the leading order evolution of the unfiltered uncertainty. Although the evolution is derived for linear models, the mechanism for error propagation can be considered a generic feature of ensemble Kalman filters. Under the condition that error evolution is weakly-nonlinear, the ensemble span will align with the span of the leading backward Lyapunov vectors; therefore, the error decomposition in the basis of backward Lyapunov vectors will be valid for the ensemble Kalman filter.

Similar to how we view AUS as a theoretical framework for understanding the properties of ensemble-based covariances in the presence of chaotic dynamics (and in the absence of model error), this work aims to be used as a theoretical explanation for the empirically observed properties of ensemble-based covariances in the presence of chaotic dynamics and additive model errors, providing a theoretical motivation for the role of covariance inflation in preventing filter divergence. We demonstrate that even when issues of sampling error, truncation errors due to nonlinearity, and misspecification of model and observation error distributions are all excluded, there is an intrinsic deficiency of the standard reduced rank error covariance recursion that leads to systematic underestimation of the forecast errors in the ensemble span. While we believe this provides a new theoretical explanation for the role of covariance inflation in the ensemble Kalman filter, we also discuss possible strategies to obviate inflation with less ad hoc methods that take into account the evolution of unfiltered errors more directly.

This paper is structured as follows: Sect.

We begin by introducing our notation and the problem formulation, including
relevant definitions. There is inconsistent use of the terminology for
Lyapunov vectors in the literature, and so we choose to use the nomenclature
of

Throughout the entire text, the conventional notation

Although much of the derivations that follow are done for linear dynamics, we
are ultimately concerned with nonlinear systems; therefore, we will assume
that Oseledec's theorem holds, even for linear model propagators. In general,
this is a non-trivial assumption, but one which can be considered generic for
the tangent-linear model of a wide class of nonlinear systems, due to the
multiplicative ergodic theorem (MET): with probability one, Oseledec's
theorem holds, the Lyapunov exponents are well defined and the values of the
Lyapunov exponents are independent of the initial condition

We order the Lyapunov exponents

Oseledec's theorem decomposes the (tangent-linear) model space into a direct
sum of time-varying, covariant Oseledec spaces, referred to as an Oseledec
splitting or decomposition. At times, we will refer to the covariant Oseledec
spaces, as well as to the covariant, and to the forward Lyapunov vectors.
These discussions will provide a deeper interpretation of our results for
those familiar with these technical points. However, these discussions are
not crucial to the understanding of our results; therefore, we limit the
use of formal definitions to the backward Lyapunov vectors. For a more formal
discussion of the Oseledec spaces, constructions for Lyapunov vectors and
related results for the full rank Kalman filter see

The backward Lyapunov vectors can be defined by a choice of an orthonormal
eigenbasis for the far past operator, and/or by recursive QR factorizations
of the (tangent-linear) model propagator

The decomposition in Eq. (

We seek to estimate the distribution of a Gaussian random vector

Suppose that

In a linear model, with known Gaussian observation and model error
distributions, the estimated error covariances of the KF are exact: the posterior
error distribution for the state is Gaussian, and the KF completely describes
the Bayesian posterior through its recursive equations for the estimated mean
and covariance. However, it is often the case that the recursion for the
posterior error distribution is approximated with a reduced rank surrogate in
which the estimated covariance,

Nonetheless, it is possible in an ideal setting to analytically describe the
error statistics of a reduced rank Kalman filter – to illustrate this,
assume that we have a linear model with known Gaussian error distributions.
Suppose we apply the analysis update in a reduced rank set of BLVs, as has
been done in extended Kalman filter (EKF)-AUS

The significance of deriving an analytical recursion for the forecast error
under the reduced rank estimator in Eq. (

Consider the forecast error recursion for the linear KF in
Eq. (

Thus, we decompose the forecast error into its orthogonal projections in the
filtered and unfiltered subspaces as

For

For an arbitrary rank filtered subspace, the reduced rank gain

For every

With the above notation, and using Eq. (

Equation (

We begin by deriving the evolution of error in the unfiltered subspace, by
verifying that it evolves according to the free evolution. Notice first the
following relation:

The error in the

We now consider the evolution of the projection of the forecast error into
the filtered space, with respect to the reduced rank gain. From
Eq. (

For

This implies that the direct application of EKF-AUS from perfect dynamics

It is important to note that the error in the unfiltered subspace moves
upward through the backward Lyapunov filtration precisely because the
unfiltered subspace is defined by the span of the trailing BLVs, governed by
the invariant upper triangular dynamics. The span of the trailing BLVs is not
equal to the direct sum of the trailing Oseledec spaces, which are themselves
covariant with the dynamics. However, this choice for the unfiltered subspace comes
naturally because the filtered subspace (the image space of

In principle, data assimilation could be designed to prevent dynamical
upwelling of unfiltered error by defining the unfiltered space to be the
direct sum of the trailing, stable Oseledec spaces. In this case, the
unfiltered error would be covariant with the dynamics and leave the filtered
error unaffected. To achieve this, the filtered space would need to be
defined by the orthogonal complement to trailing Oseledec spaces, i.e., the
span of the leading forward Lyapunov vectors

With the recursive form of the filtered error in
Eq. (

We first consider the recursion for the cross covariance. In particular, by
combining Eq. (

Having derived the exact error covariance associated to the reduced rank Kalman estimator, characteristic of the ensemble-based Kalman gain in geophysical models, we will summarize the result.

It should be noted that the KF-AUSE Riccati equation is also valid for the
exact forecast error covariance of a reduced rank Kalman filter in perfect
models, where

We emphasize that the KF-AUSE Riccati equation (

Thus, the upwelling of uncertainty from the unfiltered subspace to the ensemble
span highlights a dynamical mechanism, and provides a theoretical
motivation for why covariance inflation in the EnKF has been successful in
preventing filter divergence. In certain scenarios, due to the neglected
upwelling terms in the standard Kalman filter error recursion, covariance
inflation may emulate the process of upwelling in the ensemble span,
replicating the increased uncertainty in the ensemble span due to the
injection of terms (

Generally, the reasons for using covariance inflation in the EnKF are wide,
including the treatment of model error, sampling error, intrinsic bias and
non-Gaussianity of error distributions

Reduced rank Kalman filters have previously corrected for the upwelling of
model errors with both multiplicative and additive covariance inflation
methods. Although it was not explicitly formulated as such, the SEEK filter
of

The dynamical upwelling of model error differs from the misrepresentation of
the covariance due to truncation error or sampling error induced by nonlinear
dynamics in perfect models, treated in the modified EKF-AUS-NL

Without otherwise augmenting the ensemble-based Kalman gain, the upwelling of
uncertainty into the filtered space can, in certain scenarios, be emulated
with multiplicative inflation. In the following section, we numerically
explore the interaction of the filtered subspace rank, the stability in the
unfiltered directions, and multiplicative covariance inflation in relation to
the effect of dynamical upwelling in reduced rank Kalman filters. However,
while the results of Sect.

We will explore two different discrete model configurations in which we vary
the effect of nonlinearity. In the continuous model configuration with
stochastic differential equations, we also achieve qualitatively similar
results which will not be included. It is important to remark that the
analytic form for the forecast error in Eq. (

In the following, we use two different formulations of the standard Lorenz 96
equations (L96)

In linear experiments, we construct a discrete, linear model from the L96
system. Fixing the system dimension

In our experiments with the discrete extended Kalman filter for nonlinear
systems, we use Eq. (

Let us define the nonlinear map

The matrix

This experimental configuration is mathematically consistent with the
extended Kalman filter for a discrete nonlinear map with model error, and is
a standard formulation for model error twin experiments, utilized by e.g,
Mitchell and Carrassi (2015) and Sakov et al. (2018), with the configuration
using the circulant covariance matrix,

In a linear setting, we compute the exact forecast error covariance of
KF-AUSE via the recursive Riccati equation, Eq. (

In Fig.

Eigenvalues of the KF and KF-AUSE forecast error covariance plotted
with triangles. Projection coefficients of the KF-AUSE forecast error
covariance plotted with crosses. Dimension of the KF-AUSE filtered subspace is

It is of special interest how the projection coefficients of the forecast
error covariance relate to the dimension of the filtered subspace,

In our experiments with the discrete extended Kalman filter, we compute the
analysis root mean square error (RMSE) of each of the following: (i) the full rank extended
Kalman filter (EKF), (ii) the EKF-AUS and (iii) the EKF-AUSE, for which
Eq. (

Recall that EKF-AUS has historically only been studied without additive model
errors – we implement EKF-AUS in the presence of model error by computing a
rank

We study the performance of EKF-AUS/E
when the dimension of the filtered
subspace is greater than, or equal to, the dimension of the unstable–neutral
subspace; the case

To benchmark the performance of EKF-AUS/E, we plot the observation error
standard deviation and the analysis RMSE of the standard, full rank EKF in
horizontal lines – the algorithms for EKF-AUS/E are tantamount to a change
of basis for the EKF when the filtered subspace is equal to the full space;
thus, this is the logical point of comparison. We are interested in
finding the necessary dimension of the filtered subspace such that EKF-AUS/E
has an RMSE which (i) performs better than the observation error standard
deviation and (ii) performs comparably to filtering the entire space. When
the RMSE of EKF-AUS/E falls below the observation error standard deviation,
the filter has a forecast performance superior to initializing observations
directly in the model; when it performs similarly to the EKF, the filter can be
considered close to optimal performance, while utilizing a suboptimal
correction based on only

Analysis RMSE of EKF-AUS plotted with triangles and EKF-AUSE plotted
with crosses, varying over the rank of the suboptimal gain. Horizontal lines are
the observation error standard deviation and the EKF analysis RMSE. Note the log
scale of the

In Fig.

We look at the behavior of the local Lyapunov exponents for the L96 model to
explain the convergence of EKF-AUS to the full rank EKF. In
Fig.

Box plot statistics of the local Lyapunov
exponents, for Lyapunov exponents 14 through 29, over 10

When the filtered subspace for EKF-AUS is of dimension 19, such that the
leading unfiltered BLV corresponds to

Finally, we are interested in how analytically computing the upwelling of
error from the unfiltered subspace, as in EKF-AUSE, compares with a
homogeneous, multiplicative inflation applied to the EKF-AUS algorithm.
Multiplicative scalar inflation is among the most common approaches to
mitigate for sampling and model error in Kalman filtering methods, and it is
widely used in operational environmental forecasts utilizing the EnKF. We
define

From the results in Fig.

Analysis RMSE of EKF-AUS (

Figure

Conceptual representation of the number of
samples necessary to prevent divergence of the EnKF in different filtering
regimes. Dark green represents near-optimal filter performance and dark red
represents filter divergence. In perfect-linear models, only

However, multiplicative inflation (in the ensemble span) neglects the
fundamental issue that the unfiltered error lying outside of the ensemble
span can be the major driver of the uncertainty in a reduced rank filter with
model error. Figure

Conceptual diagram of the shape of the exact
forecast error covariance of the full rank Kalman filter and the exact
reduced rank Kalman filter. The

Generally, unless local Lyapunov exponents in the unfiltered space are
strongly stable and thereby rapidly dissipate the unfiltered perturbations of
model error, transient instabilities can make the unfiltered errors large
enough to prevent useful state estimates

This issue of instability forcing unfiltered error is even more acute in
practice. If an EnKF applies a correction of a rank less than the number of
unstable and neutral Lyapunov exponents, it has been found that the filter's
estimated error can become small while the filter permanently loses track of
the true trajectory

Hybridization of the ensemble-based gain and additive inflation of the
ensemble-based covariance are two historical methods for compensating for the
inability to correct for instabilities outside of the ensemble span. In
hybridization, the ensemble-based Kalman estimator is augmented by a static,
climatologically based estimator – using a background climatological
covariance, the rank of the estimator used for the analysis update is
increased, and has the effect of applying a correction to additional modes
outside of the ensemble span

However, there is considerable difficulty in mathematically analyzing the exact recursive form for a suboptimal augmentation of the ensemble-based covariance and ensemble-based Kalman gain. Although the dynamical upwelling of errors is a generic dynamical feature of these systems, the one-way dependence of the error in the leading BLVs on the trailing BLVs does not persist, due to the introduction of estimation errors into the trailing modes via the augmented gain. Moreover, the surrogate covariance used to constrain error in the trailing BLVs will not generally agree with the exact error covariance in the trailing BLVs, making a closed form more difficult to derive. In this setting, it may be more appropriate to derive heuristic methods which attempt to (i) provide some corrections in the trailing BLVs, albeit suboptimal, (ii) describe the dynamical upwelling of the residual error from the trailing BLVs into the leading BLVs and (iii) describe the cross covariances, between the leading and trailing BLVs, with respect to the corrections.

Multiplicative inflation may be used in this case to account for
misestimation of forecast errors resulting from these approximations, but
this misestimation can also be accounted for using less ad hoc approaches
including parameterizing this error with hyperpriors

Assimilation in the unstable
subspace (AUS) has provided a useful conceptual framework for understanding
the dynamical properties of data assimilation cycling in perfect models. Both
numerical and mathematical results have confirmed the underlying hypothesis
of Anna Trevisan: in the setting of perfect, chaotic models, the evolution of
uncertainty is confined to a space characterized by non-negative Lyapunov
exponents, typically of much lower dimension than the full model state space

This paper now demonstrates that the framework of AUS can also be used to
understand the underlying mechanisms for the evolution of uncertainty for
ensemble-based filters in chaotic models with additive errors. Due to the
high dimensional models, and unresolved physical processes, this circumstance
is ubiquitous in high-dimensional geoscience applications where standard
EnKFs are extremely rank deficient. Utilizing the Lyapunov filtration for the
backward vectors, we have shown how unfiltered error, outside of the span of
the anomalies, is transmitted by the dynamics into the filtered subspace. In
perfect models, or when stability in the unfiltered subspace is sufficiently
strong, this effect can be neglected due to the rapid dissipation of
unfiltered errors. However,

The role of inflation we describe differs from previous studies, e.g., the
work of

If we treat the standard EnKF as a Monte Carlo estimate of the error statistics
characteristic of the KF-AUSE covariance, Eq. (

Where there is dynamical chaos, AUS will continue to be a robust framework for the theory of data assimilation in physical models. Understanding the dynamical mechanisms that govern the evolution of error in fully nonlinear data assimilation, e.g., the unstable–neutral manifolds of a (stochastic) chaotic attractor, will be the subject of future research and may be considered the logical extension of the framework put forward by Anna Trevisan – her insight to the underlying processes in assimilation will continue to provide inspiration to both developers and practitioners of data assimilation methods.

Our Python code for generating the results is included in
our github. Our supplementary materials are available at:

The authors defined their problem and scientific approach together. Grudzien derived the original analytical and numerical results. These results were refined and expanded through discussion and evaluation between the three authors. Grudzien led the writing phase, with Carrassi and Bocquet contributing to editing and review.

The authors declare that they have no conflict of interest.

This article is part of the special issue “Numerical modeling, predictability and data assimilation in weather, ocean and climate: A special issue honoring the legacy of Anna Trevisan (1946–2016)”. It is a result of a symposium honoring the legacy of Anna Trevisan – Bologna, Italy, 17–20 October 2017.

This work benefited from funding by the REDDA project of the Norwegian Research Council under contract 250711. CEREA is a member of the Institut Pierre-Simon Laplace (IPSL). The authors thank the three anonymous referees, Steve Penny and Patrick Raanes, for their valuable feedback on this work. The authors would like to show their gratitude and respect for Anna Trevisan, and the impact she has had on the understanding of theoretical data assimilation. Edited by: Juan Manuel Lopez Reviewed by: Steve G. Penny and three anonymous referees