NPG Nonlinear Processes in Geophysics NPG Nonlin. Processes Geophys. 1607-7946 Copernicus Publications Göttingen, Germany 10.5194/npg-18-735-2011 Ensemble Kalman filtering without the intrinsic need for inflation Bocquet M. 1 2 Université Paris-Est, CEREA Joint Laboratory École des Ponts ParisTech/EDF R&D, France INRIA, Paris Rocquencourt Research Center, France 20 10 2011 18 5 735 750 Copyright: © 2011 M. Bocquet 2011 This work is licensed under the Creative Commons Attribution 3.0 Unported License. To view a copy of this licence, visit https://creativecommons.org/licenses/by/3.0/ This article is available from https://npg.copernicus.org/articles/18/735/2011/npg-18-735-2011.html The full text article is available as a PDF file from https://npg.copernicus.org/articles/18/735/2011/npg-18-735-2011.pdf

The main <i>intrinsic</i> source of error in the ensemble Kalman filter (EnKF) is sampling error. External sources of error, such as model error or deviations from Gaussianity, depend on the dynamical properties of the model. Sampling errors can lead to instability of the filter which, as a consequence, often requires inflation and localization. The goal of this article is to derive an ensemble Kalman filter which is less sensitive to sampling errors. A prior probability density function conditional on the forecast ensemble is derived using Bayesian principles. Even though this prior is built upon the assumption that the ensemble is Gaussian-distributed, it is different from the Gaussian probability density function defined by the empirical mean and the empirical error covariance matrix of the ensemble, which is implicitly used in traditional EnKFs. This new prior generates a new class of ensemble Kalman filters, called finite-size ensemble Kalman filter (EnKF-N). One deterministic variant, the finite-size ensemble transform Kalman filter (ETKF-N), is derived. It is tested on the Lorenz '63 and Lorenz '95 models. In this context, ETKF-N is shown to be stable without inflation for ensemble size greater than the model unstable subspace dimension, at the same numerical cost as the ensemble transform Kalman filter (ETKF). One variant of ETKF-N seems to systematically outperform the ETKF with optimally tuned inflation. However it is shown that ETKF-N does not account for all sampling errors, and necessitates localization like any EnKF, whenever the ensemble size is too small. In order to explore the need for inflation in this small ensemble size regime, a local version of the new class of filters is defined (LETKF-N) and tested on the Lorenz '95 toy model. Whatever the size of the ensemble, the filter is stable. Its performance without inflation is slightly inferior to that of LETKF with optimally tuned inflation for small interval between updates, and superior to LETKF with optimally tuned inflation for large time interval between updates.

 References Anderson, J.&nbsp;L.: An ensemble adjustment {K}alman filter for data assimilation, Mon. Weather Rev., 129, 2884–2903, 2001. Anderson, J.&nbsp;L.: Exploring the need for localization in ensemble data assimilation using a hierarchical ensemble filter, Physica D, 230, 99–111, 2007{a}. Anderson, J.&nbsp;L.: An adaptive covariance inflation error correction algorithm for ensemble filters, Tellus A, 59, 210–224, 2007{b}. Anderson, J.&nbsp;L. and Anderson, S.&nbsp;L.: A {M}onte {C}arlo Implementation of the Nonlinear Filtering Problem to Produce Ensemble Assimilations and Forecasts, Mon. Weather Rev., 127, 2741–2758, 1999. Bishop, C.&nbsp;H. and Hodyss, D.: Flow-adaptive moderation of spurious ensemble correlations and its use in ensemble-based data assimilation, Q. J. Roy. Meteor. Soc., 133, 2029–2044, 2007. Bocquet, M., Pires, C.&nbsp;A., and Wu, L.: Beyond {G}aussian statistical modeling in geophysical data assimilation, Mon. Weather Rev., 138, 2997–3023, 2010. Brankart, J.-M., Cosme, E., Testut, C.-E., Brasseur, P., and Verron, J.: Efficient adaptive error parameterization for square root or ensemble {K}alman filters: application to the control of ocean mesoscale signals, Mon. Weather Rev., 138, 932–950, 2010. Burgers, G., van Leeuwen, P.&nbsp;J., and Evensen, G.: Analysis scheme in the ensemble {K}alman filter, Mon. Weather Rev., 126, 1719–1724, 1998. Carrassi, A., Vannitsem, S., Zupanski, D., and Zupanski, M.: The maximum likelihood ensemble filter performances in chaotic systems, Tellus A, 61, 587–600, 2009. Corazza, M., Kalnay, E., and Patil, D.: Use of the breeding technique to estimate the shape of the analysis errors of the day, J. Geophys. Res., 10, 233–243, 2002. Dee, D.&nbsp;P.: On-line Estimation of Error Covariance Parameters for Atmospheric Data Assimilation, Mon. Weather Rev., 123, 1128–1145, 1995. Desroziers, G., Berre, L., Chapnik, B., and Poli, P.: Diagnosis of observation, background and analysis-error statistics in observation space, Q. J. Roy. Meteor. Soc., 131, 3385–3396, 2005. Evensen, G.: Sequential data assimilation with a nonlinear quasi-geostrophic model using {M}onte {C}arlo methods to forecast error statistics, J. Geophys. Res., 99, 10143–10162, 1994. Evensen, G.: {D}ata {A}ssimilation: {T}he {E}nsemble {K}alman {F}ilter, Springer-Verlag, 2nd Edn., 2009. Furrer, R. and Bengtsson, T.: Estimation of high-dimensional prior and posterior covariance matrices in {K}alman filter variants, J. Multivariate Anal., 98, 227–255, 2007. Gejadze, I.&nbsp;Y., Le&nbsp;Dimet, F.-X., and Shutyaev, V.: On analysis error covariances in variational data assimilation, SIAM J. Sci. Comput., 30, 1847–1874, 2008. Gottwald, G. A. and Mitchell, L., and Reich, S.: Controlling overestimation of error covariance in ensemble K}alman filters with sparse observations: {A variance-limiting {K}alman filter, Mon. Weather Rev., 139, 2650–2667, 2011. Hamill, T.&nbsp;M., Whitaker, J.&nbsp;S., and Snyder, C.: Distance-dependent filtering of background error covariance estimates in an ensemble {K}alman filter, Mon. Weather Rev., 129, 2776–2790, 2001. Harlim, J. and Hunt, B.: A non-{G}aussian ensemble filter for assimilating infrequent noisy observations, Tellus A, 59, 225–237, 2007. Houtekamer, P.&nbsp;L. and Mitchell, H.&nbsp;L.: Data assimilation using an ensemble {K}alman filter technique, Mon. Weather Rev., 126, 796–811, 1998. Houtekamer, P.&nbsp;L. and Mitchell, H.&nbsp;L.: A sequential ensemble {K}alman filter for atmospheric data assimilation, Mon. Weather Rev., 129, 123–137, 2001. Huber, P.&nbsp;J.: Robust regression: Asymptotics, conjectures, and {M}onte {C}arlo, Ann. Statist., 1, 799–821, 1973. Hunt, B.&nbsp;R., Kostelich, E.&nbsp;J., and Szunyogh, I.: Efficient data assimilation for spatiotemporal chaos: A local ensemble transform {K}alman filter, Physica D, 230, 112–126, 2007. Jeffreys, H.: Theory of Probability, Oxford University Press, 3rd Edn., 1961. Lei, J., Bickel, P., and Snyder, C.: Comparison of Ensemble {K}alman Filters under Non-{G}aussianity, Mon. Weather Rev., 138, 1293–1306, 2010. Li, H., Kalnay, E., and Miyoshi, T.: Simultaneous estimation of covariance inflation and observation errors within an ensemble {K}alman filter, Q. J. Roy. Meteor. Soc., 135, 523–533, 2009. Livings, D.&nbsp;M., Dance, S.&nbsp;L., and Nichols, N.&nbsp;K.: Unbiased ensemble square root filters, Physica D, 237, 1021–1028, 2008. Lorenz, E.&nbsp;N.: Deterministic nonperiodic flow, J. Atmos. Sci., 20, 130–141, 1963. Lorenz, E.&nbsp;N. and Emmanuel, K.&nbsp;E.: Optimal sites for supplementary weather observations: simulation with a small model, J. Atmos. Sci., 55, 399–414, 1998. Mallat, S., Papanicolaou, G., and Zhang, Z.: Adaptive covariance estimation of locally stationary processes, Ann. Stat., 26, 1–47, 1998. Mitchell, H.&nbsp;L. and Houtekamer, P.&nbsp;L.: An adaptive ensemble {K}alman filter, Mon. Weather Rev., 128, 416–433, 1999. Mitchell, H.&nbsp;L. and Houtekamer, P.&nbsp;L.: Ensemble {K}alman Filter Configurations and Their Performance with the Logistic Map, Mon. Weather Rev., 137, 4325–4343, 2009. Muirhead, R.&nbsp;J.: Aspect of Multivariate Statistical Theory, Wiley-{I}nterscience, 1982. Myrseth, I. and Omre, H.: Hierarchical Ensemble {K}alman Filter, SPE J., 15, 569–580, 2010. Ott, E., Hunt, B.&nbsp;R., Szunyogh, I., Zimin, A.&nbsp;V., Kostelich, E.&nbsp;J., Corazza, M., Kalnay, E., Patil, D.&nbsp;J., and Yorke, A.: A local ensemble {K}alman filter for atmospheric data assimilation, Tellus A, 56, 415–428, 2004. Raynaud, L., Berre, L., and Desroziers, G.: Objective filtering of ensemble-based background-error variances, Q. J. Roy. Meteor. Soc., 135, 1177–1199, 2009. Sacher, W. and Bartello, P.: Sampling Errors in Ensemble {K}alman Filtering. Part {I}: {T}heory, Mon. Weather Rev., 136, 3035–3049, 2008. Sakov, P. and Bertino, L.: Relation between two common localisation methods for the E}n{KF, Comput. Geosci., 15, 225–237, 2010. Sakov, P. and Oke, P.&nbsp;R.: Implications of the Form of the Ensemble Transformation in the Ensemble Square Root Filters, Mon. Weather Rev., 136, 1042–1053, 2008. Tippett, M.&nbsp;K., Anderson, J.&nbsp;L., Bishop, C.&nbsp;H., Hamill, T.&nbsp;M., and Whitaker, J.&nbsp;S.: Ensemble square root filters, Mon. Weather Rev., 131, 1485–1490, 2003. van Leeuwen, P.&nbsp;J.: Comment on Data Assimilation Using an Ensemble Kalman Filter Technique, Mon. Weather Rev., 127, 1374–1377, 1999. Wang, X., Bishop, C.&nbsp;H., and Julier, S.&nbsp;J.: Which is better, an ensemble of positive-negative pairs or a centered spherical simplex ensemble?, Mon. Weather Rev., 132, 1590–1605, 2004. Whitaker, J.&nbsp;S. and Hamill, T.&nbsp;M.: Ensemble Data Assimilation without Perturbed Observations, Mon. Weather Rev., 130, 1913–1924, 2002. Wick, G.&nbsp;C.: The Evaluation of the Collision Matrix, Phys. Rev., 80, 268–272, 1950. Zhou, Y., McLaughlin, A., and Entekhabi, D.: Assessing the performance of the ensemble kalman filter for land surface data assimilation, Mon. Weather Rev., 134, 2128–2142, 2006. Zupanski, M.: Maximum Likelihood Ensemble Filter: Theoretical Aspects, Mon. Weather Rev., 133, 1710–1726, 2005.