NPG Nonlinear Processes in Geophysics NPG Nonlin. Processes Geophys. 1607-7946 Copernicus Publications Göttingen, Germany 10.5194/npg-16-475-2009 The diffuse ensemble filter Yang X. 1 2 DelSole T. 1 2 Center for Ocean-Land-Atmosphere Studies, 4041 Powder Mill Rd., Suite 302, Calverton, MD, 20705, USA George Mason University, Fairfax, VA, USA 16 07 2009 16 4 475 486 Copyright: © 2009 X. Yang 2009 This work is licensed under the Creative Commons Attribution 3.0 Unported License. To view a copy of this licence, visit https://creativecommons.org/licenses/by/3.0/ This article is available from https://npg.copernicus.org/articles/16/475/2009/npg-16-475-2009.html The full text article is available as a PDF file from https://npg.copernicus.org/articles/16/475/2009/npg-16-475-2009.pdf

A new class of ensemble filters, called the Diffuse Ensemble Filter (DEnF), is proposed in this paper. The DEnF assumes that the forecast errors orthogonal to the first guess ensemble are uncorrelated with the latter ensemble and have infinite variance. The assumption of infinite variance corresponds to the limit of "complete lack of knowledge" and differs dramatically from the implicit assumption made in most other ensemble filters, which is that the forecast errors orthogonal to the first guess ensemble have vanishing errors. The DEnF is independent of the detailed covariances assumed in the space orthogonal to the ensemble space, and reduces to conventional ensemble square root filters when the number of ensembles exceeds the model dimension. The DEnF is well defined only in data rich regimes and involves the inversion of relatively large matrices, although this barrier might be circumvented by variational methods. Two algorithms for solving the DEnF, namely the Diffuse Ensemble Kalman Filter (DEnKF) and the Diffuse Ensemble Transform Kalman Filter (DETKF), are proposed and found to give comparable results. These filters generally converge to the traditional EnKF and ETKF, respectively, when the ensemble size exceeds the model dimension. Numerical experiments demonstrate that the DEnF eliminates filter collapse, which occurs in ensemble Kalman filters for small ensemble sizes. Also, the use of the DEnF to initialize a conventional square root filter dramatically accelerates the spin-up time for convergence. However, in a perfect model scenario, the DEnF produces larger errors than ensemble square root filters that have covariance localization and inflation. For imperfect forecast models, the DEnF produces smaller errors than the ensemble square root filter with inflation. These experiments suggest that the DEnF has some advantages relative to the ensemble square root filters in the regime of small ensemble size, imperfect model, and copious observations.

 References Anderson, B. D O. and Moore, J B.: Optimal Filtering, Dover Publications, 1979. Anderson, J L.: An adaptive covariance inflation error correction algorithm for ensemble filters, Tellus~A, 59, 210–224, 2007. Anderson, J L. and Anderson, S L.: A Monte Carlo implementation of the nonlinear filtering problem to produce ensemble assimilations and forecasts, Mon Weather~Rev., 127, 2741–2758, 1999. Ansley, C F. and Kohn, R.: Estimation, filtering and smoothing in state space models with incompletely specified initial conditions, Ann. Stat., 13, 1286–1316, 1985. Bishop, C H., Etherton, B., and Majumdar, S J.: Adaptive Sampling with the Ensemble Transform Kalman Filter. Part I: Theoretical Aspects, Mon Weather~Rev., 129, 420–436, 2001. Burgers, G., van Leeuwen, P J., and Evensen, G.: On the Analysis Scheme in the Ensemble Kalman Filter, Mon Weather~Rev., 126, 1719–1724, 1998. de~Jong, P.: The diffuse Kalman Filter, Ann. Stat., 19, 1073–1083, 1991. Evensen, G.: Sequential data assimilation with a nonlinear quasi-geostrophic model using Monte Carlo methods to forecast error statistics, J. Geophys. Res., 99, 1043–1062, 1994. Gaspari, G. and Cohn, S E.: Construction of Correlation Functions in Two and Three Dimensions, Q J Roy Meteor Soc., 125, 723–757, 1999. Hamill, T M., Whitaker, J S., and Snyder, C.: Distance-Dependent Filtering of Background Error Covariance Estimates in an Ensemble Kalman Filter, Mon Weather~Rev., 129, 2776–2790, 2001. Haykin, S.: Kalman Filtering and Neural Networks, in: Kalman filters, edited by: Haykin, S., chap 1, p 284, John Wiley & Sons, 2001. Horn, R A. and Johnson, C R.: Matrix Analysis, Cambridge University Press, New York, 561~pp., 1985. Houtekamer, P L. and Mitchell, H L.: Data Assimilation Using an Ensemble Kalman Filter Technique, Mon Weather~Rev., 126, 796–811, 1998. Houtekamer, P L. and Mitchell, H L.: A Sequential Ensemble Kalman Filter for Atmospheric Data Assimilation, Mon Weather~Rev., 129, 123–137, 2001. Johnson, R A. and Wichern, D W.: Applied Multivariate Statistical Analysis, Pearson Education Asia, 2002. Kalnay, E. and Yang, S.-C.: Accelerating the spin-up of ensemble Kalman filtering, Q. J. Roy. Meteorol. Soc., submitted, 2009. Klinker, E., Rabier, F., Kelly, G., and Mahfouf, J.-F.: The ECMWF operational implementation of four-dimensional variational assimilation. III: Experimental results and diagnostics with operational configuration, Q. J. Roy. Meteorol. Soc., 126, 1191–1215, 2000. Koopman, S A.: Exact Initial Kalman Filtering and Smoothing for Nonstationary Time Series Models, J. Am. Stat. Assoc., 92, 1630–1638, 1997. Lorenz, E N. and Emanuel, K A.: Optimal sites for supplementary weather observations: simulation with a small model, J. Atmos. Sci, 55, 399–414, 1998. Maybeck, P S.: Stochastic models, estimation, and control, Academic Press, 423~pp., 1979. Sakov, P. and Oke, P R.: Implications of the form of the ensemble transformations in the ensemble square root filters, Mon. Weather Rev., 136, 1042–1053, 2008. Tippett, M K., Anderson, J L., Bishop, C H., Hamill, T M., and Whitaker, J S.: Ensemble square-root filters, Mon Weather~Rev., 131, 1485–1490, 2003. Whitaker, J. and Hamill, T M.: Ensemble Data Assimilation Without Perturbed Observations, Mon Weather~Rev., 130, 1913–1924, 2002. Zupanski, M., Fletcher, S J., Navon, I M., Uzunoglu, B., Heikes, R P., Randall, D A., Ringler, T D., and Daescu, D.: Initiation of ensemble data assimilation, Tellus~A, 58, 159–170, 2006.