NPG Nonlinear Processes in Geophysics NPG Nonlin. Processes Geophys. 1607-7946 Copernicus Publications Göttingen, Germany 10.5194/npg-21-869-2014 A non-Gaussian analysis scheme using rank histograms for ensemble data assimilation Metref S. 1 Cosme E. 1 Snyder C. 2 Brasseur P. 1 CNRS – Univ. Grenoble Alpes, LGGE (UMR5183), 38041 Grenoble, France National Center for Atmospheric Research, Boulder, Colorado, USA 25 08 2014 21 4 869 885 Copyright: © 2014 S. Metref et al. 2014 This work is licensed under the Creative Commons Attribution 3.0 Unported License. To view a copy of this licence, visit https://creativecommons.org/licenses/by/3.0/ This article is available from https://npg.copernicus.org/articles/21/869/2014/npg-21-869-2014.html The full text article is available as a PDF file from https://npg.copernicus.org/articles/21/869/2014/npg-21-869-2014.pdf

One challenge of geophysical data assimilation is to address the issue of non-Gaussianities in the distributions of the physical variables ensuing, in many cases, from nonlinear dynamical models. Non-Gaussian ensemble analysis methods fall into two categories, those remapping the ensemble particles by approximating the best linear unbiased estimate, for example, the ensemble Kalman filter (EnKF), and those resampling the particles by directly applying Bayes' rule, like particle filters. In this article, it is suggested that the most common remapping methods can only handle weakly non-Gaussian distributions, while the others suffer from sampling issues. In between those two categories, a new remapping method directly applying Bayes' rule, the multivariate rank histogram filter (MRHF), is introduced as an extension of the rank histogram filter (RHF) first introduced by Anderson (2010). Its performance is evaluated and compared with several data assimilation methods, on different levels of non-Gaussianity with the Lorenz 63 model. The method's behavior is then illustrated on a simple density estimation problem using ensemble simulations from a coupled physical–biogeochemical model of the North Atlantic ocean. The MRHF performs well with low-dimensional systems in strongly non-Gaussian regimes.

 References Anderson, J. L.: A Method for Producing and Evaluating Probabilistic Forecasts from Ensemble Model Integrations, J. Climate, 9, 1518–1530, 1996. Anderson, J. L.: A local least squares framework for ensemble filtering, Mon. Weather Rev., 131, 634–642, 2003. Anderson, J. L.: A non-Gaussian ensemble filter update for data assimilation, Mon. Weather Rev., 138, 4186–4198, 2010. Anderson, J. L.: Localization and Sampling Error Correction in Ensemble Kalman Filter Data Assimilation, Mon. Weather Rev., 140, 2359–2371, 2012. Béal, D., Brasseur, P., Brankart, J.-M., Ourmières, Y., and Verron, J.: Characterization of mixing errors in a coupled physical biogeochemical model of the North Atlantic: implications for nonlinear estimation using Gaussian anamorphosis, Ocean Sci., 6, 247–262, <a href="http://dx.doi.org/10.5194/os-6-247-2010">https://doi.org/10.5194/os-6-247-2010</a>, 2010. Bertino, L., Evensen, G., and Wackernagel, H.: Sequential Data Assimilation Techniques in Oceanography, Internat. Stat. Rev., 71, 223–241, 2003. Bishop, C. H., Etherton, B. J., and Majumdar, S. J.: Adaptive sampling with the ensemble transform Kalman filter, Part I: Theoretical aspects, Mon. Weather Rev., 129, 420–436, 2001. Bocquet, M.: Ensemble Kalman filtering without the intrinsic need for inflation, Nonlin. Processes Geophys., 18, 735–750, <a href="http://dx.doi.org/10.5194/npg-18-735-2011">https://doi.org/10.5194/npg-18-735-2011</a>, 2011. Bocquet, M., Pires, C. A., and Wu, L.: Beyond Gaussian statistical modeling in geophysical data assimilation, Mon. Weather Rev., 138, 2997–3023, 2010. Brankart, J.-M., Testut, C.-E., Béal, D., Doron, M., Fontana, C., Meinvielle, M., Brasseur, P., and Verron, J.: Towards an improved description of ocean uncertainties: effect of local anamorphic transformations on spatial correlations, Ocean Sci., 8, 121–142, <a href="http://dx.doi.org/10.5194/os-8-121-2012">https://doi.org/10.5194/os-8-121-2012</a>, 2012. Buehner, M., Houtekamer, P. L., Charette, C., Mitchell, H. L., and He, B.: Intercomparison of Variational Data Assimilation and the Ensemble Kalman Filter for Global Deterministic NWP, Part I: Description and Single-Observation Experiments, Mon. Weather Rev., 138, 1550–1566, 2010. Candille, G. and Talagrand, O.: Evaluation of probabilistic prediction systems for a scalar variable, Q. J. Roy. Meteorol. Soc., 131, 2131–2150, 2005. Cohn, S. E.: An introduction to estimation theory, J. Meteorol. Soc. Jpn., 75, 257–288, 1997. Cotter, C. J. and Reich, S.: Ensemble filter techniques for intermittent data assimilation, in Large Scale Inverse Problems, Radon Ser. Comput. Appl. Math., 13, 91–134, 2013. Dee, D. and Da Silva, A. M.: The choice of variable for atmospheric moisture analysis, Mon. Weather Rev., 131, 155–171, 2003. Doucet, D., de Freitas, N., and Gordon, N.: An introduction to sequential Monte Carlo methods, in: Sequential Monte Carlo Methods in Practice, edited by: Doucet, D., de Freitas, N., and Gordon, N., Statistics for Engineering and Information Science, Springer-Verlag, New York, 2001. Dutkiewicz, S., Follows, M., Marshall, J., and Gregg, W. W.: Inter-annual variability of phytoplankton abundances in the North Atlantic, Deep-Sea Res., Pt. II, 48, 2323–2344, 2001. El Moselhy, T. A. and Marzouk, Y. M.: Bayesian inference with optimal maps, J. Comput. Phys., 231, 7815–7850, 2012. Evensen, G.: Sequential data assimilation with a nonlinear quasi-geostrophic model using Monte Carlo methods to forecast error statistics, J. Geophys. Res., 99, 10143–10162, 1994. Evensen, G.: The Ensemble Kalman Filter: theoretical formulation and practical implementation, Ocean Dynam., 53, 343–367, 2003. Fournier, A., Hulot, G., Jault, D., Kuang, W., Tangborn, A., Gillet, N., Canet, E., Aubert, J., and Lhuillier, F.: An Introduction to Data Assimilation and Predictability in Geomagnetism, Space Sci. Rev., 155, 247–291, 2010. Gordon, N. J., Salmond, D. J., and Smith, A. F. M.: Novel approach to nonlinear/non-Gaussian Bayesian state estimation, IEEE Proc. F., 140, 107–113, 1993. Gregg, W. W., Friedrichs, M. A., Robinson, A. R., Rose, K. A., Schlitzer, R., Thompson, K. R., and Doney, S. C.: Skill assess- ment in ocean biological data assimilation, J. Mar. Syst., 76, 16–33, 2009. Greybush, S. J., Kalnay, E., Miyoshi, T., Ide, K., and Hunt, B. R.: Balance and Ensemble Kalman Filter Localization Techniques, Mon. Weather Rev., 139, 511–522, 2011. Hamill, T. M.: Interpretation of Rank Histograms for Verifying Ensemble Forecasts, Mon. Weather Rev., 129, 550–560, 2001. Hamill, T. M., Whitaker, J. S., and Snyder, C.: Distance-Dependent Filtering of Background Error Covariance Estimates in an Ensemble Kalman Filter, Mon. Weather Rev., 129, 2776–2790, 2001. Harlim, J. and Hunt, B.: A non-Gaussian ensemble filter for assimilating infrequent noisy observations, Tellus A, 59, 225–237, 2007. Hoteit, I., Luo, X., and Pham, D.-T.: Particle Kalman Filtering: A Nonlinear Bayesian Framework for Ensemble Kalman Filters, Mon. Weather Rev., 140, 528–542, 2012. Houtekamer, P. L. and Mitchell, H. L.: Data Assimilation Using an Ensemble Kalman Filter Techniqu, Mon. Weather Rev., 126, 796–811, 1998. Houtekamer, P. L. and Mitchell, H. L.: A Sequential Ensemble Kalman Filter for Atmospheric Data Assimilation, Mon. Weather Rev., 129, 123–137, 2001. Kalman, R.: A new approach to linear filtering and prediction problems, Trans. Am. Soc. Mech. Mech. Eng. J. Basic Eng., 82, 35–45, 1960. Kullback, S.: Information Theory and Statistics, Wiley, New York, 1959. Lauvernet, C., Brankart, J.-M., Castruccio, F., Broquet, G., Brasseur, P., and Verron, J.: A truncated Gaussian filter for data assimilation with inequality constraints: Application to the hydrostatic stability condition in ocean models, Oceanogr. Meteorol., 27, 1–17, 2009. Lawson, W. and Hansen, J.: Implications of stochastic and deterministic filters as ensemble-based data assimilation methods in varying regimes of error growth, Mon. Weather Rev., 132, 1966–1981, 2004. Lei, J., Bickel, P., and Snyder, C.: Comparison of Ensemble Kalman Filters under Non-Gaussianity, Mon. Weather Rev., 138, 1293–1306, 2010. Lermusiaux, P. F. J.: Data assimilation via error subspace statistical estimation, Part II: Middle Atlantic Bight shelfbreak front simulations and ESSE validation, Mon. Weather Rev., 127, 1408–1432, 1999. Lermusiaux, P. F. J.: Uncertainty estimation and prediction for interdisciplinary ocean dynamics, J. Comp. Phys., 217, 176–199, 2006. Lorenz, E.: Deterministic Nonperiodic Flow, J. Atmos. Sci., 20, 130–141, 1963. Morzfeld, M., Tub, X., Atkinsa, E., and Chorin, A.: A random map implementation of implicit filters, J. Comp. Phys., 231, 2049–2066, 2012. Nakano, S., Ueno, G., and Higuchi, T.: Merging particle filter for sequential data assimilation, Nonlin. Processes Geophys., 14, 395–408, <a href="http://dx.doi.org/10.5194/npg-14-395-2007">https://doi.org/10.5194/npg-14-395-2007</a>, 2007. Pham, D. T.: Stochastic Methods for Sequential Data assimilation in Strongly nonlinear systems, Mon. Weather Rev., 129, 1194–1207, 2001. Reich, S.: A nonparametric ensemble transform method for Bayesian inference, SIAM J. Sci. Comput., 35, A2013–A2024, 2013. Sakov, P. and Bertino, L.: Relation between two common localisation methods for the EnKF, Comput. Geosci., 15, 225–237, 2010. Sakov, P., Counillon, F., Bertino, L., Lisæter, K. A., Oke, P. R., and Korablev, A.: TOPAZ4: an ocean-sea ice data assimilation system for the North Atlantic and Arctic, Ocean Sci., 8, 633–656, <a href="http://dx.doi.org/10.5194/os-8-633-2012">https://doi.org/10.5194/os-8-633-2012</a>, 2012. Simon, E. and Bertino, L.: Application of the Gaussian anamorphosis to assimilation in a 3-D coupled physical-ecosystem model of the North Atlantic with the EnKF: a twin experiment, Ocean Sci., 5, 495–510, <a href="http://dx.doi.org/10.5194/os-5-495-2009">https://doi.org/10.5194/os-5-495-2009</a>, 2009. Snyder, C.: Particle filters, the optimal proposal and high-dimensional systems, Proc. Seminar on Data Assimilation for Atmosphere and Ocean, ECMWF, Reading, Berkshire, 161–170, 2012. Snyder, C., Bengtsson, T., Bickel, P., and Anderson, J.: Obstacles to high-dimensional particle filtering, Mon. Weather Rev., 136, 4629–4640, 2008. Tarantola, A.: Inverse problem theory and methods for model parameter estimation, SIAM, Philadelphia, USA, 2005. van Leeuwen, P.: Particle filtering in geophysical systems, Mon. Weather Rev., 137, 4089–4114, 2009. van Leeuwen, P.: Nonlinear data assimilation in geosciences: an extremely efficient particle filter, Q. J. Roy. Meteorol. Soc., 136, 1991–1999, 2010. Villani, C.: Optimal transportation: Old and new, Springer-Verlag, Berlin, Heidelberg, 2009. Whitaker, J. S. and Hamill, T. M.: Ensemble data assimilation without perturbed observations, Mon. Weather Rev., 1913–1924, 2002. Whitaker, J. S., Hamill, T. M., Wei, X., Song, Y., and Toth, Z.: Ensemble data assimilation with the NCEP Global Forecast System, Mon. Weather Rev., 136, 463–482, 2008. Whitley, D.: A genetic algorithm tutorial, Statist. Comput., 4, 65–85, 1994. Wikle, C. and Berliner, L.: A Bayesian tutorial for data assimilation, Physica D, 230, 1–16, 2007.