Articles | Volume 29, issue 2
https://doi.org/10.5194/npg-29-241-2022
© Author(s) 2022. This work is distributed under
the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
https://doi.org/10.5194/npg-29-241-2022
© Author(s) 2022. This work is distributed under
the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
Research article
| 
22 Jun 2022
Research article |
| 22 Jun 2022
| 22 Jun 2022
A stochastic covariance shrinkage approach to particle rejuvenation in the ensemble transform particle filter
Computational Science Laboratory, Department of Computer Science, Virginia Tech, 2202 Kraft Drive, Blacksburg, VA 24060, USA
Amit N. Subrahmanya
Computational Science Laboratory, Department of Computer Science, Virginia Tech, 2202 Kraft Drive, Blacksburg, VA 24060, USA
Adrian Sandu
Computational Science Laboratory, Department of Computer Science, Virginia Tech, 2202 Kraft Drive, Blacksburg, VA 24060, USA
Related authors
Andrey A. Popov and Adrian Sandu
Nonlin. Processes Geophys., 26, 109–122, https://doi.org/10.5194/npg-26-109-2019, https://doi.org/10.5194/npg-26-109-2019, 2019
Short summary
Short summary
This work has to do with a small part of existing algorithms that are used in applications such as predicting the weather. We provide empirical evidence that our new technique works well on small but representative models. This might lead to creation of a better weather forecast and potentially save lives as in the case of hurricane prediction.
Andrey A. Popov and Adrian Sandu
Nonlin. Processes Geophys., 26, 109–122, https://doi.org/10.5194/npg-26-109-2019, https://doi.org/10.5194/npg-26-109-2019, 2019
Short summary
Short summary
This work has to do with a small part of existing algorithms that are used in applications such as predicting the weather. We provide empirical evidence that our new technique works well on small but representative models. This might lead to creation of a better weather forecast and potentially save lives as in the case of hurricane prediction.
Ahmed Attia and Adrian Sandu
Geosci. Model Dev., 12, 629–649, https://doi.org/10.5194/gmd-12-629-2019, https://doi.org/10.5194/gmd-12-629-2019, 2019
Short summary
Short summary
This work describes DATeS, a highly extensible data assimilation package. DATeS seeks to provide a unified testing suite for data assimilation applications that allows researchers to easily compare different methodologies in different settings with minimal coding effort. The core of DATeS is written in Python. The main functionalities, such as model propagation and assimilation, can however be written in low-level languages such as C or Fortran to attain high levels of computational efficiency.
Related subject area
Subject: Predictability, probabilistic forecasts, data assimilation, inverse problems | Topic: Climate, atmosphere, ocean, hydrology, cryosphere, biosphere | Techniques: Theory
Extending ensemble Kalman filter algorithms to assimilate observations with an unknown time offset
Applying prior correlations for ensemble-based spatial localization
Ensemble Riemannian data assimilation: towards large-scale dynamical systems
Inferring the instability of a dynamical system from the skill of data assimilation exercises
Multivariate localization functions for strongly coupled data assimilation in the bivariate Lorenz 96 system
Improving the potential accuracy and usability of EURO-CORDEX estimates of future rainfall climate using frequentist model averaging
Ensemble Riemannian data assimilation over the Wasserstein space
An early warning sign of critical transition in the Antarctic ice sheet – a data-driven tool for a spatiotemporal tipping point
Behavior of the iterative ensemble-based variational method in nonlinear problems
A methodology to obtain model-error covariances due to the discretization scheme from the parametric Kalman filter perspective
A method for predicting the uncompleted climate transition process
Statistical postprocessing of ensemble forecasts for severe weather at Deutscher Wetterdienst
Correcting for model changes in statistical postprocessing – an approach based on response theory
Brief communication: Residence time of energy in the atmosphere
Seasonal statistical–dynamical prediction of the North Atlantic Oscillation by probabilistic post-processing and its evaluation
Application of a local attractor dimension to reduced space strongly coupled data assimilation for chaotic multiscale systems
Order of operation for multi-stage post-processing of ensemble wind forecast trajectories
Elia Gorokhovsky and Jeffrey L. Anderson
Nonlin. Processes Geophys., 30, 37–47, https://doi.org/10.5194/npg-30-37-2023, https://doi.org/10.5194/npg-30-37-2023, 2023
Short summary
Short summary
Older observations of the Earth system sometimes lack information about the time they were taken, posing problems for analyses of past climate. To begin to ameliorate this problem, we propose new methods of varying complexity, including methods to estimate the distribution of the offsets between true and reported observation times. The most successful method accounts for the nonlinearity in the system, but even the less expensive ones can improve data assimilation in the presence of time error.
Chu-Chun Chang and Eugenia Kalnay
Nonlin. Processes Geophys., 29, 317–327, https://doi.org/10.5194/npg-29-317-2022, https://doi.org/10.5194/npg-29-317-2022, 2022
Short summary
Short summary
This study introduces a new approach for enhancing the ensemble data assimilation (DA), a technique that combines observations and forecasts to improve numerical weather predictions. Our method uses the prescribed correlations to suppress spurious errors, improving the accuracy of DA. The experiments on the simplified atmosphere model show that our method has comparable performance to the traditional method but is superior in the early stage and is more computationally efficient.
Sagar K. Tamang, Ardeshir Ebtehaj, Peter Jan van Leeuwen, Gilad Lerman, and Efi Foufoula-Georgiou
Nonlin. Processes Geophys., 29, 77–92, https://doi.org/10.5194/npg-29-77-2022, https://doi.org/10.5194/npg-29-77-2022, 2022
Short summary
Short summary
The outputs from Earth system models are optimally combined with satellite observations to produce accurate forecasts through a process called data assimilation. Many existing data assimilation methodologies have some assumptions regarding the shape of the probability distributions of model output and observations, which results in forecast inaccuracies. In this paper, we test the effectiveness of a newly proposed methodology that relaxes such assumptions about high-dimensional models.
Yumeng Chen, Alberto Carrassi, and Valerio Lucarini
Nonlin. Processes Geophys., 28, 633–649, https://doi.org/10.5194/npg-28-633-2021, https://doi.org/10.5194/npg-28-633-2021, 2021
Short summary
Short summary
Chaotic dynamical systems are sensitive to the initial conditions, which are crucial for climate forecast. These properties are often used to inform the design of data assimilation (DA), a method used to estimate the exact initial conditions. However, obtaining the instability properties is burdensome for complex problems, both numerically and analytically. Here, we suggest a different viewpoint. We show that the skill of DA can be used to infer the instability properties of a dynamical system.
Zofia Stanley, Ian Grooms, and William Kleiber
Nonlin. Processes Geophys., 28, 565–583, https://doi.org/10.5194/npg-28-565-2021, https://doi.org/10.5194/npg-28-565-2021, 2021
Short summary
Short summary
In weather forecasting, observations are incorporated into a model of the atmosphere through a process called data assimilation. Sometimes observations in one location may impact the weather forecast in another faraway location in undesirable ways. The impact of distant observations on the forecast is mitigated through a process called localization. We propose a new method for localization when a model has multiple length scales, as in a model spanning both the ocean and the atmosphere.
Stephen Jewson, Giuliana Barbato, Paola Mercogliano, Jaroslav Mysiak, and Maximiliano Sassi
Nonlin. Processes Geophys., 28, 329–346, https://doi.org/10.5194/npg-28-329-2021, https://doi.org/10.5194/npg-28-329-2021, 2021
Short summary
Short summary
Climate model simulations are uncertain. In some cases this makes it difficult to know how to use them. Significance testing is often used to deal with this issue but has various shortcomings. We describe two alternative ways to manage uncertainty in climate model simulations that avoid these shortcomings. We test them on simulations of future rainfall over Europe and show they produce more accurate projections than either using unadjusted climate model output or statistical testing.
Sagar K. Tamang, Ardeshir Ebtehaj, Peter J. van Leeuwen, Dongmian Zou, and Gilad Lerman
Nonlin. Processes Geophys., 28, 295–309, https://doi.org/10.5194/npg-28-295-2021, https://doi.org/10.5194/npg-28-295-2021, 2021
Short summary
Short summary
Data assimilation aims to improve hydrologic and weather forecasts by combining available information from Earth system models and observations. The classical approaches to data assimilation usually proceed with some preconceived assumptions about the shape of their probability distributions. As a result, when such assumptions are invalid, the forecast accuracy suffers. In the proposed methodology, we relax such assumptions and demonstrate improved performance.
Abd AlRahman AlMomani and Erik Bollt
Nonlin. Processes Geophys., 28, 153–166, https://doi.org/10.5194/npg-28-153-2021, https://doi.org/10.5194/npg-28-153-2021, 2021
Short summary
Short summary
This paper introduces a tool for data-driven discovery of early warning signs of critical transitions in ice shelves from remote sensing data. Our directed spectral clustering method considers an asymmetric affinity matrix along with the associated directed graph Laplacian. We applied our approach to reprocessing the ice velocity data and remote sensing satellite images of the Larsen C ice shelf.
Shin'ya Nakano
Nonlin. Processes Geophys., 28, 93–109, https://doi.org/10.5194/npg-28-93-2021, https://doi.org/10.5194/npg-28-93-2021, 2021
Short summary
Short summary
The ensemble-based variational method is a method for solving nonlinear data assimilation problems by using an ensemble of multiple simulation results. Although this method is derived based on a linear approximation, highly uncertain problems, in which system nonlinearity is significant, can also be solved by applying this method iteratively. This paper reformulated this iterative algorithm to analyze its behavior in high-dimensional nonlinear problems and discuss the convergence.
Olivier Pannekoucke, Richard Ménard, Mohammad El Aabaribaoune, and Matthieu Plu
Nonlin. Processes Geophys., 28, 1–22, https://doi.org/10.5194/npg-28-1-2021, https://doi.org/10.5194/npg-28-1-2021, 2021
Short summary
Short summary
Numerical weather prediction involves numerically solving the mathematical equations, which describe the geophysical flow, by transforming them so that they can be computed. Through this transformation, it appears that the equations actually solved by the machine are then a modified version of the original equations, introducing an error that contributes to the model error. This work helps to characterize the covariance of the model error that is due to this modification of the equations.
Pengcheng Yan, Guolin Feng, Wei Hou, and Ping Yang
Nonlin. Processes Geophys., 27, 489–500, https://doi.org/10.5194/npg-27-489-2020, https://doi.org/10.5194/npg-27-489-2020, 2020
Short summary
Short summary
A system transiting from one stable state to another has to experience a period. Can we predict the end moment (state) if the process has not been completed? This paper presents a method to solve this problem. This method is based on the quantitative relationship among the parameters, which is used to describe the transition process of the abrupt change. By using the historical data, we extract some parameters for predicting the uncompleted climate transition process.
Reinhold Hess
Nonlin. Processes Geophys., 27, 473–487, https://doi.org/10.5194/npg-27-473-2020, https://doi.org/10.5194/npg-27-473-2020, 2020
Short summary
Short summary
Forecasts of ensemble systems are statistically aligned to synoptic observations at DWD in order to provide support for warning decision management. Motivation and design consequences for extreme and rare meteorological events are presented. Especially for probabilities of severe wind gusts global logistic parameterisations are developed that generate robust statistical forecasts for extreme events, while local characteristics are preserved.
Jonathan Demaeyer and Stéphane Vannitsem
Nonlin. Processes Geophys., 27, 307–327, https://doi.org/10.5194/npg-27-307-2020, https://doi.org/10.5194/npg-27-307-2020, 2020
Short summary
Short summary
Postprocessing schemes used to correct weather forecasts are no longer efficient when the model generating the forecasts changes. An approach based on response theory to take the change into account without having to recompute the parameters based on past forecasts is presented. It is tested on an analytical model and a simple model of atmospheric variability. We show that this approach is effective and discuss its potential application for an operational environment.
Carlos Osácar, Manuel Membrado, and Amalio Fernández-Pacheco
Nonlin. Processes Geophys., 27, 235–237, https://doi.org/10.5194/npg-27-235-2020, https://doi.org/10.5194/npg-27-235-2020, 2020
Short summary
Short summary
We deduce that after a global thermal perturbation, the Earth's
atmosphere would need about a couple of months to come back to equilibrium.
André Düsterhus
Nonlin. Processes Geophys., 27, 121–131, https://doi.org/10.5194/npg-27-121-2020, https://doi.org/10.5194/npg-27-121-2020, 2020
Short summary
Short summary
Seasonal prediction of the of the North Atlantic Oscillation (NAO) has been improved in recent years by improving dynamical models and ensemble predictions. One step therein was the so-called sub-sampling, which combines statistical and dynamical predictions. This study generalises this approach and makes it much more accessible. Furthermore, it presents a new verification approach for such predictions.
Courtney Quinn, Terence J. O'Kane, and Vassili Kitsios
Nonlin. Processes Geophys., 27, 51–74, https://doi.org/10.5194/npg-27-51-2020, https://doi.org/10.5194/npg-27-51-2020, 2020
Short summary
Short summary
This study presents a novel method for reduced-rank data assimilation of multiscale highly nonlinear systems. Time-varying dynamical properties are used to determine the rank and projection of the system onto a reduced subspace. The variable reduced-rank method is shown to succeed over other fixed-rank methods. This work provides implications for performing strongly coupled data assimilation with a limited number of ensemble members on high-dimensional coupled climate models.
Nina Schuhen
Nonlin. Processes Geophys., 27, 35–49, https://doi.org/10.5194/npg-27-35-2020, https://doi.org/10.5194/npg-27-35-2020, 2020
Short summary
Short summary
We present a new way to adaptively improve weather forecasts by incorporating last-minute observation information. The recently measured error, together with a statistical model, gives us an indication of the expected future error of wind speed forecasts, which are then adjusted accordingly. This new technique can be especially beneficial for customers in the wind energy industry, where it is important to have reliable short-term forecasts, as well as providers of extreme weather warnings.
Cited articles
Aggarwal, C. C.: Neural networks and deep learning, Springer, https://doi.org/10.1007/978-3-319-94463-0, 2018. a
Anderson, J. L.: Localization and sampling error correction in ensemble Kalman filter data assimilation, Mon. Weather Rev., 140, 2359–2371, 2012. a
Asch, M., Bocquet, M., and Nodet, M.: Data assimilation: methods, algorithms, and applications, SIAM, https://doi.org/10.1137/1.9781611974546, 2016. a, b, c
Attia, A. and Sandu, A.: A hybrid Monte–Carlo sampling filter for non-Gaussian data assimilation, AIMS Geosciences, 3, 41–78, https://doi.org/10.3934/geosci.2015.1.41, 2015. a
Burgers, G., van Leeuwen, P. J., and Evensen, G.: Analysis scheme in the
ensemble Kalman filter, Mon. Weather Rev., 126, 1719–1724, 1998. a
Chen, Y., Wiesel, A., and Hero, A. O.: Shrinkage estimation of high dimensional covariance matrices, in: 2009 IEEE International Conference on Acoustics, Speech and Signal Processing, 19–24 April 2009, Taipei, Taiwan, IEEE, 2937–2940, https://doi.org/10.1109/ICASSP.2009.4960239, 2009. a, b, c
Chustagulprom, N., Reich, S., and Reinhardt, M.: A hybrid ensemble transform
particle filter for nonlinear and spatially extended dynamical systems,
SIAM/ASA Journal on Uncertainty Quantification, 4, 592–608, 2016. a
Computational Science Laboratory: DA Tools, GitHub [code],
https://github.com/ComputationalScienceLaboratory/DATools/releases/tag/2022-06-10, last access: 10 June 2022. a
Evensen, G.: Sequential data assimilation with a nonlinear quasi-geostrophic model using Monte Carlo methods to forecast error statistics, J. Geophys. Res.-Oceans, 99, 10143–10162, 1994. a
Evensen, G.: Data assimilation: the ensemble Kalman filter, Springer Science & Business Media, https://doi.org/10.1007/978-3-642-03711-5, 2009. a
Farchi, A. and Bocquet, M.: Review article: Comparison of local particle filters and new implementations, Nonlin. Processes Geophys., 25, 765–807, https://doi.org/10.5194/npg-25-765-2018, 2018. a, b
Jaynes, E. T.: Probability theory: The logic of science, Cambridge university press, https://doi.org/10.1017/CBO9780511790423, 2003. a, b
Kozubowski, T. J., Podgórski, K., and Rychlik, I.: Multivariate generalized Laplace distribution and related random fields, J. Multivariate Anal., 113, 59–72, 2013. a
Lancewicki, T. and Aladjem, M.: Multi-target shrinkage estimation for covariance matrices, IEEE T. Signal Proces., 62, 6380–6390, 2014. a
Law, K., Stuart, A., and Zygalakis, K.: Data assimilation: a mathematical introduction, vol. 62, Springer, https://doi.org/10.1007/978-3-319-20325-6, 2015. a
Liu, J. S.: Monte Carlo strategies in scientific computing, Springer Science & Business Media, https://doi.org/10.1007/978-0-387-76371-2, 2008. a, b
Lorenc, A. C., Bowler, N. E., Clayton, A. M., Pring, S. R., and Fairbairn, D.: Comparison of hybrid-4DEnVar and hybrid-4DVar data assimilation methods for global NWP, Mon. Weather Rev., 143, 212–229, 2015. a
Lorenz, E. N.: Deterministic nonperiodic flow, J. Atmos. Sci., 20, 130–141, 1963. a
Lorenz, E. N.: Predictability: A problem partly solved, in: Proc. Seminar on Predictability, 4–8 September 1995, Shinfield Park, Reading, UK, vol. 1, 1996. a
McCann, R. J.: Existence and uniqueness of monotone measure-preserving maps, Duke Math. J., 80, 309–324, https://doi.org/10.1215/S0012-7094-95-08013-2, 1995. a
McCann, R. J. and Guillen, N.: Five lectures on optimal transportation: geometry, regularity and applications, Analysis and geometry of metric measure spaces: lecture notes of the séminaire de Mathématiques Supérieure (SMS) Montréal, 145–180, https://doi.org/10.1090/crmp/056/06, 2011. a
Musso, C., Oudjane, N., and Le Gland, F.: Improving regularised particle filters, in: Sequential Monte Carlo methods in practice, Springer, 247–271, https://doi.org/10.1007/978-1-4757-3437-9_12, 2001. a
Nino-Ruiz, E. D. and Sandu, A.: Ensemble Kalman filter implementations based on shrinkage covariance matrix estimation, Ocean Dynam., 65, 1423–1439, https://doi.org/10.1007/s10236-015-0888-9, 2015. a
Nino-Ruiz, E. D. and Sandu, A.: An ensemble Kalman filter implementation based on modified Cholesky decomposition for inverse covariance matrix estimation, arXiv [preprint], arXiv:1605.08875, 28 May 2016. a, b
Nino-Ruiz, E. D. and Sandu, A.: Efficient parallel implementation of DDDAS inference using an ensemble Kalman filter with shrinkage covariance matrix estimation, Cluster Comput., 22, 2211–2221, https://doi.org/10.1007/s10586-017-1407-1, 2019. a
Nino-Ruiz, E. D., Sandu, A., and Anderson, J. L.: An efficient implementation of the ensemble Kalman filter based on an iterative Sherman–Morrison formula, Stat. Comput., 25, 561–577, https://doi.org/10.1007/s11222-014-9454-4, 2014. a
Nino-Ruiz, E. D., Sandu, A., and Deng, X.: A parallel ensemble Kalman filter implementation based on modified Cholesky decomposition, in: Proceedings of the 6th Workshop on Latest Advances in Scalable Algorithms for Large-Scale Systems, Supercomputing 2015 of ScalA '15, 15 November 2015, Austin, Texas, https://doi.org/10.1145/2832080.2832084, 2015. a
Olver, F. W., Lozier, D. W., Boisvert, R. F., and Clark, C. W.: NIST handbook of mathematical functions hardback and CD-ROM, Cambridge university press, ISBN 978-0521140638, 2010. a
Popov, A. A. and Sandu, A.: An Explicit Probabilistic Derivation of Inflation in a Scalar Ensemble Kalman Filter for Finite Step, Finite Ensemble Convergence, arXiv [preprint], arXiv:2003.13162, 29 March 2020. a, b
Popov, A. A., Sandu, A., Nino-Ruiz, E. D., and Evensen, G.: A Stochastic Covariance Shrinkage Approach in Ensemble Transform Kalman Filtering, arXiv [preprint], arXiv:2003.00354, 29 February 2020. a, b, c
Popov, A. A., Bhattacharjee, A., Subrahmanya, A. N., and Sandu, A.: ComputationalScienceLaboratory/DATools: v0.0.1 (v0.0.1), Zenodo [code], https://doi.org/10.5281/zenodo.6676745, 2022. a
Rao, V., Sandu, A., Ng, M., and Nino-Ruiz, E. D.: Robust Data Assimilation using L1 and Huber norms, SIAM J. Sci. Comput., 39, B548–B570, 2017. a
Roberts, S., Popov, A. A., and Sandu, A.: ODE Test Problems: a MATLAB suite of initial value problems, CoRR, arXiv [preprint], arXiv:1901.04098, 14 January 2019. a
Roberts, S., Popov, A. A., Sarshar, A., Gomillion, R., and Sandu, A.: ComputationalScienceLaboratory/ODE-Test-Problems: v0.0.1 (v0.0.1), Zenodo [code], https://doi.org/10.5281/zenodo.6676706, 2022. a
Snyder, C., Bengtsson, T., Bickel, P., and Anderson, J.: Obstacles to high-dimensional particle filtering, Mon. Weather Rev., 136, 4629–4640,
2008. a
Strogatz, S. H.: Nonlinear dynamics and chaos with student solutions manual: With applications to physics, biology, chemistry, and engineering, CRC press, ISBN 978-0813349107, 2018. a
van Kekem, D. L.: Dynamics of the Lorenz-96 model: Bifurcations, symmetries and waves, PhD thesis, University of Groningen, ISBN 978-94-034-0979-5, 2018. a
Van Leeuwen, P. J.: Particle filtering in geophysical systems, Mon. Weather Rev., 137, 4089–4114, 2009. a
Van Leeuwen, P. J., Künsch, H. R., Nerger, L., Potthast, R., and Reich, S.: Particle filters for high-dimensional geoscience applications: A review, Q. J. Roy. Meteor. Soc., 145, 2335–2365, 2019. a
Villani, C.: Topics in optimal transportation, American Mathematical Soc., 58, ISBN 978-1-4704-6726-5, 2003. a
Zhang, Y., Liu, N., and Oliver, D. S.: Ensemble filter methods with perturbed observations applied to nonlinear problems, Computat. Geosci., 14, 249–261, 2010. a
Short summary
Numerical weather prediction requires the melding of both computational model and data obtained from sensors such as satellites. We focus on one algorithm to accomplish this. We aim to aid its use by additionally supplying it with data obtained from separate models that describe the average behavior of the computational model at any given time. We show that our approach outperforms the standard approaches to this problem.
Numerical weather prediction requires the melding of both computational model and data obtained...
