Articles | Volume 29, issue 1
https://doi.org/10.5194/npg-29-77-2022
© Author(s) 2022. This work is distributed under
the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
https://doi.org/10.5194/npg-29-77-2022
© Author(s) 2022. This work is distributed under
the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
Research article
| 
18 Feb 2022
Research article |
| 18 Feb 2022
| 18 Feb 2022
Ensemble Riemannian data assimilation: towards large-scale dynamical systems
Department of Civil, Environmental and Geo-Engineering and Saint Anthony Falls Laboratory, University of Minnesota-Twin Cities, Twin Cities, Minnesota, USA
Ardeshir Ebtehaj
Department of Civil, Environmental and Geo-Engineering and Saint Anthony Falls Laboratory, University of Minnesota-Twin Cities, Twin Cities, Minnesota, USA
Peter Jan van Leeuwen
Department of Atmospheric Science, Colorado State University, Fort Collins, Colorado, USA
Gilad Lerman
School of Mathematics, University of Minnesota-Twin Cities, Twin Cities, Minnesota, USA
Efi Foufoula-Georgiou
Department of Civil and Environmental Engineering and Department of Earth System Science, University of California Irvine, Irvine, California, USA
Related authors
Sagar K. Tamang, Ardeshir Ebtehaj, Peter J. van Leeuwen, Dongmian Zou, and Gilad Lerman
Nonlin. Processes Geophys., 28, 295–309, https://doi.org/10.5194/npg-28-295-2021, https://doi.org/10.5194/npg-28-295-2021, 2021
Short summary
Short summary
Data assimilation aims to improve hydrologic and weather forecasts by combining available information from Earth system models and observations. The classical approaches to data assimilation usually proceed with some preconceived assumptions about the shape of their probability distributions. As a result, when such assumptions are invalid, the forecast accuracy suffers. In the proposed methodology, we relax such assumptions and demonstrate improved performance.
Sagar Kumar Tamang, Wenjun Song, Xing Fang, Jose Vasconcelos, and J. Brian Anderson
Proc. IAHS, 379, 131–138, https://doi.org/10.5194/piahs-379-131-2018, https://doi.org/10.5194/piahs-379-131-2018, 2018
Short summary
Short summary
The opening of the bridge changed from several feet to a few inches due to sediment deposition. The hydrologic model was used to simulate discharges and then sediment generation from the watershed. The study watershed is ungauged and does not have data for model calibration, therefore, a parameter transfer method is used. Basically, a hydrologic model was developed and calibrated for a nearby watershed with streamflow data, and then model parameters are transferred to the ungauged watershed.
Tianjia Liu, James T. Randerson, Yang Chen, Douglas C. Morton, Elizabeth B. Wiggins, Padhraic Smyth, Efi Foufoula-Georgiou, Roy Nadler, and Omer Nevo
Earth Syst. Sci. Data, 16, 1395–1424, https://doi.org/10.5194/essd-16-1395-2024, https://doi.org/10.5194/essd-16-1395-2024, 2024
Short summary
Short summary
To improve our understanding of extreme wildfire behavior, we use geostationary satellite data to develop the GOFER algorithm and track the hourly fire progression of large wildfires. GOFER fills a key temporal gap present in other fire tracking products that rely on low-Earth-orbit imagery and reveals considerable variability in fire spread rates on diurnal timescales. We create a product of hourly fire perimeters, active-fire lines, and fire spread rates for 28 fires in California.
Nicholas Williams, Nicholas Byrne, Daniel Feltham, Peter Jan Van Leeuwen, Ross Bannister, David Schroeder, Andrew Ridout, and Lars Nerger
The Cryosphere, 17, 2509–2532, https://doi.org/10.5194/tc-17-2509-2023, https://doi.org/10.5194/tc-17-2509-2023, 2023
Short summary
Short summary
Observations show that the Arctic sea ice cover has reduced over the last 40 years. This study uses ensemble-based data assimilation in a stand-alone sea ice model to investigate the impacts of assimilating three different kinds of sea ice observation, including the novel assimilation of sea ice thickness distribution. We show that assimilating ice thickness distribution has a positive impact on thickness and volume estimates within the ice pack, especially for very thick ice.
Nicholas J. Kedzuf, J. Christine Chiu, V. Chandrasekar, Sounak Biswas, Shashank S. Joshil, Yinghui Lu, Peter Jan van Leeuwen, Christopher Westbrook, Yann Blanchard, and Sebastian O'Shea
Atmos. Meas. Tech., 14, 6885–6904, https://doi.org/10.5194/amt-14-6885-2021, https://doi.org/10.5194/amt-14-6885-2021, 2021
Short summary
Short summary
Ice clouds play a key role in our climate system due to their strong controls on precipitation and the radiation budget. However, it is difficult to characterize co-existing ice species using radar observations. We present a new method that separates the radar signals of pristine ice embedded in snow aggregates and retrieves their respective abundances and sizes for the first time. The ability to provide their quantitative microphysical properties will open up many research opportunities.
Concetta Di Mauro, Renaud Hostache, Patrick Matgen, Ramona Pelich, Marco Chini, Peter Jan van Leeuwen, Nancy K. Nichols, and Günter Blöschl
Hydrol. Earth Syst. Sci., 25, 4081–4097, https://doi.org/10.5194/hess-25-4081-2021, https://doi.org/10.5194/hess-25-4081-2021, 2021
Short summary
Short summary
This study evaluates how the sequential assimilation of flood extent derived from synthetic aperture radar data can help improve flood forecasting. In particular, we carried out twin experiments based on a synthetically generated dataset with controlled uncertainty. Our empirical results demonstrate the efficiency of the proposed data assimilation framework, as forecasting errors are substantially reduced as a result of the assimilation.
Sagar K. Tamang, Ardeshir Ebtehaj, Peter J. van Leeuwen, Dongmian Zou, and Gilad Lerman
Nonlin. Processes Geophys., 28, 295–309, https://doi.org/10.5194/npg-28-295-2021, https://doi.org/10.5194/npg-28-295-2021, 2021
Short summary
Short summary
Data assimilation aims to improve hydrologic and weather forecasts by combining available information from Earth system models and observations. The classical approaches to data assimilation usually proceed with some preconceived assumptions about the shape of their probability distributions. As a result, when such assumptions are invalid, the forecast accuracy suffers. In the proposed methodology, we relax such assumptions and demonstrate improved performance.
Sagar Kumar Tamang, Wenjun Song, Xing Fang, Jose Vasconcelos, and J. Brian Anderson
Proc. IAHS, 379, 131–138, https://doi.org/10.5194/piahs-379-131-2018, https://doi.org/10.5194/piahs-379-131-2018, 2018
Short summary
Short summary
The opening of the bridge changed from several feet to a few inches due to sediment deposition. The hydrologic model was used to simulate discharges and then sediment generation from the watershed. The study watershed is ungauged and does not have data for model calibration, therefore, a parameter transfer method is used. Basically, a hydrologic model was developed and calibrated for a nearby watershed with streamflow data, and then model parameters are transferred to the ungauged watershed.
Sara A. Kelly, Zeinab Takbiri, Patrick Belmont, and Efi Foufoula-Georgiou
Hydrol. Earth Syst. Sci., 21, 5065–5088, https://doi.org/10.5194/hess-21-5065-2017, https://doi.org/10.5194/hess-21-5065-2017, 2017
Short summary
Short summary
Humans have profoundly altered land cover and soil drainage for agricultural purposes in the Midwestern USA. Here we investigate whether climate alone can explain recent increases in observed streamflows throughout the region. Using multiple analyses, including statistical tests and water budgets, we conclude that historical drainage installation has likely amplified the streamflow response to regional precipitation increases. We stress that better documentation of artificial drainage is needed.
Zeinab Takbiri, Ardeshir M. Ebtehaj, and Efi Foufoula-Georgiou
Hydrol. Earth Syst. Sci., 21, 2685–2700, https://doi.org/10.5194/hess-21-2685-2017, https://doi.org/10.5194/hess-21-2685-2017, 2017
Short summary
Short summary
We present a multi-sensor retrieval algorithm for flood extent mapping at high spatial and temporal resolution. While visible bands provide flood mapping at fine spatial resolution, their capability is very limited in a cloudy sky. Passive microwaves can penetrate through clouds but cannot detect small-scale flooded surfaces due to their coarse resolution. The proposed method takes advantage of these two observations to retrieve sub-pixel flooded surfaces in all-sky conditions.
Sara A. Kelly, Zeinab Takbiri, Patrick Belmont, and Efi Foufoula-Georgiou
Hydrol. Earth Syst. Sci. Discuss., https://doi.org/10.5194/hess-2016-571, https://doi.org/10.5194/hess-2016-571, 2016
Manuscript not accepted for further review
Short summary
Short summary
Humans have profoundly altered land use and drainage of the Midwestern USA. We examine whether historical precipitation increases alone can explain large increases observed in streamflows throughout the region. Using multiple analyses, statistical tests and a water budget we determine that streamflow increases have been driven by combined effects of precipitation and agricultural drainage installation. We argue that better documentation of artificial drainage is greatly needed.
Related subject area
Subject: Predictability, probabilistic forecasts, data assimilation, inverse problems | Topic: Climate, atmosphere, ocean, hydrology, cryosphere, biosphere | Techniques: Theory
Prognostic assumed-probability-density-function (distribution density function) approach: further generalization and demonstrations
Bridging classical data assimilation and optimal transport: the 3D-Var case
Improving ensemble data assimilation through Probit-space Ensemble Size Expansion for Gaussian Copulas (PESE-GC)
Evolution of small-scale turbulence at large Richardson numbers
How far can the statistical error estimation problem be closed by collocated data?
Using orthogonal vectors to improve the ensemble space of the ensemble Kalman filter and its effect on data assimilation and forecasting
Review article: Towards strongly coupled ensemble data assimilation with additional improvements from machine learning
Toward a multivariate formulation of the parametric Kalman filter assimilation: application to a simplified chemical transport model
Data-driven reconstruction of partially observed dynamical systems
Extending ensemble Kalman filter algorithms to assimilate observations with an unknown time offset
Applying prior correlations for ensemble-based spatial localization
A stochastic covariance shrinkage approach to particle rejuvenation in the ensemble transform particle filter
Inferring the instability of a dynamical system from the skill of data assimilation exercises
Multivariate localization functions for strongly coupled data assimilation in the bivariate Lorenz 96 system
Improving the potential accuracy and usability of EURO-CORDEX estimates of future rainfall climate using frequentist model averaging
Ensemble Riemannian data assimilation over the Wasserstein space
An early warning sign of critical transition in the Antarctic ice sheet – a data-driven tool for a spatiotemporal tipping point
Behavior of the iterative ensemble-based variational method in nonlinear problems
A methodology to obtain model-error covariances due to the discretization scheme from the parametric Kalman filter perspective
A method for predicting the uncompleted climate transition process
Statistical postprocessing of ensemble forecasts for severe weather at Deutscher Wetterdienst
Correcting for model changes in statistical postprocessing – an approach based on response theory
Brief communication: Residence time of energy in the atmosphere
Seasonal statistical–dynamical prediction of the North Atlantic Oscillation by probabilistic post-processing and its evaluation
Application of a local attractor dimension to reduced space strongly coupled data assimilation for chaotic multiscale systems
Order of operation for multi-stage post-processing of ensemble wind forecast trajectories
Jun-Ichi Yano
Nonlin. Processes Geophys., 31, 359–380, https://doi.org/10.5194/npg-31-359-2024, https://doi.org/10.5194/npg-31-359-2024, 2024
Short summary
Short summary
A methodology for directly predicting the time evolution of the assumed parameters for the distribution densities based on the Liouville equation, as proposed earlier, is extended to multidimensional cases and to cases in which the systems are constrained by integrals over a part of the variable range. The extended methodology is tested against a convective energy-cycle system as well as the Lorenz strange attractor.
Marc Bocquet, Pierre J. Vanderbecken, Alban Farchi, Joffrey Dumont Le Brazidec, and Yelva Roustan
Nonlin. Processes Geophys., 31, 335–357, https://doi.org/10.5194/npg-31-335-2024, https://doi.org/10.5194/npg-31-335-2024, 2024
Short summary
Short summary
A novel approach, optimal transport data assimilation (OTDA), is introduced to merge DA and OT concepts. By leveraging OT's displacement interpolation in space, it minimises mislocation errors within DA applied to physical fields, such as water vapour, hydrometeors, and chemical species. Its richness and flexibility are showcased through one- and two-dimensional illustrations.
Man-Yau Chan
Nonlin. Processes Geophys., 31, 287–302, https://doi.org/10.5194/npg-31-287-2024, https://doi.org/10.5194/npg-31-287-2024, 2024
Short summary
Short summary
Forecasts have uncertainties. It is thus essential to reduce these uncertainties. Such reduction requires uncertainty quantification, which often means running costly models multiple times. The cost limits the number of model runs and thus the quantification’s accuracy. This study proposes a technique that utilizes users’ knowledge of forecast uncertainties to improve uncertainty quantification. Tests show that this technique improves uncertainty reduction.
Lev Ostrovsky, Irina Soustova, Yuliya Troitskaya, and Daria Gladskikh
Nonlin. Processes Geophys., 31, 219–227, https://doi.org/10.5194/npg-31-219-2024, https://doi.org/10.5194/npg-31-219-2024, 2024
Short summary
Short summary
The nonstationary kinetic model of turbulence is used to describe the evolution and structure of the upper turbulent layer with the parameters taken from in situ observations. As an example, we use a set of data for three cruises made in different areas of the world ocean. With the given profiles of current shear and buoyancy frequency, the theory yields results that satisfactorily agree with the measurements of the turbulent dissipation rate.
Annika Vogel and Richard Ménard
Nonlin. Processes Geophys., 30, 375–398, https://doi.org/10.5194/npg-30-375-2023, https://doi.org/10.5194/npg-30-375-2023, 2023
Short summary
Short summary
Accurate estimation of the error statistics required for data assimilation remains an ongoing challenge, as statistical assumptions are required to solve the estimation problem. This work provides a conceptual view of the statistical error estimation problem in light of the increasing number of available datasets. We found that the total number of required assumptions increases with the number of overlapping datasets, but the relative number of error statistics that can be estimated increases.
Yung-Yun Cheng, Shu-Chih Yang, Zhe-Hui Lin, and Yung-An Lee
Nonlin. Processes Geophys., 30, 289–297, https://doi.org/10.5194/npg-30-289-2023, https://doi.org/10.5194/npg-30-289-2023, 2023
Short summary
Short summary
In the ensemble Kalman filter, the ensemble space may not fully capture the forecast errors due to the limited ensemble size and systematic model errors, which affect the accuracy of analysis and prediction. This study proposes a new algorithm to use cost-free pseudomembers to expand the ensemble space effectively and improve analysis accuracy during the analysis step, without increasing the ensemble size during forecasting.
Eugenia Kalnay, Travis Sluka, Takuma Yoshida, Cheng Da, and Safa Mote
Nonlin. Processes Geophys., 30, 217–236, https://doi.org/10.5194/npg-30-217-2023, https://doi.org/10.5194/npg-30-217-2023, 2023
Short summary
Short summary
Strongly coupled data assimilation (SCDA) generates coherent integrated Earth system analyses by assimilating the full Earth observation set into all Earth components. We describe SCDA based on the ensemble Kalman filter with a hierarchy of coupled models, from a coupled Lorenz to the Climate Forecast System v2. SCDA with a sufficiently large ensemble can provide more accurate coupled analyses compared to weakly coupled DA. The correlation-cutoff method can compensate for a small ensemble size.
Antoine Perrot, Olivier Pannekoucke, and Vincent Guidard
Nonlin. Processes Geophys., 30, 139–166, https://doi.org/10.5194/npg-30-139-2023, https://doi.org/10.5194/npg-30-139-2023, 2023
Short summary
Short summary
This work is a theoretical contribution that provides equations for understanding uncertainty prediction applied in air quality where multiple chemical species can interact. A simplified minimal test bed is introduced that shows the ability of our equations to reproduce the statistics estimated from an ensemble of forecasts. While the latter estimation is the state of the art, solving equations is numerically less costly, depending on the number of chemical species, and motivates this research.
Pierre Tandeo, Pierre Ailliot, and Florian Sévellec
Nonlin. Processes Geophys., 30, 129–137, https://doi.org/10.5194/npg-30-129-2023, https://doi.org/10.5194/npg-30-129-2023, 2023
Short summary
Short summary
The goal of this paper is to obtain probabilistic predictions of a partially observed dynamical system without knowing the model equations. It is illustrated using the three-dimensional Lorenz system, where only two components are observed. The proposed data-driven procedure is low-cost, is easy to implement, uses linear and Gaussian assumptions and requires only a small amount of data. It is based on an iterative linear Kalman smoother with a state augmentation.
Elia Gorokhovsky and Jeffrey L. Anderson
Nonlin. Processes Geophys., 30, 37–47, https://doi.org/10.5194/npg-30-37-2023, https://doi.org/10.5194/npg-30-37-2023, 2023
Short summary
Short summary
Older observations of the Earth system sometimes lack information about the time they were taken, posing problems for analyses of past climate. To begin to ameliorate this problem, we propose new methods of varying complexity, including methods to estimate the distribution of the offsets between true and reported observation times. The most successful method accounts for the nonlinearity in the system, but even the less expensive ones can improve data assimilation in the presence of time error.
Chu-Chun Chang and Eugenia Kalnay
Nonlin. Processes Geophys., 29, 317–327, https://doi.org/10.5194/npg-29-317-2022, https://doi.org/10.5194/npg-29-317-2022, 2022
Short summary
Short summary
This study introduces a new approach for enhancing the ensemble data assimilation (DA), a technique that combines observations and forecasts to improve numerical weather predictions. Our method uses the prescribed correlations to suppress spurious errors, improving the accuracy of DA. The experiments on the simplified atmosphere model show that our method has comparable performance to the traditional method but is superior in the early stage and is more computationally efficient.
Andrey A. Popov, Amit N. Subrahmanya, and Adrian Sandu
Nonlin. Processes Geophys., 29, 241–253, https://doi.org/10.5194/npg-29-241-2022, https://doi.org/10.5194/npg-29-241-2022, 2022
Short summary
Short summary
Numerical weather prediction requires the melding of both computational model and data obtained from sensors such as satellites. We focus on one algorithm to accomplish this. We aim to aid its use by additionally supplying it with data obtained from separate models that describe the average behavior of the computational model at any given time. We show that our approach outperforms the standard approaches to this problem.
Yumeng Chen, Alberto Carrassi, and Valerio Lucarini
Nonlin. Processes Geophys., 28, 633–649, https://doi.org/10.5194/npg-28-633-2021, https://doi.org/10.5194/npg-28-633-2021, 2021
Short summary
Short summary
Chaotic dynamical systems are sensitive to the initial conditions, which are crucial for climate forecast. These properties are often used to inform the design of data assimilation (DA), a method used to estimate the exact initial conditions. However, obtaining the instability properties is burdensome for complex problems, both numerically and analytically. Here, we suggest a different viewpoint. We show that the skill of DA can be used to infer the instability properties of a dynamical system.
Zofia Stanley, Ian Grooms, and William Kleiber
Nonlin. Processes Geophys., 28, 565–583, https://doi.org/10.5194/npg-28-565-2021, https://doi.org/10.5194/npg-28-565-2021, 2021
Short summary
Short summary
In weather forecasting, observations are incorporated into a model of the atmosphere through a process called data assimilation. Sometimes observations in one location may impact the weather forecast in another faraway location in undesirable ways. The impact of distant observations on the forecast is mitigated through a process called localization. We propose a new method for localization when a model has multiple length scales, as in a model spanning both the ocean and the atmosphere.
Stephen Jewson, Giuliana Barbato, Paola Mercogliano, Jaroslav Mysiak, and Maximiliano Sassi
Nonlin. Processes Geophys., 28, 329–346, https://doi.org/10.5194/npg-28-329-2021, https://doi.org/10.5194/npg-28-329-2021, 2021
Short summary
Short summary
Climate model simulations are uncertain. In some cases this makes it difficult to know how to use them. Significance testing is often used to deal with this issue but has various shortcomings. We describe two alternative ways to manage uncertainty in climate model simulations that avoid these shortcomings. We test them on simulations of future rainfall over Europe and show they produce more accurate projections than either using unadjusted climate model output or statistical testing.
Sagar K. Tamang, Ardeshir Ebtehaj, Peter J. van Leeuwen, Dongmian Zou, and Gilad Lerman
Nonlin. Processes Geophys., 28, 295–309, https://doi.org/10.5194/npg-28-295-2021, https://doi.org/10.5194/npg-28-295-2021, 2021
Short summary
Short summary
Data assimilation aims to improve hydrologic and weather forecasts by combining available information from Earth system models and observations. The classical approaches to data assimilation usually proceed with some preconceived assumptions about the shape of their probability distributions. As a result, when such assumptions are invalid, the forecast accuracy suffers. In the proposed methodology, we relax such assumptions and demonstrate improved performance.
Abd AlRahman AlMomani and Erik Bollt
Nonlin. Processes Geophys., 28, 153–166, https://doi.org/10.5194/npg-28-153-2021, https://doi.org/10.5194/npg-28-153-2021, 2021
Short summary
Short summary
This paper introduces a tool for data-driven discovery of early warning signs of critical transitions in ice shelves from remote sensing data. Our directed spectral clustering method considers an asymmetric affinity matrix along with the associated directed graph Laplacian. We applied our approach to reprocessing the ice velocity data and remote sensing satellite images of the Larsen C ice shelf.
Shin'ya Nakano
Nonlin. Processes Geophys., 28, 93–109, https://doi.org/10.5194/npg-28-93-2021, https://doi.org/10.5194/npg-28-93-2021, 2021
Short summary
Short summary
The ensemble-based variational method is a method for solving nonlinear data assimilation problems by using an ensemble of multiple simulation results. Although this method is derived based on a linear approximation, highly uncertain problems, in which system nonlinearity is significant, can also be solved by applying this method iteratively. This paper reformulated this iterative algorithm to analyze its behavior in high-dimensional nonlinear problems and discuss the convergence.
Olivier Pannekoucke, Richard Ménard, Mohammad El Aabaribaoune, and Matthieu Plu
Nonlin. Processes Geophys., 28, 1–22, https://doi.org/10.5194/npg-28-1-2021, https://doi.org/10.5194/npg-28-1-2021, 2021
Short summary
Short summary
Numerical weather prediction involves numerically solving the mathematical equations, which describe the geophysical flow, by transforming them so that they can be computed. Through this transformation, it appears that the equations actually solved by the machine are then a modified version of the original equations, introducing an error that contributes to the model error. This work helps to characterize the covariance of the model error that is due to this modification of the equations.
Pengcheng Yan, Guolin Feng, Wei Hou, and Ping Yang
Nonlin. Processes Geophys., 27, 489–500, https://doi.org/10.5194/npg-27-489-2020, https://doi.org/10.5194/npg-27-489-2020, 2020
Short summary
Short summary
A system transiting from one stable state to another has to experience a period. Can we predict the end moment (state) if the process has not been completed? This paper presents a method to solve this problem. This method is based on the quantitative relationship among the parameters, which is used to describe the transition process of the abrupt change. By using the historical data, we extract some parameters for predicting the uncompleted climate transition process.
Reinhold Hess
Nonlin. Processes Geophys., 27, 473–487, https://doi.org/10.5194/npg-27-473-2020, https://doi.org/10.5194/npg-27-473-2020, 2020
Short summary
Short summary
Forecasts of ensemble systems are statistically aligned to synoptic observations at DWD in order to provide support for warning decision management. Motivation and design consequences for extreme and rare meteorological events are presented. Especially for probabilities of severe wind gusts global logistic parameterisations are developed that generate robust statistical forecasts for extreme events, while local characteristics are preserved.
Jonathan Demaeyer and Stéphane Vannitsem
Nonlin. Processes Geophys., 27, 307–327, https://doi.org/10.5194/npg-27-307-2020, https://doi.org/10.5194/npg-27-307-2020, 2020
Short summary
Short summary
Postprocessing schemes used to correct weather forecasts are no longer efficient when the model generating the forecasts changes. An approach based on response theory to take the change into account without having to recompute the parameters based on past forecasts is presented. It is tested on an analytical model and a simple model of atmospheric variability. We show that this approach is effective and discuss its potential application for an operational environment.
Carlos Osácar, Manuel Membrado, and Amalio Fernández-Pacheco
Nonlin. Processes Geophys., 27, 235–237, https://doi.org/10.5194/npg-27-235-2020, https://doi.org/10.5194/npg-27-235-2020, 2020
Short summary
Short summary
We deduce that after a global thermal perturbation, the Earth's
atmosphere would need about a couple of months to come back to equilibrium.
André Düsterhus
Nonlin. Processes Geophys., 27, 121–131, https://doi.org/10.5194/npg-27-121-2020, https://doi.org/10.5194/npg-27-121-2020, 2020
Short summary
Short summary
Seasonal prediction of the of the North Atlantic Oscillation (NAO) has been improved in recent years by improving dynamical models and ensemble predictions. One step therein was the so-called sub-sampling, which combines statistical and dynamical predictions. This study generalises this approach and makes it much more accessible. Furthermore, it presents a new verification approach for such predictions.
Courtney Quinn, Terence J. O'Kane, and Vassili Kitsios
Nonlin. Processes Geophys., 27, 51–74, https://doi.org/10.5194/npg-27-51-2020, https://doi.org/10.5194/npg-27-51-2020, 2020
Short summary
Short summary
This study presents a novel method for reduced-rank data assimilation of multiscale highly nonlinear systems. Time-varying dynamical properties are used to determine the rank and projection of the system onto a reduced subspace. The variable reduced-rank method is shown to succeed over other fixed-rank methods. This work provides implications for performing strongly coupled data assimilation with a limited number of ensemble members on high-dimensional coupled climate models.
Nina Schuhen
Nonlin. Processes Geophys., 27, 35–49, https://doi.org/10.5194/npg-27-35-2020, https://doi.org/10.5194/npg-27-35-2020, 2020
Short summary
Short summary
We present a new way to adaptively improve weather forecasts by incorporating last-minute observation information. The recently measured error, together with a statistical model, gives us an indication of the expected future error of wind speed forecasts, which are then adjusted accordingly. This new technique can be especially beneficial for customers in the wind energy industry, where it is important to have reliable short-term forecasts, as well as providers of extreme weather warnings.
Cited articles
Altman, A. and Gondzio, J.: Regularized symmetric indefinite systems in interior point methods for linear and quadratic optimization, Optim. Method. Softw., 11, 275–302, 1999. a
Anderson, J. and Lei, L.: Empirical localization of observation impact in ensemble Kalman filters, Mon. Weather Rev., 141, 4140–4153, 2013. a
Anderson, J. L.: An ensemble adjustment Kalman filter for data assimilation, Mon. Weather Rev., 129, 2884–2903, 2001. a
Anderson, J. L.: Reducing correlation sampling error in ensemble Kalman filter data assimilation, Mon. Weather Rev., 144, 913–925, 2016. a
Bigot, J. and Klein, T.: Characterization of barycenters in the Wasserstein space by averaging optimal transport maps, ESAIM-Probab. Stat., 22, 35–57, https://doi.org/10.1051/ps/2017020, 2018.
a
Bishop, C. H., Etherton, B. J., and Majumdar, S. J.: Adaptive sampling with the ensemble transform Kalman filter. Part I: Theoretical aspects, Mon.
Weather Rev., 129, 420–436, 2001. a
Borobia, A. and Cantó, R.: Matrix scaling: A geometric proof of sinkhorn's theorem, Linear Algebra Appl., 268, 1–8, 1998. a
Brajard, J., Carrassi, A., Bocquet, M., and Bertino, L.: Combining data assimilation and machine learning to emulate a dynamical model from sparse and noisy observations: a case study with the Lorenz 96 model, Journal of Computational Science, 44, 101171, https://doi.org/10.1016/j.jocs.2020.101171, 2020. a
Brenier, Y.: Décomposition polaire et réarrangement monotone des champs de vecteurs, C. R. Acad. Sci. Paris, Série I Math., 305, 805–808, 1987. a
Burgers, G., Jan van Leeuwen, P., and Evensen, G.: Analysis scheme in the ensemble Kalman filter, Mon. Weather Rev., 126, 1719–1724, 1998. a
Carrassi, A. and Vannitsem, S.: State and parameter estimation with the extended Kalman filter: an alternative formulation of the model error dynamics, Q. J. Roy. Meteor. Soc., 137, 435–451, 2011. a
Carrassi, A., Bocquet, M., Bertino, L., and Evensen, G.: Data assimilation in the geosciences: An overview of methods, issues, and perspectives, WIREs Clim. Change, 9, e535, https://doi.org/10.1002/wcc.535, 2018. a
Chen, B., Dang, L., Gu, Y., Zheng, N., and Prıncipe, J. C.: Minimum Error Entropy Kalman Filter, arXiv [preprint], arXiv:1904.06617, 17 April 2019. a
Chen, J., Chen, Y., Wu, H., and Yang, D.: The quadratic Wasserstein metric for earthquake location, J. Comput. Phys., 373, 188–209, 2018. a
Chen, Y., Georgiou, T. T., and Tannenbaum, A.: Matrix optimal mass transport: a quantum mechanical approach, IEEE T. Automat. Contr., 63, 2612–2619, 2017. a
Chen, Y., Georgiou, T. T., and Tannenbaum, A.: Wasserstein geometry of quantum states and optimal transport of matrix-valued measures, in: Emerging Applications of Control and Systems Theory, Springer,
139–150, https://doi.org/10.1007/978-3-319-67068-3_10, 2018. a
Chepurin, G. A., Carton, J. A., and Dee, D.: Forecast model bias correction in ocean data assimilation, Mon. Weather Rev., 33, 1328–1342, 2005. a
Chianese, E., Galletti, A., Giunta, G., Landi, T., Marcellino, L., Montella, R., and Riccio, A.: Spatiotemporally resolved ambient particulate matter concentration by fusing observational data and ensemble chemical transport model simulations, Ecol. Model., 385, 173–181, 2018. a
Cotter, C., Crisan, D., Holm, D., Pan, W., and Shevchenko, I.: Modelling uncertainty using stochastic transport noise in a 2-layer quasi-geostrophic model, Foundations of Data Science, 2, 173–205, 2020. a
Courtier, P., Thépaut, J.-N., and Hollingsworth, A.: A strategy for operational implementation of 4D-Var, using an incremental approach, Q. J. Roy. Meteor. Soc., 120, 1367–1387, 1994. a
Courtier, P., Andersson, E., Heckley, W., Vasiljevic, D., Hamrud, M., Hollingsworth, A., Rabier, F., Fisher, M., and Pailleux, J.: The ECMWF implementation of three-dimensional variational assimilation (3D-Var). I: Formulation, Q. J. Roy. Meteor. Soc., 124, 1783–1807, 1998. a
Cramér, H.: Mathematical methods of statistics, Princeton University Press, vol. 9, ISBN: 9780691005478, 1999. a
Cuturi, M.: Sinkhorn distances: Lightspeed computation of optimal transport, in: Advances in neural information processing systems,
edited by: Burges, C. J. C., Bottou, L., Welling, M., Ghahramani, Z., and Weinberger, K. Q., Curran Associates, Inc., 26, 2292–2300, https://proceedings.neurips.cc/paper/2013/file/af21d0c97db2e27e13572cbf59eb343d-Paper.pdf (last access: 9 February 2022), 2013. a, b
Cuturi, M. and Peyré, G.: Semidual regularized optimal transport, SIAM Rev., 60, 941–965, 2018. a
Dantzig, G. B., Orden, A., and Wolfe, P.: The generalized simplex method for minimizing a linear form under linear inequality restraints, Pac. J. Math., 5, 183–195, 1955. a
Dee, D. P. and Da Silva, A. M.: Data assimilation in the presence of forecast bias, Q. J. Roy. Meteor. Soc., 124, 269–295, 1998. a
De Lannoy, G. J., Reichle, R. H., Houser, P. R., Pauwels, V., and Verhoest, N. E.: Correcting for forecast bias in soil moisture assimilation with the ensemble Kalman filter, Water Resour. Res., 43, W09410, https://doi.org/10.1029/2006WR005449, 2007. a
Dobrushin, R. L.: Prescribing a system of random variables by conditional distributions, Theor. Probab. Appl., 15, 458–486, 1970. a
Evensen, G.: Using the extended Kalman filter with a multilayer quasi-geostrophic ocean model, J. Geophys. Res., 97, 17905–17924, 1992. a
Evensen, G.: The ensemble Kalman filter: Theoretical formulation and practical implementation, Ocean Dynam., 53, 343–367, 2003. a
Evensen, G. and Van Leeuwen, P. J.: Assimilation of Geosat altimeter data for the Agulhas current using the ensemble Kalman filter with a quasigeostrophic model, Mon. Weather Rev., 124, 85–96, 1996. a
Feyeux, N., Vidard, A., and Nodet, M.: Optimal transport for variational data assimilation, Nonlin. Processes Geophys., 25, 55–66, https://doi.org/10.5194/npg-25-55-2018, 2018. a, b
Fisher, M. and Gürol, S.: Parallelization in the time dimension of four-dimensional variational data assimilation, Q. J. Roy. Meteor. Soc., 143, 1136–1147, 2017. a
Fréchet, M.: Les éléments aléatoires de nature quelconque dans un espace distancié, Annales de l'institut Henri Poincaré, 10, 215–310, 1948 (in French). a
Gaspari, G. and Cohn, S. E.: Construction of correlation functions in two and three dimensions, Q. J. Roy. Meteor. Soc., 125, 723–757, 1999. a
Hamill, T. M.: Interpretation of rank histograms for verifying ensemble forecasts, Mon. Weather Rev., 129, 550–560, 2001. a
Hamill, T. M., Whitaker, J. S., and Snyder, C.: Distance-dependent filtering of background error covariance estimates in an ensemble Kalman filter, Mon. Weather Rev., 129, 2776–2790, 2001. a
Hellinger, E.: Neue begründung der theorie quadratischer formen von unendlichvielen veränderlichen, J. Reine Angew. Math., 1909, 210–271, 1909 (in German). a
Houtekamer, P. L. and Mitchell, H. L.: Data assimilation using an ensemble Kalman filter technique, Mon. Weather Rev., 126, 796–811, 1998. a
Houtekamer, P. L. and Mitchell, H. L.: A sequential ensemble Kalman filter for atmospheric data assimilation, Mon. Weather Rev., 129, 123–137, 2001. a
Houtekamer, P. L. and Zhang, F.: Review of the ensemble Kalman filter for atmospheric data assimilation, Mon. Weather Rev., 144, 4489–4532, 2016. a
Kalman, R. E.: A New Approach to Linear Filtering and Prediction Problems, J. Basic Eng.-T. ASME, 82, 35–45, https://doi.org/10.1115/1.3662552, 1960. a
Kalnay, E.: Atmospheric modeling, data assimilation and predictability, Cambridge University Press, Cambridge, ISBN: 9780511802270, 2003. a
Kapur, J. N.: Measures of information and their applications, Wiley-Interscience, https://doi.org/10.2307/2533186, 1994. a
Kolouri, S., Park, S. R., Thorpe, M., Slepcev, D., and Rohde, G. K.: Optimal mass transport: Signal processing and machine-learning applications, IEEE Signal Proc. Mag., 34, 43–59, 2017. a
Kullback, S. and Leibler, R. A.: On information and sufficiency, Ann. Math. Stat., 22, 79–86, 1951. a
Kutta, W.: Beitrag zur naherungsweisen Integration totaler Differentialgleichungen, Z. Math. Phys., 46, 435–453, 1901 (in German). a
Lei, J. and Bickel, P.: A moment matching ensemble filter for nonlinear non-Gaussian data assimilation, Mon. Weather Rev., 139, 3964–3973, 2011. a
Lei, L., Whitaker, J. S., and Bishop, C.: Improving assimilation of radiance observations by implementing model space localization in an ensemble Kalman filter, J. Adv. Model. Earth Sy., 10, 3221–3232, 2018. a
Lguensat, R., Tandeo, P., Ailliot, P., Pulido, M., and Fablet, R.: The analog data assimilation, Mon. Weather Rev., 145, 4093–4107, 2017. a
Li, L., Vidard, A., Le Dimet, F.-X., and Ma, J.: Topological data assimilation using Wasserstein distance, Inverse Problems, 35, 015006, https://doi.org/10.1088/1361-6420/aae993, 2018. a
Li, R., Jan, N. M., Huang, B., and Prasad, V.: Constrained ensemble Kalman filter based on Kullback–Leibler divergence, J. Process Contr., 81, 150–161, 2019. a
Li, T., Bolic, M., and Djuric, P. M.: Resampling methods for particle filtering: classification, implementation, and strategies, IEEE Signal Proc. Mag., 32, 70–86, 2015. a
Li, Z., Zang, Z., Li, Q. B., Chao, Y., Chen, D., Ye, Z., Liu, Y., and Liou, K. N.: A three-dimensional variational data assimilation system for multiple aerosol species with WRF/Chem and an application to PM2.5 prediction, Atmos. Chem. Phys., 13, 4265–4278, https://doi.org/10.5194/acp-13-4265-2013, 2013. a
Lin, L.-F., Ebtehaj, A. M., Flores, A. N., Bastola, S., and Bras, R. L.: Combined assimilation of satellite precipitation and soil moisture: A case study using trmm and smos data, Mon. Weather Rev., 145, 4997–5014, 2017. a
Lorenc, A. C., Ballard, S. P., Bell, R. S., Ingleby, N. B., Andrews, P. L. F., Barker, D. M., Bray, J. R., Clayton, A. M., Dalby, T., Li, D., Payne, T. J., and Saunders, F. W.: The Met. Office global three-dimensional variational data assimilation scheme, Q. J. Roy. Meteor. Soc., 126, 2991–3012, 2000. a
Lorenz, E. N.: Deterministic nonperiodic flow, J. Atmos. Sci., 20, 130–141, 1963. a
Lorenz, E. N.: Predictability – a problem partly solved, in: Predictability of Weather and Climate, Seminar on Predictability, Shinfield Park, Reading, UK, 4–8 September 1995, ECMWF, https://doi.org/10.1017/CBO9780511617652.004, 1995. a, b, c
Maclean, J., Santitissadeekorn, N., and Jones, C. K.: A coherent structure approach for parameter estimation in Lagrangian Data Assimilation, Physica D, 360, 36–45, 2017. a
McCann, R. J.: A convexity principle for interacting gases, Adv. Math., 128, 153–179, 1997. a
Nerger, L., Janjić, T., Schröter, J., and Hiller, W.: A regulated localization scheme for ensemble-based Kalman filters, Q. J. Roy. Meteor. Soc., 138, 802–812, 2012a. a
Nerger, L., Janjić, T., Schröter, J., and Hiller, W.: A unification of ensemble square root Kalman filters, Mon. Weather Rev., 140, 2335–2345, 2012b. a
Pass, B.: Multi-marginal optimal transport: theory and applications, ESAIM-Math. Model. Num., 49, 1771–1790, 2015. a
Pedlosky, J.: Geophysical fluid dynamics, Springer, vol. 710, https://doi.org/10.1007/978-1-4612-4650-3, 1987. a, b
Penny, S., Bach, E., Bhargava, K., Chang, C.-C., Da, C., Sun, L., and Yoshida, T.: Strongly coupled data assimilation in multiscale media: Experiments using a quasi-geostrophic coupled model, J. Adv. Model. Earth Sy., 11, 1803–1829, 2019. a
Pitt, M. K. and Shephard, N.: Filtering via simulation: Auxiliary particle filters, J. Am. Stat. Assoc., 94, 590–599, 1999. a
Poterjoy, J. and Zhang, F.: Intercomparison and coupling of ensemble and four-dimensional variational data assimilation methods for the analysis and forecasting of Hurricane Karl (2010), Mon. Weather Rev., 142, 3347–3364, 2014. a
Pulido, M. and van Leeuwen, P. J.: Sequential Monte Carlo with kernel embedded mappings: The mapping particle filter, J. Comput. Phys., 396, 400–415, 2019. a
Rabier, F., Järvinen, H., Klinker, E., Mahfouf, J.-F., and Simmons, A.: The ECMWF operational implementation of four-dimensional variational assimilation. I: Experimental results with simplified physics, Q. J. Roy. Meteor. Soc., 126, 1143–1170, 2000. a
Rabin, J., Peyré, G., Delon, J., and Bernot, M.: Wasserstein barycenter and its application to texture mixing, in: International Conference on Scale Space and Variational Methods in Computer Vision, Springer,
435–446, ISBN: 9783642247859, 2011. a
Rao, C. R., Rao, C. R., Statistiker, M., Rao, C. R., and Rao, C. R.: Linear statistical inference and its applications, Wiley New York, vol. 2, ISBN: 9780471708230, 1973. a
Reich, S.: A nonparametric ensemble transform method for Bayesian inference, SIAM J. Sci. Comput., 35, A2013–A2024, 2013. a
Reichle, R. H., McLaughlin, D. B., and Entekhabi, D.: Hydrologic data assimilation with the ensemble Kalman filter, Mon. Weather Rev., 130, 103–114, 2002. a
Reichle, R. H., Koster, R. D., Dong, J., and Berg, A. A.: Global Soil Moisture from Satellite Observations, Land Surface Models, and Ground Data: Implications for Data Assimilation, J. Hydrometeorol., 5, 430–442, https://doi.org/10.1175/1525-7541(2004)005<0430:GSMFSO>2.0.CO;2, 2004. a
Runge, C.: Über die numerische Auflösung von Differentialgleichungen, Mathematische Annalen, 46, 167–178, 1895 (in German). a
Shen, Z. and Tang, Y.: A modified ensemble Kalman particle filter for non-Gaussian systems with nonlinear measurement functions, J. Adv. Model. Earth Sy., 7, 50–66, 2015. a
Sinkhorn, R.: Diagonal Equivalence to Matrices with Prescribed Row and Column Sums, Am. Math. Mon., 74, 402–405, https://doi.org/10.2307/2314570, 1967. a, b
Spantini, A., Baptista, R., and Marzouk, Y.: Coupling techniques for nonlinear ensemble filtering, arXiv [preprint], arXiv:1907.00389, 30 June 2019. a, b, c
Spiller, E. T., Budhiraja, A., Ide, K., and Jones, C. K.: Modified particle filter methods for assimilating Lagrangian data into a point-vortex model, Physica D, 237, 1498–1506, 2008. a
Srivastava, S., Li, C., and Dunson, D. B.: Scalable Bayes via barycenter in Wasserstein space, J. Mach. Learn. Res., 19, 312–346, 2018. a
Tagade, P. and Ravela, S.: On a quadratic information measure for data assimilation, in: 2014 American Control Conference, IEEE, 598–603, https://doi.org/10.1109/ACC.2014.6859127, 2014. a
Tamang, S. K., Ebtehaj, A., van Leeuwen, P. J., Zou, D., and Lerman, G.: Ensemble Riemannian data assimilation over the Wasserstein space, Nonlin. Processes Geophys., 28, 295–309, https://doi.org/10.5194/npg-28-295-2021, 2021. a, b, c, d
tamangsk: tamangsk/EnRDA: Ensemble Riemannian Data Assimilation, Version v1.0, Zenodo [code], https://doi.org/10.5281/zenodo.5047392, 2021 (data available at:
https://github.com/tamangsk/EnRDA, last access: last access: 9 February 2022). a
Tang, Y., Deng, Z., Manoj, K., and Chen, D.: A practical scheme of the sigma-point Kalman filter for high-dimensional systems, J. Adv. Model. Earth Sy., 6, 21–37, 2014. a
Tian, X., Zhang, H., Feng, X., and Xie, Y.: Nonlinear least squares En4DVar to 4DEnVar methods for data assimilation: Formulation, analysis, and preliminary evaluation, Mon. Weather Rev., 146, 77–93, 2018. a
Tippett, M. K., Anderson, J. L., Bishop, C. H., Hamill, T. M., and Whitaker, J. S.: Ensemble square root filters, Mon. Weather Rev., 131, 1485–1490, 2003. a
Trevisan, A. and Palatella, L.: On the Kalman Filter error covariance collapse into the unstable subspace, Nonlin. Processes Geophys., 18, 243–250, https://doi.org/10.5194/npg-18-243-2011, 2011. a
Van Leeuwen, P. J.: Particle filtering in geophysical systems, Mon. Weather
Rev., 137, 4089–4114, 2009. a
Van Leeuwen, P. J.: A consistent interpretation of the stochastic version of the Ensemble Kalman Filter, Q. J. Roy. Meteor. Soc., 146, 2815–2825, 2020. a
Villani, C.: Topics in optimal transportation, American Mathematical Soc., Providence, RI, Volume 58, https://doi.org/10.1090/gsm/058, 2003. a, b, c
Villani, C.: Optimal transport: old and new, Springer Science & Business Media, vol. 338, ISBN 9783662501801, 2008. a
Vissio, G., Lembo, V., Lucarini, V., and Ghil, M.: Evaluating the performance of climate models based on Wasserstein distance, Geophys. Res. Lett., 47, e2020GL089385, https://doi.org/10.1029/2020GL089385, 2020. a
Yang, Y. and Engquist, B.: Analysis of optimal transport and related misfit functions in full-waveform inversion, Geophysics, 83, A7–A12, 2018. a
Yang, Y., Engquist, B., Sun, J., and Hamfeldt, B. F.: Application of optimal transport and the quadratic Wasserstein metric to full-waveform inversion, Geophysics, 83, R43–R62, 2018. a
Yong, P., Huang, J., Li, Z., Liao, W., and Qu, L.: Least-squares reverse time migration via linearized waveform inversion using a Wasserstein metric, Geophysics, 84, S411–S423, 2019. a
Zupanski, M.: Regional 4-Dimensional Variational Data Assimilation in a Quasi-Operational Forecasting Environment, Mon. Weather Rev., 121, 2396–2408, 1993. a
Short summary
The outputs from Earth system models are optimally combined with satellite observations to produce accurate forecasts through a process called data assimilation. Many existing data assimilation methodologies have some assumptions regarding the shape of the probability distributions of model output and observations, which results in forecast inaccuracies. In this paper, we test the effectiveness of a newly proposed methodology that relaxes such assumptions about high-dimensional models.
The outputs from Earth system models are optimally combined with satellite observations to...
