Articles | Volume 31, issue 3
https://doi.org/10.5194/npg-31-409-2024
© Author(s) 2024. This work is distributed under
the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
https://doi.org/10.5194/npg-31-409-2024
© Author(s) 2024. This work is distributed under
the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
Research article
| Highlight paper
| 
19 Sep 2024
Research article | Highlight paper |
| 19 Sep 2024
| 19 Sep 2024
Representation learning with unconditional denoising diffusion models for dynamical systems
Tobias Sebastian Finn
CORRESPONDING AUTHOR
CEREA, École des Ponts and EDF R&D, Île-de-France, France
Lucas Disson
CEREA, École des Ponts and EDF R&D, Île-de-France, France
Alban Farchi
CEREA, École des Ponts and EDF R&D, Île-de-France, France
Marc Bocquet
CEREA, École des Ponts and EDF R&D, Île-de-France, France
Charlotte Durand
CEREA, École des Ponts and EDF R&D, Île-de-France, France
Related authors
Wenbo Yu, Anirbit Ghosh, Tobias Sebastian Finn, Rossella Arcucci, Marc Bocquet, and Sibo Cheng
EGUsphere, https://doi.org/10.5194/egusphere-2025-2836, https://doi.org/10.5194/egusphere-2025-2836, 2025
Short summary
Short summary
We introduce the first denoising diffusion model for wildfire spread prediction, a new kind of generative AI model that learns to simulate fires not just as one fixed outcome, but as a range of possible scenarios. This allows us to capture the inherent uncertainty of wildfire dynamics. Our model produces ensembles of forecasts that reflect physically meaningful distributions of where fire might go next.
Charlotte Durand, Tobias Sebastian Finn, Alban Farchi, Marc Bocquet, Julien Brajard, and Laurent Bertino
EGUsphere, https://doi.org/10.5194/egusphere-2024-4028, https://doi.org/10.5194/egusphere-2024-4028, 2025
Short summary
Short summary
This paper presents a four-dimensional variational data assimilation system based on a neural network emulator for sea-ice thickness, learned from neXtSIM simulation outputs. Testing with simulated and real observation retrievals, the system improves forecasts and bias error, performing comparably to operational methods, demonstrating the promise of sea-ice data-driven data assimilation systems.
Yumeng Chen, Polly Smith, Alberto Carrassi, Ivo Pasmans, Laurent Bertino, Marc Bocquet, Tobias Sebastian Finn, Pierre Rampal, and Véronique Dansereau
The Cryosphere, 18, 2381–2406, https://doi.org/10.5194/tc-18-2381-2024, https://doi.org/10.5194/tc-18-2381-2024, 2024
Short summary
Short summary
We explore multivariate state and parameter estimation using a data assimilation approach through idealised simulations in a dynamics-only sea-ice model based on novel rheology. We identify various potential issues that can arise in complex operational sea-ice models when model parameters are estimated. Even though further investigation will be needed for such complex sea-ice models, we show possibilities of improving the observed and the unobserved model state forecast and parameter accuracy.
Charlotte Durand, Tobias Sebastian Finn, Alban Farchi, Marc Bocquet, Guillaume Boutin, and Einar Ólason
The Cryosphere, 18, 1791–1815, https://doi.org/10.5194/tc-18-1791-2024, https://doi.org/10.5194/tc-18-1791-2024, 2024
Short summary
Short summary
This paper focuses on predicting Arctic-wide sea-ice thickness using surrogate modeling with deep learning. The model has a predictive power of 12 h up to 6 months. For this forecast horizon, persistence and daily climatology are systematically outperformed, a result of learned thermodynamics and advection. Consequently, surrogate modeling with deep learning proves to be effective at capturing the complex behavior of sea ice.
Tobias Sebastian Finn, Charlotte Durand, Alban Farchi, Marc Bocquet, Yumeng Chen, Alberto Carrassi, and Véronique Dansereau
The Cryosphere, 17, 2965–2991, https://doi.org/10.5194/tc-17-2965-2023, https://doi.org/10.5194/tc-17-2965-2023, 2023
Short summary
Short summary
We combine deep learning with a regional sea-ice model to correct model errors in the sea-ice dynamics of low-resolution forecasts towards high-resolution simulations. The combined model improves the forecast by up to 75 % and thereby surpasses the performance of persistence. As the error connection can additionally be used to analyse the shortcomings of the forecasts, this study highlights the potential of combined modelling for short-term sea-ice forecasting.
Tobias Sebastian Finn, Gernot Geppert, and Felix Ament
Hydrol. Earth Syst. Sci. Discuss., https://doi.org/10.5194/hess-2020-672, https://doi.org/10.5194/hess-2020-672, 2021
Revised manuscript not accepted
Short summary
Short summary
Through the lens of recent developments in hydrological modelling and data assimilation, we hourly update the soil moisture with ensemble data assimilation and sparse 2-metre-temperature observations in a coupled limited area model system. In idealized experiments, we improve the soil moisture analysis by coupled data assimilation across the atmosphere-land interface. We conclude that we can merge the separated assimilation cycles for the atmosphere and land surface into one single cycle.
Wenbo Yu, Anirbit Ghosh, Tobias Sebastian Finn, Rossella Arcucci, Marc Bocquet, and Sibo Cheng
EGUsphere, https://doi.org/10.5194/egusphere-2025-2836, https://doi.org/10.5194/egusphere-2025-2836, 2025
Short summary
Short summary
We introduce the first denoising diffusion model for wildfire spread prediction, a new kind of generative AI model that learns to simulate fires not just as one fixed outcome, but as a range of possible scenarios. This allows us to capture the inherent uncertainty of wildfire dynamics. Our model produces ensembles of forecasts that reflect physically meaningful distributions of where fire might go next.
Joffrey Dumont Le Brazidec, Pierre Vanderbecken, Alban Farchi, Grégoire Broquet, Gerrit Kuhlmann, and Marc Bocquet
Geosci. Model Dev., 18, 3607–3622, https://doi.org/10.5194/gmd-18-3607-2025, https://doi.org/10.5194/gmd-18-3607-2025, 2025
Short summary
Short summary
We developed a deep learning method to estimate CO2 emissions from power plants using satellite images. Trained and validated on simulated data, our model accurately predicts emissions despite challenges like cloud cover. When applied to real OCO3 satellite images, the results closely match reported emissions. This study shows that neural networks trained on simulations can effectively analyse real satellite data, offering a new way to monitor CO2 emissions from space.
Charlotte Durand, Tobias Sebastian Finn, Alban Farchi, Marc Bocquet, Julien Brajard, and Laurent Bertino
EGUsphere, https://doi.org/10.5194/egusphere-2024-4028, https://doi.org/10.5194/egusphere-2024-4028, 2025
Short summary
Short summary
This paper presents a four-dimensional variational data assimilation system based on a neural network emulator for sea-ice thickness, learned from neXtSIM simulation outputs. Testing with simulated and real observation retrievals, the system improves forecasts and bias error, performing comparably to operational methods, demonstrating the promise of sea-ice data-driven data assimilation systems.
Simon Driscoll, Alberto Carrassi, Julien Brajard, Laurent Bertino, Einar Ólason, Marc Bocquet, and Amos Lawless
EGUsphere, https://doi.org/10.5194/egusphere-2024-2476, https://doi.org/10.5194/egusphere-2024-2476, 2024
Short summary
Short summary
The formation and evolution of sea ice melt ponds (ponds of melted water) are complex, insufficiently understood and represented in models with considerable uncertainty. These uncertain representations are not traditionally included in climate models potentially causing the known underestimation of sea ice loss in climate models. Our work creates the first observationally based machine learning model of melt ponds that is also a ready and viable candidate to be included in climate models.
Marc Bocquet, Pierre J. Vanderbecken, Alban Farchi, Joffrey Dumont Le Brazidec, and Yelva Roustan
Nonlin. Processes Geophys., 31, 335–357, https://doi.org/10.5194/npg-31-335-2024, https://doi.org/10.5194/npg-31-335-2024, 2024
Short summary
Short summary
A novel approach, optimal transport data assimilation (OTDA), is introduced to merge DA and OT concepts. By leveraging OT's displacement interpolation in space, it minimises mislocation errors within DA applied to physical fields, such as water vapour, hydrometeors, and chemical species. Its richness and flexibility are showcased through one- and two-dimensional illustrations.
Yumeng Chen, Polly Smith, Alberto Carrassi, Ivo Pasmans, Laurent Bertino, Marc Bocquet, Tobias Sebastian Finn, Pierre Rampal, and Véronique Dansereau
The Cryosphere, 18, 2381–2406, https://doi.org/10.5194/tc-18-2381-2024, https://doi.org/10.5194/tc-18-2381-2024, 2024
Short summary
Short summary
We explore multivariate state and parameter estimation using a data assimilation approach through idealised simulations in a dynamics-only sea-ice model based on novel rheology. We identify various potential issues that can arise in complex operational sea-ice models when model parameters are estimated. Even though further investigation will be needed for such complex sea-ice models, we show possibilities of improving the observed and the unobserved model state forecast and parameter accuracy.
Charlotte Durand, Tobias Sebastian Finn, Alban Farchi, Marc Bocquet, Guillaume Boutin, and Einar Ólason
The Cryosphere, 18, 1791–1815, https://doi.org/10.5194/tc-18-1791-2024, https://doi.org/10.5194/tc-18-1791-2024, 2024
Short summary
Short summary
This paper focuses on predicting Arctic-wide sea-ice thickness using surrogate modeling with deep learning. The model has a predictive power of 12 h up to 6 months. For this forecast horizon, persistence and daily climatology are systematically outperformed, a result of learned thermodynamics and advection. Consequently, surrogate modeling with deep learning proves to be effective at capturing the complex behavior of sea ice.
Joffrey Dumont Le Brazidec, Pierre Vanderbecken, Alban Farchi, Grégoire Broquet, Gerrit Kuhlmann, and Marc Bocquet
Geosci. Model Dev., 17, 1995–2014, https://doi.org/10.5194/gmd-17-1995-2024, https://doi.org/10.5194/gmd-17-1995-2024, 2024
Short summary
Short summary
Our research presents an innovative approach to estimating power plant CO2 emissions from satellite images of the corresponding plumes such as those from the forthcoming CO2M satellite constellation. The exploitation of these images is challenging due to noise and meteorological uncertainties. To overcome these obstacles, we use a deep learning neural network trained on simulated CO2 images. Our method outperforms alternatives, providing a positive perspective for the analysis of CO2M images.
Tobias Sebastian Finn, Charlotte Durand, Alban Farchi, Marc Bocquet, Yumeng Chen, Alberto Carrassi, and Véronique Dansereau
The Cryosphere, 17, 2965–2991, https://doi.org/10.5194/tc-17-2965-2023, https://doi.org/10.5194/tc-17-2965-2023, 2023
Short summary
Short summary
We combine deep learning with a regional sea-ice model to correct model errors in the sea-ice dynamics of low-resolution forecasts towards high-resolution simulations. The combined model improves the forecast by up to 75 % and thereby surpasses the performance of persistence. As the error connection can additionally be used to analyse the shortcomings of the forecasts, this study highlights the potential of combined modelling for short-term sea-ice forecasting.
Joffrey Dumont Le Brazidec, Pierre Vanderbecken, Alban Farchi, Marc Bocquet, Jinghui Lian, Grégoire Broquet, Gerrit Kuhlmann, Alexandre Danjou, and Thomas Lauvaux
Geosci. Model Dev., 16, 3997–4016, https://doi.org/10.5194/gmd-16-3997-2023, https://doi.org/10.5194/gmd-16-3997-2023, 2023
Short summary
Short summary
Monitoring of CO2 emissions is key to the development of reduction policies. Local emissions, from cities or power plants, may be estimated from CO2 plumes detected in satellite images. CO2 plumes generally have a weak signal and are partially concealed by highly variable background concentrations and instrument errors, which hampers their detection. To address this problem, we propose and apply deep learning methods to detect the contour of a plume in simulated CO2 satellite images.
Pierre J. Vanderbecken, Joffrey Dumont Le Brazidec, Alban Farchi, Marc Bocquet, Yelva Roustan, Élise Potier, and Grégoire Broquet
Atmos. Meas. Tech., 16, 1745–1766, https://doi.org/10.5194/amt-16-1745-2023, https://doi.org/10.5194/amt-16-1745-2023, 2023
Short summary
Short summary
Instruments dedicated to monitoring atmospheric gaseous compounds from space will provide images of urban-scale plumes. We discuss here the use of new metrics to compare observed plumes with model predictions that will be less sensitive to meteorology uncertainties. We have evaluated our metrics on diverse plumes and shown that by eliminating some aspects of the discrepancies, they are indeed less sensitive to meteorological variations.
Joffrey Dumont Le Brazidec, Marc Bocquet, Olivier Saunier, and Yelva Roustan
Geosci. Model Dev., 16, 1039–1052, https://doi.org/10.5194/gmd-16-1039-2023, https://doi.org/10.5194/gmd-16-1039-2023, 2023
Short summary
Short summary
When radionuclides are released into the atmosphere, the assessment of the consequences depends on the evaluation of the magnitude and temporal evolution of the release, which can be highly variable as in the case of Fukushima Daiichi.
Here, we propose Bayesian inverse modelling methods and the reversible-jump Markov chain Monte Carlo technique, which allows one to evaluate the temporal variability of the release and to integrate different types of information in the source reconstruction.
Colin Grudzien and Marc Bocquet
Geosci. Model Dev., 15, 7641–7681, https://doi.org/10.5194/gmd-15-7641-2022, https://doi.org/10.5194/gmd-15-7641-2022, 2022
Short summary
Short summary
Iterative optimization techniques, the state of the art in data assimilation, have largely focused on extending forecast accuracy to moderate- to long-range forecast systems. However, current methodology may not be cost-effective in reducing forecast errors in online, short-range forecast systems. We propose a novel optimization of these techniques for online, short-range forecast cycles, simultaneously providing an improvement in forecast accuracy and a reduction in the computational cost.
Joffrey Dumont Le Brazidec, Marc Bocquet, Olivier Saunier, and Yelva Roustan
Atmos. Chem. Phys., 21, 13247–13267, https://doi.org/10.5194/acp-21-13247-2021, https://doi.org/10.5194/acp-21-13247-2021, 2021
Short summary
Short summary
The assessment of the environmental consequences of a radionuclide release depends on the estimation of its source. This paper aims to develop inverse Bayesian methods which combine transport models with measurements, in order to reconstruct the ensemble of possible sources.
Three methods to quantify uncertainties based on the definition of probability distributions and the physical models are proposed and evaluated for the case of 106Ru releases over Europe in 2017.
Tobias Sebastian Finn, Gernot Geppert, and Felix Ament
Hydrol. Earth Syst. Sci. Discuss., https://doi.org/10.5194/hess-2020-672, https://doi.org/10.5194/hess-2020-672, 2021
Revised manuscript not accepted
Short summary
Short summary
Through the lens of recent developments in hydrological modelling and data assimilation, we hourly update the soil moisture with ensemble data assimilation and sparse 2-metre-temperature observations in a coupled limited area model system. In idealized experiments, we improve the soil moisture analysis by coupled data assimilation across the atmosphere-land interface. We conclude that we can merge the separated assimilation cycles for the atmosphere and land surface into one single cycle.
Cited articles
Alain, G. and Bengio, Y.: What Regularized Auto-Encoders Learn from the Data-Generating Distribution, J. Mach. Learn. Res., 15, 3563–3593, 2014. a
Arcomano, T., Szunyogh, I., Pathak, J., Wikner, A., Hunt, B. R., and Ott, E.: A Machine Learning-Based Global Atmospheric Forecast Model, Geophys. Res. Lett., 47, e2020GL087776, https://doi.org/10.1029/2020GL087776, 2020. a
Arnold, H. M., Moroz, I. M., and Palmer, T. N.: Stochastic Parametrizations and Model Uncertainty in the Lorenz '96 System, Philos. T. Roy. Soc. A, 371, 20110479, https://doi.org/10.1098/rsta.2011.0479, 2013. a
Bao, F., Zhang, Z., and Zhang, G.: A Score-based Nonlinear Filter for Data Assimilation, arXiv [preprint], https://doi.org/10.48550/arXiv.2306.09282, 2023. a
Baranchuk, D., Rubachev, I., Voynov, A., Khrulkov, V., and Babenko, A.: Label-Efficient Semantic Segmentation with Diffusion Models, arXiv [preprint], https://doi.org/10.48550/arXiv.2112.03126, 2022. a
Bauer, P., Thorpe, A., and Brunet, G.: The Quiet Revolution of Numerical Weather Prediction, Nature, 525, 47–55, https://doi.org/10.1038/nature14956, 2015. a
Bauer, P., Dueben, P. D., Hoefler, T., Quintino, T., Schulthess, T. C., and Wedi, N. P.: The Digital Revolution of Earth-system Science, Nat. Comput. Sci., 1, 104–113, https://doi.org/10.1038/s43588-021-00023-0, 2021a. a
Bauer, P., Stevens, B., and Hazeleger, W.: A Digital Twin of Earth for the Green Transition, Nat. Clim. Change, 11, 80–83, https://doi.org/10.1038/s41558-021-00986-y, 2021b. a
Bengio, Y., Yao, L., Alain, G., and Vincent, P.: Generalized Denoising Auto-Encoders as Generative Models, in: Advances in Neural Information Processing Systems, vol. 26, Curran Associates, Inc., ISBN 9781713845393, 2013. a
Bishop, C. H., Etherton, B. J., and Majumdar, S. J.: Adaptive Sampling with the Ensemble Transform Kalman Filter. Part I: Theoretical Aspects, Mon. Weather Rev., 129, 420–436, https://doi.org/10.1175/1520-0493(2001)129<0420:ASWTET>2.0.CO;2, 2001. a
Bocquet, M.: Ensemble Kalman filtering without the intrinsic need for inflation, Nonlin. Processes Geophys., 18, 735–750, https://doi.org/10.5194/npg-18-735-2011, 2011. a, b
Bonavita, M. and Laloyaux, P.: Machine Learning for Model Error Inference and Correction, J. Adv. Model. Earth Sy., 12, e2020MS002232, https://doi.org/10.1029/2020MS002232, 2020. a
Bortoli, V. D., Thornton, J., Heng, J., and Doucet, A.: Diffusion schrödinger bridge with applications to score-based generative modeling, Advances in neural information processing systems, Curran Associates, vol. 34, 17695–17709, ISBN 9781713845393, 2021. a
Buizza, R., Houtekamer, P. L., Pellerin, G., Toth, Z., Zhu, Y., and Wei, M.: A Comparison of the ECMWF, MSC, and NCEP Global Ensemble Prediction Systems, Mon. Weather Rev., 133, 1076–1097, https://doi.org/10.1175/MWR2905.1, 2005. a
Cachay, S. R., Zhao, B., James, H., and Yu, R.: DYffusion: A Dynamics-informed Diffusion Model for Spatiotemporal Forecasting, arXiv [preprint], https://doi.org/10.48550/arXiv.2306.01984, 2023. a
Chen, T., Kornblith, S., Norouzi, M., and Hinton, G.: A Simple Framework for Contrastive Learning of Visual Representations, in: Proceedings of the 37th International Conference on Machine Learning, 1597–1607, PMLR, 119, ISSN 2640-3498, 2020. a
Chen, T., Liu, G.-H., and Theodorou, E. A.: Likelihood Training of Schrödinger Bridge Using Forward-Backward SDEs Theory, arXiv [preprint], https://doi.org/10.48550/arXiv.2110.11291, 2023. a
Chen, T.-C., Penny, S. G., Whitaker, J. S., Frolov, S., Pincus, R., and Tulich, S.: Correcting Systematic and State-Dependent Errors in the NOAA FV3-GFS Using Neural Networks, J. Adv. Model. Earth Sy., 14, e2022MS003309, https://doi.org/10.1029/2022MS003309, 2022. a
De Bortoli, V.: Convergence of Denoising Diffusion Models under the Manifold Hypothesis, arXiv [preprint], https://doi.org/10.48550/arXiv.2208.05314, 2022. a
Demaeyer, J., Penny, S. G., and Vannitsem, S.: Identifying Efficient Ensemble Perturbations for Initializing Subseasonal-To-Seasonal Prediction, J. Adv. Model. Earth Sy., 14, e2021MS002828, https://doi.org/10.1029/2021MS002828, 2022. a
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K.: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, in: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Association for Computational Linguistics, Minneapolis, Minnesota, 4171–4186, https://doi.org/10.18653/v1/N19-1423, 2019. a
Dockhorn, T., Vahdat, A., and Kreis, K.: Score-Based Generative Modeling with Critically-Damped Langevin Diffusion, arXiv [preprint], https://doi.org/10.48550/arXiv.2112.07068, 2022. a
Dong, L., Yang, N., Wang, W., Wei, F., Liu, X., Wang, Y., Gao, J., Zhou, M., and Hon, H.-W.: Unified Language Model Pre-training for Natural Language Understanding and Generation, in: Advances in Neural Information Processing Systems, Curran Associates, Inc., vol. 32, ISBN 9781713807933, 2019. a
Efron, B.: Tweedie's Formula and Selection Bias, J. Am. Stat. Assoc., 106, 1602–1614, 2011. a
Evensen, G.: The Ensemble Kalman Filter: Theoretical Formulation and Practical Implementation, Ocean Dynam., 53, 343–367, https://doi.org/10.1007/s10236-003-0036-9, 2003. a, b
Falcon, W., Borovec, J., Wälchli, A., Eggert, N., Schock, J., Jordan, J., Skafte, N., Ir1dXD, Bereznyuk, V., Harris, E., Murrell, T., Yu, P., Præsius, S., Addair, T., Zhong, J., Lipin, D., Uchida, S., Bapat, S., Schröter, H., Dayma, B., Karnachev, A., Kulkarni, A., Komatsu, S., Martin.B, SCHIRATTI, J.-B., Mary, H., Byrne, D., Eyzaguirre, C., cinjon, and Bakhtin, A.: PyTorchLightning: 0.7.6 Release, Zenodo [code], https://doi.org/10.5281/zenodo.3828935, 2020. a
Farchi, A., Laloyaux, P., Bonavita, M., and Bocquet, M.: Using Machine Learning to Correct Model Error in Data Assimilation and Forecast Applications, Q. J. Roy. Meteor. Soc., 147, 3067–3084, https://doi.org/10.1002/qj.4116, 2021. a
Finn, T.: Ddm-attractor (Version initial_submission), Zenodo [code], https://doi.org/10.5281/zenodo.8406184, 2023. a, b
Finn, T. S., Durand, C., Farchi, A., Bocquet, M., Chen, Y., Carrassi, A., and Dansereau, V.: Deep learning subgrid-scale parametrisations for short-term forecasting of sea-ice dynamics with a Maxwell elasto-brittle rheology, The Cryosphere, 17, 2965–2991, https://doi.org/10.5194/tc-17-2965-2023, 2023. a
Gagne II, D. J., Christensen, H. M., Subramanian, A. C., and Monahan, A. H.: Machine Learning for Stochastic Parameterization: Generative Adversarial Networks in the Lorenz '96 Model, J. Adv. Model. Earth Sy., 12, e2019MS001896, https://doi.org/10.1029/2019MS001896, 2020. a
Grooms, I.: Analog Ensemble Data Assimilation and a Method for Constructing Analogs with Variational Autoencoders, Q. J. Roy. Meteor. Soc., 147, 139–149, https://doi.org/10.1002/qj.3910, 2021. a
Grooms, I., Renaud, C., Stanley, Z., and Yang, L. M.: Analog Ensemble Data Assimilation in a Quasigeostrophic Coupled Model, Q. J. Roy. Meteo. Soc., 149, 1018–1037, https://doi.org/10.1002/qj.4446, 2023. a
Hamill, T. M. and Snyder, C.: A Hybrid Ensemble Kalman Filter–3D Variational Analysis Scheme, Mon. Weather Rev., 128, 2905–2919, https://doi.org/10.1175/1520-0493(2000)128<2905:AHEKFV>2.0.CO;2, 2000. a
He, K., Zhang, X., Ren, S., and Sun, J.: Deep Residual Learning for Image Recognition, arXiv [preprint], https://doi.org/10.48550/arXiv.1512.03385, 2015. a
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, R.: Masked Autoencoders Are Scalable Vision Learners, arXiv [preprint], https://doi.org/10.48550/arXiv.2111.06377, 2021. a
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S.: GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium, arXiv [preprint], https://doi.org/10.48550/arXiv.1706.08500, 2017. a
Ho, J., Jain, A., and Abbeel, P.: Denoising diffusion probabilistic models, in: Advances in neural information processing systems, edited by: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M. F., and Lin, H., Curran Associates, Inc., ISBN 9781713829546, Vol. 33, 6840–6851, 2020. a, b, c, d, e, f, g
Hoffmann, S. and Lessig, C.: AtmoDist: Self-supervised Representation Learning for Atmospheric Dynamics, Environmental Data Sci., 2, e6, https://doi.org/10.1017/eds.2023.1, 2023. a
Holzschuh, B., Vegetti, S., and Thuerey, N.: Solving inverse physics problems with score matching, in: Advances in neural information processing systems, edited by: Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S., Curran Associates, Inc., Vol. 36, 61888–61922, ISBN 9781713899921, 2023. a
Hunt, B. R., Kostelich, E. J., and Szunyogh, I.: Efficient Data Assimilation for Spatiotemporal Chaos: A Local Ensemble Transform Kalman Filter, Physica D, 230, 112–126, https://doi.org/10.1016/j.physd.2006.11.008, 2007. a
Hyvärinen, A.: Estimation of Non-Normalized Statistical Models by Score Matching, J. Mach. Learn. Res., 6, 695–709, 2005. a
Jolicoeur-Martineau, A., Li, K., Piché-Taillefer, R., Kachman, T., and Mitliagkas, I.: Gotta Go Fast When Generating Data with Score-Based Models, arXiv [preprint], https://doi.org/10.48550/arXiv.2105.14080, 2022. a
Kingma, D. P. and Ba, J.: Adam: A Method for Stochastic Optimization, arXiv [preprint], https://doi.org/10.48550/arXiv.1412.6980, 2017. a
Kretschmer, M., Hunt, B. R., and Ott, E.: Data Assimilation Using a Climatologically Augmented Local Ensemble Transform Kalman Filter, Tellus A, 67, 26617, https://doi.org/10.3402/tellusa.v67.26617, 2015. a
Latif, M.: The Roadmap of Climate Models, Nat. Comput. Sci., 2, 536–538, https://doi.org/10.1038/s43588-022-00322-0, 2022. a
Lguensat, R., Tandeo, P., Ailliot, P., Pulido, M., and Fablet, R.: The Analog Data Assimilation, Mon. Weather Rev., 145, 4093–4107, https://doi.org/10.1175/MWR-D-16-0441.1, 2017. a
Li, L., Carver, R., Lopez-Gomez, I., Sha, F., and Anderson, J.: SEEDS: Emulation of Weather Forecast Ensembles with Diffusion Models, arXiv [preprint], https://doi.org/10.48550/arXiv.2306.14066, 2023a. a
Li, X., Feng, M., Ran, Y., Su, Y., Liu, F., Huang, C., Shen, H., Xiao, Q., Su, J., Yuan, S., and Guo, H.: Big Data in Earth System Science and Progress towards a Digital Twin, Nature Rev. Earth Environ., 4, 319–332, https://doi.org/10.1038/s43017-023-00409-w, 2023b. a
Lorenc, A. C.: The Potential of the Ensemble Kalman Filter for NWP – a Comparison with 4D-Var, Q. J. Roy. Meteor. Soc., 129, 3183–3203, https://doi.org/10.1256/qj.02.132, 2003. a
Lorenz, E. N.: Deterministic Nonperiodic Flow, J. Atmos. Sci., 20, 130–141, https://doi.org/10.1175/1520-0469(1963)020<0130:DNF>2.0.CO;2, 1963. a, b
Lu, C., Zhou, Y., Bao, F., Chen, J., Li, C., and Zhu, J.: DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps, arXiv [preprint], https://doi.org/10.48550/arXiv.2206.00927, 2022. a
Luo, C.: Understanding Diffusion Models: A Unified Perspective, arXiv [preprint], https://doi.org/10.48550/arXiv.2208.11970, 2022. a
Luo, G., Dunlap, L., Park, D. H., Holynski, A., and Darrell, T.: Diffusion Hyperfeatures: Searching Through Time and Space for Semantic Correspondence, arXiv [preprint], https://doi.org/10.48550/arXiv.2305.14334, 2023. a
Meng, C., He, Y., Song, Y., Song, J., Wu, J., Zhu, J.-Y., and Ermon, S.: SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations, arXiv [preprint], https://doi.org/10.48550/arXiv.2108.01073, 2022. a, b
Mittal, S., Abstreiter, K., Bauer, S., Schölkopf, B., and Mehrjou, A.: Diffusion Based Representation Learning, in: Proceedings of the 40th International Conference on Machine Learning, PMLR, 202, 24963–24982, ISSN 2640-3498, 2023. a
Molteni, F., Buizza, R., Palmer, T. N., and Petroliagis, T.: The ECMWF Ensemble Prediction System: Methodology and Validation, Q. J. Roy. Meteor. Soc., 122, 73–119, https://doi.org/10.1002/qj.49712252905, 1996. a
Nguyen, T., Brandstetter, J., Kapoor, A., Gupta, J. K., and Grover, A.: ClimaX: A Foundation Model for Weather and Climate, arXiv [preprint], https://doi.org/10.48550/arXiv.2301.10343, 2023. a
Oke, P. R., Allen, J. S., Miller, R. N., Egbert, G. D., and Kosro, P. M.: Assimilation of Surface Velocity Data into a Primitive Equation Coastal Ocean Model, J. Geophys. Res.-Oceans, 107, 5-1–5-25, https://doi.org/10.1029/2000JC000511, 2002. a, b
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S.: PyTorch: An Imperative Style, High-Performance Deep Learning Library, in: Advances in Neural Information Processing Systems 32, edited by: Wallach, H., Larochelle, H., Beygelzimer, A., Alché-Buc, F., Fox, E., and Garnett, R., Curran Associates, Inc., 8024–8035, ISBN 9781713807933, 2019. a
Peebles, W. and Xie, S.: Scalable Diffusion Models with Transformers, arXiv [preprint], https://doi.org/10.48550/arXiv.2212.09748, 2023. a
Perez, E., Strub, F., de Vries, H., Dumoulin, V., and Courville, A.: FiLM: Visual Reasoning with a General Conditioning Layer, arXiv [preprint], https://doi.org/10.48550/arXiv.1709.07871, 2017. a
Price, I., Sanchez-Gonzalez, A., Alet, F., Andersson, T. R., El-Kadi, A., Masters, D., Ewalds, T., Stott, J., Mohamed, S., Battaglia, P., Lam, R., and Willson, M.: GenCast: Diffusion-based Ensemble Forecasting for Medium-Range Weather, arXiv [preprint], https://doi.org/10.48550/arXiv.2312.15796, 2024. a
Radford, A., Narasimhan, K., Salimans, T., and Sutskever, I.: Improving Language Understanding by Generative Pre-Training, https://openai.com/research/language-unsupervised (last access: 18 September 2024), 2018. a
Rahimi, A. and Recht, B.: Random Features for Large-Scale Kernel Machines, in: Advances in Neural Information Processing Systems, 1177–1184, ISBN 9781605603520, 2007. a
Reich, S.: Data Assimilation: The Schrödinger Perspective, Acta Numerica, 28, 635–711, https://doi.org/10.1017/S0962492919000011, 2019. a
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B.: High-Resolution Image Synthesis With Latent Diffusion Models, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA, 2022, 10674–10685, https://doi.org/10.1109/CVPR52688.2022.01042, 2022. a
Rozet, F. and Louppe, G.: Score-Based Data Assimilation, Adv. Neur. Inf., 36, 40521–40541, 2023. a
Salimans, T. and Ho, J.: Progressive Distillation for Fast Sampling of Diffusion Models, arXiv [preprint], https://doi.org/10.48550/arXiv.2202.00512, 2022. a, b, c, d
Scher, S. and Messori, G.: Generalization properties of feed-forward neural networks trained on Lorenz systems, Nonlin. Processes Geophys., 26, 381–399, https://doi.org/10.5194/npg-26-381-2019, 2019. a, b
Schraff, C., Reich, H., Rhodin, A., Schomburg, A., Stephan, K., Periáñez, A., and Potthast, R.: Kilometre-Scale Ensemble Data Assimilation for the COSMO Model (KENDA), Q. J. Roy. Meteor. Soc., 142, 1453–1472, https://doi.org/10.1002/qj.2748, 2016. a
Sohl-Dickstein, J., Weiss, E. A., Maheswaranathan, N., and Ganguli, S.: Deep Unsupervised Learning Using Nonequilibrium Thermodynamics, arXiv [preprint], https://doi.org/10.48550/arXiv.1503.03585, 2015. a
Song, Y., Garg, S., Shi, J., and Ermon, S.: Sliced Score Matching: A Scalable Approach to Density and Score Estimation, in: Proceedings of The 35th Uncertainty in Artificial Intelligence Conference, PMLR, 115, 574–584, ISSN 2640-3498, 2019. a
Sutherland, D. J. and Schneider, J.: On the Error of Random Fourier Features, arXiv [preprint], https://doi.org/10.48550/arXiv.1506.02785, 2015. a
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A.: Going Deeper with Convolutions, arXiv [preprint], https://doi.org/10.48550/arXiv.1409.4842, 2014. a
Tandeo, P., Ailliot, P., and Sévellec, F.: Data-driven reconstruction of partially observed dynamical systems, Nonlin. Processes Geophys., 30, 129–137, https://doi.org/10.5194/npg-30-129-2023, 2023. a
Toth, Z. and Kalnay, E.: Ensemble Forecasting at NMC: The Generation of Perturbations, B. Am. Meteorol. Soc., 74, 2317–2330, https://doi.org/10.1175/1520-0477(1993)074<2317:EFANTG>2.0.CO;2, 1993. a
Van Rossum, G.: Python Tutorial, Technical Report CS-R9526, Tech. rep., Centrum voor Wiskunde en Informatica (CWI), Amsterdam, ISSN 0169-118X, 1995. a
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I.: Attention Is All You Need, arXiv [preprint], arXiv:1706.03762, https://doi.org/10.48550/arXiv.1706.03762, 2017. a, b
Vincent, P.: A Connection Between Score Matching and Denoising Autoencoders, Neural Comput., 23, 1661–1674, https://doi.org/10.1162/NECO_a_00142, 2011. a
Vincent, P., Larochelle, H., Bengio, Y., and Manzagol, P.-A.: Extracting and Composing Robust Features with Denoising Autoencoders, Proceedings of the 25th international conference on Machine learning, 1096–1103, https://doi.org/10.1145/1390156.1390294, 2008. a
Vincent, P., Larochelle, H., Lajoie, I., Bengio, Y., Manzagol, P.-A., and Bottou, L.: Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion., J. Mach. Learn. Res., 11, 3371−-3408, 2010. a
Vlachas, P. R., Pathak, J., Hunt, B. R., Sapsis, T. P., Girvan, M., Ott, E., and Koumoutsakos, P.: Backpropagation Algorithms and Reservoir Computing in Recurrent Neural Networks for the Forecasting of Complex Spatiotemporal Dynamics, Neural Networks, 126, 191–217, https://doi.org/10.1016/j.neunet.2020.02.016, 2020. a
Xiang, W., Yang, H., Huang, D., and Wang, Y.: Denoising Diffusion Autoencoders Are Unified Self-supervised Learners, arXiv [preprint], https://doi.org/10.48550/arXiv.2303.09769, 2023. a
Yang, G. and Sommer, S.: A Denoising Diffusion Model for Fluid Field Prediction, arXiv [preprint], https://doi.org/10.48550/arXiv.2301.11661, 2023. a
Yang, L. M. and Grooms, I.: Machine Learning Techniques to Construct Patched Analog Ensembles for Data Assimilation, J. Comput. Phys., 443, 110532, https://doi.org/10.1016/j.jcp.2021.110532, 2021. a
Yang, X. and Wang, X.: Diffusion Model as Representation Learner, 2023 IEEE/CVF International Conference on Computer Vision (ICCV), Paris, France, 18892–18903, https://doi.org/10.1109/ICCV51070.2023.01736, 2023. a
Zhang, Z., Zhao, Z., and Lin, Z.: Unsupervised Representation Learning from Pre-trained Diffusion Probabilistic Models, Adv. Neur. Inf., 35, 22117–22130, 2022. a
Executive editor
This paper tests the ability of Artificial Intelligence methods, and more specifically Deep Learning, to eliminate the Gaussian noise that disturbs the data of a dynamic system. The authors demonstrate this using a highly chaotic model as a hard test case.
This paper tests the ability of Artificial Intelligence methods, and more specifically Deep...
Short summary
We train neural networks as denoising diffusion models for state generation in the Lorenz 1963 system and demonstrate that they learn an internal representation of the system. We make use of this learned representation and the pre-trained model in two downstream tasks: surrogate modelling and ensemble generation. For both tasks, the diffusion model can outperform other more common approaches. Thus, we see a potential of representation learning with diffusion models for dynamical systems.
We train neural networks as denoising diffusion models for state generation in the Lorenz 1963...
