Implications of model error for numerical climate prediction

Introduction Conclusions References


Introduction
One of the principal aims of numerical climate models is to provide a reliable tool for the prediction of climate change following pre-defined future forcing scenarios.The suitability of numerical climate models to study the climate is based on two assumptions.The first assumption is the existence of a climate attractor.There is no rigorous proof that this attractor exists (e.g.Lorenz, 1991).However, the observations available on long-term records provide some certainty about the validity of this assumption (e.g.Essex et al., 1987).The second assumption is that the solutions provided by a numerical climate model lie on the actual climate attractor, or at least on the projection of the infinite-dimensional climate attractor on the model's phase space.Only under this assumption, numerical climate model solutions can be used Figures to study the properties of the climate under present-day conditions.In order to study climate change, an additional assumption must be made: it must be assumed that the model climate attractor responds in the same way as the actual climate attractor does under changing forcing conditions.The last assumption has been subject to extensive investigation through studies that have shown that, rather than producing new patterns of variability, the effects of anthropogenic forcing project onto already existing patterns of natural variability (e.g.Palmer, 1993b;Corti et al., 1999).
The presence of biases in climate models with respect to observations and reanalysis datasets under present-day conditions (Randall et al., 2007) indicates that the actual climate attractor (as inferred from observations and reanalysis datasets) and the attractors of available climate models are different even if just slightly, i.e. the second assumption is not fully satisfied.While climate models will never be perfect, we can make use of the limit posed by the second assumption to devise useful measures for the assessment of climate models.The relation between errors in the parameterisation of fast physics processes and errors in long-term simulations has been investigated through techniques such as "initial tendencies" (see Klocke and Rodwell, 2013, and references therein).However, there is no mathematically rigorous theory to explain the relationship between phenomena developing in short timescales and observed long term trends.Such a theory would also help to relate errors in weather prediction and biases in climate projections.For example, it would help to explain why climate models exhibit biases in the tilt of cyclone tracks (Zappa et al., 2013) even though they are capable to simulate realistic extratropical cyclones (Catto et al., 2010).
The objective of this contribution is to show the implications of the second assumption for long-term integrations of a "simple" dynamical system in a threedimensional phase space: the Lorenz '63 system (Lorenz, 1963).The Lorenz '63 system has been used as an archetype system in several previous studies of weather and climate (e.g.Palmer, 1993a, b;Mu et al., 2002).Palmer (1993a)  predictability of nonlinear systems.Palmer (1993a) also introduces the concept of state-dependent predictability by showing that the predictability of the Lorenz '63 system strongly depends on the initial position on the Lorenz attractor.Palmer (1993b) showed that the effects of climate change are expected to modify already existing patterns of atmospheric variability.Mu et al. (2002) investigates three problems on predictability related to maximum prediction time, maximum prediction error and maximum admissible errors in initial values and parameter values.Unlike those studies, in which the Lorenz '63 system was used to infer properties of the climate or the properties of climate models separately, in this contribution the Lorenz '63 system is used to investigate relationships between a system and an imperfect model (e.g. a model with a similar structure to that of the system but with different parameters).This is similar to the approach taken by Orrell et al. (2001), who used system/model combinations to investigate shadowing of target orbits in low-and high-dimensional systems.Instead of trajectory shadowing, the focus in this study are the cumulative effects of model error for long-term integrations, so that no orbit could be expected to shadow any target trajectory.
To avoid confusion, in this article the term "prototype system" refers to a system as part of a system/model combination.Thus, a prototype system and its model are both dynamical systems and in this work they will be instances of the Lorenz '63 system, differing only on the values of their parameters.The methodology is fully described in Sect. 2 while the results for the Lorenz '63 system are discussed in Sect.3.
Clearly, there are several important differences between the prototype system/imperfect model combination using the Lorenz '63 system and the combination formed by the climate system and numerical climate models.For example, the Lorenz '63 system is perfectly known whereas our knowledge of the climate system relies on observations which are of limited temporal extent and subject to observational error.Another important difference is that the imperfect model for the Lorenz '63 system (as constructed here) share its dimensionality whereas numerical weather and climate models have necessarily a lower dimensionality than the climate system.Despite these Introduction

Conclusions References
Tables Figures

Back Close
Full and other differences, there are several implications that can be transferred between both systems.These implications are discussed in Sect. 4. Finally, a summary and concluding remarks are given in Sect. 5.

Methodology
The Lorenz '63 system is defined by the equations (Lorenz, 1963) (2) ż = xy − bz. (3) The variables x, y and z define the phase space of the system while σ, b and r are constant parameters.For a range of these parameters the trajectories of the system tend asymptotically towards the well-known two-winged Lorenz attractor.The shape of the attractor depends on the values given to the parameters σ, b, and r.Thus, two fixed points (defined as the points in phase space for which ẋ = 0, ẏ = 0, ż = 0) for r > 1 are located at (x 0 , y 0 , z 0 ) = ± b(r − 1), ± b(r − 1), r − 1 .A third fixed point is located at the origin.The three fixed points are unstable for In this region of the parameter space the system has no other attractors but a strange attractor.
The standard values of these parameters (as used by Lorenz, 1963) are σ = 10, b = 8/3, and r = 28.In this work, the Lorenz '63 system characterized by these parameter values will be regarded as the prototype system.To construct an imperfect model of the prototype system the values of σ = 10 and b = 8/3 will be kept but the value r = 25 will be used instead.By doing so, r c = 24.74 is valid for both the prototype system and the Introduction

Conclusions References
Tables Figures

Back Close
Full imperfect model.However, the position of the fixed points differs between the prototype system and the imperfect model.For the prototype system (±8.49, ±8.49, 27) while for the imperfect model The prototype system and the imperfect model have been initialised with the same random initial conditions drawn from a uniform distribution between 0 and 1 for the three phase-space variables.Then, the system and the imperfect model have been integrated for 100 t.u.(1 t.u.= 1 time unit) with a sampling rate of ∆t = 0.01 t.u.The first 20 t.u.have been discarded from both orbits to eliminate initial transients.The remaining points in each integration are considered here as the attractors of the system and the imperfect model.
The separation between any two points in phase space is measured throughout this work using the Euclidean distance (which is referred to simply as distance).The distance between the given point and the attractor is defined here as the minimum distance between a given point in phase space and the points in an attractor.

Model error in the prototype system/imperfect model combination
The attractors of the prototype system and the imperfect model are shown in Fig. 2a.The structures of both attractors appear similar, but the size of the imperfect model attractor appears smaller.Thus, the second assumption is not satisfied in this case, i.e. the attractor of the model does not lie on the attractor of the system.
Let us assume that we observe the prototype system at regular intervals (e.g.every Introduction

Conclusions References
Tables Figures

Back Close
Full the imperfect model with respect to the prototype system's orbit moves the forecast trajectory away from the imperfect model's attractor.Figure 2b shows that the distance between observations and the imperfect model's attractor is small but not negligible.
The initialisation of the imperfect model from a point on the prototype system's attractor induces a transient period during which the orbit tends towards the imperfect model's attractor.Second, sensitivity to initial conditions will pull the imperfect model orbit away from the prototype system orbit and into a different segment on the imperfect model's attractor.This effect is particularly evident when the initial conditions are close to the imperfect model attractor, as occurs at t = 85 t.u. in Fig. 2, for example.Figure 2a also shows the orbits of the imperfect model at every forecast cycle projected onto the x subspace (red lines).These orbits appear to closely follow the orbit of the system very well for about 1 t.u.immediately after each initialisation.However, computing the distance between the prototype system's orbit and the imperfect model forecast orbit reveals that the forecast error is actually much larger than the apparent distance in x (red line, Fig. 2c).For comparison, a perfect model, given by a Lorenz '93 system with the same parameter values as the prototype system, was also used to forecast the state of the prototype system.The perfect model was initialised with imperfect initial conditions given by the observations randomly perturbed assuming that the observational error in each variable x, y, and z is independent and normally distributed with standard deviation σ O = 0.1 l.u.(1 l.u.= 1 length unit).The perfect model forecast orbits projected onto the x subspace are also shown in Fig. 2a (grey lines).Even though in this case we should expect the orbits to diverge from the prototype system's orbit due to sensitivity to initial conditions, the divergence appears to be much slower than in the imperfect model case.This effect becomes clearer when looking at the distance between the prototype system's orbit and the perfect model forecast orbits (black lines in Fig. 2c): the perfect model forecast orbits remain close to the prototype system's orbit for about 2 t.u.immediately after each initialisation.After this initial interval, sensitivity to initial conditions takes over and the perfect model forecast orbits move away from the prototype system's orbit.Introduction

Conclusions References
Tables Figures

Back Close
Full In order to show that these results are robust, similar analyses were conducted for an imperfect model with r = 27 and for initial conditions with σ O = 0.2 l.u., σ O = 0.5 l.u. and σ O = 1 l.u. Figure 3a shows the evolution of the probability density functions (PDFs), represented by median and interquartile range, of the distance between the prototype system's orbit and the forecasts obtained with these models.For long lead times (i.e.t L = 5 t.u.) the effects of a relatively large observational error (e.g.σ O = 1 l.u.) and a relatively small model error (e.g.r = 27) are apparently similar (see Fig. 3a).At shorter lead times, however, there are important behaviour differences between models.The two imperfect models show a short period of very fast divergence from the prototype system's orbit followed by a plateau and a second period of fast divergence.The first period of fast divergence is induced by the approach of the imperfect model forecast orbit to the imperfect model's attractor.In contrast, the forecasts produced by a perfect model with imperfect initial conditions show periods of slow divergence from the prototype system's for short lead times.In fact, Fig. 3b, which shows the rate of change of the median of the distance between the prototype system's orbit and model orbits with respect to forecast lead time, reveals that the perfect model runs undergo a short period during which the distance between model orbit and prototype system's orbit tends to decrease.Indeed, this shrinking period occurs as a consequence of the prototype system's orbit being part of the prototype system's attractor and having initial conditions with finite observational error.Figure 3b provides a summary of the difference between imperfect models with perfect initial conditions and perfect models with imperfect initial conditions.At the beginning of the forecast cycle, the imperfect models are characterised by a positive and comparatively large rate of change in the median of the distance between the orbits of the prototype system and the model with respect to forecast lead time; on the other hand, the perfect models are characterised by a negative and comparatively small rate of change in the same variable.
One might argue, by pointing at the gray and red lines in Fig. 3a, that having small model error (r = 27) or large observational error (σ O = 1 l.u.) leads to very similar model behaviour.Following this line of thought, one could try to eliminate the initial period Introduction

Conclusions References
Tables Figures

Back Close
Full of fast divergence in the imperfect model by following a suitable strategy to project "unbalanced" initial conditions onto the surface on which the attractor of the imperfect model evolves (e.g, the strategy suggested by Anderson, 1995).However, focusing on the distance between orbits alone gives only a partial view of the situation: two points could be at a similar distance from a third point, and nevertheless be placed at very dissimilar locations.Figure 4 highlights a different aspect of the comparison between perfect and imperfect models: the location of the attractor in phase space.This aspect is fundamental for climate prediction, in which we are not interested in predicting the state of a system at a particular time, but in the statistical properties of the system during a time interval of a given duration at a particular starting time.
Figure 4 shows the evolution of the PDF of z, represented by median and interquartile range and computed using forecasts, as forecast lead time increases.For comparison, it also shows these same quantities computed using the attractors of the prototype system and the imperfect model shown in Fig. 1.At t L = 0 t.u., both perfect and imperfect models produce very similar statistics to those produced by the prototype system.The small difference between statistics at t L = 0 t.u. is due to the difference in sample size between the prototype system's attractor and the forecasts.As forecast lead time increases the differences between perfect and imperfect model become more apparent.The imperfect model forecast orbits tend to the imperfect model's attractor so that in less than about 0.5 t.u. the statistics produced by the imperfect model for t L > 0.5 t.u. are closer to those produced by the imperfect model's attractor than to those produced by the prototype system's attractor.In contrast, the perfect model forecast orbits tend towards the prototype system's attractor.Therefore, the statistics produced by the perfect model forecasts remain around those produced by the prototype system throughout the whole forecasting cycle even though this model was initialised with imperfect initial conditions.These results show a fundamental property of a perfect model: if a model is a perfect model, then the attractor, reconstructed by sampling a collection of initialised model orbits (forecast orbits), will be invariant to forecast lead time, provided two conditions: (1) that the model is initialised with good estimates of the Introduction

Conclusions References
Tables Figures

Back Close
Full system's true state based on observations and (2) that the collection of forecast orbits is a representative sample of the region in phase space accessible to the system.This property marks a clear difference between perfect and imperfect models and provides an alternative means for climate model evaluation.Furthermore, it has the advantage over the distance between the orbits of the prototype system and the models that no prior knowledge of the prototype system's orbit is required (apart from initial states at suitable times).Moreover, it avoids the false impression that a perfect model and an imperfect model exhibit similar behaviour.

Attractor reconstruction
Reconstructing even part of the attractor of a system is equivalent to knowing at least part of its climate.It would be desirable to reconstruct the full climate attractor in order to completely know the climate.However, this task is impossible given the very large dimensionality of the climate system.In principle, it would be enough to collect a sufficiently large number of observations to be able to represent the system's attractor in phase space and infer its properties.However, if the only source of data available was the imperfect model, then the most we could achieve would be to represent the imperfect model's attractor in model phase space.This is related to the existence of biases in climate models when evaluated against observations and reanalysis datasets (e.g.Kim et al., 2009;Matsueda et al., 2009;Zappa et al., 2013).As discussed in Sect. 1, these biases are an expression of the mismatch between the climate attractor and the attractors of climate models.Evidence of the existence of biases can be found even using short-term forecasts by comparing two different times in a forecast cycle, analysis time (T + 0 d) and T + 15 d, in an analogous way to that used to study the prototype system/imperfect model combination based on the Lorenz '63 system in Sect.3. Figure 5 shows interquartile ranges of daily zonally-averaged 320-K potential

NPGD Introduction Conclusions References
Tables Figures

Back Close
Full vorticity (PV) for the period between December 2009 and February 2010 for these two lead times and for three forecast datasets produced with three different models: (1) the Met Office Global and Regional Ensemble Prediction System (MOGREPS, Bowler et al., 2008), (2) the European Centre for Medium-Range Weather Forecasts (ECMWF) Ensemble Prediction System (EPS, Molteni et al., 1996) and ( 3) the National Centers for Environmental Prediction (NCEP) Global Ensemble Forecast System (GEFS, Toth and Kalnay, 1997).These datasets have been archived by the THORPEX Interactive Grand Global Ensemble (TIGGE, Park et al., 2008).
As shown in Sect.3, if the models were perfect, the statistics between the forecasts at T + 0 d and T + 15 d would be similar or, in the limit of infinitely large samples, the same.However, the three datasets reveal clear statistical differences between analyses and T + 15 d forecasts.It must be noted that even though the three ensemble prediction systems (EPS) produce different statistics at analysis time and at T + 15 d, the deviation shown by the ECMWF EPS (Fig. 5b) seems systematically smaller than that produced by MOGREPS (Fig. 5a) or NCEP GEFS (Fig. 5c).This effect might occur as a result of the optimisation of the ECMWF model for the specific purpose of mediumrange weather prediction.However, this is only one metric and more research would be needed to give a complete comparison between these and other TIGGE models.
There are two potential caveats in these results.The first potential caveat is that the results are shown only for the season December-February (DJF) 2009-2010 in the Northern Hemisphere, which was characterized by exceptional conditions in terms of atmospheric circulation in the North-Atlantic European sector (e.g.Santos et al., 2013).However, five other DJF periods have been analysed (from 2006-2007 to 2011-2012) on both hemispheres and all of them show the same qualitative results.Moreover, recent research performed on this same data confirms the existence of systematic model error in the three datasets (Gray et al., 2014).
The second potential caveat is that only the control members (unperturbed analyses with no stochastic physics included in the forecast model) in each EPS have been considered in this analysis.However, the ensemble members tend to follow the

Conclusions References
Tables Figures

Back Close
Full behaviour of the control member.For example, Fig. 6a shows the 2-PVU contours on the 320-K isentropic surface in the analysis and the control members in five forecasts produced with MOGREPS for the same validation time (00:00 UTC, 25 November 2009) but different lead times (T + 1 d to T + 5 d). Figure 6b shows the 2-PVU contours on the 320-K isentropic surface in the analysis and the ensemble members for the T + 4 d forecast for the same validation time.There are two remarks to make regarding this figure.The first remark is that the apices of the upper-level ridge (over Scandinavia in the analysis) in the control members tend towards the southeast with increasing lead time (Fig. 6a) (Sideri, 2013).The second remark is that the ensemble at the lead time shown (Fig. 6b), and in fact any other between 1 d and 5 d, clusters around the corresponding control member while failing to include the analysis (Sideri, 2013).

Short-term forecast
It has been shown that initialising the imperfect model with perfect initial conditions with respect to the system can be viewed as initialising the model with initial conditions away from its own attractor.This induces a transient period during which the model approaches its own attractor.Data assimilation blends information from the model and observations in order to provide initial conditions for the next forecast.Using data assimilation to initialise a numerical prediction model has a similar effect to initialising the imperfect model with perfect initial conditions by moving the initial model state away from the model's attractor.This induces a transient (spin-up) period until the numerical model reaches a new balance (Daley, 1991).The new balance is achieved when the model's orbit is close to the model's attractor.
The transient period and the subsequent evolution on the model attractor imply divergence between the model's orbit and the true system's orbit.This divergence is not only due to sensitivity to initial conditions.Instead, it is partly due to fundamental differences between the system and the imperfect model.The forecast of the upperlevel ridge on 25 November 2009 introduced in Sect.4.1 provides one example of this model-error related divergence (Fig. 6).As mentioned before, the apices in the Introduction

Conclusions References
Tables Figures

Back Close
Full forecasts tend towards the southeast as lead time increases (Fig. 6a), thus indicating that the model is diverging from the system's orbit.The fact that no member in the ensemble is close to the actual behaviour of the system (Fig. 6b) might be due to the same effect: in this particular event, an ensemble around accurate initial conditions generates an ensemble forecast with every member tending towards the model's attractor and away from the true future state of the system.This occurs even though MOGREPS incorporates a representation of model error variability in the ensemble (Bowler et al., 2008).There are many other examples of this type of behaviour in other models (e.g.Rodwell et al., 2013).One might argue that even though analysis and forecasts diverge from each other, they could still be part of the same attractor.
However, the evidence presented (Fig. 5 and subsequent discussion) strongly suggests that indeed model and system have different attractors.

Summary and concluding remarks
It has been shown that, in the prototype system/imperfect model combination based on the Lorenz '63 system, imperfections in the model translated into differences in attractor structure (fixed points and apparent size) between the system and the imperfect model (Fig. 1).As a result, the second assumption for the suitability of a model (i.e. the assumption that the solutions provided by a model lie on the system's attractor, or at least on the projection of the system's attractor on the model's phase space) was not satisfied.Under these circumstances, even a perfectly accurate initialisation of the system induces a transient period during which the model orbit diverges from the system's orbit and approaches the model attractor (Fig. 2).Thus, the orbit of the model and the actual system's orbit become essentially different.This difference is purely due to model error and not to sensitivity to initial conditions.This was shown through a comparison of two imperfect models initialised with perfect initial conditions and a perfect model initialised with imperfect initial conditions subject to four levels of observational error (Fig. 3).It was shown that, even though at long lead times Introduction

Conclusions References
Tables Figures

Back Close
Full small model error and large observational error produced apparently similar results (Fig. 3a), there were noticeable differences at very short lead times: while imperfect model forecast orbits tend to quickly diverge from the prototype system's orbit, perfect model forecast orbits tend to undergo a short period at the beginning of the forecast cycle during which they approach the prototype system's orbit (Fig. 3b).However, these methods require the prior knowledge of the actual state of the system and its evolution, which is an unaffordable luxury for climate scientists, who are bound to deal with a system of very large dimensionality.
The investigation of the prototypical system/imperfect model combination provides a new, alternative framework for the interpretation of output from numerical climate models with implications for two widely recognized needs in climate science: the need for climate model improvement (e.g.Stevens and Bony, 2013) and the need for new methods for the interpretation of current available climate models when contrasted against observations (e.g.Brands et al., 2012).It has been shown that climate model biases can be interpreted as an expression of a mismatch between the climate system attractor and the numerical climate model attractor.Furthermore, it has been shown that such a mismatch can be detected even in short-term forecasts by relying on the following fundamental property of a perfect model: if a model is a perfect model, then the attractor, reconstructed by sampling a collection of initialised model orbits (forecast orbits), will be invariant to forecast lead time, provided two conditions: (1) that the model is initialised with good estimates of the system's true state based on observations and (2) that the collection of forecast orbits is a representative sample of the region in phase space accessible to the system.Deviations from this condition would constitute an alternative measure for the suitability of a climate model.This was shown for the Lorenz '63 system (Fig. 4) and for three operational ensemble prediction systems (Fig. 5).This approach provides a link between the fields of weather and climate prediction as it relies on the availability of forecast orbits produced by climate models.
The results presented in this contribution are consistent with the discussions by Judd andSmith (2001, 2004).They have shown that, given a set of imperfect observations Introduction

Conclusions References
Tables Figures

Back Close
Full in a perfect models scenario, it is possible to find a set of indistinguishable states consistent with the observations (Judd and Smith, 2001).In contrast, in an imperfect model scenario, almost certainly no trajectory of the imperfect model is consistent with any set of observations (Judd and Smith, 2004).Judd and Smith (2004) also introduce the concept of pseudo-orbits that intrinsically take into account the existence of model error.Although the discussion in Judd andSmith (2001, 2004) do require the availability of observations, the concept of pseudo-orbits might prove useful for the interpretation of climate projections; however, I can only speculate at this point, leaving this for future investigation.
Discussion Paper | Discussion Paper | Discussion Paper | used the Lorenz '63 system (including several modified versions) to investigate extended-range Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Screen / Esc Printer-friendly Version Interactive Discussion Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper |

Fig. 1 .
Fig. 1.System attractor (black) and imperfect model attractor (red).Also shown are the fixed points for the system (black x) and for the imperfect model (red x).

Fig. 2 .
Fig. 2. (a) Prototype system orbit (black), orbits of the imperfect model with perfect IC (red) and orbits of a perfect model with imperfect IC (grey).Also shown are perfect observations on the system's orbit (black dots), forecasts from the imperfect model with perfect ICs (red x) and forecasts from the perfect model with imperfect ICs (black x).Observations and forecasts are given at regular intervals of 5 t.u.(b) Minimum distance from perfect ICs to points on the imperfect model attractor (a m ) at every observation time.(c) Error between the prototype system orbit and those from the imperfect model with perfect ICs (red) and the perfect model with imperfect ICs (black).