A common classification scheme of the stably stratified atmospheric boundary layer (SBL) distinguishes between two distinct regimes, denoted the weakly and very stable boundary layers respectively, wSBL and vSBL, e.g.hereafter AM19a, AM19b, and AM19c. In this classification scheme the wSBL is characterized by weak stratification, strong wind, and shears which produce sufficient turbulence kinetic energy (TKE) to sustain continuous vertical mixing despite the stable stratification e.g.. The vSBL is characterized by strong stratification, low wind speeds, and weak or intermittent turbulence such that vertical coupling of the atmospheric layers weakens. Very stable boundary layers are also sometimes found to display so-called upside-down turbulence, in which TKE is generated aloft by strong shears and then transported downwards. Observational data as well as simulations show that, to a good approximation in horizontally homogenous conditions, the wSBL conforms to the classical understanding of turbulence in the atmospheric boundary layer, with turbulence quantities decreasing with height and near-surface profiles which are well described by Monin–Obukhov similarity theory MOST; e.g.. In the vSBL, on the other hand, turbulence profiles can decouple from the surface and MOST breaks down e.g.. Regime structures and transitions are poorly represented in weather and climate models, due both to coarse resolution (vertical and horizontal) and to an imperfect understanding of the diverse physical processes governing the SBL, particularly with regard to the vSBL to wSBL transitions e.g.. In this study we first analyze how well the statistics of SBL regime occupation and regime transitions can be described by a two-regime system, with the goal of using this information to develop a prototype explicitly stochastic parameterization of turbulence in the SBL for models of weather and climate.

As transitions between the two SBL regimes are a common feature of SBL dynamics around the globe (AM19b), a representation of the effect of these dynamics in weather and climate models is needed. The regime transitions, however, are associated with a range of different mechanisms. Over land, the wSBL to vSBL transition (which for simplicity we denote the collapse of turbulence even though turbulence does not cease entirely) is normally caused by radiative cooling at the surface increasing the inversion strength and suppressing vertical turbulent fluxes of momentum and heat. This process is relatively well understood and has been examined using conceptual and idealized single-column models or direct numerical simulations of stratified channel flows and atmospheric boundary layers e.g.. Radiative cooling leads to very shallow boundary layers which are typically not resolved well in large-scale circulation models. Another mechanism for the wSBL to vSBL transition of particular importance over water is the advection of warm air aloft (AM19c), producing vSBL conditions which are not as shallow as those driven by radiative fluxes.

The vSBL to wSBL transition (which we denote the recovery of turbulence) is less well understood. Mechanisms by which turbulence recovers include the build-up of shear resulting in instabilities or an increase in cloud cover weakening the stratification by increasing the downwelling longwave radiation (AM19b). Another potential class of processes initiating these transitions is associated with intermittent turbulence events e.g.and references within, which have been found to dominate the turbulent transport in vSBL conditions . Intermittent turbulence arises from a range of different phenomena, such as breaking gravity waves or solitary waves , density currents , microfronts , Kelvin–Helmholtz instabilities interacting with the turbulent mixing , or shear instabilities induced from internal wave propagation . It has also been suggested from direct numerical simulations that intermittency can arise as an intrinsic mode of the non-linear equations in the absence of external perturbations of the mean flow . Regardless of which process causes the recovery of turbulence, all phenomena are subgrid scale in state-of-the-art weather and climate models and are typically not included explicitly through process-based deterministic parameterizations.

Although processes in the SBL have been extensively studied, substantial errors of SBL representation persist in weather and climate models . Misrepresentation of the SBL includes unrealistic decoupling of the atmosphere from the surface (due to misrepresentations of TKE in the vSBL), resulting in runaway surface cooling , underestimation of the wind turning with height within the boundary layer , overestimation of the boundary layer height , underestimated low-level jet speed , and underestimation of near-surface wind speed and temperature gradients or their diurnal cycle .

Accurate simulations of these near-surface properties is particularly important for global and regional weather forecasts of vertical temperature structures which control the formation of fog and frost . More accurate simulations of the SBL regime behaviour are also important for better representations of surface wind variability and wind extremes ; simulation and assessment of pollutant dispersal, air quality , harvesting of wind energy ; and agricultural forecasts .

Global and regional weather and climate models often use an artificially enhanced surface exchange under stable conditions in order to improve simulations of the large-scale flow . This approach led to the introduction of long-tailed stability functions and minimum background TKE values that are not supported by observations. In such representations, turbulence is artificially sustained under very stable conditions and the two-regime characteristic of the SBL is suppressed, biasing near-surface winds and temperature profiles. Without such parameterizations the nocturnal boundary layers can experience a single turbulence collapse which persists for the entire night. Although the long-tailed stability functions in relatively coarse-resolution models are designed to mimic the grid box mean of fluxes over many subgrid-scale wSBL and vSBL patches, with increasing horizontal and vertical resolution more accurate process-based parameterizations are necessary. The occurrence of vSBL to wSBL transitions does not appear to depend deterministically on internal or external state variables (AM19a,b), indicating that parameterizations of the effects of these kinds of transitions in weather and climate models may be required to be explicitly stochastic e.g.. In particular, phenomena such as intermittent turbulence events will likely rely on stochastic parameterizations as their structure and propagation are found to be only weakly dependent on mean states e.g.. Stochastic subgrid-scale parameterizations of SBL processes have been proposed to account for the missing variability in the SBL and improve both climate mean states and forecast ensemble spread e.g.. For instance, propose an additional Markovian system to switch between times of strong and weak influence of short-timescale, non-turbulent motions on TKE production in the vSBL.

In AM19a,b,c it was demonstrated that hidden Markov model (HMM) analysis of Reynolds-averaged mean states can be used as a tool to analyze the SBL regimes at tower sites in many different settings. Independent of the surface type, the climatological setting, or the complexity of the surrounding topography, two distinct regimes in the state variable spaces of Reynolds-averaged mean states and turbulence are evident. As the HMM analyses provide climatological (that is, based on long-term statistics) transition probability matrices for a two-regime Markovian system, a natural approach to developing stochastic parameterizations of SBL regime dynamics is to investigate whether these can be based on “freely running” stationary Markov chains (FSMCs) using these transition matrices. The first goal of this study is to determine whether climatological Markovian transition probability matrices, which are by construction independent of the state of the SBL, are adequate for simulations of the SBL regime dynamics. While the HMM analyses presented in AM19a assume stationary Markov regime dynamics, statistical analyses of the estimated regime sequences show clear evidence of elevated probability of turbulence collapse close to sunset (AM19b). Furthermore, probability distributions of event durations demonstrate a localized maximum corresponding to a typical recovery time of 1 to 2 h after transitions, during which a subsequent transition is unlikely. This structure is indicative of non-Markov behaviour (AM19b). Because of these non-stationary and non-Markov behaviours, a FSMC will never exactly capture all aspects of SBL regime dynamics. However, it is a parsimonious model and might be sufficient to reproduce most statistics of interest.

In order to investigate the potential of FSMC-based parameterizations, we first analyze how well they can characterize the HMM-based regime statistics. As part of this analysis we consider the sensitivity of the regime sequence estimated by the HMM to perturbations of the persistence probabilities, allowing for a quantification of which ranges of persistence probabilities accurately describe SBL regime statistics in HMM analyses. By comparing this sensitivity analysis with a sensitivity analysis of regime statistics to varying persistence probabilities in a FSMC, we quantify which ranges of persistence probabilities are consistent with both SBL regime statistics derived from an HMM analysis and SBL regime statistics simulated in a FSMC. As we demonstrate that FSMCs cannot adequately simulate all SBL regime statistics of interest, we then investigate the state dependence of observed SBL regime transition probabilities. Finally, we develop a pragmatic prototype of an explicitly stochastic parameterization using the derived state-dependent transition probabilities and present preliminary tests in an idealized single-column model SCM;. The study is organized as follows. First a short review of the observational data used in the HMM analysis is given in Sect. , followed by a brief review of the HMM application to the SBL (Sect. ). Results of simulating statistics in FSMC are shown in Sect. , followed by the description of the prototype state-dependent stochastic parameterization and test simulations in Sect. . Conclusions follow in Sect. .

Only a brief summary is presented here because the observational data used in this study have been discussed in detail in AM19a. Observational data sets from nine different research towers measuring standard Reynolds-averaged meteorological state variables with a time resolution of 30 min or finer are considered (Table ). The observational levels of wind speeds and temperatures determine the reference state variable sets which are used in the HMM analyses to classify the data into SBL regimes (cf. AM19a, Table ). Substantial differences among the nine experimental sites exist in terms of their surface conditions, surrounding topography, and meteorological setting. As a simple classification scheme, we distinguish between land-based, glacial-, and sea-based stations.

The land-based stations can be further clustered into different subsets. Both the Cabauw and Hamburg towers lie in flat, humid, grassland areas, although the Hamburg tower is affected by the large metropolitan area of Hamburg. The Karlsruhe tower is located in the Rhine Valley, a rather hilly, forested area. The American sites, Boulder and Los Alamos, are located in relatively arid settings and are strongly affected by the surrounding topography of the Rocky Mountains.

The DomeC observatory, the single glacial-based station, is located in the interior of Antarctica and is influenced by completely different surface conditions, including high albedo and low roughness length.

The sea-based stations are the offshore research platforms Forschungsplattform in Nord- und Ostsee (FINO), located in the German North and Baltic seas. These sites are characterized by relatively homogeneous local surroundings and a large surface heat capacity. At the FINO towers nights with statically unstable conditions (defined as nights with two or more unstable datapoints in a night) are excluded as under these conditions wind speed measurements are unreliable . Furthermore, at FINO-1 nights with primary wind directions between 280 and 340∘ are excluded due to mast interference effects in the data. At the other stations such an exclusion is not necessary as three wind measurements with 120∘ separation are taken at each level.

We now present a brief overview of the HMM analysis with application to the SBL AM19a. An in-depth description of HMM analyses can be found in .

We use the HMM to systematically characterize regime behaviour in the SBL from observed data. The HMM assumes that underlying the observations is an unobserved, or hidden, discrete Markov chain (X={x1,x2,…,xT}). The analysis estimates the regime-dependent parametric probability density distribution (pdf) of the observations (described by the parameter set λ), the transition probability matrix Q, and a most likely regime path of the Markov chain (known as the Viterbi path, VP). We associate the different states of the Markov chain with the SBL regimes (wSBL and vSBL). In our analysis we use observations of the three-dimensional vector consisting of the Reynolds-averaged vertically averaged mean wind speed, wind speed shear, and stratification to define the HMM input vector Y. A detailed justification of this observational input data set is presented in AM19a. The HMM estimation algorithm makes use of the following assumptions.

Markov assumption: the current regime value it at xt depends exclusively on the previous regime of xt-1, so1P(xt=it|xt-1=it-1,xt-2=it-2,…,x0=i0)=Qitit-1∀twithi∈{0,1},where the dynamics of the SBL are governed by Q (a 2×2 matrix corresponding to the wSBL and vSBL, respectively) such that ∑itQitit-1=1.

The goal of the HMM analysis is to estimate Λ from Y. Starting from the probability of the observational time series conditioned on the parameters P(Y|Λ) and applying Bayes' theorem to obtain P(Λ|Y), the problem reduces to a maximum-likelihood estimation which can be iteratively solved to find local maxima via the expectation maximization algorithm . Having estimated Λ, the most likely regime sequence (the VP) can be calculated. Regime occupation and transition statistics can then be obtained through analysis of the VP. The estimation of the parameters in the expectation-maximization scheme for our analysis is described in detail in .

One limitation of the HMM model considered is that it assumes stationary statistics. However, non-stationarities linked to the diurnal cycle and seasonal variability are present in the regime statistics of the SBL (cf. next section, AM19b). Generalizations exist which can account for non-stationarities, such as nonhomogeneous HMMs which condition transition probabilities on the state of external variables. Other clustering techniques such as the finite-element variational approaches also relax the stationarity assumptions e.g.. In particular, the finite-element, bounded-variation, vector autoregressive factor method (FEM-BV-VARX) includes both autoregressive dynamics within each regime and a modulation of regime dynamics to external drivers. For instance, were able to use this model to identify different regimes of interaction between submesoscale motions and the turbulence. However, as our analyses find no clear relationship between external drivers (geostrophic wind and cloud cover) and transitions between regimes of Reynolds-averaged variables (AM19b), we consider stationary HMM analysis in this study in order to investigate the simplest possible approach to a stochastic parameterization of turbulence under SBL conditions. Using such a relatively simple parameterization allows us to determine where additional complexity is warranted and assess how well the dynamics are approximated by stationary Markovian systems.

To be useful as the basis of new parameterizations of turbulent fluxes in the SBL, FSMCs should model SBL regime statistics accurately. The statistics we focus on are the event durations and the probabilities of each of the occurrence of very persistent nights, of the occurrence of at least one transition within a night, and of multiple transitions within a night. Our reference FSMC models use transition matrices Qref obtained from HMM analyses in AM19a (Table ). In the HMM analysis, the matrix Q can be held fixed at prescribed values while other parameters and the VP are estimated. Repeating HMM analyses using such fixed Q perturbed from Qref, we investigate the sensitivity of the regime statistics of corresponding VPs relative to their reference VPref. Because the estimated VP is constrained by the observations, its statistics may differ considerably from a FSMC using the same Q. Evaluating the FSMC regime statistics for a range of different Q determines the ranges of persistence probabilities for which SBL regime statistics of a FSMC match those of VPref. Mathematical expressions used to compute the regime statistics of a FSMC using a given Q are presented in Appendix . These calculations require specification of the lengths of the nights. As the tower sites are located in the midlatitudes, we use a range of nighttime durations between 8 and 15 h. In this section we do not consider the glacial-based station, DomeC in Antarctica. Because the duration of the polar nights is much longer than nights at the other midlatitude stations considered, direct comparisons of regime occupation statistics within individual nights are not meaningful.

In this section, we develop a prototype explicitly stochastic parameterization for numerical weather prediction and climate models and test it using an idealized SCM. We first consider the state dependence of transition probabilities from VPref, to be used as the basis of a two-value (wSBL or vSBL) discrete SBL regime occupation variable (S). After having estimated functional forms for these conditional probabilities from fits to data, a parameterization of episodic enhancement of eddy diffusivity by intermittent turbulence bursts is developed. Finally, the application of this parameterization in the SCM is presented. We emphasize the fact that the explicitly stochastic parameterization and the tests of it presented here are intended to be a proof of concept. A formal validation of model experiments against observational data, including systematic sensitivity analyses of the free parameters and an implementation in a more complex single-column model, will be the subject of a future study.

Recent studies have demonstrated that hidden Markov model (HMM) analysis is an effective tool to classify the nocturnal boundary layer (SBL) into weakly stable (wSBL) and very stable (vSBL) conditions AM19a, AM19b, AM19c. One goal of this study is to investigate whether a two-regime “freely running” stationary Markov chain (FSMC, obtained from the HMM analysis) is able to simulate SBL regime statistics with sufficient accuracy to be the foundation of a stochastic parameterization of SBL regimes. We have assessed the performance of the FSMC (using the most likely transition probabilities from HMM analyses) relative to the observed regime statistics such as the distributions of event durations and the probabilities of occurrence of very persistent nights (nights without SBL regime transitions), of any regime transitions, and of multiple subsequent transitions.

The non-stationary occurrence probabilities of very persistent nights as estimated from the HMM analyses cannot be accounted for in a FSMC. The occurrence of regime transitions is slightly overestimated by the FSMC. Transitions subsequent to preceding ones and the mean event durations in each regime are relatively close to the statistics as estimated with the HMM across all sites and seasons. The recovery time between regime transitions, however, is not explainable by any two-regime FSMC.

By fixing the persistence probability matrix and producing new perturbed HMM regime sequences we have quantified the range of persistence probabilities that are consistent with the most likely HMM regime sequence. At all the sites considered, we find that the HMM regime sequence varies only slightly for reasonable variations of transition probabilities.

Generally, no persistence probability range can be identified for which all SBL regime statistics of a FSMC are consistent with those of the HMM analysis. This result is true even when seasonal non-stationarity is accounted for. The non-Markov behaviour of regime occupation and the fact that aspects of regime transitions such as radiatively driven turbulence collapse can be simulated by models indicate the need for state-dependent transition probabilities in any explicitly stochastic representation of SBL regime transitions.

With the exception of the sea-based stations, state-dependent transition probabilities conditioned on the bulk Richardson number (RiB) exhibit a systematic state-dependent behaviour for wSBL to vSBL transitions. Transition probabilities for turbulence recovery events, on the other hand, demonstrate approximately state-independent characteristics with little consistency across sites. The lack of robustness of the conditional transition probabilities and weak dependence of turbulence recovery on RiB imply that RiB is not an appropriate conditioning variable.

State-dependent transition probabilities conditioned on stratification, in contrast, demonstrate a systematic state-dependent behaviour for both types of transitions across all stations. The wSBL to vSBL transition probabilities conditioned on the stratification increase almost linearly up to a threshold, while the vSBL to wSBL transition probabilities show a sigmoidal behaviour.

A prototype of an explicitly stochastic parameterization is developed based on the following foundations. The explicitly stochastic parameterization uses a new local variable S tracking the SBL regime (wSBL or vSBL). At each time step, the occurrence of a transition is determined randomly using the instantaneous state-dependent transition probabilities. If S is determined to be in the vSBL, episodes of enhanced turbulent mixing are added.

Experiments in an idealized SCM confirm that such an approach provides a reasonable representation of SBL regime dynamics. VSBL to wSBL transitions are related to the occurrence of turbulence bursts, lagging these slightly. The simulated responses of temperatures and wind speeds due to the enhanced heat and momentum fluxes towards the surface are comparable to observations. For both transitions, simulated recovery times are consistent with the observed distributions.

Emphasizing the fact that the explicitly stochastic parameterization presented here is only intended to be a proof of concept, preliminary results suggest that the parameterization has the potential to simulate SBL regime dynamics in weather and climate models. The observational information on climatological regime statistics and event duration distributions (cf. AM19b, AM19c, and the present study) can be used to tune the parameterization to generate the appropriate SBL regime variability. Due to the fact that event duration distributions and stratification-dependent transition probabilities are similar between the midlatitude tower sites, we believe that the transition process at those stations can be parameterized independently of the local complexity of the surface conditions (such as surface type or topography). Although at DomeC similar stratification-dependent transition probabilities can be obtained, the altitude range used to determine stratification is different than at the other sites, suggesting that a generalized parameterization has to take additional local state variables into account. Furthermore, even though a systematic behaviour of transition probabilities conditioned on RiB across the different tower sites is absent, RiB is a coarse approximation to Ri. Analyses of other data sets (with higher spatial and temporal resolution) allowing for better approximations or an estimation of the gradient Ri or Ri flux are needed to determine whether a systematic behaviour is truly absent.

As our model analysis only considers fixed surface and upper boundary conditions, sensitivity analyses of these conditions in the idealized SCM as well as of different resolutions in both time and space must be assessed in terms of different observational case studies. Due to the fact that the first-order closure requires us to consider the effects of intermittent turbulence events as an enhancement of diffusivities for momentum and heat, we have imposed a rather synthetic space–time structure of these enhancements. As intermittent turbulence events are associated with the local enhancement of turbulence kinetic energy (TKE), our parameterization of episodically occurring turbulence bursts can more naturally be implemented in a prognostic TKE scheme as an additional TKE source term e.g.. Such an approach would allow the model to determine the space–time structure of turbulence pulses as well as the interaction of turbulence bursts. In the future, we will implement the parameterization in a more complex SCM (with and without a prognostic TKE scheme) to obtain a more comprehensive assessment of its use in numerical weather prediction and climate models.

Finally, the parameterization requires further information regarding horizontal dependence of regime statistics, as it is not reasonable to expect an entire large-scale weather or climate model grid box to always be in one or the other state. This horizontal dependence will be the subject of a future study. Assessment of the dependence length scales relative to the grid box size will allow the determination of the effects of spatial averaging to the grid box scale on the probability distribution of SBL quantities.