Our study uses an existing convective-scale 1000-member ensemble simulation described in detail by . The 1000-member ensemble applies the full-physics non-hydrostatic Scalable Computing for Advanced Library and Environment regional model (SCALE-RM) and the SCALE Localized Ensemble Transform Kalman Filter (SCALE-LETKF) DA system . Using an offline nesting approach, the 1000-member ensemble setup couples two domains with different horizontal resolutions. Ensemble forecasts in the outer domain covering central Europe (15 km grid spacing) delivered the boundary and initial conditions for the convective-scale ensemble forecasts in the inner domain covering Germany (3 km grid spacing). High-resolution short-term forecasts from the inner domain will be analyzed to evaluate correlations and localization.

Initial and boundary conditions: the data assimilation cycling has been performed in the coarse European domain assimilating conventional observations with a LETKF . A set of 1000 independent and specifically constructed ensemble boundary conditions (BCs) drives the European-scale forecasts. These BCs combine 1000 climatologically scaled random perturbations with a 20-member analysis ensemble of the NCEP Global Ensemble Forecast System (GEFS). The GEFS 20-member analysis ensemble is used 50 times to reach 1000 BCs and afterwards combined with 1000 random climatologically scaled perturbations. This approach yields 1000 independent BCs that ensure sufficient ensemble spread when combined with relaxation to prior spread RTPS. The boundary and initial conditions for the inner and convective-scale forecast domain are downscaled from 15 to 3 km resolution based on simulations in the European domain.

Our study uses the model output from the inner model domain with a 250×230 grid area centered over Germany with a 3 km horizontal resolution. This sub-domain excludes the Alps and regions within 10 grid points of the domain boundary. The model output has 30 vertical levels ranging from the surface to the model top at 16.9 km. The original vertical grid is terrain-following and has fixed height levels above the surface (m). For practical reasons, we extracted temperature (T), specific humidity (Q), and horizontal zonal (U) and meridional (V) wind components on 20 vertical pressure levels (100, 150, 200, 250, 300, 350, 400, 450, 500, 550, 600, 650, 700, 750, 800, 850, 900, 925, 950, 975 hPa). Performing our analysis on this modified grid allows horizontal averaging of data on pressure levels where needed. Overall, our examination includes 10 short-term forecasts that were initialized twice per day from 29 May to 2 June 2016 at 00:00 and 12:00 UTC. The 3 h forecasts valid at 03:00 and 15:00 UTC serve as a basis for computing and investigating background error correlations.

Atmospheric blocking over the Atlantic influenced the large-scale flow over Europe in the 5 d experimental period. The blocking led to a quasi-stationary weather pattern over central Europe with an upper-level trough over western Europe and a shallow surface low over central Europe. The low-pressure system was associated with a cold front and a warm front that moved over Germany during the period. A convergence zone over southern Germany caused large-scale lifting. Furthermore, mid-level winds advected warm and moist air masses from southern Europe towards Germany at the beginning of the experimental period. Combined with the convergence zone, atmospheric conditions led to intense convection and heavy precipitation, including hail. Weak pressure gradients and slowly moving convective cells resulted in high local precipitation rates and flash flooding. Due to these severe weather events, several studies focused on this exceptional period e.g.,. , , and provide further details on the weather situation in this period as these studies also explore the 1000-member ensemble simulation with a different purpose.

Error covariances are a key component in data assimilation and determine how assimilated information is weighted and distributed in state space. Given a sample of state vectors xn provided by a background forecast ensemble, the flow-dependent sample error covariance matrix P can be computed as follows: 1P=1N-1∑n=1N(xn-x‾)(xn-x‾)T, where N is the ensemble size and x‾ is the ensemble mean state. The covariance matrix P by definition is a symmetric, positive semi-definite matrix with variances on its diagonal entries and covariances on its off-diagonal entries. Each off-diagonal element contains a sample covariance (cov) of two state variables xi: 2cov(x1,x2)=r(x1,x2)σ(x1)σ(x2), where r∈[-1,1] is the sample correlation and σ is the sample standard deviation.

Usually, the number of affordable ensemble members is limited in NWP due to a huge state space and computational restrictions. This deficit causes severe sampling errors. Consequently, all ensemble filters require a correction of sampling errors, often referred to as localization. For example, highlighted that the sampling error in covariances is dominated by sampling error in the sample correlation r, not by sampling error in the variances. Therefore, our analysis will solely tackle sampling errors in sample correlations. Sample correlations are normalized with standard deviations and possess no unit. The normalization allows comparison or combination of correlations of different variables, facilitating the interpretation.

The implementation of localization depends on various factors determined by the type of ensemble filter. Usually, localization is applied directly to the background error covariance matrix using a Schur product: 3Ploc=C∘P, where C is the localization matrix . The matrix C consists of tapering factors α that are determined using the localization approach of choice. Based on the Schur product theorem theorem 7.5.3, positive semi-definite matrices C and P guarantee positive semi-definiteness of the localized covariance matrix Ploc. Furthermore, localization matrices should feature ones on the diagonal similar to a correlation matrix to avoid undesired inflation of variances .

The sampling noise expected for zero correlation estimates and sample size N is (N)-1 . For the 1000-member ensemble (N=1000), this estimation yields a very small sampling noise of approximately 3%. In comparison, a 40-member ensemble reveals an expected sampling noise of approximately 16%. Throughout this study, correlations computed using the full 1000-member ensemble serve as truth (r1000) for the interpretation of vertical correlations and the evaluation of sampling errors and localization in smaller subsamples of the full ensemble. We focus on vertical correlations and sampling errors in 40-member subsamples as this is a typical ensemble size applied by operational weather services such as, e.g., the Deutscher Wetterdienst. Preceding studies applied a similar approach for studying sampling errors .

The present study will adopt the subsampling approach from and . The 1000-member ensemble provides 25 randomly drawn 40-member subsamples without repetition of members (illustrated in Fig. a). We assume that the 40-member sub-ensembles of the 1000-member ensemble statically have sampling errors similar to what those independent 40-member ensemble EnKF systems would have. As mentioned above, we will analyze 10 3 h forecasts. This setup results in a sample of 250 ensemble forecasts with 40 members that we can compare to the 10 ensemble forecasts with 1000 members. The model domain has 250×230 grid points yielding 57 500 vertical columns in our domain. We will, therefore, analyze approximately 11.5×106 true and 287.5×106 40-member vertical correlation profiles per variable pair, accounting for all 20 reference levels. This data set allows robust statistical analysis of error correlations, but it should be noted that error correlations may differ for other periods and regions.

In the present study, we will analyze four prognostic variables: temperature (T), specific humidity (Q), zonal wind (U), and meridional wind (V). This setup yields 16 correlation pairs (Table ) that we will inspect for different reference levels. Furthermore, we will group correlations as “self” (e.g., temperature–temperature as shown in Fig. ) or “cross” (e.g., temperature–humidity) correlations to highlight common behavior. Subsequently, we will use the correlation coding shown in Table . For example, “TQ” combines all temperature correlations from the reference level to specific humidity at all other vertical levels in a column. Throughout the paper, we will mainly present results for the U-wind component as conclusions for the V-wind component are similar.

Our goal is to empirically find the optimal localization factor α that minimizes the sampling error or cost function J: 5J(α,t,z,p,A)=∑s=1S∑k=1Kαrs,k40-rk10002, where the minimization is done separately for each forecast time t, reference level z, pressure level p, and variable pair A. S is the number of 40-member ensembles (S=25). This is equivalent to finding the α that minimizes 6∑s=1S∑k=1Kα2rs,k402-2αrs,k40rk1000+rk10002. Taking a derivative with respect to α and finding the minimum gives us 7α=∑s=1S∑k=1Krs,k40rk1000∑s=1S∑k=1Krs,k402. In other words, the EOL minimizes the root mean square difference (RMSD) between the 1000-member correlation and all 25 40-member subsample correlations for a chosen setting. For technical reasons, we minimized the cost function using the Brents method as implemented in scipy.optimize . Note that the range of localization is not confined to [0,1], which means that the EOL could increase correlations if required. Values larger than 1 can occur when the true correlation is larger than the sample correlation. For example, this can happen when estimating the EOL after applying other localization approaches. Negative EOL values can be observed when the EOL is computed for a small correlation sample (e.g., a single vertical column), which is not the case in the present study. However, we suggest setting negative EOL values to 0 in case they might occur.

Applying the EOL by construction yields a symmetric but not necessarily positive semi-definite localization matrix. In our case, constructed localization matrices were not positive semi-definite. Depending on the data assimilation algorithm, additional steps could be required to apply the EOL results to guarantee the positive semi-definiteness of the localized covariance matrix. For this purpose, Sect.  will assess approaches for achieving positive semi-definiteness of the EOL. The assessment includes a numerical approach to approximate the EOL matrix with the nearest correlation matrix, which is SPSD.

Our approach for empirically estimating localization is inspired by , who compare two methods: the GGF and empirical localization functions (ELFs). The GGF minimizes the RMSD between the estimated regression coefficients in subsets of the ensemble using a hierarchical ensemble filter . ELFs are derived from an observing system simulation experiment (OSSE) by minimizing the RMSD between the true values of the state variables and the posterior ensemble mean . In contrast to ELFs, the GGF and EOL purely judge localization based on ensemble sampling error without an OSSE. Furthermore, in contrast to the GGF, the EOL assumes the large ensemble correlation as truth for minimizing the sampling error. The EOL presented in our study corresponds to a non-adaptive distant-dependent domain-uniform vertical localization that is common for operational convective-scale regional data assimilation systems.

Figure c displays the EOL (α(p)) as estimated for the example of TT correlations introduced above and reference level 500 hPa. The domain-uniform EOL equals 1 at the reference level 500 hPa as no correction is needed. The EOL appears broader and follows the shape of the mean absolute correlation. For example, this localization behavior was also described by . The error reduction is largest for weak and distant correlations.

As discussed in Sect. , the domain-averaged absolute vertical correlation can aid the interpretation of the EOL. For this reason, we will first evaluate the mean absolute vertical correlation and then the EOL. Figure  shows the mean absolute vertical correlation for all possible variable combinations and reference level 500 hPa. Self-correlations of the same variable all peak at the reference level. In contrast, cross-correlations are weaker and do not always exhibit a maximum correlation at 500 hPa. The TU correlation, for example, peaks around the tropopause, while the UT correlation reveals a minimum at that height. The mean vertical correlation length is variable-dependent, shortest for specific humidity and longest for wind. The domain-averaged absolute vertical correlation only exhibits a fairly small variability within the 5 d experimental period. The variability between day and night also appears to be small (not shown). Results could, however, differ for other conditions or seasons, e.g., situations with strong atmospheric stability.

Next, we focus on the EOL derived for 40-member subsamples from all forecasts. Figure  displays the EOL for all variable combinations and reference level 500 hPa. The EOL depends on the prevailing correlation but has a different shape and vertical extent. As seen for the single forecast example in Sect. , weaker correlations are more affected by sampling errors and require stronger correction. Consequently, all cross-correlations require a stronger localization. The localization for cross-correlations reveals an amplitude smaller than 1 at the reference level. Given this behavior, tapering functions for cross-correlations should not be 1 at zero distance when applying a distance-dependent localization. Self-correlations are less affected by sampling error and require only a weaker correction, especially close to the reference level.

EOLs for humidity correlations all peak at the reference level 500 hPa (Fig. a). However, temperature and wind EOLs behave differently (Fig. b, c) and do not peak at the reference level following the correlation pattern (Fig. b, c). For example, the TU EOL peaks around the tropopause, where winds are typically strongest. Wind correlations (e.g., UU; Fig. c) require only a small correction. The EOL for UV correlations is almost constant with height and does not show a distinct maximum. All self- and cross-correlations involving humidity peak at the reference level 500 hPa (for example, see UQ localization).

Overall, the variability of domain-averaged correlations from forecast to forecast is small (Fig. ). EOLs exhibit a larger variability than domain-averaged correlations. For most variables, the variability is larger close to the surface, especially for temperature correlations (Fig. b). Results should be treated with caution where changes in the EOL with height are smaller than the variability from forecast to forecast.

Subsequently, we will discuss the EOL for two additional reference levels to highlight changes in height within the troposphere. Figure  shows the EOL for a reference level 300 hPa. For reference level 300 hPa, EOLs appear to be broader compared to 500 hPa. This height dependence is in line with larger vertical correlation length scales found for the upper troposphere, in contrast to the lower troposphere, boundary layer, or close to the surface. Similar to other reference levels in the middle and upper troposphere, EOLs for correlations between wind and temperature reveal a maximum (TU; Fig. b) and minimum (UT; Fig. c) above the tropopause level.

All reference levels within the boundary layer show similar behavior of the EOL (see, for example, Fig.  using reference level 900 hPa). The EOL shows a narrow optimal vertical localization for reference levels close to the surface. In contrast to higher reference levels, the EOL of cross-correlations also peaks at the reference level (Fig. ). The EOL drops to different constant values with increasing distance. For wind and humidity, the EOL reveals an almost constant value above 550 hPa. In contrast, the EOL for temperature steadily declines with increasing distance until the domain top. Temperature self-correlations (TT) exhibit a second peak in the upper troposphere. EOLs do not converge to 0 for large vertical distances. Separation distances where the EOL converges to a small constant value could indicate suitable cut-off distances. A common aspect of the choice of cut-off distance is the signal-to-noise ratio that depends on the ensemble size and correlation strength.

Some operational DA systems apply a uniform distance-based vertical localization that does not change with time, height, variable, or observation type. In this case, appropriate localization needs to meet several requirements using a suitable uniform localization approach. Results in Sect.  showed that cross-correlations systematically behave differently than self-correlations. For this reason, we will now evaluate the mean absolute correlation and EOL of three groups of variables: self-correlations, cross-correlations, or all correlations combined. Derived EOLs now minimize the sampling error for all gathered correlations of each group.

Figure  displays the mean absolute correlation for the three groups of correlations. The results show the average correlation and its variability over the 10 forecasts. Self-correlations again highlight the height dependence of the vertical correlation length and always exhibit a peak that is 1. Cross-correlations are weaker and only exhibit a narrow peak at the reference level. For all correlations combined, the peak amplitude is closer to the peak of cross-correlations as there are more cross-correlations than self-correlations. Combining all correlations or only cross-correlations results in a peak amplitude smaller than 1 at the reference level.

In contrast, the peak amplitude of the EOL for all correlations is closer to the peak of self-correlation (Fig. ). The shape of EOLs substantially differs from the single variable pair cases. The EOL is weaker due to wind correlations that account for half of all correlations. The change in the shape of the EOL indicates that different tapering functions could be needed for different variables. Minimizing the error for grouped correlations, the strength of the EOL is always weaker than 0.4. Finally, domain-averaged absolute correlations reveal a small variability from forecast to forecast (Fig. ). The same applies to EOLs. Only the EOL of self-correlations exhibits a slightly larger variability, especially far from the reference level (Fig. ).

The EOL approach empirically yields an optimal localization by minimizing differences between sample correlations and a defined true correlation. By design, the results discussed above exclude algorithm-specific requirements to better understand how vertical localization should behave in different situations. However, further steps might be required to apply EOL results depending on the data assimilation algorithm. As mentioned in Sect. , using EOL estimates does not necessarily yield a positive semi-definite localization matrix. Consequently, EOL localization could cause non-proper mathematical covariance matrices and undermine the data assimilation process. For example, positive semi-definite covariance matrices are elemental when solving a quadratic cost function as they ensure a global minimum. For this reason, we will now discuss different approaches that allow one to achieve positive semi-definiteness of the EOL.

applied a Gaussian fitting based on the Gaspari–Cohn function to ensure SPSD. However, fitting a correlation function restricts the localization to a specific function shape. This restriction can diminish the error reduction. In our case, the error reduction achieved by an optimally tuned Gaspari–Cohn function is substantially smaller than by the EOL, as discussed in Sect.  and Fig. . This finding motivates exploration of other localization functions , including multivariate localization functions , to improve fitting approaches.

Besides function fitting, re-conditioning of matrices can help achieve positive definiteness . Given its aim, a localization matrix should have similar properties to a correlation matrix, which exhibits ones on the diagonal. theorem 2.5 states that for a symmetric matrix with t non-positive eigenvalues and diagonal elements ≥1, the nearest correlation matrix has at least t zero eigenvalues. Following this theorem, we attempted to set all negative eigenvalues to zero using an EOL-based localization matrix and eigendecomposition. Forcing negative eigenvalues to zero ensured positive semi-definiteness but led to values larger than one on the diagonal of the matrix. Further matrix transformations were required to achieve a proper correlation matrix.

Finally, our most successful attempt at achieving positive definiteness aiming for the least changes in the EOL was by searching for the nearest correlation matrix using specifically designed mathematical algorithms. For example, highlighted that the convexity properties of the problem allow one to find a unique nearest correlation matrix (NCM) for a given symmetric matrix, in our case the EOL-based localization matrix. The distance between matrices can be measured using weighted Frobenius norms. We apply the NCM algorithm for the SINGLE EOL case as a proof of concept. Employing four state variables on 20 vertical levels yields an 80×80 EOL-based vertical localization matrix for one vertical column (Fig. a). The eigenvalues of this EOL matrix range from about -0.5 to 50, with approximately half the eigenvalues being negative. The EOL localization matrix features ones on the diagonal and reconfirms that cross-correlations require stronger tapering. The nearest correlation matrix computed by the NCM algorithm exhibits small changes in off-diagonal elements compared to the EOL-based matrix (Fig. b, c). Zooming in, the SPSD matrix appears to be smoother. Comparing the error reduction (not shown), the nearest SPSD matrix performs only marginally worse than the EOL matrix. The relative difference in sampling error reduction for a test case was lower than 1 %. However, the EOL achieves the larger error reduction as it is designed to minimize the sampling error without any constraints.

This example suggests that ensuring SPSD can be achieved with minor changes to the EOL estimate. However, providing a general answer on how the EOL needs to be adapted is difficult as changes will depend on the construction of the localization matrix and its unique nearest correlation matrix. NCM algorithms can iteratively determine the nearest correlation matrix for a symmetric matrix and could be a useful tool for data assimilation. Choosing the best approach to guaranteeing SPSD is likely case-dependent given changing properties of the problem and potentially very large matrices in NWP.