In Bayesian statistics, according to , a state is “a description of the world, which is the object with which we are concerned, leaving no relevant aspect undescribed”, and “the true state is the state that does in fact obtain”, i.e., the true description of the world. The mathematical notation used is reported in Tables  and , and it is similar to that suggested by . The object of our study is the hourly precipitation field, x(), that is the hourly total precipitation amount over a continuous surface covering a spatial domain in terrain-following coordinates, r. Our state is the discretization over a regular grid of this continuous field. The true state (our truth; xt) at the ith grid point is the areal average as follows: 4xit=∫Vixrdr, where Vi is a region surrounding the ith grid point. The size of Vi determines the effective resolution of xt at the ith grid point. Our aim is to represent the truth with the smallest possible Vi. The effective resolution of the truth will inevitably vary across the domain. In observation-void regions, the effective resolution will be the same as that of the numerical model used as the background, which is approximately o(10–100km2) for high-resolution local area models . In observation-dense regions, the effective resolution should be comparable to the average distance between observation locations, with the model resolution as the upper bound.

The analysis is the best estimate of the truth, in the sense that it is the linear, unbiased estimator with the minimum error variance. The analysis is defined as xa=xt+ηa, where the column vector of the analysis error at grid points is a random variable following a multivariate normal distribution ηa∼N0,Pa. The marginal distribution of the analysis at the ith grid point is a normal random variable, and our statistical interpolation scheme returns its mean value xia and its standard deviation σia=Piia.

As for linear filtering theory , the analysis is obtained as a linear combination of the background (a priori information) and the observations. The background is written as xb=xt+ηb, where the background error is a random variable ηb∼N0,Pb. The background PDF is determined mostly, but not exclusively, by the forecast ensemble, as described in Sect. . The forecast ensemble mean is xf=k-1Xf1, where 1 is the m vector, with all elements equal to 1. The background expected value is set to the forecast ensemble mean, xb=xf. The forecast perturbations are Af, where the ith perturbation is Aif=Xif-xf. The covariance matrix is as follows: 5Pf=(k-1)-1Af(Af)T, and plays a role in the determination of Pb, as defined in Sect. .

The p observations are written as yo=Hxt+εo, where the observation error εo∼N0,R is the observation operator that we consider as a linear function that maps Rm onto Rp.

Our definitions of the error covariance matrices follow from a few general principles that we have formulated. For P1 (i.e., general principle 1; hereinafter the same definition applies for other references to P), background and observation uncertainties are weather and location dependent. For P2, the background is more uncertain, where either the forecast is more uncertain or observations and forecasts disagree the most. For P3, observations are a more accurate estimate of the true state than the background. We want to specify how much more we trust the observations than the background in a simple way, such as, for example, “we trust the observations twice as much as the background”. For P4, the local observation density must be used optimally to ensure a higher effective resolution, as it has been defined in Sect.  where more observations are available. For P5, the spatial analysis at a particular hour does not require the explicit knowledge of observations and forecasts at any other hour. However, constants in the covariance matrices can be set, depending on the history of deviations between observations and forecasts. P5 makes the procedure more robust and easier to implement in real-time operational applications.

P1 and P4 led to our choice of implementing Algorithm 1 by means of a loop over grid points. P2 will lead us to the identification of the regions in which the uncertainty on the input data is greatest. P3 will be used to define the observational uncertainty with respect to that of the background.

A distinctive feature of our spatial analysis method is that the background error covariance matrix Pib is specified as the sum of two parts, namely a dynamical component and a static component. This choice is consistent with P1 and P2. The dynamical part introduces nonstationarity, while the static part describes covariance stationary random variables. This choice follows from P1, and it has been inspired by hybrid data assimilation methods . The dynamical component of the background error covariance matrix is obtained from the forecast ensemble. Because the ensemble has a limited size, and often the number of members is quite small (order of tens of members), a straightforward calculation of the background covariance matrix will include spurious correlations between distant points. Localization is a technique applied in DA to fix this issue . The static component has also been introduced to remedy the shortcomings of using numerical weather prediction as the background. There are deviations between observations and forecasts that cannot be explained by the forecast ensemble. A typical example is when all the ensemble members predict no precipitation but rainfall is observed. In those cases, we trust observations, as stated through P3. Then, the static component adds noise to the model-derived background error, as in the paper by . In , the static component is referred to as a scale matrix, since it is used to scale the noise component of the model error, and we adopt the same term here. In scale matrix, the term scale is not associated with the concept of spatial scales; instead, it refers to a scaling (amplification or reduction) of the uncertainty. We will also refer to this matrix, and its related quantities, with the letter u to emphasize that this component of the background error is unexplained by the forecast.

Pib is written as follows: 6Pib=Γi∘Pf+σiu2Γui. The first component on the right-hand side of Eq. () is the dynamical part. Pf is the forecast uncertainty of Eq. (), Γi is the localization matrix, and ∘ is the Schur product symbol. The localization technique we apply is a combination of local analysis and covariance localization, as defined by . In the local analysis, only the closest observations are used, and we have implemented it by considering only observations within a predefined spatial window surrounding each grid point, up to a preset maximum number of pmx. The covariance localization is implemented through the element-wise multiplication of Pf by Γi, which has the form of a correlation matrix that depends on distances and is used to suppress long-range correlations. The second component on the right-hand side of Eq. () is the static part. The scale matrix is expressed through a constant variance σiu2, which modulates the noise, and the correlation matrix Γui, which defines the spatial structure of that noise. In the examples of applications presented in Sect. , both Γi and Γui are obtained as analytical functions of the spatial coordinates. In Algorithm 1, Γi and Γui have been specified through Gaussian functions; other possibilities for correlation functions have been described, for instance, by . We have chosen not to inflate or deflate Pf directly and to modulate the amplitude of background covariances only through the terms of Eq. (). In this way, we reduce the number of parameters that need to be specified. As a matter of fact, for the combination of observations and background in the analysis procedure, the m by m covariance matrices are never directly used. Instead, the matrices used are the covariances between grid points and observation locations, Gib=PibHTi (specifically only the ith row of this matrix is used), and the covariances between observation locations Sib=HiPibHTi. Hi is the local observation operator, which is a linear function, i.e., Rm→Rpi.

The local observation error covariance matrix Ri is written as the constant observation error variance σio2 multiplying the correlation matrix Γio as follows: 7Ri=σio2Γio. Γio is often the identity, but other choices are possible. For instance, if some observations are known to be more accurate than the average of the others, then the corresponding diagonal elements of Γio can be set to values smaller than 1. The observation uncertainty can vary in time and space, accordingly to P1; however, its spatial structure is fixed and depends on the analytical function chosen for Γio. Note that the observation error is not only determined by the instrumental error, but it also includes the representativeness error , which is often the largest component of the observation error. The representative error is a consequence of the mismatch between the spatial supports of the areal averages reconstructed by the background and the almost point-like observations.

The spatial structures of the error covariance matrices are determined through the matrices in Eqs. () and (). At this point, we need to set σiu2 and σio2 to scale the magnitude of the covariances. In the process described below, we will see that the two variances are completely determined by two scalars, ε2 and ν, also defined below, that we assume to be known before running the spatial analysis. This prior knowledge defines the constraints that the solution has to satisfy and allows us to choose one particular solution among all the possibilities. σiu2 and σio2 characterize the region around the ith grid point as a whole, without distinguishing between the individual observations. We introduce two relationships linking σiu2 and σio2 through two additional variances, both expressing the uncertainty of a quantity over the same region around the ith grid point. σib2 is the average background error variance, and σif2 is the average forecast error variance. The two relationships are as follows: 8ε2=σio2/σib29σib2=σif2+σiu2. ε2 is a global variable, and it is the relative precision of the observations with respect to the background. Equation () implements P3, and ε2 should be set to a value smaller than 1. For example, ε2=0.1 means that we believe the observations to be 10 times more precise an estimate of the true value than the background. Equation () is an adaptation from Eq. (). The next two relationships we introduce have the objective of estimating σif2 and the empirical (i.e., based on data, not on theories) estimate of σiob2, which is the sum of σio2 plus σib2, taken directly from the forecasts and the observed values. σiob2 is used to obtain a reference value to judge if the ensemble spread is adequate. The equations are (the averaging operator … is defined as in Algorithm 1) as follows: 10σif2=νdiagSif11σiob2=νyio-yib2. ν is an inflation factor that can be used to obtain better results (e.g., via the optimization of cross-validation scores or other verification metrics). In addition, ν is introduced because Eq. () is sensitive to misbehavior in the data when it is applied using data from one single time step. Proper estimates of σif2 and σiob2 would require more than just one case, and the ideal situation would be to consider numerous situations characterized by similar weather conditions. Instead, we prefer to stick to P5. The estimation of σiob2 is not resistant in the sense defined by . A few outliers in Eq. () may have a significant impact on σiob2. The introduction of ν makes the estimation procedure more resilient in the presence of outliers and other nonstandard behavior. Equation () is used for diagnostics in data assimilation , and it is consistent with P2. The combination of Eqs. () and () returns a rough empirical estimate of σib2 that is as follows: 12σib′2=νyio-yib21+ε2.

As a final step, to set σiu2 and σio2, we distinguish between three situations. The first situation is when the ensemble spread is likely to underestimate the actual uncertainty because the background is missing an event or the spread is too narrow. The test condition is σif2<σib′2. We will refer to this situation as the ensemble being overconfident or underdispersive. This is the case when a positive σiu2 is needed in Eq. (), and we set its value such that σib2 in Eq. () is equal to σib′2 in Eq. () in the following: σiu2=σib′2-σif213=ν(yio-yib)2/(1+ε2)-diag(Sif)14σib2=ν(yio-yib)2/(1+ε2)15σio2=ε2ν(yio-yib)2/(1+ε2).

The second situation is when the ensemble spread is consistent with the empirical estimate of σib2. The test condition is σif2≥σib′2 and σif2>0. We will refer to this situation as the ensemble spread being adequate. In this case, the background information is given by the ensemble, without adjustments, and is as follows: 16σiu2=017σib2=σif2=νdiag(Sif)18σio2=ε2σif2=ε2νdiag(Sif). Equations ()–() have been written with many details, in a somewhat pedantic way, to emphasize the differences between those two situations. When the ensemble is underdispersive, the sum σio2+σib2 is bounded by the upper limit σiob2. This is not the case when the ensemble is adequate. It is worth remarking that the test conditions are independent of ν. In fact, for instance, the test condition for the first situation can be equivalently written as yio-yib2>(1+ε2)diag(Sif).

The third situation is the special case in which the background is deemed as perfect; that is, when all the observed values and all the forecasts, at all observation locations, have the same value. In practice, this occurs in the case of no precipitation. In this case, σif2=0 and σib′2=0. Errors are not Gaussian in this case, so then Eqs. () and () are not needed anymore, as discussed in the next section (Sect. ).

With reference to the working assumptions stated at the beginning of this section, they can now be reformulated in more precise mathematical terms by referring to the above definitions and equations. P1 led us to Eqs. () and () and supported our choice of a grid point by grid point implementation of the algorithm. P2 led us to Eq. () and subsequent equations, including the term σiob2. P3 led us to the introduction of ε2 in Eq. (). P4 is also a key reason for having an algorithm that can be optimized as a function of the grid point under consideration. Other than that, P4 has not been used explicitly in this section, since it will, in general, affect the specification of Γui in Eq. (). In this section, we do not postulate any formulation of Γui as being preferable to another; this depends on the application. P4 led us to the specification of Γui in Algorithm 1 as a location-dependent matrix through Di, which is the length scale determining the decrease rate of the background error unexplained by the forecasts. This length scale is set in both Algorithm 1 and Sect.  as a function of the observational network density in the surrounding of the ith grid point. In this sense, Di is dependent on the characteristics of precipitation as they can be observed by our network. This point is discussed further in Sect. . As far as we know, and as stated in the introduction, this is an innovative part of our interpolation scheme since most of the other schemes do postulate that a single analytical correlation function or semi-variogram is valid for the whole spatial domain considered. P5 led us to the introduction of ν in Eqs. () and ().

The expressions for the analysis and its error variance are direct results of the linear filter theory , and they are derived in several books based on different formulations e.g.,. The analysis at the ith grid point is equal to the background plus a weighted average of the pi innovations, while the analysis error variance is derived from the error covariance matrices as follows: 19xia=xib+Gii,:bSib+Ri-1yio-yib20(σ2)ia=Piiib-Gii,:bSib+Ri-1Gii,:bT. Equations () and () are also typical of optimal interpolation, and the formulation used is similar to the one adopted by , which follows from . It is worth remarking that the background used in Eq. () is the ensemble mean, since we have assumed xb=xf in Sect. . The ensemble members are used to determine the background error covariance matrices. The method is a modified version of EnOI , where an ensemble of synchronous realizations is considered instead of a time-lagged ensemble approach. As an additional difference between EnSI-GAP and other methods, it should be noted that the grid point by grid point implementation makes it possible to modify the interpolation settings to adapt them to the different regions in the domain, as discussed in Sect. .

The special case of a perfect background, as introduced in Sect. , leads to a perfect analysis of xia=xib. Because all the information available shows an exceptional level of agreement, we have chosen to set the analysis error variance to zero (i.e., background is the truth), such that for those points the analysis probability distribution functions (PDFs) are Dirac's delta functions, and this has consequences for the inverse transformation, as discussed in Sect. .

A one-dimensional grid with 400 points and a spacing of 1 spatial unit, or 1u, is considered. The domain covers the region from 0.5 to 400.5 u, and the generic ith grid point is placed at the coordinate iu. A simulation begins with the creation of a true state, and then observations and ensemble background are derived from it.

The simulation presented here is shown in Fig. a. For each grid point, the true value (black line) is generated by a random extraction from the gamma distribution, with the shape and rate set to 0.2 and 0.1, respectively. To ensure spatial continuity of the truth, an anamorphosis is used to link a 400-dimensional multivariate normal (MVN) vector with the gamma distribution. The samples from the MVN distribution, with a prescribed continuous spatial structure, are obtained from the descriptions by in chap. 12.4. The MVN mean is a vector with 400 components all set to zero, and the covariance matrix is determined using a Gaussian covariance function with 10 u as the reference length used for scaling distances. The effective resolution (Sect. ) of the truth is then 10 u.

The ensemble background (gray lines in Fig. a) on the grid, with 10 members, is obtained by perturbing the truth. The background values at the observation locations are obtained from the members using nearest-neighbor interpolations. For each member, the truth is perturbed by shifting it along the grid by a random number between -10 and +10 u, thus simulating the misplacement of precipitation events. Then, the effective resolution of the member is set to be coarser than that of the truth. The true values are multiplied by coefficients derived from a uniform distribution, with values between 0.05 and 2 and a spatial structure function given by a MVN with a Gaussian covariance function, with a reference length extracted from a Gaussian distribution with a mean of 50 u and a standard deviation of 5 u. Two special regions are considered, and they are shown with the bright shading in Fig. . In region R1, between 50 and 150 u, each background member follows an alternative truth (i.e., it is literally being derived from a different truth) that is everywhere different from 0 mm. In R1, the background is neither accurate nor precise, and this leads to the occurrence of misses and false alarms. In region R2, between 200 and 300 u, none of the ensemble members simulate precipitation while the true state reports precipitation. In this region, the background is precise but not accurate since the ensemble is missing, or poorly representing, an event which is otherwise well covered by observations. Because we had to ensure the continuity of the background, we have enforced smooth transitions between the two regions and their surroundings. For example, R2 is actually beginning a bit after 200 u and ending a bit before 300 u.

The number of observations (blue dots in Fig. a) is set to 40. The observed value at a location is obtained as the true value of the nearest grid point, plus a random noise that is determined as a random number between -0.02 and 0.02 that multiplies the true value. The procedure is consistent with the fact that observation precipitation errors should follow a multiplicative model . The observation locations are randomly chosen. There are five between 1 and 100 u, 30 between 101 and 300 u, and five between the 301 and 400 u. The distribution is denser in the central part of the domain and sparser closer to the borders.

The effect of the Gaussian anamorphosis is shown in Fig. b. The transformed precipitation varies within a smaller range than the original precipitation, thus effectively shortening the tail of the distribution, reducing its skewness, and making it more similar to a Gaussian distribution.

An example application of EnSI-GAP is presented in Algorithm 1. The choices that are kept fixed and that will not vary for the whole Sect.  are described in this paragraph. The localization matrix Γi of Eq. () is specified using Gaussian functions, taking the form of those used in Algorithm 1 for Zi and Vi, with Li=25u for all the grid points. The sensitivity of the results to variations in the specification of the scale matrix will be investigated in Sect. ; nonetheless, the strategy for determining Di will always be the same whether we choose to use a Gaussian function, as in Algorithm 1, or an exponential function. Di is determined adaptively at each grid point, as shown in Fig. c, as the distance between the ith grid point and its third-closest observation location. In addition, Di has been constrained to vary between 5 and 20 u. The tool used to quantify the impact of the spatial distribution of the observations on the analysis is the integral data influence IDI;; this is a parameter that stays close to 1 for observation-dense regions, while it is exactly equal to 0 in observation-void regions. In practice, the IDI at the ith grid point is computed here as the analysis in Eq. (), when all the observations are set to 1 and the background is set to 0. IDI has been adapted to EnSI-GAP in the sense that only the scale matrix is considered in the calculation of Pbi in Eq. () because the part of Pbi, taking into account the atmospheric dynamics, does not depend on the observational network. Where the IDI is close to zero, the analysis is as good as the background. Figure d shows the IDI when Di is set as the distance between the ith grid point and its third-closest observation location. EnSI-GAP is very sensitive to the tuning of Di, and its estimation is further discussed in Sect. .

The evaluation of analysis versus truth at grid points are evaluated using two scores that are applied over precipitation values. The mean squared error skill score (MSESS) quantifies the agreement between the analysis expected value and the truth. The continuous ranked probability score (CRPS) is a much used measure of performance for probabilistic forecasts. The definitions of both scores can be found, for example, in . The MSESS has been used for studies on precipitation by, for example, , while applications of CRPS to precipitation can be found, for example, in the paper by . The definitions adapted to our case are reported here, in the following: 23MSESS=1-1m∑i=1m(x̃ia-x̃it)21m∑i=1m(x̃it-c)2;c=1m∑i=1mx̃it. 24CRPS‾=1m∑i=1m∫0∞[Fa(αia,βia;y)-Ft(x̃it;y)]2dy. CRPS‾ is the CRPS averaged over all the grid points. The squared difference is between the continuous cumulative distribution functions (CDFs), namely Fa(αia,βia;y), which is the gamma analysis CDF at point i, with the indicated shape and rate parameters, evaluated at the value y; Ft(xit;y) is the Heaviside function, which is equal to 0 when y<xit and equal to 1 when y≥xit.

A sensitivity analysis on variations in the scaling parameters ν, ε2, and in the correlation function defining Γiu is presented. At the same time, the operation of EnSI-GAP is shown step by step.

The sensitivity study considers three situations which are also used in Figs. –. A reference setup is defined with ν=0.5 and ε2=0.5. Then, we consider a perturbed situation in which ε2=0.1 and the observations are assumed to be 10 times more precise than the background. Finally, a situation is considered with ν=0.1, where only a small part of the ensemble spread determines σib2. In addition, two different functions are used for the specification of the scale matrix Γiu, namely a Gaussian function and an exponential function. In the scientific literature, both functions have been used to specify correlations for spatial analysis of precipitation. For instance, the Gaussian function is used by and the exponential function by .

The scaling of the covariances, which in turn determines the weights used in the analysis, is determined by σio2 (Eq. ) and σiu2 (Eq. ), which are related to σib2 and σif2. In Fig. , the variances are shown, and their values do not depend on the correlation functions; they depend only on ν and ε2. In the reference situation, Fig. a, the ensemble spread is adequate (σiu2=0) for 58 % of the grid points, and it is overconfident between points 160 and 300u; most of these points are in R2. σib2 and σio2 are larger in R1 and R2 than outside those two special regions, as expected, and in R2 σib2 is almost equal to σiu2 because the ensemble is missing the precipitation event. On average, σib2=0.15 and σio2=0.07. In the case of ε2=0.1 in Fig. b, the percentage of points in which the spread is adequate decreases to 27 %, such that the scale matrix is used more than in the reference situation. The mean values become σib2=0.19 and σio2=0.02. In the case of ν=0.1 in Fig. c, the reduction in σib2 is evident and, on average, σib2=0.03 and σio2=0.015. The percentage of points for which the spread is adequate is determined by ε2, then, in Fig. c, it is the same as in the reference situation.

The transformed precipitation analysis is shown in Fig. , and the analysis in the original precipitation space, after the inverse transformation, is shown in Fig. . The layout of the figures is organized such that each row corresponds to the same row in Fig. . In the left column a Gaussian function has been used in Γiu, and in the right column an exponential function has been used.

By comparing Figs.  and with Fig. , it is possible to study the impact of different choices on the analysis in the transformed space. By increasing (decreasing) the error variances, the analysis spread increases (decreases) too. The comparison between Gaussian versus exponential correlation function shows that, given the same values of ν and ε2, the exponential function shows a larger analysis spread. The analysis expected value does not vary significantly among panels that are on the same row, thus indicating that the expected value is not that sensitive to the correlation function chosen. For instance, the MSESS for Fig. c is 0.78, while for Fig. d it is 0.77. The CRPS‾ for Fig. c is 0.43, while for Fig. d it is 0.44. For the other panels of Fig. , the MSESS and CRPS‾ have lower values. The comparison to the reference situations of Fig. a and b show that the analysis expected values in Fig. c and d fit the observations better, and the analysis spread is more likely to include the true values. The situation is the opposite in Fig. e and f, where the reference setup performs better.

The analysis of over 100 simulations confirms the considerations we have made above on the basis of a single simulation. If we consider 100 simulations, the results are shown in Table  in the EnSI-GAP column. The configuration leading to the best results, in terms of both MSESS and CRPS‾, is the one shown in Fig. c. The worst results were obtained when ν=0.1.

In Fig. , the EnSI-GAP results are shown for the same settings used in Sect. , without applying data transformations and just interpolating the original precipitation values. The layout of Fig.  is the same as in Fig. . The best results are found in the configurations of Fig. c and d, as in Fig. . The agreement between analysis expected values and true values is similar to those shown in Fig.  in that the differences are small. For instance, the MSESS of Fig. c is 0.76. The most evident difference is in the spike in analysis spread between 50 and 100u, which is present in Fig.  and absent in Fig. . This may indicate that, without data transformation, it is more likely for one to obtain unrealistically large analysis spread.

The comparison of the analysis spread between Figs.  and shows also that, without data transformation, it is more likely that the true values fall outside the analysis spread shown in the figure. For example, in Fig. d the analysis spread includes the true values for 75 % of the grid points when precipitation is higher than 1 mm; with respect to Fig. d, that percentage is 53 %. The CRPS‾ for Fig. c is 0.59, while for Fig. d is 0.56.

If we consider 100 simulations, the results are reported in the column “no transformation” of Table . The MSESS is often comparable or even slightly higher than using EnSI-GAP, which confirms that the analysis expected value provides a good fit of the truth – even without data transformation. The benefits of the data transformation are in the better representation of the analysis PDF, as can be seen by comparing the CRPS‾, because the analysis with the data transformation performs better for all configurations.

In Fig. , the results are shown when the ensemble background is not considered; instead, a single member or the ensemble mean are considered. In this case, in Eq. (), Pif is not considered, and Pib is determined only by Γiu. Note that, in R2, the differences between Figs.  and are very small, since in R2 Pib is almost equal to Γiu anyway. In the figure, Γiu is specified only through an exponential function, which shows better results than the Gaussian function as in the previous two sections. In the left column, the results are shown when the best ensemble member is chosen as the background. The best member is defined as the one that fits the observations better in terms of minimizing the squared deviations between the background and observed values. In the right column, the ensemble mean is chosen as the background.

When comparing the three different configurations, the general considerations are the same as in Sects.  and . The best results have been obtained with ε2=0.1 and ν=0.5, in Fig. c and d. In particular, the analyses based on the ensemble mean perform better than with the best member, which may sometimes deviate significantly from the truth as it happens between 50 and 100 u. The scores support this conclusion. For Fig. c, MSESS=0.64 and CRPS‾=0.54. For Fig. d, MSESS=0.72 and CRPS‾=0.50.

If we consider 100 simulations, the results are reported in the column labeled no ensemble in Table , and this is only for the case when the ensemble mean is considered as the background. EnSI-GAP performs better than in the case of a deterministic background for almost all configurations. Only in the cases of ε2=0.1, ν=0.5, and Γiu defined through an exponential function, does the analysis performs better without considering the ensemble. In fact, the MSESS and CRPS‾ mark this configuration as the one returning the best results among all configurations.

If we consider the 100 simulations on the one-dimensional grid, the comparison of results in Table  between the different implementation modes (EnSI-GAP, no transformation, and no ensemble) brings us to the following conclusions on the benefits of EnSI-GAP. The use of Gaussian anamorphosis ensures a more accurate probabilistic analysis than without any data transformation, as demonstrated by the fact that EnSI-GAP shows the best CRPS for almost all the configurations. The use of an ensemble in the definition of Pib allows the analysis to be more resistant to misbehavior in the background, as shown by the better scores obtained by EnSI-GAP for most of the configurations.

The comparison between exponential and Gaussian correlation functions in Γiu favors the exponential function. From geostatistics, we know that a Gaussian variogram model is infinitely differentiable at the origin . This imposes unrealistic smoothness constraints on the analysis and, as a side effect, causes an overconfidence in the analysis, leading to an underestimation of the analysis uncertainty and a tendency to produce high and low values outside the range of observations. Those effects are more evident in places where the observational network is sparse and the spatial analysis scheme is less constrained by the observations. The risks related to the use of a Gaussian covariance are described by .

The EnSI-GAP implementation in Algorithm 1 requires the specification of four parameters, namely D, L, ν, and ε2. In the previous sections, the last two parameters are considered in the sensitivity study. In the last paragraph of this section, some general considerations on the setup of D and L are presented. The optimization of Di is an important part of EnSI-GAP, as remarked in Sect. . There are classical methods for estimating the statistical structure of background errors as a function of observation location separation , based on minimizing the deviations between theoretical structure functions and empirical estimates from data. When the variation is bounded, the covariance function is equivalent to a variogram, which is used in geostatistics . Often, one single value of Di is considered valid for the whole domain, as, for instance, by . In accordance with P4 of Sect. , we want Di to be dependent on the spatial location. The blending of different variograms, using regional weights, has been done for temperature by . For precipitation, the method described by adapts the estimation of variograms for daily precipitation anomaly fields to the density of the observational network. In this document, we follow a simple procedure in which each time step is considered independently from the others (P5; Sect. ), and we take advantage of the choice to implement the algorithm based on a grid point by grid point elaboration. The observations and background are combined into the analysis because we want some observations, not just one observation, to have an impact on the analysis in the surrounding of a point. In Sect. , the IDI has been introduced, and it is shown in Fig. d. We have configured the simulation such that the IDI is almost always larger than 0.8, which can be roughly interpreted as having at least one observation, or possibly a few, significantly influencing the analysis everywhere over the domain. The procedure we suggest for setting Di is the following: have the objective of your investigation clear in your mind; choose a functional form of the scale matrix that suits your objective; test different strategies for the determination of Di, based on an inspection of the IDI, showing the regions of the domain that would be more influenced by the observations; select the range of values for Di that may lead to acceptable results in the spatial analysis; and refine the optimization of Di by evaluating EnSI-GAP performances on the basis of skill scores that serve your goals.

Li depends on the characteristics of the background used, and it should reflect the size of typical precipitation events occurring in a region. If we assume that it is reasonable to use the observational network to refine the effective resolution of the background, then we can imagine that Li should be set to values larger than Di.

Algorithm 1 has been used over a grid with 2.5 km of spacing, which is the resolution of the MEPS grid (see Sect. ). The parameters are ε2=0.1, ν=0.1, pmx=200, and Li=50km constant. A Gaussian function has been used in Γiu. Di is estimated adaptively on the grid as the distance between the grid point and the 10th closest observation location with upper and lower bounds of 3 and 10 km, respectively. The settings are such that the analyses would stay much closer to the observations than to the forecasts, where observations are available. The analysis uncertainty will reflect, locally, both the forecast ensemble spread and the averaged innovation. The two parameters of pmx and Di are used to limit the number of observations that can influence the analysis at a grid point. The localization parameter, Li, is set to a rather large value, such that the dynamics of the forecasts ensemble are evident in the results. The observation error covariance matrix of Eq. () is defined with a diagonal Γio, which is a situation where radar-derived and in situ observations are assumed to have the same precision; moreover, we are ignoring the spatial correlation of radar-derived observation errors. An investigation of spatially correlated radar-derived observation errors is outside the scope of this study. Note that those settings are useful for the illustration of the method, while for operational applications other settings may be more appropriate, such as a smaller value of Li or a more sophisticated characterization of the observation errors, for example.

As an example of application, the Gaussian anamorphosis described in Sect.  is applied here to the transformation of hourly precipitation over Sogn og Fjordane on 30 July 2019 at 15:00 UTC. The procedure is sketched in Fig. . In Fig. a, the distribution of values for an arbitrary ensemble member is shown. In Fig. b, the empirical CDFs of the 10 ensemble members are shown as gray dots, and the pink lines represent the gamma CDFs that better approximate each empirical CDF. The values of the gamma shape and rate are then averaged to obtain αD and βD, which are reported in the figure. Figure c displays the CDF for the standard normal, which is the target CDF in our transformation scheme. Finally, Fig. d shows the distribution of the transformed values for the background ensemble mean, which is used as the background for the analysis in Eq. (). In Fig. a and d, the distribution of values for the observations is also shown, though the values are not used for the estimation of the gamma parameters. The effects of the Gaussian anamorphosis in adjusting the distribution of values into a bell-shaped distribution are clearly evident.

The four different steps of the data transformation for an arbitrary value, at approximately 2 mmh-1, are also highlighted with circles to guide the reader in the order of the application of each step.

Figure  shows the hourly precipitation data for 30 July 2019 at 15:00 UTC over South Norway. The observational data are shown in Fig. a. For each grid box, the average of the radar-derived precipitation and in situ measurements within that box is shown. Note that the box-averaged observations are used only for illustration because the analysis is using each observation. Grid points that are not covered by observations are marked in gray. In Fig. b, the background ensemble mean derived from a 10-member ensemble forecast is shown, while six of the 10 ensemble members are shown in Fig. . The 10-member ensemble shows realistic precipitation fields; moreover, they are rather similar, at least in terms of the weather situation at the meso-β scale. Weather forecasters can be quite confident in stating that heavy precipitation is likely to occur over western and southern Norway, while is less likely over eastern Norway. The forecast uncertainty is large enough that it is difficult to predict exactly which subregion will be affected by the most intense showers. The observations confirm that showers occur along the coast of western Norway, and that the most intense precipitation event is located in Sogn og Fjordane (the black box in Fig. ; note that approximately half of the box is not covered by observations). Figure c shows the analysis, specifically the analysis mean at each grid point. In this case, the spatial analysis acts almost as a gap filling procedure to fill in empty spaces in between observations with the most likely precipitation values. The analysis of precipitation is consistent with the impacts of the intense weather event described in the report by . As prescribed by our EnSI-GAP settings, the analyses over observation-dense regions are not that different from the observed values.

One of the main innovations of EnSI-GAP, compared to traditional spatial analysis methods , is the specification of anisotropic background error covariances between grid points through nonstationary covariance matrices. Two visual representations of the correlations associated with those covariances are shown in Fig.  for points A and B. With reference to Pib, the background error correlations between the generic ith grid point and the other grid points, evaluated at the ith grid point, are the ith row (or column) of the correlation matrix Γib, which is obtained as follows: 25Γibi,:=Pibi,:Pibi,idiagPib. The correlations are shown instead of the covariances because we are interested in the shape of the covariance patterns, and correlation is a quantity which is then more correct to compare between the two points. For visualization purposes, in Fig.  the correlations have been downscaled over a finer-resolution grid to highlight asymmetries. The closest 200 observations are shown with different symbols, depending on rain occurrence. The two maps in Fig.  are rather different. For point A, the correlation extends more to the west than to the east. The point is located in a valley floor, which is rather sheltered from the main atmospheric flow, and this seems to be represented in its correlation pattern which rapidly decays as we move upwards. The area where the correlation is higher than 0.6 is confined within approximately 5 km in any direction from point A. At point B, the situation is different, and the correlation extends more to the east than to the west. The point is located on a plateau at 911 m and the correlation pattern follows the main atmospheric flow from west to east. The no precipitation observations 20 km northeast of point B have correlations that are comparable to those of observations at 10 km west of B.

The evolution in time of the hourly precipitation fields is shown in Fig.  for observations, background, and analysis at three different times, namely 14:00, 15:00, and 17:00 UTC. It is worth noticing that the example used to illustrate the data transformation process in Fig.  refers to the Sogn og Fjordane domain at 15:00 UTC. The background is smoother than the observed field and shows scattered showers for 14:00 and 15:00 UTC; then, a wider precipitation cell over point B is shown at 17:00 UTC. The observed fields show a large variability over short distances, and the difference between two adjacent points can be as large as 30 mmh-1. According to P4 of Sect. , in data-dense areas we would like the analysis to stay closer to the observed value than in data-sparse areas. Point A is in a densely observed area, while point B is almost in the middle of the observation-void region, and the closest observations are located at a distance of approximately 10 km. Di at point A is closer to 3 km, while at point B it is closer to 10 km. This ensures a higher effective resolution at point A than at point B. At 14:00 UTC, the observed value at point A (from radar-derived estimates) is over 30 mmh-1, and a sharp gradient from southwest to northeast is evident. The gradient is so intense that the nearby points southwest of point A, only 3 km apart, show almost no precipitation. The background indicates that a maximum of the field can occur between point A and B. The analysis matches the observations, smoothing out their spatial variability, such that at point A the analysis value is less than 10 mmh-1. A precipitation maximum of more than 30 mmh-1 has been reconstructed in the analysis between points A and B, which is consistent with the gradient in the observations and the pattern in the background. At 15:00 UTC, the radar-estimated precipitation at point A is again over 30 mmh-1, but there are several points in its surroundings with similar values, such that the local gradient of the field is less steep, and it shows a decrease in precipitation east of point A. The background also shows that it is more likely to find intense precipitation immediately to the west of point A than to the east. A second precipitation maximum is found in the background, north of point B. The analysis ignores this second precipitation maximum, since it is not supported by observations. The analysis around point A closely matches both the observed values and the gradient, such that the field in the observational-void area does not show significant local extremes. The shape of the area with precipitation rate higher than 30 mmh-1 around point A is similar to the pattern of point A correlations higher than 0.6 in Fig. . At 17:00 UTC, all the observations report values smaller than 20 mmh-1, and the analysis reconstruct a maximum of over 30 mmh-1 at point B. In this case, the observations and background precipitation yes/no patterns are similar, and they both show a southeast to northwest gradient. The analysis estimates a narrow band of precipitation around point B, where values of more than 20 and up to 30 mmh-1 are extrapolated. The extrapolated values are consistent with the effects of the extreme event reported by MET Norway .

The time series of hourly precipitation at points A and B are shown in Fig. . At point A, the graphs show the time series of the (aggregated) observation, background, and analysis, together with the estimated uncertainties. Note that the observation is used in the analysis at point A. At point B, observations are not available. For the background, the percentiles are derived from the 10-member forecast ensemble through a linear interpolation of the empirical cumulative distribution function. For the analysis, the percentiles are derived from the estimated parameters of the gamma distribution representing the marginal probability density function (PDF) of the analysis at the points. In general, EnSI-GAP forces the analysis to follow the observations more closely than the background, and the analysis uncertainty is smaller than that of the background. As a consequence, the timing of the precipitation onset is also better represented in the analysis. At point A, the PDF of the precipitation analysis, between 10:00 and 13:00 UTC, indicates with certainty that it is not raining. From 14:00 UTC onward, the analysis PDF is a gamma. From 14:00 to 23:00 UTC, the observed values are within the analysis envelopes shown in Fig.  for 50 % of the hours, which is a consistent improvement compared to the background. For the other 50 % of the hours, the observed values lie outside the envelopes, and 14:00 and 19:00 UTC are the 2 h for which the deviations between observations and analyses are the most evident. For those 2 h, the local variability in the precipitation field is extremely large, as shown in Fig.  for 14:00 UTC, and the observed values at point A are outliers, if compared to their neighbors. With respect to the precipitation yes/no distinction, from 14:00 to 23:00 UTC, the analysis clearly shows that precipitation is occurring at the point, while the background is more uncertain. At point B, the analysis uncertainties between 10:00 and 12:00 UTC are so small that the analysis is exactly 0 mmh-1, despite there are no observations exactly located at that point. From 13:00 UTC onward, the analysis follows a gamma PDF and the spread is wider at point B than at point A. The increased analysis spread reflects the increase in the uncertainty in predicting the tails of the PDF where no observations are available. It is perhaps remarkable that, even for observationally dense regions such as at point A, the analysis spread remains quite large.

EnSI-GAP can support weather forecasters and civil protection by filling in the empty spaces in the observational networks. The analysis seamlessly merges the high-resolution NWP models with observations, and it remains closer to the observed values where they are available. The predicted fields are easy to interpret by experienced staff that are aware of the spatial distribution of the observations and the characteristics of the NWP considered. The analysis is more precise and accurate than the background where observations are available, as at point A in Sect. , and also for the onset of precipitation. Uncertainty on the estimate at a point increases as the number of nearby observations decreases. The analysis procedure also modifies the field where observations are not available in a credible way, as at point B in Sect. . The uncertainty estimates can be used to have an idea of the extreme values that may occur in a region, which is useful information both for the nowcasting of an event and in the subsequent reporting phase.

In Sect. , the observed values show strong gradients over small distances. The spatial analysis finds the best estimates of true values, which are areal averages, as discussed in Sect.  and defined in Eq. (), with spatial supports determined by the EnSI-GAP settings. In Fig. , at 14:00 and 19:00 UTC, the representativeness errors of the observations at point A are particularly large with respect to the spatial supports of the true values, such that the corresponding observations are filtered out as outliers by the analysis, and their values are unlikely to occur according to the analysis PDF. If the ensemble is overconfident, according to the definition of Sect. , it is, in principle, possible to modify the analysis PDF by reducing Di such that the analysis spread would become larger, which in this case would correspond to a reduction in the spatial support for the true values, and the analysis envelope would be more likely to include the observations. However, when a single observation is an outlier with respect the neighboring observations, as in Fig.  at 14:00 and 19:00 UTC, the tuning of Di to include the observation in the analysis PDF may lead to unrealistic discontinuous patterns in the analysis due to the sudden jump in the spatial supports used in the definition of true values. In general, a very dense observational network, with observations that are closer than the effective resolution of the background, has the following two effects on the analysis where precipitation varies significantly over small distances: (i) it forces the analysis expected value to stay close to the areal average of the observations, and (ii) it increases the observations and background error variances because of the increased value of the term yio-yib2 in Eqs. ()–(); this will, in turn, increase the analysis uncertainty in Eq. (). The trade-off between the accuracy and precision of the analysis at a point ultimately depends on the objective of an application.

The EnSI-GAP Algorithm 1 has been used. The spatial analysis predicts values at those station locations used for cross-validation. The fixed parameters in this implementation are pmx=200 and Li=50km. A Gaussian function has been used in Γiu. Di is estimated adaptively at each location as the distance between that point and the 10th closest observation location, with upper and lower bounds of 3 and 10 km, respectively.

The parameters that are allowed to vary and that are the objective of the sensitivity analysis that follows are ε2 and ν.

There is an important difference here with respect to Sect. ; in this example, the radar-derived estimates are assumed to be less precise than the in situ observations but more precise than the background. The in situ observations are assumed to be 10 times more precise than the background; thus, ε2 is set to 0.1, as in Sect. . However, the radar-derived observations are assumed to be only two times more precise than the background, or, in other words they are five times less precise than the in situ observations, and the elements of the diagonal matrix Γio corresponding to radar observations are set to five, instead of one as for the in situ observations. The background ensemble and analysis PDF values considered are those extracted at the locations of stations used for the cross-validation.

Figure  shows the distribution of values for selected percentiles of the background ensemble and analysis PDF as a function of the independent observations. The distribution of the observed values has been divided into intervals; they are (units of mmh-1) 0–0.1, 0.1–0.5, 0.5–1, 1–2, 2–3, 3–5, 5–10, and 10–35. The number of samples within each interval is shown in Fig. a. Note the logarithmic scale on the y axis. Most of the observations are smaller than 1 mmh-1; nonetheless, there are still more than 1000 values that are greater than 1 mmh-1. Considering an arbitrary observation interval for each probabilistic prediction, either for background or analysis, we have computed the following percentiles: 10th, 25th, 50th, 75th, and 90th. The black line in Fig.  shows the average median within each interval, while the regions between the 90th and the 10th percentiles and the 75th and the 25th percentiles are shown with gray shading. The diagonal (1:1) ideal line is shown as a dashed line. The background is shown in Fig. a. The background envelope deviates significantly from the diagonal, especially for values greater than 2–3 mmh-1. The analysis PDFs are shown in the other panels for different EnSI-GAP configurations that are clearly indicated within each panel. The angular coefficients of the regression lines that better fit the analysis medians are reported in Fig. b–d. In all cases, the medians are closer to the diagonal for the analyses than for the background. The analysis biases that are conditional to the observations are always smaller than that of the background. As expected, by giving more weight to the observations, with ν=1 and ε2=0.1, the analysis bias conditional to the observations decreases. If we compare different analysis configurations, the medians vary less than the other percentiles, and this indicates that variations in the EnSI-GAP configuration impact on the spread of the analysis PDF (i.e., analysis uncertainty) more than on its central moment. Figure b and c show the two extreme situations, while Fig. d displays an intermediate situation. The uncertainty is more sensitive to variations over ν than over ε2. In the case of ν=0.1 and ε2=0.1, the angular coefficient of the regression line approximating the analysis median reaches its the best value; however, the analysis spread is small, and the independent observations fall above the 90th percentile. In the case of ν=0.1 and ε2=0.1, the angular coefficient of the regression line has the best value. For the two cases with ν=1, the independent observations fall into or around the 90th percentile of the analysis. Once again, it is the specific application that would determine the best combination of parameters to use.

Figure  shows the equitable threat score (ETS) for the background and analysis means. A total of four different analysis configurations are shown. The independent observations are used to judge if events have occurred. The condition defining the “yes” event for either observation or prediction is that the corresponding value must be higher than the precipitation threshold specified on the x axis. For all predictions, it is more likely that a predicted “yes” event corresponds to an observed “yes” event for smaller thresholds than for the higher ones. The added value of the analysis over the background is evident for all configurations. The two configurations with ν=1 present similar ETS curves, though the one with ε2=0.1 performs better. The same holds true when ν=0.1, though, in this case, the ETS is more sensitive to variations in ε2, and the analysis performance decreases faster with the increase in ε2.