The properties of high-dimensional spaces often defy our intuition based on two and three dimensions . Apart from the well-known fact – sometimes called the empty space phenomenon – that the number of samples needed to obtain a given coverage grows exponentially with dimension, there are other less appreciated features of high-dimensional spaces . For example, almost every point is an outlier in its own projection, and independent vectors are almost always orthogonal. The latter property is called waist concentration and, more precisely, states that when the dimension increases, the angles between independent vectors become narrowly distributed around the mean π/2, with a variance that converges towards zero.

The properties of high-dimensional spaces are sometimes called the curse and sometimes the blessing of dimensionality, depending on the considered problem. In the present context these properties turn out to be a blessing as they strongly simplify the analysis and make analytical results possible.

As a simple example we consider a cube in N dimensions with side d and centred around 0. The cube has 2N vertices with the positions d(±1/2,±1/2,…±1/2). The distance between each vertex and the centre is dN/2. The volume of the cube within a distance ϵd of the edge is (dN-(d-ϵd)N)/dN=1-(1-ϵ)N and the volume of the inscribed sphere is πN/2(d/4)N/Γ(N/2+1). The situation is shown in Table  for a unit cube (d=1) for different values of N. For N=100 there are more than 1030 vertices

Comparable to the number of atoms in 30 t of water or the number of bacteria on the Earth: a factor of 1 million larger than the estimated number of stars in the universe.

We first look at a very simple example to describe the general idea of concentration of measures. Consider N iid random variables xi, i=1,2…N, each with mean μ and variance σ2. For the sum of the variables, ∑ixi, we have for the expectation and variance 1E∑ixi=∑iE(xi)=Nμ and 2Var∑ixi=∑iVar(xi)=Nσ2. Thus, Var(∑ixi)/E(∑ixi)=1Nμσ. Therefore, when N grows, both the expectation of the sum and the width of its distribution will grow, but the relative width will decrease. We can therefore, with some reason, say that the distribution of the sum becomes more and more sharply defined around its mean. If we normalize the sum with N to get the mean, x‾=∑ixi/N, we have E(x‾)=μ and Var(x‾)=σ2/N. Therefore, the mean becomes increasingly narrowly distributed around the constant μ. Thus, for large N we can in many situations treat the mean, x‾, as a constant.

The considerations above are basically the rationale behind the law of large numbers and are also closely related to the central limit theorem which states that (x‾-μ)/N/σ converges towards a standard Gaussian distribution, N(0,1). The concentration of measures can be extended beyond the iid situation (see Sect. ) as indicated by the quotation from in the introduction.

Let us organize the random variables into an N vector x=(x1,x2…xN). Now ||x||2=x12+x22…xN2 is a sum of independent variables and will therefore – according to the arguments above – for large N be approximately a constant.

Let us consider a multi-variate standard Gaussian distribution P(x)=(2π)-N/2exp⁡(-∑n=1Nxn2/2). The surface area of a hyper-sphere with radius r in N dimensions is SN-1=2πN/2rN-1/ΓN2. So, as a function of r=||x||, we get the χ distribution 3P(r)=SN-1P(x)=21-N/2ΓN2rN-1exp⁡(-r2/2). The maximum of P(r) is reached for r=N-1 and the width (standard deviation) of the peak converges quickly with N towards 1/2. This is illustrated in Fig. .

The concentration of measures is the backbone of statistical mechanics. As a simple example, we consider the canonical ensemble of weakly interacting identical particles. This ensemble describes a system with a constant number of particles, N, in a heat bath. All particles have the same spectrum of energy states, Ei, and the probability of a particle being in the ith state is proportional to exp⁡(-βEi). The total energy grows like N, while the fluctuations (standard deviation) in the total energy grow like N. Thus, the relative fluctuations in the total energy go as ∼1/N, and in the thermodynamic limit, N→∞, these fluctuations and fluctuations in other macroscopic quantities can be neglected. This holds also for non-identical and interacting particles seefor a recent discussion, just as the concentration of measures can be extended beyond the iid situation.

Let us take a brief look at waist concentration. Consider two independent unit vectors a and b. Without lack of generality we can set a=(1,0,0…). The dot product then becomes b1. It is therefore easy to see that a⋅b has zero mean and that its spread converges to zero as ∼1/N. This result does not require Gaussianity; see e.g. for a general derivation. The angle ϕ between a and b will therefore converge towards π/2 as cos⁡ϕ=a⋅b. This is illustrated in Fig.  for Gaussian-distributed vectors for different values of N: for N=2 the distribution of angles is flat, but for larger values of N it becomes increasingly peaked around π/2.

The topic of concentration properties is an active mathematical field with focus on probabilistic bounds on how quickly empirical means converge to the ensemble means for different classes of random variables, including non-iid variables . Such bounds include Bernstein's and Hoeffding's inequalities and give strict mathematical meaning to the looser considerations above. As an example, the Hoeffding bound states that for all t≥0, 4P1N∑(xi-μi)≥t≤exp⁡(-Nt2/(2σ2)). This holds for independent random variables drawn from a distribution which tails decay at least as quickly as the tails of a Gaussian distribution sub-Gaussian,. Here μi is the mean of xi and σ is a constant. For the angle ϕ between two independent vectors, we have similarly for all t≥0 from 5Pcos⁡ϕ≥t≤2exp⁡(-Nt2/2). Such relations are also central to the related field of large-deviation theory, which specifically studies the exponential decay of probabilities of large fluctuations. See  for a general review and for an application to weather extremes.

In this subsection we directly investigate the distributions of the lengths of the sample members and the distributions of the angles between them. The results from this and the following subsections are summarized in Table .

We centre the sample, xk, k=1,…K, to the sample mean, x‾=∑kxk/K, and calculate the lengths as the square root of ||xk-x‾||2/N for each sample member. The angle ϕ between two sample members, k and l, is given by (xk-x‾)⋅(xl-x‾)=||xk-x‾||||xl-x‾||cos⁡ϕ. This gives us K lengths and K(K-1)/2 angles. This centering – the subtraction of the sample mean – is not important for the calculation of the lengths, as we explain at the end of this subsection.

We first consider the near-surface temperature and precipitation fields from the AgERA data set. Figure  shows the lengths and angles for daily means taken every fifth day for June in the period 1980–1990. The 11 years give us 66 samples. We see that for temperature the lengths are relatively tightly distributed around 4.36 K (σ in Eq. ) with a standard deviation of 0.58 K. The angles are likewise distributed around π/2 with a standard deviation of 0.21. For precipitation the distributions are somewhat narrower, in particular for the angles. This is what we would expect due to the larger number of effective degrees of freedom compared to temperature. However, this effect is reduced as we include both dry and wet days in the analysis. While the precipitation amount on wet days has a short decorrelation length, this does not hold for the spatial field indicating wet/dry days. Note also that the distribution of precipitation is extremely non-Gaussian. These results indicate that the concentration of measures and the waist concentration hold at least to some extent for these fields.

Figure  shows the lengths and angles for the monthly seasonal cycle in near-surface temperature, 1980–2015, for the multi-model CMIP5 ensemble. The models have been regridded to a common 144×73 grid, so N=144×73×12. The sample has a size of 45 and consists of one ensemble member from each of the models. The lengths are distributed around 2.57 K with a standard deviation of 0.46 K and the angles around π/2 with a standard deviation of 0.28. Thus, compared to the example in Fig. , the distributions are less tightly distributed. The main explanation is probably that the effective degrees of freedom in the monthly climatology is smaller than that of the daily fields. However, there are also reasons to believe that the multi-model ensemble is not totally independent . Note the negative skewness in the distribution of the angles. Angles close to zero indicate pairs of models that are almost parallel and therefore strongly dependent. These pairs correspond to variants of the same model, such as MIROC-ESM and MIROC-ESM-CHEM, which are well known to be close in the model genealogy . A simple comparison between the distributions of ϕ in Figs.  and  with the distributions in Fig.  (from Gaussians) shows that the effective dimension is between 25 and 50 for temperature and several hundreds for precipitation.

Results for the MPI-GE 100-member initial condition ensemble are shown in Table . Here we have 192×96 grid points, so N=192×96×12 and the sample size is 100. The distributions of lengths and angles are now narrower compared to the multi-model CMIP5 ensemble. This corresponds to a larger effective dimension in the monthly climatology which now reflects only different initial conditions and not model differences. Also in this example are the distributions for precipitation narrower than those for temperature.

Reducing the spatial area decreases the effective dimension. As an example we have included in Table  the results for the AgERA when applied to northern Europe (50–65∘ N, 0–25∘ E). As expected, we see an increase in the width of the distributions for both precipitation and near-surface temperature.

In the analysis above we centred the sample to the sample mean before calculating the lengths; i.e. we used ||xk-x‾||2/N instead of ||xk||2/N. However, these expressions only differ by the length of the mean, ||xk||2/N=x‾2/N+||xk-x‾||2/N, as xk-x‾ and x‾ are orthogonal in high dimensions (due to waist concentration). The absence of centering makes most sense for precipitation that has a natural zero point. For AgERA precipitation ||x‾||/N is 3.35, the mean of ||xk-x‾||/N is 4.34, and the mean of ||xk||N is 5.48 (all mmd-1), fulfilling the Pythagorean relationship (5.482=3.352+4.342).

If the sample members are drawn independently from the same distribution in high dimensions, they have approximately the same length, and we can write 6||xk-x‾||2/N=σ2. For the distance between two different sample members, we get 7||xk-xl||2/N=||(xk-x‾)-(xl-x‾)||2/N=2σ2, where we have used the fact that xk-x‾ and xl-x‾ are orthogonal.

Therefore, the distance between two sample members is a square root of 2 larger than the distance between a sample member and the sample mean. The geometric interpretation is that the sample mean and any two sample members form an isosceles right triangle with the right angle at the sample mean . The factor of 2 then comes from Pythagoras' equation. It is worth noting that the sample mean is special and is not drawn from the same distribution as the sample members. As mentioned when discussing the example of the high-dimensional unit cube from Sect. a, the sample members would be located in the spikes, while the sample mean would be close to the centre.

Figure  demonstrates this in the simple situation where 50 N-dimensional vectors are drawn from prescribed distributions. The results are shown as a function of N. When N increases, the spread of the distances decreases, and for large N the factor of 2 is clearly seen. This holds both for simple spherical Gaussian-distributed vectors (left panel) and for Gamma-distributed vectors with strong anisotropy in the components (right panel), although the convergence is faster in the Gaussian case.

Figure  shows the distances for AgERA daily mean precipitation for June (left panel) and for the monthly climatology of near-surface temperature for the CMIP5 ensemble (right panel). The distribution of the distances between sample members is shown together with the distribution of the distances between the sample members and the sample mean. The mean and width of these distributions are also shown in Table  for both these and the other data sets. In all cases the factor of 2 is clearly seen for the mean values, although the widths of the distributions are substantial in all cases. For the AgERA daily precipitation (Fig.  left), the two distributions are almost separated, while this is not the case for the CMIP5 ensemble.

The indistinguishable interpretation claims that observations are drawn from the same distribution as the ensemble members. With this assumption and the considerations above, explained the ubiquitous observation that the error (compared to observations) of the ensemble mean often is 30 % smaller ((1-2)/2) than the typical error of the individual ensemble members e.g.. We also explained why the ensemble mean very often has a smaller error than all individual ensemble members .

The results in this subsection and Sects. .2 and 4 not only hold for the Euclidean (square) norm distance, but also for e.g. the maximum norm distance and the correlation distance (1-r2, where r is correlation).

Error correlations and correlations between model differences are important when studying the structure of a model ensemble and when comparing an ensemble to observations .

We have in general corr(xk,xl)=1-12||x^k-x^l||2/N, where ^ indicates variables standardized to zero mean and unit variance. Therefore, with ϵk=xk-xm we have ϵ^k=(xk-xm)/2/σ. We now get corr(xk-xm,xl-xm)=corr(ϵk,ϵl)=1-12ϵ^k-ϵ^l2/N=1-12xk-xl2σ2/N8=1-12=12, where in the last step we have used Eq. (). Thus, in high dimensions the correlation between sample differences is 1/2.

Replacing xl with the sample mean, we get 9corr(xk-xm,x‾-xm)=1-12xk-xm2σ-x‾-xmσ2/N10=1-12xk-x‾2σ+(1-1/2)xm-x‾σ2/N11=1-12(1/2+(1-1/2)2)=1/2. In the last step we used the independence of the two terms and applied Eq. () to each.

Figure  shows the correlations for the simple random vectors as also used in Fig. . For all N the correlations are distributed around 1/2 and 1/2. For small N the spread is large, but it decreases when N increases, and for large N the correlations are very narrowly distributed around 1/2 and 1/2≈0.71.

The correlations for AgERA daily mean precipitation for June and for the CMIP5 monthly climatology of near-surface temperate are shown in Fig. . The mean values are close to the high-dimensional values from Eqs. () and (), although the spread is rather high. This is also the case for the other fields as reported in Table .

If we again assume that the observations are drawn from the same distribution as the ensemble members – the indistinguishable interpretation – the error correlation is 1/2 (Eq. ). On the other hand, if observations are near the ensemble mean – the truth-centred interpretation – the error correlations will be zero as xk-x‾ and xl-x‾ are orthogonal. Error correlations around 1/2 have been observed in many studies of climate models e.g., providing evidence for the indistinguishable interpretation.

With xm replaced by observations, Eq. () gives the correlation between individual model errors and the model mean error. This quantity is shown in Fig. 2 of  for the climatology of different variables in the CMIP3 multi-model ensemble, and it is always close to 1/2≈0.71, as predicted by Eq. ().

We now consider how the sample mean depends on the sample size. The ensemble mean is often used to estimate the forced response from initial condition and multi-model ensembles , and it is of interest to know how large an ensemble is needed for the estimation to be saturated .

Letting x∞‾ represent the true (i.e. the distribution) mean of the sample, we get in the high-dimensional case (reformulating Eq. ) 12||x‾||2/N=μ2+σ2/K, where μ2=||x∞‾||2/N. Thus, ||x‾||2 converges like 1/K, and the convergence is slowest where the sample spread is largest. Similar results have been presented by and based on other arguments. See also for the decay of the error of the ensemble mean when compared to observations.

The practical way to estimate the effect of sample size is to apply a bootstrap procedure to a large sample of size K0. From this sample we draw (with replacement) a number of sub-samples of size K, K=1,…K0. From these sub-samples we calculate the mean and spread of ||x‾||2/N for each K.

The mean is shown as a function of K – using the bootstrap procedure – in Fig.  for the simple examples with N=100 and N=10. For N=100 (black curves) ||x‾||2/N is narrowly distributed around the theoretical mean (Eq. ) for both the Gaussian- (left) and Gamma-distributed samples (right). For N=10 (cyan curves) ||x‾||2/N is also distributed around the theoretical mean, but with larger spread.

In the three previous subsections we studied the samples of daily June temperatures and of monthly climatologies. In the former the N vectors consisted of spatial maps and in the latter of combined spatial climatologies for all 12 months. However, we also work in high dimensionality when considering a single long time series. The left panel in Fig.  shows time series of the annual NH mean near-surface temperature for the MPI-GE 100 member initial condition ensemble and the CMIP5 45-member multi-model ensemble for the period 1961–2005. Both ensembles have been centred to their ensemble means in the first 10 years. Both ensemble means agree on a forced response consisting of an overall trend with some signals of volcanic eruptions after 1982 (El Chrichón) and 1991 (Mount Pinatubo). The spread of the multi-model ensemble is much larger than the spread of the initial condition ensemble.

The right panel shows ||x‾||2/N as a function of K. As expected from Eq. (), the initial condition ensemble converges more quickly than the multi-model ensemble due to its smaller variance. Note the excellent agreement with Eq. () (red curves), where σ2 has been estimated as the variance over time and all ensemble members and μ2 likewise estimated from the ensemble mean over all ensemble members. The large spread for the CMIP5 ensemble is due to the well-known fact that the bias in global mean temperature is different for different models , which led to a breakdown of the condition of independence. This is not the case for the initial condition ensemble (see also Table ). Smaller spread is obtained for the CMIP5 ensemble if each model is centred to its own (and not the ensemble) mean in the first 10 years.