Relative impact of model quality and ensemble deficiencies on the performance of ensemble based probabilistic forecasts evaluated through the Brier score

Abstract. The relative impact of model quality and ensemble deficiencies, on the performance of ensemble based probabilistic forecasts, is investigated from a set of idealized experiments. Data are generated according to a statistical model, the validation of which is achieved by comparing generated data to ECMWF ensemble forecasts and analyses. The performance of probabilistic forecasts is evaluated through the reliability and resolution terms of the Brier score. Results are as follows. (i) Resolution appears essentially attributable to the average level of forecast skill. (ii) The lack of reliability comes primarily from forecast bias, and to a lower extent from the ensemble being systematically under-dispersive (or over-dispersive). (iii) Forecast skill contributes very little to reliability in the absence of forecast bias, and this impact is entirely due to the finiteness of the ensemble population. (iv) In the presence of forecast bias, reducing forecast skill leads to improve the reliability. This unexpected feature comes from the fact that lower forecast skill leads to a larger ensemble spread, that compensates for the strong proportion of outliers consequent to forecast bias. (v) The lack of ensemble skill, i.e. non systematic errors affecting both ensemble mean and ensemble spread, contributes little, but significantly, to the lack of reliability and resolution.