Skip to main content

Research Repository

Advanced Search

Towards improving the framework for probabilistic forecast evaluation

Smith, Leonard A.; Suckling, Emma B.; Thompson, Erica L.; Maynard, Trevor; Du, Hailiang

Towards improving the framework for probabilistic forecast evaluation Thumbnail


Leonard A. Smith

Emma B. Suckling

Erica L. Thompson

Trevor Maynard


The evaluation of forecast performance plays a central role both in the interpretation and use of forecast systems and in their development. Different evaluation measures (scores) are available, often quantifying different characteristics of forecast performance. The properties of several proper scores for probabilistic forecast evaluation are contrasted and then used to interpret decadal probability hindcasts of global mean temperature. The Continuous Ranked Probability Score (CRPS), Proper Linear (PL) score, and IJ Good’s logarithmic score (also referred to as Ignorance) are compared; although information from all three may be useful, the logarithmic score has an immediate interpretation and is not insensitive to forecast busts. Neither CRPS nor PL is local; this is shown to produce counter intuitive evaluations by CRPS. Benchmark forecasts from empirical models like Dynamic Climatology place the scores in context. Comparing scores for forecast systems based on physical models (in this case HadCM3, from the CMIP5 decadal archive) against such benchmarks is more informative than internal comparison systems based on similar physical simulation models with each other. It is shown that a forecast system based on HadCM3 out performs Dynamic Climatology in decadal global mean temperature hindcasts; Dynamic Climatology previously outperformed a forecast system based upon HadGEM2 and reasons for these results are suggested. Forecasts of aggregate data (5-year means of global mean temperature) are, of course, narrower than forecasts of annual averages due to the suppression of variance; while the average “distance” between the forecasts and a target may be expected to decrease, little if any discernible improvement in probabilistic skill is achieved.


Smith, L. A., Suckling, E. B., Thompson, E. L., Maynard, T., & Du, H. (2015). Towards improving the framework for probabilistic forecast evaluation. Climatic Change, 132(1), 31-45.

Journal Article Type Article
Acceptance Date May 17, 2015
Online Publication Date Jul 17, 2015
Publication Date Sep 1, 2015
Deposit Date Jul 31, 2018
Publicly Available Date Aug 21, 2018
Journal Climatic Change
Print ISSN 0165-0009
Electronic ISSN 1573-1480
Publisher Springer
Peer Reviewed Peer Reviewed
Volume 132
Issue 1
Pages 31-45
Related Public URLs


Published Journal Article (368 Kb)

Publisher Licence URL

Copyright Statement
© The Author(s) 2015 Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (, which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

You might also like

Downloadable Citations