Skip to main content

Research Repository

Advanced Search

Beyond Strictly Proper Scoring Rules: The Importance of Being Local

Du, Hailiang

Beyond Strictly Proper Scoring Rules: The Importance of Being Local Thumbnail



The evaluation of probabilistic forecasts plays a central role both in the interpretation and in the use of forecast systems and their development. Probabilistic scores (scoring rules) provide statistical measures to assess the quality of probabilistic forecasts. Often, many probabilistic forecast systems are available while evaluations of their performance are not standardized, with different scoring rules being used to measure different aspects of forecast performance. Even when the discussion is restricted to strictly proper scoring rules, there remains considerable variability between them; indeed strictly proper scoring rules need not rank competing forecast systems in the same order when none of these systems are perfect. The locality property is explored to further distinguish scoring rules. The nonlocal strictly proper scoring rules considered are shown to have a property that can produce “unfortunate” evaluations, particularly the fact that the continuous rank probability score prefers the outcome close to the median of the forecast distribution regardless of the probability mass assigned to the value at/near the median raises concern to its use. The only local strictly proper scoring rule, the logarithmic score, has direct interpretations in terms of probabilities and bits of information. The nonlocal strictly proper scoring rules, on the other hand, lack meaningful direct interpretation for decision support. The logarithmic score is also shown to be invariant under smooth transformation of the forecast variable, while the nonlocal strictly proper scoring rules considered may, however, change their preferences due to the transformation. It is therefore suggested that the logarithmic score always be included in the evaluation of probabilistic forecasts.


Du, H. (2021). Beyond Strictly Proper Scoring Rules: The Importance of Being Local. Weather and Forecasting, 36(2), 457-468.

Journal Article Type Article
Acceptance Date Dec 9, 2020
Online Publication Date Mar 8, 2021
Publication Date 2021-04
Deposit Date Oct 16, 2021
Publicly Available Date Oct 18, 2021
Journal Weather and Forecasting
Print ISSN 0882-8156
Electronic ISSN 1520-0434
Publisher American Meteorological Society
Peer Reviewed Peer Reviewed
Volume 36
Issue 2
Pages 457-468
Related Public URLs


Accepted Journal Article (357 Kb)

Copyright Statement
© Copyright 2021 American Meteorological Society (AMS). For permission to reuse any portion of this work, please contact Any use of material in this work that is determined to be “fair use” under Section 107 of the U.S. Copyright Act (17 U.S. Code §?107) or that satisfies the conditions specified in Section 108 of the U.S. Copyright Act (17 USC § 108) does not require the AMS’s permission. Republication, systematic reproduction, posting in electronic form, such as on a website or in a searchable database, or other uses of this material, except as exempted by the above statement, requires written permission or a license from the AMS. All AMS journals and monograph publications are registered with the Copyright Clearance Center ( Additional details are provided in the AMS Copyright Policy statement, available on the AMS website (

You might also like

Downloadable Citations