The Brier Score is a strictly proper score function or strictly proper scoring rule that measures the accuracy of probabilistic predictions. For unidimensional predictions, it is strictly equivalent to the mean squared error as applied to predicted probabilities.
The Brier score is applicable to tasks in which predictions must assign probabilities to a set of mutually exclusive discrete outcomes or classes. The set of possible outcomes can be either binary or categorical in nature, and the probabilities assigned to this set of outcomes must sum to one (where each individual probability is in the range of 0 to 1). It was proposed by Glenn W. Brier in 1950.
The Brier score can be thought of as a cost function. More precisely, across all items in a set of N predictions, the Brier score measures the mean squared difference between: The predicted probability assigned to the possible outcomes for item i The actual outcome
Therefore, the lower the Brier score is for a set of predictions, the better the predictions are calibrated. Note that the Brier score, in its most common formulation, takes on a value between zero and one, since this is the square of the largest possible difference between a predicted probability (which must be between zero and one) and the actual outcome (which can take on values of only 0 or 1). In the original (1950) formulation of the Brier score, the range is double, from zero to two.
The Brier score is appropriate for binary and categorical outcomes that can be structured as true or false, but it is inappropriate for ordinal variables which can take on three or more values.
Definition
The most common formulation of the Brier score is
in which is the probability that was forecast, the actual outcome of the event at instance ( if it does not happen and if it does happen) and is the number of forecasting instances. In effect, it is the mean squared error of the forecast. This formulation is mostly used for binary events (for example "rain" or "no rain"). The above equation is a proper scoring rule only for binary events; if a multi-category forecast is to be evaluated, then the original definition given by Brier below should be used.
Example
Suppose that one is forecasting the probability that it will rain on a given day. Then the Brier score is calculated as follows: If the forecast is 100% ( = 1) and it rains, then the Brier Score is 0, the best score achievable. If the forecast is 100% and it does not rain, then the Brier Score is 1, the worst score achievable. If the forecast is 70% ( = 0.70) and it rains, then the Brier Score is (0.70−1)2 = 0.09.
In contrast, if the forecast is 70% ( = 0.70) and it does not rain, then the Brier Score is (0.70−0)2 = 0.49. Similarly, if the forecast is 30% ( = 0.30) and it rains, then the Brier Score is (0.30−1)2 = 0.49. If the forecast is 50% ( = 0.50), then the Brier score is (0.50−1)2 = (0.50−0)2 = 0.25, regardless of whether it rains.
Original definition by Brier
Although the above formulation is the most widely used, the original definition by Brier is applicable to multi-category forecasts as well as it remains a proper scoring rule, while the binary form (as used in the examples above) is only proper for binary events. For binary forecasts, the original formulation of Brier's "probability score" has twice the value of the score currently known as the Brier score.
In which is the number of possible classes in which the event can fall, and the overall number of instances of all classes. is the predicted probability for class is if it is class in instant ; , otherwise. For the case Rain / No rain, , while for the forecast Cold / Normal / Warm, .
Decompositions
There are several decompositions of the Brier score which provide a deeper insight on the behavior of a binary classifier.
3-component decomposition
The Brier score can be decomposed into 3 additive components: Uncertainty, Reliability, and Resolution. (Murphy 1973)
Each of these components can be decomposed further according to the number of possible classes in which the event can fall. Abusing the equality sign: