Brier Score and Calibration, explained.
The binary Brier score is the mean squared difference between forecast probabilities and zero-or-one outcomes; smaller values are better.
Why it happens
Ten independent outcomes use known probabilities 0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,0.5. Commit forecasts before revealing outcomes. Compare realised scores, expected score p(1−p)+(q−p)², and observed rates within three forecast bins.
The binary Brier score is the mean squared difference between forecast probabilities and zero-or-one outcomes; smaller values are better.
Read the result
Commit before outcomes appear, then compare realised and expected scores. A single lucky batch can reward an overconfident forecast even though it has worse expected loss.
A worked example
Scoring an honest probability
Use the generating probabilities as forecasts, recording that an event with probability eighty percent can still fail to occur.
Commit and reveal all ten outcomes. Its contribution is 0.04 if it occurs or 0.64 if it does not.
Inspect the mean score and sparse calibration bins. They describe this small batch, not a definitive judgment of forecasting ability.
OPTIONAL DEEPER DETAILGo deeper: inside the model
Inside this model
Ten independent outcomes use known probabilities 0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,0.5. Commit forecasts before revealing outcomes. Compare realised scores, expected score p(1−p)+(q−p)², and observed rates within three forecast bins.
Where this idea is useful
A practical use
A team can keep a probability journal and score forecasts rather than retrospectively remembering only confident successes.
A common misconception
“An eighty-percent forecast is wrong whenever the event does not happen.”
A probability forecast allows both outcomes. Evaluate many comparable forecasts with a proper score and calibration checks instead of treating each one as a deterministic prediction.
What this explanation leaves out
- Ten events cannot establish calibration. A good probability can lose on one outcome. Scores depend on the event mix, and calibrated forecasts need not be informative; no actual weather or market forecast is presented.
Can a calibrated forecaster still be unhelpful?
Yes. Constant base-rate forecasts can be calibrated while distinguishing no cases. Calibration and useful discrimination are separate properties.
Which clearly resolvable event could you forecast before knowing the outcome and score later?
Associated thinkers
Associations marked provisional are awaiting source review.
Further reading
Explore the original research or the teaching reference behind this experiment.