Occam's Razor, explained.
Occam's razor is a family of principles favouring economy of explanation when other relevant considerations are equal. Simplicity is a guide to inquiry rather than a guarantee that a short account is true.
Why it happens
Several rules can fit the same finite observations exactly. A more elaborate rule may add unsupported flexibility, yet real structure can also be complex. A prediction on new data supplies information that training fit alone lacks.
Occam's razor favours avoiding unnecessary explanatory complexity when other things are appropriately equal. It is a guide to inquiry, not a theorem that the shortest story must be true.
Read the result
Compare the rules on the three training points, commit to a prediction at x=4, and reveal its error. Switch teaching worlds to see a case where the shorter rule succeeds and a case where it misses real structure.
A worked example
Two exact fits and one new observation
The observations at x=1, 2, and 3 are two, four, and six. Both displayed rules match all three.
At x=4, the short rule predicts eight while the elaborate rule predicts fourteen.
Reveal the teaching world. An exact training fit did not distinguish the candidates, and the shorter rule is not guaranteed to match the new observation.
OPTIONAL DEEPER DETAILGo deeper: inside the model
Inside this model
Training observations are (1,2), (2,4), and (3,6). Candidate rules are 2x and 2x+(x−1)(x−2)(x−3); both fit exactly. At x=4 they predict eight and fourteen. Straight and Curved worlds supply either outcome. Commit to a rule before revealing held-out error; switch worlds to inspect when simplicity helps or misses structure.
Where this idea is useful
A practical use
A perfect fit on a small sample can leave very different predictions for unobserved cases.
A common misconception
“The simplest explanation must be true.”
Simplicity can be a reason to prefer a candidate provisionally, but evidence and explanatory adequacy remain necessary. Choosing an easy story does not eliminate alternatives.
What this explanation leaves out
- The worlds and rules are authored, not sampled evidence about real learning algorithms. Description length, parameter count, and truth are distinct; a complex world is possible.
Does this prove that complex models always overfit?
No. The Curved world makes the extra structure useful. The toy comparison shows why limited observations and a simplicity preference cannot substitute for testing.
Which extra assumption in an explanation makes a testable prediction, and which merely decorates the story?
Associated thinkers
Associations marked provisional are awaiting source review.
Further reading
Explore the original research or the teaching reference behind this experiment.