Regression to the Mean

Galton noticed this with peas. It applies to almost everything that varies.

The phrase comes from Francis Galton, a Victorian polymath who in 1886 was measuring the heights of parents and their grown children. He noticed something odd. The tallest fathers tended to have tall sons — but their sons were, on average, less tall than they were. The shortest fathers had short sons whose sons were, on average, taller than they were. Both ends of the distribution drifted toward the middle in the next generation. Galton called this "regression to mediocrity" — later softened to "regression to the mean."

What was actually happening was not biological. It was statistical. Any measured trait that combines a stable underlying value (genetic potential) with a noisy short-term reading (one measurement, one game, one season) will look like this. The reading sits on top of the value. The reading varies. The value does not. When you observe an extreme reading, you are observing both a high value and a positively-biased noise term. The next reading does not have that same positive noise term. It regresses.

The applications are everywhere. The player who has a career year hits worse the following year — we call this the Sports Illustrated Cover Curse. It is not a curse. The career year was partly true talent and partly favorable noise. The next year is the noise leaving. The team that wins ninety games regresses toward eighty. The team that loses ninety regresses toward seventy-five. The rookie of the year almost never repeats. The breakout pitcher coughs up some of the breakout. None of these is a punishment. They are the gravity of the average doing its job.

The interesting trap, and the one most analysts walk into, is mistaking regression for causation. The player who slumps after his career year did not get worse because his swing broke down. He had a career year because of favorable noise, and the favorable noise is gone. The pitcher who got better after spring training did not get better because of the new pitching coach. He had been unlucky, and the bad luck has run its course. These are statistical drifts wearing the costume of stories. The job of the careful reader is to peel the costume off.

None of this means coaching does not matter, or that adjustments are fake, or that talent is just luck. It means: when something extreme happens, your default prior should be that some of it was the dice. The follow-up observation will tell you how much. If a player who had a career year keeps performing at that elevated level the next year, the career year was probably mostly real. If he reverts halfway back to his prior baseline, it was probably mostly noise. The pattern is your friend. Your friend is patient.

One last thing. The reason this concept matters so much for The Sports Page — and the reason we use the Bayesian framework everywhere — is that the right way to estimate a player’s true talent is to regress their observed performance toward a prior. The prior is the league mean, weighted by how much information you have. Three at-bats? Regress almost all the way to the league mean. Three thousand? Regress barely at all. The math is doing what your gut would do if your gut were honest with itself.