Abraham Wald Looked at the Planes That Came Back. Sports Media Almost Never Does.
Wald’s insight is one of the cleanest pieces of statistical reasoning produced in the twentieth century, and it has a short technical name: survivorship bias. The general form is the mistake of drawing a conclusion from a sample of survivors when the data you actually need would include the non-survivors that the sample has already filtered out. The returning bombers were the data the Air Forces could see. The missing planes were the data the military needed. Wald’s contribution was to notice that the absence of a hole on a returning bomber was, itself, the most important observation in the dataset — precisely because the planes that did get hit there were not in the photograph.
This is the shape, in miniature, of about a third of the inference errors that pass through sports writing on any given week. The general move is: identify a person, team, or strategy that succeeded; identify their distinctive features; recommend that other people emulate those features. The move ignores everything that happened to the people, teams, and strategies who shared the same features and failed, because those people are not on the panel show, are not on the magazine cover, and frequently are not on a roster anymore. The survivors get written about. The non-survivors do not. The reader learns from the survivors. The reader is, as a consequence, learning from a sample that has been pre-filtered for the very thing the conclusion is supposed to predict.
The Wald Diagram, in Sports Terms
Three Sports Survivorship Errors, Currently in Circulation
The pattern below is the same shape every time. A successful person, team, or method gets identified. The features of the success get catalogued. The features get presented as a blueprint. The blueprint is recommended, in print or on the radio, as a lesson for someone else to follow. In every case, the features of the thousands of equivalent people, teams, and methods that shared the same features and failed are absent from the analysis, because those equivalents are not the subject of any column. Three current examples follow.
| Shape of the Error | How It Reads in a Headline | What’s Missing from the Picture |
|---|---|---|
| The Brady Argument | “Tom Brady was the 199th pick in 2000. Late picks are undervalued.” | Brady is one of roughly twenty-five players selected at that slot since the AFL-NFL merger. The other twenty-four had careers ranging from short to forgettable. The sample of all 199th picks does not support “late picks are undervalued.” The sample of one 199th pick who became Tom Brady does. The column is using a sample size of one and presenting it as a population finding. |
| The Late Bloomer Pattern | “Player X didn’t break out until age 25 or 26. Don’t write off late developers.” | For every star whose age-25 season was the breakout, there are dozens of age-25 players who were sent to Triple-A, traded to a non-contender, or released. Their stories are not the ones we tell because they did not break out. The breakout cases are visible by construction. The non-breakout cases drop quietly off the back of every roster every spring and are forgotten by July. A base rate of “how often does a player still working at 25 become a star” cannot be computed from highlight reels. |
| The Successful Tank Blueprint | “The Astros tanked from 2011 to 2013 and won the 2017 World Series. Tanking works.” | The 2011 Houston Astros are visible because they later won a championship. The 2011 Cleveland Browns, who have tanked through at least three rebuilds in the same window without producing a comparable franchise quarterback, are visible to Cleveland fans and almost nobody else. The Pyrrhic March, which ran in this newsletter for six weeks, made the same point with five additional case studies of teams whose tanks did not pay off in any way the column would write about. Successful tanks survive into the next news cycle. Unsuccessful tanks become “rebuilding under new leadership.” |
In each of these cases the column being criticized has not made an error of fact. Tom Brady was the 199th pick in 2000. The cited late bloomer did break out at 25. The Astros did tank and the Astros did win. The error is in the population from which the conclusion is drawn. The columnist has implicitly chosen a sample — successful 199th picks, late bloomers who broke out, tanks that ended in titles — that has been pre-filtered for the very feature the conclusion is supposed to predict. The reasoning is, formally, a tautology: the players who succeeded had the features the writer is now recommending, because the writer chose to examine players who succeeded. The non-success players, who likely had similar features, are missing from the photograph. They are the planes that did not come home.
The Vocabulary the Reader Takes Home
Three terms. They are slightly more technical than the temporal-contiguity vocabulary from Part I and they are worth the slightly higher cost.
1. Survivorship bias. The general name for any inference drawn from a sample that has been pre-filtered for success. The phrase carries the warning in its own etymology: the sample contains only the survivors. The non-survivors are missing. Any conclusion that does not account for what is missing is incomplete by construction.
2. Selection mechanism. The diagnostic question the reader should ask of any sports column that draws a lesson from a small set of successes: What decided which cases I am being shown? If the answer is “the writer chose them because they succeeded,” the conclusion is doing less work than it appears to be doing. The selection mechanism is what produces the bias.
3. The Wald check. An informal habit the reader can run on any blueprint argument. Identify the cases the column presents. Then identify, by name if possible, the equivalent cases that share the same features but failed. If you cannot name any, ask whether the column has actually looked for them. If the column has not looked for them, the reader should not be persuaded by what the column found.
“The absence of bullet holes on a returning bomber was the most important observation in the dataset. The absence of failed late bloomers, of failed tanks, of forgotten late picks is the most important observation in roughly a third of sports columns. The data is missing on purpose. The conclusion does not survive its absence.”
— The Sports Page, on Wald and what gets photographedThe Series, Continued
This is Part II of the newsletter’s recurring thread The Reader’s Defense. Part I (David Hume, temporal contiguity, the Issue #70 errata) introduced the project. The next entries already named: regression to the meani, the Galton finding that explains why hot streaks are usually borrowed from cold streaks waiting to happen; base-rate neglect, the Kahneman–Tversky finding that explains why one upset feels more important than the historical record of upsets ever does; and cherry-picked outliers, which Issue #51 of this newsletter (“Three Lies, Thirty Dots”) treated graphically and which the series will revisit verbally. Each entry will give the reader a name for an error, an illustration from current sports coverage, and a small vocabulary to carry to the next column they read.
The premise, as in Part I, is not that sports media is uniquely bad. The premise is that sports media uses the same inferential patterns everyone uses, and that those patterns have failure modes named centuries ago by people who could not have anticipated a single televised sport. Hume could not have anticipated a draft lottery. Wald could not have anticipated a 199th pick becoming a seven-time Super Bowl winner. The reasoning errors are older than the games. The reasoning errors persist because the games make them feel new. The defensive reader learns the errors by name and stops feeling that any of them are new.
A note on the data: the 1943 bomber-armor analysis is conventionally attributed to Abraham Wald and the Statistical Research Group at Columbia University; the canonical narrative source is Marc Mangel and Francisco Samaniego’s 1984 article “Abraham Wald’s Work on Aircraft Survivability” in the Journal of the American Statistical Association. Tom Brady’s draft slot (199th overall, sixth round, 2000) is documented across every NFL reference. The 2011–2013 Houston Astros’ teardown and 2017 championship were the subject of Part IV of the Pyrrhic Victory March in this newsletter. The general principle — that a sample pre-filtered for success cannot be used to predict success — is older than statistics as a discipline and recurs in every domain that produces survivors and non-survivors. The reader who has finished this issue should be able to detect it in roughly half the sports columns they read this week.
