Our Model Went 15 for 15. That Is How We Knew It Was Broken.

A perfect backtest is not a triumph; it is a bug report. Rebuilt so that cheating was impossible, the same method beat the spread 51.0% of the time across 2,140 games — and the interval around that contains 50.

The Sports Page · The Professor · marking our own homework, again

The question: can our ratings beat a betting line?

No. And for one afternoon we had convincing evidence that we could.

Why that is surprising. Not the answer — the answer is what anyone should expect. What is surprising is how good the wrong evidence looked, and that the thing which exposed it was the result being too good.

What you get: the smell test that catches this in your own work.

The result that was too good

We keep a ledger of picks in a family pool. Wanting to know whether our method was any use, we ran it backwards over a week of games already played.

It went 15 for 15. Every pick, against the spread, in the correct order of confidence.

Now, that should feel wonderful for about four seconds. Fifteen coin flips landing the same way is roughly a one-in-thirty-three-thousand event. Which is the point: when a test hands you a number that good, the model is almost never the explanation.

So we measured the thing directly. Across all 51 games that week, the correlation between our edge and what actually happened against the spread was +0.842.

Nobody predicts closing lines at 0.84. A genuinely good model sits near +0.10. That number is not a discovery; it is a confession.

What we had done

The ratings we fed the test were not the ratings that existed before the games. They were today’s.

The data provider serves one set of ratings and overwrites them as results arrive. We had noticed the drift ourselves two days earlier — one club had moved from 32.7 to 30.0 — and then ran the test with the updated numbers anyway.

So the model was asked to predict games it had already watched. It did brilliantly. Anyone would.

Doing it so cheating was impossible

The fix was to stop borrowing anybody’s rating. We built our own from scratch, and rebuilt it every week of the season using only the games played before that week. A rating for week seven has never seen week seven. It cannot.

Then we let it pick against the closing line for every game of 2024 and 2025 — 2,140 games.

It won 51.0% of them. The 95% interval runs from 48.9% to 53.1%. Break-even against a standard line is 52.4%.

The interval contains 50. We cannot reject the possibility that our model knows nothing whatsoever about football.

Look at what that figure does to the idea underneath the whole method. We rank our confidence by how far our number sits from the line — the bigger the disagreement, the more sure we are.

If that were sound, the bars would climb. They sit flat. A six-point disagreement wins 51.1% of the time; so does a disagreement of more than ten. The one band above break-even, at three to six points, is a lonely bar among four and we are not going to build anything on it.

Why the market is so hard to beat

Worth saying plainly, because it is not a failure of our arithmetic.

A betting line is not a prediction. It is the price at which the money balances. Every injury report, every bit of local knowledge, every professional model already sits inside it. To beat it you do not need to be good; you need to know something several thousand motivated people have missed.

We did not. Almost nobody does.

What to take home: be suspicious of your best result

Everyone knows to check a result that looks wrong. Almost nobody checks the one that looks wonderful, and that is where the errors live, because a mistake that flatters you does not feel like a mistake.

So keep one rule: when a number comes back better than it has any right to be, treat it as a bug until you have proved otherwise.

The specific version, for anything predicting the future: ask what your inputs knew, and when they knew it. A rating that updates is a rating that has read the newspaper. A survey scored after the outcome is a survey that saw it. The past is remarkably easy to predict once it has happened.

We now write the ratings down every Thursday, before kickoff, and keep them. In November we will be able to answer this honestly. Today we cannot, and saying so costs less than the alternative.