Our New Rule Beat Our Old One. Random Numbers Beat Them Both.
Does the order you put your picks in matter? We run an entry in a family pick'em pool. You choose fifteen games and hand out fifteen point values, 15 down to 1. We wanted to know whether a smarter ordering would win more.
So we tested one. Over three weeks it scored 213 where we had scored 202. It would have moved us up in two of the three weeks. That looks like a win.
Then we asked the question we should have asked first. What does the order score if you pull it out of a hat? The answer is 224. Our clever rule lost to the hat by eleven points. So did our real entry, by twenty-two.
Most people assume a rule that beats what you were doing before is an improvement. It is not. It is only an improvement if it beats doing nothing in particular. Ours did not. By the end of this you will know the one question to ask any winning idea.
Why a hat is the fair test
Here is the key fact. We got 28 of 45 picks right. Those 28 right answers were going to carry whatever point values we happened to stick on them. Hand out the numbers 15 to 1 at random and, on average, each correct pick carries 8 points. Twenty-eight times eight is 224.
So 224 is not a target; it is the middle of a pile. Watch what that pile actually looks like.
The curve is every way the weights could have fallen. It is wide; the spread is about 15 points. That width is the answer. Two orderings of the same picks can differ by thirty points for no reason at all.
Now find our three marks. Every one sits left of center. The rule we were about to adopt sits at 213, below where a hat lands. We did not find a way to score more. We found a way to lose less.
A rule that beats what you were doing is not an improvement. It is only an improvement if it beats doing nothing in particular.
— The Sports PageAnd the reason it “worked” is false
The rule had a story behind it. It said the crowd overrates its favorites, so put your big points on the games where the crowd is against you. Sensible enough; lots of people believe it.
It is not true in our own data. Look at the picks we made where the crowd disagreed with us, and the picks where the crowd agreed:
| our picks | record | hit rate |
|---|---|---|
| crowd was against us | 7 of 12 | 58% |
| crowd agreed with us | 21 of 33 | 64% |
The games the crowd liked were the ones we got right more often. The rule moves points away from those. Its whole idea runs backwards in the very numbers used to build it.
So where did the eleven points come from? Four games. Northwestern alone was worth ten of them. Move one Saturday and the rule vanishes.
This is building the rule after seeing the answeri. We sorted three weeks of noise until a pattern fell out, then wrote a reason for it. The pattern was real; the reason was invented.
What to take home
Ask one question of any winning idea, yours or anyone else's: compared to what? Not compared to what you did before. Compared to doing nothing thoughtful at all — a coin, a hat, the alphabet. Most claims never face that test, and a surprising number fail it.
Then ask the second one. Does the reason hold up on its own? Our rule gained points while its story was false. If the story is wrong, the gain was luck, and luck does not come back next week.
We have sealed this rule and will grade it on week four against 224, not against ourselves. If it loses again, we will say so here. You can read the pool guide for the arithmetic, and the rest of the paper at thesportspage.net.
Argued about something you could not settle? Send it to us · what makes a good question