Don’t Get Fooled By Variance

In the excellent book, Thinking, Fast and Slow by Daniel Kahneman (a book all poker players should read), the author presents the following question.

A study of the incidence of kidney cancer in the 3,141 counties of the United States reveals a remarkable pattern. The counties in which the incidence of kidney cancer is lowest are mostly rural, sparsely populated, and located in traditionally Republican states in the Midwest, the South, and the West. Now, what do you make of this information?

What might it be about these counties that causes the low rates of kidney cancer? Take a moment to think about this before you keep reading the article.

The author then tells you that counties with the highest incidence of kidney cancer meet the exact same criteria—they are mostly rural, sparsely populated, and located in traditionally Republican states in the Midwest, the South, and the West.

What explains that these same types of counties feature among those with the lowest rates of kidney cancer and also among those with the highest rates?

Again, try to come up with an answer before you continue reading.

Most people answer this question by building narratives around some of the features of these counties. They’re rural, so the air is clean—or maybe it’s not clean. They’re Republican so something about health care maybe? Or they’re in the Midwest, the South, or the West, and there’s something in the soil?

In fact, the key information about these counties is that they are “sparsely populated.” Say a county has just 100 residents. If no one has kidney cancer in the county, the rate is zero, and it is among the counties with the lowest incidence of kidney cancer.

If one person in the county has kidney cancer, the rate is 1%, which is likely among the counties with the highest incidence of kidney cancer. No matter what, this county will appear on either the lowest or the highest lists purely by virtue of its small population.

When you also consider that most counties are sparsely populated (at least compared to the relatively few highly populated urban and suburban counties), it should be obvious that the top and bottom of nearly any incidence-based ordering of counties should be dominated by these rural, sparsely populated counties.

A common strategy for people trying to improve at something is to make a list from best to worst of others who do that something. Then select only those at the top of the list—that is, those with the best results—and try to find a common thread. “What is it that these people are doing that makes them so successful?” is the common idea.

This strategy, however, is extremely prone to failure. Hopefully it is obvious why—the people at the top of such a list nearly always have a relatively sample size from which they’ve derived their success. This is true even when you understand this problem and try to correct for it.

Say, for example, you are trying to find the best poker players. So you take a list of online results and sort by winrate. Naturally you will see that the players at the very top of the winrate list have probably played a single hand and won it.

Well, that’s obviously not what you want. So you say, “Ok, I only want players who have played at least five thousand hands.” While this will eliminate the most extreme effects, the players at the top of the list will still be heavily biased toward those with small samples. This is true both because of the small sample size effect as well as that there are simply more players with smaller samples than with the really big samples.

You won’t really have a list of the best players. You will have a list of some of the players with smaller samples who happen to have run good. Trying to figure out anything useful from this list will be a fool’s errand.

This “strategy” to learn about the world is used all over the place and, as I said before, is usually garbage. Now that I’ve pointed it out, pay attention to how often people try to use this method to learn about something and realize that they’re very likely drawing poor conclusions.

Problems With Pet Plays

Unfortunately, whether you know it or not, you probably use this strategy to figure out good poker plays. That’s because this is essentially the strategy of trial and error when it’s applied to a game with random outcomes like poker.

Say you happen to like to raise your flush draws all-in on the turn. Why is this a thing you tend to do?

The answer is probably because you tried it a few times, and it worked in most of the cases. So you mentally filed this play as a “good” one and you looked for more opportunities to try it.

Or perhaps you never make this play. It could be that you tried it a few times, it backfired a lot, and you filed it away as a “bad” play, never to be tried again.
All poker players have their own short sample histories with all sorts of different plays. Alan tried to bluff a few monochrome flops once and failed, so he doesn’t do it anymore. Bettina on the other hand had her best session ever just blasting through monochrome board after monochrome board.

Chris called it off with pocket kings, an overpair to the board, a few times and ran into sets every time and now he just folds out. Alan caught Dina trying to bluff with her kings and so calls down tenaciously now.

I’m oversimplifying the process, but in essence this is how most people learn the game. It’s not effective. Instead of separating the good plays from the bad plays, you are separating plays into the ones that happened to work a few times and ones that happened to fail a few times.

You aren’t selecting for good plays, therefore. But even worse, you’re making your overall play more predictable, because you are choosing to overuse certain plays and underuse (or never use) others.

Final Thoughts

The way to fight against the human tendency to shape your game this way is to relentlessly analyze your hands. Write down any significant hands you’ve played and go through the math at home. If a play went bad, try to figure out whether it was bad play or bad luck. Just as important, go through the plays that went well and make sure that they were good plays rather than just good luck. Just because you got a fold doesn’t mean that you should keep making that play every chance you get.

The variance in random events fools us in ways we often never even notice. Don’t be fooled. Analyze your game rigorously and root out as much of the fuzzy thinking as you can. ♠

Ed’s newest book, The Course: Serious Hold ‘Em Strategy For Smart Players is available now at his website edmillerpoker.com. You can also find original articles and instructional videos by Ed at the training site redchippoker.com.