Perverse Statistics: Regression Towards the Mean

Blissful ignorance or intentional denial?

One of the ideas I keep coming across in my readings is regression towards the mean, the phenomena that if a variable is extreme on its first instance, it will be closer to the mean on the next. Seems quite straightforward, but this simple little fact of statistics is a very perverse happening.

What does it this mean in practice?
Well, everybody has some decent understanding about the nature of bell curves, and intuitively understands the curse of the mediocre. So, if you’re parents are extremely tall, you are likely to be tall, but shorter than them. If you are a musical genius, your kid will probably play a guitar better than most of us, but he’s unlikely to match your talent.

alt
Intuitively, this makes perfect sense. In a world without regression towards the mean, we’d seek out those with similar talents and soon live in a world of the tall, the small, the smart, the fast, the extremely analytical and what other extreme tribe we can come up with. But no, we live in a world where reversion to mediocrity keeps us modest, alike, boring, at a safe distance from the amplified extreme versions we could have been.

This beautiful fact of life is just as persistent as our inability to understand it. Take the movement of stock prices for example. Hersh Shefrin (in his book Beyond Greed and Fear) explains how when a stock price is very low, the most likely outcome would be for it to regress back towards its mean price. Thus, the most probable price in the next period will fall somewhere between the extremely low one and the mean value. And what happens in practice? We expect losers to remain losers and do even worse, so we shy away from that stock, while the more ambitious of us expect the price to outperform the mean, again, taking “towards the mean” and transforming it in “above the mean.” How fickle is our understanding?

To some degree, I accept our reluctance to embrace regression towards the mean because life works in ironic ways when it comes to this one. Daniel Kahneman (in Thinking, Fast and Slow) gives the examples of flight instructors praising cadets for extremely good performance, only for them to perform less well on the second run. Similarly, commenting on bad performance seems to be followed by improved performance. Logical conclusion? Praise reduces performance, punishment increases it. Is this the correct conclusion though? Not exactly. An extremely well or bad performance is likely to be followed by a mediocre one.

See the dilemma? We are statistically punished for being nasty, and perversely rewarded for being nice. Life, why do you make it so hard for us to learn our lessons?

Of course, the whole thing gets even more troublesome the moment you realise this says a lot about the fine line between skill and luck. By luck, I mean a random event that happens in your favour. If you did exceptionally well on an exam, was that luck or skill? They don’t call it reversion to mediocrity for nothing... I’m not even gonna go into that whole discussion of baseball statistics and all the fallacies associated with sport performance. I’m going to focus on that lovely race horse that wins race after race. On the gymnast who works her ass off and takes home the medals year after year. You see, even though I understand that performance is a combination of skill and “luck”, I blindly focus on one and refuse to let go of my belief in talent, in hard work and earned skill.

As for luck, I can only hope that randomness will work in my favor, that passion and focus can shift basal conditions and increase the probability of positive events. And despite the reasonably logical statement I just wrote, in my heart, I just hope I’m lucky.

References:

Beyond Greed and Fear: Understanding Behavioral Finance and the Psychology of Investing
Hersh Shefrin, 2000

Thinking, Fast and Slow
Daniel Kahneman, 2011