Imagine for a moment that we need to transfer 30 files from my documents directory on a medium that’s limited to 16 MB in size.2² Maybe an old usb stick. Will we be able to fit them in, if we don’t yet know which 30 files will be selected?

The mean size of all files is 160 kB, and the standard deviation is 0.9 MB. Since we are talking about 30 files, if the LessWrong post is to be trusted, their sum will be normally distributed as

\[N(\mu = 30 \times 0.2,\;\;\;\; \sigma = \sqrt{30} \times 0.9).\]

With these parameters, 16 MB should have a z score of about 2, and my computer3³ Using slightly more precise numbers. helpfully tells me it’s 2.38, which under the normal distribution corresponds to a probability of 0.8 %.

So that’s it, there’s only a 0.8 % chance that the 30 files don’t fit. Great!

Except …

In this plot, the normal density suggested by the central limit theorem is indicated by a black line, whereas the shaded bars are the actual sizes of random groups of 30 files in my documents folder4⁴ Using find and perl to get it into R..

There are two things of note:

1. The plot extends out to 50 MB because when we pick 30 files from my documents directory, one plausible outcome is that they total 50 MB. That happens. If you trust the normal distribution, you would consider that to be impossible!
2. If we count the area of the bars beyond 16 MB, we’ll find that the actual chance of 30 files not fitting into a 16 MB medium is 4.8 %.

The last point is the most important one for the question we set out with. The actual risk of the files not fitting is 6× higher than the central limit theorem would have us believe.

If we adhere to the Gaussian hypothesis, we will think we can pick 30 files every day and only three times per year will they not fit. If we actually try to do it, we will find that every three weeks we get a bundle of files that don’t fit.

Using the same procedure but with data from the S&P 500 index, the central limit theorem suggests that a drop of more than $1000, when looking at a time horizon of 30 days, happens about 0.9 % of the time.

This fit might even look good, but it’s easy to be fooled by eyeball statistics. If we count the proportion of the area under the bars we discover the real probability is 1.9 %, i.e. more than twice as often. Also note that in the real data5⁵ Downloaded from Yahoo Finance., a month with a $3000 drop is possible, contrary to what the normal distribution would suggest.6⁶ All of this said, I have to admit that the normal distribution fits this data closer than I would have expected. Random walks in finance are notoriously jumpy and rarely converge quickly according to the central limit theorem. (See Taleb, Mandelbrot, and others for more on this.) I think there are two reasons our S&P 500 data is well-behaved: it’s a composite of multiple stocks, meaning we’re not really looking at 30 things but 15000 things, and the data comes from a relatively stable period (year to date.) Note the consequences of this: fairly well-behaved data that optically looks like a decent fit can still result in a 2× error in the tail!

Maybe a bit convoluted, but imagine we’re studying the wildlife in Finnish lakes. There are a lot of Finnish lakes, so we’ll study a random sample of 30 of them only. Our budget only covers 7000 km² of lake – what’s the risk we blow our budget?

Going by the clt, the risk is a measly 0.4 %. That we can live with.

But according to the real data7⁷ Sampled from Wikipedia., the probability is over 3× as high, at 1.4 %.

Much like in the previous example, this might look like a good fit if we just eyeball it. But if there’s something we should eyeball, it’s the tails. The fit is not good in the tails. That’s what will cause us problems down the road.

According to the clt, the probability of the next 30 armed conflicts costing more than 250,000 lives yearly is 2.5 %, but according to the real data8⁸ Also Wikipedia., there’s a 4.8 % risk.

Again, we underestimate the risk by almost 2× when we blindly use the central limit theorem. In this case, the Gaussian is also visually a bad fit.

In connection to the 2007–2008 financial crisis some executives complained that they were seeing “price movements that only happen once every hundred thousand years.”11¹¹ Something like that – I don’t remember the details. This makes it seem like they were modeling price movements using a normal distribution, and saw 9-sigma movements.

If only they had known Cantelli’s inequality. Since we do, we can tell that 9-sigma movements can happen as often as four times a year. They usually don’t, but it would be imprudent to not consider the possibility.

Here’s a handy table:

Practical statistics often uses three standard deviations as the limit for what’s normally possible. That is reasonable under a normal distribution, where only a fifth of a percent of data points fall beyond it. It’s not true in general, though, because we can find as many as one in ten data points outside of three standard deviations if we’re unlucky.

Since the Cantelli inequality is true even for the most extreme data, it’s necessarily quite conservative. In most real-world cases, we won’t see 4 % of values outside of five standard deviations. A mental heuristic I sometimes apply is to assume a value somewhere between the Gaussian probability and the Cantelli bound. The more reason I have to believe the distribution is Gaussian, the closer to that bound I pick.12¹² This process is just as unscientific as it sounds like, and it looks like it could be possible to extract better heuristics from the Wikipedia page for Chebyshev’s inequality.

As an illustration of how slowly heavy-tailed data converge, here are 1000 convolutions of the file size data.

It starts to look better, but the tails are still not a great fit. Quite a ways away from the 30 suggested by the LessWrong article!13¹³ And this is from a fixed set of 560 files – if we were dealing with an actual theoretical distribution able to generate an infinite number of different values, it would converge even slower.

When hearing “the more reason I have to believe the distribution is Gaussian” it can be tempting to eyeball the data in front of us and see if it sort of looks like a bell curve. That is a mistake. Heavy-tailed data generate disproportionately two things, compared to the normal distribution:

1. Values far out in the tail, and
2. Central values.

In other words, a heavy-tailed distribution will generate a lot of central values and look like a perfect bell curve for a long time … until it doesn’t anymore. Annoyingly, the worse the tail events, the longer it will keep looking like a friendly bell curve.14¹⁴ Intuitively, heavier tails means probability mass is shifted from the shoulders of the distribution into the center and tails, but more into the center than the tails, on aggregate. However, the tails still do expand. This is the sort of thing that might take an animation to explain properly, but I don’t know how to make those.

The best way I know to figure out if data is heavy-tailed or Gaussian is to be careful and apply theoretic knowledge judiciously. Defer to real-world data when possible.

Many of the theoretical distributions we know, including every single one used in the LessWrong article, are classified as thin-tailed. This means their tail decays faster than or equal to the exponential distribution, which is the threshold between thin- and heavy-tailed distributions.

This explains why the author found the distributions converged so quickly according to the central limit theorem – thin-tailed distributions are very well behaved under repetition: you get the central values and maybe something shallow into the tails, but never a deep tail event. Those deep tail events are what makes the heavy-tailed distributions converge very slowly.