In case you know the drill already, just paste your data into the tool and get on with your life. I wish I had made this long ago! That tool shows a q-q plot for visual comparison, it computes Kuiper’s test statistic, and has a handy reference table for significance thresholds.

It lets you choose between multiple theoretical reference distributions. Or you can input your own reference distribution, either as an empirical distribution function known exactly, or a reference sample with embedded uncertainty. The theoretical reference distributions that support it come with aic values for model selection.

If none of the above made any sense to you, read on.

Many quantitative questions in life take the same general shape:

1. We have some maybe-special data.
2. We know such data commonly follows some distribution.
3. We want to know if the maybe-special data belongs to the common case, or if it is indeed special in some way.

We tend to have a good idea of what happens generally – this is a reference distribution we know with certainty (in our example, what the scores are for the population in general). We then have a smaller set of data points – a sample – which we think might be special in some way (in our example, the scores of a treatment group), but we’re not sure. To get more sure, we want to perform some sort of test that can tell us if our sample is actually special. If the test fails, we’ll assume our sample is a small manifestation of the reference distribution.

Such a test would be like a black box with two slots and a light bulb. Into the big slot, we put a reference distribution. In the smaller slot, we slide in our sample. Then the box hums for a brief moment, and when it goes silent, one of two things can happen. Either

• the light bulb lights up! In which case the box has determined, with high certainty, that the sample did not come from the reference distribution1¹ Okay, this is a frequentist test, so technically what the box has determined is that if repeated, similar-sized draws were made from the reference distribution, the test statistic would be less extreme in more than 95 % of them. But frequentist confidence intervals commonly approach Bayesian posterior credible intervals, so I’m going to gloss over this difference. Albers, Kiers, and Ravenwaaij’s 2018 argument is the commonly cited defense.; or
• the light bulb stays off, in which case the box cannot tell the sample and the reference distribution apart.

It’s important to note that in the second case, the box does not claim the sample came from the reference distribution. That’s impossible to prove, because it may come from something that’s so subtly different it isn’t detectable with limited data. The lamp staying dark only means the sample wasn’t different enough from the reference distribution to tell them apart. This can happen because they are the same, or because they are very-similar-but-not-the-same, but also because the sample is too small to achieve certainty in the difference, even if the difference is quite large.

For better or worse, there aren’t very many tests that fit our criteria.

• The Kolmogorov–Smirnov test is simple and very popular. The test statistic is computed by taking the greatest difference between the distribution implied by the sample and the reference distribution and scaling it by the number of values in the sample. There are many tables available for comparing this value to the appropriate confidence threshold.

• The Cramér–von Mises and Anderson–Darling tests are both popular special cases of a family of tests. In this family, the test statistic is computed from all sample values, rather than only the greatest difference. This makes these tests strictly more powerful than Kolmogorov–Smirnov, but also more complicated to compute.

  The Cramér–von Mises statistic weighs all parts of the sample equally, whereas the Anderson–Darling test derives more of the statistic value from the tails of the sample. This makes Anderson–Darling more powerful in many cases, but again, slightly more complicated to compute.

All three of these tests have one drawback: they are not invariant under rotation. This means that if the measurement of interest has a cyclic distribution (e.g. hours of the day, wind direction, hue in colour spaces, edge direction in images, etc.), then these tests can detect spurious differences that aren’t really there, and only appear due to accidents of rotation.2² If something happens a lot around midnight, for example, these tests will see that as it happening “at the start of the day” and separately “at the end of the day” rather than as one lump of events that just happens to straddle a cut-off point. Fortunately, there are tests that are invariant under rotation.

• Kuiper’s test is like the Kolmogorov–Smirnov test except the test statistic is based on the sum of the greatest deviations in both directions from the reference distribution. This makes Kuiper’s test more powerful than Kolmogorov–Smirnov, and somehow it also makes it invariant under rotation.3³ This is particularly neat because rotational invariance would typically trade away some power, by virtue of being more general. But in this case we get both rotational invariance and greater power. Kuiper’s test is only a tiny bit more complicated, and otherwise a strict improvement over the Kolmogorov–Smirnov test.
• The Watson U² test is an adjustment to the Cramér–von Mises family of tests that makes them rotationally invariant. This means the Watson U² test is more powerful than Kuiper’s test, but also more complicated to perform.

I think Kuiper’s test really hits a sweet spot between power, appropriateness for cyclic and non-cyclic distributions alike, and simplicity. We’ll use that going forward. My implementation of the test is based on the procedure used in Stephens’ modification4⁴ Use of the Kolmogorov–Smirnov, Cramér–von Mises, and related statistics without extensive tables; Stephens; Journal of the Royal Statistical Society; 1970. of the test, mainly because the paper where it’s introduced happened to be a good, concise reference.

Recall the plot we were confused by. We had a grey reference distribution, and on top of it we overlaid a sample that had undergone treatment. The treatment sample looks like it comes from a different distribution, but does it?

We can imagine stretching, warping, and squeezing this plot so that the grey reference line becomes a straight diagonal line. I took a screenshot of the plot above, opened it in Gimp, stretched out some parts and compressed others, until the grey line almost became a straight diagonal.

Clearly, I’m not very good at this. Fortunately, we can ask the computer to do this transformation instead. The result is what’s called a q-q plot, because the quantiles of the sample are plotted against the quantiles of the reference distribution.

The q-q plot is a little harder to interpret than the raw distributions, mainly because it sacrifices information about the scale of the measurement. But when it comes to comparing distributions, the q-q plot does just as well as the raw distributions, only warped to be viewed from the perspective of the reference distribution.

That’s in fact what makes the q-q plot so useful when eyeballing differences: we don’t have to keep track of two distributions simultaneously, but instead we can focus on the relationship between the sample and the fixed diagonal. The q-q plot is also convenient for getting the data needed to perform Kuiper’s test: we simply record the largest distances under and over the diagonal.

In case you’re not yet convinced, here are some more examples of when we might want to perform a test like this.

• We have noticed that the birthdays’ of people at a social gathering seem to cluster toward the end of the year rather than be roughly uniformly distributed as would be expected from the general population. Maybe that’s just random chance, or there’s something at play which selects for people born at the end of the year.5⁵ This was actually the motivating case for making the tool presented here. I didn’t have a way to test this hypothesis on the go, so I made one later when I got home. Yes, I’m that fun at parties.
• We have recorded the times between incidents in a software service. If these times can be drawn from an exponential distribution, that means the incidents occur independently of each other. That has huge consequences for how we choose to deal with them.6⁶ But the reason I cared was for a forecasting competition.
• Someone said commits sizes are log-normally distributed. We can get some commit sizes from e.g. nginx and test if they agree with the log-normal distribution.
• From a previous article, we have the ages at which Supreme Court justices retire. We speculated these followed a Weibull distribution, but it would be nice to see if Kuiper’s test agrees.
• We find some data which we think corresponds to biweekly maxima of daily violations of Taiwanese airspace. If they are, they should fit a Fréchet or Gumbel distribution fairly well. If not, we might be looking at some other data than we thought.

Well, we can find out.

I have had the Kuiper Q-Q plot tool for a while now, and it’s come in handy very often. After all, this is the second most common hypothesis test I find myself wanting to perform.

The other kind of test I often need is effectively the same, except when the values are discrete. Creating that tool in a way that’s simple, generic, and still usable has proven to be slightly more of a challenge. Maybe some other time.