Classical hypothesis testing involves a frequentist–objectivist procedure:

1. Before running the experiment, we set a confidence level at which we would like to perform the test.4⁴ This is commonly something like 5 %, often for no good reason at all. Let’s say we are happy with results at a 10 % confidence level for our product future.
2. We run the experiment, and discover that variant A was +37 better than B (on whatever dimension we were measuring) .
3. The difference of +37 happens just by random chance (sampling error) about 23 % of the time if we would run experiments similar to ours many times.5⁵ We usually don’t run the same experiment many times – instead we base this probability on the assumption that the sampling distribution is approximately Gaussian, or Student t, or some other theoretical distribution. Sometimes there’s good reason to believe this, sometimes there’s not.
4. Since 23 % is greater than the 10 % confidence level we picked before the experiment, we conclude that A is not better than B6⁶ Or, if we’re more careful, that we have failed to prove that A is better than B..7⁷ Had the false positive risk been lower than the confidence level, we would have concluded that A is definitely better than B. The idea behind this is that randomisation ought to ensure that any contributing factors not under study are equally divided between the groups, and the only thing separating the groups is the thing under study.
5. Having learned this, we treat A and B as definitely equal in future decisions, including this one: we continue to maintain both variants despite the cost of maintenance.

Note the unfortunate situation we are forced into by frequentism–objectivism. Since the probability we get is the risk of type I errors, we cannot use it meaningfully in decision problems. Hence we are forced to make decisions pretending our hypothesis is either definitively true, or definitively false. And we have to make this judgment based on fairly limited information (type I error) to boot.8⁸ In practise, even frequentists maintain an internal judgment of the likelihood of the hypothesis, informally weighing together the results of similar studies. But that’s not how the strict process of hypothesis testing proceeds under the frequentist–objectvist framework, because that happens outside of the framework.

For someone with a more subjective view on probability, the process is a little different.

1. We run the experiment and discover that variant A is +37 better than B (on whatever dimension we are measuring).
2. This difference leads to us thinking there’s a 77 % chance variant A is actually better than B.9⁹ You might think there’s some relationship between this number and the 23 % the frequentist got. There’s not. It’s pure coincidence. Do not mistake the risk of type I errors for the likelihood that the hypothesis is false!
3. Now we know that if we discontinue maintenance of B, there’s a 77 % chance that we make a $12,000 profit, and a (100 - 77) = 23 % chance that we make a $18,000 loss. This is a plain expected value calculation: 0.77×12,000 - 0.23×18,000 = $5,100 profit.10¹⁰ Note the tricky wording here: we are not guaranteed a profit. In fact, there’s a fair chance (about one in four) we’ll be wrong and make a loss on this decision. But if we continue to use this framework for future decisions, we’ll maximise our profit in the long run. Thus, we discontinue maintenance for variance B.

Note the fuzziness of these steps! We never have to claim that the effect is definitively real, because we have the probability we assign to it being real. In decisions, we can then account for both possible outcomes, weighted by their appropriate probabilities.

I have received some comments from readers claiming that we also have to step outside of the Bayesian–subjectivist to make decisions. These comments explain that the Bayesian hypothesis test is an application of Bayes’ rule, and the prior probability cannot be determined within the Bayesian–subjectivist framework – it needs to come from… somewhere else.

I disagree. There is nothing about the Bayesian–subjectivist hypothesis test that requires an application of Bayes’ rule. If it is applied, it may just as well be applied backwards, e.g. going from posterior (based on the experiment) to what the prior must have been. This may seem like an unfamiliar view, but the basic idea is that a Bayesian–subjective probability doesn’t have to come from any procedure; it’s just there in the minds of the person doing the evaluation, and that’s fully within the framework itself. I have an upcoming article with more details on this.