Researchers have long understood that more information is not necessarily better. Instead, they would argue that biases are the problems behind weak forecasts. When we make forecasts, we are falling for mental and external biases, like narrative bias or management pressure. Training in prediction focuses on reducing vulnerability to biases – and as we saw above – that training works; it improves the Brier score by about 10 %! All evidence points to bias being the problem.

Then Kahneman wrote an entire book5⁵ Noise: A Flaw in Human Judgment; Kahneman, Sibony, Sunstein; William Collins; 2021. on why bias is not as big a deal as we make it – in practise, noise is often large enough that its effects overwhelm those of any bias present6⁶ Fans of statistical process control will recognise this tune: we focus on reducing noise first, because noisy measurements hide any improvements to the trend we want to make.. But noise is also difficult to measure, so it, and its effect, is understudied.

In the bin paper, the authors found out that the training that aimed at reducing bias actually appears to reduce noise and that’s why it works. For all practical purposes, the entire effect of training is reducing noise. Oops. But also good news, because now we know why training works, and that noise has a significant effect.

The authors find also that the majority of the improvement from teaming is reduced noise. The members of the team can temper each other and prevent over-reactions to insignificant data. Superforecasters again have reduced noise compared to regular teams. Most of the improvements as we go up the ladder of better forecasting comes from reduced noise.

However, when comparing individuals to teams there are also two other effects in play. First up is information. Aside from the main effect of teaming, which is reduced noise, teaming also improves accuracy on short-term forecasts (a month or less) due to improved information. This makes sense: when multiple people get together and second-guess each other’s forecasts, clearly some people will sit on information other people have missed. No mystery there.

What’s interesting to me7⁷ This conclusion is my own, not the authors. Since the authors do not point this out, either it is already established in the field, or I’m missing some flaw in my argument that invalidates it. Let me know if you know! is that this improvement due to added information peters out at about 30 days, meaning we can classify forecasts into short term where injecting additional information produces an improvement, and long term where additional information does not help accuracy, and the threshold to long term sits around 30 days.

Teaming also reduces bias somewhat. The paper speculates that some of this might be because team members collaborate by catching each other’s mental biases, which are notoriously hard to self-monitor for.

But here’s what’s odd: the debiasing effect is very small for short-term forecasts, and becomes significant only in long-term forecasts. The authors do not explain this. I can only speculate from personal experience that maybe biases compound as the time horizon extends, so that the debiasing effect of teaming exists also in short-term forecasting, but biases themselves have a comparatively small effect in short-term forecasts.

Superforecasters exhibit basically the same improvements as teaming8⁸ Up to and including the contribution differences in short-term/long-term forecasts., except even stronger. In Superforecasting9⁹ Superforecasting: The Art and Science of Prediction; Tetlock & Gardner; Cornerstone Digital; 2015., Tetlock describes superforecasters as performing the job of teams except in one brain, so it’s not too surprising that they have similar improvement patterns.