First Impressions of Metaculus

Prediction markets have some problems

I was fairly sure I would dislike Metaculus1¹ 70 %. I have known about it for a long time2² 2–6 years at 90 % confidence but never bothered to try it until now. The hesitation was because I used to think prediction markets were the better approach to amateur forecasting3³ 85 %. Metaculus is not a prediction market, it’s a forecasting platform and community.4⁴ 99.5 % but I will stop quantifying statements now.

Both prediction markets and forecasting platforms have their flaws. Let’s look at prediction markets first.

Betting is important and prediction markets simplify it

One of the key points in the rant on guesses, predictions, and bets is that betting serves an important function in civilisation; betting is how differences in belief can be resolved peacefully and it can reduce the risk of adverse events. Humans also generally tend to make better forecasts when they are forced to bet on them5⁵ How to Measure Anything: Finding the Value of Intangibles; Hubbard; Wiley; 2014., because … I don’t know, psychology?6⁶ Somehow, even just imagining loss makes us more acutely aware of the uncertainty of the situation. You’d be surprised how quickly people go from loudly confident to “Yeah, but you never know, you know?” once you start talking money.

Prediction markets structure betting by ensuring that all bets have the same standard format, rules, and amounts, which makes it easier to find counterparties to the transaction.7⁷ A prediction market is to events what futures contracts are to commodities. Prediction markets make betting on events accessible and easy for regular people, and they encourage people to put their money where their mouth is.

Naturally, I’m a fan of prediction markets. I like prediction markets so much that I managed a primitive prediction market for colleagues at a previous workplace.

Prediction markets have leverage and collaboration problems

That said, prediction markets are not perfect. One of the major flaws is that they are fundamentally adversarial: if I know something you don’t, it would be stupid of me to share information because any money you would earn from that comes out of my pocket.

There are also other problems mainly related to impossible leverage requirements. Under plausible assumptions of wallet thickness, no single actor can update the price to correspond to their actual probability assessment. A prediction market only allows actors to indicate in which direction the price should move. This means prices can take a while to converge to the market assessed probability.

The above is related to risk management: I may not want to sink my entire wealth into an outcome, however wrong I think the market is.8⁸ This gets us into the Kelly criterion, which will have to be a separate article at some point in the future. Related to risk management, the price in a prediction market may also go in the wrong direction when people use it to hedge; and when stakes get higher outright manipulation becomes a possibility.

I had not thought about these problems very hard, because I didn’t think there was a way around it. Turns out the Metaculus team has found a way.

Metaculus somehow solves honest forecasting

The main problem prediction markets solve – and the reason I kept coming back to them – is that of honest forecasting. It’s actually quite hard to get people to forecast honestly. People tend to do it either with their hearts rather than their brains9⁹ E.g. suggesting that their favourite sportsball team will win even though the odds are stacked against them. or to look good rather than be accurate10¹⁰ Especially in politics there’s a lot of virtue signaling forecasting going on.. On social media, the clickbait type of forecast has also become a thing.

Prediction markets avoid all of that by forcing people to stake something of value to their forecast. Once something real is on the line, people increasingly drop the pretenses and forecast their honest beliefs.

Metaculus manages to get people to forecast honestly despite not having anything real on the line. I don’t know exactly how that happens, but part of this is that there’s really nothing to gain from forecasting dishonestly – your actual forecast is hidden to everyone, so you’re only being dishonest to yourself. And then your track record will take a turn for the worse – and that can be seen by other users (though it’s not visible by default.)

In fact, the people on Metaculus place great pride in and respect toward a solid track record of accurate prediction.

Metaculus does other things well too

Here are some other observations on things that make Metaculus good.

The scoring system is designed to incentivise predicting early and updating often. This means even if you honestly don’t know whether the S&P500 index will reach a new all-time high this year, you’re better off putting in a non-committal 50 % now, and then if you get more useful information, you can refine it later.

This is in contrast to a prediction market where it really only makes sense to purchase a contract if you think the market consensus is wrong.
Admins are very active in conversations around resolution criteria and question descriptions. This means ambiguous questions can be improved quickly, which leads to more interactions with them.
There’s a strong community of sharing information and explaining reasoning. This isn’t just dumb luck: many tournaments have financial prizes not only for the best forecasters, but also for the people who write the most informative comments.

The tendency toward dichotomous thinking

One of the classic markers of an expert is that they see things more along a continuum, rather than divide things into dichotomies.11¹¹ One thing I learned from studying a little ethnography is that if you ever find yourself theorising about a dichotomy, that’s a good sign you need to go back to the field and learn more about the phenomenon, because almost nothing is dichotomous in real life. I find myself making this mistake quite often on Metaculus. On questions like whether Iran will re-enter the jcpoa before some date, I find myself going, “yeah, it sounds remote but plausible that they will enter back into some sort of deal” – but then someone points out that there are ongoing talks for a much smaller version of said deal, and it hits me that while re-entering the full deal is reasonable, much more reasonable is something in between, which means I should adjust my probabilities for the full thing lower.

I hope to learn more about my forecasting mistakes from my continued interaction with the Metaculus community!