As before, non-binary questions where we need to forecast an entire probability distribution are more difficult. The contribution from each type of these questions to my average log score was:

By looking at a histogram of the scores split up into the two types of question, we can also see that the continuous questions have wider variance, i.e. they were wilder guesses on my part.

That said, they are not as broadly distributed as in the previous tournament4⁴ here theyW ranged from -1.2 to +0.3 so at least I’m a little more careful now. Some of the problems I had with continuous questions this year were actually not a failure to forecast, but a failure to input the distribution I had in mind. (See the New York rain question under Lessons Learned.)

In the previous tournament I discovered that quantitative modeling is where I have an edge over the community, and I wanted to do that for more questions this time around. I did, and while the edge has persisted, it is smaller.

I think the decrease in the average edge is to be expected, because I broadened the scope for which questions I considered quantitative modeling. Even questions that don’t seem like they are amenable to quantitative reasoning can be forecast using e.g. a tallying heuristic over a small ensemble of forecasts, where we make two or three different types of forecasts and then take the average of those to get the final number (see the Scottish human chain question under Lessons Learned for an example). I did that type of light quantitative modeling for more questions than last tournament, so naturally the effectiveness of what counts as quantitative modeling will go down.

Here’s the density of scores for forecasts, broken into groups based on whether they were modeled quantitatively or I picked a forecast from thin air.

What concerns me in particular about this plot is that while quantitatively modeled forecasts seem to consistently profit a small number of points, the gut feels are all over the place. That suggests any apparent improvement in gut feel is more about luck than anything else, and I should continue to find ways to build forecasts from numbers.

However, I also tried a new qualitative way to model forecasts: using some sort of actor–motivation framework, where I try to think of which people have a bearing on the outcome, and what their respective priorities are. I read about someone using this type of reasoning a while ago, but I have – annoyingly – forgotten where.5⁵ If you happen to know, please email me! I want to learn more.

We’ll compare this actor–motivation framework to quantatively modeling (which we know has a meaningful effect) to get a sense of whether its effect is meaningful.

Although the data is shaky because I didn’t use it for very many questions, the effect looks comparable to quantitative modeling, which is neat.

There might be something to it, although it is worrying this framework is responsible for my greatest loss on any question this tournament. I’ll continue to explore it, but prudently.

My forecast: 25 %, when the community forecast was 50 %.

The Chain of Freedom Scotland organised an event with the purpose of people holding hands all the way across the narrowest part of Scotland. This question would resolve positively if the human chain did indeed stretch across Scotland – although people were allowed to cheat and hold hands through banners.

My first attempt at modeling this was based on the following assumptions:

• Size of Scottish population: 5,000,000
• Fraction of Scottish population supportive of the cause: 45–60 %
• Fraction of humans who are interested in this sort of event: 20–50 %
• Fraction of interested humans actually intending to attend: 2–20 %
• Fraction of intended attendants who show up: 70–95 %
• Number of people who need participate for success: 50,000

I plugged these numbers into Precel to compute the final percentage and it responded with 42 %.

However, I also noticed in the description of the event that they had divided the width of Scotland into 11 subsections. That made me realise that the success rate of the entire endeavour is capped at the success rate of the least-likely-to-succeed subsection. In other words, even if 10 subsections are guaranteed to be successses – as long as just one subsection has a success chance of 30 %, the entire question will have a resolution chance of 30 %.

Since much of the planned chain went through well-populated areas, I speculated the average successs rate for a subsection could be as high as 90 %, but I imagined that there is likely one trouble spot that has a success rate of just 30 %. That means the total chance of success of the endeavour is 14 %.

The average of 42 % and 14 % is 28 %, but the shape of the problem (all subsections must be successful for the question to resolve positively) made me lean slightly lower, so I forecast 25 %.

My forecast: 40 %, when the community forecast was 50 %.

The Mid Bedfordshire constituency has been held by the UK Conservatives for many decades, but polling indicated it would go to Labour in this by-election.

I don’t know why the community forecast was as high as 50 %. Polling was very clear in this case, and the only reason I didn’t go lower than 40 % was that I was concerned I had misunderstood something about UK politics. Maybe that was true for the rest of the community also.

This was one of the first questions for which I tried the actor–motivation model. I reasoned that both sides would most likely want to avoid the drawn out process of the last election, and a candidate would not be proposed unless they had already found broad agreement. In other words, I expected the disagreements to turn into delays, rather than more ballots.

Compared to the community, I had my money on “it will either take just one or two ballots (because they are careful), or more than 10 ballots (because they throw candidates at the wall to see which one sticks).” Turns out they did something in between: they carefully threw candidates at the wall, which made one stick after four ballots.

That was an outcome that seems obvious in hindsight, but I under-weighted it at the time.

This question I lost points on due to technical difficulties, rather than a bad forecast. New York City had seen many consecutive weekends of rain during the autumn, and the question was when the rainy weekends would stop.

Because of a missing feature of Metaculus that has been added since this question6⁶ At the time, Metaculus did not have a multiple-choice question type, so questions like these needed to be asked with a continuous distribution of dates., answering this question required carefully aligning pixels using slider inputs. I accdentally imisaligned my pixels in a way that made the system think I had forecasted the rainy weekends to end on a Monday morning rather than the last hour of a Sunday, which is when the weekend actually ends. I lost some points on this tecnicality.

I’m glad Metaculus now has multiple-choice questions so we don’t have to align pixels as often.

My forecast: 95 %, when the community forecast was 75 %.

The appropriate forecast for this would actually have been something closer to 97 % than 95 %. The reason I submitted a lower forecast was the spotty and difficult-to-use historic data that was available. There were two main problems:

• Missing data was sometimes encoded with an “n/a” type value, sometimes as a 0, and sometimes that row was just not present. This made it hard to align the data from multiple years without writing a script for it.
• The data from the current year was split into two different files, and it took some time before I realised how they were related.

Had I taken the time to sort out those data problems earlier, I would have submitted a more confident forecast. I suspect the same problem affected the community forecast, though, so I didn’t suffer much from it.

My forecast: 50 %, when the community forecast was 60 %.

The community did seem fairly comfortable “betting on the incumbent in an African election”, as a commenter put it. Maybe 60 % was the correct forecast, I don’t know.

The reason I bring this question up is to note how many points one can get from a non-committal 50 % forecast when the community is just 10 percentage points on the wrong side of maybe. +0.21 corresponds to more than a sixth of my total score. It really pays not to be too confident.

I have this idea that national populist politicians get more votes in the real election than polling indicates. I don’t know why that is, nor from where I got the idea, but it contributed to my score on this question.

The bigger contribution, though, was a very wide probability distribution. I don’t know why but it’s extremely easy to submit too narrow distributions on continuous questions.7⁷ Although I have no evidence of it, I secretly speculate that maybe something about the algorithm that computes the community prediction does so in a way that makes it seem like the community is more certain than it is.

My forecast: 80 %, when the community forecast was 95 %.

I think the community had the right forecast here, but I understood so little about the matters involved – and I didn’t have time to research it either – so I opted for a more conservative forecast of my own. Despite losing some points on this, I think it was the right thing to do.

My forecast: 79 %, when the community forecast was 38 %.

This was a question where I had a strong sense it was likely to resolve positively, but it was based mainly on gut feel. There was also some debate around what would really count as a positive resolution, further increasing my uncertainty. Thus, when the community forecast was revealed, I quickly reverted down to something very close to it.

I don’t know if that reversion was a mistake. It is annoying to look back on the times when my gut was right but I didn’t listen to it. On the other hand, the statistical evidence above suggests that it’s maybe a losing policy to pay too much attention to my gut as a rule, and the correct thing is to revert closer to the community forecast unless I have stronger reasons not to.

I had the same problem when evaluating whether any shipping company would return to routing ships through the Red Sea after the Houthi attacks intensified. (They did, and I strongly suspected they would, but I wasn’t confident enough to stray very far from the community prediction.)

My forecast: 5 %, when the community forecast was 3 %.

I know the death rate of the US military is very low, even among troops in contact with hostile forces. So I just guessed fairly wildly at 5 %. Then halfway through, someone prompted me to look at the data and at that point I realised how low the death rate really is, and I updated to 1 % immediately.

Clearly, I should have just taken three minutes to look up the data from the start and opened at 1 %. But I’m lazy! Here’s that information deficiency again.

My forecast: 6 %, when the community forecast was 20 %.

The question was whether Claudine Gay would resign before January 1, 2024. I opened way below the community, and even as time went on and the community updated downwards, I did so more aggressively, and ended up at 1 % when the community was at 4 %.

I was lucky. Claudine Gay did not resign before January 1, 2024 – she resigned on January 2.

What confuses me about this question is that a part of me still thinks my forecast was more correct than the community. Yet – had she resigned just a couple of days earlier, I would have lost a lot of points on this one. Did we actually nearly witness a 1 % probability event in a forecasting tournament, or was the community forecast more accurate?

I honestly don’t know, but it seems important for my future performance that I figure it out.