ACX 2025 Prediction Contest Retrospective
The last few questions of the acx 2025 prediction contest just resolved. This means we can take stock of what went right and what went wrong.
My preferred metric of forecasting performance is the average blind Brier score, since this is what’s popularly used as a reference for comparison across forecasting contexts. In the acx 2025 contest, I achieved a Brier score of 0.21 – the same as in the 2024 edition!
| Tournament | Brier score | Placement |
|---|---|---|
| 2023 Q3 | 0.17 | #31 / 842 |
| 2023 Q4 | 0.14 | #2 / 838 |
| ACX 2024 | 0.21 | #138 / 2087 |
| 2024 Q1 | 0.28 | #552 / 709 |
| 2024 Q2 | 0.20 | #17 / 1002 |
| 2024 Q3 | 0.15 | #16 / 760 |
| 2024 Q4 | 0.23 | #17 / 378 |
| Vox 2025 | 0.12 | #39 / 2983 |
| ACX 2025 | 0.21 | #230 / 4557 |
There were 32 binary questions that resolved with a definitive outcome, and my forecast was closer than the community aggregate forecast to the true outcome in 13 of them. The community aggregate got a very impressive Brier score of 0.17.
(What about the Vox 2025 Future Perfect competition? It shared some questions with acx 2025, and the rest were signficantly easier, resulting in a better Brier score both for me and the community which ended up at something like 0.07.)
Specific questions
The raw per-question Brier score is not a good indicator of which questions anyone did well or badly on, because the high Brier scores may belong to questions that were simply difficult. What we can do is take the difference of our per-question Brier scores to the score of the community, to find questions where we did unusually well or badly.
In my case, I did badly (Brier score difference less than -0.15) on these four questions:
Will the poverty rate in Argentina be lower in the first half of 2025 compared to the second half of 2023?
I forecast on the wrong thing. I forecast on the question “will people’s living standards materially change in the span of a year?” and that rarely happens. But this question was really about “is it possible for the poverty index to move quickly from one reading to the next?” and that both is possible and happened.
Will any rationalist, effective altruist, or AI safety researcher go on the Joe Rogan Experience before 2026?
Here I think I misjudged the industriousness of Joe Rogan. He speaks to an awful lot of people, bringing up the probability that any of them fit a specific description. Maybe there’s also an element of me forgetting that “ai safety researcher” was going to become a more common job title during this year.
Will the US government release additional Epstein documents in 2025?
I have no explanation for why I was so confident they would not.
Will Elon Musk cease to be an advisor to Donald Trump and face public criticism from Donald Trump before 2026?
I knew Trump was volatile. I did not know he was that volatile. Also I assumed he wanted to stay close to the Musk wealth, but I suppose there are other wealths he can stay close to.
On the other hand, I did unusually well (Brier score difference greater than +0.15) on these questions:
On December 31, 2025, will Google, Meta, Amazon, Tesla, or X accept crypto as a payment?
This seemed like a ridiculous idea to me, and I’m not sure why the community was so bullish on it. It’s a very specific set of companies. Even if suddenly 5 % of the largest 100 technology companies start accepting cryptocurrency, there’s still only a 23 % chance any of the listed five companies are in that set. For the community prediction to be correct, you have to assume nearly one in five large technology companies have started accepting cryptocurrency. That does not happen in just a year.
Will the 12-month percentage change in the US Consumer Price Index be lower in November 2025 than it was in November 2024?
This is the “is inflation going down?” question. I reasoned that we have configured our economy to produce a particular level of inflation. There was a shock to the system after 2020, but after that it would slowly return to the level it is calibrated for.
Will at least twice as many deportations by U.S. ICE occur in Fiscal Year 2025 compared with Fiscal Year 2024?
It is very rare for a bureaucracy to simply turn a knob and double its output in a year. Not “it happens every third time” rare, but closer to “it happens every 13 times” rare.
Next up: The 2026 prediction contest!
The acx 2026 prediction contest with a prize pool of $10,000 accepts new entrants for a few more days. I have just finished up all my predictions for it, and in a future email newsletter I will share the forecasts and my reasoning for all questions. I won’t turn it into a full article because it’s so sloppily written and illustrated.
I have also started participating again in the Metaculus Cup, with this Spring 2026 edition. I won’t spend as much time and effort on it as I have in the past, but I’ll try to submit at least one prediction for each question.
Come join me, and happy forecasting!