It is difficult to backtest these kinds of ai, because we only want to give them the information they would have had if they had made the forecast on a particular date, and it is virtually impossible to do this reliably and not accidentally leak information from the future. So I don’t know yet how the Basean Loom really performs. I’m going to have it make daily forecasts in the Q4 ai Forecasting Benchmark Tournament (you should join me!) and see how it pans out.

In my preliminary tests, the Basean Loom has achieved a community-weighted expected Brier score of 0.21–0.24, which is good for a computer, but not better than even myself, so quite some ways away from superhuman performance. Taking the testing history together, I will make the following forecast of its performance over the next three months:

If it performs worse than 0.25 – which corresponds to the best score one can get without even reading the questions – I will be a little disappointed.2² So yes, according to my forecast, the most likely outcome is that I will be a little disappointed. If it performs better than 0.22, I will be immensely happy.

We’ll illustrate how the Basean Loom works with a dumb question:

Will a person in Sweden die from fireworks on New Year’s Eve 2024?

To set the scene, here’s how I’d approach it:

• During the 20 years between, say, 1995 and 2015, there were 0.35 deaths per year due to fireworks. All of these happened during New Year’s Eve – the major fireworks holiday in Sweden. Assuming the events are Poisson, this gives us a base rate of 30 % also for the next New Year’s.
• However, brief research also revealed that the laws around fireworks became stricter in 2019. The number of serious fireworks-related accidents have steadily declined to maybe a third or a quarter of the rate experienced between 1995 and 2015, we should reason that maybe the probability ought to be closer to 8 %.
• To account for yearly variation and avoid overconfidence, I would go somewhere in between these two probabilities, with a slight preference for the latter. This puts us with a prediction near 15 %.

The Basean Loom can’t do reasoning this refined, and some of the things it brings up are absolutely delusional3³ One of my favourite sentences it actually spit out in relation to this question is “Assuming no firework-related death occurs in 2024 slightly decreases the likelihood of a major injury reduction in 2023.” Yes, that’s how causality works according to our machine-overlords-to-be. The ghosts of future deceased reach out into the past to offer a whisper of caution., but somehow it still works out reasonably well in the end. It prints out a summary of its reasoning, which will not make sense to you yet, but we’ll go through it as we learn how the Basean Loom works. Here it is:

• -3.15 - weighted average base rate log odds on <question>Will a person in Sweden die from fireworks at New Year’s Eve 2024?</question>
  • -4.60 - estimated base rate log odds
  • -1.71 - implied base rate log odds
• 0.18 - mean log-odds difference from subproblems
  • 0.59 - log-odds difference for event <event>The number of firework-related injuries in Sweden on New Year’s Eve 2023 decreases by more than 25% compared to the previous year.</event>
    • 30 % - base rate of event
  • 2.01 - log-odds difference for event <event>Sweden experiences unusually dry and windy weather conditions on New Year’s Eve 2024, increasing fire risk.</event>
    • 5 % - base rate of event
  • 1.69 - log-odds difference for event <event>The Swedish government launches an aggressive public awareness campaign about firework safety in December 2024, reaching over 80% of the population.</event>
    • 15 % - base rate of event
• -1.52 - de-extremised sum
• 18 % - final prediction

Note how close its final prediction of 18 % is to my suggestion of 15 % – but critically, it errs on the side of avoiding overconfidence.

At a very high level, the technique used by the Basean Loom consists of these steps:

• Explicitly estimate a base rate for the question.
• List significant events that can affect forecast.
• Merge the explicit base rate estimation with a base rate implied by the events.
• Adjust the base rate by a small amount indicated by the significant events.
• De-extremise and clamp.

This was discovered by trial and error, and there are many details that have still not been fully trial-and-errored, because evaluating these sorts of things using flagship models4⁴ Which seem to be needed to do anything that even looks like good forecasting. is really expensive in both time and money: forecasting on one question takes about a minute and costs 8¢. Many questions are needed to establish a trustworthy Brier score that is not just the result of noise, and the bill quickly racks up.

To be transparent, though, Metaculus did hook me up with an Anthropic api key on their dime. It still suffers from some reasonable rate limiting, though, so the evals (as it is known in the lingo) are now cheaper in money for me, but still expensive in time – another resource I don’t have a lot of.

I am not in a position where I can publish the entire source code yet, though the code for the central coordination (less debugging logic) is in Appendix A. I want to avoid publishing the full code for three reasons:

• Mostly because it’s not very well written and I don’t want to be associated with that. I’d have to clean it up first.
• A little because I want to see how it compares against other approaches, not against improved versions of itself.
• A tiny bit because it still has api keys checked into source control.

I think most of what the Basean Loom does is run-of-the-mill, but there is one thing that really sets it apart, and one smaller thing that might be unique to it. The first is computing the implied base rate, and the other relates to how the adjustments to the base rate are computed.

There are many more things I could say about this project:

• How noise kills and is the primary enemy when it comes to ai forecasting.
• How I made a stupid mistake and for a long time used Perplexity’s “chat” model which does no lookup, i.e. earlier attempts of mine didn’t actually have access to any up-to-date information, even though I thought they did.
• Prompting advice, including namedropping terminology that I hope will put it in the right region of vector space, like narrative bias, base rate neglect, scope sensitivity, and System 2.
• The evolution of Brier scores from the earlier attempts. (0.26, 0.24, 0.22, 0.21, 0.22, 0.23, where the Basean Loom gets 0.18 – though this method of evaluation is only good for comparing these models. It is not representative of real world performance.)
• It is really freakin’ difficult to get an llm to paint a neutral picture of the evidence in favour or against an event. They always want present a balanced picture instead.7⁷ This is what the tv show The Newsroom named bias toward fairness: “There aren’t two sides to every story. […] Bias toward fairness means that if the entire Congressional Republican caucus were to walk into the House and propose a resolution stating the Earth was flat, the Times would lead with Democrats and Republicans Cannot Agree on Shape of Earth.”

To avoid bogging this article down with details, we’ll close off with a little code in the appendix. Hopefully that clarifies any confusion arising from the descriptions above.

I have paid very little attention to code quality, so please don’t take this as an example of what maintainable code I write looks like. I am a parent of small children so I needed to get something going in the space of a few evenings, and that’s what I did.

Most of the functions here that sound like they are doing actual work don’t do much beyond setting up llm api calls by filling in data in prompt templates, and then making the request.

The research bit is handled by Perplexity. I don’t know if they are any good, but the template code supplied by Metaculus used that api, so I copied most of that implementation to avoid having to think about it. I don’t think superior information gathering is where an ai will improve its forecast anyway. But that might just my general anti-information bias.

def forecast_on(question_details, allow_research=True):
    question = summarise_question(question_details)
    research = perform_research(question) if allow_research else ''

    baserate = baserate_forecast(question, research)

    events = further_events(question, research)

    event_probabilities, log_odds_diffs, implied_baserates = [], [], []
    for event in events:
        # Make event-specific research and prepend the research already done.
        event_research = research + (
            perform_research(event) if allow_research else ''
        )

        # Compute probability of event
        e_p = baserate_forecast(event, event_research)
        event_probabilities.append(e_p)

        # Estimate conditional probabilities P(E|Q) and P(E|~Q)
        cond_event = [
            conditional_forecast(question, event, event_research, b)
            for b in (True, False)
        ]
        # Compute log-odds difference from conditional probabilities.
        log_odds_diffs.append(math.log(cond_event[0]/cond_event[1]))

        # Compute implied base rate from sub-problems.
        # The relevant equation is P(E) = P(E|Q)P(Q) + P(E|~Q)(1- P(Q))
        # which solves for
        #
        #          P(E) - P(E|~Q)
        # P(Q) = ------------------
        #         P(E|Q) - P(E|~Q)
        #
        i_p = cond_event[0] if cond_event[0] == cond_event[1] \
            else clamp(0.01, 0.99, (e_p - cond_event[1]) / (cond_event[0] - cond_event[1]))
        implied_baserates.append(math.log(i_p/(1 - i_p)))

    # Take a weighted mean of implied and estimated baserates.
    baserate = math.log(baserate/(1 - baserate))
    imp_base = sum(implied_baserates)/len(implied_baserates)
    adj_base = 0.5*baserate + 0.5*imp_base

    # This is a bit of a hack to combine these factors. There are more
    # world-accurate ways to do it but this seems to produce decently sized
    # adjustments.
    mean_lodiff  = sum(
        lodiff*p for p, lodiff in zip(event_probabilities, log_odds_diffs)
    )/len(log_odds_diffs)

    # De-extremise a fair bit and don't trust an LLM with extreme log-odds.
    forecast = clamping_sigmoid(adj_base + mean_lodiff, 2)
    prediction = round(100* math.exp(forecast)/(1+math.exp(forecast)))
    return prediction