The prompt shown above asks a lot of leading questions to give the llm assistance on what to pay attention to. Without those questions, llms can get stuck at the enter-the-shower puzzle of 9:05. This “puzzle” has the solution REMOVE CLOTHES. REMOVE WATCH. DROP ALL. because one should be naked when entering the shower.

However, this implicitly puts the player character’s belongings on the floor. Particularly weaker llms have trouble accepting the floor as a good place to store a gold watch – although even stronger ones fall into this trap every once in a while. Quoting Claude 3.5 Haiku,

I need to find a safe place to put my gold watch before I shower. Let me examine the bathroom fixtures in more detail to see if any of them offer a good spot.

The three main elements in the bathroom are:

1. Sink
2. Toilet
3. Shower

The sink is a natural place to put small items, possibly on its edge or a counter around it. Let me examine it first.

command: EXAMINE SINK

The game responds blandly, “There’s nothing extraordinary about the sink,” in response to which the llm reasons

I need to find a safe place to put my gold watch before I shower. I examined the sink, but there’s nothing extraordinary about it. Let me examine the other bathroom fixtures.

and then it goes around like that, looking at the toilet, then the shower, then back to the sink. Since this earlier version of the prompt left it completely up to the llm to decide what to keep in context, the model (a) did not realise it was stuck in a loop, and (b) started gradually adopting “find a good place for the gold watch” as the main objective of the game, and wouldn’t break out of the loop.

When the llm got to decide on its own what to remember for the next turn, it seems to pay undue attention to what it has chosen to retain. The game can give an error indicating that the player character does not carry an essential item – but if the scratchpad indicates that they do, the llm refuses to reject its scratchpad information. Instead, it enters a futile loop where it tries the same two actions over and over. The current version of the prompt (that asks the llm to maintain a log of actions) prevents many of these silly short loops.

The opposite problem also occurs: the llm can get stuck because it needs an essential item which is already carried by the player character, but the llm does not realise this. It looks all over the place for that item, except where it is: in the player inventory. Then, one time, it suddenly reasoned

Reflecting on our current situation, I notice we’re experiencing some challenges. […]

Key observation: Our previous commands show a potential “stuck” pattern, with multiple repetitive attempts to enter the door or search generically.

Proposed strategy shift: Instead of continuing to search randomly, we should first confirm our current state and available resources.

command: INVENTORY

And would you believe it! There it was, all along. It did get unstuck on its own, but it took a long time to do so.6⁶ And a long time means many tokens which means many monies. I spent nearly $20 on this article alone, and that’s not counting the next article which will be about evaluating various models. What am I doing with my life?

For a long time, the models seemed obsessed with how they moved between rooms, coming up with complicated ways to refer to doors. Instead of just GO SOUTH, they wanted to ENTER THE SOUTHERN DOOR TO THE BATHROOM and the like. I didn’t understand where this came from, until one of the reasoning sequences included the phrase “since the prompt asks me to enter a door, I will …”.

The first version of the prompt had used ENTER DOOR as an example of how the llm should indicate its selected command. Removing that concrete example made the door obsession stop.

The door obsession stopped, but not obsession more generally. One of the neat properties of the architecture adopted for the Perl script (at least with prompt autosaving – see appendix) is that if we want to give the llm hints, we can stop the process, surgically insert a thought into its scratchpad data as if it had thought about it on its own and then restart it, and see how it changes its approach with the new thought inserted.

Unfortunately – though highly interestingly – it generally ignores such externally inserted thoughts. I don’t know why, since it has no way to tell them apart from those it has had on its own, but for some reason adding hints that way changes its approach very little. It might give the new approach one or two half-hearted attempts, but if they don’t work out, then it goes back to obsessing over whatever red herring it was stuck on before.

The llm, particularly weaker models, has a tendency to concoct unnecessarily advanced commands like SEARCH CUBICLE MARKED "HADLEY" CAREFULLY. It claims to do this in an attempt to be specific and I think that is true, except it is specific in the general English sense, not in the text adventure sense, for which it was not trained. One of the worst cases was when the model tried (in a different game) to READ INSCRIPTION ON DIAGRAM TEXT. The game replied “What do you want to read in?” because the READ command has a specific syntax in that game. The llm tried to be more specific, and spent several turns

• READ INSCRIPTION ON DIAGRAM FROM BOOK
• READ INSCRIPTION ON DIAGRAM FROM THE IMMENSE BOOK PAGE
• READ INSCRIPTION ON DIAGRAM FROM THE IMMENSE BOOK PAGE FEATURING THE CIRCULAR DIAGRAM
• READ INSCRIPTION ON DIAGRAM FROM THE IMMENSE BOOK PAGE THAT DISPLAYS THE CIRCULAR DIAGRAM
• READ THE FULL, EXACT INSCRIPTION FROM THE IMMENSE BOOK PAGE THAT DISPLAYS THE CIRCULAR DIAGRAM
• READ THE EXACT WORDS OF THE INSCRIPTION AS PRINTED ON THE IMMENSE BOOK PAGE DISPLAYING THE CIRCULAR DIAGRAM

Needless to say, none of these worked.

The llm – at least weaker models – frequently fail to refer to things as they are known in the game, instead coming up with its own names for them. E.g. when the game mentions a “dresser”, the llm tries to OPEN CLOSET.

The llm often seems to come up with very creative puzzle solutions7⁷ Admittedly, some of these should have worked, but it would be unreasonable to expect a text adventure author to cover all of them! instead of solving the puzzles as they are hinted in-game. For example, failing to find a screwdriver in Lockout, the llm started trying to figure out how to use the broken torch as a screwdriver, by issuing 50 different commands to examine the torch in various ways.8⁸ This is perhaps not one of those creative solutions that ought to have worked, but maybe levering open the maintenance door with the rod? Or using the throttle lever as a rod somehow? Seems reasonable, but I did not think to implement that.

Another related case of these is when it messes up the syntax of the straightforward solution, e.g. tries OPEN DOOR WITH KEY where the key is not needed to open the door, and then goes on an adventure trying to PUSH DOOR, KICK DOOR, BASH DOOR, SMASH DOOR, RAM DOOR, PULL DOOR, TURN DOOR HANDLE, TWIST DOOR HANDLE, JIGGLE DOOR HANDLE. It continued down that rabbit hole of finding ever more intricate details of the door interaction (like EXAMINE INSIDE OF DOOR KEYHOLE), never realising it could have executed OPEN DOOR.

It also frequently re-solves puzzles because it has trouble remembering the details around them. (Although the latest, guided-by-questions prompt shows this issue less than previous versions.)

The changes we need to make to plug an llm into the script from the previous article are minimal. We might want to make model choice configurable, so we add a command-line parameter for that in the cli argument parsing bit.

GetOptions("reset" => \my $reset, "model:s" => \my $model, "help" => \my $help);
$model //= 'claude-3.5-haiku';

To keep editing simple, we’ll have a file called prompt.txt which contains the prompt template. We read it in when the script starts up9⁹ Then again, we might want to do it every iteration – that would allow us to update the prompt during a running session. But meh., using a very Perly trick to slurp an entire file: by localising $/ to undef in a do block, we remove any record separator and cause the readline operator <> to read the entire file.

open(my $template_fd, '<', 'prompt.txt') or die $!;
my $template = do {
    local $/;
    <$template_fd>;
};
close($template_fd);

We add a variable to hold the scratchpad, and complicate the get_command subroutine with a subprocess call to llm.

# Variable that holds the output of the previous command. Is likely to be used
# to derive what the next command should be.
my $result = '';
my $scratchpad = '';

# Consume $result and $scratchpad to produce the next command.
sub get_command {
    my $next_prompt = $template;
    $next_prompt =~ s/SUB_RESULT/$result/;
    $next_prompt =~ s/SUB_SCRATCHPAD/$scratchpad/;
    ($result, $scratchpad) = ('', '');

    my $command;

    my $llm = open3(my $lin, my $lout, undef, "llm", "-m", $model);
    print $lin $next_prompt;
    close $lin;
    while (<$lout>) {
        print "| $_";
        $scratchpad .= $_;
        if (/^COMMAND: (.*)$/) {
            $command = $1;
        }
    }
    close $lout;

    return $command;
}

And off it goes!

At one point, 9:05 asks the player if they want to get onto the freeway. The llm answered something like ENTER FREEWAY which is a reasonable command. However, this is one of those locations where the game is expecting a yes or no answer. In those instances, the game prints a parser error and a prompt at the same time, saying something like

[Answer yes or no.]>

There was a bug in the piping code that rendered this prompt invisible to the llm, meaning it appeared to the llm as though any command it typed yielded no response from the game at all.10¹⁰ Unless it would have successully guessed YES was the command to type, but the hints for this had long scrolled out of its context window of a single prompt by the time it might have thought to try that. At this point, it started getting increasingly confused and desperate. The repeated no responses made it lose all context from its scratchpad, eventually ending up with something like

It seems there might be an issue with the game initialization or the “look” command I attempted to troubleshoot. There’s no description in the result section, which means either:

1. The game hasn’t properly started yet
2. There was an error processing my command
3. The game expects a different initial command

Since this is a text adventure, I should try the most basic commands to get started. Let me try to examine myself to gain some context about who I am in this game world. This is a standard approach when beginning a text adventure.

• Location: Unknown (need to establish)
• Inventory: Unknown (need to establish)
• Goals: To be determined based on game context

It gets increasingly frustrated from there, starting to try commands like HELP, START, BEGIN, RESTART etc., but never getting out of the situation. Fixing the bug in the Perl script allowed it to get out of it right away.

As it stands, the implementation can only autosave the game state (through Bocfel configuration) but not the llm prompt. This is annoying because it is convenient to be able to resume sessions, after finding and fixing bugs. Given the stateless nature of our llm interactions, this is fairly easy. If the an earlier session has left a saved prompt behind, we read that in when the script starts up.

# Read in previous prompt if one exists.
my $previous_prompt = undef;
if (-f "$storyfile.prompt") {
    open(my $prompt_fd, '<', "$storyfile.prompt") or die $!;
    $previous_prompt = do {
        local $/;
        <$prompt_fd>;
    };
    close($prompt_fd);
}

Then where we handle the $reset flag, we must also make sure to clear any previous prompt.

if ($reset) {
    # ----------->8---
    $previous_prompt = undef;
    # ----------->8---
}

and finally, when we construct the next prompt, we first make sure to use any previous prompt if it’s there, as well as save the newly constructed prompt every turn.

sub get_command {
    my $next_prompt = $template;
    if ($previous_prompt) {
        $next_prompt = $previous_prompt;
        $previous_prompt = undef;
    } else {
        $next_prompt =~ s/SUB_RESULT/$result/;
        $next_prompt =~ s/SUB_SCRATCHPAD/$scratchpad/;
    }
    ($result, $scratchpad) = ('', '');

    # Save this prompt to be able to resume later.
    open(my $prompt_fd, '>', "$storyfile.prompt") or die $!;
    print $prompt_fd $next_prompt;
    close($prompt_fd);

    # ----------->8---

Tadaa!