Some commits are obviously new development, or obviously maintenance. It would be nice if we could classify those through a more primitive – and faster – way, and only ask the llm about the more difficult cases. Here’s the pseudo-code for such an algorithm:

sub labeling {
    my (msg) = @_;

    # Take the word frequencies of the commit message
    # as the features for the fast classifier.
    my $features = bag_of_words($msg);

    # Use the fast classifier when it has seen enough
    # training samples, but skip it sometimes to
    # prevent accidentally learning p=0 or p=1 based
    # on long runs of a single label.
    if ($fast_classifier->{samples} > 50 && rand() < 0.95) {
        my $p = $fast_classifier->predict($features);
        # Use the labels predicted by the fast
        # classifier only if it is very confident.
        return 'maintenance' if $p < 0.08;
        return 'new development' if $p > 0.92;
    }

    # If we have insufficient samples or fall through,
    # use the LLM classifier to get a label.
    my $label = classify($msg);

    # Since we got a real label, use it to train the
    # fast classifier.
    $fast_classifier->refine($label, $features);

    return $label;
}

The question is what object implements the interface we have assumed for the fast classifier. We have called two non-trivial methods on it:

• predict($features): return a probabilistic prediction of the features being labelled as new development.
• refine($label, $features): add the association between the features and the label to the training set of the classifier.

An obvious choice is naïve Bayes. The drawback is that it assumes independence between features, which generally won’t be the case for us. Another alternative is a logistic regression.

A logistic regression is generally calibrated when fully trained, but some things can knock it out of calibration, such as regularisation (which my instance had), or small data (which it also has early on in the process).4⁴ When I checked, this showed up as a 0.2 log-odds bias toward the positive label, and log-odds 10 % more extreme than desired. These are small effects in the grand scheme of things, and we probably shouldn’t care about them, but since we’re having so much fun anyway we might see what we can do.

A common way to improve the calibration of classification models is through Platt scaling, which means fitting another logistic regression on top of the output of the first one to neutralise the bias and noise. Doing this complicates the labelling loop a bit, because the logistic regression will be overfit on its training data, so we need an independent judge (a separate stream of data) to train the scaler. We can implement this by siphoning off, say, 10 % of the training samples to the Platt scaler instead.

sub labeling {
    my ($msg) = @_;

    my $features = bag_of_words($msg);
    my $z = $fast_classifier->score($features);
    my $p = $platt_scaler->predict({_INTERCEPT => 1, z => $z});

    # Use a lower threshold of Platt scaler samples
    # because by the time the scaler has seen this many
    # samples, the classifier has seen even more.
    if ($platt_scaler->{samples} > 10 && rand() < 0.95) {
        return 'maintenance' if $p < 0.08;
        return 'new development' if $p > 0.92;
    }

    # If we have insufficient samples or fall through,
    # use the LLM classifier to get a label.
    my $label = classify($msg);

    # Start by feeding training data only to the fast
    # classifier. Once it has seen a few samples, use
    # every tenth label to train the Platt scaler
    # instead.
    if ($fast_classifier->{samples} > 10 && rand() < 0.1) {
        $platt_scaler->refine($label, {_INTERCEPT => 1, z => $z});
    } else {
        $fast_classifier->refine($label, $features);
    }

    return $label;
}

Both of $fast_classifier and $platt_scaler are instances of that LR type we made! The feature observed by the Platt scaler is the raw log-odds emitted by the fast classifier. (This was why the score() function was exposed separately in that class.)

I thought it was interesting to peek into the fast classifier to see how it discriminated between maintenance and new development commits. It was trained on commit messages as bags-of-words, so it’s basically associating individual words with the probability of that word occurring in the commit message for new development work, and vice versa.7⁷ Though unlike naïve Bayes, the logistic regression accounts for some interdependence between words.

Here are words commonly used in maintenance commits:

• fix (or fixes, fixed)
• use
• move (or refactor)
• cleanup (or remove)
• bug

On the opposite side, these words are often used in commits with new development:

• add (or implement)
• support
• allow
• command (or directive)
• parameter (or option, api)

For my project, I trained the classifier separately on multiple open source repositories, and it learned these same words for all of them. It seems like developers, even across different projects, use many of the same words when indicating something is new development or maintenance. Which also means training jointly on all repositories would have given me a better and more robust classifier, but that would have required rewiring some of the code I had already written and I can’t take the time.

Another optimisation, which probably had a greater effect than the fast classifier in this article, is that all of those rand() calls in the code above are actually calls to a custom function called commit_rand($commit) which turns the commit hash into a uniform probability. This means the randomness is always the same for each commit. This is particularly useful in the loop that samples which commits to label from each repository, because it will always sample the same commits every time it runs.

This, in turn, means it becomes meaningful to cache the labels generated by the expensive classifier. The part of the labelling loop that does this replaces the call to classify($features) with

my $label = $cached_labels{$commit->{hash}};
if (! $label) {
    $label = classify($commit->{msg});
    open(my $cache, '>>', "labels.csv");
    printf $cache "%s,%s\n", $commit->{hash}, $label;
    close($cache);
}

This stores the label for a commit immediately when it has been retrieved. When the script starts up, it reads all the cached labels from before into a dictionary.

my %cached_labels = ();
if (-r "labels.csv") {
    open(my $cache, '<', "labels.csv");
    while (<$cache>) {
        $cached_labels{$1} = $2 if /^(.*),(.*)$/
    }
    close($cache);
}

printf STDERR "Reclaimed %d cached labels.\n",
  scalar(keys %cached_labels);

Couldn’t be simpler!

This was the prompt I used for the llm classification. It is indicative of my typical prompting style, and I performed no separate evaluations of prompt variants since the first one I thought of was good enough.

You will be presented with a git commit message. Your job is to determine whether the commit is new development work or maintenance work.

Things that are considered new development: adding features, adding cli flags, adding api endpoints, removing limitations on usage of features. New development is anything developed because it was requested by a user to help them perform their job to be done.

Things that are considered maintenance: bugfixes, performance improvements, removing features, adding diagnostics, stability improvements, observability improvements, security fixes. Maintenance is any development that needs to happen in support of the features the user need to perform their job to be done.

== START OF COMMIT MESSAGE ==

$msg

== END OF COMMIT MESSAGE ==

It may not be clear whether this is new development or maintenance. Yet you must pick one label, whichever you feel fits best. Respond only with the label, either “new development” or “maintenance”. No other text is allowed.