Stacked diffs with Maiao
I have been using the community fork of Maiao for a little while now, and it is excellent. For anyone who prefers working with stacked diffs, but needs to interact with a more traditional forge such as GitHub, try using Maiao for a bit. It just does the right thing in almost every situation. It’s cut my frustration with GitHub at least in half, if not more.
For the small group of readers who are stoked about stacked diffs, that’s probably enough of a pitch. For the rest of the world, let’s back up a little.
The pull request as the unit of change
On traditional forge platforms1 Like GitHub, Codeberg, GitLab, BitBucket, etc., pre-commit2 Pre-commit because it’s the review that happens before the code is merged (“committed”) to the upstream branch, in contrast to e.g. audits which are a kind of post-commit review. code review happens on pull requests (sometimes called merge requests), where each pull request is created from a git branch.
Users that are new to git can then think of a branch as one development effort toward a small goal. They keep adding commits to that branch as long as its tip does not represent the goal state. At the end, when everything looks right and the change is approved, they perform an explicit merge. This means they create a merge commit on the upstream branch that references both the old upstream and the branch they worked on.
This lets beginners work in smaller increments while not dealing with the intricacies of rewriting git history3 Sometimes the beginner is instructed to perform a squash merge instead, which typically means the forge platform creates a single, regular commit on the upstream branch, referencing only the old upstream, and containing all the changes implied by the tip of the development branch. Although it sounds like this rewrites history, it doesn’t really – the old development branch is still exactly the same as it was before. The difference is that the changes implied by it were copied over to the upstream branch without reference to the development branch.; the workflow lowers the barrier to collaborating with others using git, which is nice.
It also means the reviewing interface in a traditional forge doesn’t have to deal with the intricacies of rewriting git history. This means the reviewing ui typically only shows the differences between commits still on the development branch, and not what changed between two versions of “the same” commit, when one of them is no longer on the branch.
To be more concrete, here’s an example of a part of the history graph of a project that uses this workflow.4 The commits are not shown in chronological order. I reordered them (where it was possible to do so without changing the structure of the graph) to make the graph read more clearly.
▲ (upstream) │ │ ▲ (input-validation) │ │ │ │ ▲ (priv-cache) │ │ │ │ ● │ Tighten input validation │ │ │ │ │ ● Rename cache variables │ │ │ │ │ ● Cache user permissions ├───╯ │ ●◀──╮ │ Merge flaky-test-fix (PR #835) │ ├───╯ │ ● Fix timezone offset instead │ │ │ ● Disable login test │ │ ●◀──┼───╮ Merge log-migration (PR #834) ├───╯ │ ●◀──╮ │ Merge avatar-upload (PR #833) │ │ │ │ ● │ Fix typo in readme │ │ │ │ ● │ Clean up unused imports │ │ │ │ ● │ Add avatar upload ├───╯ │ ●◀──╮ │ Merge parser-null-check (PR #832) │ │ │
We can see a total of 12 commits. Four of them are merge commits on the
upstream branch, and two of those merge commits correspond to branches we can
see the work on (avatar-upload and flaky-test-fix). We can see each of those
two branches had three and two commits of actual work on them, respectively.
Then there are two more branches (input-validation and priv-cache) which are
still open (not merged with upstream). One of them (priv-cache) has two
commits of work, and it’s not based on the upstream branch, but on the final
result of one of the earlier branches (flaky-test-fix), before it got merged.
The commit as the unit of change
We don’ thave to use git that way. We can instead choose to have only one
development branch. I call it something generic like dev. On that branch, each
goal state is represented by one commit. So instead of multiple branches (one
for each goal) with multiple commits on them (steps toward the implementation),
there is just one branch (a generic development branch) with multiple commits on
it (one for each goal).
Fixing up existing commits
With that workflow, if we discover one of the commits don’t represent the goal
state of that change, we edit that commit5 Technically speaking, git
doesn’t let users edit commits. What happens under the hood is the commit is
replaced with a new one. But with a change id in the commit message, we can
still tell that they represent the same atomic change. instead of adding more
commits.6 What if we need to revert to an old commit? Fortunately, git
maintains a reflog with old commits. Those stick around until git decides to
garbage collect dangling refs. Then each atomic change in progress is
implemented in exactly one commit, all on one development branch. The dev
branch may have many commits, but they represent different atomic changes, not
progress toward one goal state.
This way of working is also known as making stacked diffs, and I find it easier and faster than having a separate branch for each atomic change. Many experienced software engineers agree.7 I should say I disagree with Gergely Orosz when he says it’s impossible to make a pull request that depends on changes in an earlier, unmerged pull request. It’s possible (and even easy in most forges) but it’s a lot of extra work to juggle the branches and merge targets. Fundamentally, there’s no deep incompatibility between pull requests and stacked diffs. The reason people prefer tooling made for stacked diffs is that there’s less paperwork needed to satisfy git when working with those tools.
Merging stacked diffs
The way stacked diffs are merged is ideally by copying them over to the upstream branch, either through cherry-picking, or by choosing the rebase and fast-forward merge type in the forge ui. This produces a history of the upstream branch that very clearly lists each atomic change in the order it occurred, without the false steps and mistakes along the way.
Beginners may think this is dishonest. The the point is not to hide one’s mistakes, but to present a history that another developer can read and follow along with three years into the future, when they want to understand how the project ended up in the state it is in. For that purpose, only what eventually went into the upstream branch matters.
Revisiting the history graph
The example history graph from above would, under a stacked diffs-type workflow, look more like this:
▲ (upstream) │ │ ▲ (dev) │ │ │ ● Tighten input validation │ │ │ ● Cache user permissions │ │ ●◁┄┄┄◌ Fix timezone offset in login test (PR #835) ├───▶┤ ● │ Migrate to structured logging (PR #834) │ │ ●◁┄┄┄◌ Allow users to upload avatars (PR #833) ├───▶┤ ● │ Check for null in order parser (PR #832) │ │
Two commits on the upstream branch (null check and timezone offset) came from
other people. In between them, we updated our long-lived dev branch to reflect
the latest upstream, and then copied the finished avatar commit over to the
upstream. The same process happened for the flaky test fix a little later. We
worked on both user permission caching and input validation in the same dev
branch instead of making two separate, new branches. Once those commits are
approved and have passed through ci, they will get copied to the upstream
branch, like the ones before them.
All of these commits needed amendment just as many times as in the previous graph, only we do it by replacing the commits instead of adding more commits.
This is what people mean when they talk about how “one atomic change is one commit, and one commit is one atomic change”.
The drawback of stacked diffs
There is one unfortunate consequence of the stacked diffs workflow: when all commits are in one branch, they look like they depend on each other in the order they exist on the branch, even if they actually contain independent changes. When all commits are on one branch, there’s no way to see which commits are independent without (a) reading the code, and (b) trying to cherry-pick one of the changes.8 Both are needed, because cherry-picking or rebasing can conflict in git even though changes are independent (when they happen to appear close to each other in the source code), and dependent changes can generate a clean cherry-pick as far as git is concerned, even if it breaks the code (when the changes happen in different locations in the code, but one of them assumes the other has happened already).
If it’s important to document to ourselves that we have two parallel workstreams
that are indepedent, we can use two local generic development branches (I call
mine dev and, imaginatively, dev2). Each branch is still treated as a stack
of diffs, i.e. each commit represents one atomic change, but now it’s abundantly
clear that commits cannot depend on each other across the two branches.
Stacked diffs are annoying with pull requests
At this point, you can probably guess why the pull request workflow of traditional forges is annoying. The pull request assumes we have one branch for each atomic change, but with stacked diffs, we only have the one branch, where we want each commit to become a separate pull request, building on the one before it.
The way I used to handle it was exactly through the tedium you wouldn’t expect:
- I manually created one branch for each commit.
- I pushed all those branches.
- I went to the GitHub web ui and made a pull request for each branch, taking care to set the merge target of each to the branch that came before it.
- When merging, I again had to be careful to merge in the right order.
This is a lot of work.9 Still less work in total than having a branch per atomic change, but just barely! Maiao solves that.
Maiao is automation at its finest
These days, when I feel ready to create pull requests for my commits, I trigger
the git review command. Maiao then automatically does all of the things I did
manually before, better than I could. It fetches the latest upstream, rebases
the current branch onto it, creates disposable branches for each commit, pushes
them, opens up a pull request with the right merge target for each disposable
branch through the GitHub api, and turns them into a native GitHub stack of
pull requests, meaning they get merged in the right order, and can be merged all
at once when they are ready.
If code review or the ci build uncovers things I need to change, I do what I
always did: I replace the commit that needs changing. Then I run git review
again, and Maiao just does the right thing: it rebases onto the latest
upstream again, it moves the disposable branches to the new commits,
force-pushes them, updates the pull request descriptions with any changes to the
commit messages, updates any merge targets that have changed, etc.
Maiao is one command (git review). When run, it makes the GitHub pull request
stack match our local development branch. That’s all it does, and that’s all it
needs to do.
Dealing with multiple branches and unfinished work
In cases where we have multiple local development branches (because the work is
independent and parallel), git review can be run on either or both of them
independently of the other, and Maiao will create one stack for each branch, as
one would expect.
Note also that Maiao doesn’t have to create pull requests for the entire
development branch. Maybe we’re still working on the top few commits. Then we
can create a temporary branch pointing to the topmost of the commits we do want
pull requests for. When we run git review with that branch checked out, Maiao
makes pull requests only out of the commits included in that branch. Then when
we finish those top commits, we can run git review on the development branch,
and Maiao understands that the topmost commits build on the commits that already
have pull requests, so it stacks new pull requests for the new commits on top of
the pull requests that already existed for the prior commits.
All of the above to say Maiao doesn’t care about how you branch locally. A
decent mental model is that it creates and updates pull requests for each of the
commits between head and the master branch so the stack on the forge matches
the local commits. That’s all.
And it’s awesome.