To put you in the right mindspace, imagine a human three-year old, that grows to become five years, nine years, sixteen years, and eventually 25 years. All along, they mature and their body undergoes changes. What causes these changes in a human? Not a trick question. Most people would say it’s an intrinsic drift toward the more mature. Humans are pre-programmed to grow and mature a particular way.

This was how a lot of people looked at evolution in the time of Darwin. The evolution of species was an intrinsic drift in some direction. Species were created – much like Pokémon – pre-programmed to evolve a particular way.

Then Darwin came around and said, no, you’ve got it all wrong. There are four elements that make up the evolution we observe:

1. Species have large numbers of offspring, who inherit many of the traits of their parents;
2. Except that the traits of the offspring are subtly different in all directions;
3. Natural selection eliminates more of the individuals with bad changes; and
4. This process has had a very long time to work its course.

Darwin was saying that the apparent drift toward the more adapted is entirely due to compounding selection bias.3³ Note that Darwin may not have been entirely correct in this, but correct in broad enough strokes to lay the foundation for the field even today. He did this even without knowing about the gene! He never postulated a mechanism for inheritance, only offered the observation that a calf is consistently somewhat like its cow.

It’s not hard to see how this can be applied to other areas than biological evolution. Darwin himself was inspired by similar ideas in geology and economics. Richard Gabriel views innovation and product development through this lens. The second world war planes-with-bullet-holes story is another typical example of how the correct solution needs to account for selection bias.

Think of how natural it was for humans before Darwin to consider evolution as intrinsic to species, and how wrong they were. Humans quickly default to intrinsic explanations, that particular behaviours come from within the thing behaving. What are all the things where we default to an intrinsic explanation today, in our daily lives, when selection effects may be dominant?

Aside from selection effects more generally, we can run Darwinianism both backwards and forwards, and in either case, we’ll find that the requirements are relatively strict. Let’s first run it in the same direction as Darwin, and then see what happens when we reverse it.

Another important contribution from Darwin4⁴ At least the way Gould tells it. are his methods of historical inference. Think of a time when you stumbled over Chesterton’s fence at work; the way something is makes no sense to you, but you’re afraid to change it in case it performs an important function you just don’t know about. You’ve asked around, but nobody knew why it is the way it is.

What can you do? These four methods are listed from more concrete and directly applicable to fuzzier and less helpful.

Uniformity

Carefullly observe all the processes going on today, and try to extrapolate their results; into the past and future. Does this produce something that resembles the Chesterton fence you’re looking at? Then maybe that process – or something similar to it – produced the fence and you have more clues as to its purpose.

Example: I inherited a complex set of rules according to which people were allowed to spend a consultant’s hours, but I didn’t understand what problem that solved.

I allowed a few people to use consulting hours freely and carefully observed what happened: it turned out people were offloading more and more of their tough problems onto the consultant. This was a barely perceptible effect, but let’s extrapolate boldly: over the years, offloading tough problems will make people less confident5⁵ Tough problems is how we build expertise, after all., which will make them ready to offload more tough problems, in a reinforcing feedback loop. At some point, it might become a problem big enough to come up with a complex rule set around.

Sequencing

If you take a collection of seemingly unrelated things and arrange them in sequences, does one sequence make them seem like stages in a single process? If yes, does the Chesterton fence you’re looking at fit into that sequence somewhere? Then maybe it’s a byproduct of a stopped process that started with the earlier things in the sequence, and would have produced the later things if it wasn’t interrupted.

Example: I stumbled over an inconsistent configuration file. I didn’t know what to make of it, so I let it be.

As I spent more time at that workplace, I observed how people went about editing configuration files, and learned what their edits looked like at various stages in the process. I eventually figured out the inconsistent configuration file was in an intermediary stage on an uncommon type of edit branch, which was why nobody knew why it was inconsistent in that way.6⁶ What’s more interesting is that the person who left the configuration in an inconsistent state couldn’t even explain themselves why it looked the way it did, because they viewed their editing sequence as one smooth flow, and didn’t think of how things appeared when frozen mid-flow.

Consilience

Aim for a single parsimonious explanation that can account for many seemingly disparate things. That makes it more likely to be true, even in the absence of a large volume of evidence on any single one thing.

Example: I encountered a lot of weird policies and incentives and meetings and culture that seemed out of place in the company at the time.

I eventually figured out that the purpose of the company was not what they said it was – it must have shifted into generating short-term profits. That might sound like an obvious realisation, but management in the company worked really hard to prevent people from realising that7⁷ And there may have been no small element of self-delusion involved..

Discordance

If something seems especially weird when you look at it today, imagine there must have been a good reason for it in the past, and that reason has gone away. If you can guess what that reason was, then you know something about what has changed in the environment.

Example: When I was hired at one of my past employers that had just three programmers aside from myself, there was a lot of process in place for onboarding new programmers. This is an oddity that, if you can guess the explanation, can clue you in to other ways this explanation has shaped the environment.

I didn't figure that puzzle out on my own, but after being there for a while, I learned that they had planned on doubling the number of programmers every six months for two years. However, just before I was hired, an investor withdrew which made that plan unrealistic.8⁸ This was also why they were in the middle of a large project of decoupling software components and improving the deployment experience!

When I was talking to one of the founders of a previous company I worked for, I had to ask what he thought made the company successful. He went silent for a few seconds, then shrugged and suggested, “Luck?”

This was a refreshingly honest contrast to all the people who stand ready to list the steps in their recipes for success, and, of course, the mountains of people who followed that recipe to the letter but still didn’t end up successful.

The system theorists constant mantra is “It’s not intrinsic, it emerges from the interaction between individual and environment.”

But! We can choose whether we view those interactions as plain luck, or whether we – like Darwin – try to dig deeper into what drives those interactions and how they appear to go in some directions but not others.