What is wrong with the world today? The short answer is that the world is too complex. One way to look at this problem is to differentiate between accidental and essential complexity. Essential complexity is unavoidable and inherent to the core functionality of the system whereas accidental complexity is unnecessary cruft that has built up over time. When we say that the government or a large corporation is too bureaucratic, we mean that there is too much accidental complexity within the system.
However, this lens is misleading. The complexity in our world today is not so easily separable into essential or accidental. For example, the planning system in Britain is a clear example of a system that is excessively complex and imposes enormous costs on the British economy. As the Economist has argued, the planning system has turned Britain into a vetocracy that is unable to build. So how did we get here? Strangely enough, we got here not by a series of mistakes but by a series of changes to the system, each of which in isolation is logical but which in combination constitutes a dysfunctional mess. The Economist provides a perfect example of this (emphasis mine):
For a development of 350 houses in Staffordshire, a developer had to provide a statement of community involvement, a topographical survey, an archaeological report, an ecology appraisal, a newt survey, a bat survey, a barn owl survey, a geotechnical investigation to determine if the ground was contaminated, a landscape and visual impact assessment, a tree survey, a development framework plan, a transport statement, a design and access statement, a noise assessment, an air quality assessment, a flood risk assessment, a health impact assessment and an education impacts report. These are individually justifiable, yet collectively intolerable.
This is how systems become excessively complex and sclerotic. A series of micro-optimisations leads to macro-hell. Every improvement to the system makes sense at the time. Yet, the collective impact over time is to create an overly complex legal and dysfunctional system where nothing gets done, and everything costs too much.
And this problem is not just restricted to just one country. A recent article showed how the National Environmental Policy Act (NEPA) has undergone the same evolution as the British planning system with “average NEPA review times estimated to be increasing 39 days per year”.
So what is the common explanation that explains this evolution in systems and institutions across the world today? The answer is that this move towards more complexity is built into the very structure of human progress, especially modern progress. The phenomenon can be summarised in a simple principle universally true for all codified systems: Each solution is the source of the next problem1. Each “patch” to the codified systems that we call civilisation solves one problem but lays the ground for more problems in the future. Fred Brooks identified this problem in software development, but the pattern he describes is universal to all codified systems:
All repairs tend to destroy the structure, to increase the entropy and disorder of the system. Less and less effort is spent on fixing original design flaws; more and more is spent on fixing flaws introduced by earlier fixes. As time passes, the system becomes less and less well-ordered.
How does this process start? Paradoxically, this entire process that culminates in crippling complexity starts with a heroic simplification of the system. This is especially true of systems in the modern world, but it was also true of systems in the pre-modern world. For example, agriculture was a radical simplification of the hunter-gatherer world2. Property laws in many countries started out as an externally imposed simplification that replaced a “thicket of common property and mixed forms of tenure”3. Any new software project starts out as a simplified representation of its domain.
Initially, this simplification is often very successful. The canonical example of modern simplification in James Scott’s ‘Seeing Like A State’ is the radical simplification of German forestry and its eventual catastrophic failure. But the initial results were a stunning success4. But the system does not remain simple for long. Over time, the accretion of solutions and the problems they create increases complexity and sooner or later, the system ends up being even more complex than it was when it was initially simplified.
This in a nutshell is the lifecycle of codification and the progress of humanity itself. At this point when the system has become excessively complex, we either get a collapse or a breakthrough that simply makes the sclerotic system and its kludges irrelevant. In the longer arc, the succession of breakthroughs that define human history and long-run progress constitute a ratchet, where every breakthrough increases the stakes of collapse. As Ruth DeFries puts it in ‘The Big Ratchet’:
Ratchet, hatchet, pivot; ratchet, hatchet, pivot. In every cycle, the stakes get higher, as our species expands in numbers and in the extent of its reach across the world. In every cycle, new obstacles emerge. And in every cycle, millennium after millennium, humanity as a whole has muddled through.
So what can we do to sustain progress and avoid collapse? What will not work is a naive system rewrite/reform. The system is simply too complex and fragile to reform without risking collapse. This is why it is a mistake to think that incumbent elites oppose change to protect their own interests. Often, incumbent elites understand the need for change as well as anyone else. However, they also realise that the system is fragile enough that radical change risks collapse. After all, every piece of the complex system was put there for a reason. Even if the end result is dysfunctionality, how does one figure out what can safely be removed? Even if one thinks they understand what one part of the system does, those parts may do more than is apparent. Reforming the system by removing components is like removing Jenga blocks. Collapse is almost inevitable over time. The best example of the risks of reform is the collapse of the Soviet Union. Gorbachev knew that the system needed to be reformed but all that reform achieved was systemic collapse. In fact, this principle is well-known in software and can be summarised as follows: Never attempt a system rewrite.
So what can we do? This is a question that I will answer in much greater detail in future essays, but the brief answer is as follows. If the system is not yet dysfunctional, a commitment to periodic refactoring is essential. Most software developers realise that “a sustained commitment to refactoring can keep a system from subsiding into a big ball of mud”. The same is true of all codified systems - laws, regulations, bureaucratic processes etc. When it is too late to refactor the system (as is the case now), we need to grow new subsystems/services that function in parallel and gradually take over functionality so that the incumbent monolith can wither away.
a phrase introduced by Donald Gause and Gerald Weinberg in their book ‘Are Your Lights On?’
In James Scott’s words, “Cultivation is simplification. Even the most cursory forms of agriculture typically produce a floral landscape that is less diverse than an unmanaged landscape.” (in his book ‘Seeing Like A State’)
James Scott in ‘Seeing like A State’.
In Scott’s words, “Originally , the Norway spruce was seen as a restoration crop that might revive overexploited mixed forests , but the commercial profits from the first rotation were so stunning that there was little effort to return to mixed forests.”