All systems must be efficient, robust and evolvable

How all complex systems can and must be efficient, robust and evolvable to thrive

Our institutions can no longer get things done. Many key institutions seem unable to either do their job efficiently in normal times, cope with an unanticipated change in their environment or innovate. In other words, they are failing along all three dimensions that matter:

1. Efficiency/Productivity

2. Robustness

3. Evolvability/Ability to innovate

For example, the National Health Service (NHS) in the United Kingdom is hardly a paragon of efficiency even though the British state has been trying to make it more efficient for more than three decades. But despite being inefficient, the NHS neither has the slack to cope with large increases in demand (like Covid) nor does it rank highly in healthcare innovation. In other words, the NHS is neither efficient, robust or evolvable. This is not a problem unique to the NHS. You could say the same for most government institutions across the world.

And the problem isn't even restricted to just the government. Sooner or later, most private institutions fall into the same malaise. Even the trillion-dollar technology companies of today are indistinguishable from any other large bureaucratic organisation and suffer from the same problems. For example, Google's internal promotion process is clearly designed to encourage its employees to chase after the next big thing i.e. maximise innovation. But the end result of the system is inefficient and wasteful theatre that fails even at this limited objective.

This is the unique pathology of our times. Our institutions and our systems are neither efficient, robust or evolvable.

No tradeoff between efficiency, robustness and evolvability

Conventional wisdom would tell you that there are unavoidable tradeoffs between the three dimensions of efficiency, robustness and evolvability. In other words, surely we need to sacrifice exploration (that provides evolvability) and excess capacity (that provides robustness) if we are to be maximally productive and efficient? And to be robust and protect existing system functionality, surely we need to maintain some slack and inefficiency? These almost universally held notions regarding the tradeoffs between efficiency, evolvability and robustness are wrong. Successful systems can and must simultaneously achieve a high level of efficiency, evolvability and robustness. The only way to succeed in any one dimension is to succeed in all three dimensions.

In fact, attempts to target only one of the three dimensions at the expense of the others typically backfires such that even the limited, narrow objective is not achieved. Excess capacity and slack often lead to increased system fragility and increased risk of catastrophic outcomes. Preserving existing functionality without adequate exploration leads to a progressive deterioration in the very functionality that is sought to be preserved. Transformation of the system to focus solely on maximising efficiency paradoxically leads to reduced productivity over time, as the system accumulates kludge after kludge on the initial “optimisation”.

On the other hand, system features such as the presence of components with partially overlapping functionality (known as degeneracy and explained in a previous essay) improve the system's performance across all three dimensions. Much of this is common knowledge in biological systems e.g. the pervasive presence of degeneracy, the complementarity between robustness and plasticity, and the near-optimal efficiency of many of the sub-systems in our body. But these concepts and their relevance are almost completely unknown in other domains.

In future essays, I will analyse why our institutions and systems are so dysfunctional and how we got here. As well, I will explain what efficient, robust and evolvable systems look like, how such systems can be created, and how we can transform our existing dysfunctional systems into functional and healthy systems.