The lifecycle of a form: from simple and effective to complex and dysfunctional
how all codified processes become complex and dysfunctional over time
In a recent article, David Bier showed how the length of a US green card application has increased dramatically over the last two decades.
This phenomenon is not restricted to immigration or even to state bureaucracies. For example, doctors today fill out an increasing number of forms for every task that they perform. Richard Cook describes a typical experience1:
At one point I started counting the checkboxes and menu drop-downs needed to complete an ‘ordinary’ formal anesthetic record. I gave up when I came to 50.
So how did we get here? We got here by starting with a small, simple form and then “improving” it over time as failures are identified. Every failure prompts us to add more fields to the form to prevent that failure in the future. What could be more natural than that? However, every solution to a failure makes the form more complex and creates the next problem.
Forms start out as simple, but they never remain simple. The temptation to add fields is too strong, especially when things go wrong. And even when the process is widely accepted to be too complex, removing and simplifying the process is impossible because every addition has a valid reason, a valid and real failure that it is trying to prevent. Any attempt at subtraction is accused of being irresponsible and risking disaster.
Soon the form is so complex and time-consuming that one of two things happens.
If the consequences of not filling the form correctly are significant, the user is forced to use professional help at significant cost, e.g. lawyers to fill out green card applications and tax accountants/tax software to fill out tax returns. Soon, these professionals have a vested interest in maintaining this complexity and even increasing it over time.
If, however, the consequences of not filling the form correctly are trivial, the form becomes a tick-box exercise that no one takes seriously. Paradoxically, computerisation and automation make forms more complex, but they also help in making form-filling a pointless exercise. Richard Cook again gives us an example of how, even though the form for an anaesthetic record is long and cumbersome, it can be filled without giving it much thought.
Thankfully, the computer has a templating mechanism that allows me to specify all the checkboxes and menu selections so that I can populate most of these entries in the database record for a given anesthetic by pressing a single button. Yet herein lies a paradox: most of these things can be done with the push of a single button. I need not read or select the specific items to create the formal record—indeed it would be almost impossible to do so in the time available. The machinery of compliance and attestation is so awkward and large that it has spawned its own remedy—its own ‘workaround.’ A review of the formal record will show that I have done all the steps but someone watching me would see me choose one item from a dropdown menu and proceed.
Here, the system only continues to function because the codified process is effectively ignored. If the official process was actually followed to the letter, the system would grind to a halt. If such workarounds are not built into the system, the transition from paper forms to computerised forms can also bring the system to a halt, as I described in a previous essay:
Bureaucrats have always been sticklers for the process but in the era of paper forms, they had the option of fudging and ignoring the strict requirements of the process in an emergency or if they simply felt that some part of the process was not relevant to the case in question. But this is no longer possible when the process is enforced by data validation. An employee on the ground can fudge a paper form, but there is no getting around a ‘required’ field on an online form.
Any formal order depends on these informal hacks and fudges to function. In James Scott’s words: “The more highly planned, regulated, and formal a social or economic order is, the more likely it is to be parasitic on informal processes that the formal scheme does not recognize and without which it could not continue to exist, informal processes that the formal order cannot alone create and maintain.”
As control becomes more all-encompassing in the modern data-driven computerised era, these fudges, hacks and informal processes get squeezed out. The system becomes even less nimble and adaptable than it was in the era of paper bureaucracy.
But there is a deeper problem that arises from this combination of an increasingly complex form and its workarounds. The codified process, which started out as an attempt to transform an illegible and opaque process into a legible and transparent process, becomes opaque and illegible again. Richard Cook again describes how “the content of many patients’ records are so byzantine that clinicians often have to resort to actually talking to the patients in order to figure out what has been happening with them”.
What I have described above is not unique to forms. It is the lifecycle of every codified process, whether it is the lifecycle of Google search, the Soviet economic experiment, or the evolution of the NHS. All codified processes - software, laws, bureaucratic procedures and checklists - start out as simple and effective but eventually become complex, opaque and dysfunctional.
in the foreword to Sidney Dekker’s ‘Compliance Capitalism’