Metrics, codification and objectivity

The rise of objectivity and the decline of discretion and subjectivity

Last week, the long-time captain of the South African women’s cricket team, Dane Van Niekerk, was dropped from their T20 World Cup squad because she failed to clear just one of the mandated fitness tests. Specifically, Van Niekerk failed to run 2km within the demanded standard of 9 minutes 30 seconds, missing out by 18 seconds in a personal best time of 9 minutes 48 seconds.

What is interesting about this episode is not whether the fitness standards are appropriate but the method by which these standards are applied, which is through objective and quantitative metrics that are free from any human discretion or subjectivity. In fact, the South African selectors made precisely this argument for why they could not make an exception for Van Niekerk even though she is, by any estimate, one of South Africa’s best cricketers. As the Cricket South Africa (CSA) Chief Medical Officer put it,

"What message would that send to the other players who all met the minimum standard?" CSA chief medical officer Dr Shuaib Manjra added regarding exemptions to the rules. “For years and years, we've battled and struggled to give women's cricket in the country professional status. Let's adopt a professional standard.”

The episode reflects a fundamental trend in the modern world - the displacement of discretionary, subjective and often qualitative control by objective, quantitative control free of any human discretion. Theodore Porter has written an excellent book on the rise of objectivity in the modern era, and the trend has, if anything, accelerated in recent times. The primary driver of this trend towards objectivity is to ensure that the control process appears unbiased. Whether the process is effective or reaches the truth of the matter is secondary. In Porter’s words,

Mechanical objectivity has been a favorite of positivist philosophers, and it has a powerful appeal to the wider public. It implies personal restraint. It means following the rules. Rules are a check on subjectivity: they should make it impossible for personal biases or preferences to affect the outcome of an investigation. Following rules may or may not be a good strategy for seeking truth. But it is a poor rhetorician who dwells on the difference. Better to speak grandly of a rigorous method, enforced by disciplinary peers, canceling the biases of the knower and leading ineluctably to valid conclusions.

Whether the process is truly unbiased has, of course, very little to do with its quantitative nature, as bias can just as easily be baked into a process of quantitative control as it can be within a process of qualitative control. Moreover, objective and quantitative control also removes personal accountability from the system. By removing the subjective element from the process which disqualified Van Niekerk, the CSA also escapes being held accountable if the results are not satisfactory.

This is the primary reason why objective, quantitative control is preferred in the modern era. By removing themselves from the direct act of control and decision-making, controllers avoid being held directly responsible for the system's failures. Even if a system ends up being dysfunctional, all that typically occurs is an official review of the system by a committee rather than those deemed to be directly responsible for the decisions that led to failure being sacked (which is what would typically happen in the prior discretionary regime).

What is supposedly gained in the transparency of control causes a more than corresponding loss in accountability of those in positions of control and power. James Scott illustrates this phenomenon well in his book ‘Two Cheers for Anarchism’:

When they are successful politically, the techniques of the SAT, cost-benefit analysis, and, for that matter, the Intelligence Quotient appear as solid, objective, and unquestionable as numbers for blood pressure, thermometer readings, cholesterol levels, and red blood cell counts. The readings are perfectly impersonal and, so far as their interpretation is concerned, “the doctor knows best.” They seem to eliminate the capricious human element in decisions. Indeed, once the techniques with their deeply embedded and highly political assumptions are firmly in place, they do limit the discretion of officials. Charged with bias, the official can claim, with some truth, that “I am just cranking the handle”—of a nonpolitical decision-making machine. The vital protective cover such antipolitics machines provide helps explain why their validity is of less concern than their standardization, precision, and impartiality.

Strictly quantitative metric-driven control also suffers from an unavoidable problem that I have discussed earlier - over time, the control regime becomes more complex and opaque as Goodhart’s Law takes hold, and those being controlled learn to game the system. Adding human discretion to make the control “fuzzy” would solve this problem. However, the protection that a purely objective regime provides to those wielding power means that this is never an option. For example, the logical “fix” to the CSA’s fitness standards to come up with a system where failure on one metric is not fatal is to come up with a more complex system which evaluates the fitness of the player across metrics, possibly combining these metrics into a more elaborate metric. As I explained in a previous essay, Amazon tried precisely this approach within their organisation only to realise that the cost paid in increased complexity and opacity is not worth it. However, this is precisely what happens in every quantitative, codified system of control. Every edge case and every way in which the system is gamed is dealt with by making the process more and more complex until the system no longer functions at all.

This process of “improvement” is a common thread in the inexorable increase in complexity and dysfunctionality within all our legible, codified systems - software, bureaucracy, government, or large corporate organisations. At this point, following the rules is itself the cause of the dysfunctional behaviour of the system and keeps the system in a state of permanent mediocrity.