Metrics and their limits

Stefan Collini summarises the most important weaknesses of using quantifiable data - metrics - as described in Jerry Z Muller's book The Tyranny of Metrics: 

Misdescription of purpose is fundamental: in the attempt to find outcomes that are measurable, complex characterisations of purpose are replaced by quantifiable results. ‘Goal displacement’ is also a major problem: where a metric is used to judge performance, energy will be diverted to trying to improve the scores at the expense of the activities for which the metric is supposed to be a proxy. ... But there are less obvious effects too, such as discouraging risk-taking, undervaluing co-operation and common purpose, and the degradation of the experience of work. Kept alive for thirty days, Stefan Collini, 'London Review of Books' (subscription), 8 November

I see the need to rely on metrics as a product of big government, itself a function of the highly complex, highly aggregated societies in which we live. What is termed multiculturalism doesn't help either. Lacking a common history and common values, our needs are less readily appreciated by any large centralised body. So our politicians have to rely on quantifiable data. In this, our national governments have more in common with other big organisations (corporations, trade unions, other governments) than with the people it's supposed to represent. All these institutions have two main things in common: first, that their over-arching objective is that of self-perpetuation, about which I've writen before. Second, that they rely heavily on easily quantifiable data.

Things were simpler in earlier days:

[W]e prefer to take our chance of cholera and the rest than be bullied into health. A leading article from the [London] Times, 1 August 1854, in response to government measures to provide basic sanitation.

Problems themselves were more obvious; the causes of problems could be more readily identified, and so could solutions to some of them. Governments were largely successful in their policy interventions on behalf of the disadvantaged: they instituted basic health and education for their own populations. They provided other public goods, such as law and order, and sanitation. And they did so with great success and sometimes, as the quote shows, against strident opposition.

In our industrial societies, with their large, complex economies, government bodies have far more complicated tasks, but they still believe that the best way solving problems is to look for causes and try to treat those. And they still believe that they are best placed to perform these tasks. Government has enlarged its role and largely supplanted families, extended families and local people in supplying a range of welfare services to those who need them. Increasingly government is turning to numerical indicators to manage its resource allocation.

But this use of indicators is relatively recent, unsystematic and unsophisticated, as Messrs Collini and Muller relate. Few indicators are targeted explicitly for a sustained period: the targeted range of inflation in, for instance, the UK, is a rare (and not especially helpful) exception. Other indicators, such as the size of hospital waiting lists, don’t measure what matters to people, or are prone to manipulation. Even when numerical goals are clear and meaningful they are rarely costed, they are almost always too narrow and short term, and they are largely driven by existing institutional structures. Those broad targets that are targeted with some degree of consistency tend to be economic aggregates, such as the inflation rate, or the rate of growth of Gross Domestic Product — which appears to be the de facto measure of success of rich and poor countries alike.

But GDP’s shortcomings as a single indicator of the health of an economy are well known: amongst other failings, GDP does not take into account changes in the quality of the environment, or the distribution of income, it ignores human capital (the education and skills that are embodied in the work force) and leisure time, and it ignores such social problems as crime and homelessness.

Much of what matters most to us - family, relationships, connection with nature, meaningful work etc - is impossible to quantify. Unfortunately, the underlying assumption, as Mr Collini tells us, is that:

...the right structure of incentives and penalties will ultimately improve the bottom line of any business. Organisations whose rationale is not the maximisation of profit, such as schools, hospitals, universities, museums and so on are a challenge to this idea because their ‘product’ does not take financial form. So some equivalent has to be found – the numbers passing certain exams or being treated within certain times – on the basis of which quantitative targets can be set and performance rewarded or punished accordingly.

What's to be done? My suggestion is that we target broad, long-term goals that are meaningful to ordinary people. They should be quantified by indicators or targets that are inextricably linked to people's well-being. Achievement of these goals must be exactly congruent with achievement of society's wishes, as articulated by democratic governments in consultation with its citizens. And the achievement of these goals should be contracted out in a way that lets market forces - the most efficient way yet devised of allocating our scarce resources - play their role in maximising efficiency.


I suggest that we issue Social Policy Bonds to target such goals as: stabilising the climate, achieving universal literacy, improving citizens' health, and preventing nuclear conflict. By issuing the bonds, a government, or coalition of governments and others, need not specify how such long-term goals are to be achieved, nor who shall achieve them. Broad national or global goals can be more reliably quantified and are much less costly to monitor than narrow, short-term goals which, amongst other deficiencies, allow problems to be shifted to non-targeted areas or time periods.

What about things that cannot be readily or reliably quantified? Family, mental health, some aspects of the Commons? Perhaps government's role here should be that of laying down minimum standards based on those criteria that can be measured, and then backing off: making sure it doesn't discourage diverse approaches, ensuring that its regulations do not favour the large at the expense of smaller concerns, be they voluntary, non-profit, co-operative or smaller businesses. Neither this approach, nor a Social Policy Bond regime, will come about easily. They would mean government relinquishing some of its power to create bodies, allocate funding, and have a large say in how to solve our social problems. But, in our ever more complex world, it's not particularly good at these things, and a bond regime would still need government to articulate society's wishes and to raise the revenue for their achievement. These necessary functions, democratic governments are actually quite good at. If all this sounds far fetched the question to ask is: in a world of increasing social and environmental complexity, facing urgent, huge challenges (climate change, nuclear proliferation etc), and rising political impotence and extremism... what is the alternative?