Hidden metrics

Hidden metrics, not to be confused with the hidden variables of quantum mechanics, are one possible way of preventing the gaming that could otherwise result from targeting specific quantitative outcomes.

Let me explain. Say we are targeting female literacy in Pakistan. We issue bonds that will be redeemed when the literacy level of 15-year old girls in Pakistan exceeds 95 percent each year over a ten-year period. Campbell's Law tells us, rightly I think, that:

The more any quantitative social indicator (or even some qualitative indicator) is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor. Source

What metrics could we deploy that measure, accurately and inevitably, that which we want to achieve: namely, near-universal literacy for 15-year old Pakistani girls?  We could subject every girl of that age to a standard reading test, but we know that academic tests and their results, even in developed countries, are manipulated. Girls could be taught to the test (by, for instance, being given the test in advance and taught to memorise it by rote). Or anyone up the ladder from the administrators of the test to the collators of the results could be bribed to alter the data at any stage.

So instead we could take a random sample of, say, 100 schoolgirls from 100 districts around Pakistan at ten yearly intervals, and test them in non-standard ways for their literacy. We don't reveal which districts, let alone which girls, will be chosen; nor do we release exact details of the test. That makes it much harder to game the outcome.

More complex is the goal of peace. Now, nuclear peace is relatively easy to target: we can issue bonds that will not be redeemed, say, until a thirty-year period has elapsed during which there has been no detonation (accidental or not) of a nuclear weapon that kills more than 100 people. But what about peace more generally? Hot wars are fairly easy to recognise and define and so would be correspondingly easy to deter using a bond regime. But there are more nebulous ways of fomenting conflict between states (Ukraine), or of ratcheting up tension to levels that severly curtail quality of life.  In these instances we could use an array of metrics such as: trans-border movement of weapons, numbers of people killed or fleeing their homes. Even the lack of information could also be an indicator of conflict that could find its way into our calculations. We could also use survey data, such as attitudes about potential enemies, or expressions of fear. Other potential indicators are disruptions to food, water or electricity supplies and other results of damage to infrastructure. 
 
The important point is that in this as in other goals, we need not and indeed should not specify in advance exactly which combination of metrics and indicators will be used to determine whether or not the bonds' redemption terms shall be deemed satisfied. In general, we try, as far as possible, to target metrics that are, or that are inextricably linked to, exactly what we want to achieve. Where that is difficult, we try to prevent gaming by not specifying too far in advance the exact redemption terms of the bonds. The aim, at all times, is for bondholders to comply with the spirit as well as the letter of the goals set by the backers of the bonds.

Irrational health funding

This is irrational, but not surprising:

The [US] National Institutes of Health last month published a startling analysis of how it allocates its funding: in 2010, HIV research received nearly $3.1 billion in funding, while a deadly lung disease that has more than six times the health toll in the United States got only $118 million. Two diseases with a similar health burden, breast cancer and chronic liver disease, received wildly different levels of support: $763 million for the cancer best known for iconic pink ribbon awareness efforts, versus $284 million for a disease commonly caused by alcohol abuse. Autism receives more than five times the funding of eating disorders, but their impacts on health, measured in years of disability and premature death, are quite close. Why the diseases that cause the most harm don’t always get the most research money, Carolyn Johnson, washingpost.com, 17 July

I've inveighed against this sort of bias in government policy for years. In health, as in other policy areas, I am sure that government is well meaning and hard working. But it suffers from its inherently uniform, top-down approach. Government can also be short term in its thinking, reactive rather than proactive, and disdainful of innovation while favouring tried, tested but failed approaches. It has to make its resource allocation decisions on the basis of data that are necessarily incomplete. How can it know in detail the effect that spending on, say, cancer diagnostic machinery will have on the overall health of the nation, as compared with subsidising the cost of nicotine chewing gum?

So, by default, health expenditure is influenced by groups of medical specialists with little incentive or capacity to see improvements in the overall health of a large population as an objective. Funding decisions are also heavily influenced by the public profile of a disease or its victims, rather than on what would best meet the needs of society. It’s also a question of diet, exercise, transport, and culture. Recent research shows, for instance, the beneficial effects on health of green spaces in our cities (see here (pdf) for instance). The way government is structured, with its discrete funding bodies, makes it unlikely that such benefits will influence funding decisions.

We cannot expect a government nor any single organization to identify the huge numbers of variables, with all their time lags and interactions, that influence the nation’s health. We can, though, devise a system that rewards people who explore and implement the most cost-effective health solutions, even when circumstances and knowledge are changing continuously. I have tried to do this with my essay on Health Bonds, which would aim to distribute scarce government funds to where they would do most good, as measured by such indicators as Disability Adjusted Life Years. One small caveat though: I'm assuming that most of us - at least in our most rational moments - favour such a distribution of health resources. But there is a possibility that the current (mis)allocation of resources originates in unvoiced but widely held preferences. Ms Johnson quotes one expert: "...we tend to underfund things where we blame the victim". It's unlikely that the large disparities in health funding outlined above do reflect such deliberate choices. But even if they do, it would be better to be explicit about it.

Targeting surrogate indicators

In the absence of broad, agreed, explicit goals, governments fine it convenient to target surrogate outcomes. Climate change is one example: instead of targeting the adverse effects of climate change, governments target greenhouse gas emissions, which may or may not have something to do with climate change, and they target them in ways that even in their own terms achieve nothing. More generally, instead of targeting anything as meaningful as human well-being, the de facto target of most governments is gross domestic product (or GDP per capita).

It's not just governments. It's big companies too. Instead of targeting anything that's inextricably linked to physical health (Quality Adjust Life Years, for instance), they try to get us to focus on surrogate outcomes. Cholesterol levels are one such surrogate outcome. Statins have been the preferred - by the pharmaceutical companies - vehicle for reducing cholesterol levels but, now that their patents are running out, a new class of drug, PCSK9 inhibitors, is on its way.

Reducing cholesterol levels might bring about a reduction in heart attacks, but it appears to do nothing for overall mortality. Indeed, overall mortality is not really a concern of many medical specialists, nor of big pharma. It's government that should be articulating our broad health goals. But in this, as in climate change and other policy areas, it's instead taking instruction from the wealthy and powerful, at the expense of the people it's supposed to represent.

Greenhouse gas emissions accelerating

Michael Le Page, writes in New Scientist (18 July) about the 'coal renaissance' and in particular about the heavy investments in coal power of poor, fast-growing countries in Asia and Africa. The result is that 'global CO2 emissions are rising faster than ever. And they are likely to continue to grow.' Indeed 'not only are carbon emissions rising, the pace has accelerated since 2000'. A graph (in the print issue) illustrates this: emissions rose at an average rate of 1.3 percent a year from 1970 to 2000, and have been rising at 2.2 percent since then.

I have for years been pointing out the futility of the Kyoto agreement and other apparent attempts to cut greenhouse gas emissions. I favour a clearer identification of what we actually want to achieve. Is it reduced greenhouse gas emissions? Even with that exceptionally narrow and arguable goal (can we really identify all the greenhouse gases and weight their contribution to climate change accurately?), Kyoto, as Mr Le Page tells us, is a failure. Will reducing greenhouse gas emissions, even were we to be successful in doing that, actually affect the climate? To what end? Nobody knows. But perhaps Kyoto and its interminable followups is intended to be a respectable-sounding forum within which bureaucrats from all countries can talk to each other and syphon off enough taxpayers' money to bring up their families. Ok, yes, it's had more success there. But that's probably not an achievement that taxpayers support with enthusiasm.

We do need to focus on what we want to achieve. Climate Stability Bonds, which I advocate, don't need to be exclusively, or even partly, about aiming to reduce the variability of the earth's climate. We could target some physical and biological indicators of climate change, but we could instead, or as well, target social and financial measures, such as the numbers of people killed or made homeless by adverse climatic events and the cost of compensating them, or the costs of preventing the negative impacts from arising at all.

A bond regime would be sufficiently versatile to include all these indicators, and more. It wouldn't even require people to agree on the - still questioned - thesis that the climate is actually changing or that, if it is changing, it's something for which we are responsible. It would, though, require clarity and honesty about what we want to achieve. Hmm, perhaps that's why it's gone nowhere in the 25 years or so it's been in the public arena. But all is not lost, yet. For more, see this page, and the links therefrom.

Health: it's complicated

At lower levels of general health we have a good idea about what's needed: basic sanitation, inoculations, and education about hygiene, for starters. A benign government with funds can get things done. It gets a lot more complicated at the health levels prevailing in western countries. Here cause and effect are far more difficult to identify; there are huge numbers of, and possible interactions between, lifestyles and interventions that affect health. And these are changing constantly as our scientific knowledge grows. As well, lobbies are adept at influencing policy in their favour, often at the expense of the general health of the population. This is where an outcome-based approach can succeed where existing policy seems to have lost its way and is likely to be generating diminishing - even negative - returns.

Jerome Burn, here, points out some of the flaws of evidence based medicine as practised in the rich countries. He quotes Dr David Unwin, a general practitioner in Liverpool, UK:

We had to balance evidence based medicine – you come with a problem; I give you a solution – with evidence based practise. That means drawing on my years of clinical experience, rather than just relying on guidelines, and applying it to patient’s own experience. They are the expert on their lives, what they need and what works for them. Without taking that into account you are not going to change anything.

Mr Burn continues:

Even if charities or the government dug deep into their pockets and began to run many more RCT’s [Randomised Control Trials] on lifestyle changes, they are the wrong tool to use. The lifestyle approach we need to integrate much more effectively into medicine doesn’t involve just changing one thing – drug or no drug – it involves doing lots of things at once – for example: different diets and more exercise combined with psychological techniques such as stress reduction. RCTs have difficulties with such multiple interventions. Yet when they are tested they often turn out more effective than drugs.

The existing rich countries' healthcare systems don't encourage the approach that Dr Unwin and Mr Burn are advocating. What's more, they cannot do so. Drug companies' priority is to deliver returns to shareholders. One way of doing this is to influence government, which could not anyway gather, collate and exploit the data necessary to optimise the general health of the population.

Outcome-based policy, and Social Policy Bonds in particular, could be the answer. Improving rich countries' health is complicated and long term in nature. Existing policy isn't working very well. Broad metrics for physical health are fairly well established and robust. A gradual transition in the rich countries to a Social Policy Bond regime would reward efficient existing approaches and channel our limited funding into the most promising new ones. For more, see my essay on Health Bonds.

Scary

 Max Fisher concludes his long piece, How World War III Became Possible:

We may have escaped the Cold War, but we have not escaped the nuclear threat, which not only remains but is growing. The sense that this danger is resigned to history books, common in Washington and other Western capitals, is precisely part of its danger. It is another echo of the months and years before World War I, when the world drifted unknowingly toward disaster.

In April of last year, just after Russia had annexed Crimea, the London-based think tank Chatham House published a report on the dangers of unintended nuclear conflict. It was not pegged to the events in Ukraine, and at that point few people, including the report's authors, saw Crimea as the potential beginning of a larger conflict. Even still, it was dire in its warnings. "The probability of inadvertent nuclear use is not zero and is higher than had been widely considered," it stated. "The risk associated with nuclear weapons is high" and "under-appreciated." Their warnings were widely ignored. As the report itself noted, the world has concluded, wrongly, that nuclear weapons no longer pose an imminent threat. Attention has moved on. But the seeds of a possible war are being sown in Europe. How World War III Became Possible, Max Fisher, 29 June

You don't have to agree with all Mr Fisher's contentions to be scared by his discussion about how Russia and NATO might be sleepwalking toward devastating conflict. Still less should we be concerned to allocate blame. Probably there are faults on both sides, but trying to weigh them so as to have an opinion about which side is worse than the other is something a self-indulgence; understandable perhaps, given how powerless we ordinary people feel.

But I'd say we can do better than watch and debate on the sidelines. I cannot suggest a way out of any impending nuclear conflict, but what I do suggest is that we offer incentives for people to find ways of avoiding such a conflagration. Rather than leave everything to the politicians, ideologues, military men and the war-gamers, we could encourage people to back Conflict Reduction Bonds that will be redeemed only when there has been a sustained period of nuclear peace. Backers could include any combination of governments, international organizations, non-governmental organizations and philanthropists, and their funds could be swelled by contributions from the rest of us.

The maintenance of nuclear peace is a goal that is ideal for solution via Social Policy Bonds:

• it has an unambiguous, verifiable metric
• existing policy doesn't seem to be working
• nobody now knows the best ways of achieving the goal
• the goal is long term
• the goal is likely to require a multiplicity of diverse, adaptive approaches.   

Of course, the bond approach can run in parallel with existing policy, such as it is. It's likely to strengthen the hand of those whose activities seem to be working, as well as encourage new approaches, including those beyond current conception.

SIBs can be better than existing policy

Though still skeptical about Social Impact Bonds - the non-tradeable version of Social Policy Bonds - I do now think that under some circumstances they will be an improvement over existing policy. My main concern with SIBs is the danger that, being necessarily more narrow and short term than Social Policy Bonds, they will stimulate achievement of their targeted goals at the expense of people whose well-being isn't targeted for improvement. This possibility is lessened when the beneficiaries of a SIB regime are markedly disadvantaged. I am thinking now of the use of SIBs to target goals such as improving the well-being of the mentally unwell. (Here are arguments for and against the issuing of SIBs to brighten the employment prospects of mentally unwell people in New Zealand.) There will need to be provisos and careful monitoring, of course, as not all attempts to game a SIB regime can be foreseen and written into the redemption conditions of the bonds. But at this point SIBs are controversial, which is a positive in that it means that bondholders will want to ensure compliance with the bond issuers' intent and not only the bonds' legal stipulations.  

The advantages of a SIB regime over existing policy arise from their rewarding of outcomes rather than activities. Not as effectively as a pure Social Policy Bond regime, to be sure, but they should nevertheless stimulate greater efficiency in current activities and some innovation of new activities - a big plus where existing policy is thought to be failing.

You'll notice that I still have my reservations about Social Impact Bonds. Being better than existing policy is not always a great recommendation, though it could mean a lot to those individuals who (as in the New Zealand example) are in urgent need of new approaches. A pure Social Policy Bond regime would represent a drastic change from current policy with, I believe, commensurate benefits for all. SIBs can be a useful intermediate step towards such a regime.

Bonds in New Zealand

This post is a follow-up to my previous post, where I discuss why I don't like Social Impact Bonds. Today, Sunday, there was a short tv debate shown in New Zealand about using Social Impact Bonds to address mental health problems. My name is invoked at the five-minute mark. I would say that both protagonists are not quite correct. I'm against these SIBs only because they are not tradeable, because then there won't be much of the (necessary) innovation that the pro-SIB man, Stephen Franks, was talking about.

Why I don't like Social Impact Bonds

When I first came up with the idea of Social Policy Bonds, their tradeability was an intrinsic part of their identity. Without tradeability, the bonds would be little more than prizes awarded to existing service providers for doing their jobs a bit more efficiently than previously. There's obvious scope for gaming here: perform badly, wait till the government issues SIBs, perform better, then cash in. But perhaps more important is the necessity for allowing new service providers - or, as I term them, outcome achievers - to buy the bonds at any stage and receive extra rewards, in the form of a rise in the market value of their bonds, for performing well. Without tradeability, that won't happen.

The way of the world is that big business and government go hand-in-hand. The way of Social Impact Bonds is is equally familiar and equally dubious: government thinks it knows best who will deliver certain services and rewards them if they improve their performance. These will be existing service providers or, just possibly, new bodies upon which the government is equally keen to bestow favour for some reason or other. The whole point of my Social Policy Bond idea is to bring about creative destruction into the achievement of social and environmental outcomes. Under a Social Policy Bond regime new approaches and new organizations will be rewarded if, and only if, they are more efficient than anyone else. Social Impact Bonds won't do that: existing organizations will be favoured; they will have some incentive to perform their existing functions more efficiently, but little incentive to look at new approaches. For one thing, innovation risks rocking the boat. For another, SIBs, because they are not tradeable, are going to be restricted in their time horizon to that of existing organizations: they will take a short-term view, relative to the complexity of our social and environmental problems. This is a major deficiency: it restricts the scope of the bonds to narrow, short-term goals. As well, the costs of monitoring progress toward such goals is going to be large relative to sums at stake.

Contrast this with Social Policy Bonds, which are not limited by the prejudices of government (or anyone else) as to how our goals shall be achieved nor who shall achieve them. So we can target long-term social goals that will inevitably require diverse, adaptive approaches: reduced crime rates, for example, or improved health. We can even target global goals, like human development or the sustained avoidance of violent political conflict. Governments that issue Social Policy Bonds don't have to take a view as to which organizations are best placed to achieve such goals, nor as to how they shall go about achieving them. That work is done by bidders for the bonds who will be motivated by the rewards they get for helping achieve the targeted outcome. Note the word 'helping': a single organization need not achieve the entire goal to be rewarded under a Social Policy Bond regime: helping raise the likelihood of early achievement of the goal will generate reward in the form of the increased value of the bonds they (or a contracting agent) owns. Social Policy Bonds encourage a coalition of interests, whose composition and structure will change in response to changing circumstances, to co-operate in exploring and implementing a diverse, adaptive array of initiatives with the aim of achieving our broad social goals, even ones as seemingly remote and unattainable as universal female literacy or world peace.

For more on this see my short article: why the bonds must be tradeable. Also previous blog posts. For another critical view of SIBs (in New Zealand) see here. Contact me if you would like to know more.

Chopping down trees to fight climate change

It's inefficient, at best, for central government to try to prescribe how to achieve most social or environmental goals. Society's just too complex for any large, fixed organization fully to understand. But politicians aren't known for their humility. This is what happens when government thinks it knows how best to achieve its environmental goal of reducing greenhouse gases (not to be confused with trying to prevent climate change, but that's another story):

For the sake of a greener Europe, thousands of American trees are falling each month in the forests outside this cotton-country town [Oak City, North Carolina, US]. .... Each day, dozens of trucks haul freshly cut oaks and poplars to a nearby factory where the wood is converted into small pellets, to be used as fuel in European power plants. Soaring demand for this woody fuel has led to the construction of more than two dozen pellet factories in the Southeast [of the US] in the past decade, along with special port facilities in Virginia and Georgia where mountains of pellets are loaded onto Europe-bound freighters. European officials promote the trade as part of the fight against climate change. Burning “biomass” from trees instead of coal, they say, means fewer greenhouse gases in the atmosphere. .... But ...scientists say, Europe’s appetite for wood pellets could lead to more carbon pollution for decades to come, while also putting some of the East Coast’s most productive wildlife habitats at risk. How Europe’s climate policies led to more U.S. trees being cut down, Joby Warrick, 2 June

It's a typical government intervention; confusing the causes of a problem (climate change) with the supposed causes of a problem (coal burning) and imposing a restricted (geographically limited to the EU) pseudo-solution to a problem that goes beyond politicians' sphere of influence. The result: government intervention makes things worse even on its own terms.

The same confusion arises in other areas, especially those where experts think they have all the answers. So we see the targeting of a surrogate indicator like blood cholesterol in medicine, through the mass prescription of statins, which might, but probably won't, do anything to reduce the overall health of a population - with large upfront and certain costs in terms of finance and side effects. Of one thing we can be certain: government will not (pdf) look back at its policies and learn from its mistakes.

The solution, I think, is for governments to stick to what they are good at: articulating society's goals and raising the funds to achieve them. But, for complex social and environmental goals, government should contract out the actual achievement to a motivated coalition of interests, which will have incentives to explore diverse, adaptive ways of achieving our goals. A Social Policy Bond regime would effect such a contracting out. The bonds, being tradeable, would imply that this coalition could change, always to be composed of those who will be most efficient at achieving our goals. This, in turn, would mean that we could target long-term goals, including those, like climate change, that transcend national boundaries.