Showing posts with label literacy. Show all posts
Showing posts with label literacy. Show all posts

21 August 2017

What exactly is the problem?

'When greenbacks are on offer, American schoolchildren seem to try harder': the Economist discusses how financial incentives can change test results:
[R]esearchers conducted an experiment in secondary schools in Shanghai and America. In each place pupils were split into two groups. The first answered 25 maths questions that had appeared in PISA. The second took the same test, but before the pupils did so, they were presented with an envelope with 25 dollar bills or the equivalent in yuan. The teens were told that for every wrong answer they would be docked a dollar. [The researchers] found that the ploy boosted scores among American students relative to their compatriots without a cash incentive, but not among the Chinese ones. Effort, not ability, may explain the gap between American and Chinese pupils, 'the Economist', 19 August. PISA is the Programme for International Student Assessment
The boost was significant: 'According to some rough calculations, if extrapolated to the main PISA test, the improvement in performance would have moved America from 36th to 19th in the ranking [of 69 countries], in which Shanghai came top.' It's an interesting result, and one that should make us question what the tests are trying to measure, and whether there are wider policy implications.

Test scores will most likely continue to play a big role in determining policy and the allocation of resources in education. That would be true whether or not we issue Social Policy Bonds that aim to achieve, say, universal literacy. So, in the light of the research results summarised above, should we offer cash incentives to pupils about to take literacy tests?

I'm inclined to think not, at least when it comes to measuring basic literacy. One reason is that motivation, or the lack of it, are significant in themselves. If children of school age find a basic reading test too burdensome to pass without a financial incentive, then that in itself can  be seen as a problem that needs to be solved. The nature of that problem might be a general cultural one, or one that's specific to certain classes of pupil. We might even interpret the difference between performance with a financial incentive and without as a social problem; and aim to narrow it.

There's no definitive answer though. Much depends on how we're going to use test scores, and whether there are other indicators that can usefully be targeted at the same time. As a society, we do need to think carefully about what we are trying to achieve. 'Teaching to the test' is problematic in itself and, while I do think universal literacy and numeracy are valid goals in themselves, and not very susceptible to the effects described above, we might do better to target for reduction as well as, or instead of, illiteracy, the social problems of unemployment and poverty.

23 May 2016

Blockchain could guarantee integrity of redemption terms

I've written before about hidden metrics as applied to Social Policy Bonds. Briefly, the aim is to overcome the sort of gaming that we see nowadays with certain micro-objectives: here, for example. Campbell's Law, if we're not careful, could apply. One answer, as I wrote, would be to target broad outcomes without specifying exactly how we shall determine whether our target has been achieved. So, say, we are aiming for 99 percent literacy of 15-year-olds in a country. We could issue Literacy Bonds, redeemable only when surveys show that that goal has been satisfied. We could say at the outset what the broad outlines of these surveys would look like: say a minimum of 250 boys and 250 girls randomly sampled from another random sample of 10 regions out of 100 regions. The identity of these regions need not be revealed or even determined until the bond market reckons that the literacy target has been achieved. At that point, the regions to be surveyed could be randomly chosen. That would work, provided the randomizing could be seen to be done transparently and fairly.

But the current Economist points to what might be a more elegant solution. It discusses using blockchain technology to distribute public keys to the protocols of the many thousands of medical trials carried out annually to ensure that the pharmaceutical companies sponsoring these trials cannot alter trial protocols. This, the sponsors have been known to do because the drugs being trialed have some positive effects, but not those specified at the outset of the trial. Such 'hidden outcome switching' can exaggerate the value of the drugs being trialed.
The blockchain is a database that acts as a public ledger of all transactions with the currency, and is thought to be almost completely tamper-proof because it is validated and stored independently on thousands of different computers worldwide. This provides a way ... to check that results have not been fudged. Better with bitcoin, the 'Economist', 21 May
Anyone with a copy of the trial protocol could generate the original private key, and check that it generates the same public key when it's decrypted using the same algorithm that encrypted it. 'Public keys for protocols should be uploaded to trial registries ... and included in research papers. Researchers and medical journals could speedily check whether the right results were being reported.'

More important, from the Social Policy Bond point of view, is this possibility, which concludes the Economist article:
Another benefit, paradoxically, is that the protocol for studies could be hidden until completed. This might be useful for commercially sensitive trials of new therapies. As long as the public key was uploaded to a registry when the trial began, the protocol could be verified later without the worry that it had been changed during the study.
If, instead of 'protocol for studies', we substitute 'detailed redemption terms', then we could use the same technology to could guarantee to investors in Social Policy Bonds that the backers of the bonds they own would not attempt to evade their obligations by manipulating the terms under which the targeted goal shall be deemed to have been reached.



06 August 2015

Hidden metrics

Hidden metrics, not to be confused with the hidden variables of quantum mechanics, are one possible way of preventing the gaming that could otherwise result from targeting specific quantitative outcomes.

Let me explain. Say we are targeting female literacy in Pakistan. We issue bonds that will be redeemed when the literacy level of 15-year old girls in Pakistan exceeds 95 percent each year over a ten-year period. Campbell's Law tells us, rightly I think, that:
The more any quantitative social indicator (or even some qualitative indicator) is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor. Source
What metrics could we deploy that measure, accurately and inevitably, that which we want to achieve: namely, near-universal literacy for 15-year old Pakistani girls?  We could subject every girl of that age to a standard reading test, but we know that academic tests and their results, even in developed countries, are manipulated. Girls could be taught to the test (by, for instance, being given the test in advance and taught to memorise it by rote). Or anyone up the ladder from the administrators of the test to the collators of the results could be bribed to alter the data at any stage.

So instead we could take a random sample of, say, 100 schoolgirls from 100 districts around Pakistan at ten yearly intervals, and test them in non-standard ways for their literacy. We don't reveal which districts, let alone which girls, will be chosen; nor do we release exact details of the test. That makes it much harder to game the outcome.

More complex is the goal of peace. Now, nuclear peace is relatively easy to target: we can issue bonds that will not be redeemed, say, until a thirty-year period has elapsed during which there has been no detonation (accidental or not) of a nuclear weapon that kills more than 100 people. But what about peace more generally? Hot wars are fairly easy to recognise and define and so would be correspondingly easy to deter using a bond regime. But there are more nebulous ways of fomenting conflict between states (Ukraine), or of ratcheting up tension to levels that severly curtail quality of life.  In these instances we could use an array of metrics such as: trans-border movement of weapons, numbers of people killed or fleeing their homes. Even the lack of information could also be an indicator of conflict that could find its way into our calculations. We could also use survey data, such as attitudes about potential enemies, or expressions of fear. Other potential indicators are disruptions to food, water or electricity supplies and other results of damage to infrastructure. 
 
The important point is that in this as in other goals, we need not and indeed should not specify in advance exactly which combination of metrics and indicators will be used to determine whether or not the bonds' redemption terms shall be deemed satisfied. In general, we try, as far as possible, to target metrics that are, or that are inextricably linked to, exactly what we want to achieve. Where that is difficult, we try to prevent gaming by not specifying too far in advance the exact redemption terms of the bonds. The aim, at all times, is for bondholders to comply with the spirit as well as the letter of the goals set by the backers of the bonds.