What exactly is the problem?

'When greenbacks are on offer, American schoolchildren seem to try harder': the Economist discusses how financial incentives can change test results:

[R]esearchers conducted an experiment in secondary schools in Shanghai and America. In each place pupils were split into two groups. The first answered 25 maths questions that had appeared in PISA. The second took the same test, but before the pupils did so, they were presented with an envelope with 25 dollar bills or the equivalent in yuan. The teens were told that for every wrong answer they would be docked a dollar. [The researchers] found that the ploy boosted scores among American students relative to their compatriots without a cash incentive, but not among the Chinese ones. Effort, not ability, may explain the gap between American and Chinese pupils, 'the Economist', 19 August. PISA is the Programme for International Student Assessment

The boost was significant: 'According to some rough calculations, if extrapolated to the main PISA test, the improvement in performance would have moved America from 36th to 19th in the ranking [of 69 countries], in which Shanghai came top.' It's an interesting result, and one that should make us question what the tests are trying to measure, and whether there are wider policy implications.

Test scores will most likely continue to play a big role in determining policy and the allocation of resources in education. That would be true whether or not we issue Social Policy Bonds that aim to achieve, say, universal literacy. So, in the light of the research results summarised above, should we offer cash incentives to pupils about to take literacy tests?

I'm inclined to think not, at least when it comes to measuring basic literacy. One reason is that motivation, or the lack of it, are significant in themselves. If children of school age find a basic reading test too burdensome to pass without a financial incentive, then that in itself can  be seen as a problem that needs to be solved. The nature of that problem might be a general cultural one, or one that's specific to certain classes of pupil. We might even interpret the difference between performance with a financial incentive and without as a social problem; and aim to narrow it.

There's no definitive answer though. Much depends on how we're going to use test scores, and whether there are other indicators that can usefully be targeted at the same time. As a society, we do need to think carefully about what we are trying to achieve. 'Teaching to the test' is problematic in itself and, while I do think universal literacy and numeracy are valid goals in themselves, and not very susceptible to the effects described above, we might do better to target for reduction as well as, or instead of, illiteracy, the social problems of unemployment and poverty.