29 June 2007

Forecasts by scientists versus scientific forecasts

One of the virtues of the Social Policy Bond approach is that we can insure against things that may or may not be happening. Take climate change: if the market believes the climate is not becoming more unstable, then Climate Stability Bonds targeting the current level of stability, would be valued at something close to their redemption value when floated. So the bonds' backers (most probably taxpayers) would lose very little. For this reason I'm in the happy position - intellectually - of not needing to have an opinion about what is happening to the world's climate. Climate Stability Bonds would be the most cost-efficient way of dealing with climate change whether we accept the scientific consensus or not. Correction: whether we accept the consensus of scientists or not. There is a difference, as the recent paper Global Warming: Forecasts by Scientists Versus Scientific Forecasts (pdf) by J Scott Armstrong and Kesten C Green makes clear:

The forecasts in the [Intergovernmental Panel on Climate Change’s Working Group One] Report [2007] were not the outcome of scientific procedures. In effect, they present the opinions of scientists transformed by mathematics and obscured by complex writing. We found no references to the primary sources of information on forecasting despite the fact these are easily available in books, articles, and websites. We conducted an audit of Chapter 8 of the Report. We found enough information to make judgments on 89 out of the total of 140 principles. The forecasting procedures that were used violated 72 principles. Many of the violations were, by themselves, critical. We have been unable to identify any scientific forecasts to support global warming. Claims that the Earth will get warmer have no more credence than saying that it will get colder. ...

Prior to conducting an audit, one might ask policy makers to say what information would be sufficient to change their opinions. People who are able to specify such evidence are often able to change their opinions. When we have used this question among academic researchers and students, we find that many of them are willing to specify such information. Disturbingly, however, many others are unable to even imagine that any information could possibly change their minds.

This paper could be seen as reassuring or threatening, depending on your viewpoint. Regardless, though, it is, as the authors say in the quote above, disturbing that basic scientific forecasting principles appear to have been violated. (See here for an article inspired by the paper.)
Online Dating


Frank said...

Hi Ronnie,

It's funny, I came across Armstrong's article a few days ago.
I haven't read it entirely, so I can't really discuss it. What put me off is that he (as so many others) claims to use scientific principles whereas others do not. In the chapter "On the value of forecasts by experts" he then quotes -what he calls- forecasts which turned out to be wrong. Does he really compare those to the climate predictions the IPCC is based on. Also he argues that Watt's statement from 1970 was a (wrong) forecast about temperature changes, when if fact Watt just said what would happen "[i]f present trends continue".
He is also cherry picking his references and grossly mis-interprets statements, e.g. says that Jim Renwick's statement about seasonal forecasting is a statement about global climate models in general.

Note that I do not want to make a judgement as to whether his conclusion is wrong or right, but his methods do not strike me as particularly scientific.

I have the impression that he is just frustrated because the scientific community ignores his "forecasting principles".


Ronnie Horesh said...

Thanks Frank for your comment. I emailed Gavin at the excellent RealClimate website, asking for his views on the Armstrong and Green paper. I'm sure he won't mind my publishing the gist of his response here:

"We saw this, but it's very badly done. The authors ignorance about how modelling is done or what is actually being forecast is evident. For instance, they only looked at one chapter of the IPCC report when scattered throughout the three volumes are all the issues that they felt were ignored. 'Out of sample' tests? There are hundreds, but Armstrong et al gave it a -2 rating - that can only be because they don't know what the 'sample' is or how climate models are built and tested.

"Given the low level of diligence, plus the parrotting of some of the least well supported contrarian literature (Carter et al? please!), one gets the sense that the biggest point of contention is that IPCC didn't cite Armstrong's book."

It seems you were right to be skeptical. Regards

Harald Korneliussen said...

If a social policy bonds regime came into action, and climate think tanks started investigating actions against climate change... well, one thing is certain, I wouldn't be investing in your climate think tank!

On climate science, it seems to me you still have a bit of that "sell the shovels, buy ditch digged bonds instead!" attitude.

Ronnie Horesh said...

Thanks Harald. I don't think I'd set up a climate change think-tank. Your choice would be whether to invest in the bonds or not. Any organization that resuts from the bonds would have as its objective that of achieving climate stability. Running a think-tank of any sort, if it were to happen at all, would be a byproduct of that goal.

But your main point is valid: I am more concerned about setting up policies that would encourage people to dig ditches, rather than to dig ditches myself. I could - and would try to - do both of course, but my comparative advantage is in policy, rather than climate change science. Actually I don't see the two activities as mutually exclusive alternatives. I think setting up systems that reward the achievement of climate stability will motivate, and enlarge the pool of, the people interested in actaully achieving it. Regards