Economist Richard Tol has been taking another look at everyone's favourite mega-document, the 2007 Intergovernmental Panel on Climate Change (IPCC) report. In guest posts on blogs here and here, he argues that while one section of the report (produced by Working Group 2) "appears to have systematically overstated the negative impacts of climate change," another section (written by Working Group 3) appears to have systematically understated the costs to society associated with emissions reduction.
Click image for larger version. From p. 7 of a Dec. 2009 document issuedby the US Environmental Protection Agency (39-page PDF here)
At this juncture it's worth remembering that the IPCC's chairman, Rajendra Pachauri, has repeatedly claimed that the IPCC relies solely on peer-reviewed material to make its case. By now we know this isn't remotely true. Tol highlights passages in Chapter 11 of the Working Group 3 report that further demonstrate this.
On this page, the IPCC discusses emissions reduction studies. Tol points out that although the third paragraph cites three documents – Stern (2006), Anderson (2006) and Barker (2006a) – not one of them has been peer-reviewed. Indeed, of the seven studies mentioned in total on this page only one was published in a peer-reviewed journal. (All reference material for that chapter is listed here.)
Tol further notes that on another page, devoted to the rather important question of what effect reducing emissions might have on employment (in the US climate change policies are currently being sold to be public as job creation plans), a total of six "studies are cited to support the notion that emission reduction creates jobs. Only one of the six is peer-reviewed."
If this seems rather sloppy, Tol says it gets worse. The academic literature in this area, he says, suggests that the relationship between emissions reduction and job creation is a weak one, and that job growth only occurs in certain circumstances – namely when government policies are "smart and well-designed." If "emission permits are given away for free – as is common," he points out, "no positive impact on employment" is achieved. The IPCC report mentions none of this, however.
Tol doesn't talk about it in these blog posts, but he was an IPCC expert reviewer for this chapter. After reading the first draft, he raised a number of concerns. Below are some choice remarks appearing on pages 2-3 of the 65-page PDF of reviewer comments available here:
In a number of instances, authors mainly quote their own work. This is unworthy. In a number of instances, authors mainly quote other IPCC material. This is incestuous. The quoting of IPCC material is most pronounced in the scenario discussion, which can be summarised as "We, the IPCC, declare that all previous IPCC work is great." This is silly.Tol complains repeatedly elsewhere in the reviewer comments. In one instance he says that "much of the material is based on papers that have not been peer-reviewed." In response, one of the chapter's authors acknowledges this to be true:
…In many places, the authors are out of their depth; the selection of papers is haphazard, the assessment superficial. I also found too many references that are simply wrong; the authors cannot have read these papers. For a supposedly expert panel, this is very serious.
…In a number of instances, the draft material reads like a political manifesto rather than a scientific document. In other instances, the authors have tried to hide their political message in pseudo-scientific language. For a supposedly independent panel, this is very serious.
…Part of the literature review is haphazard; it seems as if the authors have not systematically searched the literature, but simple [sic] quote a few papers that happened to lie around. Another part of the literature review is severely biased; the authors quote their own work, and that of their friends, but systematically ignore the work of many authors. This is particularly true in the presentation of model results; results are shown for a subset of models only…
There is much material that is grey literature in the chapter. This is generally in the form of research reports to governments (e.g. US Congressional papers, such as the Lasky review)…Specific grey literature will be reviewed by Chapter 11 authors and made accessible following IPCC procedures. [bold added]Annoyed that a European Union document is cited by the IPCC as proof of a particular point, Tol protests:
EU communications are not peer-reviewed. On the contrary, the Commission is known for manipulating research and hiring manipulable researchers. [sic]Two responses appear on page 36 of the PDF. The first, in a normal-coloured font (after two initials that apparently identify the writer), rather incoherently says: "We will try." An unidentified person, using red-coloured font and underscoring, then adds:
(not an answer! The point by Tol is perhaps a brutal statement but needs consideration because it is true in some way. Some people at some directorates prefer ‘willing’ consultants above ‘independent’ analysis, and IPCC should be cautious in accepting this grey literature! [bold added, closing parenthesis missing in the original]There is no guarantee, of course, that research papers published in peer-reviewed journals are correct. (As climatologist Phil Jones recently testified, no peer reviewers even asked to see his raw data.) Conversely, just because a study hasn't been peer-reviewed doesn't mean it's wrong. But if peer-reviewed literature exists and the IPCC chooses instead to cite non-peer-reviewed sources produced by potentially politically-motivated government bodies, surely that's a problem.
Tol wasn't the only person expressing these concerns during those early days (the second draft and a second round of reviewer comments were still to come). On page four of this PDF, Mohammed Alfehaid observes:
Chapter 11 is one of the most important chapters as it is supposed to be the crux of [Working Group 3] where cross sectoral mitigation should have been adequately addressed. Only positive potential views are represented while the costs and adverse effects are mostly neglected…There is not adequate references, reference mostly chosen on a personal preference rather on scientific finding and unbiaseness. [sic]The chapter authors' response includes the following:
The team will seek to ensure that the references are balanced and adequate in that all are peerreviewed, or otherwise acceptable to the team. [sic]Another reviewer, Bert Metz, asked: "Why is only one EU study discussed here. There must be many more similar studies…" Another, Jim Ragland, commented: "This section…presents a one-sided review of the literature which ignores or distorts the views of researchers who have alternative views on the subject."
Given that so much concern was expressed about the quality of the sources being cited one would think the ratio of peer-reviewed to grey literature citations would be high in the final product.
But an examination of all the references ultimately cited shows something rather different. Tol and others may have kicked and screamed, metaphorically speaking, about the lack of peer-reviewed sources. They may have done so since the earliest opportunity afforded to them by the IPCC process. The authors of Chapter 11 may have acknowledged these concerns. But because it's the authors themselves who ultimately decide what gets included and what gets left out, little progress seems to have been made.
I counted the references cited in the final, published version of Chapter 11 and got a tally of 330. Of those, fully 139 – or 42 percent – were non-peer-reviewed grey literature.
This is worth repeating: despite vigorous protests from its own expert reviewers, in this chapter only 58% of the documents cited by the IPCC were peer-reviewed.
..
>> Let there be no more scientific consensuses
>> The battle for the soul of science
>> More dodgy citations in the Nobel-winning climate report
>> Greenpeace and the Nobel-winning climate report
>> Yes Virginia, the climate bible relies on newspaper clippings
