A year later, in June 2008, during a visit to New Zealand, Pachauri told a journalist: "People can have confidence in the IPCC's conclusions…Given that it is all on the basis of peer-reviewed literature."
A few weeks afterward, in San Francisco, he again told an audience that IPCC reports are "based on peer-reviewed literature." On that occasion, he mocked the idea that his organization might "pick up a newspaper article and, based on that, come up with our findings." IPCC reports rely, he insisted, "on very rigorous research which has stood the test of scrutiny through peer reviews." [hear a sound clip here] [see the video here – these remarks begin at 1 min, 15 seconds]
The more one examines IPCC publications, however, the more evident it becomes that we've all been told a fairy tale. Andreas Bjurström of Sweden's Göteborgs Universitet, had a guest post on Roger Peilke Jr.'s blog yesterday regarding the previous IPCC report. Among his startling findings: only 62 percent (less than two-thirds) of the sources cited by the IPCC back in 2001 were peer-reviewed.
In fairness, Dr. Pachauri didn't become chairman until 2002. So while we may accuse him of paying little attention to his organization's previous publication, it’s only the 2007 report for which he bears responsibility. A couple of days ago I blogged about a chapter in the latest report which, I discovered, relies on peer-reviewed sources only 58 percent of the time. That number seems shockingly low when one considers that the IPCC's expert reviewers complained bitterly about the quality of the citations at the time the report was being written.
Yet that may be the IPCC on a good day. Chapter 5, from Working Group 3's report - which I randomly chose to examine next - is far worse. Only 61 of the 260 references relied on in that chapter have their feet firmly planted in peer-reviewed literature – an abysmal 24 percent. Put another way, three-quarters of the material cited there is grey literature. In a chapter devoted to something as tangible as the transportation sector. [CORRECTION: 64 references were peer-reviewed, bringing the overall percentage to 25 percent. Please accept my apologies.]
What's bizarre is that an examination of the comments submitted by IPCC reviewers following both the first and second draft of Chapter 5 - and the responses to them - suggests that those involved appear to have taken part in a shared hallucination. A great deal of lip service was paid to peer-review, but in practice it was a next to meaningless concept.
When Takayuki Takeshita, a researcher associated with the University of Tokyo, suggested that a presentation he'd helped prepare be cited by the IPCC, the chapter authors advised him that "the use of a presentation would not satisfy the requirement for published literature." This is all well and good, but had that standard been applied uniformly the list of references at the end of the chapter would contain closer to 64 entries than 260.
Elsewhere, when Takeshita said he considered a statement in the chapter to be "doubtful" and noted that it conflicted with almost "all of the literature I have ever read" on the topic, he was told: "Rejected; text simply quotes the study, and good chance the study is correct."
The full citation for that study looks like this:
MIT, 2004: Coordinated policy measures for reducing the fuel consumption of the U.S. light-duty vehicle fleet. Bandivadekar, A.P. and J.B. Heywood, Massachusetts Institute of Technology, Laboratory for Energy and the Environment Report LFEE 2004-001, 76 pp.Despite the fact that it is not peer-reviewed, the chapter authors think there's a "good chance" it's correct – and that's the end of the matter.
This is the celebrated IPCC internal peer review process in action. The reviewers don't get to make their case to a neutral editor who then arbitrates. Instead, the authors are at perfect liberty to ignore comments submitted in good faith by expert reviewers - to decide that the source they've cited is probably right.
The shenanigans don't end there. Another reviewer, Danny Harvey from the University of Toronto, pointed out that the descriptive text written by the IPCC's authors was at odds with the labeling on a graph. The authors' response: "Rejected. The figure came from [non-peer-reviewed] literature, it was not built by the authors."
On two separate occasions, John Kessels from the Energy Research Center of the Netherlands, complained that press releases were being cited to support statements of fact. "Not referenced adequately, is a press release scientific literature?" he asked, and then again: "[this] is a press release not a journal or scientific literature."
In the first instance, the authors replied: "We will add ref for cost estimate." In the second case, he was advised: "Rejected, information on this type of high-edge technologies is not gotten from the scientific journals."
Would it surprise you to learn that both press releases are, in fact, relied on in the final published version (Power System, 2005 and Yuasa, 2000)? Would it surprise you to learn that while this travesty was allowed to occur, a comment submitted by the government of Australia was summarily dismissed because, according to the chapter's authors, Australia's contention wasn't "supported by available literature."
There is no doubt about it. Pachauri has repeatedly misled us. The IPCC does not rely solely on peer-reviewed literature - not by a long shot. Moreover, his penchant for declaring that he "can't think of a better process" than the one employed by the IPCC and that there "is not a parallel on this planet" starts to look pathological once one has peered under the hood.
Others also have some explaining to do. According to the IPCC (see a graphic here), 2,500 people served as expert reviewers on the 2007 report. Eight hundred more were contributing authors, and another 450 were lead authors. That's roughly 4,000 souls who were in a position to know that the claim that the IPCC report is based solely on peer-reviewed literature is absolute fiction.
How – and why - have we been deceived about this for so long?
If the amalgam of people and institutions that comprise the IPCC can't be trusted to tell the truth about something so simple, why should we pay attention to anything else the IPCC says?
..
IPCC reviewer comments and author responses may be accessed here. Takeshita's first-draft comments appear on pp. 15 & 87 of this 91-page PDF. All other comments are from this 186-page PDF. Harvey's appears on p. 88. Kessels' are on pp. 102 & 104. See p. 159 for Australia's comment.
>> Let there be no more scientific consensuses
>> Systemic failure: invasion of the drama queens
>> We're always out-of-touch with the future
>> Almost half non-peer-reviewed
>> More dodgy citations in the Nobel-winning climate report
