The Intergovernmental Panel on Climate Change (IPCC) had an experience like that. It was so impressed by one edition of the academic journal Climatic Change that it cited 16 of the 21 papers published that month. The journal editors should take a bow. When three-quarters of a single issue of your publication is relied on by a Nobel-winning report, you're doing something right.
Except for one small problem. The issue in question - May 2007 - didn't exist yet when the IPCC wrote its report. Moreover, none of the research papers eventually published in that issue had been finalized prior to the IPCC's cutoff date.
As the IPCC chairman recently reminded us, that organization's 2007 report:
...was based on scientific studies completed before January 2006, and did not include later studies...That's what the rules say. And that's what was supposed to have happened. But according to the online abstracts for each of the 16 papers cited by the IPCC and published in the May 2007 issue of Climatic Change (see my working notes here):
- 15 of them weren't accepted by the journal until Oct. 17, 2006
- the other wasn't accepted until May 18, 2006
So how could 16 papers, accounting for 39 new citations across fours chapters and two working groups, have made it into this twice vetted, next-to-finalized IPCC report? Those citations don't reference research papers the wider scientific community had already digested. They don't even reference papers that were hot off the press. Instead, in 15 of 16 cases, no expert reviewer could possibly have evaluated these papers since they hadn't yet been accepted for publication by the journal itself.
Where do these 39 citations of the May 2007 issue of Climatic Change turn up in the IPCC report? [working notes here]
- Chapt. 11 by Working Group 1 references ten papers (20 citations in total)
- Chapt. 12 by Working Group 2 references nine papers (15 citations in total)
- Chapt. 2 by Working Group 2 references two papers (2 citations in total)
- Chapt. 3 by Working Group 2 references two papers (2 citations in total)
- see the first abstract here (cited twice as Jacob et al. 2007 on this page of the IPCC report)
- second abstract is here (cited as Déqué et al. 2007 on this page)
- third abstract is here (cited as Christensen et al. 2007 on this page)
Welcome to the strange world of the IPCC. Whenever one turns over a new rock there's something shady beneath.
Coming soon: the research paper that wasn't accepted for publication until May 2008, yet got cited seven times in the IPCC's 2007 report
>> Cutoff dates, what cutoff dates?
>> What the IPCC learned from press releases
>> The Stern Review scandal - IPCC breaks 3 of its own rules
>> Climate bible gets 21 'F's on report card