SeekerBlog

A caveat before proceeding: your writer considers climate change research and related policy options to be a serious issue. Climate science has become highly politicized – some of the scientists involved have become political actors. Meanwhile political actors cherry-pick each new climate science publication to see if it can be interpreted to support their preferred policy. In the excellent A Third Way on Climate?, Prometheus analyzes an 11/30/04 essay by Hans von Storch, Nico Stehr and Dennis Bray:

This then is the classical set-up of a modern science with high policy relevance – high stakes (implementation of Kyoto and beyond) and high uncertainty (in the assessment of ongoing change and in perspective of what may come)….

Judgments of solid scientific findings are often not made with respect to their immanent quality but on the basis of their alleged or real potential as a weapon by "skeptics" in a struggle for dominance in public and policy discourse.

Bubbling the best policy options to the top of this witches’ brew is going to be a challenge – it is becoming clear that changes are needed in the way climate science and policy generation is done. For now I’m trying to understand the current state of the science and how science and policy are coupled (there must be some coupling right?)

This post is mainly about quality control issues – with examples of what can happen when broken quality control is combined with politicized science. Traditional quality control procedures like "peer review" turn out to be much too weak – at least in this superheated environment. It’s obvious we need truly independent review, verification of high-impact results, and at least the possibility of audit (that means all the data sets, software, working papers, etc. must be safely archived so that if an audit is merited, then an independent team can do so promptly). Future posts will struggle with the harder question "how do we fix it?"

Key points first:

• The 1000-year N. American temperature reconstruction "hockey stick" graph, turns out to be "an artifact of poor mathematics"{1}.
• But this work was peer-reviewed, it must be OK!
• Not so. Peer-review doesn’t mean what we thought it meant. In the case of climate research, it specifically does not mean that another researcher verified your results (or even checked your data and calculations).
• Much more robust standards for review and audit of scientific results are required.

We have developed a short-list of objective climate science sources (seriously, I truly believe these sources are as clean as you can find – it is quite difficult to locate sources that are NOT already agenda-promoters). There are many other good sources not mentioned here, it just takes a lot of effort to wash each one down. Rather than trying to summarize or rewrite what has already been well-written, I would like to motivate you to read the following, presented in suggested reading order (that is, decreasing clarity , e.g., the Von Storch results are more difficult than the McIntyre and McKitrick results):

1. Global Warming Bombshell Richard A. Muller {1}, October 15, 2004 MIT Technology Review. I’ve selected two of Prof. Muller’s recent articles as he sets such a high standard for clarity. If you read only one reference, please read this one. From this article I understood what before looked like it was going to require more statistics than I was willing to invest.
2. Medieval Global Warming Richard A. Muller, December 17, 2003. This is Muller’s first article where he survey’s the emerging questions about the 1998 Mann paper (by Michael Mann of the University of Virginia and his coauthors Raymond Bradley and Malcolm Hughes, commonly referred to as MBH98 for short).
3. What Is Climate Change? R. Pielke, Jr., Summer 2004, Issues in Science and Technology. Roger Pielke, Jr. directs the University of Colorado’s Center for Science and Technology Policy Research at Boulder, CO.
4. Prometheus: McIntyre on Climate Science Policy Prometheus is Roger Pielke, Jr.’s blog, a resource to bookmark for anyone seriously interested in science policy. That is what they do at the center – work out how to make good science policy happen.
5. Some Thoughts on Disclosure and Due Diligence in Climate Science, Steve McIntyre at Climate Audit. Canadian scientists Stephen McIntyre and Ross McKitrick are one of the teams that have found serious errors in the MBH98 work.
6. Kyoto Protocol Based on Flawed Statistics Feb. 1, 2005 edition of Natuurwetenschap & Techniek (NWT), a prominent European science magazine. This survey of the McIntyre and McKitrick (M&M) research is longer and more detailed than Muller’s for MIT Technology Review. Well worth a read – the history of science politics will give you a better understanding of the social problems to be solved.
7. See also in the same issue Science popes and conflict of interests and a lead editorial.
8. Hockey Sticks, Principal Components and Spurious Significance has been accepted for publication in Geophysical Research Letters, the link is to the pre-publication copy – further reproduction or electronic distribution is not permitted.
9. The M&M Critique of the MBH98 Northern Hemisphere Climate Index: Update and Implications has been accepted for publication by Energy & Environment and is available here by kind permission of the publisher.
10. Re-constructing Past Climate from Noisy Data (Science Express, Sept. 30, 2004), Hans von Storch et al. The original article is subscriber-only, this Soon & Legates article is a useful summary, who quote Von Storch as follows: Von Storch bluntly summed up his results with the following comment reported in Der Spiegel on October 4:"We were able to show in a publication in Science that this [hockey stick] graph contains assumptions that are not permissible. Methodologically it is wrong: Rubbish."

I would like to emphasize the value of Roger Pielke, Jr.’s Prometheus Blog. I am continuing to work my way through the archives. It is a rich store of resources on science & technology policy in every domain, not just climate change. For those who are not motivated to go read the sources, here is a bit more background on the above, using the same paragraph numbering, emphasis mine. First, in the captioned Global Warming Bombshell column, Prof. Muller explains the science clearly in a few words:

(1) Progress in science is sometimes made by great discoveries. But science also advances when we learn that something we believed to be true isn’t. When solving a jigsaw puzzle, the solution can sometimes be stymied by the fact that a wrong piece has been wedged in a key place.

In the scientific and political debate over global warming, the latest wrong piece may be the “hockey stick,” the famous plot (shown below), published by University of Massachusetts geoscientist Michael Mann and colleagues. This plot purports to show that we are now experiencing the warmest climate in a millennium, and that the earth, after remaining cool for centuries during the medieval era, suddenly began to heat up about 100 years ago–just at the time that the burning of coal and oil led to an increase in atmospheric levels of carbon dioxide.

I talked about this at length in my December 2003 column. Unfortunately, discussion of this plot has been so polluted by political and activist frenzy that it is hard to dig into it to reach the science. My earlier column was largely a plea to let science proceed unmolested. Unfortunately, the very importance of the issue has made careful science difficult to pursue.

But now a shock: Canadian scientists Stephen McIntyre and Ross McKitrick have uncovered a fundamental mathematical flaw in the computer program that was used to produce the hockey stick. In his original publications of the stick, Mann purported to use a standard method known as principal component analysis, or PCA, to find the dominant features in a set of more than 70 different climate records.

But it wasn’t so. McIntyre and McKitrick obtained part of the program that Mann used, and they found serious problems. Not only does the program not do conventional PCA, but it handles data normalization in a way that can only be described as mistaken.

Now comes the real shocker. This improper normalization procedure tends to emphasize any data that do have the hockey stick shape, and to suppress all data that do not. To demonstrate this effect, McIntyre and McKitrick created some meaningless test data that had, on average, no trends. This method of generating random data is called “Monte Carlo” analysis, after the famous casino, and it is widely used in statistical analysis to test procedures. When McIntyre and McKitrick fed these random data into the Mann procedure, out popped a hockey stick shape!

That discovery hit me like a bombshell, and I suspect it is having the same effect on many others. Suddenly the hockey stick, the poster-child of the global warming community, turns out to be an artifact of poor mathematics. How could it happen? What is going on? Let me digress into a short technical discussion of how this incredible error took place… (continued in the original)

In this important paper, What is Climate Change, Roger Pielke, Jr. explains some of the issues that are hindering progress, and why there is such a bias towards mitigation rather than including adaptation policies.

(3) Believe it or not, the Framework Convention on Climate Change (FCCC), focused on international policy, and the Intergovernmental Panel on Climate Change (IPCC), focused on scientific assessments in support of the FCCC, use different definitions of climate change. The two definitions are not compatible, certainly not politically and perhaps not even scientifically. This lack of coherence has contributed to the current international stalemate on climate policy, a stalemate that matters because climate change is real and actions are needed to improve energy policies and to reduce the vulnerability of people and ecosystems to climate effects.

…Effective climate policy will necessarily require a combination of mitigation and adaptation policies. However, climate policy has for the past decade reflected a bias against adaptation, in large part due to the differing definitions of climate change.

The bias against adaptation is reflected in the schizophrenic attitude that the IPCC has taken toward the definition of climate change. Its working group on science prefers (and indeed developed) the broad IPCC definition. The working group on economics prefers the FCCC definition; and the working group on impacts, adaptation, and vulnerability uses both definitions. One result of this schizophrenia is an implicit bias against adaptation policies in the IPCC reports, and by extension, in policy discussions…

In this Prometheus post, Roger Pielke, Jr. comments on the quality-control issues raised by M&M (empasis added):

(4) Here at Prometheus we don’t do hockey sticks. (Astute readers will find one oblique reference to it in this paper – PDF.) However, the debate over the hockey stick is worth our attention not only for what it says about the state of climate science and politics, but also because it is significant for how we think about climate science policy. Climate science policy refers to those decisions that we make about climate science, including priorities for research and processes of scientific assessment and evaluation.

Steven McIntyre has posted his thoughts on climate science policy arising from his experiences with taking on the hockey stick. He writes,

IPCC proponents place great emphasis on the merit of articles that have been "peer reviewed" by a journal. However, as a form of due diligence, journal peer review in the multiproxy climate field is remarkably cursory, as compared with the due diligence of business processes. Peer review for climate publications, even by eminent journals like Nature or Science, is typically a quick unpaid read by two (or sometimes three) knowledgeable persons, usually close colleagues of the author. It is unheard of for a peer reviewer to actually check the data and calculations.

This observation has also been made in the Bulletin of the American Meteorological Society in a 2000 commentary by Ron Errico, who writes,

Too frequently, published papers contain fundamental errors? How can a piece of work be adequately evaluated or duplicated if what was really done or meant is not adequately stated?… My paramount recommendation is that our community acknowledges that a major problem in fact exists and requires ardent attention. Unless this is acknowledged, the community will likely not even consider significant changes. I suspect that too many scientists, especially those with the authority to demand changes, will prefer the status quo.

Errico’s paper, titled "On the Lack of Accountability in Meteorological Research", is well worth reading in full. He makes several recommendations that are completely consistent with McIntyre’s recommendations.

McIntyre also comments on the incestuous structure of the IPCC,

The inattentiveness of IPCC to verification is exacerbated by the lack of independence between authors with strong vested interests in previously published intellectual positions and IPCC section authors… For someone used to processes where prospectuses require qualifying reports from independent geologists, the lack of independence is simply breathtaking and a recipe for problems, regardless of the reasons initially prompting this strange arrangement.

From the just-published Feb. 1, 2005 Kyoto Protocol Based on Flawed Statistics in Natuurwetenschap & Techniek, we get some reaction from referees of the M&M Geophysical Letters paper (emphasis added):

(6) Turning Tide: In January 2005,an adapted version of McIntyre and McKitrick’s paper was accepted for publication by Geophysical Research Letters (GRL). The issue will come out in early 2005, as will a new issue of Energy and Environment containing a second publication by the two Canadians on implications of their GRL findings. Judging by the reactions of the referees of GRL, which McIntyre made available to us, the tide may be turning in the climatology field. One referee stated: “S.McIntyre and R.McKitrick have written a remarkable paper on a subject of great importance. What makes the paper significant is that they show that one of the most important and widely known results of climate analysis, the “hockey stick”diagram of Mann et al.,was based on a mistake in the application of a mathematical technique known as principal component analysis (PCA).”

The same referee also writes:“McIntyre and McKitrick found a non-standard normalization procedure in the Mann et al.analysis.Their paper describes this procedure;it was an apparently innocent one of normalization, but it had a major effect on their results. The Mann et al. normalization tends to significantly increase the variance of data sets that have the hockey-stick shape. In the Mann et al.data set, this turned out to be bristlecone pines in the western United States. Thus the hockey stick plot, rather than representing a true global average of climate for the past thousand years, at best represented the behavior of climate in the western US during that period.This is an astonishing result. I have looked carefully at the McIntyre and McKitrick analysis, and I am convinced that their work is correct.”

The referee ends with:“I urge you not to shy away from this paper because of its potential controversy. The whole field of global warming is currently suffering from the fact that it has become politicized. Science really depends for its success on an open dialogue,with critics on both sides being heard. McIntyre and McKitrick present a cogent analysis of the global warming data.They do not conclude that global warming is not a problem;they don’t even conclude that the medieval warm period really was there. All they do is correct the analysis of prior workers,in a way that must ultimately help us in our understanding of past climate,and predictions of future climate.That makes this a very important paper. I strongly urge you to publish it.”

{1}Dr. Richard A. Muller is a physicist at UC Berkeley Dept of Electrical Engineering, Faculty Senior Scientist at Lawrence Berkely Laboratory. He teaches the marvelous "Physics for Future Presidents" course – from which he has derived a monthly column for MIT Technology Review since 2001. Visit Dr. Muller’s homepage for an index to those columns – you will learn something new from every essay.