This science policy post by Roger Pielke Jr. is a gem. You’ll want to keep these principles in mind whenever you read new research press releases (much of the science reporting you read in the media is regurgitated press releases). Here’s an excerpt:
(…) The problem of “false positive science” is of course not limited to the discipline of psychology or even the social sciences. Simmons et al. provide several excellent empirical examples of how ambiguity in the research process leads to false positives and offer some advice for how the research community might begin to deal with the problem.
Writing at The Chronicle of Higher Education, Geoffrey Pullam says that a gullible and compliant media makes things worse:
Compounding this problem with psychological science is the pathetic state of science reporting: the problem of how unacceptably easy it is to publish total fictions about science, and falsely claim relevance to real everyday life.
Pullam provides a nice example of the dynamics discussed here in the recent case of the so-called “QWERTY effect” which is also dissected here. On this blog I’ve occasionally pointed to silly science and silly reporting, as well as good science and good reporting — which on any given topic is all mixed up together.
When prominent members of the media take on an activist bent, the challenge is further compounded. Of course, members of the media are not alone in their activism through science. The combination of ambiguity, researcher interest in a significant result and research as a tool of activism makes sorting through the thicket of knowledge a challenge in the best of cases, and sometimes just impossible.
The practical conclusion to draw from Simmons et al. is that much of what we think we know based on conventional statistical studies published in the academic literature stands a good chance of just not being so — certainly more than the 5% threshold used as a threshold for significance. Absent solid research, we simply can’t distinguish empirically between false and true positives, meaning that we apply other criteria, like political expediency. Knowing what to know turns out to be quite a challenge.