The File Drawer Effect

This is not an academics-only esoteric debate. It is about whether new, effective drugs are released. It is about how accurate is our knowledge of reality. Here is astronomer Phil Plait quoted in this excellent essay on Survivorship Bias:

For far too long, studies that fizzled out or showed insignificant results have not been submitted for publication at the same level as studies that end up with positive results, or even worse, they’ve been rejected by prominent journals. Left unchecked, over time you end up with science journals that only present the survivors of the journal process – studies showing significance. Psychologists are calling it the File Drawer Effect. The studies that disprove or weaken the hypotheses of high-profile studies seem to get stuffed in the file drawer, so to speak. Many scientists are pushing for the widespread publication of replication, failure, and insignificance. Only then, they argue, will the science journals and the journalism that reports on them accurately describe the world being explored. Science above all will need to root out survivorship, but it won’t be easy. This particular bias is especially pernicious, said Plait, because it is almost invisible by definition. ”The only way you can spot it is to always ask: what am I missing? Is what I’m seeing all there is? What am I not seeing? Those are incredibly difficult questions to answer, and not always answerable. But if you don’t ask them, then by definition you can’t answer them.”

We subscribe to the AllTrails network – I can’t recommend their work highly enough. See also Publication bias in Wikipedia and All Results Journals.