The File Drawer Effect

This is not an academics-only esoteric debate. It is about whether new, effective drugs are released. It is about how accurate is our knowledge of reality. Here is astronomer Phil Plait quoted in this excellent essay on Survivorship Bias:

For far too long, studies that fizzled out or showed insignificant results have not been submitted for publication at the same level as studies that end up with positive results, or even worse, they’ve been rejected by prominent journals. Left unchecked, over time you end up with science journals that only present the survivors of the journal process – studies showing significance. Psychologists are calling it the File Drawer Effect. The studies that disprove or weaken the hypotheses of high-profile studies seem to get stuffed in the file drawer, so to speak. Many scientists are pushing for the widespread publication of replication, failure, and insignificance. Only then, they argue, will the science journals and the journalism that reports on them accurately describe the world being explored. Science above all will need to root out survivorship, but it won’t be easy. This particular bias is especially pernicious, said Plait, because it is almost invisible by definition. ”The only way you can spot it is to always ask: what am I missing? Is what I’m seeing all there is? What am I not seeing? Those are incredibly difficult questions to answer, and not always answerable. But if you don’t ask them, then by definition you can’t answer them.”

We subscribe to the AllTrails network – I can’t recommend their work highly enough. See also Publication bias in Wikipedia and All Results Journals.

 

Down With the Luxury Journals?

Dereck Lowe:

Nobel laureate Randy Schekman has stirred up a lot of controversy with his public declaration that he will send no more manuscripts to Nature, Science, Cell and such “luxury journals”.

. . .The prevailing structures of personal reputation and career advancement mean the biggest rewards often follow the flashiest work, not the best. Those of us who follow these incentives are being entirely rational – I have followed them myself – but we do not always best serve our profession's interests, let alone those of humanity and society.

We all know what distorting incentives have done to finance and banking. The incentives my colleagues face are not huge bonuses, but the professional rewards that accompany publication in prestigious journals – chiefly Nature, Cell and Science.

These luxury journals are supposed to be the epitome of quality, publishing only the best research. Because funding and appointment panels often use place of publication as a proxy for quality of science, appearing in these titles often leads to grants and professorships. But the big journals' reputations are only partly warranted. While they publish many outstanding papers, they do not publish only outstanding papers. Neither are they the only publishers of outstanding research.

These journals aggressively curate their brands, in ways more conducive to selling subscriptions than to stimulating the most important research. Like fashion designers who create limited-edition handbags or suits, they know scarcity stokes demand, so they artificially restrict the number of papers they accept. The exclusive brands are then marketed with a gimmick called “impact factor”. . .

Making sense of uncertainty – why uncertainty is part of science

Here we have 28 pages of goodness – a very compact users' guide for reading scientific product.

Put crudely, scientists tend to think science is about things we don't know fully. Journalists and politicians (and to a large extent, many people) think it is about things we do know, and they're impatient with ‘maybe's.

The problem here is that not all scientific knowledge is the same, so treating it as though it were is misleading

For your reference library.

 

 

Michael Eisen: ‘The Past, Present and Future of Scholarly Publishing

Very important manifesto – from PLOS co-founder Prof. Michael Eisen's lecture at the California Commonwealth Club:

…Universities are, in essence, giving an incredibly valuable product – the end result of an investment of more than a hundred billion dollars of public funds every year – to publishers for free, and then they are paying them an additional ten billion dollars a year to lock these papers away where almost nobody can access them.

It would be funny if it weren’t so tragically insane.

To appreciate just how bizarre this arrangement is, I like the following metaphor. Imagine you are an obstetrician setting up a new practice. Your colleagues all make their money by charging parents a fee for each baby they deliver. It’s a good living. But you have a better idea. In exchange for YOUR services you will demand that parents give every baby you deliver over to you for adoption, in return for which you agree to lease these babies back to their parents provided they pay your annual subscription fee.

Of course no sane parent would agree to these terms. But the scientific community has.

…So what would be better? The outlines of an ideal system are simple to spell out. There should be no journal hierarchy, only broad journals like PLOS ONE. When papers are submitted to these journals, they should be immediately made available for free online – clearly marked to indicate that they have not yet been reviewed, but there to be used by people in the field capable of deciding on their own if the work is sound and important.

The journal would then organize a different type of peer review, in which experts in the field were asked if the paper is technically sound – as we currently do at PLOS ONE – but also what kinds of scientists would find this paper interesting, and how important should it be to them. This assessment would then be attached to the paper – there for everyone to see and use as they saw fit, whether it be to find papers, assess the contributions of the authors, or whatever.

This simple process would capture all of the value in the current peer review system while shedding most of its flaws. It would get papers out fast to people most able to build on them, but would provide everyone else with a way to know which papers are relevant to them and a guide to their quality and import.

By replacing the current journal hierarchy with a structured classification of research areas and levels of interest, this new system would undermine the generally poisonous “winner take all” attitude associated with publication in Science, Nature and their ilk. And by devaluing assessment made at the time of publication, this new system would facilitate the development of a robust system of post publication peer review in which individuals or groups could submit their own assessments of papers at any point after they were published. Papers could be updated to respond to comments or to new information, and we would finally make the published scientific literature as dynamic as science itself. And it would all be there for anyone, anywhere to not just access, but participate in.

…If we all do this, them maybe the next time someone like Aaron Swartz comes along and tries to access every scientific paper every written, instead of finding the FBI, they’ll find a giant green button that says “Download Now”.

The lecture transcript.

 

UC research should be free

Michael B. Eisen is a personal hero of mine. Co-founder of the biggest Open Access journal PLOS ONE, he campaigns every day to liberate research publications from being imprisoned behind the ridiculous paywalls of the publishers. This op-ed neatly summarizes the case for open access in the University of California system: 

It is a felony to share knowledge created by the faculty, staff and students of the University of California with the public.

Wait. What?

In 2011, online rights activist Aaron Swartz was accused of using the MIT computer network to download millions of scholarly journal articles with the intent of freely sharing them with the public. Federal prosecutors aggressively pursued charges against him, and, earlier this month, with a trial looming, Swartz killed himself.

The Justice Department has faced intense scrutiny for its senseless decision to turn this victimless act into a major case, but the real culprits in this tragedy are all the universities across the world that allowed articles that rightfully belong to the public to fall into private hands in the first place.

Every day, faculty, staff and students of the University of California hand over control of papers describing their ideas and discoveries to publishers, most of whom immediately lock them up behind expensive paywalls. They do this not only with the university’s knowledge— they do it with its complicity.

That the public does not have unlimited access to the intellectual output of academic scholars and scientists is one of the greatest-ever failures of vision and leadership from the men and women who run our research universities — all the more so at a publicly funded institution like the University of California.

When the Internet began to take off in the mid-1990’s, it created the opportunity to do something scholars had been dreaming of for millennia — to gather all of the writings of scholars past and present together in a single online public library — a free, globally accessible version of the ancient library in Alexandria.

But 20 years on and we are barely any closer to achieving this goal. Instead of posting their work online, scholars send them to journals, most of which condition publication on receipt of the authors’ copyright. These journals then exercise their exclusive rights to distribute these works by demanding payment for access to their collections.

If you have not yet published in a scholarly journal, you may not realize just how absurd this transaction is. Scholars at the UC system and every other research university on the planet voluntarily hand over control of their work to publishers, work that the same universities have to immediately turn around and buy back. And this is not a minor transaction — revenue for scholarly journals exceeds an estimated $9 billion per year.

Hiding your research behind a paywall is immoral

Michael Eisen linked the captioned Guardian article by Open Access activist Dr. Mike Taylor. This is a target-rich environment for Taylor and Eisen. I highly recommend you just dive into Mike’s essay, which begins with a conversion confession:

Publishing science behind paywalls is immoral. There, I said it.

I know, I know. It’s an easy trap to fall into – I’ve done it myself. To my shame, several of my own early papers, and even a recent one, are behind paywalls. I’m not speaking as a righteous man to sinners, but as a sinner who has repented.

Having started my scientific life from rather a conventional stance, it took me a while to come around to this position. (You can watch my position evolve, if you care to, through this chronological series of blogposts: “Choosing a journal”, “Free work”, “Collateral damage”, “Private-sector”, “RCUK submission”, “Irritation”, “Versus everybody” and “Making public”.) But I’ve finally arrived. And it’s great that the UK government has arrived in the same place.

If you are a scientist, your job is to bring new knowledge into the world. And if you bring new knowledge into the world, it’s immoral to hide it. I heartily wish I’d never done it, and I won’t do it again.

But aren’t there special cases?

I really need to publish in Science/Nature/Cell for my career …

No. Michael Eisen, cofounder of the Public Library of Science (PLOS), doesn’t believe this is true and makes a strong case that we’re confusing correlation with causation. He notes that fewer than half of biology hires at Berkeley in the last decade have published in Science, Nature or Cell. Berkeley!

(…) 

But I can’t afford article processing charges (APCs) …

No. First of all, more than half of open-access journals don’t charge a fee at all. Among those that do, the average fee is $906 (£563) – a tiny proportion of most research grants. PeerJ, which launches this month, charges a one-off fee of $299 for a lifetime’s publications. Most fee-charging open-access journals offer waivers – for example, the no-questions-asked waiver at PLOS, where the philosophy is explicitly that no one should be prevented from publishing by lack of funds.

Tim Gowers, who is leading a boycott against the publisher Elsevier and is starting the new Forum of Mathematics journal says it “will not under any circumstances expect authors to meet APCs out of their own pockets, and I would refuse to be an editor if it did”.

(…) 

Great article, great comments, so many substantive comments that Mike followed up with this post.

Is it immoral to hide your research behind a paywall?

Mike Taylor’s followup post, which has also attracted lots of excellent comments. Here’s a sample

As noted a few days ago, I recently had an article published on the Guardian site entitled Hiding your research behind a paywall is immoral. The reaction to that article was fascinating, exhilarating and distressing in fairly equal parts. Fascinating because it generated a fertile stream of 156 comments, most of them substantial. Exhilarating because of some very positive responses. And distressing because some people who I like and respect absolutely hated it.

Those people’s main objections were nicely summed up by a response piece by Chris Chambers, published a few days later on the same site: Those who publish research behind paywalls are victims not perpetrators. It’s a good, measured article, and I highly recommend it — not least because it’s apparent that while Chris thinks my tactics are all off, he makes it clear that he shares the goal of universal open access and further significant reform in scholarly communications.

So I’d like to clarify a couple of points that I didn’t make clearly enough in the original articles (but which I addressed in two separate comments on Chris’s article); and then I want to throw the floor open to see if we can hack through the more difficult issues that it raises.

A clarification

Chris wrote:

Do scientists who follow accepted publishing practices deserve to be labelled ‘immoral’, as Taylor claims?

The intention of my original article was not to say that the individuals who allow their work to go behind paywalls are immoral people, but that the act it itself immoral. If that feels like a fine distinction, it’s not. For a variety of pragmatic reasons, essentially moral people commit immortal acts all the time. At the trivial end of the scale, something as insignificant as not bothering to sort the recycling; at the other end, while no-one would claim dropping atomic bombs on civilian populations is an essentially moral act, many people would accept that in the context of WWII, the Hiroshima and Nagasaki bombs were justified or even necessary. (And please: no-one cite this as ‘Mike says publishing behind a paywall is exactly like nuking civilians’!)

So my goal in the original piece was not to castigate individuals as immoral people, but to push us all into deliberately thinking through the moral implications of our publication choices — decisions that all too many scientists still make without thought for the accessibility or otherwise of their work. I stand by my original assertion that it’s immoral to accept public funding to do research, then hide the fruits of that research from the public that paid for it. But that doesn’t mean that I am ‘labelling’ anyone. My apologies if that distinction wasn’t clear.

To summarise the intent of my article: the decision of where to publish is a moral one. Please, all you moral people out there, make a moral choice.

The curse of  journal prestige

And so we come to the vexed subject of journal rank. First of all, it’s encouraging to see that most people seem to agree at least that the effects of journal rank are A Bad Thing — that judging scientists by what journals they have published in is at best corner cutting, if not outright dereliction. This is not controversial any more, if it ever was: the ridiculous experience of PLOS Medicine as they  negotiated (yes, negotiated) their initial impact factor tells you all you need to know about such metrics.

As Chris wrote in his article:

In many (if not most) fields, the journals in which we publish are judged to be an indicator of professional quality. […] Science is bad at being scientific: the actual quality takes second place to the perception of quality, which is so strong that journal rank creates its own biosphere.

The problem here seem to be one of wrenching an entire community out of a delusion at once. Because the things I hear over and over again are: 1. ‘Of course, I personally would never judge a paper by what journal it’s in, or judge a scientist by what journals her papers are in’. And 2. ‘I need to get my papers in glamourous journals so that people will judge me well’. Everyone is worried about being judged by the very criterion that they insist they would never judge by.

I don’t pretend to have a solution to this absurd circle. Well, I do: we should all just stop it. But I don’t have a strategy for reaching that solution. One thing that is infuriating to see is that even when the REF and the Wellcome Trust so very explicitly say ‘We don’t care what journal your work is in’, researchers continue to disbelieve them. I would love to hear constructive thoughts on what can be done about this.

Read the whole thing and the comments, where Michael Eisen closes the circle with this:

Can somebody please call snopes.com? Everybody here is do sure that you have to have a Science, Nature or Cell paper to get a fellowship/job/tenure because “somebody told them so”. So scientific of y’all. How about somebody look at some actual data? I have. I will post more details on my blog when I get a moment, but the basic things I have found are:

1) I looked at new hires in the top 10 biology departments in the US. All had recent assistant professor hires who had not published Science, Nature or Cell papers prior to getting their jobs. A fair number of new hires published their postdoctoral work in OA journals.

2) I looked at recent awardees of several major postdoctoral fellowship programs, and again, while there were certainly many people with SNC papers, in all cases there were exceptions.

3) I looked at two prominent young investigator awards, and, surprise surprise, there were a fair number of awardees without glamour mag publications (including one whose work was published in PLoS ONE).

So let’s cut out the “somebody told me” crap. Whether people said it or not, it’s simply not true. Is there a correlation between getting jobs and fellowships and SNC publications? Yes. Of course. Because, first of all, everybody believes it’s essential and works hard to get their papers there. And the same basic criteria for cutting edge science are used by the editors of these journals as are used in hiring/fellowship decisions. So there would be a correlation even if there were no causal relationship.

Do I think it helps to have glamour mag publications if all else is equal? Yes. It probably does a little.

Peter Kareiva: Myth-busting scientist pushes greens past reliance on ‘horror stories’

Stewart Brand has “green creds” with most of the environmentalists that we know. Since Stewart posted a Google+ note on a longish E&E profile on Peter Kareiva I thought I should have a look. I’ve mostly read Kareiva’s essays and lectures, but haven’t looked much at his history. If you have been flustered by Kareiva’s criticisms of the old-school Sierra Club/FOE types, you are not likely to find this profile comforting. I think is that Kareiva is spot on – e.g., his Breakthrough joint paper Conservation in the Anthropocene- Beyond Solitude and Fragility. See what you think…

Peer-reviewed and truth are not synonyms

Russ Roberts interviews Brian Nosek in this Econtalk podcast: Nosek on Truth, Science, and Academic Incentives. I learned quite a bit about flaws in the academic incentive structure, and how challenging it is to make structural changes in the publishing pipeline that will impact on the quality of papers (and the research).

Brian Nosek of the University of Virginia talks with EconTalk host Russ Roberts about how incentives in academic life create a tension between truth-seeking and professional advancement. Nosek argues that these incentives create a subconscious bias toward making research decisions in favor of novel results that may not be true, particularly in empirical and experimental work in the social sciences. In the second half of the conversation, Nosek details some practical innovations occurring in the field of psychology, to replicate established results and to publicize unpublished results that are not sufficiently exciting to merit publication but that nevertheless advance understanding and knowledge. These include the Open Science Framework and PsychFileDrawer.