Peer review: If no one else reforms it, could Twitter do the job?

In "Trial by Twitter",Apoorva Mandavilli for Nature (19 January 2011) tells us that "Blogs and tweets are ripping papers apart within days of publication, leaving researchers unsure how to react."

"Scientists discover keys to long life," proclaimed The Wall Street Journal headline on 1 July last year. "Who will live to be 100? Genetic test might tell," said National Public Radio a day later.

These and hundreds of similarly enthusiastic headlines were touting a paper in Science1 in which researchers claimed to have identified a set of genes that could predict human longevity with 77% accuracy — a finding with potentially huge implications for medicine, health policy and the economy.

[ ... ]

This resulting critical onslaught was striking — but not exceptional. Papers are increasingly being taken apart in blogs, on Twitter and on other social media within hours rather than years, and in public, rather than at small conferences or in private conversation.

Some, of course, are calling for "a new set of cultural norms."

Personally, I think the right lesson is this: The tweet beat means that researchers can't just float a trial balloon like "Evolution explains why men like dumb women"or "Religious people have embezzlement gene", and then run off to collect their tenure or whatever other reward, before anyone gets around to reading their stuff months later and shouting, "Rubbish! Compost! Anyone have a shovel handy?"

Neuroscience journal changes policy on last-minute add-ins

John Maunsell, editor-in-chief of The Journal of Neuroscience advises that his publication is ending the practice of permittting the submission of publication of additions to submitted materials online. He explains,

We recognize that this is a major change that will set The Journal apart from most neuroscience journals, but the Society for Neuroscience Council has approved this step because supplemental material has begun to undermine the peer review process in important ways. We believe that the changes described here are our best option for protecting peer review and maintaining our leadership in publishing articles of the greatest significance and highest quality. Because not all of the problems associated with supplemental material will be obvious to readers, we explain them here.

[ ... ]

Although The Journal, like most journals, currently peer reviews supplemental material, the depth of that review is questionable. Most well qualified reviewers are overburdened with requests to review manuscripts, and many feel that it is too much to ask them to also evaluate supplemental material that can be as extensive as the article itself. It is obvious to editors that most reviewers put far less effort (often no effort) into examining supplemental material. Nevertheless, we certify the supplemental material as having passed peer review.
Another troubling problem associated with supplemental material is that it encourages excessive demands from reviewers. Increasingly, reviewers insist that authors add further analyses or experiments "in the supplemental material." These additions are invariably subordinate or tangential, but they represent real work for authors and they delay publication. Such requests can be an unjustified burden on authors. In principle, editors can overrule these requests, but this represents additional work for the editors, who may fail to adequately referee this aspect of the review.

Reviewer demands in turn have encouraged authors to respond in a supplemental material arms race. Many authors feel that reviewers have become so demanding they cannot afford to pass up the opportunity to insert any supplemental material that might help immunize them against reviewers' concerns.

Supplemental material also undermines the concept of a self-contained research report by providing a place for critical material to get lost.

It probably also functions as a way to front nonsense, but Maunsell would be too polite to say so.

Peer review: Gold standard or gold in them thar hills?

I've been busy writing, and now with radio, so I haven't been blogging for a couple of days, but here is a piece I just put up elsewhere, in five parts, on peer review.

Introduction

Part One: If peer review always worked before, why doesn't it work now?

Part Two: How bad can it get? Pretty bad.

Part Three: How the system is slowly becoming more open and dynamic, whether anyone wants it to or not

Part Four:How will we know if a more open system works better?

Back to blogging tomorrow, I hope.