Tuesday, February 11, 2014

Review Or Else

In a correspondence published in Nature not long ago, the associate editor of a journal grappled with the fact that some prolific authors are apparently not doing their share of reviewing. Let's assume that the review-slackers are not doing major reviewing for other journals. Let's also assume that the associate editor is not sending these authors review requests for papers quite far outside their expertise (I got one of these requests just the other day; I can't even imagine what combination of keywords and/or hallucinogens might have resulted in my name turning up as a likely reviewer on a topic so far from anything on which I have ever worked, much less published).

With those assumptions in mind, we are left to consider a person who accepts that others spend time reviewing their manuscripts but who is not willing to take the time to review the work of others. What, if anything, to do about this imbalance?

I have touched on this and related topics before without proposing a solution. In the Nature correspondence, the associate editor of a journal wonders whether a solution to the problem (described as: "the biggest consumers of peer review seem to contribute the least to the process") could be that "journals should ask senior authors to provide evidence of their contribution to peer review as a condition for considering their manuscripts."

There is much that is appealing about that idea, although in practice there are some likely complications (perhaps these could be dealt with in a flexible system). My preference, however, would be for a system that involved carrots but no sticks. I don't think we want to force people to review manuscripts just so they can accumulate enough reviewer points to get their own manuscripts into the system. That does not sound like a good recipe for thoughtful, thorough peer review.

Another reason for avoiding a system that would automatically reject the submissions of slacker-reviewers is that some of the slacker's co-authors would likely be innocent victims in such a scheme.


Is it possible to create an incentive system that is fair to all, is consistent with the philosophy of peer-review, and encourages more authors to contribute their time (thoughtfully and objectively) reviewing the work of others?

[I am ignoring in this discussion the fact that some people resent volunteering their time to feed the bank accounts of certain big publishing companies that profit from the federally-funded results of uncompensated authors and suck dry the meager budgets of university libraries, though I know some people have strong feelings about that.]

In my previous post on the topic of slacker-reviewers, I explained that I try to be as efficient as possible with the contributions of authors who are diligent reviewers. They don't get any sort of preferential treatment in terms of likelihood of acceptance of their manuscripts, but I do try to move things along for them as best I can, at least for the parts of the process I control (for example, how quickly I get to the manuscript once it is returned by reviewers). I don't know if that is enough of a "perk", but I hope it helps a bit.

I don't like slacker-reviewers, but I always try to remind myself that I don't know what their reviewing history is with other journals, I don't know what else is going on in their life, and I certainly don't want to do anything to harm their co-authors. It is not fair that some people are consumers of and not contributors to the peer-review system, but maybe we don't really want such people reviewing our papers(?).




19 comments:

nicoleandmaggie said...

I will not be reviewing for nature publishing corp until they take real steps to fix their problems with misogyny. I do not see that happening any time soon.

GMP said...

the associate editor of a journal grappled with the fact that some prolific authors are apparently not doing their share of reviewing.

You don't get very far ahead of the pack in any profession without being self-serving. I know very few highly accomplished scientists for whom "jerk" would not be a fitting description.

Anonymous said...

I had this debate with a group of friends recently. They seemed shocked that I had a 'rule' that I need to do 3 reviews for every paper I submit. Many did 1 review for every paper, etc. It had never dawned on them that the math doesn't work out unless you review more than you submit.

This rule sadly means I owe a few reviews right now due to a paper that has had to be submitted to a few places. Lesson learned about submitting to the wrong target journal...

Strung out cyclist said...

I'm unemployed, yet I still get the occasional review showing up in my inbox. What do I get from it? Oh right, I get to put a little note at the bottom of my resume.

Scientific publishing is a real racket: the institutions pay through the nose to publish papers and subscribe to the journals so that the scientists can sign away their copyright. As you've already pointed out. And as it's currently practised, it's completely obsolete.

In this internet age, why do we need separate journals for each subfield and sub-subfield? (Plus they're multiplying all the time.) Electronic indices or wiki pages or similar things can encompass the whole lot. For that matter why do we even need to submit scientific work in the form of packaged-up print articles? One could imagine submitting software, video, interactive content--the sky is the limit.

I'd like to see the shift in science to the truly electronic age. What we need is some kind of index or file-sharing system or wiki page where people can submit new scientific work and have it peer reviewed online and also collaborate with other, interested parties. One could imagine two types of review processes: formal and informal. A formal review would be just like current peer review except automated by the software. In an informal review, you'd just leave it up online and let people comment on it.

I'm thinking something like Wikipedia except for original research.

David Stern said...

I get 1-2 review requests a week from serious journals (as well as lots of spam from predatory journals). I do about 2 reviews a month which is a lot more than my fair share of reviews based on how many reviews my submissions to journals require (3*submissions/author number). But still I am sure some editors think I am a slacker on that basis.

Old Biddy said...

I've done approximately 3 reviews per submitted paper for the last few years. Before that it was probably more like 6 or more reviews per published paper, since I was in industry and couldn't publish as much.

Anonymous said...

I have never gotten a review request. I have published in the last five years but I am currently in industry - what is the experience of other industry scientists? Do we matter in this discussion?

Shaun Griffith said...

Scholarpedia has recently crossed my vision: http://www.scholarpedia.org/article/Main_Page. We'll have to wait and see if it works out.

For the current situation, I would think the time is ripe for a paper review marketplace. Perhaps a website where editors can register authors and reviewers, and enter requests for reviews. Reviewers get points for reviews, and debits for papers reviewed. What the review:submission ratio should be is a good question, and depends on the publication. Different points for different papers is also appropriate, both for reviewers and for authors.

This kind of marketplace would have a tendency to increase the number of authors on a paper, to spread the review debt.

Anonymous said...

I wonder how field-dependent this problem is. In my relatively small field, I've not heard this mentioned before, although I do know that it often takes a few tries to get a reviewer. However, most of my submissions have a confirmed reviewer within a week, maybe 2, suggesting that it's not that big of a problem. Since it's a small field, we're often acquainted with the editors, which also may make it harder for some people to say no. Finally, our journals only require 1 referee. I myself try to be reasonably cognizant of reviewing if I've been submitting, and I also have a policy of not reviewing for journals that I never submit to. I generally use the same policy for other review panels. If the norm in other fields is 3 referees per paper, then it would seem reasonable to do 3 jobs for each submission.

Maybe it would help to work on the culture in the field? For example, editorials from the professional society, etc, and have graduate programs instill more of a sense of duty and giving back? Generating discussion, for starters. The grads will soon be postdocs, and I did a lot more refereeing as a postdoc (it was more interesting and educational then, and it was before I learned to say no).

Anonymous said...

I really like the idea someone left in the comments of the original piece - journals provide carrots like free color printing, reduced open access charges, etc. that accrue as you review. From the commenter, that seems to be happening in some countries already. Seems like a great carrot and helps alleviate some of the 'predatory' criticism (that journals are making money off the free work provided by authors and reviewers).

EliRabett said...

Money talks

inBetween said...

One of my most successful and prominent colleagues claims that they can't review papers because they are dyslexic. This dyslexia doesn't seem to keep them from reading the literature or writing papers, but apparently, completely inhibits them from reviewing manuscripts or, for that matter, helping with departmental service. I think the situation is more like GMP points out... this person is really a selfish jerk and is rewarded well for it.

pyrope said...

I frequently these days have to send out 12+ requests to get two reviewers to agree on a paper...and this is for a top tier disciplinary journal.
I find that I can get higher agreement if I scrounge out the postdocs, but then also a higher likelihood of getting a super bitchy review...which is generally unfair for the authors. (as an aside, I was a terrible reviewer as a postdoc too, it's just part of the process).
Some journals that I know give discounts on color figures in future publications...the times when I could have taken advantage of that, I forgot to.

Anonymous said...

inBetween suggested that a dyslexic person who turns down reviewing jobs for that reason "is a really selfish jerk". That's pretty harsh, considering that most dyslexics must spend far more time on reading/writing than other people. Just because this person can manage with reading and writing enough to succeed doesn't mean that refereeing wouldn't be an unreasonably onerous task for them.

Anonymous said...

I have a question for those of you that are editors... I'm a postdoc and so far I have accepted every review I've been offered (which isn't that many). However I tend to... procrastinate/be busy, alternatingly... and usually get my review in just around the deadline.

Am I a slacker reviewer? Should I try for a faster turn around, or am I actually doing the right thing by not seeming TOO eager?

Female Science Professor said...

You are most definitely not a slacker reviewer.

Anonymous said...

I think the best solution would be to make you an associate editor after X reviews, for a suitable number X depending on the field and the venue. If a journal trusts enough your judgement to keep coming back to you for X times, then surely you can be an editor too.

Anonymous said...

In addition to important points Anonymous (6 p.m.) makes on InBetween's post, the colleague's dyslexia may seriously impede their ability to do an adequate job reviewing a paper, given the kind of attention to detail required. In addition, you don't know what kinds of assistance this colleague uses for keeping up with the literature (for instance, text-to-voice software) or writing papers (help from friends or spouse with proofing).

Anonymous said...

Just did the math, and I've done 5-6 reviews per first-authored paper I've published over the last 5 years. As time has passed I've also turned down more and more reviews as I get asked more and more.

So even though I'm doing my fair share by any metric, I'm now also turning down about 80% of all review requests. It might seem to editors that I'm shirking my duty, but the truth is that now I only review papers that I would read anyway when they get published, and only for fairly well-known journals.

I know a colleague who accepts all reviews and reviews exactly one paper a week. New requests are told the waiting time for the review and can wait or decline.

I can only imagine how many review requests the eminent people in my field get: probably hundreds every year.