Wednesday, April 14, 2010

Preferred People

Some journals ask authors for a list of possible reviewers, and some also ask for a list of reviewers to avoid. Are these lists useful to editors? Who should be on these lists? If someone is listed as a person who should not be asked for a review, can the editor still ask that person for a review?

In my experience, these lists are advisory only. They are suggestions of some people with the necessary expertise to review the manuscript. If the author has particular opinions about issues other than expertise (e.g., objectivity), these may also be a factor in deciding which individuals to list as preferred reviewers and which, if any, to request not be asked to review.

If authors list the obvious reviewers, I have no problem picking at least one from their list of suggested reviewers, maybe even two if that makes sense and I would have asked these people anyway. In most cases, though, I try to find at least one reviewer who is not listed. If you want to increase the chances that a certain expert person will be selected, a possibly good strategy is to leave that name off your list. Whether this strategy is good or not depends on the editor's editorial habits. You may well not be aware of what these are, so in fact probably the best strategy, if required to provide a list of suggestions, is to make a sincere list of expert people.

As an editor, if I don't think the list of suggested reviewers is reasonable, I may not select any from the list. Some authors seem to believe that the editor will or must pick from their list, and they stack the list with their close colleagues and friends in their own department. This is not a good strategy because (1) it probably won't work; and (2) why give the editor reason to question your judgment and ethics?

There are certain research topics that involve feuding research groups, and the authors in each group believe (in some cases with justification) that those in the other group(s) will not give an objective review. These groups always list those in the other(s) in the do-not-review column. If I know something about the debate and the people involved, I either respect this request or I don't, depending on the people and the situation. In most cases, however, I respect the list, but if I do happen to select a reviewer from the do-not-review request list, I remain alert (as always) for any possible problems with the review, including apparently unsupported negative criticisms or what seem to me to be overly harsh reviews of matters for debate (as opposed to problems with the data or observations).

Reviewers do not decide whether a manuscript is accepted or rejected; they make recommendations. The editor decides whether the reviews are reasonable and makes the decisions, so a negative review may not be fatal.

The manuscript management system with which I am most familiar makes it very clear what author preferences are in terms of reviewers to select or avoid. Not long ago, however, I was a guest editor for a journal that used a different system, and I found this other system difficult to navigate. At first I didn't even see that there was a place for authors to list their reviewer preferences; the list was rather hidden. Also, the journal had an inefficient system for selecting reviewers for manuscripts in guest-edited issues, so I was unable to select reviewers directly. For two different manuscripts, reviewers who were on the authors' lists of reviewers to avoid were in fact asked to review the manuscripts. Both returned very constructive, overall positive reviews in which each also made their identity obvious to the authors.

In one case, the authors were enraged that a reviewer had been selected against their recommendations. They complained to the main editor. They wrote a rather nasty letter to me. This was puzzling because the reviewer gave them a very useful and positive review; there was absolutely nothing in that review that indicated a problem with objectivity. Did they just dislike on principle the fact that this person had been allowed to give input? There was nothing time sensitive in their manuscript; nothing that needed to be kept secret from certain people in advance of publication; nothing even very controversial. In fact, it was a solid but rather prosaic paper that I spent a huge amount of time editing because the writing was so bad. The authors were in error to believe that just by listing the objectionable person on a list meant that the editors were bound to comply. I probably would have respected their wishes had I known them and had I ultimate control over reviewer selection, but I was not compelled to do so.

In the other case, the authors were pleased that they had gotten such a positive review from someone they feared would not be objective. They later started corresponding with the reviewer about their mutual research interests, and sent a nice thank you letter commending the editorial work of my colleagues and me.

Some journals ask for a reason why an individual is included on a to-be-avoided list of reviewers; some funding agencies request this information as well. If there is a reasonable explanation for why an individual should not review a manuscript, then an editor may be more inclined to comply with this request.

It can be difficult to explain some of these reasons, though. Unless you are an established researcher with a long track record of publishing, you probably don't want to write "This person hates all of my papers and always gives a negative review" because it raises the question of whether all of your papers are really bad. Something vague like "conflict of interest", however, is an all-purpose, vague-but-possibly-effective reason, should you be required to provide one.

So:

Are these lists useful to editors? Yes and no. Not really. Sort of.

Who should be on these lists? People with expertise. People who are not your close friends and colleagues.

If someone is listed as a person who should not be asked for a review, can the editor still ask that person for a review? Yes.

14 comments:

Venkat said...

If you don't want person X as a reviewer (possibly because you know he thinks your model is wrong), I assume you'd include X in the list of persons you don't want as a reviewer.
But, will you be hesitant to do this as you are concerned that the editor may tell person X at some point (and hence bad blood between you and X)?
I know that its supposed to be confidential. But, how commonly is this trust breached in practice?

Anonymous said...

I dont get it. How did the author know that someone he/she had opposed had been selected as a reviewer? Aren't reviewers supposed to be anonymous?

mixlamalice said...

"Reviewers do not decide whether a manuscript is accepted or rejected; they make recommendations. The editor decides whether the reviews are reasonable and makes the decisions, so a negative review may not be fatal."

I guess that was what was intended when peer-review began. Nowadays, when there are 30000 journals ans when "publish or perish" rules, I am not so convinced.
I have a limited experience but have never seen the case when an editor decided against the reviewer, even when the review was obviously flawed (two examples in my short life: a one sentence review which was arguing the length of the manuscript. In another case, I got one reviewer who said "this is new but the model has to be rethought" and the other said "this is good work but not new enough to be published in this journal": both rejected it for incompatible reasons but it didn't seem to puzzle the editor).

I usually put some referees suggestions, but I do it in an honest way: I put people with expertise on the field, mainly because my "field" is rather large. I don't even think of the idea of putting my former advisor or things like that, even though I know some of my papers were reviewed by some his friends or colleagues.
The fact that I am not a very well known or powerful person makes it also easier: I don't know a lot of people, and they don't know me either. I usually don't make a list of people I do not want as reviewers...

a physicist said...

Some journals definitely honor the lists of suggested referees. There is one prominent PI I know moderately well (we are not close personal friends), and I get a lot of his papers to referee from one particular journal (perhaps 1-2 per year from the same person over several years). I hope I provide a fair critique; given the research topic and my supposed expertise, I can see why he'd suggest my name. Still, kind of amusing.

Anne M. Archibald said...

On a one paper we suggested some reviewers because our research group was so sprawling that it might have been hard to think of someone with relevant expertise who wasn't at the same institution as one of the eighteen authors. Was this concern really reasonable? That is, is there a presumed conflict of interest in reviewing the paper of anyone at the same institution?

EliRabett said...

A list of suggested reviewers can also push the editor in the direction the authors think useful (eg listing statisticians for a paper where data analysis is crucial) but might not have occurred to the editor.

chemprof said...

As an Editor of a journal that requires that authors recommend atleast 3 potential referees, I am usually disappointed at the poor recommendations made. It is rare that I can even use one of the suggestions as they are either (i) colleagues from the authors' institution, (ii) close collaborators or (iii) have no experience in the research field. As well as making my job more difficult, it always leaves me wondering if the authors really have any understanding of the field - after all if they can't name anyone who could reasonably review their work, how well do they know the field? I am also amazed at how often authors recommend reviewers who then harshly reject their papers.

My advice to all authors is to choose recommended reviewers carefully. If you make a sensible choice, then more often than not a busy Editor will use these recommendations.

John V said...

I generally assume my paper will be accepted, and suggest reviewers that have the greatest insight into the problem, the best reputation, and ones who are more passionate, whether I think they will be advocates or not.

In the long run, getting a good set of comments gives me feedback more valuable than a little less trouble in the review process.

It is not trivial to get all the right people to comment incisively on one's work outside the proposal and manuscript review process.

In addition, from experience, 90% of the maneuvering for more favorable reviews and guesswork about who did previous hatchet reviews doesn't work.

Anonymous said...

Thank you for this post, FSP. It is very relevant to some of my experiences as a newby editor.

Comrade PhysioProf said...

in their own department

For all of the journals whose editorial policies I am aware of, even being at the same institution as an author is considered an absolute conflict of interest, and a complete bar to serving as a reviewer.

Anonymous said...

I am interested in your opinion on whether reviewing a paper from your Ph.D. advisor represents a conflict. I recently got a paper from his lab, and felt pretty uncomfortable because I keep in touch with him and his current students (I actually help them a little bit with the research, although in very generic terms), and didn't feel I could be neutral. I contacted the editor, who said there was no conflict, so I did my best, but I am still disappointed he suggested my name as a reviewer. I am not the best expert in the field, and therefore it was obvious that my name was suggested by the authors.

Anonymous said...

Anonymous at 12:16, you can't necessarily assume that your advisor suggested you. If s/he did, I agree it would be strange. In my limited experience as an editor, it can been very difficult to get people to agree to review manuscripts. It is really discouraging. In some cases, I have gone through the entire list of obvious, qualified, appropriate, no-COI people and NO ONE will take it on. I then start looking at potential reviewers who are more junior, who don't have direct experience with the study system, who potentially might be too close to the authors....

I have one ms right now that I can't find anyone to review. Why? I have no idea. It looks like an easy one to me. I could review it but I'll be damned if I go down that road. I hadn't thought about the author's previous students though....hmmm....

Anonymous said...

even being at the same institution as an author is considered an absolute conflict of interest, and a complete bar to serving as a reviewer.

Over the last year I got two such requests. I promptly informed the editors about the COI and still they insisted I review the manuscript.

Kevin said...

I don't mind journals allowing authors to list reviewers they feel would not be objective---there can be all kinds of reasons apart from science why two people don't get along.

I am somewhat troubled by journals allowing authors to recommend referees, since that increases the chances of "old-boy" networks dominating the review process.

I am very upset with journals that insist on authors providing at least three referees' names in order to submit to the journal (a fairly common setting on manuscriptcentral.com). I've taken to submitting the name "Decline to Provide" with email "no-one@no-where.com".

It isn't that I couldn't think of anyone in my field (though I do have trouble remembering names), but that I regard it as inappropriate for authors to list referees.