Thursday, February 10, 2011

Wearing Two Hats: Editor & Reviewer

A few times since I have been a journal editor or guest editor, authors of rejected manuscripts have written angry e-mails railing against the injustice of the negative decision by me or by one of my fellow editors. I have written about various aspects of this before, but today I was thinking about one of the reasons that some rejected authors focus on as evidence that their article was misjudged/mishandled: the editor also acted as a reviewer.

In most cases, this particular reason should not be a cause for anger. In fact, in some cases, you may be lucky that an editor also acted as a reviewer.

Common reasons why an editor would also act as reviewer are: (1) it was impossible to get enough reviewers to commit to a review owing to (a) random bad luck/timing, (b) the topic of the paper was so highly specialized that the reviewer pool was small; or (2) one or more agreed reviewers didn't return a review and an editor decided not to hold the review process up any longer. In the case of having an insufficient number of reviews, an editor can make a decision based on the review(s) in hand, but can also decide to act as an additional reviewer.

How does this work? What are the implications of having a review by someone who acts as reviewer and editor?

I can only speak about my own experiences, but from what I've seen and done, this means that the editor provides more detailed and substantive comments on the article, rather than mostly providing context and guidance based on the reviews provided by others. Of course, some editors routinely act as a reviewer in this way as well, no matter how many reviews were received.

When my editor-colleagues and I act as reviewers in addition to our editorial roles, we inform the author that we are doing this and sign the review that is ours. It would make no sense to provide an anonymous review and then agree with it as editor. We also inform the author as to the reason for our "reviewing" the manuscript -- again, this is typically because we couldn't get a sufficient number of reviews in a reasonable amount of time.

Understandably, some authors may blame an editor for a rejection, so the fact that the editor was reviewer and ultimate decider makes it seem like the deck was stacked. In fact, from what I've seen in journals with which I have been associated, manuscripts that are reviewed by editors are not rejected at a higher rate than those in which there are multiple reviews by non-editors.

I doubt if editors act as reviewers to make sure that a particular paper is rejected. If an editor wanted to sink a paper, there are much more efficient ways to do this than to spend the extra time required to expand editorial comments into a review.

Speaking again from my own experience, editor-reviewers are likely to provide useful reviews because we know what constitutes a constructive review. And if you don't get a detailed, substantive review from an editor/reviewer, the manuscript probably didn't have much of a chance to be accepted anyway.

The most recent incident involving an angry almost-author was not directed at me, but at one of my hardest-working, most diligent, and most thorough fellow editors. If this particular editor takes the time to act as a reviewer and editor, the author is actually lucky to get the extra attention. The author is assured of a fair and thoughtful review from this editor.

The fact that it didn't turn out well for the author in this case is unfortunate, but I would advise authors to wait a little while and calm down before firing off a rude e-mail accusing an editor of incompetence because s/he was unable to find enough reviewers for a manuscript. It would be better to consider carefully other possible reasons for the rejection of the manuscript (and difficulty in finding reviewers) before focusing on the dual editor/reviewer role as the favored explanation for the rejection.

20 comments:

Dr. Confused said...

In my field, it's typical to get three reviewers, sometimes two if one is too slow. When I published my PhD work, the journal paper came back with four reviews. One consisted of just "You should add citations to these papers: (insert a few papers on really quite distantly related topic by associate editor for my sub-field of the journal field)". My supervisor rolled her eyes but we twisted the lit review and put in the mostly irrelevant citations.

So you may be right for your journal, but there is at least the potential there for some conflict of interest.

SoloGen said...

The possible problem is the framing of other reviews based on the editor-reviewer's own review. In this case, the editor "interprets" other reviews in a way that is biased.
I expect that there are not many editor-reviewers who "reject" the paper when they act as the reviewer but ultimately accept the paper because other reviews have been positive.

Female Science Professor said...

Here's an example from the last time I was both editor and reviewer of a ms: I did not yet have the minimum number of reviews in hand for a ms, but a late reviewer said he could send the review in 2 weeks. He said that he had the review mostly done and just need a bit more time but he was traveling and couldn't finish it right now. I said OK. Two weeks later, he asked for another week (or two), with a promise that this was the last delay. He again said that the review was almost done. I asked him to upload his existing review, even if he felt it was incomplete, rather than delay further. He did not want to do this. A week later, he said it would take a few more days. Then he said "no later than tomorrow". The next day, it was "this afternoon or evening". After that, he stopped responding to my e-mails and phone calls. Nothing. I waited a bit. Nothing. What to do? Start the review process again with an entirely new reviewer, with the possibility of dragging out the review for months? I did the review. The late reviewer never sent any review at all.

As an author in a case like that, would anyone rather wait for another independent review rather than having a review by the editor? Does your answer depend entirely on whether the editor recommends acceptance or rejection?

Anonymous said...

I'm always happy to read these posts on your experiences as an editor. It makes me feel like maybe I'm not entirely hopeless, as a relatively new associate editor. I have already reviewed many manuscripts that I have handled as an editor, because I can't get a second review in reasonable time, because the two reviews strongly disagree, because both reviews are superficial, because I suspect the ms might not be a good fit for the journal but lack confidence that the reviewers will pick up on this. My input ranges from a formal review to "In addition to the reviewers comments, you also need to address X, Y, and Z." In fact, right now on my to do list is "Just review the damn thing myself."

And seriously. Bias? This is nothing compared to potential bias when I decide if the ms goes out to review, when I pick the reviewers (including reviewers that you oppose), when I give the reviewers special instructions, when I interpret the reviews and make the final decision. That's the bias you should worry about!

Monisha said...

I serve as an associate editor for a journal in my field; when i was recruited to the position i was told that the editor-in-chief wanted 'strong editorial hands' - meaning that i am expected to make decisions based on my own review and the outside reviewers. In my own career, I have always preferred this experience as an author, because it does so well at creating clarity amidst conflicting reviews. As an editor, I like it because sometimes reviewers miss a larger, big-picture issue, particularly if they are very closely engaged in the authors field, and it allows me to raise that issue. I have not received angry tirades over my few years in this job, although I did get an angry reaction to a manuscript that I triaged.

Anonymous said...

SoloGen,

I write fence-sitting reviews in a way that requires the editor to make a decision based on their interpretation of either the paper or the journal or, possibly, both. This is particularly true for "applied" papers that are "theoretically applicable" at best. I leave it to the editor to make an editorial decision. I believe that is their job and that some subsequently rejected authors would say the editor was "biased" in their interpretation of my mostly positive review.

Anonymous said...

A week later, he said it would take a few more days. Then he said "no later than tomorrow". The next day, it was "this afternoon or evening". After that, he stopped responding to my e-mails and phone calls. Nothing. I waited a bit. Nothing. What to do?


Why didn't you call the Dept and ask to speak to the Professor? (pretend to be a student so he would pick up)

Would be funny to chat to the guy in person or if you can get him on the phone?

"What happened?"

Anonymous said...

@Anonymous above: you are a very unhelpful referee, why don't you just tell the editor you don't want to do the job, rather wasting your time giving he/she no information on which to make a decision. The best reviewer is one who gives a firm *opinion* backewd up of course by evidence. The editor's job is to evaluate your evidence, and hence your opinion, and that of the other referees. Not to start again from scratch.

Anonymous said...

FSP: When I review papers for journals where I usually publish, I always wonder if the fact that I'm a good reviewer (on time and constructive) helps me with the editors whenever I make my own submissions. I am not implying I am expecting a freebie, but I always wonder if editors are more patient with authors that are good reviewers for the journal and less patient with people like the one you described in your comment.

Female Science Professor said...

That's a great topic for a post.. maybe next week.

SoloGen said...

FSM: Even though I haven't been an associate editor for a journal, but I can imagine the frustration of such a situation, which I believe is far from being rare. That's yet another reason your job as an associated editor is difficult, and I empathize.

Regarding your question, I would prefer the editor to write a positive review than to wait for much longer. But my interest is not the same as the interest of the peer-reviewing system.


Anonymous (2/10/2011 10:43:00 AM): I agree that there are various sources of bias. The goal of a peer-reviewing system, one could say, is to have a low-bias, low-variance assessment of the quality of the paper. Other sources of bias are existent for sure.

Let's see the situation here: Roughly speaking, adding the number of reviewers (including the editor herself) decreases the variance of the final result, but it adds to the bias. Which is the more dominant term?

I think one can empirically study it in a simulated peer-reviewing process.
A simple example might be like this: The researcher picks a text that states some facts. He asks reviewers to read it and answer a questionnaire. Then we have two scenarios: In the first, the editor reads the text then reads the answers to the questionnaire from the reviewers, and then fills the questionnaire herself. In the second experiment, the editors reads the text, answers the questionnaire herself, reads the questionnaire of the others, and then re-answer the questionnaire.
If there are objective answers to the questions, then one can potentially measure the effect of this bias.



Anonymous (2/10/2011; 11:01:00AM): I almost do the same. When I review, I see my job as evaluating pros and cons of the paper and providing evidence for them. Then the goal of the editor, I assume, is to compare my evidence with other reviewers', and aggregate them in an unbiased way. The editor should understand the paper to the extent that she can understand the points of the reviewers, but having a personal opinion changes everything in an unpleasant way.
It is like asking the judge to be the prosecutor or defense lawyer too.

mathgirl said...

Continuing with the topic of anonymous 11:01AM and 12:33AM, I have a question.

As a reviewer, I try to give an opinion whenever I can. Sometimes it's hard to give an opinion because even though I understand the paper, and I can check it's correct, it's hard to me to judge whether it's appropriate for the particular journal. I still think such a review is helpful, since I took the time to check all the math in it, something that is not trivial at all (if it's trivial, then it's safe to say that the paper should be rejected!)
Am I wrong in thinking this?

In other words: Is a detailed review that gives no final decision useless?

Warlaw said...

I'd love to be the reviewer and editor of my papers. I'd give myself an A+ every time haha. It's too bad I have to work with B's and C's, but I guess that's life. Also, I think added unneeded citations is kind of a waste of time, but what can you do? Anyway, great read.

Anonymous said...

Anon at 12:33:00 PM,

I am very happy and capable to judge the merits of a paper but not necessarily the alignment of said paper with the editorial guidelines of the journal. I see nothing wrong with saying "The analysis in this paper is solid and interesting and novel but the topic is pure conjecture so I am not sure if it is suitable for publishing in Journal of Concrete Applied Science". The editor can then make an editorial decision about suitability. Many journals that I regularly publish in (not just referee for) change their editorial slant faster than I notice.

But, as I indicated, I have seen responses from rejected authors saying "Editor is biased! Referee said the analysis was novel ... "

Anonymous said...

As ANON 11:01:00 AM,

I should point out that not all of my reviews are fence sitting. I simply mean that when I do need to sit on the fence I am providing information about the paper but expecting the editor to make a decision based on an opinion about the journal in addition to my conclusions about the paper.

My bad for being too brusque earlier!

Fahad Saeed said...

"If an editor wanted to sink a paper, there are much more efficient ways to do this than to spend the extra time required to expand editorial comments into a review"
FSP, it would be interesting to hear about this in one of your blogs;

Female Science Professor said...

OK, I'll do one or more editor-themed posts next week.

Anonymous said...

Do you ever get such complaints from female academics? What is the gender distribution of complainers would you say?

Anonymous said...

After waiting forever and a day for initial reviews and then re-reviews, I would kill for an editorial review and a decision...

Funny Researcher said...

May be the reviewers can sink the paper as well, just because of competition?