In my role as editor, I have the ability to rescind a review that is 'inappropriate', and the definition of inappropriate is left to my discretion. I have never used this option, and have chosen to deal with the occasional inappropriate review comments in other ways.
For example, in a recent case, an otherwise acceptable review in terms of content and tone had a completely inappropriate statement that demonstrated the discomfort of the reviewer, who was from a western European country, with the government of the author's native country, which is east of western Europe. The review comment had absolutely nothing to do with the scientific content of the manuscript, and the reviewer's recommendation was bizarre. I sent the review to the manuscript authors, but mentioned in my editor's comments that the authors could disregard that particular recommendation by the reviewer.
I recently heard about a more egregious case than anything I have encountered. If I ever received such a review as an editor, I would rescind it. In this case, a reviewer, angry that the manuscript under review disagreed with some of his prior work on the topic, recommended that the manuscript be rejected and included an explicit racist comment about the first author. The editor rejected the manuscript on the basis of this review.
I'm not involved in the research and have no direct interest in the research, but I am familiar with the work in the rejected manuscript and know it to be of high quality. It is shocking that the manuscript was rejected on the basis of a review that clearly demonstrated an extreme lack of objectivity. Just as shocking is that an editor would let an explicitly racist review be sent to the authors.
This is not about political correctness. This is about being ethical. This is about being professional. This is not a marginal case in which a reviewer unfortunately chose to express himself in an awkward way. This is a disgusting case of a reviewer who was so lacking in objectivity about a perceived assault on his exalted reputation that he responded by denigrating an author's ethnicity. And he somehow felt comfortable doing so. The editor's behavior confirms that the reviewer was justified in this feeling of comfort.
I think that the peer-review system works quite well overall, but there are instances that seem to argue otherwise.
Reviewers and manuscript authors don't have many 'rights'. Authors would like to have the right to a rapid and fair review. Reviewers would like to have the right to have their comments taken seriously by authors and editors. In real life, the review process can take days to years; reviewers may or may not be as objective as they could be; authors may or may not take seriously the comments of reviewers when revising manuscripts; and editors may make decisions by flipping a coin or consulting their pet beagle.
I think that reviewers who include explicitly offensive comments about an author's race or other personal characteristics (gender, religion etc.) lose the 'right' to have their review sent to the author. This seems reasonable to me, and doesn't infringe at all on the time-honored tradition of implicitly or explicitly insulting the intelligence of the author.
14 years ago
22 comments:
Dear FSP,
I meant to write you a while ago to ask your opinion about the peer-reviewed system as oppose to open access to everyone's preprints (I'm thinking of the ArXiv) and this post seems like a good opportunity to do so.
I don't know what is your exact field of expertise, and maybe you don't want to answer this post because your answer might require you to give that information up, but I would be very interested in hearing your opinion.
In my field (Theoretical Physics) there is a wide usage of the ArXiv. As you might know, this means that anyone in the community can upload their preprints there and everyone else has immediate open and free access to it. This system has by now been accepted by the journals as well and they are happy to accept papers which have previously appeared on the ArXiv. In my opinion this has the following advantages/disadvantages:
On the plus side communication of results to the world is much faster than through the review system. It also allows you to gauge the impact of your result in a very short time and receive comments, ideas, etc as you go on your research. On top of that is a free service, which is important especially for people who work in medium size/small universities that do not have enough resources to subscribe all journals. This last point worries me because I find that some journals charge extremely high fees for providing services where most of the job is done by the academia community for free (as you know reviewers are not paid to do their job and certainly scientists are not paid by the journals to write papers).
On the minus side, sometimes this leads to people uploading to the ArXiv the first idea that crosses their mind without enough thinking about it. The high speed at which the systems moves due to the ArXiv makes people fear that they will be scooped at any time and they rush to publish their results. Also the lack of peer-reviewed system means that you have to be more critical when you read the pre-prints because there might be a higher density of mistakes/incorrections that in a paper published on a journal.
On the last point though, I think that on the ArXiv the whole community peer-reviews each paper. If a paper is not very good, then it will not receive much attention in subsecuent papers and so on. Of course this process does not always work. For example there might be very good papers on topics that very few people are working on and hence they would have very few citations. with this I mean that not always the number of citations is an indication of the quality of the paper, but the fact that everybody can in principle peer-review any other paper might solve some of the problems you point out in your post today about peer-reviewed systems.
As you can see I am a bit confused as to which system is best and that is why I would appreciate very much your opinion on the topic. I apologize for a far too long comment, but this topic has worried me in the last weeks and I wanted to discuss it with other scientists outside of my department to see what you think.
Thank you very much for your blog, it is very inspiring and helpful,
a Female PhD Student
A reviewer who makes a racial attack is clearly unethical. But it is not uncommon for reviewers to diss papers that disagree with them or even just do something in or near their territory. Most of these reviewers are careful not to use racial or political slurs and their reviews are taken seriously.
At least it would seem that if the authors receive the review that includes the racist reference, they have the ability to appeal the rejection of the manuscript on the basis of bias of one of the reviewers, and request another reviewer when they resubmit? Or am I naive? If it is known who the reviewer is, it would seem as if the solution would be to blacklist the reviewer by all editors.
Chem Prof.
If I were the rejected author, on the basis of a racist review, I would threaten to sue the editor and the journal. Simple as that. Next they'll be able to turn down papers because of any other non-scientific reason, if that's allowed to persist.
I think that reviewers who include explicitly offensive comments about an author's race or other personal characteristics (gender, religion etc.) lose the 'right' to have their review sent to the author. This seems reasonable to me, and doesn't infringe at all on the time-honored tradition of implicitly or explicitly insulting the intelligence of the author.
Not only that, but in my opinion, editors should forward such reviews to the provosts (or other responsible officials) of the reviewers' institutions, as well as the presidents of the reviewers' scientific societies. The authors should also be informed of the name of the reviewer, so that they can seek to ensure that appropriate action is taken by the reviewer's institution and scientific society.
If you make blatantly racist, misogynist, or otherwise discriminatory remarks in a review, you forfeit your right to anonymity. Reviewers who do this, or those who may be tempted to do so, need to be made aware of the fact that they cannot hide explicit improper discrimination behind the cloak of anonymous peer review.
I do not think peer review works at all.
In the top tier journal in my field, the reviewer sees the authors' names, but the authors do not see the reviewers' names.
I think the identities should be masked in both directions or not at all. Otherwise, it is too easy for the reviewer to be biased, even if the bias is not conscious.
As an example, I made a small discovery in my PhD work that contradicted what was generally thought to be true. It was not earth-shattering, just one little part of the puzzle, but it was novel and unexpected.
The paper was rejected because one reviewer basically said s/he did not beleive it. We were not given the chance to share our data.
It is such a crapshoot. Anyone who has been around a while has had the experience of seeing something you really dig rejected and some little thing you tossed off accepted.
Grrrr. Write to the editor. If they are going to send those comments to the authors, they have the duty to publish them. With a very strongly worded dismissal to that reviewer from all future reviewing duties. And hopefully the entire field will see it and refuse to work with such scum. I'm disugusted.
(p.s. yes, I know the world doesn't work this way. I'm just very bothered there are no clear consequences for this kind of behavior)
zoelouise, masking names in both directions was tried in American Mathematical Society journals about 30 years ago. It did not work, since very often the combination of subject matter and writing style made the author's identity obvious to the referee. Maybe it could work in a field with a much larger population of workers.
"I sent the review to the manuscript authors, but mentioned in my editor's comments that the authors could disregard that particular recommendation by the reviewer."
Is it alright not to include parts of a review? I would have probably sent in the review to the authors, but deleted that comment, and said so to the reviewer. Why would sending them a questionable comment that has absolutely nothing to do with science and is distracting help in any way? Unless you are required to keep reviews in their entirety.
I'm sorry zoelouise, but your comment doesn't make any sense. It is clear to whomever reads a manuscript, even if they don't have the names given to them who the submitting author is based upon the fact that most people reference themselves to provide background to their current papers.
I think the anonymous reviewer system should stay in place is it allows younger faculty to honestly and critically review a manuscript without fear of being black-listed or alienated by a more well known player in their field.
However, I am actually shocked that a reviewer, regardless of what they actually think in their head, would include a racist comment in a review. This is definitely something that should be brought to the attention of the editor of the journal. That being said though, based upon years of living in europe, the PC culture has not been pounded into the heads of a lot of europeans as of yet and said reviewer might not even be aware that what he/she commented was totally inappropriate.
Don't be fooled by how a lot of Europe is portrayed; prejudices against people based upon their nationality, gender, sexual orientation etc are alive and well and in my experience more accepted than in the US. I'm not saying that Americans don't think or vocalize the same things but there is definitely more of a PC culture in the US of what is acceptable to say out loud without being judged harshly for it.
This is an incredibly good post, and I'm not surprised by the numerous comments it has stirred up.
I have had several experiences, on both sides, which I have questions/comments about and I would like your insights (the general blog community).
1) I recently (June 3) submitted my first manuscript without my thesis adviser's name on it. Note: I am now an Assistant Professor, no longer in his group and the work was not done in collaboration with him nor did he contribute to the development of the idea. Still, I requested that the manuscript not be sent to him for review. 1 wk ago, I heard from a former colleague that my adviser was "very upset" that I was publishing without him. I contacted the journal to confirm that the manuscript wasn't sent to him (per my request), and it wasn't. Which means that one of the reviewers (there are three) contacted my adviser and told him about the paper. Now, I don't know the spirit of the reviewer's contact - maybe he thought my adviser would be proud... But, his contact still violates the concept of confidential review. And now I have angry - though passive/agressive - adviser to deal with. Advice?
2) While I agree with the concept of open review - everyone knowing who everyone is. I think it might be even better if the whole process was blind. The science in the paper is what should matter, not the scientists performing the work. After reading journals like Nature and Science, it often seems like once you publish once, the "bar to entry" gets lowered. This simply isn't right - the bar should be same every time. The only way to maintain this threshold is to have a truly blind review.
In any case, I read your blog everyday, and I really appreciate all of the time you put into it. I have gotten a lot of advice out of it on topics like choosing graduate students, navigating the academic culture and dealing with students in classes. As the only female professor in my department, your blog is invaluable. Thanks.
I agree with PhysioProf that this is such an egregious deviation from acceptable practice that there should have been additional consequences for the reviewer. I am not sure notifying the University is appropriate, given the desire to maintain anonimity in review, but a permanent ban from reviwing for that journal would certainly be warranted. I also think it's beyond the pale that an editor would accept such a review.
However, I also think this is a fairly unusual case that should not be viewed as part of the normal review process. As a monitoring editor of a journal in my field for a number of years, I have never seen any review that included any sort of personal attack. That doesn't mean bias doesn't enter the process, but it is much more likely to be scientific bias or personal animosity translated into scientific terms. That sort of behavior, while also regretable, is not in the same league as a racist comment, or even a comment that is explicitly political rather than scientific.
Mark P
The comment by your reviewers was non-germane and hurtful. You passed it on and therefore assumed responsibility for it. You had several choices.
1. Can the review and get another reviewer
2. Excise the non-germane slur and send the review on after carefully checking that the review was correct.
3. What you did.
1 in my opinion was the best choice because it removed any doubt of bias. 2. probably would have been ok, but what you did was both hurtful and suspicion generating.
In my field, philosophy, papers are almost always double-blind reviewed. In submitting a manuscript, one must remove all self-identifying references, the list of acknowledgements, etc. Being an unestablished female, I'm very happy to have this protection in place. But I understand why this might not be realistic for the sciences, where there are multiple authors and labs are easily identified. But then there ought to be some other kind of protection in place. Just counting on the good will of individual editors is not enough.
To clarify: In the example I gave in which I was involved as editor, the irrelevant comment was not "hurtful" or a "slur". The comment was a recommendation for a change to the wording of some text, and was not a personal comment directed at the authors. I cannot excise single comments from a review anyway; it's all or nothing. This review was constructive overall, but had one stupid comment that I was able to deal with in my editorial letter. If the comment had been a slur, I would have rescinded the review.
And now I have angry - though passive/agressive - adviser to deal with. Advice?
Tell him to grow the fuck up. You are an assistant professor running your own research program, and your former advisor has no proprietary interest whatsoever in the work that you have done independently of him since you left his lab. And tell him PhysioProf said to grow the fuck up, too!
I don't know if anyone else had this experience...I am a female math physicist and have got referee reports with lines like `He should change lines so-and-so' or `he should include x paper as reference'.....! Needless to say, these reports were for single-author papers written by me. The referees apparently could not imagine a woman writing them. Perhaps journals ought to formulate a set of rules referees should be mindful of, irrespective of their personal biases.
Andrea: Your former advisor should be happy for you that you are submitting independently and that things are working out for you.
If they want you to be successful in your tenure process, they should be allowing you to distance yourself not crying because you aren't.
The authors who received the review with the racist comment should complain to the editor-in-chief of the journal. If it were me, I would send it to the board as well. That is beyond unacceptable.
The editorial policy of the Astrophysical Journal, for example, says that the scientific editor may remove identifying information and "inappropriate remarks" from the review before forwarding it to the author. I believe this includes inappropriate personal criticism and would cover comments such as the one having to do with the east-of-western-Europe government. However, excessively personal comments, let alone racist ones, should disqualify the referee outright.
Regarding the first comment and the arXiv, I think that even though people often put manuscripts on it before submission, they do at least some clean-up of the manuscripts to improve their chances of getting past the referee. That is, if we had just the arXiv and no peer review, I think the manuscripts would be sloppier. In that sense, there is not a choice of arXiv versus peer review; they complement each other.
FSP,
Responding just to your bit about the lack of rights of reviewers and authors:
For what it's worth, some people have tried to codify some rights. Computing's main professional organization, the ACM, worked on this, and the result was an article in CACM (the ACM's flagship rag—emphasis on rag, though some sharp people are trying to fix that now). Here's the document, and here's the same with very useful commentary. (Sadly, you need a subscription to view the latter, but the former link contains the policy.) The policy establishes rights and responsibilities for everyone you can imagine, except the poor printer's devil.
Needless to say, virtually nobody even knows of the existence of this document. For instance, one of the rights in it is the problem with the time for fixing journal galleys (as the commentary says, “The time needed to fix galleys is a pet peeve of authors”, and goes on to describe a depressing familiar situation). Yet I was harrassed by the ACM's own chosen publisher, for a paper at the flagship ACM journal in my area, in direct violation of this policy. When I complained (five days to proof a 40-page document with lots of mathematics...while I was abroad with a tiny laptop screen), the publisher essentially told me to stop whining.
I ended up on that very journal's editorial board some months later. I've tried to complain about this. Everyone is sympathetic; nobody expects anything to happen.
But hopefully any computer scientists reading this blog who are in a similar situation will find the above links useful. At least the ACM's intent is in the right place.
"I cannot excise single comments from a review anyway; it's all or nothing."
Well, then that is a bad policy that is set by the journal. As a later commenter noted, it's not the same policy in all scientific journals accross the board; it should be possible to change it. This just gives an excuse for reviewers to include personal or political messages and get away with it in the future under the cover of a 'meaningful review'. Since you let them get away with it, you can expect more of the same in the future.
Just saying.
I recently had a paper rejected. The letter from the editor was moderately encouraging and made reference to three reviews. I only got two reviews. One was neutral-negative, short and superficial; the other one was glowingly positive. I replied to the decision email saying I was missing one review, and got back the missing review from the journal manager. Oh boy. It was nasty. I wonder if the editor made the decision to not send it in the first place because of its bigoted nature, and I wonder to what extent the editor's decision to reject was based on that review.
Post a Comment