Tuesday, April 13, 2010

Let's Review

Why review manuscripts?

Reviewing can be a deeply unsatisfying experience if you end up mired in bad writing, spending a lot of time making detailed and insightful comments for an editor and authors who will ignore all of your criticisms and suggestions. Reviewing can also be a very positive experience in which you help improve the content and presentation of a possibly interesting and important (or at least useful) paper, learning something in the process, and accumulating the respect of authors and editors.

In the comments of yesterday's post, Dr. Doyenne provided a nice list of reasons why reviewing can be worthwhile for the reviewer. You should check out that list, but I have also included some of the items, in modified and supplemented form, here. Some of them are most applicable to new researchers, but can apply to reviewers at any career stage.

1. You learn a lot useful things about writing, science (or whatever), and publishing. You may even learn more about the technical aspects of manuscript writing, organizing, and publishing from a bad manuscript than from a good one, so don't assume that you've wasted your time just because the manuscript your reviewed was a travesty.

2. You will learn what is appropriate (or at least typical) in terms of length, level of detail, and subject matter for particular journals. You should get an idea about these things by reading published papers, but reviewing can help make all this more clear. For example, a common criticism of manuscripts submitted by recent PhD students is that a manuscript "reads too much like a thesis chapter". This isn't an issue if you write your thesis chapters as manuscripts, but in some cases the thesis contains much more detail than a manuscript should. Even though doctoral students surely read many published articles and should get some idea of what is an acceptable level of background material, somehow it is different when you are writing your own manuscript. If you start looking at manuscripts (even your own) through the eyes of a reviewer, you get better calibrated, saving yourself some criticism when you submit your own manuscripts.

Furthermore, some journals provide a checklist or series of questions for reviewers to fill out as part of the review. As an editor I find these lists rather useless, but they do provide some structure and continuity to a review. For the reviewer, these may give an idea for what kinds of things are considered important, in general or for a particular journal. You may also learn about this kind of thing from other means during the review process, and these may help you organize and target your own manuscripts better.

3. Once you have some experience reviewing, you can better deal with reviews of your own manuscripts, especially if there are (very) negative comments. If you are encouraged to resubmit a manuscript, you may have a better idea for how to craft the letter to the editor, explaining any reviewer comments you decided to ignore. If the editor knows you as a diligent and thoughtful reviewer, this opinion might help you in discussions with that editor about the fate of your own manuscripts.

4. If you are a reliable and thoughtful reviewer, editors may return the favor by trying to make the manuscript review process as efficient as possible for you. If you submit a lousy manuscript, the editors are unlikely to give you a break just because you are a good reviewer, but a grateful editor might feel inclined to handle your manuscript soon after the reviews come in rather than letting it languish in their inbox for a while.

5. In some fields, for some journals, editors are influential people and it can be a good thing to come to their attention as a bright and thoughtful person. This can be helpful not just with your own manuscript submissions, but just in general in terms of your visibility and professional standing in your field.

Note, however, that not everyone shares this view that editors can be important people in their fields. A few years ago, I was in a committee meeting during which one of my fellow committee members, an outspoken professor with strong opinions, tried to sink the nomination of a particular candidate for an award by noting that the candidate was an editor of a journal and this meant that the candidate was a "has-been" whose research career was so washed up that he was now an editor instead of a real researcher. He announced that most editors are "losers". There was silence in the room, so he said "I'm guessing that there are a lot of editors in this room and that you all now hate me." I said "I don't know about the others, but I'm an editor and I hate you." (In fact, I didn't hate him at all, but it felt good to say it.)

6. And finally, reviewing is considered Professional Service, a component of a faculty job at many institutions. You can list on your CV or faculty activity report or tenure dossier the journals for which you have done reviews. If editors are requesting reviews from you, this is a sign of professional visibility, and in many fields that is a good thing.

As I've written before, if you submit manuscripts, you should review manuscripts. If you review manuscripts, you should do a careful and constructive job, finding a good balance between sharing your expertise and not sharing any irritation or biases that are not appropriate to display in a review. If you are angry at an author because they gave you a bad review or you hate their former adviser or whatever, don't do the review.

I hope this is a semi-convincing list of the various reasons why reviewing can be useful for the reviewer beyond the content of the manuscript. These reasons will be cold comfort when you are deep into a ghastly manuscript, but realize that somewhere, somehow, someday, it is better to be a constructive and thorough reviewer than a mean and/or lazy reviewer.

15 comments:

Kevin said...

I agree with most of your reasons for being a reviewer, but I find the checklists provided by journals seriously counter-productive as a reviewer. They rarely ask questions that are relevant to the paper being reviewed.

Even more seriously, I object to the journals that require authors to submit the names of reviewers. It is a serious flaw in the peer review process if the author gets to nominate the reviewers. (I realize I'm in the minority in holding this ethical position---I couldn't convince the rest of the editorial board I'm on to stop the practice for even one journal.)

cookingwithsolvents said...

thank you for providing another incredibly useful series of posts.

RE: listing reviewing activities on the CV.

How much detail?

Reviewer: Journal X, Journal Y

vs

Reviewed abc articles for Journal X (2009-present, asdf per year)
Reviewed xyz articles for Journal Y (2010-present, asdf per year)

Or somewhere in between?

Thank you!

Anonymous said...

Kevin, I suspect that many editors ignore the list of suggested reviewers. I know I do (I already ranted about this yesterday). It has the potential to be very helpful in moving the ms along, but too many authors abuse it.

Mitch said...

Do other fields of science require reviewers to assign numerical scores to manuscripts? Several of the journals I've reviewed for require a score between 0 and 100 with no guidance whatsoever as to how to determine this score. I find that requirement incredibly silly and can't imagine the editors find it useful.

Anonymous said...

Kevin/anon@6:52 - I am also an editor and I also fairly routinely ignore and/or consider with great care any referee suggestions made by authors. They can be useful starting points (in searching for relevant reviewers), and useful for drawing new people to our attention in general, but that's about it.

Anonymous said...

I have heard that some editors (or journals?) will edit reviews before sending them back to the authors, specifically removing inappropriate content.

This strikes me as not entirely ethical, though I suppose the journals get to decide their own procedures. I guess. Does anyone have any insight into this? How common it is?

Anonymous said...

I don't like the practice of suggesting reviewers. I hope that in my field and/or for my editors it is just a formality/guideline and not really taken seriously.

Maybe I like to make my life harder for myself, but when I submit papers, I prefer to have truly honest opinions of my work show up in my inbox. If Famous Prof. Y doesn't like my paper, I'm happy to take into consideration any and all comments that he/she may have. I see it as a way to become a better scientist.

In my opinion, if X suggests Y because they are former colleagues and mutually agree to give each other good reviews, we all lose. Unfortunately, I have seen this happen.

And @Anon 9:18, I *wish* my editors would edit some of the reviews that I get. A reviewer's use of 50 cent words in attempt to insult me is neither useful nor productive.

amy said...

Reviewing papers has been very helpful for me, for all the reasons mentioned. I only wish I had started earlier. My dissertation committee members were not very good about training students in practical, professional activities, so they never showed me any papers they were reviewing, nor recommended me as a possible reviewer for any papers. For my first several years as an asst. prof. I received no manuscripts to review. At some point I mentioned this to an older person in the profession and she was shocked. Within a couple of months I started receiving manuscripts -- I think she must have contacted some editors she knew and told them that I would be a good reviewer. Anyway, I learned very quickly what makes a publishable paper (as opposed to a dissertation chapter), and it really helped me with my own papers, which I had been having a lot of trouble getting published.

Anonymous said...

When I review an article, I submit two files: (1) a recommendation to the editor, and (2) comments for the author. The authors receive only the second document.

(I'm not sure how common this is, but it was the advice I was given for my first review assignment.)

someone said...

About self-plagiarism: you can read the paper C. Collberg and S. G. Kobourov, "Self-Plagiarism in Computer Science," Communications of the ACM, v. 48, no. 2, p. 88-94, 2005. http://portal.acm.org/citation.cfm?id=1053291.1053293

Anonymous said...

Reviewers should have a reputation measure that is similar to sellers on eBay. Authors can then ranked, via an anonymous ID, the constructiveness of a reviewer's feedback.

John V said...

No editor worth their salt uses only the suggested reviewers, nor forgets which reviewers were suggested. Suggestions are quite useful, usually editors know who else has the same specialty as the suggested people.

Editors do not edit reviews, so far as I know - that's unethical, but do offer to keep some comments for their eyes only.

Anonymous said...

Is it common practice to pick reviewers from the set of authors cited in the manuscript? I feel that I not uncommonly am invited to review a paper because my work is cited, even if the citation is tangential to the main topic of the paper.

Anonymous said...

I agree that reviewing has a number of good side effects.

However, the moral argument--that somehow just because I submit papers to a journal means I have an obligation to review them is silly. Just because you benefit from s service does not mean you should be expected to contribute to it.

Anonymous said...

Excellent timing! Just a few days ago I received my first review request (and a few hours later my second).

Both were journals that I'd submitted to in the year or so since I received my PhD. I'm guessing this is not a coincidence.