Measure for Measure of Success

Something that I have been seeing more and more in grant proposal reviews (my own and those of colleagues who have shared theirs with me) is the idea that it's not enough to have a record of success advising grad students, undergrads, and postdocs in research -- you have to understand and explain your advising techniques and you have to have a plan for assessing and improving.

OK, I get that, but even when I attempt to do those things, it isn't good enough for some reviewers. They think that I (and my colleagues) are relying too much on past success and traditional measures of success (degrees, publications, conference presentations, post-graduation employment). They are not convinced that that is sufficient. They want something different. Apparently, unless you change something, you are not improving and therefore are not being transformative, or something.

Example reviews (comments condensed/reworded to remove any identifying vocabulary):

A highly qualified PhD student has already been identified for this research but the mentoring of this student and an undergraduate is largely assumed based on prior experience of the PIs. The PIs have records of successful advising but should include in the proposal a more intentional discussion of how they plan to train the next generation of scientists. The mechanism for success is not explained and there is no plan for assessing success of their mentoring. How will successful training of the graduate student be determined other than by the record of publications, presentations, and completion of the thesis? Although the research is potentially transformative and this is an excellent team of researchers, because of these shortcomings in the broader impacts I have given the proposal a lower rating.

That makes no sense to me. I am definitely not saying that we all deserve to have all of our grants awarded just because we have had past success. However, I think that if the proposed research is deemed excellent by a reviewer and the PI has a demonstrated record of success with advising, it does not make sense to downgrade a proposal rating for the reasons given in the example review above, contributing to the rejection of the proposal and therefore a lack of funding for the graduate student.

Here's another:

[From a review of a proposal that included one week of salary for a soft-money research scientist who runs a lab in which students would do some analyses for a proposed project]: Description of the mentoring of the postdoc is not well developed. There is no mention of career counseling. Mentoring in professional activities such as writing proposals and papers is confined to discussions and support for participation in conferences and workshops. There is no mention of how the postdoc will be mentored to collaborate with diverse groups of researchers and students. There is no description of the postdoc's career path in the context of developing an effective mentoring plan for him.

And this:

[From a review of a proposal that included a substantial component of support for undergraduate research]: These PIs have a long record of success in advising undergraduate students in research but no evidence is presented for how the field of research on undergraduate research will be advanced. 

These are just anecdotes, of course, plucked from reviews of different proposals by different PIs. At least one of the proposals even involved a colleague who does research on teaching and learning. It wasn't enough. Some of us PIs have attended national and local workshops on teaching and learning, read some of the relevant literature, even co-authored papers (some with education specialists) in science ed journals. It's not enough.

I think that giving attention to effective advising is an important component of research (and therefore grant proposals), but I also think these and similar reviews show that certain reviewers have run amok and are harming the very people (students, postdocs) they think they are helping.

Ab,surd

A colleague recently shared a review of his (rejected) proposal. The most negative review contained this statement:

There were a number of editorial errors found throughout the proposal. Some were missing commas and the like,

The mind, like, boggles.

Now I need to know: have you ever

• commented on punctuation in a review? (minor errors, egregious errors, pet peeves?) -- did it affect your overall rating of the proposal, as far as you know?

• received a comment on punctuation in a review? (minor errors, egregious errors, pet peeves?)

And if there were punctuation, grammar, and/or spelling errors in a review that criticized you for real or perceived punctuation errors, how did you feel about that?

Did you think, "Oh well, the reviewer is just bashing out a review on a webform and of course shouldn't be held to their own high standards for punctuation in proposals. Perhaps that missing comma on page 10 of the proposal indicates that I am a sloppy and untransformative scientist and therefore didn't deserve a grant that would have supported student research."

Or did you think something a bit more negative about the reviewer's punctuated hypocrisy?

Review Or Else

In a correspondence published in Nature not long ago, the associate editor of a journal grappled with the fact that some prolific authors are apparently not doing their share of reviewing. Let's assume that the review-slackers are not doing major reviewing for other journals. Let's also assume that the associate editor is not sending these authors review requests for papers quite far outside their expertise (I got one of these requests just the other day; I can't even imagine what combination of keywords and/or hallucinogens might have resulted in my name turning up as a likely reviewer on a topic so far from anything on which I have ever worked, much less published).

With those assumptions in mind, we are left to consider a person who accepts that others spend time reviewing their manuscripts but who is not willing to take the time to review the work of others. What, if anything, to do about this imbalance?

I have touched on this and related topics before without proposing a solution. In the Nature correspondence, the associate editor of a journal wonders whether a solution to the problem (described as: "the biggest consumers of peer review seem to contribute the least to the process") could be that "journals should ask senior authors to provide evidence of their contribution to peer review as a condition for considering their manuscripts."

There is much that is appealing about that idea, although in practice there are some likely complications (perhaps these could be dealt with in a flexible system). My preference, however, would be for a system that involved carrots but no sticks. I don't think we want to force people to review manuscripts just so they can accumulate enough reviewer points to get their own manuscripts into the system. That does not sound like a good recipe for thoughtful, thorough peer review.

Another reason for avoiding a system that would automatically reject the submissions of slacker-reviewers is that some of the slacker's co-authors would likely be innocent victims in such a scheme.


Is it possible to create an incentive system that is fair to all, is consistent with the philosophy of peer-review, and encourages more authors to contribute their time (thoughtfully and objectively) reviewing the work of others?

[I am ignoring in this discussion the fact that some people resent volunteering their time to feed the bank accounts of certain big publishing companies that profit from the federally-funded results of uncompensated authors and suck dry the meager budgets of university libraries, though I know some people have strong feelings about that.]

In my previous post on the topic of slacker-reviewers, I explained that I try to be as efficient as possible with the contributions of authors who are diligent reviewers. They don't get any sort of preferential treatment in terms of likelihood of acceptance of their manuscripts, but I do try to move things along for them as best I can, at least for the parts of the process I control (for example, how quickly I get to the manuscript once it is returned by reviewers). I don't know if that is enough of a "perk", but I hope it helps a bit.

I don't like slacker-reviewers, but I always try to remind myself that I don't know what their reviewing history is with other journals, I don't know what else is going on in their life, and I certainly don't want to do anything to harm their co-authors. It is not fair that some people are consumers of and not contributors to the peer-review system, but maybe we don't really want such people reviewing our papers(?).

Fake Review Contest : A Summary

Today's post refers to recent posts involving the so-called Fake Review Contest, even though most of the reviews ended up being not-so-fake; they were, or were based on, real reviews that their contributors had received. Alas.

A possibly-interesting exercise, though not necessarily an uplifting one, is to see how many of the 21 submitted reviews resemble real reviews you have received. I estimate that I have received reviews similar to about half of the contest entries. I hasten to mention that this is over the course of more than 20 years of publishing papers and writing proposals; such reviews are rare, in my own experience. They do tend to be the memorable ones though.

Anyway, in the Fake Review Contest, every entry got at least one vote. Yay, everyone wins, but some more than others. Late-entry #21 (This work disagrees with several unpublished results and is unsuitable for publication in this journal or ANY OTHER ONE [sic] on the planet) is the vote leader. 

It is clear, however, that #9 (Dear Editor, In the future please waste your own time) also resonated with voters, as did #14 (These authors would be well advised to wait until my paper is published, and then they can cite it) and #17 (vague review Sent from my iPhone).

And at least 5 other entries have so far received double-digit numbers of votes.

I recently found myself writing a review and striving semi-valiantly to avoid being one of Those reviewers -- the kind who object to how their own published work is presented, who appear to be quashing an attempt by others to publish competing ideas, and who come across as patronizing and hyper-critical snobs about technical aspects of writing. I am not sure if I succeeded in avoiding those things, but perhaps one positive effect of compiling a list with so many examples of mean and unhelpful reviews was to remind me to be constructive, detailed, and as objective as possible in my own reviewing. I recommended that the paper be rejected, but at least I explained carefully and (I think) politely why I came to that conclusion. Now it is up to the editor.. a friend of the authors.

 

Fake Review Contest : The Vote

Is it time to vote on your favorite (define that however you want) entry in the Fake Review Contest, even though many of the entries turned out to be real (disturbing)? You can review the entries in their entirety by scrolling through recent posts, or you can refer to the handy list of excerpts that I have included in this post. The poll is below. You have to choose one.

1. if this paper had been better, I would have read it and loved it

2. [this paper] is filled with dangerous ideas crammed into a package with a mundane exterior

3. plagiarism example

4. Why didn’t the author [insert thing that is indicated clearly in the title]?

5. I don’t know anything about the topic of this paper but I’m going to review it anyway

6. Since the author is a woman, I had lowered my expectations accordingly, but the author did not even meet those.

7. something is wrong somewhere

8. There was no point in my reading the entire paper because .. the authors assert with no supporting evidence that the world is round.

9. Dear Editor, In the future please waste your own time.

10. The experiments seem to be toys

11. I couldn’t help but noticed that my work is not cited in this paper

12. If the authors had high-quality data, interesting ideas, and an understandable discussion and conclusions, I would write a positive review. In the absence of those items, I regret that I must hate this paper.

13. one of the authors is the nephew of [someone important to me and so] this is a definite accept

14. These authors would be well advised to wait until my paper is published, and then they can cite it.

15. Presumably you are doing something smarter than what you describe, so you should explain that.

16. Mad Libs review (The authors describe how they (phrase from abstract) etc.)

17. vague review Sent from my iPhone

18. However, I don't think the method is correct in principle. 

19. Although we made you do two revisions which cost you time .. we still will not accept your paper [for political reasons]

20. This paper is transformative and should be published immediately with only minor revisions.

21. This work disagrees with several unpublished results, and .. is unsuitable for publication in this journal or ANY OTHER ONE [sic] on the planet.

Which is your favorite?
	1
	2
	3
	4
	5
	6
	7
	8
	9
	10
	11
	12
	13
	14
	15
	16
	17
	18
	19
	20
	21
pollcode.com free polls<

Late Entry - Fake Review Contest

21.  PI (another real review, alas)

This work disagrees with several unpublished results, and in its current form, is unsuitable for publication in this journal or ANY OTHER ONE [sic] on the planet."

Fake Review Contest Entries #17-20

These may be the last ones(?). Voting to come soon.

-->

17. TC

The topic of this paper is not of critical importance to furthering knowledge.

Sent from my iPhone

18. DM (generalized from real review, with no further elaboration of what is not correct)

The authors proposed to study phenomenon A by using methodology XYZ. The calculated results were largely consistent with experimental results. However, I don't think the method is correct in principle. 

19 (formerly 21). FR (another mostly-real one)

Dear Authors -

We regret to inform you that we cannot accept your paper. Although we made you do two revisions which cost you time (4 months) and energy; we still will not accept your paper.

The reason we cannot accept your paper is because a "prestigious professor" [PP] (who I, the editor, wants to collaborate with in the future) at a famous university thinks that you should had done XX (minor technique) in the way that professor described. Although four other reviewers thought that the paper was sound and "accepted without revision" after the second revision; I am not foolish enough to piss off PP in any way. I will be kissing his ass for foreseeable future.

Your truly

Ass-kissing-associate-editor

20. IDL (this one is apparently entirely fictitious)

I am very pleased that I accepted to review this fascinating and compelling paper. Below are some constructive comments to help you make your significant paper a bit more clear for readers. Mostly, however, I found the paper to be extremely well written. I thoroughly checked all equations and found them not only flawless but also instructive. The discussion section should be essential reading for all of us in this field. This paper is transformative and should be published immediately with only minor revisions.

Fake Review Contest Entries #13-16

More entries below. It is not too late to submit your own!

13. EFSP (based on a real situation, alas)

Yes, I know that this paper is on a topic that isn't even closely related to the broad and sweeping topic of our conference. But it has some pretty pictures in it. And one of the authors is the nephew of the president of Obscure Foreign University and they are going to award me an honorary doctorate this coming summer. So this is a definite accept and darn the others who vote for definite reject on this! I'm the chair of this conference and the paper gets accepted because I say so!

14. DS

I am afraid I cannot support the publication of this paper. The authors present an idea that I myself have had for some time and just haven’t gotten around to writing it up yet. I think that when I write my paper, it will be better than this one. These authors would be well advised to wait until my paper is published, and then they can cite it.

15. FF1

Presumably you are doing something smarter than what you describe, so you should explain that.

16. FF2 (Mad Libs review)

The authors describe how they (phrase from abstract). It is on the relevant and timely topic of (paper keyword). However, they should really address (title of paper section) and (title of a different section).

Fake Review Contest Entries #9-12

Here are a few more "fake reviews" for the Fake Review Writing Contest:

9. ER

Dear Editor,

In the future please waste your own time.

10. YN (modified only slightly from a real review)

I'm suspicious of claims to categorize behavior in the vague terms of these authors, or as has been done in the past, using the very popular statistical framework X, which claims that the only way to rationality is to have a prior distribution on possible hypotheses for every problem. The experiments seem to be toys.

11. SW

I couldn’t help but noticed that my work is not cited in this paper on (a topic that has nothing to do with my work). If my work is not going to be cited, why would I spend time reviewing this work? If the authors add citations to my papers, if only the recent ones but perhaps also one of my classic papers and ideally my 2010 paper because I just need a few more cites on that one to raise my h-index by a point, I will be happy to read past the introduction and reference list.

12. NR

If the authors had high-quality data, interesting ideas, and an understandable discussion and conclusions, I would write a positive review. In the absence of those items, I regret that I must hate this paper.

Fake Review Contest Entries #5-8

Here are a few more 'fake' reviews (even though some are apparently word-for-word or only slightly modified from real reviews):

-->

5. CF

Paper title: Graph analysis of System ABC shows differences in A-B connections during Condition X vs. Y

This paper uses a novel method to study ABC system dynamics. However, I don't know anything about graph analysis so I'm going to interpret the paper through the lens of a different analysis method I do know something about. Thus the only constructive feedback I can give is that this paper is a poorly written explanation of a structural equation model of XYZ - the conclusions drawn make little sense given the data, i.e. they frequently refer to "graphs" but all I see are these pictures of circles and arrows.

6. SS (modified slightly from a real review)

Since the author is a woman, I had lowered my expectations accordingly, but the author did not even meet those. Even though I have never done experiments in my life and have never used any of these techniques, the experimental results presented were completely misinterpreted in my distinguished opinion. The largest error was the omission of any background information; the authors did not cite enough of my papers or work in the background, specifically my articles in “Science” and “Nature”.

7. MC (verbatim real review)

Why is this result not loudly proclaimed as a triumph of predictive modeling? I can think of several reasons, such as (a) the authors are saving this for another paper, or (b) some one else has this result in press already and the authors don't want to deal with the politics, or (c) the one that worries me, the only explanations known are ones that cast serious doubt on the other analysis in the paper, so something is wrong somewhere.

8. EWH

There was no point in my reading the entire paper because in the introduction on page 2 the authors assert with no supporting evidence that the world is round. That the world is round is simply stated. What is the evidence for this? Without a detailed and convincing explanation, with compelling evidence, the rest of the paper is worthless.

Fake Review Contest Entries #1-4

The first entries in the reviewing writing-fest are fascinating because they are all apparently based on real reviews. In some cases, I think only a few identifying details have been changed. Perhaps there is no need to have a creative writing exercise to craft a fake review introduction; the real ones are strange enough to provide plenty of fodder.

There will likely be a vote on the entries at some point, so I am numbering the entries and adding an author pseudonym for each (in most cases at the request of the author):

1. kamikaze

Taking into account that this paper forms the basis of Ms HopefulAuthor's PhD thesis, I would have loved to love this paper. But I don't. I hate it so much I don't even want to read it properly. Therefore, I will reject it without any other argument than the fact that if this paper had been better, I would have read it and loved it. Ms HopefulAuthor had better hope I'm not on her committee.

2. mixedmetaphor

This paper is like a car-bomb headed for a building or a wall or something; it is difficult to be sure what or where it is going. Will it explode or will it be a dud? Neither has a good outcome, nor does this paper. It is filled with dangerous ideas crammed into a package with a mundane exterior.

3. JT

I have completed my review of the manuscript by XYZ et al.  This manuscript must be rejected on grounds of plagiarism - significant sections of the text were copied verbatim from a previously-published manuscript [ABC et al.].  I attached a PDF of ABC et al.'s paper and the XYZ et al. manuscript marked to indicate the plagiarized text (you will note that all but the first two paragraphs were copied).  I am very disappointed that the authors chose to represent another group's work as their own.

4. GR

Proposal Title:  Linear and Nonlinear Methods to solve XXX

Reviewer 3 (it's always reviewer 3):

"Why is the approach limited to linear methods, and the PI does not propose nonlinear methods?"

Fuzzy COI

In Scientopia, I use a reader's question about whether to serve as guest-editor of their advisor's and other close colleagues' manuscripts (short answer: no) to discuss more ambiguous examples of possible conflicts of interest (COI).

Take Down

For some reason -- or, possibly, for no reason -- the manuscripts I have been sent to review lately have been of a certain type: the type of paper that has an interesting dataset or idea that would make an interesting focus for the paper, but the authors instead choose to spin the paper as an attack on someone else's idea/s. That is, the papers seem to be aimed mostly at criticizing someone else's published work and proposing something different instead, even if that something:

• isn't all that different from the original idea they are trashing;

• isn't nearly as interesting as what they could focus on instead (says me); and/or

• isn't supported by their own data.

These cases are different from those in which there is clear evidence that a published idea or dataset is wrong and that wrong needs righting.

I have disproved the work of others before (including one of my early grad advisors, who hated me for it), but I don't derive any particular pleasure from it -- at least, not on a personal level. As a scientist, I can appreciate the sweeping away of an old, bad idea and replacing it with a beautiful new idea that explains things, and I feel satisfaction and pride if it's my research that does this or helps with this process, but I don't enjoy an attack for the sake of an attack.

For that reason, I find it hard to understand when someone else chooses -- and it is a choice -- to go that route when there really isn't much of a point to doing so. That is, when some researchers try very hard to find something, anything, no matter how unimportant, to tear down, and focus on that so much that the rest of their work is subsumed.

Yes, I know that sometimes there is personal animosity involved, but in the cases I recently encountered, the people involved actually get along quite well, at least as far as I know. The attacks in the manuscripts under review are not vicious or personal; they seem almost formulaic, as if the primary authors were told that this was the best way to write a paper that will be noticed (cited) or that they should be sure to distinguish their work from that of others.

In fact, the primary authors of these manuscripts have all been PhD students or postdocs. Maybe they are trying to make a splash? I think the papers could be really nice contributions if the focus were more on the substance of the research, not on some far-fetched or unfounded undermining of a minor point in some other publication. 

Probably my reviews will sound patronizing to the authors, and of course they and the editors can ignore my advice, but I think it is a mistake to go negative when there is nothing to be gained by doing so.

If you have gotten advice, particularly as an early-career researcher, about the best way to set up a paper, did that advice include anything about this issue? For example: framing a paper as an argument or attack is a good way to write a paper (no matter what), this is a bad way to write a paper, only do this if you think you are totally justified and it is an important issue etc.?

S/he

Can you think of a (good) reason why a journal would need to know the gender of authors and reviewers?

A reader wrote to me with this question, as a result of being required to select a gender-revealing option when registering on a journal's website. This option was separate from one asking for the professional title (Prof., Dr., Prof. Dr. etc.). It was not possible for this person, even after communicating with journal staff, to register without checking this box. In fact, the journal staff insisted that this information was essential because otherwise communication was too difficult and would involve the awkward use of "he/she" in letters, or perhaps embarrassing mistakes if the gender of the person wasn't clear from their name.

I can think of reasons why a woman would not want reviewers and editors to know her gender, but I can't think of a good reason why reviewers and editors would need to know the gender of an author or reviewer. It occurred to me that a journal might want to keep track of how many papers are published by male vs. female authors (or lead authors, in fields that make this distinction), but that is not the reason the journal gave to the reader who wrote to me about this issue. In that case, the concern was making embarrassing mistakes in using pronouns in correspondence or that someone would be offended if referred to as he/she instead of by the correct pronoun.

If a journal did want to keep track of gender data, those data could be separated from individual papers, so that editors and reviewers did not see it for any particular individual or paper. 

If you are writing to someone whose gender you do not know, why would you even use he/she or his/her in direct correspondence with them? This is a real question. Am I overlooking something?

In my role as editor and reviewer, I do not need it; 'you' is nice and direct, or I use the person's name or title. In correspondence about someone, I can use their title, a term such as "Reviewer 1" (if they are anonymous, you shouldn't use a pronoun anyway), or I refer indirectly to "the author/s", depending on context. The journal with which I am most closely involved is based in Europe, with close ties to Asia, North America, and Australia. It is more formal than many North American-based journals in its correspondence traditions, but even so, we do not need to know the gender of authors or reviewers.

Yes, I know about Frau Professor, Herr Professor etc., but those can be options for those who prefer those titles. There should not a requirement to inform a journal of your gender before you submit a paper or review an article.

Reviewerzilla

Today in Scientopia, I post a reader's question about what to put in a review for one of the High-Impact Journals. This reader wants to avoid being one of 'those' awful reviewers that bloggers and commenters complain about.

Take Out

Summer is not a great time for editors of journals to find reviewers for manuscripts. At least, that is my experience. In the summer, many academic scientists (and others) are extremely busy trying to get as much research done as possible before the beginning of the academic year. Many of us travel (conferences, research visits etc.). Some even take a vacation.

And yet, many people submit manuscripts for review in the summer, when they have time to complete projects, so the peer review process cannot take the summer off.

With effort, I have been able to find a sufficient number of reviewers for most submitted manuscripts that I have to handle as editor, but I have also received a large number of excuses from people explaining why they can't accept my request to review a manuscript.

Of course, no one has to give a reason, but many choose to do so, no doubt feeling the invisible sting of my editorial frustration and fearing my wrath the next time they submit a manuscript to the journal I edit. Or something.

Most excuses are routine and uncreative:

• I have too many other reviews to do at this time (in fact, that's the one I use the most when declining review requests from editors of other journals).

• I am traveling non-stop for the next n weeks (further admission: I've used that one as well, but only when it's true).

• The subject of this manuscript is beyond my expertise. etc.

Boring.

This summer, I got this one from a potential reviewer:

My wife made me take her on vacation.

Shall we parse that? It's summer, we have nothing better to do, let's do it:

My wife.. It's her fault! Not mine! I really really wanted to do this review but..

made me.. I had no choice! I was coerced! She was going to refuse to make my dinner and wash my socks if I didn't accede to her demand..

take her.. ugh, this is the part I dislike the most in this sentence. Why did she need to be taken on vacation? Did he carry her? Strap her to the car roof? Does "take" mean that he paid, drove, or both, and she sat passively while he took her places?

on vacation.. because of course, given his druthers, as a serious scientists, he would not go on vacation, he would do the review. But, alas, he was not allowed to have druthers. His wife took them.

Surely we can come up with better excuses than the boring ones and the wife-made-me one. They need not even be true, as long as they are not boring.

I am requesting that each of you provide a creative, entertaining excuse for declining a request to review. Do not decline this request even though it is summer. I am quite sure that you do not have any other similar requests at this time, you can easily type one in the comment box even if you are traveling non-stop for the next n weeks, and I happen to know that your wife (or whoever) wants you to take this challenge and entertain the readers of FSP.

Cite-Me

Today in Scientopia, I discuss a question about how/whether to mention in a manuscript review that the authors should cite one (or more) of your papers.

Review Well or Die

In the course of some recent discussions about Journal Editing, the question arose as to whether someone's reviewing skills should be a factor in tenure and promotion decisions. In particular, should a tenure-track professor's failure to be a good (reviewing) citizen of their academic discipline be a major, minor, or no factor in a tenure decision?

Some commenters said that it should be a major factor, but I think that some of these arguments are based on the assumption that a lousy reviewer will also be a substandard researcher and a bad advisor, in which case, their failure as a reviewer is the least of their problems.

Being a good reviewer is important. Our current system of peer review depends on there being many reviewers willing to spend the time to provide thorough, thoughtful, critical reviews, and to return these reviews in a reasonably timely way. I am not, however, convinced that someone who is a bad and/or disorganized reviewer (turning in reviews late, if at all, for example) is also likely to be a lousy researcher or teacher or advisor. In fact, as an editor, I have had many experiences to the contrary with delinquent reviewers who do outstanding research and who are excellent mentors to their graduate and undergraduate students and postdocs. Some are also excellent teachers, although I don't have this information about most reviewers.

Different institutions have different ratios of expected research : teaching : service by faculty. At my university, reviewing manuscripts and proposals is "service". Service is by far the smallest of the three components, and reviewing is just one part of "service". Perhaps being an excellent (or, at least, not terrible) reviewer is more important at other types of institutions.

Typically, faculty and committees reviewing the files of tenure and promotion candidates at a research institution have a list of journals and funding agencies for which the candidate did reviews, but no indication of the quality of the reviewing efforts. I suppose quality issues could come up in an external letter, but I can't recall having seen any examples of that.

Therefore, at institutions like mine, the quality of someone's work as a reviewer is a non-issue in tenure and promotion decisions unless it is symptomatic of their approach to research and advising, in which case the fact that they are a lousy reviewer is swamped by their deficiencies in these other aspects.

Let's assume that we somehow know whether a tenure candidate is a good or bad reviewer. Should it be a factor in employment decisions? I say no: if an individual excels at research and teaching but is a lousy reviewer, I think they should get tenure and/or be promoted based on the fact that they are doing well at the most important aspects of their job.

Being a reviewer shows a commitment to professional service and is also an indication of how well respected and visible someone is in their field. Visibility and respect are important at all career stages, but can be particularly important in promotion to professor at a research university. Even so, there are other indicators of this.

But let's say we still want to know if someone is a good or bad reviewer. One possible indirect indicator of reviewer quality might be the number of reviews someone is asked to do. Those who are lousy reviewers may not get asked to do m/any reviews, and this fact may show up in the numbers, although there are other explanations for minimal reviewing activity.

Can reviewing activity over time be an indicator? Not necessarily. For example, if someone's CV shows that they used to do a lot of reviews and now do not, there are several possible explanations:

- They did such bad or late reviews, or never returned promised reviews, that editors stopped inviting them.

- The topics in which they are most expert used to be well represented in the literature, but now are not.

- Their field of research got more crowded with time, so there is a larger pool of possible reviewers.

- They got too busy with other things and stopped accepting most requests to review.

It may be impossible to tell which of these, if any, is the relevant explanation, so I would hesitate to make inferences from a CV or other document that records a decrease in reviewing activity over time.

Bad reviewers and review-shirkers are annoying, especially since they rely on the reviewing work of others to get their own papers published, but I don't think there should be any major overt punishment of bad reviewers. Although it may seem that review-shirkers are getting away with something, they are losing the opportunity to play a role in the peer review process, to be a constructive influence in their field, and to be respected by editors and others for their reviewing wisdom and efforts. That's their loss, and an appropriate consequence for being a bad reviewer.

Militantly Ignorant

Today in Scientopia, I discuss the phenomenon of Militantly Ignorant Reviewers, and worry about becoming one of them.

Orders of Magnitude

Thanks to all who sent me links to the recent PNAS article by Ceci & Williams on "Understanding current causes of women's underrepresentation in science". I had read reports of the results, and now have read the article itself.

The authors of the article focused on some of the most commonly cited reasons for the underrepresentation of women in "math-intensive fields of science": discrimination in reviewing of proposals and manuscripts; being selected for interviews in faculty positions; and in hiring. They propose that current data do not support these as primary causes for the underrepresentation of women today.

Their main conclusion is that "differential gendered outcomes in the real world result from differences in resources attributable to choices, whether free or constrained", and that underrepresentation would be best alleviated through changes that take into account "differing biological realities of the sexes."

I think they make some important points with their study, and I believe that the current situation for women being evaluated for jobs, grants, and publications is better than it has ever been. However, I continue to see and hear examples of discrimination in reviews and hiring committees -- faculty who doubt that women have their own ideas (but have no trouble believing this about male candidates), or who don't like "aggressive" women (but think this is a fine trait in a man), are alive and well. These issues are not as obvious and widespread as in the past, but neither are they isolated, rare incidents that can safely be ignored as irrelevant to current practices.

I was surprised that the study focused so much on databases related to the life sciences, a realm of science in which, as the authors note, women now make up the majority of PhD recipients. I realize that some biological sciences are quite "math-intensive", but the authors seem to use this term to refer specifically to the non-biological sciences. It would therefore make sense to base conclusions primarily on studies other than those involving the life sciences.

(A quibble: In the sentence about how the number of PhDs awarded to women in the life sciences has increased, 13% is described as "orders of magnitude less" than 52%. It is not.)

In the section on Discrimination Against Women in Journal Reviewing, the authors rely in part on a study of acceptance rates for the journals Behavioral Ecology, the Journal of Biogeography, and Nature Neuroscience. Those all seem kind of life sciencey to me.

Similarly, in the section on Discrimination Against Women in Grant Funding, the authors rely on studies of databases of Medical Research Councils and similar organizations of various countries, including the NIH. There is also mention of NSF and the Australian Research Council, both of which cover a wide range of fields in science, engineering, education, and beyond. If possible, it would have been interesting to see a separate analysis of recent data for the "math-intensive" sciences.

Having seen such data for my own field, I believe that NSF works -- with some success -- to provide a "gender-fair grant review process", but I don't think the authors of this particular study have demonstrated that with their chosen databases. [I chose the phrase "works to provide" rather than a simple "provides" based on experiences such as this (which referred to, but did not specify, that the problem was offensive sexist comments by a program officer) and this and this and this, all of which the excellent and enlightened staff at NSF can and do filter so that there are no deleterious effects on female PIs].

The authors note that there are "more women in teaching-intensive, part-time posts where research resources are scarce", and attribute this to life-style choices or constraints. When discussing the situation at major research universities, the authors cite an NRC task force report that concluded that women were not at a disadvantage for interviews and offers (and may have a slight advantage) in a study of 6 fields of "natural science". That's great, but I think it is premature to propose, based on these data, that universities should discontinue efforts to train hiring committees to avoid bias (explicit or implicit).

Although ultimately not as convincing as it could be (owing to the datasets used), this is a useful study in that the authors try to focus our collective attention on additional factors that affect the underrepresentation of women in math-intensive fields of science, and suggest that universities explore new options for career tracks ("The linear career path of the modal [sic?] male scientist of the past may not be the only route to success.."). I agree; just don't throw out the methods that seem to be working so far. That would make the situation orders of magnitude worse than it already is for women in math-intensive fields of science.