Tuesday, January 18, 2011

Compared to What?

It is not uncommon for a request for a letter of reference -- for a faculty position or as part of an evaluation for tenure and/or promotion -- to include a specific appeal for a comparison of the candidate with others

Here are some examples of how this request is worded in instructions to external letter-writers:

Standard model/general: How does Dr. X compare with other individuals in his/her field at similar career stages?

Standard model/more specific: How does Dr. X compare in terms of research achievements, standing, and potential with other individuals in his/her field at similar career stages?

Advanced model/domestic: How does Dr. X compare with the best people in the US at similar stages in their careers?

Advanced model/global: How does Dr. X compare with the best people in the world at similar stages in their careers?

Advanced model/comprehensive: How does Dr. X compare with leaders in his/her field with respect to both current and potential future standing in the field, nationally and internationally?

I have worded these as questions, but the same thing may also be worded as a statement: "It would be very helpful to our evaluation if you would compare Dr. X with ..."

A common variant requested of current or former advisors of an applicant for a faculty position is for the advisor to compare the applicant with other advisees.

I don't have any particular compelling reason to argue against this practice, but, when writing letters, I ignore this request.

Many people comply with the request, and that is fine, but as a reader of many letters containing comparisons, I can say that I rarely find the comparisons useful, even if I know all the people involved (applicant, letter-writer, noted peers).

In my previous experiences on committees evaluating candidates for tenure, promotion, or awards in disciplines other than my own, the names of compared-peers are typically meaningless to me, and are used only for the departmental evaluation stage. I can maybe get something out of the peer's institution, but even here, I'd have to know the field fairly well because there are universities that have top-ranked departments in some fields but low-ranked departments in others. This applies to all types of universities, of varying levels of prestige and ranking overall.

One thing I find fascinating when I read these letters is that some candidates/applicants are compared to a completely different set of peers in each letter. For tenure/promotion, there might be 6 or 10 or more letters containing these comparisons for one candidate, each one listing different people as peers. That means that there is little agreement about what constitutes a peer and what constitutes an individual's "field".

This is particularly true in science and engineering fields that are highly interdisciplinary and that may involve very large numbers of researchers. Letter writers may be selected for their expertise in different aspects of an applicant/candidate's research interests, and so the letter writers may be familiar with different sets of peers.

Some letter-writers who make comparisons are quite general in their response: "Dr. X's research accomplishments are comparable to W (University of A), Y (University of B), and Z (C University)."

Others are more detailed: "Dr. X's research accomplishments are greater than those of W at the University of A, but s/he has not been as productive as Y at the University of B. Nevertheless, I rate his/her creativity as superior to that of Z at C University, although all of these are second-tier compared to T at the University of R." It is rarely clear what the basis of such detailed comparisons is.

Except when it is very clear, such as when the letter-writer specifies that s/he used a specific metric, such as number of citations, papers, or the h-index.

I am usually quite skeptical about claims that someone is "the next [insert name of genius-successful-famous person]". If the claim is based on the types of problems or methods used, or is at least backed up by some achievements to date, OK.. maybe. If, however, the letter is a vague prose-poem to someone's awesomeness, these comparisons are as believable as book blurbs.

Dr. X's last article in Nature could have been written by a young Charles Dickens!

Dr. X really knows how to write a proposal -- like Marcel Proust if restricted to a 15-page limit!


Dr. X's CV is an electrifying page-turner! The 2008 article in The Journal of Rigorous Science could have been written by Stieg Larsson!


Dr. X is that rare anomaly: an evocative and inventive scientist and prose magician, like Nikola Tesla and Virginia Woolf rolled into one!

19 comments:

mOOm said...

I've only once been requested to write a letter for someone who was not one of my former advisees etc. looking for an entry level job. They wanted me to evaluate someone for promotion to full professor. They sent me lots of information on the individual but as I remember it no criteria. So apart from just discussing the person's contributions all I could say was "This person would not be promoted at my university, but my university is very different to yours, so I don't know if they are promotable at your university". Checking that university's website just now I see that he was promoted.

Anonymous said...

Perhaps this is the reason why TT faculty are averse to taking any risk in following an unconventional path. At least till they get tenure, they have to bring in $$, publish papers and earn the respect of the established—all of which are negative incentives for going against the grain.

Comrade PhysioProf said...

One hilarious case I encountered included in the P&T letter packet both a letter from a competitor at a similar stage to the candidate--no idea why this person was asked for a letter, btw--stating that the accomplishments of the candidate were modest, and several letters from substantially more senior people in the field stating that the candidate's accomplishments were greater than those of the competitor who wrote the lukewarm letter.

Anonymous said...

I will be sure to include at least 1 of your suggested "blurbs" in each letter I write in the future!

plam said...

My advisor refuses to include comparisons in letters.

Two situations which I've experienced: 1) reading a letter from advisor which says "Dr. X is the first to do Y," where Y is a description fitting something that I've done (we did not interview Dr. X); and 2) reading a comparison to a group which is clearly supposed to include you.

Sharon said...

I haven't seen these comparisons much in faculty letters. However, I do find them helpful in grad applications. Even at a very general level (e.g., top 1%, 5%, 20% of students in our department tells a lot). Also more specifically, e.g., someone coming out of the same undergraduate lab, "She is comparable to X who started in your program 5 years ago." Those kinds of comparisons to students from the same UG school already in (or through) our program have been very helpful.

Anonymous said...

I actually think these comparisons are very useful. There are peer groups in productivity, and if I don't know the candidate, then it is very helpful to me to see the peer group in which the recommender considers the candidate to be. Is this person one of the Movers and Shakers? One of the Enthusiastic Wanna-be's? One of the Solid but Yet to Show Genuine Creativity? Or Quiet but Brilliant Marching to Own Drummer? So it's helpful to name several people for comparison, not just one. These comparisons are also extremely illuminating because they can say things that the writer may be reluctant to say directly.

Anonymous said...

yes, but to be useful in that way, you'd need to specify that you consider peer-X to be a creative genius etc. Not everyone may agree about what it means if Candidate is the peer of X without explanation. Someone considered by you to be brilliant might be considered a wannabe-advisorclone by someone else.

Anonymous said...

Ok, now I am totally developing a complex. I have the rec letter for fellowship application written by a collaborator who also sat on my PhD committee. He has not compared me to anyone!

I can't wait until I'm the one writing these damn things instead of asking for them.

gasstationwithoutpumps said...

"I can't wait until I'm the one writing these damn things instead of asking for them."

Writing recommendation letters is no fun at all. When I have to write one (not the anonymous tenure review letters but one requested by person being recommended), I demand a first draft from the person being recommended. It then usually takes me another 2 hours to turn it into a reasonable letter.

I almost never do comparisons by name with others. First, I never remember names. Second, I've rarely found head-to-head comparisons like that of any use. I'd much rather have a description of the strengths and weaknesses of the person than who they might be similar to.

EngineeringProf said...

Hmm. I guess I'm the odd one out. I've found comparisons helpful. The main value of comparisons is that they address the calibration problem: when you say the candidate is great, ok, how great? If there's a comparison to other folks in the field, then that helps pinpoint just how positive a letter it is. Otherwise, I'm stuck trying to read the tone of the letter to deduce exactly how positive the letter-writer intended to be, which is no fun and error-prone.

I can understand how these comparisons would be less useful to folks at the campus level, but at the departmental level, they're gold.

What strategies do you have for dealing with the calibration problem?

Anonymous said...

I wrote one of these recently. Here's a flavor of how it sounds, for those of you wondering.

"A at Auburn started there in 2001, and was promoted a few years ago. He had a more prolific pre-publication record, but I think Candidate's work will be more high profile than A's has been.

B at Brandeis started there in 2007 and has done very well. I think she's been doing more high-profile research than Candidate, but only by a hair. After Candidate's papers get published I think his reputation might become comparable to B.

C at Chicago started in 2003 and is now an associate professor. I think C is doing better than Candidate: he's had a steady stream of publications, from Chicago, over many years. But then, I think the quality of graduate students at Chicago is better than at Candidate's institution. So that is to say, Candidate is not at the level of C (one of the best young people in the field right now), but he's close.

D at Duke has been there since 2005 and is now an associate professor. I definitely rank Candidate higher than D, Candidate is doing more creative work and I think will have a much bigger impact on the field."

etc etc. So, using the comparisons to say more about the candidate.

gasstationwithoutpumps said...

EngineeringProf said"The main value of comparisons is that they address the calibration problem: when you say the candidate is great, ok, how great?"

That only works if you are in a field so small and homogeneous that everyone knows the people being compared to and agrees on their worth. I've never seen that happen in the engineering departments I've been in. Hell, people don't even agree on the relative ranking of the people in the *department* much less the thousands of people in the field.

Aisling said...

"I demand a first draft from the person being recommended"

Doesn't this defeat the whole purpose of recommendation letters, which is to find out what a third party thinks of the candidate? I find it very awkward to have someone put themselves in your shoes and try to second guess what your opinion of them is. At this point, why not stick to a cover letter admittedly written by the candidate?

In my experience, the editing of drafts I supplied went as far as comma addition and replacement of my name by a pronoun (or the other way around). Frankly, I can't say that these letters did constitute a "third party opinion" on my application. I'm more tempted to wonder what value search committees insist on attributing to these letters.

Anonymous said...

I am Anon 2:29,

GasStation (and others who don't make comparisons in their letters): thanks for making me feel a little bit better (seriously), although on the flip side I do see the utility in making direct comparisons. I guess I'm just overly sensitive to what makes a good letter because I need good letters right now, and I'm left wondering what the letter readers are going to think of me when direct comparisons are left out...

And actually when I made my comment yesterday, I was in the middle of writing a portion of my letter for said committee member/collaborator. Nothing makes me feel worthless like having to write my own letter! And my experience is similar to Aisling's, my recommender used my text verbatim, typos and all.

gasstationwithoutpumps said...

I do a lot more to drafts I'm given than just punctuation and grammar fixes (as desperately needed as those usually are). I did two recommendation letters yesterday for undergrads applying to grad school or summer programs. Even with initial drafts, the letters took me about 2 hours each to finish.

Aisling said...

gasstationwithoutpumps: well, I knew you were going to say that tremendous editing happens to the drafts you request. That is why I mentioned my experience of virtually no editing being made to drafts I supplied, and apparently this is not an isolated experience.

I realize how much time goes into writing a recommendation letter when done properly - it did take quite a while to write my own drafts, based on previous, older, versions.

But I still fail to understand the rationale of having a candidate write a draft for their own recommendation letters. If you're going to spend the time required by the task, why is a draft from the candidate better for you to work with rather than, let's say, a letter previously written by you on another occasion? Even with appropriate editing, I still think a letter based on a draft by the candidate is at best a biased third party opinion.

Back on the topic of comparisons, I think focusing on the candidate's achievements and competence is better in most cases, as no two candidates or individuals ever have truly comparable track records. Any evaluation has some subjectivity in it, but comparisons seem more subjective than other evaluation criteria.

gasstationwithoutpumps said...

"But I still fail to understand the rationale of having a candidate write a draft for their own recommendation letters. If you're going to spend the time required by the task, why is a draft from the candidate better for you to work with rather than, let's say, a letter previously written by you on another occasion?"

If I have previously written a letter for the student, I of course use that as my starting draft, though if it is old I request an update from the student first.

Starting a cold draft from nothing has a big activation barrier for me. If a student can't be bothered to write a few paragraphs for me, then they probably don't care enough about the recommendation letter to be worth 2-4 hours of my time.

Also, I tell the students to put into the draft the specific things that they have worked on with me. Often, on reading these drafts, I think "oh yeah, I should mention that project from 4 years ago---I forgot all about it!"

I don't send students into the task cold: I have them read the assignment from my class that tells them a little about how to write the draft of a recommendation letter:
http://www.soe.ucsc.edu/classes/bme205/Fall10/fellowship.html

Anonymous said...

I'm re-reading these old posts (to make myself nervous as I prepare my tenure packet, I guess) and just thought I'd add a note about writing your own recommendation letter. When I was a student, I felt much like the trainees commenting on this thread: if so and so thought I was worth anything, he'd take the time to write a letter for me. It seemed kind of insulting and pointless to be asked to write my own letter. Now that I'm on the other side of things, though, I realize that it's actually a golden opportunity (how I wish I could write my own tenure letters! ;). I don't always ask trainees to draft their own letters, but when I do, it's actually a compliment. It means that I trust them to know what needs to go into the letter. It's a great opportunity for them to highlight the really important accomplishments in the right way and to remind me of that key result they worked so hard for. So if someone asks you to draft your own letter of recommendation, recognize it as an opportunity to toot your own horn and as a compliment, not as a sign that the referee can't be bothered. Put in all of the great things you've done, everything you think is important, and they can edit it down a bit if they like. If the referee didn't think highly of you, you can be sure they wouldn't let you draft the letter!