Thursday, May 17, 2007

Evaluating Others

Speaking of evaluations.. like many university faculty in the past month or two, I recently filled out a survey as part of the NRC rating process. Earlier in the year I filled out a form involving information to be used by the NRC for ranking my own department; this latest survey involved my ranking 15 other departments (the so-called 'reputational ranking' part of the process).

I've talked with other colleagues about how we each went about our rankings; not about our actual rankings, but about our methods. We each got a wide-ranging list that included at least one of the supposed top-ranked departments in our field as well as some schools we didn't even know had a Ph.D. program in our field. The information provided up front included the number of graduate faculty, various data about Ph.D.'s awarded, and demographic data about the number of women and ethnic minority faculty.

Some of my colleagues said they did their survey very quickly because they already had opinions about the places on their lists and they didn't need to look at the data provided or read information on a department's webpages. This raises questions about how to rank unfamiliar departments. That is, if you don't know anything about a particular Ph.D. program, couldn't name any faculty in that department, and don't know anyone who got a Ph.D. there, does that mean the program is deficient or does it mean that you should check the 'no opinion/insufficient info' option because perhaps the program is strong in a sub-field different from your own?

Others spent a lot of time reading all the information provided and following links to other webpages. I didn't spend a huge amount of time on the survey, but I also didn't blast through it without considering the data. I had opinions about all but perhaps one place on my list prior to getting the survey, but I didn't want to fall into the trap of rating a program high just because it was always rated high. I wanted to think about each place in terms of its program in recent years, not what it was like > 10 years ago.

Some faculty approached their surveys by considering each of the programs on their list in the context of the 15 programs on the list, and some viewed each in a context beyond the list of 15. I did the latter, but I assume that the ultimate dataset will be large enough so that these variations won't matter in the end. It might not matter anyway if each list is constructed to include a wide variety of programs.

Another issue we discussed is whether the demographic data about faculty gender and ethnicity is an indicator of the quality of a program, and why those data are reported up front in this survey and not information about faculty publications and funding. We all agreed that we would have liked more information on traditional measures of a department's research activities.

What do you think? All other things being equal between 2 large departments, should a total lack of women faculty in one result in its getting a lower ranking, and, if so, a slightly lower ranking or a substantially lower ranking?


Anonymous said...

Slightly lower in my opinion, not of any substantial degree- even if this is a specified goal one tries to pursue.

But in your field it seems (perhaps naturally enough) to be few women. Is it then expectable that the percentage of women at faculty level being high at all? Or maybe one could turn it the other way around: those fac. with high representation of women score high, others having low representation score neutral?

One should base empl.ments on qualification and merits- and not gender in my opinion. But if qualifications being the same-- one could favour women if more equal genderrepresentation is a specified goal.


Anonymous said...

I don't think that the gender or racial makeup of a department needs to impact its ranking, but obviously we've all seen that rankings are more or less meaningless without knowing exactly what was used to do said ranking, and the people who finally see the composite number might not know all the bases and weights used to create the final scalar. Is gender and racial information something people might want to know about that department? Surely that is possible for a wide variety of reasons.

I think what might be more useful, as problematic as getting enough respondents can be, is making sure people understand they should check a box that says something alone the lines of 'I do not feel qualified/knowledgeable enough to rate this category', and making sure that box gets included.

mike3550 said...

I think that it is funny that you are talking about this survey and asking the same question that some university presidents are about the U.S. News rankings. How, exactly, does one rank colleges and universities that one knows nothing about?

On a side note, it seems like the NRC would be more careful to instruct people on how to fill out the survey. All of the results are essentially unreliable if they aren't measuring the same thing -- I mean what does it tell you about a certain program if you filled it out based on ranking all the universities, another colleague filled it out based on just the fifteen on their list while some only filled it out based on 10 and marked the other 5 as "insufficient knowledge." In order to get reliable results, it seems like better instructions would have to be given.

Ms.PhD said...

Yes. Having fewer women should result in a much lower ranking.

Did you see the article in Science about the very low # of women elected to the NAS this year, and the pathetic excuses the head guy made for why that would be?

Departments should be smacked, hard, for not hiring women. That's all there is to it.

Anonymous said...

my home dept when I was a grad student had zero tenure track female faculty, one female adjunct. During the time I was there, they hired two female faculty, but they both left after a 2-3 of years (to get tenure in equally good universities, I should add). It is hard to retain female faculty if there is a perception of hostility or scepticism to female competency. It seems that people who have never been on the "other side" do not appreciate the benefits of diversity, it's all chips on the shoulder and just get over it.

Global Girl said...

What Ms. PhD said. Yes, and much lower. Objectivity matters, and if a department can't be objective enough to hire women, then it's just not a good department.

Anonymous said...


Context: I'm in grad school in a department with no tenured or tenure-track women; the only one who was above postdoc is leaving after this year.

And I've become so cynical about it all that my instant reaction was, "The faculty here would probably rate departments with no women higher."

They wouldn't actually. I mean, they wouldn't consciously claim to believe that fewer women make for a stronger department, but...they would certainly be annoyed that that information is presented as if it should affect the ranking. And I can actually imagine the numbers triggering subconscious biases. The faculty here really aren't willing to admit to having any, let alone be willing to examine them. Because they try so hard to hire women, it can't possible be their fault they haven't actually succeeded yet!

It is definitely weird that they don't include publishing or funding information, though. That seems, you know, important.

sandy shoes said...

All things being equal, a total lack of women faculty warrants a lower ranking. I think there's something seriously wrong with a physical science department that hasn't yet managed to hire a good percentage of women. In my (admittedly second-hand) observation, people pay lip service to the issue in faculty meetings (some more sincerely than others), but at the end of the day it can be business as usual with no consequences. Very frustrating.

In terms of the numbers of qualified female applicants, we've come a long way since even 20 years ago. There's no excuse any more for these overwhelmingly male departments, except that it reflects a systemic sexism that nobody cares to address.

Departments that recognize it is important to correct this should be recognized; those that don't should be called on it.

Mr. B. said...

What to say, what to say...

First, Mr. B. has been in the racket long enough to have heard (as a graduate student) that "women cannot be faculty members in a chemistry department."

Second, BigU was forced by the courts to hire women in tenure track faculty positions. Beeg mess. Nothing terrible seems to have happened. And if it had not been for court intervention it would have taken a hell of a lot longer for women to be hired.

Having said that, I don't think that Universities should be penalized in the NRC rankings for perceived or actual discrimination.
This is not the place.

But it is not hard for female applicants to count. When you visit a grad school as a prospective, note the number of female faculty members especially if there are few or none. Talk to the female faculty, post-docs, and grad students about the environment for women at the school.

And if you don't like what you hear, don't go there. Vote with your feet. Most of the holdouts who did not hire female faculty - because they couldn't find qualifed applicants - are starting to find that they can. I believe that they have learned that some of the best students are female and if they want to compete for them, they better start hiring female faculty.

A hopeful Mr. B who believes that incremental progress is better than none at all.

Anonymous said...

I think to answer to this question about evaluations it is important to figure out who is the audience for the rankings. If it's a grant giving organization or grant reviewers, they probably already have opinions about the school without the ranking. I doubt rankings are for faculty recruitment since I hope no PhD would choose a job based on its ranking.
This leaves rankings targeted to potential students (grad and undergrad). Although all the criteria listed might be more relevant to grad students, they're missing some of the most obvious. Do student's get jobs when they graduate from a program? Do they get the type of jobs they want. For the gender questions, does the gender ratio radically change between people enter and complete a program? There are some reasonable and bad reasons a program might have only 10%-20% female faculty, but I'd be very concerned if female students are disproportionally leaving a program before completion.

Ms.PhD said...

It's actually very hard to find out about graduate students who were in a program and left without graduating. So while I applaud the idea of tracking that, I'd submit that it's difficult, if not impossible, to assess that from a distance.

Graduates count. Drop outs are viewed as failures, not victims.

,, said...

I vote substantially lower ranking, because a lack of female and minority faculty is prima facie evidence for their willingness to make hiring decisions on the basis of their own sexist and racist bigotry.

This is is illegal discrimination and I predict that they will be too busy dealing with Title IX issues in the very near future to get any research done -- so if their rankings aren't low now, they soon enough will be.

But, honestly, how can anyone take them seriously as competently educated individuals (much less scientists), when actions as basic to the functioning of an intellectual community as who is hired -- are apparently driven primarily by racist and sexist bigotry?