After reading the recent NY Times article (and associated commentary in the blogosphere) about discrepancies in proposal success rate of black vs. white PIs at the NIH, I tried in a rather feeble way to find these data for NSF. I know NSF collects demographic data, and I have seen some of these when I have been on NSF committees, but the easily accessible online data seem to focus on other PI characteristics (institution, state).
Does anyone know if NSF has similar discrepancies in proposal success rate? Does NSF have a similar issue?
If it does, this might help point at an explanation for the discrepancy (and therefore a solution). If it does not, ditto.
As is apparently the case for NIH proposals, NSF proposals have PI names and institutions (and year of PhD), but no other demographic data. In a small field like mine, I typically know (by name, if not by sight) the PIs of proposals I review, so ethnicity is not an unknown. I know nothing of the NIH and the size of the various populations submitting proposals to particular programs, but it is possible that in at least some programs, the ethnicity of PIs is known to all or most reviewers and panelists. Is this a factor? I hope not, but it is one of the things that will be looked into, according to what I have read.
In any case, does anyone know these data for NSF, foundation-wide or for particular programs?
Even if you found this data for NSF, I don't know that simple success rate data would be as meaningful as the analysis published in Science last week. They really did their comparisons carefully, controlling for variables like previous accomplishments and training, nationality, employer characteristics, etc. It's a really impressive piece of work that answers most imaginable objections. I'm sure somebody could still come up with an objection, but it's still a pretty good piece of work, and far more illuminating than just looking at success rates for white applicants and black applicants.
ReplyDeleteAll that said, I'm going to make a prediction: I'm going to predict that when all the data is in it will turn out that NSF is worse than NIH. I say this mostly to be contrarian--NSF talks about diversity more than NIH does (Broader Impacts and all that), and hypocrisy on moral issues is an ubiquitous human trait.
I'd be delighted to be proven wrong.
I'm going to guess the opposite -- that NSF is better than NIH. I can't find the data either though.
ReplyDeleteOn a side note, I wonder if there have been any studies on correlations with student evaluations and the professor's ethnicity.
ReplyDeleteYou can check out the NSF's annual reports. Here is a link to the the 2008 version.
ReplyDeleteOn page 7 or 8 they give the funding rates for a number of different groups. I hope this helps.
http://www.nsf.gov/nsb/publications/2009/nsb0943_merit_review_2008.pdf
There's a small mention of discrepancy in the Science "News and Analysis" editorial: http://www.sciencemag.org/content/333/6045/925.full.
ReplyDeleteThis seems to suggest that the NSF tracks this information in some way, though I doubt it was subjected to the thorough analysis as the NIH funding.
I apply to both NIH and NSF. I would say that in general the NIH pool of applicants is larger such that you wouldn't know everyone's ethnicity for sure. The study did speculate that ethnicity could be assumed from name in some cases or undergraduate institution (particularly true for those who went to the historically black institutions). I would hope that doesn't play a role but I'm sure that there are people who it does.
ReplyDeletelooking at the annual report suggested above (nsb 0943), it does appear that NSF is doing better.. however, the NSF analysis lumps African American and Hispanic, which the NIH report did not - and NIH found that the bias was specific to African American PIs.
ReplyDeleteI would be very interested in this sort of data. I usually look to NIH for funding, as do most people in my department. But NSF is certainly an option.
ReplyDeleteAs for how the ethnicity of the applicant could enter into the equation. Academia is a small world. Certain folks tend to stick out. There are often clues on the CV. There certainly is on mine.
I'm a Hispanic female and one thing I have noticed in NSF panels is that people expect much more in the broader impacts section from minorities. It's OK for a white old guy to write a standard statement about mentoring students, but a black PI is expected to solve all the injustices in the world.
ReplyDeleteIn NSF, the first page created by fastlane has the ethnicity of the PIs (together with disabilities and residence status), but that information does not go to the reviewers.
I feel the same way about broader impacts and gender. It's OK for a man to say that he is going to mentor women, but I am held to a higher standard because I am a woman (which I feel I should mention specifically in the proposal even though it is of course obvious). My male colleagues get positive reviews on their BI's, which are the lame "I will advise female students", and I get average to critical review comments for the same thing.
ReplyDeleteIf it makes anon at 11:05 and 11:10 feel any better, I also promised to mentor female and minority research students in my latest proposal (and I have the track to back up that plan), and I got criticized for my Broader Impact as well. They looked at my long list of plans (which also included developing educational tools based on my research and a detailed dissemination plan) and said "Well, that's nice, but why not also provide research opportunities for undeclared majors who aren't sure if they want to be in STEM?" Um, yeah, anything else you'd like? Maybe a public lecture series to go with that? A community college partnership? Something involving elementary school kids?
ReplyDeleteSometimes I really hate Broader Impact.
I agree with Alex, but I still think females and ethnic minorities are held to higher standards. What really kills me is to see people in panels criticizing the broader impacts of somebody like Alex while having nothing better to show for themselves. Last time I went on an on about mentoring Hispanic students with hard evidence of past activities and a clear plan. I'm Hispanic, and my university is located in a state with a huge Hispanic population. A reviewer said I was being racist by excluding other minorities. The chances that this reviewer does more for minority students than I do are very slim, believe me.
ReplyDeleteMy prediction would have been that NSF would be better than NIH for the reasons that Alex stated -- plus I actually think NSF is the least hypocritical of the agencies. We all know that NSF studiously tracks data on minorities, women and other groups, but it is self-reported, which means a lot of data is missing. But I can provide a tiny bit of data. I served on a COV (unit external review panel) a couple years ago. The template report form included questions about the success rate for underrepresented groups. The data for this particular unit, in this particular review period, showed identical success rates for women and minorities at around 31% compared to 36% overall, and around 20% or less for new investigators. These numbers are not bad, I think, and I was heartened that the minority success rate was identical to women. However, the numbers for minorities are subject to HUGE stochasticity, since there were only a handful of successful proposals. BTW, this means underrepresented minorites, so not including Asian Americans. This review was not for my home NSF unit. I also served on a COV several years before that, for my home unit, and the same template was required. However, I don't have the data for that report, and we were much more cursory in filling in the template -- we just put "appropriate" for everything where there was no problem. This question was one of them, but we did note there was a lack of representation from HBCU's and MSI's.
ReplyDeleteBut I agree with others that Broader Impacts is arbitrarily applied, totally panel-dependent, and not well implemented.
In any case, if you really want to know, then ask someone in your directorate. They may only have time to get you someone's recollected knowledge, but they do seem to pay attention to these things and the executive officers may well know off the top of their head.
One problem with Broader Impact is that we don't have the same sort of "norming" that we have for research quality. There might be debates over research quality, but there's a lot of stuff that we all get because we all read the same journals and go to the same conferences as everybody else in our subfields. We might dissent and argue that, say, some of the stuff in the GlamourMagz is over-rated and not as well-done as the stuff in top society-level journals, but we still know what the hierarchy is, and we know what the community looks for. And we know how many papers/year is typical in our sub-field and our tier of institution. We might dissent for any number of reasons, or we might still argue whether a particular project or paper really is good enough, but we at least know what the community looks for.
ReplyDeleteBut we aren't normed on Broader Impact. I might know that X number of papers over 3 years will impress a panel given my field and institutional tier, but what number Y of under-represented students is considered "good enough"? Or what if I am likely to have less than Y under-represented students in any given time interval, but I am also developing an educational module? What is the "exchange rate" between educational modules and under-represented students (or public lectures, or K-12 outreach, or whatever) so that I know that my combined package is "good enough"?
Or, to go to the point of Anon at 12:19, is a plan targeted at one under-represented group a good broader impact because it is likely to succeed, or is it (almost by definition) not "Broad" in its Impact because it is targeted? What's the rule here? In research, if Anon@12:19 proposed to study a single system but study it exhaustively in some experimental tour-de-force, we'd probably agree that it's at least as good as a project that studies a range of systems in less detail, but we have no yardstick for saying the same in Broader Impact.
What I don't understand at all, is why do they have a bias? Why would I care if the applicant is black, white, pink or something else? I would evaluate scientific merit of the proposal, not genetic background.
ReplyDeleteI know there is a strong bias in Europe for foreign PIs, but I did not know about US.
-------
Post hoc ergo propter hoc
Some of the explanation may be a presumption that African-American applicants achieved what they did in part because of affirmative action programs, not entirely on their merits. It could be an unintended consequence of trying to right past wrongs.
ReplyDeleteI have the feeling that I posted a similar comment in this website years ago when discussing NSF panels...Anyway, regarding the comment of anon at 01:28, I witnessed first hand in an NSF panel how somebody disqualified an excellent proposal by a female PI on the grounds that she had achieved her position thanks to affirmative action. The female PI had an excellent track record (numerous awards, etc), and the panelist had achieved much less in the same field. The argument didn't fly, and was promptly dismissed by the person who was running the panel. Still, I think some damage was done.
ReplyDeleteMy take on this is that minority PIs often do not have the networking resources that others do. When I started my position I noticed that my white male colleagues constantly got unsolicited help from more senior professors in the form of advice, sample proposals, etc. I had to ask people for help, conveying the false idea that I was having more trouble getting funded. I haven't read the study yet, but is it possible that the proposals by black PIs were not as polished as others due to the lack of mentoring and other resources non-minorities get much more easily?
FSP said: "...it is possible that in at least some programs, the ethnicity of PIs is known to all or most reviewers and panelists. Is this a factor? I hope not...."
ReplyDeleteI hope not, too. But in general, the sub-fields for NIH are often-but-not-always small enough that you may know (or know of) the PI. However, even if you didn't know the PI, it wouldn't be unusual at all to google the PI and look at their departmental or lab webpage (which nearly always has a picture)...
Anon 8/25 1:03pm said: "What I don't understand at all, is why do they have a bias? Why would I care if the applicant is black, white, pink or something else?"
Umm....well, right. Wikipedia defines discrimination as "the prejudicial treatment of an individual based on their membership in a certain group or category."
I think Anon at 1:28 makes a good point - as a woman (non-minority), I have been told that I was getting faculty interviews because engineering departments are 'always looking to hire women'. So, if the same pinhead who said that reads my application, maybe he assumes
ReplyDeleteI got a fellowship or a job because of that and knocks those accomplishments lower than a white male. I can see that being done with minorities, and unfortunately, since some programs DO specifically give awards for women/minorities those thoughts aren't going away any time soon. There are tons of studies showing bias manifesting in this way, so while it's hugely disappointing it's not shocking. The interesting thing is that NIH has outlined some plans to try to rectify this - will be interesting to see if any work.
Timely. I just completed an REU re-application, and was wondering the same thing about NSF rates and race. I work at a school that has a high URM enrolment, so our “diversity” numbers look good, and I have a surname that is ethnically ambiguous.
ReplyDeleteSince this is *F*SP blog, do similar number exist for gender break down as well?
It’s also very interesting to read other people’s interactions with the “broader impact” section. It’s not my favourite section at all. Too many grants fall into 1) highly specialized research instrumentation with little real broad impact, and 2) those grants where the broader impact should be obvious to a blind goat.
It would be nice to live in a world, and I think we ARE getting there, where factors like ethnicity and gender don’t factor into objective science; intentionally or not.
I'm assuming this is the same Bashir as posted above, who found said data....
ReplyDeletehttp://jbashir.wordpress.com/2011/08/25/nsf-vs-nih-in-grant-racial-disparities/
I think it is very, very likely that NSF results will be similar to NIH because the review is not blind. Once the name and institution are added to the application, there are unconscious biases that come into play. The reason that we can't understand them anon 1:03 is that they are not conscious decisions but unconscious ones. There is lots of literature to back this up. For NSF panels that I have served on recently, they do some bias awareness training that combats this to an extent but the ad hoc reviewers don't receive this and their scores are given as much or more weight than the panelists.
ReplyDeleteNSF program officers have access to diversity data which is submitted with the proposal. Sometimes it is obvious in the proposal esp. in the broader impacts section.
ReplyDeleteAs a panel member I have seen good reception of outreach plans to such under-represented minorities as home schoolers, rural students and more. The panels, at least, are very receptive to outreach that identifies and targets underserved areas and appreciate that the PI has thought about such things.
This is one of the reasons I would like for us scientists to use "unique author identification number (UAIN)" rather than our names for scientific publication, grant proposals, etc. This way, our genders and ethnicities make no appearance and our work can be judged on its own merit.
ReplyDeleteIs age considered? I don't know the facts, but my guess is that more younger/new PIs tend to be minorities than established. Comparing funding of someone's first grant proposal or something like that might be more accurate.
ReplyDeleteBroader impact in academia - HaHa.
ReplyDeleteThis section should by reported by the school/department, as measured for student success as related to ethnic and minority status. Furthermore, the PI can be objectified with an author identification number, so the reviewers can be in an atmosphere of maximum fairness. The publications would also have to converted to a value based on a predefined points system.