Proto-Broader Impacts

As I've described in previous posts, I am on board with the mandatory "broader impacts" aspect of NSF grant proposals, and I think that activities beyond the basic doing-of-the-Science should be given some serious thought by researchers. I don't think the definition and review criteria for broader impacts should be extreme -- that is, I think that reviewers and panelists and program officers should re-read the examples in the NSF grant proposal guide every once in a while and not downgrade a proposal for only including a thoughtful but standard description of activities involving education, mentoring, and so on.

And I am even on board with the general idea that we don't suddenly learn how to care about Broader Impacts (BI) when we become faculty and need to put something in a grant proposal. This is something that should be learned along the way, and grad students should be aware of these other elements of research, including the importance of communicating research results to a broad audience and considering the education/training aspects of research as part of the overall "impact" of the work.

Where is this post heading? Am I about to complain/whine about getting bad reviews for the broader impacts component of a proposal? No, not this time.

Instead, I am going to opine that, although graduate students should start learning about broader impacts (what they are, why they are important), we should not expect that students will have much experience with BI during their student years. I am talking/complaining about overly severe expectations that students will have impressive BI credentials, e.g., when writing their own grant proposals to NSF or other funding agencies with BI-like components.

Certainly there are BI activities that grad students can do as part of their research, and many do these routinely (e.g., mentor undergrads, visit schools). That's great, but from what I've seen recently, it's not considered enough by some reviewers and panels.

I don't think that's fair to the students. We expect a lot from grad students in terms of research and communicating research results (publishing, going to conferences); there's a lot to learn and a lot to do. Right now, expecting serious BI participation from students seems like adding time and effort without changing the number of hours in the day. Oh yes, and we also want to get the time-to-degree statistics for grad students down at our institutions.

Is it just me, or have others noticed that BI expectations are quite stringent even for student proposals?

There should be a way to teach students about BIs, encourage them to be involved and contribute their ideas and energy (as time permits) to BI activities (keeping in mind, of course, that the students themselves are BIs for their advisors), but that expectations should be reasonable for what students can and should be doing at this stage of their careers.

The Odd Women

A scientist put together a research team to do some Awesome Science, and they wrote a grant proposal together. The team involved three PIs at different universities, their graduate students, postdocs, undergrad research students, and so on: the usual elements of a research proposal. The proposed research was excellent, and all 3 PIs are talented, respected scientists. Even so, some people doubted whether the proposal would be funded.

There are lots of reasons to doubt whether any particular proposal, however excellent, will be funded, but in this case, there was a reason unrelated to the significance of the proposed work, the research qualifications of the research team, the technical merits of the proposal, or even the lack of enough money to fund all excellent grant proposals. So what was the reason for doubt in this case?

The 3 PIs are women.

The doubt had nothing to do with the qualifications of any one of the PIs (all are highly qualified), but rested on the fact that all 3 PIs are female.

Of course, in the entire history of humankind, including today, no one would blink an eye if there were 3 male PIs. In the physical sciences, most scientists are men, and this has always been the case, so an all-male team of PIs is unremarkable, even today.

Oh sure, nowadays male PIs might describe in a proposal how they would involve female students or postdocs in the research, and they might write about how some of them have even advised females in the past, but the proposal would be funded or not funded depending on the scientific merit of the proposed work. I see proposals like this all the time.

But, even considering that this field of science is dominated by men, especially at research universities, is it so strange that a research project might be led by three women?

And, even if it is unusual, is it a problem?

In fact, it was not a problem. The pessimists were wrong, and the proposal was funded. And it was funded because the science was great and the PIs are all leaders in their field, with substantial track records of excellence and productivity.

Now, some would think the proposal was funded because the 3 PIs are women.

Are you following along? It's confusing, I know, but here is a handy summary:

If the proposal is not funded, it might be because the 3 PIs are women.
If the proposal is funded, it might be because the 3 PIs are women.

Fortunately(?), these statements refer to perceptions, not reality. Today, projects led entirely by women, even in STEM fields in which women are vastly underrepresented, are funded by the NSF if the proposed research is excellent.

We are not yet, however, at the stage where an all-female team of PIs is unremarkable. We are still at the stage where some people wonder if women get grants because grant agencies have to fund some females to meet diversity quotas or worry about projects that involve too many women. Maybe we shouldn't worry about this. Maybe we should just be happy (for now) to get grants. Maybe it is asking too much to have grants and respect?

This started me thinking about what an unremarkable proportion of women would be on a project. Is it equal to or less than the proportion of women in a particular field, or can we crank that number up a bit? In a 3-person PI team in a field in which women represent much less than 1/3 of scientists at research universities, we'd have to round a fractional woman down to zero if we used the proportional scheme.

I'm going to propose that, in fields such as this, ~25 +/- 10 % would be a non-threatening, unremarkable % of women involved in a research team. At this level, most people wouldn't worry that the science won't be any good or that talented men are being excluded for unfair reasons that have nothing to do with their research skills or experience.

But actually, my preference would be to assume that my not-entirely-serious, cynico-sarcastic analysis is flawed -- the proposal with the 3 women PIs was funded, after all -- and let some people go ahead and worry that maybe it was funded because NSF has to toss some money to the girls now and then.

Eventually (soon?), anyone who remarks in a negative way on the oddness of an all-woman PI team will be told that they, in fact, are the odd ones to think there is anything remarkable or problematic about an excellent and productive research team that just happens to involve only women.

Note: The title of this post is an oblique reference to the George Gissing novel, The Odd Women, in which "odd" doesn't mean that the women are bizarre, but refers instead to the fact that, in late 19th century England, there were more women than men of marriageable age; i.e., an "odd" number of women. The title of this post therefore refers to a possibly problematic circumstance in which there are too many women.

CAREEReview

Owing to the apparent youth of much of the academic segment of the blogosphere, it is not difficult to find posts by assistant professors describing their activities and thoughts during various stages of preparation of CAREER proposals, and their elation/dejection upon receiving news of NSF's decision.

My own CAREER grant expired long (long) ago, and now my primary experience with the CAREER program is as a reviewer of proposals and as an occasional mentor (in real life) to colleagues who are preparing their own such proposals.

Fellow reviewers of CAREER proposals (or whatever equivalent there might be in the NIH university): let's discuss how we go about reviewing these things.

If you have reviewed one or more CAREER proposals, do you:

1. Ignore the fact that these proposals are a bit different from regular NSF proposals, even though you know better, and review them much as you would any proposal (perhaps accounting somewhat for the relative career youth of the PI). That is, do you mostly ignore all the extra educational components of the proposal and focus on the Science?

2. Pay some attention to the education and/or outreach plan, but focus mostly on the Science?

3. Give serious (perhaps equal) thought to the plan for educational/outreach activities, provide detailed comments on these in your review, and factor in your opinion of these in your overall rating of the proposal?

- Other..?

From what I've seen through various intersections with the CAREER program over the years, different fields have different philosophies about these grants and how important they are. I think everyone agrees that it is a good thing to get these grants, owing in part to their being of longer duration (up to 5 years) than a typical grant (2-3 years). There seem, however, to be differences in practice about when to submit the proposal, e.g.:

The CAREER grant can/should be the first grant obtained by an assistant professor, and therefore should be applied for early, even if the individual has no previous grant track record,

vs.

A CAREER proposal should be submitted after there is some grant track record, even if after only one other NSF grant.

And I have heard rumors of fields or subfields in which a CAREER grant is essential for tenure at some institutions, but I have not seen evidence that these rumors apply to anyone I know or to any department at my university as long as there is otherwise a solid record of funding.

When I review a CAREER proposal, I certainly look at the required education/outreach parts and I like to see a sincere effort with this part of the proposal, but I must admit that I think it is asking a bit much of assistant professors to have a particularly sophisticated plan. It's great when someone really does have a creative and detailed plan, but, as long as there seems to have been a sincere effort with this part of the proposal, I don't penalize PIs whose broader impacts aren't awesome. That doesn't mean I don't value education or outreach; it just means that I think we have to be reasonable about expectations for already overburdened early-career faculty at research universities.

So, I guess I'm mostly a #2 in the list above, with the caveat that the education/outreach plan has to have some substance to it. After all, that is part of what distinguishes this grant from others, and partly what justifies the longer duration.

Another issue arises when reviewing CAREER proposals from faculty at different types of institutions. This is a general issue when reviewing proposals from, say, faculty at research universities vs. faculty at small liberal arts colleges, but it's a particular issue for CAREER proposals because the education activities might be more of an expected and valued part of the job for some faculty than for others. I actually don't think it's a major big-deal issue because we routinely deal with these types of differences n reviewing proposals, but perhaps someone disagrees with that opinion?

So, how do you review CAREER proposals?

In Which I am Thanked for not Being Biased Against Men

An alternative title for this post is:

Now I've Seen Everything (in a review)

To explain:

One possible example of a broader impact that can be mentioned in an NSF proposal is the extent to which the proposed research activities broaden the participation of underrepresented groups.

A topic of past discussion in this blog is whether to list yourself as a broader impact, if you are the PI or co-PI and are a member of an underrepresented group.

I don't mention myself as a broader impact in proposals anymore. At an earlier stage of my career, on at least one occasion I included myself, as one item in a list of BIs, and I got slammed in review for it by someone who took it as evidence that I was using my gender as an unfair tactic to get a grant. Never mind that I wasn't telling anyone anything they didn't already know (i.e., that I am female) and that it was therefore not a particularly clever tactic; including this factoid as an item in a list enraged at least one reviewer.

I stopped listing myself, not because I was afraid of getting such a negative reaction again, but because I changed my approach to the BI section. I used to adopt a comprehensive approach of noting all BIs relevant to my proposal. Now I just focus on what I think are the most important BI activities or elements of my proposed research.

Also, I got older, and it's not clear whether a tenured female professor fits the BI goal of broadening participation etc. anyway, except in a rather indirect way.

That brings us up to the present situation:

In one review of one of my recent proposals, I was thanked by one reviewer for not mentioning myself or other women involved in the project as a broader impact. The reviewer was very happy to see that my proposal was therefore not obviously biased against men.

OK... you're welcome.. but you know what? Even if I wrote in the BI section that the proposed research involved female investigators and therefore in some way helped broaden the participation of an underrepresented group, this does not demonstrate bias against men. It would be stating something that is part fact (I am the female PI whether I mention it in the proposal text or not) and part opinion (my involvement in research broadens the participation etc.); no men were excluded or oppressed to produce this proposal.

I think the reviewer was mostly expressing relief at not having to read what is apparently an annoying/enraging statement about the underrepresentation of women in Science; clearly the reviewer has seen such statements in other proposals. The reviewer comment was meant as a compliment, but I think the comment was inappropriate in a review and makes me wonder at the reviewer's comments in proposals by women who do mention that they are a broader impact. If I had mentioned in the proposal that women investigators were involved in the research, would the reviewer have rated this proposal lower, just because this obvious fact was explicitly stated?

I don't know, but, at the very least, thanking a woman PI in a proposal review for not mentioning that she is female is kind of weird.

Too Broad for Impacts?

A recent article in naturenews (Nature, 26 May 2010) discusses NSF's "broader impacts" criterion for proposals and notes that the definition of "broader impacts" is so broad that many PIs are confused. Although NSF does provide a handy list of examples, questions remain about what types and amounts of BI activities are most appropriate for a particular project (and PI).

[Note: Here are links to a few old posts on Broader Impacts: 1/10, 11/09, 10/08, 6/06]

Although some PIs do innovative BI activities, many do not. Apparently, the #1 BI in proposals submitted to NSF's chemistry division involves the training of graduate students and postdocs; an important goal, but that many of us do anyway. Hence the perennial question: Do BI activities have to go beyond what we usually do, or do we just have to do a better job at the advising/mentoring aspects of our job, or at least say that we will pay more attention to these things?

I don't know, but I found one of the relevant anecdotes in the article strange and unconvincing. The story involves the admirable efforts of a physicist, Diandra Leslie-Pelecky, who conducted a project in 2001:

In many ways, it was typical of the kinds of things that NSF-funded researchers do to fulfil [sic] their broader-impacts requirement. She took three female graduate students on weekly visits to local classrooms, where they spent 45-minutes leading nine- and ten-year-old children in practical activities designed to teach them about electricity and circuits. The visitors also talked about their lab work and careers. In addition, Leslie-Pelecky did something less typical of broader-impacts efforts: she brought along education researchers to study the effect of this interaction on the children's perception of scientists.

Those assessments were startling, she says. After three months, most of the students said that they still weren't sure who these young 'teachers' were – except that they couldn't possibly be scientists. In their minds, scientists were unfriendly, grey-haired old men in white lab coats.

"And that's what I worry about with broader impacts," says Leslie-Pelecky. "There are a lot of people putting time and effort into [these sorts of activities] and they have no idea if they're making any difference or not."

The anecdote ends there, leaving me a bit confused. I can see being concerned that these kids didn't understand that the graduate students were scientists, but that doesn't make me worry about "broader impacts" activities. That makes me even more convinced that we need programs of this kind, perhaps with a different approach to explaining (in this particular case) that the young women were scientists. What else, other than meeting real live examples would convince these kids that scientists don't necessarily fit their image of scientists as cranky old men -- an image that surely discourages a lot of kids, especially girls, from considering science as a career option?

The article mentions the idea that the goal of BI might be better achieved if BI activities were organized at the level of the institution, involving people with expertise in such things, who would then involve certain faculty in particular programs.

There are some aspects of that idea that I like, but I think it would be a mistake to remove all responsibility for BI from the individual investigators. The BI criterion, despite the confusion surrounding it, is a move in the right direction: that of asking investigators to think broadly about their research, to pay attention to the training aspects of our research, and possibly to consider how our research affects the non-academic world.

Furthermore, in the individual project-based system for BI, there are many opportunities for creative integration of the "intellectual" and "broader" aspects of research. A centralized office of people who are experts on "broader impacts"-type activities, but who don't necessarily understand the science and engineering research itself, might not innovate in the same way that individual PIs can.

I also would be concerned about a system in which this centralized office called on "certain faculty" to be involved in BI. I have a feeling I know who these faculty would be and that level of involvement would vary drastically across the institution, with confusion about whether this type of activity was valued by departments, was appropriate for tenure-track faculty etc. etc.

I agree, though, that if the intent is to do effective outreach on a large scale, then that type of BI activity should involve those who have specific expertise and time to devote to such things. Perhaps there is a way to have the best of both types of systems: e.g., to have more institution-based support systems to help PIs with BI activities (some universities have this; some don't), yet without completely relegating the organization and implementation of BI activities to the institution.

Brain Wave

It's interesting how the policies of funding agencies can have a pervasive effect on our thought processes. I am thinking of a specific example: broader impacts.

When BIs were first introduced as a required component of proposals, they were commonly an afterthought or just a statement of what PIs were already doing (advising grad students). With time, the BI component of some proposals that I reviewed became more interesting and innovative; some involved more outreach, e.g. interaction with K-12 teachers or the public, and some demonstrated a sincere commitment to increasing diversity of various sorts. Not most, or even many, but some did.

Perhaps there have been studies of the impact, if any, of the broader impacts proposal requirement on outreach, education, and so on. The effects might be hard to measure, but it would nevertheless be interesting to know if there has been any observable change in academic culture.

The BI requirement has had one observable effect on me, as demonstrated by the anecdote that motivated this post.

As I've written about before, I (and many of you) often receive invitations from random people to do random things that, in some cases, I can't imagine wanting to do, and, in others, I can't imagine having the time to do. Now and then something worthwhile comes along, but these are rare.

But: I recently got an e-mail with a request for me to do unspecified things involving communicating about some aspects of my research that might somehow inspire a contemporary dance project and/or a project involving children, literature, and culture by someone I don't know. I was about to delete the e-mail, having absolutely no idea what this person was asking me to do, but then I paused and thought "Hmm... broader impacts... might this be considered an interesting and innovative type of outreach?"

Is that thought totally corrupt or creative?

Broader Impacts R Us

If you are a female PI on a proposal that requires text about "Broader Impacts" (to use the NSF term), which may include the extent to which the proposed research activities increase (broaden) the participation of underrepresented groups, and if you are a member of an underrepresented group in your field, do you explicitly mention in the proposal that you are a broader impact?

Or, if this fact is obvious, do you not mention it and focus instead only on other examples of how your research will fulfill the Broader Impacts criteria?

I get asked about this a lot.

Last year I wrote about how I got blasted by one proposal reviewer who was extremely disgusted by my inclusion, at the end of a list of all my proposed research's broader impacts, that the project would support the research of a female scientist. I don't even know why I mentioned such an obvious fact; I was mostly just being systematic about going through the possibilities.

The NSF program officer put a line through these hostile reviewer comments and said they were ignored, but the overall review, including that reviewer's ranking, was considered. It was the only negative review but it was enough to sink the proposal out of the fundable range.

That was an extreme example, but I have seen cases in which male PIs who write about how they will involve female students in their research get higher marks for broader impacts than female PIs who are broader impacts. Some program officers view as inappropriate the criticism that female PIs are using their gender as a grant-getting tactic, but if one or more reviewers knock their ranking down a notch (or two) in anger about female-PIs-as-broader-impacts, the overall consequences for a proposal can be dire.

Of course there is more to "broader impacts" than involvement of underrepresented groups. And female PIs have to do more than just be passive "broader impacts". As is the case for any PI on an NSF proposal, we need organized and serious plans that recognize the importance of educating and training students and postdocs, that enhance connections with industry or government agencies, that promote the communication of scientific results to the public, and/or that benefit society in any of a number of other important ways. In my research, a significant broader impact typically also involves my close collaborations with international colleagues and students.

I am on board with all that.

I am curious, however, as to whether female PIs (or other members of underrepresented groups) deliberately mention/don't mention themselves as a broader impact. Owing to the lack of women in my field, I seldom review proposals by other women, so I don't know what others typically do. I now leave it off my list of broader impacts in proposals because (1) it's obvious and (2) it might be a magnet for the hostile women-have-an-unfair-advantage reviewers.

Female Impact

Every once in a while, there is an offensive statement in a review of one of my proposals. I am not talking about a "This research is not worth doing" type of statement, or even a "The PI has absolutely no qualifications to do this kind of research" type of statement. I mean statements that, in the view of the funding agency, inappropriately comment on an aspect of the proposal or the PI.

Some program directors at funding agencies will redact portions of reviews that they consider inappropriate. I know one program director who deletes such review comments and, in the specific place where the text has been deleted, inserts a statement that this has been done. He also includes an explanatory note that the comments were inappropriate and were not considered in the final decision about the proposal. The comments are not actually deleted from the system, but the PI can't see them.

I think that in some cases these comments are inappropriate because they are offensive, and in others they are inappropriate for technical reasons.

Another option that program directors seem to have is to strike-through the offending text. This leaves the text still visible to the PI but lets the PI know that the program director thought the comment was not appropriate. The overall ranking of the proposal by that reviewer may still be considered, even if particular comments are ignored.

I have seen both types of deletions of review comments in my proposals, but the strike-through method was used in the specific example I am going to mention today.

The specific example I wish to consider involved a review comment on the Broader Impacts section of one of my NSF proposals.

NSF's guidelines for what might be considered Broader Impacts include this statement:

How well does the proposed activity broaden the participation of underrepresented groups (e.g. gender, ethnicity... etc.).

In a one page description of the Broader Impacts of my proposed research, amidst a long summary list of activities of a Broader Impactish sort, was this item in a bulleted list (to be exact: the last item in the list):

and research support for a female faculty member (the PI).

The Broader Impacts part of my proposal was quite favorably reviewed by all but one reviewer. That last statement set off one of the proposal reviewers, who expressed severe disgust and revulsion for this kind of tactic. The program director had put a line through this strongly worded opinion about my use of my gender to get a grant. The reviewer also expressed a similar but less harshly worded opinion in other comments (not lined-through) in the same review, and his/her disgust seems to have affected the overall ranking, which was the lowest of the set.

Of course I didn't need to mention in the proposal that funding this proposal would support the research of an FSP. So why did I include it in my proposal?

I must admit that I didn't give it a lot of thought when I was writing the proposal. To the extent that I had a motive, I suppose I added that last item to my bulleted list summarizing all the Broader Impacts of the project because it made the list complete (note use of defensive font). If someone was pre-inclined to be suspicious of women PI's, I can see why they would be offended by the inclusion of such an obvious statement, although of course one would hope that such a reviewer could ignore their anger and revulsion and at least give an objective review of the Intellectual Merit of the proposal.

In discussing this situation with colleagues, I have heard different hypotheses for the reason behind the reviewer's high level of disgust, which went far beyond being uncomfortable about the concepts of gender diversity or Broader Impacts, but all these hypotheses involve the same central element: the reviewer, male or female, hates the idea that someone might get a grant (or job or other 'privilege') just because she is female, and not because of intrinsic merit, whatever that is.

Somewhere out there is a reviewer who ignored 99.9% of the proposal and got incensed about one item in a summary list in the Broader Impacts section. I hope that the program director will remember that this reviewer is hostile to proposals that mention gender diversity in their Broader Impact statements, at least when it is a woman PI mentioning this.

Perhaps I will leave such an obvious statement out of my proposals in the future, as it doesn't add anything substantive. Even so, most proposals that I review are by male PI's who are involving women students in research, and who mention their efforts to increase gender diversity in the sciences as part of their Broader Impacts statement. Perhaps this is OK with my hostile reviewer as long as these female students don't grow up to become FSPs who write their own proposals.

Broader Impacts (2)

Lately I have been reviewing a lot of proposals. All of them note that female students are involved in the research and associated classes and internships, and in most cases the % of female students is > 50%. Every single proposal is by a male scientist or group of male scientists, including one that involves 12 men and no women.

I review the proposals based on the science and not the gender of the people involved, but it makes me wonder whether these guys are aware of the disparity. I've reviewed proposals by some of these men for years, and for years they have been writing in the *broader impact* sections of their proposals about the large number of women students in their programs. I just wonder if they look around their all-male or almost-all-male departments and wonder why there aren't more women colleagues.

In my own department, the male faculty definitely think that we're in good shape because we have so many women undergraduates and graduate students. They have said this directly in faculty meetings: we don't have a problem with diversity, we have lots of female students. Then they scratch their beards contentedly. [I might be stereotyping just a bit there; some of them have moustaches and no beards]

Broader Impacts

I recently finished a grant proposal to the National Science Foundation (NSF), and it was actually great fun putting it all together. I like writing proposals because putting the whole package together really makes you think about what you are doing and why, and I like the challenge of making the most compelling case possible within the space limitations of the proposal (15 pages). I do not like constructing the budget, but the writing part is interesting. I've been very lucky with grants in the past decade or so, and always feel optimistic when I send a new proposal off, even if it is semi-delusional to hope it will be funded the first time it is submitted.

In addition to writing about the scientific work to be done, proposal-writers must also address what is called Broader Impacts. Advising graduate and undergraduate students is a Broader Impact, so that's an easy one, as it's just part of our jobs as teachers and researchers at a university. In the past, I've tried to come up with new and inventive ways to broaden the impact of my research, which is rather more of the curiosity-driven, basic science type of research than the applied kind with obvious and direct benefit to society.

However, no matter how elaborate the broader impact component of my proposals are (e.g., collaborations with K-12 or with local teacher training programs), I always get an 'average' score from reviewers and the panel for BI. I wondered what my colleagues were doing and if they got similar reviews for BI, as it didn't seem like most of them were doing much in the way of novel activities outside their basic research. To my surprise (speaking of being delusional), I found that one of my (male) colleagues got rave BI reviews for saying he was going to supervise an undergraduate in the summer. I supervise undergrads in the summer, and have never gotten any positive comments from reviewers or panels for it. Another (male) colleague got positive comments for saying he was going to incorporate his research in his classes. I do that too! I do not have enough 'data' to figure out if these differences are random, owing to the vagaries of reviewers/panels, but it adds an extra component of uncertainty for me when I write about the broader impacts of my research.

In my more paranoid moments, which are, I think, still relatively infrequent, I wonder if it relates to the fact that teaching is an expected part of women's jobs, but when men write about doing teaching AND research, it seems more special. It would be interesting if funding agencies compiled statistics on how reviewers/panels rated different Broader Impact activities. If there are discrepancies, it is quite clear that they do not result in women being funded at a lower rate than men -- NSF is an extremely fair organization with an excellent track record of supporting women in the physical sciences -- but it might show something interesting about people's perceptions and expectations regarding men and women scientists.