As I've described in previous posts, I am on board with the mandatory "broader impacts" aspect of NSF grant proposals, and I think that activities beyond the basic doing-of-the-Science should be given some serious thought by researchers. I don't think the definition and review criteria for broader impacts should be extreme -- that is, I think that reviewers and panelists and program officers should re-read the examples in the NSF grant proposal guide every once in a while and not downgrade a proposal for only including a thoughtful but standard description of activities involving education, mentoring, and so on.
And I am even on board with the general idea that we don't suddenly learn how to care about Broader Impacts (BI) when we become faculty and need to put something in a grant proposal. This is something that should be learned along the way, and grad students should be aware of these other elements of research, including the importance of communicating research results to a broad audience and considering the education/training aspects of research as part of the overall "impact" of the work.
Where is this post heading? Am I about to complain/whine about getting bad reviews for the broader impacts component of a proposal? No, not this time.
Instead, I am going to opine that, although graduate students should start learning about broader impacts (what they are, why they are important), we should not expect that students will have much experience with BI during their student years. I am talking/complaining about overly severe expectations that students will have impressive BI credentials, e.g., when writing their own grant proposals to NSF or other funding agencies with BI-like components.
Certainly there are BI activities that grad students can do as part of their research, and many do these routinely (e.g., mentor undergrads, visit schools). That's great, but from what I've seen recently, it's not considered enough by some reviewers and panels.
I don't think that's fair to the students. We expect a lot from grad students in terms of research and communicating research results (publishing, going to conferences); there's a lot to learn and a lot to do. Right now, expecting serious BI participation from students seems like adding time and effort without changing the number of hours in the day. Oh yes, and we also want to get the time-to-degree statistics for grad students down at our institutions.
Is it just me, or have others noticed that BI expectations are quite stringent even for student proposals?
There should be a way to teach students about BIs, encourage them to be involved and contribute their ideas and energy (as time permits) to BI activities (keeping in mind, of course, that the students themselves are BIs for their advisors), but that expectations should be reasonable for what students can and should be doing at this stage of their careers.
And I am even on board with the general idea that we don't suddenly learn how to care about Broader Impacts (BI) when we become faculty and need to put something in a grant proposal. This is something that should be learned along the way, and grad students should be aware of these other elements of research, including the importance of communicating research results to a broad audience and considering the education/training aspects of research as part of the overall "impact" of the work.
Where is this post heading? Am I about to complain/whine about getting bad reviews for the broader impacts component of a proposal? No, not this time.
Instead, I am going to opine that, although graduate students should start learning about broader impacts (what they are, why they are important), we should not expect that students will have much experience with BI during their student years. I am talking/complaining about overly severe expectations that students will have impressive BI credentials, e.g., when writing their own grant proposals to NSF or other funding agencies with BI-like components.
Certainly there are BI activities that grad students can do as part of their research, and many do these routinely (e.g., mentor undergrads, visit schools). That's great, but from what I've seen recently, it's not considered enough by some reviewers and panels.
I don't think that's fair to the students. We expect a lot from grad students in terms of research and communicating research results (publishing, going to conferences); there's a lot to learn and a lot to do. Right now, expecting serious BI participation from students seems like adding time and effort without changing the number of hours in the day. Oh yes, and we also want to get the time-to-degree statistics for grad students down at our institutions.
Is it just me, or have others noticed that BI expectations are quite stringent even for student proposals?
There should be a way to teach students about BIs, encourage them to be involved and contribute their ideas and energy (as time permits) to BI activities (keeping in mind, of course, that the students themselves are BIs for their advisors), but that expectations should be reasonable for what students can and should be doing at this stage of their careers.
Look, if we have to suffer, they have to suffer. I'm sitting here being told that "Yes, it's great that you will turn your research software into educational modules disseminated via NSF-funded cyberinfrastructure, and it's great that you have a demonstrated track record of mentoring female and minority students in research and getting them into grad programs, and it's great that you are at a minority-serving school, substantially increasing your odds of continuing to recruit minority students into your group, and it's great that one of your undergraduate research students even got a national award in an open competition against PhD students. But, you know, why aren't you also active in k-12 outreach? Why aren't you also reaching out to students who haven't declared a STEM major and bringing them into your research group?"
ReplyDelete(And this wasn't for a CAREER or a special outreach proposal. This was for a regular research proposal.)
If we have to face unrealistic demands, might as well get the students used to it! Maybe if enough people face unrealistic BI demands, there will be a revolt, a sort of "Academic Spring", if you will.
(Yes, I'm sleep-deprived, teaching too many courses, and preparing my tenure application. What gave it away?)
Holy Moly yes! Last year an undergraduate who had worked with me, published two first-author papers and had a 4.0 GPA had his NSF fellowship application turned down due to "insufficient" broader impact. (Thankfully he received a fellowship from another agency, and is now off to get his Ph.D. at the top program in our field.)
ReplyDeleteA first-year grad student in my lab (also with two first-author publications) also had his fellowship application turned down, due to lack of broader impact, this despite the fact that his work had clear implications for broader society.
In the first case, I think it is fair to say that the student did not make a case for broader impact but the second case clearly indicates that even the reviewer did not understand the meaning of broader impact.
I agree, even though I benefited from this. (I got an NSF fellowship based on my Extreme BI.)
ReplyDeleteGranted, the reviewers feel obligated to grade on some sort of curve, and there will be a lot of people in the pool with the standard "I mentor undergrads and do science shows for kids". But...if we're actually looking at the impact an average working scientist can have, I'm not sure there's anything wrong with this. Not everyone wants to start a new shiny outreach program, nor should they have to.
I do wish there was a way to reward people who go above and beyond, thinking creatively about how to bring their science to the public, without punishing those who take the standard options. Unfortunately, when the pool of money is limited, that's not easy.
Students can submit grants? Huh, I never knew that. Yeah, asking students to do BI stuff seems a bit much to me.
ReplyDeleteAlex: I'm curious -- who is telling you that? (Not being snarky, I'm honestly curious.) Is that coming from an NSF program officer? From past NSF reviews? From collaborators giving advice?
ReplyDeleteI noticed the expectations for some of my fellow reviewers at a NSF GRFP panel were rather ridiculous - they seemed to expect a kid to have already initiated a huge outreach program in their first 2 months on campus as a grad student.
ReplyDeleteOn the other hand, other panelists pointed out that many of the applications are really good. And there is never enough money. So BI's are an easy way to separate the great from the pretty darn good. Which I would be more ok with if the fellowships were ever tracked to demonstrate they did any of their BI's.. right now you can propose everything and do nothing. How is that right?
In reality, I noticed that BI's actually separate students who have exposure to strong NSF mentors and those writing it on their own who don't know what the real expectations are. Not sure that's the best criteria to pick the winners - the ones who got the most help rise to the top again.
If graduate school is a "training program" and broader impacts are an expectation of professional scientists, then why wouldn't you want your graduate students to do these types of things? Through these activities, students may learn a lot of really important things, like teaching and communication skills, awareness of the issues facing under-represented groups. They may improve their ability to communicate about science to non-science audiences. I think that a job applicant down the road who has experience and skills beyond the bench is a very attractive candidate (but then again, I'm at a PUI).
ReplyDeleteChris,
ReplyDeleteThat is from an actual review. Reviewer #3 said it and the panel summary concurred.
That isn't the main reason that the proposal was not funded, but it added insult to injury.
As a graduate student I have to agree with FSP. I feel my first two proposals by the NSF were turned down in large part due to BI. It seems as a graduate student the easiest way to prove BI is to be a minority or serve minorities. (White female in a male-dominated field doesn't seem to count.) Teaching is rarely enough.
ReplyDeleteA lot of times you are penalized for not having "future plans", which as another commenter mentioned penalizes those who don't have good mentorship. It's easy to come up with a "plan" that they never check up on. My guess is that few graduate students follow through on that plan so it seems weird to judge on that.
Although I was praised in comments for outreach to middle school girls, it would have been better if they were under-represented minorities. Although I articulated well how my research would impact the field, it would have been better if I was having under-represented minorities do the research with me.
My final point is that it's contradictory on the NSF's part. We're looking for fellowships in most cases so we can teach less and devote more time to research (which is what the NSF seems to want- see the debacle of earlier this year). When exactly are we supposed to be doing all this BI work, most of which involves teaching?
I have sat on student and post-doctoral fellowship funding committees (not NSF but another) and as mentioned by others, the applicants are stellar. So the use of Broader Impacts is one way to delineate between excellent candidates.
ReplyDeleteAs a supervisor of students my direct interaction has actually been to try to tone down their desire for broader impact work. My field of study can eventually , in a long long term, translate into treating patients. Students often want to put the cart before the horse and jump into patient clinical trials when the reality is that they are looking at gene expression in mice. Trying to explain to them the protocols and procedures that have to be in place before embarking on this campaign is like talking to a wall.
We get volunteered to do high school outreach. Every few months, we get a Friday off work to go visit a local high school. It's a pretty sweet deal, except for that part where school starts way earlier than we want to wake up...
ReplyDeleteMy whole beef with this is that a lot of the outreach is haphazard and not highly productive. To really run a good program reaching out to, say, minority middle school students, one has to know a lot about not just science but also teaching and devote a lot of time and energy to this. Doing that well is a career on its own. While I completely agree it's noble to do small things on the side, one cannot be both a highly productive researcher and independently run a successful, data-driven, effective outreach program. Instead NSF policies are encouraging half-measures. We won't select the best researchers like this and because each researcher must do something on his or her own and researchers are not experts on outreach or teaching, we won't have effective outreach either. It is the worst of both worlds.
ReplyDeleteI served on an NSF GRFP panel once, and I agree with the implication of HRF that "extreme" BI is expected for an application to be highly ranked. Many other reviewers felt that participating in existing programs that involve outreach to public school students, for example, is not enough, but that the applicant must initiate a new program for it to count. This situation has obviously come about because students at some institutions, who receive good coaching and many BI opportunities, so they no longer stand out from the crowd on BI grounds.
ReplyDeleteI turned down a request to serve on one of these panels the next year, largely because I felt that the system has gone overboard on this criterion.
Note that I just posted a comment but accidentally hit Enter before I completed the Name box. It probably has my name as Fem or Fema . . . Please attach my full "name" to that comment, if possible.
ReplyDeleteI think the whole problem with BI is the inconsistent standards and the complete panel-dependence of what is acceptable and what isn't. I have no problem with students doing BI, and I think they should -- but just like our proposals, they should be held to a clear and reasonable standard, not something arbitrary.
ReplyDeleteI am completely unconvinced that the current emphasis on broader impact works at all and suspect it is counter-productive: my understanding of the history of science suggests that broader impact often comes through serendipity and is virtually impossible to predict in real-time for an individual research project.
ReplyDeleteFurther, I feel that the entire point of NSF is to support basic research because the OVERALL broader impact is clear whereas individual basic research efforts are hard to justify in industry or in other government agencies with specific missions. Thus, the emphasis on broader impact seems to me to actually erode NSF's mission.
I have said this to program managers but it's mainly met with shrugs. I have also contemplated making a boiler plate version to put in every broader impacts section I review, but I worry that it might count against the proposal I'm reviewing.
Finally, let me say that this is my strong opinion despite the fact that I often do fairly applied research and tend to benefit from the broader impacts rating.
On my NSF GRFP reviews, the first reviewer thought my BI's were "excellent." My second reviewer gave my BI's a "poor." The reason? I was entering graduate school many years post-undergrad and was doing a major switch in fields. During the in between years I did many things, but science outreach was not one of them. So I wrote my BI's to emphasize the sorts of things I had done to in my non-science (athletic and non-profit) life to help and include "underrepresented groups." Reviewer 1 thought this was great. Reviewer 2 didn't think any of it counted at all, since it wasn't science. I was lucky and Reviewer 3 agreed with Reviewer 1 and I got the fellowship. But it could just as easily have gone the other way.
ReplyDeleteI think the main problem with BI's as they stand now is that the interpretation of BI's is so variable. How many of us try to come up with flashy, but low-commitment BI's just to get a chance of future funding? That's the game to play if you're a grad student these days.
Yes, yes, yes, FSP! I tend to get high marks for BI and have tried to help my students as well, but especially for newcomers, there is just so much they can do. And as others point out, grad school should be a place to focus on research, not have to suddenly in the first year develop all kinds of outreach programs.
ReplyDeleteAnon 9:12 - I don't think anyone here (in the post of comments) is arguing that students shouldn't learn about BI and participate to some extent. The point is about the level of expectation and whether BI should be the deciding factor in selecting among excellent NSF GRF proposals.
ReplyDeleteEli is hatching a plan just along these lines for his next center proposal. . . . :)
ReplyDeleteI couldn't agree more. It is borderline ridiculous to require BI at all for student fellowships (though like FSP I am ok with requiring it of profs, within reason). The whole point of requiring BI of researchers is to share the excitement of doing cutting edge science with those lower on the educational ladder. But there has to be a period where people are learning how to do the cutting edge science before they can share it.
ReplyDeleteIf the trend continues, you can imagine programs where the K-12 students will have to do BI themselves, reaching out to students still in utero.
I think what I object to the most is the idea that contributing to an existing program is somehow lazy or uncreative. When I was a student, there were a few outreach programs in the department. I wasn't one of the main people involved, but I saw that the people involved worked incredibly hard to keep it going and make it better. A student who joins a program like that, learns the ropes, becomes a leader, uses their energy to keep it going and keep people involved, and innovates in some way to improve the program, is making a tremendous contribution. To the extent that the person takes advantage of existing resources while doing that, so much the better. The biggest insult would be to pass over those students and reward the person who decides to ignore their hard work and instead re-invent their own wheel. Plugging into an existing program is NOT taking the lazy way out. If anything, the lazy way out is to promise some wheel re-invention, do just enough to show that you tried, and then let it predictably fizzle.
ReplyDeleteFrom what I've seen, the main impact of the BI requirements has been to put in place a bunch of rather useless little programs, sapping any faculty interest in participating in any established programs that might actually do some good.
ReplyDeleteI'd much rather see NSF go back to funding research and outreach separately, as they currently are damaging both the selection of good research and the selection of good outreach for stuff that does both less well.
As a white male, this has factored into my not bothering with several "big" fellowship applications (not NSF).
ReplyDeleteI can't disagree with the importance of outreach. So many people have no idea what scientists do all day, and/or hold crazy stereotypes about the kind of people who are scientists. But, I'm a first year grad student in the middle of writing an NSF GRFP and have been advised that the BI aspect of my essays may be more important than the proposal itself. Surprising, but possibly true? Faced with a bunch of solid proposals, I suppose reviewers will parse out the great from the excellent by using the BI?
ReplyDeleteI agree with several comments that say the problem with BI is that it is reviewed in a very inconsistent way. For the most part, on the (astro) panels I've sat on, I think the panels have struck a fair balance between understanding that some activity should be done, but that it needn't be all-consuming. On the other hand, I have sat on at least one panel where a reviewer said at the beginning "I don't believe in BI, so my reviews don't factor that part in." That was ridiculous and (in my eyes) unethical.
ReplyDeleteI think the NSF would be well-served by really strengthening the description of what is expected, and making those expectations very clear to panels (along the lines of a judge giving pre-deliberation instructions to a jury!). I think they do try and do this, which is good; it just needs to continually be emphasized.
I agree with the several commenters who would rather the NSF fund outreach separately. At the big R1s I've been affiliated with, the school has hired outreach coordinators to package BI programs for its professors, so they can compete better for grants. That's nice and all, but if the NSF is going to pay for that (they are, indirectly) they might as well direct the resources where they will do the most good. Would you rather have an outreach coordinator in Cambridge or Palo Alto, or at a HBCU or rural land grant? Right now we have the former, and it's yet another disadvantage for the latter.
ReplyDeleteI also agree that the lack of standards is frustrating. We are all socialized to have some sort of conversion table for IM-type achievements...one Nature article = X J.Subfield articles = Y invited talks, and so on. BI, not so much. Some people care that you've checked all the boxes (like Alex #1 got), some people care about new and flashy, others don't care at all and will only go after your BI if they didn't like your grant to begin with. Maybe this will settle down with time, but it hasn't yet.
I will say, though, that information on this is widely available. I didn't have any help whatsoever on the NSF - I took several years out after undergrad, so all I know I learned from the internet. I learned how important BI was, and wrote to the criteria; my personal statement was mostly given over to this, and I hit it hard in the other sections also. People can find these things out if they look.