A recent article in naturenews (Nature, 26 May 2010) discusses NSF's "broader impacts" criterion for proposals and notes that the definition of "broader impacts" is so broad that many PIs are confused. Although NSF does provide a handy list of examples, questions remain about what types and amounts of BI activities are most appropriate for a particular project (and PI).
[Note: Here are links to a few old posts on Broader Impacts: 1/10, 11/09, 10/08, 6/06]
Although some PIs do innovative BI activities, many do not. Apparently, the #1 BI in proposals submitted to NSF's chemistry division involves the training of graduate students and postdocs; an important goal, but that many of us do anyway. Hence the perennial question: Do BI activities have to go beyond what we usually do, or do we just have to do a better job at the advising/mentoring aspects of our job, or at least say that we will pay more attention to these things?
I don't know, but I found one of the relevant anecdotes in the article strange and unconvincing. The story involves the admirable efforts of a physicist, Diandra Leslie-Pelecky, who conducted a project in 2001:
In many ways, it was typical of the kinds of things that NSF-funded researchers do to fulfil [sic] their broader-impacts requirement. She took three female graduate students on weekly visits to local classrooms, where they spent 45-minutes leading nine- and ten-year-old children in practical activities designed to teach them about electricity and circuits. The visitors also talked about their lab work and careers. In addition, Leslie-Pelecky did something less typical of broader-impacts efforts: she brought along education researchers to study the effect of this interaction on the children's perception of scientists.
Those assessments were startling, she says. After three months, most of the students said that they still weren't sure who these young 'teachers' were – except that they couldn't possibly be scientists. In their minds, scientists were unfriendly, grey-haired old men in white lab coats.
"And that's what I worry about with broader impacts," says Leslie-Pelecky. "There are a lot of people putting time and effort into [these sorts of activities] and they have no idea if they're making any difference or not."
The anecdote ends there, leaving me a bit confused. I can see being concerned that these kids didn't understand that the graduate students were scientists, but that doesn't make me worry about "broader impacts" activities. That makes me even more convinced that we need programs of this kind, perhaps with a different approach to explaining (in this particular case) that the young women were scientists. What else, other than meeting real live examples would convince these kids that scientists don't necessarily fit their image of scientists as cranky old men -- an image that surely discourages a lot of kids, especially girls, from considering science as a career option?
The article mentions the idea that the goal of BI might be better achieved if BI activities were organized at the level of the institution, involving people with expertise in such things, who would then involve certain faculty in particular programs.
There are some aspects of that idea that I like, but I think it would be a mistake to remove all responsibility for BI from the individual investigators. The BI criterion, despite the confusion surrounding it, is a move in the right direction: that of asking investigators to think broadly about their research, to pay attention to the training aspects of our research, and possibly to consider how our research affects the non-academic world.
Furthermore, in the individual project-based system for BI, there are many opportunities for creative integration of the "intellectual" and "broader" aspects of research. A centralized office of people who are experts on "broader impacts"-type activities, but who don't necessarily understand the science and engineering research itself, might not innovate in the same way that individual PIs can.
I also would be concerned about a system in which this centralized office called on "certain faculty" to be involved in BI. I have a feeling I know who these faculty would be and that level of involvement would vary drastically across the institution, with confusion about whether this type of activity was valued by departments, was appropriate for tenure-track faculty etc. etc.
I agree, though, that if the intent is to do effective outreach on a large scale, then that type of BI activity should involve those who have specific expertise and time to devote to such things. Perhaps there is a way to have the best of both types of systems: e.g., to have more institution-based support systems to help PIs with BI activities (some universities have this; some don't), yet without completely relegating the organization and implementation of BI activities to the institution.
I'd say one problem in the whole balancing BI activities between PIs and universities is the question of funding.
ReplyDeleteI suspect that, in this tight economy, increasing BI support at an institutional level would be accompanied by a request for a greater percentage of grant funds.
Since PIs are being charged with the responsibility of implementing BI activities, why doesn't NSF simply require submission of an evaluation rubric at the time of grant submittal? In the summarized story, perhaps the rubric would be "X% of students agree with the statement: 'scientist is a job that both girls and boys can excel at'"
These sorts of "breaking misconceptions" really do need to start with the students identifying their misconceptions first. (lots of educational research done on that!)
ReplyDeleteSo, for example, to get across that scientists are not just gray haired old guys, you have to have the students articulate first what they think a scientist looks like (draw a picture, talk about it, etc.). Then you can address that scientists can look like anyone.
I agree that broader impacts shouldn't be run at the institution level but instituitions can help. For example, our university organises an annual community day when the public visit and we scientists all put on demos of our equipment/expts. I also think more sharing of information about BI activities - what worked, want the kids understood etc would be great. It would mean each lab doing BI doesn't have to start from scratch but can learn from others. But a good system for that is not necessarily easy to implement in practice.
ReplyDeleteI think that one problem raised by those results is that the students aren't likely to consider what those women showed them to be "science" and that attitude would only snowball as they moved along in age.
ReplyDeleteStill, the "broader impact" mandate is so inexact that it has to be frustrating. I know that when I've gone out to grade school and high schools in an attempt to share my research and expertise, that I have to dial back that information to such a basic level ("No, Sebastian, World War II wasn't part of the Middle Ages!") that it can be daunting!
My concern with situations like this anecdote is that there weren't "impacts" in the "broader actions." Surprisingly, scientists assume that their actions (ie teaching science at a school) has an impact. I find this ironic, given that as scientists you'd expect all of us to hypothesize an impact to our actions and test this hypothesis. Since we all have such limited time to spend, stories like this should remind us that we need to evaluate the effect of our actions. I can't imagine how a centralized office would help; however, a couple of education research consultants (not busy education profs) could be a great asset to science (and most other) profs interested in the effects of their outreach.
ReplyDeleteMy university has a number of proghrams that the PI can plug in if they want and reach out to K-12 or the broader community. I think those are well run, and people in them seek a balance between education professionals and scientists, which is the way it should be, right?
ReplyDeleteUnfortunately, I have found at the NSF that some program managers in certain divisions consider this to be "lazy" and not creative enough and want you to do a lot of legwork on your own. For other divisions it works great. So go figure... But I agree with FSP's general sentiment that it should be made easier for PI's to participate in meaningful and worthwhile BI activities, with a sensitivity to how much time is being spent. It is clear that institutional support for BI activities is then be crucial.
The 9 and 10 year olds were pretty smart. I don't know what grad students are, either, but they aren't exactly scientists. Oh, I know we call them scientists and they do as much science as anybody else. But we are also happy to exploit their dreams and treat them like dirt. We'll let them postdoc but 90% of them will never be able to get a permanent research position. Pretending that grad students have real careers is just propaganda and I am glad the kids saw through it.
ReplyDeleteGreat post.
ReplyDeleteAnother federal agency I write grants for insists that education and outreach activities associated with the otherwise totally research-based projects must contain an assessment component. It's not enough just to DO the outreach, you have to also try to figure out the impact.
ReplyDeleteThat makes perfect sense to me, and I have, when appropriate and relatively easy, engaged some relevant assessment experts in proposals & projects. But sometimes (many times) you are not really breaking new ground in the assessment, and as long as you've been through some assessment projects before, you can do a good enough job to at least draw some conclusions about the value of your efforts.
I'm neutral on the benefits of requiring a broader impact statement in NSF proposals. I view them as simply requiring the PI to put forth their case to be relevant and useful to society, in addition to good science. That's a good exercise, and everyone should recognize that quantifying the quality of BI is somewhere between not worth the trouble and impossible.
ReplyDeleteStill I can argue with several statement here.
1. Repeatedly people complain that the public has the inaccurate image of scientists as old white guys. Repeatedly people here complain that scientists ARE largely old white guys. These conflict.
2. I disagree that much outreach should come from the majority of individual scientists. We scientists are partly in this field as eggheads who passed on the chance to be in the more social fields of law, medicine, and engineering. While some scientists are great communicators, most are not, and many never could be.
Organization at the college and university-wide level is more effective, especially at large state schools with a mandate and resources to boost education at K-12 as well as higher levels.
This is OT to the BI question, but Bitter Anonymous at 9:08 says that a graduate student is not a scientist. I don't see it that way. The parallel might be that a medical student is not a doctor, but the MD is a professional degree, with board certification, licensing, etc. To be a scientist is to have a certain view of the world. Graduate training hones that worldview, or is supposed to, and provides tools, both intellectual and technical. Students may become more effective scientists, but if they weren't scientists to begin with, they wouldn't have come to graduate school.
ReplyDeleteI had a seminal experience as an undergraduate and it came in my first biology class. I was an English major, but the college was so small that our liberal arts requirements were met by taking the real introductory courses designed for the majors. This was not a "rocks for jocks" approach. The professor of that course only had a Master's, but she was a biologist. It was part and parcel of her worldview, the scientific, rational, "test it" approach ran deeper in her than in many people I've met who hold PhDs and work in labs.
In my opinion, those grad students are scientists as much as someone with the degrees, position, publication record, and grant funding. I considered myself a biologist long before I got the letters after my name. Grad school made me a more effective biologist, but if I hadn't thought of myself that way to begin with, I would never have gone.
Good point, Shippen.
ReplyDeleteJohn V: There's no conflict. Most scientists are white guys (maybe not necessarily old), but the perception that ALL scientists are white guys needs to change (especially when someone -- young or old -- is presented with specific evidence that a woman -- young or old -- is a scientist).
It seems to me that "broader impact" activities include both activities directly related to the project and also "good citizenship" activities that a public-spirited scientist should do but are not related to the project.
ReplyDeleteFor instance, going to an elementary school classroom: Great idea, and I laud those who do it. However, as Janice said, there's only so much from your own research that you can bring to them. This isn't a natural extension of the project (compared with, say, producing college course materials that directly incorporate cutting edge research at the undergraduate level, or mentoring under-represented students, or bringing undergrads into the project, or letting high school students use some of your equipment for their science fair projects, or public lectures that bring in your own research at the level of an educated adult, or whatever).
Also, as geekmommyprof said, if you are doing this as part of somebody else's outreach program, it isn't even clear that your research grant was directly relevant or necessary for you to go to that classroom and do demos or whatever. However, it might be useful if the university accountants that FSP has lamented start asking why you spent an hour in an elementary school classroom. Of course, you could always go and reinvent a wheel just to show that the grant really was necessary for outreach, but wouldn't it have been a better use of everyone's time (and NSF's resources) if you had just plugged into somebody else's existing program and augmented it?
Also, it's interesting to consider this: I have a heavy teaching load. The majority of the students in my classes would fit one or more of the following categories: Ethnic minority, economically disadvantaged, female, first generation college student. So let us compare two statements of how one spends time, and consider which one makes a better broader impact statement.
1) "When the PI is not in the lab doing this project, and not teaching one course per semester at an expensive university with mostly traditional students, he/she will spend 2 hours per week doing science demonstrations in elementary school classrooms."
2) "When the PI is not in the lab doing this project, he will spend 12 contact hours per week helping more than 100 undergraduates prepare for STEM careers. A majority of these students are ethnic minorities, and/or women, and/or economically disadvantaged, and/or first generation college students."
The second BI statement would probably not read as well, and might be summarized as "Doing his job." The first would be considered extraordinary effort. So the second person would probably be asked to design an additional project.
As a reviewer, the best BI statements I have seen are from people who have access to public outreach facilities at their (usually large) institution. A museum, an active education school, etc. I wish I had access to those kinds of resources, because reinventing that particular wheel is very daunting.
ReplyDeleteA couple times I have seen amazing plans for BI accompanied by a request in the budget for a full-time "tech" who does nothing but the BI activity for a year or two. Pretty gutsy, considering budget limitations, but seriously, why should we expect that BI activities cost nothing to implement? Or that they don't take time away from the other activities in the project?
FSP
ReplyDeleteWhile I like the quantitative approach of Buck's study, we're concerned because 15 "disadvantaged" 10-yr-olds are claimed to have trouble distinguishing between the vocations of teachers and scientists, given that they observed junior scientists AS teachers? Table 2 in the paper, the only mention of the significance of the result, is utterly uninterpretable in context. It would not surprise me if the standard deviation in the statistics, sketchy as they are already, assume each response is independent, clearly not correct for a cohort in two classrooms.
The conclusions are entirely believable, but the study doesn't show much.
But that wasn't really your point. I'd still argue that the public's perception of scientists as predominantly men is accurate, and their perception of us as more coldly rational than humanist is accurate, no matter if we take off our lab coats to feed them a PR campaign about really having friendly attitudes and diverse populations. We should improve the situation, but I see the intelligence of the public being undersold here.
I typically sit in two NSF panels from different divisions, and I'm appalled by how different the two program managers are in regards to what they expect from the BI section. One (incidentally a woman), takes it very seriously and wants the panelists to discuss the BI section during the grant discussion. The other (incidentally a man), says that we shouldn't worry about it and he'll take a look at the applications that end up at the top to see that they have 'something decent'.
ReplyDeleteI personally like the idea of BI, and I think it forces some people to think about some issues that they would never think about otherwise. However, the way this is implemented if feels like a lottery, and often a huge waste of the applicant's time.
I saw the same discrepancies regarding follow-up after the award. One program manager couldn't care less if I did what I promised in the BI section (even for my CAREER), and the other one is known to check on PIs more closely.
I think the NSF should clarify what they want from the BI internally, before they attempt to clarify it to the rest of us.
My last successful NSF grant was before the era of "broader impacts". Instead I've been having to deal with NIH "significance".
ReplyDeleteEveryone here seems to be interpreting "broader impacts" as "outreach to minorities and women" rather than as looking at the impact of their research. I think I like the NIH approach better here, in that they are asking the researchers what effect their research will have, rather than asking them to be incompetent elementary-school teachers.
Re assessment: there are whole fields (or so I've heard) that study the subtle effects of wording on surveys and polls. I'm all for assessment, but I don't think a simple "Whaddya think?" questionnaires designed by a non-expert is the best way to go about it. Maybe a sociologist can weigh in on this.
ReplyDeleteWhat's with the bashing of scientists visiting elementary schools? My kids go to a good school with dedicated teachers. But frankly most of the teachers are not comfortable with many scientific subjects and, in my experience, most welcome the involvement of local scientists. Most of us volunteers are also parents, and we have experience talking to munchkins, after all.
ReplyDeleteIt's true that I don't talk about the intricacies of my NSF-sponsored research, but I can talk about underlying scientific concepts: evolution by natural selection (150+ years old, and still cutting edge in much of the country!), the scientific method, genetics and inheritance, GMOs, etc. etc. etc. I have colleagues who volunteer to tackle sex ed and STDs. The problem of scientific illiteracy in this country doesn't just start with high school kids!
It's true that I don't talk about the intricacies of my NSF-sponsored research, but I can talk about underlying scientific concepts: evolution by natural selection (150+ years old, and still cutting edge in much of the country!), the scientific method, genetics and inheritance, GMOs, etc. etc. etc. I have colleagues who volunteer to tackle sex ed and STDs. The problem of scientific illiteracy in this country doesn't just start with high school kids!
ReplyDeleteAnd that's great stuff that a good citizen-scientist should be applauded for doing. The question, though, is whether that is a broader impact of the research project. Do they want you to have something tightly integrated with your research, or do they want you to be a good citizen-scientist who is public-spirited in outreach while productive in research? And whatever it is that they want, do they want you to do it on your own (with the inevitable wheel reinvention by non-experts) or do they want you to plug into existing efforts (with the charge of "laziness" that geekmomprof highlighted)?
Shippen: "To be a scientist is to have a certain view of the world. Graduate training hones that worldview, or is supposed to, and provides tools, both intellectual and technical. Students may become more effective scientists, but if they weren't scientists to begin with, they wouldn't have come to graduate school.
ReplyDelete...
In my opinion, those grad students are scientists as much as someone with the degrees, position, publication record, and grant funding."
In context, you are wrong. This was about encouraging kids to consider science as a *career option*, not as a worldview. Your worldview doesn't make you a scientist in terms of having a career. No more than having a worldview as a doctor makes you a doctor, or having a lawyerly worldview makes you a lawyer. Calling doctors and lawyers "professionals" doesn't help your case.
Medical students are much, much closer to being doctors than science graduate students are to being scientists (in terms of a career, in terms of worldview they are hopefully the same).
I think the quickest and easiest way to change perceptions would not be through broader impacts, but by a cartoon heroine like Dora the Explorer.
ReplyDeleteI think a cool science cartoon heroine with mentors who are female would make an impact - perhaps the next Disney movie? Or perhaps Pixar?
"What's with the bashing of scientists visiting elementary schools?"
ReplyDeleteI'm not against scientists visiting schools! Personally, I spent a couple years running an after-school tech club for 2 hours a week, taught a week-long programming class last summer, and coached a math team for an hour a week this year. None of that had anything to do with any of my research, though, and I don't see its relevance in a grant application. Might as well ask me if I recycle and ride my bike to work---it's just good citizenship and has nothing at all do with my research proposals.
To me it seems like a big problem with the "broader impact" requirement is that it's ill-defined and therefore not uniformly enforced. Also, putting it into a research grant doesn't really seem the way to go. The best would be if our community actually valued community outreach, and until it's taken into account for hiring decisions, it simply won't be taken seriously.
ReplyDeleteI do lots of outreach because I find it very important and I enjoy it, but I'm told by my colleagues that spending time explaining what I do to the public is a hobby, not work, and it doesn't count towards my productivity. It isn't valued when I apply for jobs anymore than the fact that I like to run and brew my own wine. The truth is that spending two more hours per week in the lab would advance my job prospects more than spending that time justifying the public funding of science to the public. Until that changes, people are always going to try to squeak by with the minimum required.
I too find the Broader Impacts business ill-defined.
ReplyDeletePart of the problem is that BI is evaluated by grant evaluation panels, i.e., reviewers from the research community. My impression is that no one in the community really understands what the NSF wants in the way of BI, so grant-writers try to guess based upon their own preconceptions. Panelists/reviewers rate grants based upon their preconceptions, and the resulting standards are inconsistent. I find it peculiar.
I'm sure you know the joke: How do you figure out how long the Emperor of China's nose is, when no one is allowed to see the Emperor? You ask a thousand people and take the average of their guesses. The point is, taking the average opinion of a bunch of people who have no information still doesn't give you any information -- and that goes for NSF grant evaluation panelists and their evaluation criteria for BI.
We sound like a bunch of scientists - let's find a way to repeatably and objectively measure Broader Impact.
ReplyDeleteHowever, merely planting the idea that science should have a broader impact is the majority of the goal in requiring a BI statement, I'd guess.
I'll go out on a limb to assert it would be a waste of proposers, reviewers, and panelists time to draw up an extra set of guidelines, metrics, and a calibrated scale to capture BI, which I take to mean some combination of long-term improvement of social welfare, entertainment of the public with wowwy-zowwy science, development of the scientific workforce and infrastructure, and education of the masses.
Does anyone really want to have to tackle this in well-defined detail in every NSF proposal? I think I can't define BI, but I know it when I see it.
I think I can't define BI, but I know it when I see it.
ReplyDeleteJohn V,
I completely agree with you.
And this would be perefect if BI was scored on a binary, pass/fail scale (e.g., you do or you don't have what feels like broader impact).
I think most people are wondering how much and what type of BI is enough in order to ensure that your BI doesn't sink your entire proposal. Different PM's vary widely in their view of adequate BI, variations among divisions can be like night & day, not to even mention different panelists... With some PMs, BI becomes *the* gatekeeper for the science that is to be done, so having some mechanism to nail down the moving target that is BI would be appreciated...
Anon: Being a doctor is a profession. Being a professor or researcher is a profession. Being a scientist is a bit, in my opinion, like being a humanist or a $religion-ist: it's all about the world view. But we're probably splitting semantic hairs. I just find it sad that you don't seem to place much valence on the love of science and reason that brought most of us (who once were students) to the table. Of course, there's always Matt Cartmill's quote, and he seems to use the word the way you do: "As an adolescent I aspired to lasting fame, I craved factual certainty, and I thirsted for a meaningful vision of human life -- so I became a scientist. This is like becoming an archbishop to meet girls."
ReplyDeleteAnd on reflection, this does have a bit to do with Broader Impacts. I had no idea I was that interested in science until I met a real scientist.
I have sat on NSF panels in several different programs, and each panel had a different requirement for BI. Some reviewers seemed to think that only "creative" BI proposals were worthwhile, but I completely disagree. Why should faculty re-invent the wheel? It's more time- and cost-effective if a faculty member can take part in a proven program that really works; that is the type of program that's most likely to make an actual impact. It's silly to have over-worked faculty (and especially junior faculty who need to focus on setting up a lab and developing new classes) with no educational training be responsible for creating new educational training programs. This is a waste of tax-payer money and faculty time. Advanced degree programs in education exist because creating these types of outreach programs requires specific expertise in education. I believe NSF and universities should employ educational experts to create effective programs in which science/engineering faculty can participate. If NSF wants to require that STEM faculty provide outreach, I think that's a great thing - but the outreach should be made to be as effective as possible. Otherwise it's just a waste.
ReplyDeleteFor those of you for whom this is tl;dr: The bottom line is that Broader Impacts are relevant to the mission of NSF. This is from my reading, not from any conversation with NSF program officers. Also, "Broader Impacts" is misread by some reviewers as a code word for promoting racial diversity. It's meant to be more than that.
ReplyDeleteThe NIH version is the Relevance or Narrative (depending on whether it's the PHS398 or the SF424 form), not the Significance. Significance is, as was noted, a measure of how the work will move the field forward. The relevance is what the work has to do with public health, because the NIH is funded as part of the Public Health Service. In other words, the research funded should be relevant to the mission of the funding agency. "The National Institutes of Health (NIH), a part of the U.S. Department of Health and Human Services, is the nation’s medical research agency-—making important medical discoveries that improve health and save lives." Thus, even work on the 1000-celled worm C. elegans has to be related in some way to human health in a way that can be credibly stated in lay language.
The mission of NSF? "To promote the progress of science; to advance the national health, prosperity, and welfare; to secure the national defense...." That mission has evolved such that NSF is "responsible for the overall well-being of science and engineering across all disciplines." (1997 GAO report)
The report goes on to say, "NSF is also committed to help ensure the nation’s supply of scientists, engineers, and science educators through the financial support of education and research." Why is this outdated GAO report relevant to the discussion? Because NIH was "dinged" for not having any measurable way to assess how "influences outside their control" would impact NSF's overall goals. The report notes, "... the extent to which schools and universities emphasize mathematics and science or subsidize faculty research are influences outside of NSF’s control. These influences could affect NSF’s realization of its goal aimed at improving the achievement of mathematics and science skills needed by all Americans [emphasis mine], or its goal of making discoveries at and across the frontier of science and engineering." In other words, Broader Impacts are a way that NSF tries to have a positive impact on influences outside their control. Better overall science literacy in the US will enhance the ability of NSF to achieve its goals.
NSF makes investments. Your research grant is an investment NSF has made in you in order to advance the foundation's goals in science and technology. Because another of NSF's goals relates to overall science literacy and creation of the pipeline of new researchers, then you can think of the Broader Impacts requirement as the way the foundation leverages its investment in your research to advance more than one goal.