Something that I have been seeing more and more in grant proposal reviews (my own and those of colleagues who have shared theirs with me) is the idea that it's not enough to have a record of success advising grad students, undergrads, and postdocs in research -- you have to understand and explain your advising techniques and you have to have a plan for assessing and improving.
OK, I get that, but even when I attempt to do those things, it isn't good enough for some reviewers. They think that I (and my colleagues) are relying too much on past success and traditional measures of success (degrees, publications, conference presentations, post-graduation employment). They are not convinced that that is sufficient. They want something different. Apparently, unless you change something, you are not improving and therefore are not being transformative, or something.
Example reviews (comments condensed/reworded to remove any identifying vocabulary):
A highly qualified PhD student has already been identified for this research but the mentoring of this student and an undergraduate is largely assumed based on prior experience of the PIs. The PIs have records of successful advising but should include in the proposal a more intentional discussion of how they plan to train the next generation of scientists. The mechanism for success is not explained and there is no plan for assessing success of their mentoring. How will successful training of the graduate student be determined other than by the record of publications, presentations, and completion of the thesis? Although the research is potentially transformative and this is an excellent team of researchers, because of these shortcomings in the broader impacts I have given the proposal a lower rating.
That makes no sense to me. I am definitely not saying that we all deserve to have all of our grants awarded just because we have had past success. However, I think that if the proposed research is deemed excellent by a reviewer and the PI has a demonstrated record of success with advising, it does not make sense to downgrade a proposal rating for the reasons given in the example review above, contributing to the rejection of the proposal and therefore a lack of funding for the graduate student.
Here's another:
[From a review of a proposal that included one week of salary for a soft-money research scientist who runs a lab in which students would do some analyses for a proposed project]: Description of the mentoring of the postdoc is not well developed. There is no mention of career counseling. Mentoring in professional activities such as writing proposals and papers is confined to discussions and support for participation in conferences and workshops. There is no mention of how the postdoc will be mentored to collaborate with diverse groups of researchers and students. There is no description of the postdoc's career path in the context of developing an effective mentoring plan for him.
And this:
[From a review of a proposal that included a substantial component of support for undergraduate research]: These PIs have a long record of success in advising undergraduate students in research but no evidence is presented for how the field of research on undergraduate research will be advanced.
These are just anecdotes, of course, plucked from reviews of different proposals by different PIs. At least one of the proposals even involved a colleague who does research on teaching and learning. It wasn't enough. Some of us PIs have attended national and local workshops on teaching and learning, read some of the relevant literature, even co-authored papers (some with education specialists) in science ed journals. It's not enough.
I think that giving attention to effective advising is an important component of research (and therefore grant proposals), but I also think these and similar reviews show that certain reviewers have run amok and are harming the very people (students, postdocs) they think they are helping.
OK, I get that, but even when I attempt to do those things, it isn't good enough for some reviewers. They think that I (and my colleagues) are relying too much on past success and traditional measures of success (degrees, publications, conference presentations, post-graduation employment). They are not convinced that that is sufficient. They want something different. Apparently, unless you change something, you are not improving and therefore are not being transformative, or something.
Example reviews (comments condensed/reworded to remove any identifying vocabulary):
A highly qualified PhD student has already been identified for this research but the mentoring of this student and an undergraduate is largely assumed based on prior experience of the PIs. The PIs have records of successful advising but should include in the proposal a more intentional discussion of how they plan to train the next generation of scientists. The mechanism for success is not explained and there is no plan for assessing success of their mentoring. How will successful training of the graduate student be determined other than by the record of publications, presentations, and completion of the thesis? Although the research is potentially transformative and this is an excellent team of researchers, because of these shortcomings in the broader impacts I have given the proposal a lower rating.
That makes no sense to me. I am definitely not saying that we all deserve to have all of our grants awarded just because we have had past success. However, I think that if the proposed research is deemed excellent by a reviewer and the PI has a demonstrated record of success with advising, it does not make sense to downgrade a proposal rating for the reasons given in the example review above, contributing to the rejection of the proposal and therefore a lack of funding for the graduate student.
Here's another:
[From a review of a proposal that included one week of salary for a soft-money research scientist who runs a lab in which students would do some analyses for a proposed project]: Description of the mentoring of the postdoc is not well developed. There is no mention of career counseling. Mentoring in professional activities such as writing proposals and papers is confined to discussions and support for participation in conferences and workshops. There is no mention of how the postdoc will be mentored to collaborate with diverse groups of researchers and students. There is no description of the postdoc's career path in the context of developing an effective mentoring plan for him.
And this:
[From a review of a proposal that included a substantial component of support for undergraduate research]: These PIs have a long record of success in advising undergraduate students in research but no evidence is presented for how the field of research on undergraduate research will be advanced.
These are just anecdotes, of course, plucked from reviews of different proposals by different PIs. At least one of the proposals even involved a colleague who does research on teaching and learning. It wasn't enough. Some of us PIs have attended national and local workshops on teaching and learning, read some of the relevant literature, even co-authored papers (some with education specialists) in science ed journals. It's not enough.
I think that giving attention to effective advising is an important component of research (and therefore grant proposals), but I also think these and similar reviews show that certain reviewers have run amok and are harming the very people (students, postdocs) they think they are helping.
Some people have drunk way too much kool-aid.
ReplyDeleteA thought about why this might be: the current job market is atrocious and it has changed a lot in recent times. Here in the UK you need teaching qualifications (almost) as much as good research experience. It's possible the funding agencies are being proactive in trying to ensure there is a specific career development plan in place that reflects the current climate. If so, hurray!
ReplyDeleteThat may be wishful thinking, of course, but it just occurred to me as I read this that as a post-doc, I would have quite liked to know there were specific measures in place to develop my career beyond participation in the research programme.
I saw this on a proposal review here in Australia this year and was surprised by it. But apparently this is common.
ReplyDeleteThis is awful. I have not yet seen this problem in the areas of NSF I deal with (DMR, CBET). We're certainly required to write about these issues but in general "track record" seems to be strong evidence in the grants I submit or the panels I've been on as a referee.
ReplyDelete"How will successful training of the graduate student be determined other than by the record of publications, presentations, and completion of the thesis? "
ReplyDeleteWhat else do they want? A report card? Aren't successful outcomes indicative of success? I'm a little baffled.
I think the magic word is "portfolio". I'm not entirely sure what a portfolio is, but the assessment police seem to like it even if they can't explain what one is either.
ReplyDeleteBTW, I suspect that the person collecting one week of salary for a small part in this project will spend 40+ hours doing paperwork and sending emails to transfer their effort to the grant in question and then transfer their effort to some other grant and handle the inevitable snafus. Perhaps a good mentoring plan would mention training in bureaucratic hamster wheel running.
ReplyDeleteI can certainly see how this would be frustrating, but I do think as another commenter noted, the state of the job market is driving this set of newish requirements. The funding agencies want to know that the funding will benefit trainees who are going to stay in the system and have success themselves as future PIs. I completed two postdocs before leaving academia, and I think I would have felt better (and perhaps fared better) if there were very specific mechanisms in place to ensure that I was getting the mentoring that is foundational to future success. This of course all assumes that good mentoring automatically leads to future success---I don't think this is universally true, but good mentoring certainly can't hurt.
ReplyDeleteI see a couple of different ways one could look at this. On the one hand, perhaps there were other proposals in the pile from excellent researchers that were also potentially transformative and included well reasoned mentoring plans. On the other hand, perhaps the PI's track record is not as great as she/he thinks it is, and the reviewers are trying to help the PI realize that it can't just be business as usual in the current job market.
ReplyDeleteLet's be honest - how many of us have 100% of our former students/postdocs employed in the field of their choice? Have any of us NOT had trouble placing recent grads? I've been thinking fairly often about how to help my people succeed, and about how many new scientists I want to throw into the mix. If someone has a strong "portfolio" or plan for maximizing the chances of success, I'd sure like to hear about it! Because the last thing I want is for the people I train to languish underemployed in the current job market.
It sounds like this could also be the case when reviewers literally have to come up with reasons not to give funding to otherwise excellent proposals. The amount of funding has dramatically decreased in some programs, yet researchers are getting better at grant-writing skills and if there are "too many" excellent proposals that make it to the top pile, some have to be cut out, somehow. For some reason, it feels like these reviews could reflect such a situation.
ReplyDeleteDon't worry too much. Reviewers sometimes say stupid things. Sensible committees ignore them.
ReplyDelete
ReplyDeleteOff topic - but I read this and thought you'd be interested.
http://www.theguardian.com/commentisfree/2011/oct/14/does-a-question-get-science-paper-cited
This is just NSF trying to split hairs between proposals that should be funded because there is not enough money to fund even the top tier
ReplyDeleteI'm with ID. People are looking for reasons to reject otherwise great proposals because there is not enough money.
ReplyDeleteAlso, while mentoring is great, it does not solve the NO JOBS problem. I am seeing people with good pedigrees and great mentoring, but the only thing that's going to increase their job success is academic nepotism. With so many good people competing for so few jobs, what really sets a person apart? Sometime excellence. Sometimes relationships.
It is really quite simple, the reviewers were looking for a plan/ideas that could be shared with others to improve training and benefit multiple students and advisers. Given such a plan they would like to see examples of how the principles you have developed worked out in practice. They would also like to see how you will do new things that you believe would work even better and how you will monitor the effect of those things. Eli would have written the same review and suspects you are being rather clueless about all this.
ReplyDeleteThat is a very interesting point of view. Yes, a well-established and published lab should indicate a good place for a graduate student to grow, however, I think that a mentoring plan has its place in the review process as well. See my thoughts on your post at Robust Mentoring Plan is Critical for Graduate Student Success. I welcome your comments!
ReplyDeleteWhile Eli and the European Biotechnologist do make a good point, the conversation reminds me of one I've had with peers: one must always be improving in academia, and if not improving, churning out bull*(&^ phrases that nod to improvement. That 20-year track record of success is not the important thing in the grant application, but rather what you write about it.
ReplyDeleteOn the one hand this is good. People can't rest on their laurels, they're pushed to improve, and they need to articulate what they do and plan to do. We always tell students on exams in the US that it's not what they know, but what they write down for evaluation, that's being graded.
On the other hand, this is bad. People who do have a great track record of actually doing the mentoring but who do not have the currently valued turn of phrase will not succeed. People who are able to write the more compelling grant will get the money, whether or not they effectively carry out the interpersonal duties outlined. (You can check all the boxes and be a bad mentor.) And change is valued for changes' sake. When 'no one can rest on their laurels,' people who do good things can't simply continue to do them. They must change even if it makes things worse.
It is something I hate about the current incarnation of academia, this change for changes' sake. It's elsewhere, too. For instance, in some place if you'd like to earn a master's in education you need to implement some reform. So everyone is implementing reforms to check off a box even if things are going well. In the VA medical system, each administrator has to implement some reform to show they're being useful -- not simply do a good job, but change something. And so performance measures for physicians change every single year and sound good on paper but aren't necessarily thought out well. A friend of mine is at a SLAC and she's a great teacher. She'd like to swing some of her energy to research now that she's established her great teaching methods. But for every performance review she's got to come up with another teaching technique to change (clickers, flipped classroom, IBL, something!). These are all fine techniques, but do we really have to switch from fad to fad every year, with the additional demand that someone justify the fad with an eloquent grant proposal?
From talking to friends in the corporate world, I get the impression that the chasing of fads is not so different from their world. They have consultants and workshops and hot books full of buzzwords and managers talking about strategic plans, just like us. And most corporate employees hate that stuff as much as we academics do, but their managers eat it up as much as our deans do.
ReplyDeleteI'm in agreement with those that say this is hair-splitting to find some basis on which to reject the proposal because the funding agency has a lowered budget. And I'm betting that if you were Prof. Bigshot from the Massachusetts Institute of Berkeley or whatever, they wouldn't be going over the broader impacts with a fine-toothed comb.
ReplyDeleteI wrote earlier that it is important to have a mentoring plan which will benefit the student. However, we must also consider that despite our training as scientists, we are all really in the business of sales and marketing!
ReplyDeleteI'm pretty new to submitting grant proposals and was surprised to get really negative feedback on the "broader impact" part of an NSF proposal, because I proposed that I would join a highly successful established outreach program in my city. Some reviewers thought this wasn't "innovative" enough. I can understand wanting research to be innovative, but now our outreach is also supposed to be innovative? "Stick with what works" isn't acceptable, I guess.
ReplyDeletePersonally, I'm heartened to hear that funding agencies are insisting that PI's put substantial thought into how to help their students and postdocs succeed in the job market. I just completed my second postdoc in mathematics and am looking for nonacademic employment, having failed to find a tenure track position. My two postdoc departments had night and day differences in how they treated their postdocs. In the first position, every postdoc had at least one mentor that they worked with on research. My PI helped me connect with opportunities to work with different groups, mentor others in the department, and promote myself. In the second math department, postdocs were totally on their own. Most didn't find local collaborators. The grad students in the department were treated like the department's children. PI's met regularly with their grad students, made sure they were finding interdisciplinary collaborators, and helped them market themselves on the job market. The department's "alumni" website proudly lists placement information for each PhD grad student, but just PhD school and disertation topic for the poor postdoc alums (wish I had seen it as a red flag). The second postdoc did provide me with time and funding to pursue my research, but they really could have done a lot more to help me prepare for an academic career. I'd be thrilled if funding agencies held the second department accountable for helping postdocs develop as interdisciplinary researchers. For some departments it's tempting to focus all resources on their students and postdocs can get the short end of the deal.
ReplyDeleteI get these hyper-critical comments on the broader impacts section of my proposals but my husband (in the same field/same institution) doesn't. He either gets praise (for example just for advising female students) or he gets a blanket 'broader impacts are good' summary for his cursory broader impact statements. I have advised more students (and all have jobs) and yet I have to prove myself more than he does.
ReplyDeletePart of it is that NSF now asks for reviewers to comment on mentoring plans when they recruit reviewers following the new GPG. Yet, they aren't given much guidance on what to look for and haven't read many yet. As a result, many reviewers look at the list of what might be included in a mentoring plan and see it as a checklist rather than a list of potential options. This is not all that different from the transition to Broader Impacts that we saw a decade or so ago. Reviewers still don't seem to know how to review that section well. I my experience reviewing NSF's review process, program managers help calibrate the sometimes crazy reviews of the BI sections and these scores are not at all correlated with the funding outcomes. I suspect they are now doing the same with mentoring plan statements although these are not as distinct as the BI reviews. That said, it doesn't mean that a reviewer's feelings about the MP don't affect their overall review in ways that aren't easy to identify.
ReplyDeleteAnon 6/11/2014 03:41:00 PM, try explaining how you will add value to the established program and how the program will support your efforts. You did include a letter of support from the program?
ReplyDeleteMentoring for postdocs is often sorely lacking because many PIs don't actually do any mentoring of postdocs. They just treat postdocs as highly skilled lowly paid employees to work in their lab. that's fine if that's what you want, but then don't claim that you are 'mentoring' them. So, what would mentoring of postdocs look like? Do you actively include your postdocs as co-PIs in your grant proposals so they can participate not just in the technical writing but in all the 'behind the scenes' activities such as talking to program managers, in some cases (especially in my field) schmoozing the program managers? Do you give your postdocs opportunities and help with getting their own grants or showing him how to promote themselves the way you do? Also, how about doing something novel like having your students and postdocs give anonymous feedback to the granting agency about YOUR performance as their "mentor"? How about setting up a mechanism where they can do this without you seeing what they have to say about you? If you (not you specifically, FSP, but PIs in general) think you're such a good mentor, let's see what your current and past "mentees" have to say about you and their experiences in your lab, but to be meaningful they have to be free to say the truth which means anonymity and that you (the PIs) don't get to see what they say about you. Simply showing that students and postdocs have published papers doesn't mean a PI is a good mentor. I know PIs who yell and curse at their students, insist they work long hours, do not allow them to take weekends or vacations, all so the PI can get the papers he/she wants. so the student's track record looks OK - shows publications, but the experience was terrible and was more like a regular job rather than 'mentoring' experience.
ReplyDeleteThis is insane! And you know, the only thing that will change is that someone will write some boilerplate language that everyone will use that will describe the details of the ostensible mentoring that is taking place.
ReplyDeleteHow infuriating.
With acceptance rates trending below 10%, sometimes panels have to come up with some comments for rejected proposals that they find perfectly fundable, just below the cut. Broader impacts is often a good area to put these comments if the intellectual merit is considered strong. I tend to suspect these kind of comments would drop if funding rates were back up at 30%.
ReplyDeleteI think anonymous at 9:09 hit the nail on the head. Its consistent with my experience in panel reviews in the last few years. There are now a large number of excellent proposals which are not being funded. There is little meaningful distinction between these proposals and those that do get funded.
ReplyDeleteI think reviewers are stretching to find excuses ...
Having had advisors that could have cared less about my career and professional development and training, I wish they'd had to have given some thought as to how they'd train me. I'm with the earlier poster that the changes in the job market, and less academic positions, means that methods of mentoring 20 years ago may not be working now. If your student/post-doc wants to pursue a non-academic career, do you have the tools to guide them in this direction? Can the project be guided in this direction etc...
ReplyDeleteIf the proposal is pitching undergraduate research as a significant educational activity and contribution, then the actual *education* needs to be described. There is no correlation in the literature between scholarly productivity (or not) from an undergraduate research experience and the quality of the educational experience. Au contraire - the greatest learning from undergraduate research comes from having the chance to try stuff and screw it up... (again, see the literature). So I too would sniff around the 'track record' for evidence of the educational benefits, IF that is the claim put forth in the proposal.
ReplyDeleteAnon at 6/15/2014 03:40:00 PM has it right. The practical solution to this is some boilerplate language. I love reading FSP's posts and (some of) the comments, but the most useful thing at this point would be if someone would post the right boilerplate language to use! It must exist already.
ReplyDeleteI'm not in a position to use post docs, but many of the comments have the right of it.
ReplyDeleteSince I've been writing grants, I have seen an increasing requirement, not for good science, but for "other crap." Mind, it's well intentioned, and there for a reason, but just adds more labour to the grant author. Data management plan, mentoring plan, broader impact, etc. The actual SCIENCE part is getting squeezed out to the point where I feel that unless you have some bat-poo insane research, or made some personal enemies on the panel it doesn't really matter what you propose. And that annoys me.
The copy paste boiler plate will come for this too. I wish that we could just write in "as per the standard plan" and only elaborate if the mentoring plan (or even broader impacts) would be a truly *important* or innovative part of the project.
My thoughts are to develop an Individual Development Plan (there are some available online) where you meet with your student or postdoc, discuss career goals and determine where your student/postdic needs to improve and where they are doing a good job. Help them identify resources where they can improve their skills, (for example, going to the University Writing Center). My university sponsors career development workshops on scientific writing, nonacademic job tracks, collaboration, ethics in peer review, etc. I suspect that the grant reviewers are looking for language like that?
ReplyDelete-A Postdoc