What do you do if one of the program officers at a funding agency to which you send proposals is not as professional, sane, objective, and/or non-hating-of-your-guts as you might wish them be?
How common is this?
I suspect it isn't very common in its most severe form. I can think of maybe 3 colleagues in the past 15-20 years who have had this experience at such a serious level that they had to change their research topics because there was no way they were going to be funded in a certain program by a certain program officer no matter how awesome their proposals.
Even so, I am wondering how common it is for someone writing a proposal to feel some anxiety about the program officer's personal opinion of them, objectivity about certain types or subfields of research, or other aspects that don't strictly involve the "intellectual merit" of the proposed research -- whether or not the anxiety is based on experience or even reality.
There are many things to be anxious about when submitting a proposal. Where does anxiety about program officers rank among them? I refer here specifically to anxiety about program officers, not reviewers or panel members or others who might be in the reviewing chain.
For me, anxiety about program officers has always been extraordinarily low or non-existent, but lately it has shot up to close to the top of the list. I hope this is temporary, and I hope certain program officers are very temporary.
14 years ago
24 comments:
I stopped sending to NSF for a few years, because i believed that the program officer was strongly biased against me. Since then I have made it a point not to know the program officers, so that I can continue to write proposals under the belief that it is the merit of the proposal that matters. (I also mainly submit to NIH these days, though I'll probably try NSF this summer also, since I'm completely out of funds.)
I was wondering if FSP or another reader could comment a little on who program officers typically are? How would they run into and come to hate specific faculty members? Are they full-time NIH/NSF/etc employees, or is it a part-time position faculty members take on similar to being on an editorial board? (If it's the latter, I certainly understand how some old rivalries would brew to the surface and prevent impartial decisions.)
Our funding agencies run slightly differently here, so we have different problems. I found out after having a proposal rejected that the program manager had just finished a PhD in a different field of Science to the one in which I work. And this person is qualified to oversee review of my proposals how exactly?
You realize that your readers are now dying to hear the details, right?
This can be a problem especially given the stimulus funding and a lot more people applying for grants. One would expect NSF profram officers and reviewers to be objective and maybe they are more objective than most but only to a point. As with everything else it is who you know that matters more than what you know given equally qualified proposals.
When I was a new professor, I was afraid of program officers (in the same way that I was afraid of all Famous Scientists in my field, including my graduate committee). That's different from stress related to knowing and disliking the officers, though.
I have not been impressed with NSF PO's so far.. I'm trying to figure out which program officer I should talk to about my CAREER proposal. Of the 3 I've contacted, only one has replied.. and in a 2 sentence statement said essentially 'you might be under me, but try the other guys first.'
Great. way to be helpful for the new kids..
Many NSF people I have encountered are there on a one-year "sabbatical" from a faculty job, and others seem to have a high turnover rate, so it may be a temporary problem. While I don't have NSF funding myself (except a recent meeting grant, in which my experience was very good), a colleague recently had a bad situation occur, with a program officer who was new, and told him a grant was funded, only to back down several weeks later.
The discretion program officers have at NSF versus the more numbers based approach at NIH is both a strength and a weakness.
Mark P
My goodness, that never even occurred to me as something to worry about. But I haven't been submitting proposals for that many years. And, I'm bad about making those phone calls to the program managers to get advice that you're supposed to do.
Last year I was co-PI on a proposal that was recommended for funding by the panel but declined by the program officer. That was a clear sign to me that there was no point in resubmitting the proposal (again, because of course it sometimes takes a few tries) until the program officer changed.
I've never had this experience. It sounds horrible. DoD funds for a while, maybe?
Fuck, that sounds horrible! I have never submitted to NSF, but all of my NIH program officers have been really nice, stand-up people with a friendly demeanor and an understanding of and appreciation for my science.
very timely topic. I am waiting to hear from my PO at NSF, and I'm freaked out (every day with no news is a bad day). Anyhow, this proposal was assigned to somebody else on the first submission. It got very consistent reviews, and talking with the PO I got a good feel for what I needed to do. Clearly the PO liked the proposal, and was v. encouraging. I resubmitted. It was assigned to a different PO. Reviews were all over the map. Panel v. positive, but no funds; they asked specifically for prel res on X. I talked to the new PO, who is not encouraging at all and in fact tells me to not rush to resubmit. ???? Quick explanation on the realities of tenure clock, students needing summer RA, etc. Went back to lab myself to get results on X; everything is peachy, and in fact I'm preparing a manuscript on X. I resubmitted, but this time I felt like I had no feedback on what was the problem. And it looks like it wasn't funded. This PO clearly did not like the topic, and I suspect didn't like that the other PO had been encouraging. Quick ISI search reveals that PO #2, a faculty on rotation at NSF, has a total of y of papers in marginal journals, on a topic not related to the panel's focus. Y is smaller than what I had as a graduate student.
Meanwhile, my competitor got NIH money on a related topic. Am I paranoid in thinking 1- NSF will never fund this, as long as PO#2 is there; 2- i'm wasting my time. it's scary that somebody with that much power has little track record. A colleague of mine got a proposal funded in spite of "medium priority" recommendation from the panel, because his PO "liked it". geeeee.
Disclaimer: although this particular proposal is not going well, I have two federal grants (including one NSF) and a foundation grant. It's not just sour grapes :-)
Just as a counterpoint, I've generally had very good interactions with my P.O.'s at NSF. The one for my research grant can be a little terse, but everything he has ever told has been pure gold. The P.O. for my REU Site grant is very forthcoming and replies to all queries very quickly. Both of those P.O.'s are permanent. I have heard a few bad (and a few very good) things about some of the temporary* P.O.'s.
* NSF is staffed with both permanent and temporary P.O.'s. The temporary ones are generally faculty who spending a year or so as a P.O. I'm not entirely sure how they are chosen.
I'd encourage people submitting proposals to avoid blaming program officers for their proposals' failure to gain funding.
From the comments I've heard and the panels I've been on, there is almost a complete disconnect between perceived problem personalities and the real reason a proposal did not reach the fundable rating. Unless one directly heard the inside scoop from someone on the panel (even then the message often gets garbled).
Most readers probably know this, but proposals depend (1) on the panel of experts, (2) on the mail in reviews, and (3) least on the program officers. Often, the panels recommend many more proposals for funding than there are funds, and half of all reviewers are simply out to lunch, either in praise or criticism, and the panel knows it.
Further, proposals in the grey area on the edge of funding are hard for POs to deal with, and if a proposal moves from likely to be funded to not funded, it is not usually the PO's doing.
Another detail - doing what improvements one panel recommends for a rejected proposal usually doesn't bring it up to being funded the next round. It doesn't usually hurt, of course.
The PO job is very different between NIH and NSF (I've been on about a dozen panels for each, and have some professional colleagues who are permanent as well as past NSF rotators).
NSF has good POs and bad POs. A subset of the good and the bad know how to manipulate the review process so they have a large say on what gets funded (depending on the division, this subtlety is often lost on the panelists). A subset of NSF program officers think they know a lot about their field but are 20 years out of date, and some are downright crazy and/or have an agenda (NSF doesn't have a process, as far as I can tell, of seriously evaluating them). Some are excellent, provide great customer service, and take their job seriously. So they've got them all. I think that is part of the problem - no standards or quality control.
NSF culture, and even the process for reviewing grants and deciding on funding, varies dramatically between directorates. Some use panels, some use ad hoc reviewers, some use ad hoc reviewers followed by a panel. At the end of the day, the PO has a degree of discretion in funding decisions within a given classification (Highly Competitive, Competitive, Not Competitive).
It is my impression that many NSF POs were not exactly successful academics (but some were). However, the two do not necessarily correlate. I've known at least one mediocre academic who became an excellent PO.
I've seen POs run panels that are stretched well beyond the panelists' expertise comfort level, I've also encountered panelists who were borderline incompetent and had done nothing in a field in 15+ years, but were considered experts on that topic for reviewing those proposals.
Some NSF POs are rotators (1-2 years on leave from academia), and others are permanent. Some directorates utilize the rotators as their recruiting ground for the permanent positions. (In general, I think this is a good thing, but I don't know how common the practice is).
I might be sounding cynical about NSF. I believe in their mission, but think they are seriously understaffed, need some clear vision and direction (like NIH did with their Roadmap), and some operationally-minded folks to put processes in place to evaluate programs as well as the performance of program officers.
NIH program officers have much less say in what gets funded (the council sets priorities), and you live and die by the score a lot more. However, a good NIH PO will shop a borderline fundable proposal to other institutes and try to get it funded. POs do not run the review process (the SRAs do), but a good PO will attend a panel and give you a briefing of how the discussion went. Sometimes the summary the panel writes does not really convey the weighting of the issues and how they were discussed.
It is also my impression that NIH SRAs and POs have much more time for career development, staying on top of their field, and doing a good job, while NSF POs are spread rather thin and don't have enough hours in their day. I've known of good NSF POs who left for NIH and are much happier there.
Then there is DoD, where the POs get to basically call all the funding shots. Unlike NSF (as far as I can tell), DoD actually holds POs accountable for the success or failure of their portfolios.
Anything I say here could be wrong, and is based on personal observation, experience on panels, and buying people beers.
I have no personal evidence of a PO (NIH in my case) being out to get me, I find them to be informed and professional. The only thing close would be me wanting them to go out of their way to help and them not doing so- but that is a far different matter.
One thing for which I have theoretical concern is the presence of POs at study section reviews (in person or on the phone). If a reviewer were seen by a PO as regularly making their life difficult by beating up grants they wanted to see funded...that may lead to something like what you are proposing here.
Oh, FSP, I hope this person goes away soon for your sake.
Really useful comments here, too, very interesting discussion!
I had a funny experience recently where someone from a grant agency, not realizing I am still a postdoc, asked me to serve on a review committee, and then revoked the offer upon seeing the title line in my email signature file.
Suffice it to say, it was all I could do not to jump down this poor person's throat. Apologies aside, I do wonder if it's going to work against me that they said they will follow my career and invite me again later. After what I said, I don't know if that will actually happen!
My other program officers have just confused me. My NRSA SRA person was no help; fortunately my advisor knew what to do, or I would have been up a creek without a paddle.
When I called to ask why my K-grant didn't get funded and what I could do to improve it, the verbal answers I got on the phone were nothing like the written reviews. In other words, the person told me to basically IGNORE the written reviews and address a different problem altogether in my revision (which I couldn't do).
Which left me wondering what the hell went on in the discussion of my grant, would I have heard the real story if I had been in the room? And afterwards I wondered if I should have ignored the program officer instead of taking their word as gospel?
I think these people have too much power, it's one of the weakest links in our current "system" that the written reviews of the study section aren't required to stand on their own, and that there isn't really any quality control on who gets these positions.
Program Officers are people, too, which means that there are good ones and there are not-so-good ones. And some of them do most definitely play favorites. In my experience, it is much better if a PO likes you and your work, and it can be deadly if the PO decides - for whatever reason - that they don't like you. Even at the NIH, where the POs have less leeway than they do at NSF, they can pull grants up to the payline and move grants down off the payline. There are almost always ways to justify these decisions.
I am not making this stuff up. I speak from experience with both agencies. The best advice I can give anyone is to nurture their relationship with a PO. They have a lot of power. Of course, it always helps to score in the top 2%, but if you're in that 15th-percentile zone, the PO can make or break you.
I'd suggest some people may blame the PO for funding problems when the trouble lies with the panel, the reviewers, or most commonly, shortcomings in the proposal itself.
Very few scientists submit proposals they consider undeserving of funding. With regard to questioning the qualifications of POs, I'd offer that given a good panel and a couple of thorough reviews, most people off the street could rank proposals accurately.
Even if some POs play a role, the case in which they can affect the outcome is also the case in which their influence is difficult to discern. In my experience, panels rebel against prejudicial interference from POs.
I had a person come up to meet at a meeting and tell me that they had reviewed my last 4 proposals submitted to this call at NSF. They happened to be someone who I turned down a job to work with as a Post Doc (they offered, I declined). I don't think it had anything to do with it. They told me that they didn't like my grammar. Then they told me that my repeated submissions just sounded like I read over the comments from the panel and then responded to those and submitted the proposal again. Hmmm... ok... That is what I did.
Now this person is the program officer for my field. Only for two year, which is good.
Another program officer has always supported me and has funded every proposal that I have submitted to them as a PI. So, I can't really complain.
Dear John, I'm one of the anonymous with problems with some Pos and their qualifications. I have been on several panels. Have you ever tried to look up who got funded from your panel? I have. I have been surprised by this exercise. One person who did not fare well at all in the panel, but had good friendly reviews from a narrow group of reviewer (people in the same field, linked by having been postdoc/graduate of each other, or at the same time). That person got funded. Other people who had great ideas etc, but maybe not enough prel res, did not, in spite of ranking higher.
One issue you're forgetting is that the PO chooses the reviewers. And the panelists.
Yes, I realize the PO picks reviewers, favorable reviews can lead to favorable initial placement on the list for the panel to consider, and there is inertia for proposals to stay near their initial ranking during panel discussions.
And I did not check proposals funded against our ranked list at the end of the panels, however, I trust the particular PO in my NSF panel to stick to our rankings.
I did see the reviewers chosen for the 6 rounds of (100+ proposals each time) during my 3-year NSF term, and did not see any obvious sandbagging or boosterism by selective choice of reviewers. Or maybe the PO and I shared the same narrow point of view. I did hear a number of people griping groundlessly about the process.
Granted other panels and POs might be run differently.
My ex (male) had this problem and commented about it more than just in passing. His field has some serious disagreements and well-entrenched camps, some of whom are peer-reviewers for articles and some of whom are w/in NSF. He hasn't gotten grants for a few years, but I don't know whether that is because he changed schools or money is tighter, or what. But his complaints about reviews and program officers in the "other" camp were frequent enough to stick in my mind, six years on.
Post a Comment