A few years ago, some colleagues and I pitched a 'high-risk' idea to program directors at a funding agency. The program directors were receptive, but funds were scarce (of course) and they suggested that we submit a full proposal through the usual channels, get destroyed in review, and then they could use the short-sighted negative review comments to support a case for circumventing peer and panel review.
Well, they didn't describe it exactly like that, and they did hold out the possibility that our pessimism about taking this idea through normal review channels might be unfounded -- perhaps we could get the full proposal funded on the first try!
So we wrote the proposal and were pretty thoroughly destroyed in review. Reviewers were convinced our central idea was misguided, even absurd. (spoiler alert: They were wrong.).
But we got a pilot project funded, and gave the project a try. I was convinced that the research was worth doing, but I thought it would be more difficult than it turned out to be (smug alert: I was wrong). In fact, within 6 months we had the results we needed to validate the idea and show that it was worthwhile to move forward with this line of research.
I don't really blame the skeptical reviewers. Although sometimes it seems like PIs need to do most (or all) of a research project before we can convince reviewers and panels that the project is doable (and worth doing), I admit that in this case, the idea was a bit surprising.
Nevertheless, armed with our new and exciting data, we started writing a full proposal again. The new proposal was even more fun to (re)write than the first proposal because we had the new preliminary data as strong and (I think) compelling core of the proposal. As the proposal-writing progressed, I kept reminding myself to look over the old reviews in case there was something worthwhile in there that should be dealt with in the proposal rewrite. Even in situations where reviews are overall not very useful, there might be something useful that can be mined from them.
For example, there might be some minor points that need attention. These might be minor, but it's a good idea to take these into consideration in case the proposal goes back to the same reviewers.
But I kept putting it off. At pretty much the last minute, I skimmed the reviews of the old, rejected proposal.
I am not usually such a coward about reading or re-reading negative reviews, and I didn't really feel particularly wounded by the reviews, even the first time I read them. I just felt like it would be a real downer to read those reviews again, and I was in a very good mood about the new proposal. Yes, I know we might be eviscerated in review again, but at the time of (re)writing, I was in that happy delusional state when I really like the proposal and am excited about the prospect of doing the project. I was reluctant to put a dent in that happy mood. And I knew that one of my colleagues had re-read the reviews in detail.
I mentioned to another colleague -- one who is not involved in this particular project -- that I needed to re-read some negative reviews of a proposal I was rewriting, and he said that he would rather have his fingernails ripped out than re-read old reviews of a rejected proposal.
That seems a bit extreme, but sort of captures the essence of my feelings about the matter.
I am not advising that anyone not re-read negative reviews when it is clearly in one's best interests to do so. It must be done, however unpleasant, but I think it is also important to find a way to minimize any damage such a (re)read might do to the positive energy that is driving you as you revise a proposal (or manuscript).
It occurred to me that, had I been thinking ahead when I read the reviews the first time, I could have jotted down any points worth addressing in a future proposal, and thereby avoided the full re-read later.
But if, like me, you don't think about that in time and you have to re-read the stupid reviews, perhaps the re-reading is best done in a pleasant cafe (or bar), or at home with your most sympathetic pet nearby, with your favorite music playing. Or you could re-read the annoying comments on a crowded bus or subway or at the dentist; that is, some place with unpleasant distractions. Or read them with a good colleague, take whatever is useful from them, and laugh about the rest.
My advice: Read them, but don't let them make you defensive (in your proposal) and don't let them stop you from enjoying writing your cool new proposal.
14 years ago
19 comments:
What would happen if you traded review-reading duties with a friend (possibly the colleague you mentioned)? Your friend could read the negative reviews you got, jot down any points that are still of importance, and present this summary to you. In return, you could do the same for old negative reviews of your friend's proposals.
I re-read reviews of rejected proposals all the fucken time. It is extremely useful. Why do you think football coaches and players spend huge amounts of time watching game film? Hint: it ain't to watch the amazing catch over and over.
negative reviews/comments hurt my ego but it only have an effect on the first go with me. If you just use them as data points I don't really see why the emotional attachments....maybe I haven't had it as bad yet? I don't know.
When it comes to negative reviews of proposals or papers, they always seem terrible when I first get them. But when I go back and reread them, like you, after some time and the rejection blow have passed, I actually never find them to be quite as bad as they first seemed. I think it's never really about the content of the reviews, but about the emotional impact of rejection (no matter how much we may have been prepared for it).
You make it sound like reviews actually take time to read. In my experience to date (~20 proposals across NSF, NIH, DoD, and private foundations), my average reviews are about 1 page of text across 2-3 reviewers. And if I cut that down to actual criticisms/suggestions (rather than the overview statement, evaluation of me) it's less than half a page. Also, many of the 'new investigator' awards come back with NO review released.
Needless to say, this is incredibly frustrating - I don't expect to get everything (or even many things) funded, but I used to expect reviewers to take a little time so that I could hopefully build from the reviews for the future submissions as you suggest. I no longer do.
I don't mind re-reading negative reviews when it gives me the opportunity to write something along the lines of "While the prevalent opinion in the field is that A, B, and C could not be done due to X, Y, and Z, our preliminary results clearly show that given the right expertise and an innovative approach, these obstacles can be successfully overcome" in the grant/manuscript re-write. I am currently in the wonderful position of writing a manuscript about the results of an incredibly successful experiment that all 3 grant reviewers were sure would be impossible. And, since the result involved a patentable invention, we may use the negative reviews in our patent application as proof that our discovery is "non-obvious." :)
My PhD advisor would wait a week after rejection before reading the reviews. He knew the wound was fresh and he would be less receptive to the comments, even if they were helpful.
At the time it drove me nuts because I just had to know why right then and there why the paper or grant was rejected, but now that I am older (wiser?) I understand. I don't wait a week, but after I have accepted the rejection I read the reviews and find I am much calmer and open to the comments.
"Although sometimes it seems like PIs need to do most (or all) of a research project before we can convince reviewers and panels that the project is doable"
This particularly rankles me. They might as well give cash prizes for finished research.
Perhaps CPP has more rejected proposals and so has more practice at this.
Of course I have tons of rejected proposals, dumshitte. If you don't have tons of rejected proposals, you don't have tons of granted proposals.
I read the reviews of my rejected proposals multiple times, and even look at them again months or even years later. It is extremely instructive in terms of gaining an understanding of your weaknesses in grantsmanship. Without understanding your weaknesses, you can't possibly attempt to overcome or work around them.
it was a joke. anyway.
"They might as well give cash prizes for finished research."
That is, in fact, becoming a major model in computer science. Think of the DARPA autonomous vehicle prizes or the Neflix contest. Each of them got the prize-giver easily 100x as much research done as direct funding would have, and there was no need to read any proposals.
Of course, this also means that a lot of researchers worked for free.
I got a major revision for a paper that we thought was a wonderful thing with results n all....was bummed but you made my day ! kudos to all the dumb reviewers :D
So you re=read the rejections, and answer everything and they bring on another set of monkeys.
Totally know how that goes. I usually try to do it at a time I can be annoyed and then do some mindless work afterward, so that it doesn't hurt my productivity. I just need to be annoyed for a while then I can come back and reread them with the mentality that I just need to make these points more clear to that even idiot reviewer X can get the idea. It's taken me a while to get here though.
Does CPP always put things into sports analogies? Can he be any more testosterone-infused? Can we request that he just leave those sports analogies for his blog so I can read this blog without having to think about football?
Or, he could just substitute "ballet" whenever he feels the need to compare science to a male sport. I already do that when I read his comments.
This discussion is surprising to me. Either people are more sensitive than I am (truly hard to believe), or reviewers' comments in other fields are far more harsh than they are in mine (astronomy). I think most of my colleagues in the field are fairly careful to write constructive criticism, mindful that a real person will be reading the criticisms. We get referee's reports on manuscripts all the time -- these are no worse, are they? The consequences are worse, but the comments surely are no worse to read?
Cash prizes: I've always thought that we should have at least some grants given out this way (like, smaller ones and more of them than the Nobel Prize). People who do great science should be rewarded and enabled to do WHATEVER other great science they feel like doing. I've always felt frustrated that reviewers are most eager to fund you for projects in which you have an established track record, which could well be a topic that you are now bored with, rather than the fresh new ideas and new territory. Besides, we often modify what our grants fund as the research evolves, so that much of what is originally proposed is superseded - so after all the haggling over the details of the proposal during review, the grant may fund something somewhat different anyway.
FSP, I really appreciate this post (I've been out of town and am just catching up on my important blog reading!). I've been struggling to get funding the past little while, and tried one of these "preliminary grants at the discretion of the program officer" approaches last summer and got the same response (submit a full proposal and we'll see what happens). I just got the reviews back and, no surprise, they panned me. I hadn't even considered using this as ammunition to go back to the PO and see if they'll reconsider that high-risk granting mechanism. But it's now order of business #1. Also, if you have any suggestions for getting out of a funding slump, I'd love to hear them!!!
I once saw a presentation by a PI who recounted his many attempts to obtain R01 funding for a particular project. He may have submitted variations of the proposal ~!0 times, and he listed the main concern of the reviewers for each submission. The reviewers were beyond skeptical about every aspect of the project. It was not until the work in question had been approved by the FDA that study section finally awarded him this R01. Hahaha. I guess he loves to re-read those old reviews and think about what boneheads all the reviewers were.
Post a Comment