The blogosphere has done a rather thorough job of thrashing the recent *scathing* report by Senator Coburn (R-Oklahoma) on supposed mismanagement and waste by the NSF. I will therefore quell the urge to point out how poorly researched, written, and organized the report it. It's a good thing the report wasn't funded by NSF or it would indeed be an excellent example of wasted effort and money by that agency.
I was, however, bothered by a particular section that has not received as much attention: the part about the supposedly shocking number of late and never-turned-in project reports:
A 2005 audit found that “[a]pproximately 47 percent of the 151,000 final and annual project reports required in the past 5 years were submitted late or not at all.”
There is a handy chart, organized by Directorate, showing the % final reports submitted on time, late, or not at all. I have some points to make about this:
1 - It is not surprising that so many reports are late. Reports are due 90 days before the end of the budget period, and at that time some of us are still scrambling to submit additional grant-related publications to include in the report. I would rather submit a late but more impressive report than an on-time report with less substance. One of the main motivators for me to file my reports (annual or final) sooner rather than later is that it is necessary to be up-to-date with reports in order for the paperwork for new grants to be processed, but if there is no pressure from that, I will wait until I can write a more complete and informative report.
2 - According to the chart in Coburn's document, although there are many late reports, the number of missing (never submitted) reports is low. It is misleading to combine late + missing reports and then focus on that number as if there is a major problem.
3 - The reports are administratively important for NSF and are one way that the programs/directorates can see what has been produced from grants (publications, education, outreach etc.), but the reports are not the actual "products" of the research and therefore are an imperfect measure of project success. It would be more relevant (but perhaps more difficult) to look at the actual outcomes of a grant -- as opposed to whether a project report has been filed on time -- and to look at contributions and impact beyond the expiration date of the grant.
I also wondered about the numbers related to unspent grant money apparently lying around, uncollected by NSF. Does this include money that PIs retain as part of no-cost extensions that give more time to do the research, or is that apparently leftover money accounted for because the expiration date of the grant changes when a no-cost extension is approved? If you only consider the original expiration date of the grant, it would look like there is unspent grant money. I couldn't tell from the report whether the unspent money was real or not.
And did anyone else think it was strange, given Coburn's concern for duplication and overlap among programs that fund STEM education for underrepresented populations, that the subtitle of the report, "Under the Microscope", is also the name of an NSF-funded website created by The Feminist Press to encourage girls and women to be involved in STEM fields? How do we all feel about this duplication of titles? I don't have any real data, but I am sure I am against it.
I find his report a blatant ploy to further divert funds away from building US infrastructure to large corporations. Many of the "useless studies" he cites are taken out of context.
ReplyDeleteFor example, I believe a machine learning group recently published an algorithm and used "predicting winning teams for college basketball" as an example result. While I don't think he cites this specific study, he seems to allude to a study involving march madness as a waste.
So, if IBM's Watson had been developed academically, I'm sure he would cite "a computer that plays Jeopardy" as a waste. But as long as the taxpayer doesn't have to pay for it, there are tv documentaries about how Watson is a major advance in artificial intelligence.
The report is nothing but disgusting spin.
And now I feel guilty about my two end-of-the-year reports that I am tardy on.
Who cares?
ReplyDeleteWhile you point out some very key observations, you're trying to apply logic (that one might learn in a STEM education) to a political argument that is for grand-standing and quite often illogical.
ReplyDeleteI know if we (as researchers relying on NSF funds) don't stand up, we're going to get steamrolled. But just pointing out the grand canyon sized holes in their argument isn't going to win a primarily political battle.
I noticed the late report thing too, and it ticked me off; the problem arises only when the report is overdue, and, as you said, there's a 90-day period between due and overdue in which people a trying to wrap up a few more papers and submit.
ReplyDeleteBut as Gears said below, this is a political ploy. It's not like reason has anything to do with it. It's not about factual inconsistencies, it's about spinning through taking things out of context (an excellent alternative to lying btw) and thus scoring cheap points. The NSF is an awesome scapegoat: its budget is really a trivial percentage of the federal budget, yet bashing it scores points for "curbing wasteful government spending" and vilifying those elitist scientists, while not really endangering any conservative's core principles (instead of, God forbid, touching the untouchables: raising taxes on businesses and cutting defense spending).
The NSF has a very effective way of insuring that reports are filed, they simply will not fund any annual supplement or new grant where the PI or any of the co-Is has missing reports. Concentrates the mind.
ReplyDeleteAs to the 1.7 of unspent funds that was also confusing. If you take the Coburn report at its word, this is for grants that have been closed out. Since no cost extensions extend the grant term, grants with NCE are not closed out.
FSP, are you a hypocrite for expecting your students work to be turned into you on time, but you give your reports late?
ReplyDeleteI suspect most taxpayers would view this discussion with anger. Scientists are given thousands to millions of dollars, with a stipulation to provide a couple-page report at the end, and large numbers can't even do that on time, nor in language making plain why the public should pay for the work.
ReplyDeleteExcuses that fulfilling the contractual NSF requirements would impede the effort to get the next increment of taxpayer money would not be compelling.
Many of my own reports are also late, but I don't get righteous when people complain.
Yes, it's way more important for the bean counters to have all their beans on time than to submit a complete and substantive report that gives them the information they really need. And unless you always give your students an extension on homework, you are a self-righteous hypocrite. Have a great weekend!
ReplyDeleteI think that the best framing is not one of outrage at being questioned on this, but rather an aw-shucks approach. "Well, you know, I always want to do things on time, but with these long and complex projects, sometimes the key pieces of data aren't all in until after the arbitrary date set by the bureaucrats. Those darn bureaucrats! I sure try to get things done on time, but you know how those bureaucrats are with timelines that make no sense..."
ReplyDeleteAn assumption in this discussion is that reports are late because people want to write complete and substantiative reports.
ReplyDeleteMy experience is the opposite - people who can't meet report deadlines corollate with those will not get it together to write complete and substantiative reports. And corollate with people soaking up disproportionate amounts of NSF time chasing their late reports and never-arriving proposal reviews.
The wording "some of us" (submit late reports so that they will be more complete) hardly seems like a sweeping assumption. I am sure it applies to more than one person. Actually, make that two people.
ReplyDeleteMake that at least three people. I'm actually just submitting my late (but not overdue!) report today. I checked FSP while NSF was processing my pdfs.
ReplyDeleteMy understanding was always that "late" means "overdue" -- and as such it carries penalties in terms of NSF looking at your other proposals etc.
ReplyDeleteThe 90 days between "due" and "overdue" is simply an expected report submission window.
NSF annual reports are not a couple of pages. They are about as long as a proposal and take 2-4 weeks to put together and they had better be good or you ain't gonna get your annual supplement
ReplyDelete@ anons around 1:20pm
ReplyDeleteThe wording "some of us" (submit late reports so that they will be more complete) hardly seems like a sweeping assumption.
I read the post as, in part, discussing the meaning of the "shocking number of late and never-turned-in project reports" cited by Coburn. So I don't think 1 or even 3 cases of late-report explanation would dominate that statistic.
Again, I send in plenty of late reports, but consider it a bad habit, and usually observe that it takes more effort and adrenaline to do it late than just doing it on time.
Why the hostility among "some" commenters about late reports (and those who submit them) anyway? I think the reports should be done within the 90 day window if possible, but the important thing is that they are well done and informative.
ReplyDelete@ 8:58
ReplyDeleteSorry about the hostility - it's been a paperwork-bound week topped off by 2 conference calls and a faculty meeting, sending in grant reports seems like the least of the chores we face, and I don't begrudge people like Coburn timely accounting of why our work is important and useful.
Plus, I doubt our reports read more than occasionally. I don't recall that any panel I've been on, NSF, USGS, and several others, mentioned the contents of a prior final report, only if the reports never came in or had to be forcibly extracted. Maybe it's different in other fields.
I have a report due, but have been delaying because I want to submit at least one more paper first.
ReplyDeleteI was under the impression that the final reports were never read by anyone, but were purely a bureaucratic hassle created just to irritate people, because getting money from the government is supposed to be an unpleasant experience.
ReplyDeleteI have had annual reports rejected, so now I stress out about them. If I turn them in on time but there isn't enough information (including conference presentations, although these are not a big deal in my field), they may be rejected. If I turn them in late but there is more to report, they are accepted. So I turn them in late if necessary to report more contributions.
ReplyDelete"I have no real data, but I know I am against it"
ReplyDeleteYeah....that really about summed up FSP's argument about why the liberal establishment that runs academia need not justify itself.
Now I am not an intelligent liberal like most of you folks, but I know that wherever there is a lack of accountability of the rulers, terrible abuses follow. The same goes for the academic ruling class. Unlike FSP, I have the data to back up the claim that a ruling group that is not accountable leads to evil.
Since there were a lot of words in my post, and I know that can be confusing to some, let me summarize the main points so that those who think I was justifying why the liberal establishment that runs academia need not justify itself can see that they are reading a bit too much into the post:
ReplyDelete1 - It is not surprising that so many reports are late.
2 - It is misleading to combine late + missing reports and focus on that number as the relevant one.
3 - The reports are not the actual product of the research; the actual products are the papers, the students trained, the outreach activities etc.
I hope that helps clarify the key points I was trying to make.
Yet again, a hatred of dogs is also very obvious from this post, which is clearly a subtle (but not subtle enough) attempt by the feline establishment to justify its dominance.
ReplyDeletefor FSP@6pm
ReplyDeleteWhile there were a lot of words in the post, I thought I saw these predicating point #3: The reports are administratively important for NSF, leading to the thoughts in the post and comments that more comprehensive late reports are better than on-time reports, despite Coburn's angst and program director's extra effort to collect the reports.
My view is that the timeliness is more helpful than the completeness - reports are basically a box that is checked off and never looked at again. None of the 50 or so of the sketchy to skeletal reports I've submitted raised even a later mention.
The corollary is our time is better spent on papers and the next proposal that dallying to polish up a complete report.
You clearly disagree, and I may easily be wrong, but to impugn my reading comprehension seems harsh.
John V,
ReplyDeleteMy "harsh" comments were for Anon 6:07, so unless that was you, I did not criticize your reading comprehension or your interpretation of the post.
In any case, I do all my reports and I take them seriously. I just turn some of them in a bit late if I think I can add to their content.
Since there were a lot of words in my post, and I know that can be confusing to some...
ReplyDeleteHehe -- I am personally thrilled to see FSP's fangs and claws come out on occasion.
FSP - Oops, apparently it is hard for me to follow multiple strands of a conversation. Keep up the good work.
ReplyDeleteYes, FSP, as a scientist, I have learned not to trust a load of words. The best ideas are usually succint. Nothing you say justifies slackers who cannot be bothered to submit just an annual report on time and plead that the dog ate their homework.
ReplyDelete@GMP
ReplyDelete"Hehe -- I am personally thrilled to see FSP's fangs and claws come out on occasion."
Honestly, I thought FSP was being genuinely apologetic about her long winded defence of slacking. What line of science are you in GMP, where verbosity is encouraged?