The National Science Foundation has taken a bad procedural turn in the last few months. This post is intended to bring this change to the attention of a broader audience and maybe ignite a bit of pushback.
First, some background. Until a few months ago, a typical constraint on proposals for large, cross-cutting awards at NSF was as follows: The number of preliminary proposals per institution was capped, typically at three. Full proposals were by invitation only, with invitations issued on the basis of peer review of the preliminary proposals.
At a large institution like mine, these constraints implied a three stage selection process: (1) Interested faculty submit briefs to an internal competition. (2) Winners of the internal competition are permitted to submit preliminary proposals to NSF. (3) Those who are invited submit full proposals to NSF.
This was, in my opinion, basically a sound process. By limiting the number of preliminary proposals per institution and controlling the number of invitations issued for full proposals, NSF had fairly good control on their reviewing workload. Preliminary proposals required significantly less time and effort to prepare than full proposals, and internal proposals significantly less than that. And the amount of time required to review preliminary (and internal) proposals was also significantly reduced. So, the amount of time invested in the process was roughly proportional to the likelihood of success for the proposer, and the amount of review time per proposal was also roughly proportional to the expected investment in the project at each stage.
Furthermore, I liked the preliminary proposal process. Preliminary proposals were peer reviewed, and I always received excellent feedback on them. (Despite having not yet been invited to submit a full proposal in response to one of my preliminary proposals!)
I don’t much like the internal proposal selection process, but it is serviceable. The problem with the internal process is that there are a limited number of experts in a given area within a university, and all of the experts in a particular area tend to have significant conflicts of interest with each other. So, it is difficult to provide a high quality internal review process. But, with three slots (or so), I always felt like I had a fair chance to make my case. (A typical number of entrants in these internal competitions at my institution, at least the ones that I was a part of, was 6–12.)
Unfortunately, in the last six months or so, NSF has changed the constraints on several programs. (IGERT was one that I encountered earlier in the year with this new process, but it was not the first. PIRE is the most recent victim.) Specifically, they have eliminated the preliminary proposal phase. Instead, they have limited universities to, in the case of both IGERT and PIRE, one full proposal per program.
From talking to my sources at NSF, a driving factor for this change was complaints received from people who didn’t like the preliminary proposal process. Specifically, they felt that the time invested in the preliminary proposal was too great for them not to have the opportunity to submit a full proposal. If true, then this is a case of NSF listening to squeaky wheels. Isn’t it better to spend the moderate amount of time on a preliminary proposal and get some feedback than to spend the enormous amount of time required for a full proposal only to still end up with no money? The people who complained about this are idiots.
Now, personally, I have to believe that the NSF may also have been looking to reduce the amount of resources required for reviewing these programs. While reviewing a preliminary proposal is less work than reviewing a full proposal, it’s probably still at least ⅔ the work of a full proposal to review. So, the savings in reviewer and program officer time and effort by eliminating preliminary proposals are surely significant.
But the cure is much worse than the disease. It shifts the burden of reviewing onto the internal review process, which was already the weak link. In addition to the difficulties cited above, the internal process also has no mechanism for balancing national interests and priorities the way that NSF can, as it must now select a single winner. Thus, it is now left to a university research office to decide whether to put forward a proposal to revolutionize automobile safety or one to study the chemistry of water treatment or one to explore the mathematics of finite fields. This is not a decision that a university research office is well positioned to make, and, despite their best efforts, it probably leads to supporting flashier research at the expense of potentially fundamental advances.
So, memo to NSF: Ignore the complainers, invest in the peer review process, and bring back the preliminary proposal!