Helpful gatekeepers: positive management of the limited submission process.

Porter, Robert

Introduction

With sponsor budgets flattening as universities ramp up their research agendas, intensified competition has become the norm. In 2003 NSF underwent a near budget freeze, while the number of proposals increased 14 per cent, thus lowering their average success rate from 25 to 20 percent in one year (NSF 2004). Similarly, universities are witnessing an increasing number of limited submission grant programs with more internal candidates competing for each opportunity. In January of 2005, for example, the research office at Virginia Tech posted twelve programs on its limited submission calendar for the month, one of which had eleven research teams wing for a single slot! In this environment, research administration is under heightened pressure to manage limited submissions in a manner that is perceived as fair by all constituencies. The following are ten rules for implementing a positive management philosophy in this sensitive arena.

Ten rules for positive management

As a grounding principle, the entire limited submission process should mirror, as closely as possible, the best qualities of the peer review system now in place with most major sponsors, a system that continues to get high marks from most researchers (NIH 2001).

Rule I: Cast a broad net

Limited submissions have always presented management challenges, the first being the difficulty in flagging them accurately and in a timely fashion. Recurring programs such as NSF's Major Research Instrumentation (MRI) or American Honda's Grants in Scientific Education present few difficulties. But, like wayward meteorites, new limited submissions can swoop into view with precious little warning. Online database services such as Community of Science and InfoED can be programmed to issue alerts, but their performance with new programs can be spotty. Likewise, researchers who become aware of a new limited submission may or may not choose bring it to the attention of the research office until they're ready to deliver the final proposal. (Why invite competition?) To cast as broad a net as possible, grants specialists and all pre-award staff" should report any new program to a single coordinator who is responsible for immediate communications to researchers.

Rule 2: Communicate in multiple channels

Researchers who belatedly find themselves excluded from the limited submission process often complain they weren't aware of it. To fight this, recall an old rule of organizational communications: Send important information through at least three channels. For limited submissions, the big three are: a) web site calendars with internal deadlines going forward several months, b) individual e-mail alerts to researchers, department heads and deans; and 3) periodic postings in printed newsletters.

Rule 3: Set workable deadlines

Maintaining workable deadlines while trying to balance the conflicting needs of researchers, sponsors, the university and the research office can often seem like mission impossible. Stay focused on the primary goal: To assure the selected investigator(s) has sufficient time to prepare a high quality proposal, a task that requires at least five weeks after a project has been selected for submission. Working backward from the sponsor's deadline, the initial program posting should go out about 12 weeks ahead. Internal notices of intent should be required nine weeks before the sponsor deadline, with preproposals (if necessary) due about two weeks after that. Of course, all the timelines suggested below must be adjusted to fit the academic calendar, as well as the sponsors' choices in setting dates for the initial program announcement and the submission deadline. Finally, granting exceptions to any of these deadlines is a dangerous practice and will almost always be regretted.

Rule 4: Provide a concise policy statement to investigators

Even with the best of communications, some researchers will remain unaware of the institution's need to systematically manage the limited submission process. A concise policy and procedure statement, posted on the web site and repeated periodically through other communication channels, should reduce the number of uninformed. The statement should include the key steps to be followed, as well as the respective responsibilities of the PI, research administration, and leaders of the academic units involved.

Rule 5: Require notices of intent and structured preproposals

As the sole purpose of written notices of intent is to determine whether an internal competition will be necessary, the notices need only include a brief statement of the project title, a sentence or two about its scope, and the names of investigators. Should the notices of intent exceed the submission limit, an internal competition is called for, and the next communication is critical. Each PI should receive a list of all notices that have been received, together with explicit instructions for preparing and submitting their preproposal. Full disclosure to all of the investigators involved and their project titles can have beneficial results, as it signals the candidates of the level of internal competition, and on occasion can trigger collaborations and/or early withdrawals.

Requirements for preproposals should be clearly spelled out, and they should be tailored to the needs of each specific grant program. Preproposals are just that: concise project summaries that give reviewers enough details to judge their relative merits. A maximum of three pages is sufficient for most programs, and PI's should be instructed to reference the program announcement in two important ways: (a) show specifically how the project will meet the sponsor's pro gram goals and objectives; and (b) use the sponsor's instructions for proposal preparation to outline the preproposal. This forces investigators to scrutinize the sponsor's requirements in greater detail and assures a solid head start for the PI who is selected to write the full proposal. PI's should be reminded at the outset that the selection committee consists of their own hard working colleagues who do not appreciate small fonts, crowded margins, lack of subject headings or documents that exceed page limits. In other words, when writing preproposals (or full proposals for that matter) more is not more.

Rule 6: Choose selection committees carefully

Selection committees are the foundation of the whole process. Functioning as internal proposal review panels, each group must include the appropriate range of scholarly expertise, and their deliberations must result in the best possible outcomes for the university, i. e., they must consistently select those proposals with the best chances for success. Any evidence of bias or inability to properly evaluate the merit of proposed projects would seriously undermine the integrity of the entire enterprise. Deans and department heads should take the lead in putting the panel together; they are the best judges of who should or should not serve on any given committee, and they have a vested interest in maintaining quality and fairness over the long run. Selecting the right panel members can be challenging even in large universities, as Faculty with the appropriate expertise often have conflicts of interest. Moreover, as sponsors increasingly stress interdisciplinarity, some proposals can have a breadth of scope that stretches beyond the abilities of a small group of reviewers to evaluate fairly.

Rule 7: Nurture the selection committee

The research administrator serves in a classic staffing role to the selection committee. This means taking every opportunity to inform, assist, and simplify lift for every member. Among the helping steps that make this assignment less onerous to committee members are: (a) sending an initial note thanking them for offering to serve and instructing them as to the committee's working procedures and probable calendar; (b) delivering a package with hard copy of all preproposals together with the program announcement to their offices (saves them time from downloading, printing and compiling the correct stack of documents); and (c) assuring that their meeting room is as convenient and comfortable as possible. Amenities such as morning coffee or box lunches are always wise investments.

Rule 8: Be a catalyst, not a participant in the selection process

During the committee meeting, the research administrator should be a facilitator, not a voter. This means guiding the discussion in a nondirective, yet structured manner aimed at achieving a consensus ranking of the preproposals. Always start by reviewing the essential features of each grant program, with special emphasis on program goals and review criteria. Resist any group's tendency to move too quickly to a voting mode which can be done by encouraging general discussion of each preproposal beforehand. Here is a sequence of facilitator prompts that can help move the committee toward consensus:

Round one: General discussion of each proposal. Facilitator prompt:

Our work today really has two goals. Of course we want to agree on which proposal(s) should be submitted based on their likelihood for success, but we also have an obligation to provide feedback to all the PI's who submitted preproposals. There's a great opportunity here to help improve their future proposals, whether they were selected or not. So let's list the overall strengths and weaknesses we see in each preproposal, plus our recommendations for improvement, before we start narrowing down.

During the discussion that follows, the facilitator should take notes on a flipchart or greaseboard to make sure key points are recorded and are clearly visible to the committee. Round two" Pick the extremes. Facilitator's prompt:

Based on our discussion, and looking at the notes, do we see any preproposals that stand out, either as being quite strong, or conversely, quite weak? Let's try to justify our choices based on the points we've already discussed.

This instruction goes a long way in building consensus, and the ensuing discussion rarely ends with more than two closely ranked preproposals still open for discussion as to which one should go forward.

Round three: Review and test for consensus. (In this example, two submissions are allowed.) Facilitator's prompt:

Let's review our choices. First we eliminated (name proposals and reasons for elimination). Then we decided the strongest proposal was (name proposal and strong points). That will be the first submission. For the second submission, we had some difficulty choosing between (name proposals), but we finally decided that (PI's name) proposal was stronger because (reason). So that will be the second submission. Are we all still agreed on these choices?"

Such a summary may seem like needless repetition of the obvious, but it has a powerful effect on the group, as it reaffirms the rationale for their choices and cements their ownership of same. Prior to adjourning, be sure to collect any written comments committee members made for individual proposals, as these will help in writing summary notes to the PI's. Finally, a warm thank you note is in order.

Rule 9: Provide written feedback to investigators

The most important step in positive management of limited submissions, and the one most easily overlooked, is providing feedback to PI's. Whether their proposal was selected or not, PI's invariably benefit from constructive feedback, and the research administrator has an excellent opportunity to be seen as a helper and a coach and not just a traffic cop who signals stop or go.

Setting up meetings with selected PI's is relatively easy, as they are usually on the lookout for tips to improve their chances. Getting an audience with rejected PI's is often more difficult. Some will want to argue the committee's decision, some will ignore the invitation for a meeting, and some will ask that you simply send the notes by e-mail. Whenever possible, try to set up a face to face meeting, as this has far more impact and provides an excellent opportunity to discuss other possible funding sources. Always provide the PI with neatly written notes, as this lends weight to the discussion that cannot be achieved by an informal chat alone (see sample notes, Appendix A).

Do a funding search beforehand and take along a sheaf of grant summaries--there is a strong possibility that among them are one or two potential sponsors the researcher was not aware of, and this can change the tone of the whole session. Finally, you can draw upon your broader knowledge of faculty expertise and award history to suggest possible collaborators or mentors, and this is too is greatly appreciated, especially by younger faculty.

Rule 10: Be prepared to swing into contingency mode

Expect snags to crop up. Consider this scenario: a new limited submission program has slipped under the radar screen and never appears on the research office's posted list. Shortly before the sponsor's deadline, a PI appears in your pre-award office with a sketchy draft proposal. Being first in line and with the deadline approaching, she's given the go ahead. The next day, a more polished proposal comes in, with a second PI anxious to submit. What to do? In this situation, the best course is to seek shared decision making. Convene a quick meeting of the principals (PI's plus department heads or deans), and start the discussion by (a) acknowledging the lapse in communications, and (b) reminding the group that the ultimate purpose of the limited submission policy is to assure the best proposal goes forward while being fair as possible to all participants. Then ask them which proposal they think should be submitted (the documents should be distributed to all before the meeting). Given that both PI's were lax in communicating their intent, the tentative approval given the first PI becomes moot, and you have at least a reasonable chance that the group will agree to send the better proposal. Many variations to this scenario exist, but the point is to act quickly and to take responsibility immediately for any shortcoming(s) on the part of the research office.

Summary

The expanding gatekeeper role forced by more limited submission programs provides rich opportunities for research administration to be seen, not as a reluctant enforcer, but as a conscientious supporter of the university's--and the faculty's--best interests. To do this effectively, a positive management philosophy must be articulated, backed by systematic procedures that assure fairness and consistent benefit to the principal stakeholders. Above all, constructive feedback to all PI's can turn a difficult process into a powerful tool for faculty development.

Author's Note: Contact Robert Porter, Ph.D., Program Development Manager; Research Division, Virginia Tech, 340 Burruss Hall, Blacksburg VA 24061. Ph: (540) 231-6747. E mail: reporter@vt.edu.

Appendix A Sample Feedback Notes to PI

Limited Submission Program: NSF, Major Research Instrumentation Selection Committee Meeting: December 8, 2004 Project Title: "Acquisition of Advanced Mass Spectroscopy Instrumentation to Support Bio/Nanotechnology Laboratory"

Overall Strengths of the Preproposal:

* A well conceived and well written document; the logic is easy to follow

* Proposed project supports the university's strategic plan and research priorities

* Co-Pl's are well qualified, with a strong research record

* Instruments requested could be used to support a variety of interdisciplinary projects (though only 2 researchers are mentioned in proposal)

* Some components of the proposed Bio/nanotech laboratory are already in place

Areas needing improvement:

(Note: Some of the following comments may be due to the abbreviated nature of the preproposal as reviewed by the Committee)

How other users would be able to access the equipment is not clear. This could cost points in an NSF review, as a multidisciplinary need is not demonstrated. More Co-Pl's and their lines of research should be mentioned

The "broader impacts" statement is not convincing. Similarly, the education/outreach components are not well developed. These criteria are becoming increasingly important as competition heats up at NSF

* In general, the narrative is well written, but the budget appears to be a hasty, last minute effort

* The small font and narrow margins make the document hard for reviewers to read. Enlarge the font or use a two column format

Committee recommendation:

Proposal not approved for submission to NSF. There are strong qualities in this proposal, as it ranked fourth out of the nine submitted, coming just behind the three that were approved. But the weaknesses cited above lowered the committee's overall score. Given the importance of this research to the university's current priorities, and the existing infrastructure to support the requested equipment, the Pl's are encouraged to pursue funding. If future proposals to NSF are anticipated, they should seek assistance with the education/outreach components. (Note: A workshop on this subject is being offered by the research office next month.)

References

National Institutes of Health, Center for Scientific Review. (2001). Study section member satisfaction survey final report: Executive summary. Retrieved 15 February 2005 from http://www.csr.nih. gov/events/ExecSumm.pdf

National Science Foundation. (2004). NSB-04-43: Report to the National Science Board on the National Science Foundation's merit review, process, Fiscal Year 2003. Retrieved 15 February 2005 from http://www.nsf.gov/nsb/documents/2004/MReport_2003_final.pdf

Robert Porter

Virginia Tech

Table I--A Typical Limited Submission Calendar

Limited Internal Selection Preproposals
Submission Notices of Committee Due
Announced Intent Due Identified Selected

12 weeks 9 weeks 8 weeks 7 weeks
ahead ahead ahead ahead

Limited Winning
Submission Preproposal(s) Sponsor
Announced Deadline

12 weeks 5 weeks
ahead ahead