"How am I ever going to get through all of these grants? There must be more than 200 pages here. I didn't think there would be this much reading, and I only have 2 weeks before I have to send my comments in!" you exclaim to your colleague with the curly red hair. "Relax," he replies, "all you have to do is spend a day on each, and you're done. Piece of cake."
Your colleague is right about the time required to review a grant application. Careful planning is crucial if you're going to complete your reviews and still have time to get your own work done. Furthermore, it's a difficult job; because each grant application is distinct in its goals, approaches, and significance, you need a method to distinguish among the grants and rank the "apples and the oranges."
In this article, we will discuss:
The time involved in reviewing an application; The characteristics of grant applications that allow you to rank them in relation to one another; What is expected in the written review; and How to prepare for a grant-review session.
The time involved in reviewing an application;
The characteristics of grant applications that allow you to rank them in relation to one another;
What is expected in the written review; and
How to prepare for a grant-review session.
At the outset, it is important to consider why you should volunteer your time to serve as a grant reviewer. First, it's for the good of science. The complex system of public and private funding for scientific research relies on expert peer reviewers--people like you--to evaluate grant proposals and direct money to the very best and most important scientific and medical projects. The other reasons are less altruistic. Reviewing grants is a terrific way to gain grantsmanship skills. Grant reviewers learn from the mistakes they observe in the grant proposals they review, as well as from the discussion of the shortcomings--and successes--of the proposals during the peer-review meetings. Serving as a reviewer will also provide you with a crash course on each scientific topic proposed, a benefit to seasoned investigators who serve as reviewers. Lastly, you will get to experience what really happens at a grant-review panel meeting. This experience will be invaluable to you as you prepare your own proposals.
Reviewing Grant Applications
Now that you are convinced that you should participate in grant reviewing, you need to decide if you have the time. The first thing to determine is whether you are being asked to serve as an ad hoc member for just one set of grants at one peer-review meeting, or whether you are being invited to become a standing member on a review team. As a standing member, you will be expected to participate in the process several times a year. Over the course of your career, this could cost you months of time. If you are a new faculty member, this may not be the best use of your time when you are trying to get your lab going.
On the other hand, being an ad hoc reviewer for a national agency or even for an institutional funding program can be a good opportunity for a new investigator to gain experience in an amount of time that most young scientists can spare while establishing their research programs.
How much time does it really take to review an application?
Depending on the funding agency, you may be recruited to review more than one grant at a time. Sometimes reviews require only a written critique; other times you may be asked to participate in a conference call, a site visit, or a panel-review meeting. Additionally, the page length of the application and what the funding agency wants you to critique differ among the types of grants.
You need to know all these particulars to estimate how much time you will need to commit to the reviews, but here are some rough guidelines. A seasoned reviewer needs a minimum of 2.5 to 3 hours of uninterrupted time to read a 25-page, single-spaced NIH application. If the application requires that you do literature searches and find additional background material, the time increases dramatically. Some grants take all day to read. Many people read a grant on one day and write the critique the next morning. It typically takes about 2 hours to write the critique. You should therefore expect to spend at least one full workday on each grant application you have been assigned.
Typically, reviewers receive their applications 4 to 6 weeks before the review or meeting date. As soon as you get your proposals, look at them. Waiting until the 11th hour to begin the process is a bad idea. If you procrastinate, you will end up miserable, so give yourself plenty of time. That's why they sent them to you a month ahead of time.
Hey, I know that guy!
Avoiding conflicts of interest with grant applications is important. All funding agencies have rules and guidelines specifying what constitutes a conflict of interest. These include co-authorships, training relationships, and financial relationships, to name a few. If you have a conflict with any of the investigators or applications, be sure to let the agency know right away. Scientific conflicts with competitors must be evaluated on a case-by-case basis. If you feel that reading a competitor's application will lead you to change your scientific goals, then you should decline; otherwise, you should review the application, which may be the reason that your expertise was sought in the first place. We all review our competitors' applications and papers. Establishing the ethical line and treatment of this confidential information is part of the career-development process.
Whoa . . . it's not my field!!!
All reviewers occasionally get stuck having to review an application that is out of their fields of expertise. If you feel that you cannot possibly review a grant, contact the program officer who assigned it to you as soon as possible. Here's another reason to look at your assignments as soon as they arrive. It may help your cause if you see a grant in the group that you weren't assigned that you could trade for. If you can't get out of it, set aside an additional day and be prepared to read some long review articles.
What Should You Look for in an Application?
Along with your stack of applications, each funding agency will provide criteria by which the grants should be judged and ranked. For research grants, many of the criteria are the same. Ranked in importance, research grant applications typically have seven important parameters: 1) significance, 2) novelty, 3) experimental design, 4) preliminary results/proof of principle, 5) progress (if the application is a renewal), 6) the presentation, and 7) the qualifications of the applicant.
Judging the strengths and weaknesses of each of these parameters allows you to rank each of the grant applications you are reviewing relative to the others. It is important to note that we use the word "rank" rather than "fund"; reviewers do not fund grants, they are just asked to rank them. Funding agencies may provide you with a guess at their paylines: the cutoff between proposals that are funded and those that aren't. You're there to sort the grants by priority so that they can fund the best grants first and then work their way down the rank-ordered list.
Using these seven parameters, a "great" grant application has the following characteristics: It addresses a very important problem; introduces novel concepts and tools; has clear hypotheses and uses a clear, straightforward experimental approach; provides clean, useful preliminary data that prove each principle being tested; is written clearly; and is from a person fully qualified to do the work, working at an institution that has the resources to fulfill the requirements of the project. That was easy, right?
So much for the ideal grant; what about those that are not perfect?
The first point to consider is the importance of the problem being addressed. If the problem is at the top of your list of important questions and issues that this field of science should address, then it makes sense to cut the principal investigator (PI)/applicant some slack if the proposal has a few nonfatal shortcomings. If you think the problem is pedestrian, then there needs to be something else in the proposal to increase your enthusiasm and keep it off the bottom of your rank-ordered list. Although this sounds easy, most applications fall in the middle of the "importance" scale, requiring often difficult apples-and-oranges comparisons. The other parameters gain in importance as the significance wanes.
Proposals that introduce and test truly novel concepts should be greeted with enthusiasm -- if the applicant justifies the new concept and proposes a way of testing his or her new ideas. Because we tend to be rooted in the dogma of our fields, outside-the-box proposals often do not receive the high rankings they deserve. The key to deciding whether novelty is truly a virtue is preliminary data.
Depending on the type of grant, the quality -- and sometimes the quantity -- of preliminary data are crucial to your evaluation. Any novel tool or reagent should be well characterized, with adequate controls. If a grant aims to test a new principle, then a proof-of-principle experiment must be provided. The work doesn't have to be in final publication form -- that's why they call it preliminary data -- but it should be clear to you that the system is working.
Proof of principle is particularly important when some of the aims are dependent on each other for success. For example, if Aim 1 seeks to create and characterize a new tool, which will then be used in Aims 2 and 3, the ability to create the tool and use it for the intended purposes must be demonstrated. Preliminary data can also serve other essential purposes. If an investigator is using a technique for the first time, preliminary data can be used to demonstrate his or her mastery. If the approach proposed is questionable, preliminary data can demonstrate that the approach is sound, or at least promising.
Preliminary data provide proof that the investigator's ideas are justifiable and doable. Lack of key data could legitimately diminish your enthusiasm for an aim of the proposal.
Competing Renewal Applications
Competing renewal applications are a special category. These applications represent a continuation of a research project that may have very broad and long-ranging goals. Because the investigator has had time to perform all or part of a previously funded set of aims, progress on the previous application needs to be evaluated. In the ideal case, the investigator has completed all the aims of the previous application and published the results in, say, a dozen papers over the last 5 years, with several of those appearing in top-of-the-heap journals like Science, Nature, or Cell (see our Next Wave article, " Publishing at the Top of the Heap"). This ideal case would be considered outstanding progress.
In contrast, a worst-case scenario would be that the aims were not completed and no papers were published, or, if there were published papers from the work, they appeared in subspecialty-level journals and had little impact on the initial goals of the previous application. Obviously, this is poor productivity, and your confidence in the ability of the investigator to do the work proposed should be diminished.
What about the 80% of renewal applications that are neither exemplary nor obviously deficient? What is considered solid productivity? As a rule of thumb, one or two papers a year in society-level journals--or better--is considered solid productivity. You must then ask yourself, what if the investigator has fewer papers but has one in a top-of-the-heap journal and covers one or two aims of the grant? This is where journal ranking and perception of scientific reputation come into play. Papers in higher impact journals are presumed to undergo greater scrutiny, and the work is assumed to represent a major advance in a field. So, extra credit can be given to the investigator for articles in top-of-the-heap journals. Each application will be different, and you will have to judge whether the investigators have done the best job possible with the project they proposed previously.
Experimental Design and Approach
The experimental approach should make sense and stay focused. Although it is often useful for this section to be organized by hypotheses, this style isn't required; some PIs like to pose questions instead. Regardless of the style, the investigator should make it clear what she is going to do and convince you that she has the right controls in place to interpret the data. The investigator should also say what she will do if things don't come out the way she plans. If you can't follow the logic of an experiment, or if the obvious alternatives are lacking, then it is reasonable for your enthusiasm for this aim or subaim to diminish.
Many research plans are designed to test more than one idea and hypothesis. You should judge the significance/rationale, approach, interpretations, and alternatives for each aim separately. You may also have to consider whether the completion of one aim depends on the completion of another. (This is why a "fishing trip" for a new factor or gene at the beginning of a project typically leads to a poor ranking; if the factor isn't identified, the grant can't be completed successfully.) In the end, it is important to determine whether the aims of the proposal are synergistic or antagonistic.
Qualifications of the Applicant
Is the applicant qualified to do the work? First-time applicants, junior faculty, and senior investigators are evaluated differently. In all cases, the publication record of the PI is the key to making this determination. For first-time investigators, the following questions are important: Do they have first-author papers? Many, or only a few? Are their papers in the same field as the field of the application? Were the technologies used in the new proposal also used in earlier papers?
First-time applicants' postdoctoral work is used to evaluate their probability of being able to perform the proposed work. The degree of independence of first-time applicants is also considered. Is he true faculty or a senior postdoc in a senior investigator's lab? Although your enthusiasm for the latter applicants should not be dismissed, you need to be convinced that they will function independently of their bosses and begin to establish their careers. Their resources--allocated space and access to necessary equipment--should be defined clearly.
For scientists who have been independent for 2 to 5 years, the ability to publish on their own is a key criterion. Even if the publication numbers are low, a faculty member who has been in a position for 3 years or so should have a record of publication without his former mentor, and he should be first or last author on some peer-reviewed papers. If the junior faculty member already has a grant, has he published on the topic? This is key to determining whether he can function independently, and whether he can handle more than one research program.
For senior investigators, qualifications may be an issue if the application is in a new field, or if the investigator has not been productive for several years.
Although the productivity issue is difficult for everyone to overcome without new publications, gaining expertise in a new field is not. The simplest way to build credibility for a proposal in a new area is for the applicant to enlist the help of collaborators with strong and published expertise in the new field or technique. Collaborators listed as co-investigators, who will receive part of their salaries from the grant, are best because you know that the PI will expect them to work. Issue covered! Collaborators listed with support letters need to be evaluated based on their letters, published expertise, and what they will be contributing. All collaborators should detail their intents and roles in their letters of collaboration.
Are the required resources readily available? Most of the time this is not a problem, although this can be an issue for new investigators who are still in their mentors' spaces. Regardless of the PI's position, all PIs should have their own spaces and should have access to all the necessary equipment and core facilities.
Grant applications do not have to read as if they were written by Hemingway, but they need to be clear, logical, and free of glaring science errors. Although brownie points are not officially awarded for elegance or the overall style, a sloppy application suggests to many reviewers that the investigator is also sloppy with his or her experiments. Still, don't forget that it's the science you're reviewing, not the writing.
Writing a Review
Now that you have studied the application and have developed an opinion about all of the parameters described above for each of the application's aims, it's time to write your critique. Keep in mind, as you read what follows, that at this stage you have already evaluated the proposal's strengths and weaknesses; you have already decided which proposals you like most and which ones you like least.
The style, format, and specific content of the review will depend on the funding agency, but written critiques typically serve three purposes. The first is to convince the panel that the applications you like best deserve to be funded and that the ones you like least don't. Second, the critiques provide the funding agency with documentation -- and the applicant with feedback -- as to why the grant received the ranking that it did. Third, written reviews provide you with a synopsis of the key points of the application, its highlights, and your concerns when it is your turn to present the review to your peers at the reviewers' meeting.
In preparing your review, start by outlining the major categories you are asked to comment on. Some of these categories are similar to the parameters discussed above, but each agency has its own particular approach. Avoid spending time rewriting the grant for the applicant. And, although you should provide clear guidance, avoid the temptation to describe all of the experiments you recommend in detail.
After an initial assessment of the significance of the proposed science, most of your critique will address the experimental design. If your enthusiasm for a proposal is very high, it makes sense to emphasize the proposal's strengths and not its weaknesses. Your written critique should be scholarly and measured, but the goal of the written review is to bring others -- other committee members and those who will make the ultimate funding decisions -- around to your point of view.
However, if the proposal has significant strengths but has aspects that temper your enthusiasm, you will need to indicate the strengths but concentrate on the major weaknesses that you think need to be addressed for the project to be successful. This is important because the applicant will use your written critique to address the concerns when it comes to time for resubmission.
Writing a good critique is difficult. It takes several hours to write a 2- or 3-page review.
The Reviewers' Meeting
The peer-review process is at its best when reviewers are asked to justify their ranking of an application to their colleagues on a panel of experts. The process can be intimidating for first-timers. The best way to ensure a good experience is to be prepared.
The first step in preparing is to rank your grants in order of your enthusiasm. This is essential because no one will take you seriously if you rate all the proposals you reviewed as "outstanding." Once the proposals are ranked, choose your top two grants to champion. Not all the proposals you like will succeed, so you need to make sure that the one(s) that you think are the best get your full support at the meeting.
To prepare your presentation, make a bulleted outline of your review, emphasizing the points that you want to discuss. To refresh your memory, read the abstract and your reviews the night before. This sounds more and more like preparing for qualifying exams, doesn't it?
In the presentation itself, stress the significance of the work, the novelty of the system, what the applicants hope to learn or solve, a clever experiment or two, and why you think this applicant is the best person -- or team -- to do the work. The last item is easily addressed by evidence of past productivity.
For high-ranking applications, concentrate on strengths and do not dwell on weaknesses. Review panel members will use your presentation to rank the grant being presented with the ones they themselves have reviewed. For your low-ranking applications, justify your poor rankings by explaining the problems with the application. Your verbal report should take less than 10 minutes. If you are the second or third reviewer to present and you have nothing new to add to the presentation, just say so, especially if the proposal is not one that you are championing. If it is The One you've decided to champion, make sure the other panelists are aware of your enthusiasm for the proposal, even if you have nothing of substance to add to the previous reviews. Consider repeating something if you think that it will enhance the enthusiasm of the review team for the application.
Occasionally, reviewers will be split in their opinions. This is never good news for the application. How you deal with a situation like this will depend on how strongly you feel about the application. If you have ranked it on the high side and are convinced of its great value, take a stand. Stress the work's importance and argue that the investigator will be able to address the weaknesses raised by the other reviewer. If you have ranked it on the low side and feel certain that the work cannot be done, you will need to explain why, in scientific terms.
Whichever side you're on, always listen carefully to the other reviewers' arguments and be willing to bend if they are convincing. If you don't feel strongly about an application, don't hold up the meeting. Tell the others that you ranked it somewhere in the middle, raise your score if necessary, and let the meeting move along.
Participating in the grant-review process is a thankless job but one that is valuable not only for you but for the whole scientific community. Give yourself plenty of time to review the applications and write the critiques. Be as fair as you can while remembering that you are not responsible for what is written in the application. Prepare for your presentations ahead of time and be sure to champion the best proposals. In the end, the review experience will increase your scientific expertise and raise your own grant-writing skills.
Good luck and enjoy your role in awarding the grant money!
Jeremy M. Boss, Ph.D.
Professor of Microbiology and Immunology, Emory University, School of Medicine
Susan H. Eckert, Ph.D.
Associate Dean for Administration, Emory University, School of Nursing