A few months ago I had the honor to be asked to be part of a National Science Foundation (NSF) review panel. I had been asked a couple of times before but for personal reasons (a small child, a very heavy teaching load) had always said "no." This year, at the advice of a good friend, I said "yes." I am very glad I did. If you have the opportunity to participate in a grant-reviewing panel, at NSF or elsewhere, I encourage you to accept. But before you do, here are a few things that I learned about becoming an NSF grant reviewer.
Get a clear picture of the scope and time frame for the reviewing process from the program officer before you say "yes." I said "yes" without really having any sense for what I would be doing or how much work would be involved in the review process. It turned out that the panel I was part of was charged with evaluating many complex proposals in a very short time period. In retrospect, I wish I had had a better sense of the time frame so that I could have done a better job clearing my plate in the week that I had to read proposals. Then again, if I had known how much work it would be I might not have accepted, and I would have eventually regretted that decision.
Get a clear picture of exactly what your role will be in the panel, and how the panel will function, before you start reviewing. I suspect that the only way to really know how it works is by doing it; the first time will always be a learning experience. But the more you can learn about the process before you start reading, the better. I started reading 13 complex proposals, some of which covered topics outside my areas of expertise, and became concerned that I would need to develop expertise in areas like theoretical chemistry in a week. ... It is obvious to me now that NSF could not reasonably expect me to have expertise in all these areas but at the time I wasn't sure; perhaps they assumed that most chemists already knew that stuff. They didn't. But having a picture of the review process would have helped me better focus my energies. This, by the way, is something that NSF could do better.
Review the program guidelines and evaluation criteria. Take them seriously. Although NSF has always judged proposals according to their "broader impacts," this criterion has recently taken on greater significance at NSF. Broader impacts can be different things--the "impact" can be directly on science, or on diversifying the scientific workforce, on science education and literacy, or on society. When you're reading over grant proposals before the panel meets it's important to do so with "broader impacts" and the other key NSF review criteria in mind.
Dress nicely. This surprised me: This being a meeting of scientists, I wasn't really expecting high fashion. But I'm glad I packed nice clothes--appropriate business attire. At the very first meeting almost all the men in the room had ties on, although they shed them later in the week. Still, overall the dress was much more businesslike than at a typical professional meeting.
Not only did I learn a lot about being a panel member, I also learned a lot about writing grant proposals. It was really very interesting and educational to read a large number of proposals from a reviewer's perspective. I'm sure that the experience will make me a better proposal writer.
The science is, of course, important, but I learned that, even if the science is strong, poorly written proposals generally don't fare well:
Clear, concise, well-written proposals that identify an interesting problem or question early and frame it in terms of the fundamental science fare better than proposals that are vague and unfocused.
Proposals that test hypotheses are stronger than proposals that appear to be "fishing expeditions," even if the fish are likely to be big. It is nice to know that the experiments being funded will yield useful results even if things don't go as planned, which is easier to be confident of when there is a carefully framed question motivating the experiments.
It is helpful to have the key criteria outlined and the overall shape of the proposal clearly explained in the first few pages of the proposal. As a reviewer it is easier to refer back to a few pages rather than having to search through 15 pages to find answers to the questions NSF asks us to answer.
If there are potential problems or controversies in a field, it is better to address them head on rather than ignore them. You might get through without an outside reviewer picking up on an unaddressed question, but if somebody notices and you haven't addressed it the reviewers are likely to assume that you haven't thought of it.
Suggest alternative approaches to key steps in the proposal so that the reviewer can be confident that if one approach doesn't work, you've got other strategies.
Developing a track record of scientific and research-training accomplishments is important.
Articulate clear plans for dissemination and outreach. This shouldn't be a mere afterthought, tacked on at the last minute. These things really do make a difference.
Call your program officer. During the meeting I got a strong sense for the humanity and professionalism of program officers. When I first began my faculty position I was a little afraid to call a program officer--fearing that they would consider it their job to note the inexperience in my voice and permanently cast me down into some sort of funding purgatory from which I would never emerge. In fact, it was clear that their goals are to fund good science and to develop scientific infrastructure. And they really like to fund new investigators.
As a scientist and a taxpayer, I was inspired by the seriousness and good will with which reviewers approach the job and the thoughtfulness of NSF and the program officers I interacted with. This small part of the taxpayer's dollar is in good hands. On the panel I took part in, people worked every evening and every morning. Reviewers had read the more important cited papers and brought copies with them to the panel. Program officers were careful to remind us to judge the science, not the investigator, and to remind us not to look for "perfect" proposals but for excellent ones, nudging us out of the scientist's natural tendency to be overly critical. On the panel that I was part of I saw no mean-spiritedness or pettiness. I saw a group of people who believed in what they were doing and worked together hard to make thoughtful choices. Reviewers showed a remarkable diversity and degree of independence; despite notable differences in rank, experience, and status, panel members all took seriously their responsibility to think for themselves and to make their opinions heard. At the same time, I was pleased to see how rarely people talked just to hear themselves or impress their colleagues. If only faculty meetings could be so pleasant, professional, and productive. ...