To Accept or Reject?
"What do you think I should do about this paper?" you ask your colleague. "There are some obvious problems that are fixable, but it is written so poorly, it's going to take me 3 days to figure out how to tell them what they need to do." Your colleague with the curly red hair leans back in his chair. "You think you've got problems?" he says. "I have a set of grants that I need to read for study section and it's taken me all day just to get through one guy's preliminary data. Let's go get a beer!"
Reviewing papers and grants is a time-consuming and sometimes frustrating task. What is expected? How do you decide whether to accept or reject a paper or to recommend funding of a grant proposal? Why should you even volunteer to take on such a difficult task when there are papers and grant proposals of your own to be written?
Good questions. We've got answers in this article, part one of a two-part series. This part addresses the "why" question, provides strategies to help you make those crucial decisions on acceptance/rejection for the manuscripts you are asked to review, and offers tips on improving your efficiency at reviewing. A subsequent article will address reviewing grants.
Why participate in the review process?
Peer review is the dynamic process that allows science to progress with the appropriate checks and balances. In order to support great science, we need a system of objective review by scientific experts. The alternative to the peer review system would be a closed network of self-referent researchers whose work would not serve the long-term advancement of science.
Moreover, there are direct and indirect benefits to scientists who participate as peer reviewers. First, reviewing manuscripts will keep you up to date on developments in your field. Manuscript reviewers have the opportunity to read about the work in their field months before the work is published. Second, reviewers have the opportunity to support and advance the work of their field in peer-reviewed journals. Finally, the fact that you were asked and participated in providing "expert" review on a manuscript is an acknowledgement that you have established a national/international reputation in your field. Your career advancement, whether you are in academic science, government or in private business, depends on such recognition.
How to review manuscripts
All journal editors want to publish the best and most timely work--the "hottest" work--that falls within the scope of their journals. It's also important that the work they publish be well performed, controlled, and that the conclusions are supported by the data. Reviewers are also asked to comment on whether a work fits the scope or is appropriate for the journal in question.
To provide such a recommendation, you must read the manuscript--obviously--and closely examine the data it presents. But when you are acting as a reviewer, you aren't just reading a paper to absorb the information as you might in reading an already-published journal article. You are making judgments about rigor and quality that scientists reading published articles rely on.
Almost all journals have guidelines for reviewers either on their website or accompanying the manuscript. If you are a first time reviewer for the journal, you should read the guide to see what the editors want you to focus on.
Typically, a reviewer is requested to respond to the following questions:
Is the work novel?
Is this an important contribution to the field?
Does the work invoke a new paradigm or is it simply an incremental advance?
Do the experiments answer the questions being addressed or test the hypotheses proposed?
Do the data and analysis justify the conclusions drawn?
Are the appropriate controls in place to justify the interpretations offered?
Is the work within the scope of the journal or better suited for another journal?
Is the work appropriately referenced?
Is the presentation brief and clear?
Can the manuscript be shortened?
Are alternative interpretations considered?
Now that you have read the paper and answered these questions you will need to decide if you think that the paper is 1) acceptable with no changes; 2) should be returned for revisions; or 3) should be rejected outright. To accept "as is," the experimental design needs to be sound and the data need to be clean and accurate enough to support the conclusions drawn. However, your job is not done here. Higher-ranked journals want you, the expert, to tell them how important and novel the work is, how much it contributes to advancing the field, and whether the conclusions shift the current status--the "dogma"--of the field. "Top of the heap" journals (see "Publishing at the Top of the Heap," July 11, 2003) like Science want their published papers to be at the very top of the field but also to be of general interest to all their subscribers. Society-level journals want many of the above characteristics in their papers too, but it may not be necessary for the work to shift the current paradigms.
Assuming the work falls within the journal's scope, manuscripts should be returned for revisions if the work is of importance and a limited number of additional experiments or controls can be added to tighten up the work or provide a truly novel spin. Difficulties arise when some of the work is solid and some of it is not. In such a case, the referee needs to determine whether the work supports the broad conclusions the authors draw. If some of the conclusions are not fully supported, you may suggest an approach/experiment to strengthen the conclusions. Other times, the work may be very good but not novel -- others may have already published much of it, or its equivalent.
In the latter case, because important work needs to be validated and researchers get scooped all the time, the reviewer needs to consider two things in making a decision: 1) how much time has passed since that other paper presenting (some of) the same results; and 2) how much new information is added in the "we-did-it-too" manuscript you are reviewing. If the time lag is relatively short (months, not years) and there is new information that expands the observations, then you should consider giving the authors the benefit of the doubt. If the time line moves to years and there is not a lot of new information, then it may be time to reject.
An outright rejection would also be warranted if the conclusions are not supported by the data. This could be because the work was not performed well and the data are not convincing, or the controls that ensure that the system is working are missing. Other causes for rejection could be if the work provides only an incremental advance and is therefore not appropriate to the Big-Time journal where it was submitted. Manuscripts that are purely descriptive, do not address a scientific question or hypothesis, or just describe a new reagent or technique may be more appropriate for a sub-specialty journal than a mainstream society-level journal. Such works have no place in Top-of-the-Heap journals.
In general, there are two types of evaluations: a numerical/qualifier evaluation, where you are asked to check a box or choose a descriptor rating the paper in a variety of categories; and a written critique. The descriptors are very important and are used by the editors to determine the overall ranking of a manuscript. For example, as a reviewer you may be asked to rank the "originality" of a manuscript on a scale of 1-5 (1 being the best). If you choose 2, you have put the manuscript between the 21st and 40th percentile. Not too bad, but for a highly ranked journal, such a score would probably end up with a rejection.
Comments to the Editor: Most journals have a place for the referees to provide "top-secret" comments to the editors. Here, you can state how you really "feel" about the manuscript. If you are on the fence on whether to recommend accept/revision/reject, briefly tell the editor why. Championing a manuscript for acceptance, especially without revision or in a top-of-the-heap-journal, is not as easy as simply writing: "This is good stuff and should be published in this journal." You will need to briefly state why it's good stuff in your comments to the editor. "Whys" could include the novelty of the work, its importance to science, or its likelihood of shifting a paradigm.
Comments to the Authors: Now that you have decided what you are going to recommend, it is time to write an informative critique. Do not spend your time correcting spelling errors, misplaced commas, or grammar, because most journals pay people to do this once the paper has been accepted. Instead, focus on what is novel about the manuscript, the degree to which the paper's conclusions are supported by the data, and which parts you believe are not supported. If additional experiments are needed to convince you that the conclusions are correct, briefly outline the experiments. If the authors left some critical information out, mislabeled a graph/figure, did not use appropriate statistics to analyze their data, or forgot a key reference, then point this out. Be specific, so that the authors can revise their paper accordingly. Even if it doesn't come back to the same journal (as it probably won't if you or another reviewer recommend rejection), you might be asked to review it for another journal, and you will want to see that they made an attempt to improve the quality of the work. If you like the science but the manuscript is poorly written, tell the authors that they need to improve the clarity of the writing.
Sometimes in our eagerness to be critical, we forget about the people who will read the critique. Avoid emotionally charged or gratuitous criticisms. Remarks like "The data presented in figure 8 are meaningless," can be replaced with "The quality of the experimental data in figure 8 needs to be improved before the interpretation of the authors can be accepted."
If you have recommended that the manuscript be returned for revisions, you are likely to see it again. Usually, reviewers expect the authors of a revised manuscript to address the concerns raised in the critique. Written critiques for revised manuscripts are typically very short. For papers in which the authors have thoroughly addressed the previous concerns of the reviewer, the written critique can say just that. The paper is now ready for acceptance. That was easy!
Just because the authors were allowed to resubmit a revised manuscript doesn't mean that the paper has to be accepted on this round or any subsequent round. The manuscript must be sound and fulfill the journal's requirements as stated in their guidelines. Thus, if you really want to see the inclusion of another control in the experimental design, or if you still don't believe the conclusions based on the data, you will need to say so in your comments to the editor and the authors. Occasionally, new issues emerge due to either the inclusion of new data in the revision or because you or another reviewer missed a critical point. This being said, it is not fair to the authors for you to come up with a long list of new concerns that were not raised in your previous reviews.
How much time should you allocate to manuscript review?
Reading and writing a critique of a manuscript takes several hours. If you have to do some literature searches or pull some additional references in order to fully understand the work, this will increase the time you have to spend. Journals give you between 7 and 21 days to review a manuscript and submit your review. You don't want to get a reputation for accepting the assignment and then not submitting your review in a timely manner. So, if you know you can't possibly get the review done in time, tell the journal when you think you can do it, or decline the invitation.
You also don't want to provide a mediocre, unhelpful, or biased review. The editors in some journals rank the reviews themselves by the quality of their scholarship. This way, they know who the best reviewers are in a field. Some journals also use this data to consider whether someone may be appropriate to add to the editorial board. While such positions usually come as you mature in your career, editorial board positions are a sign of national recognition that will enhance your portfolio and benefit your development as a scientist.
The manuscript-review process is the cornerstone to scientific inquiry and results reporting. As a citizen of the scientific community, it is important that you participate in this process and that you take your responsibility seriously. By focusing on the questions that the editors want to know the answers to, you can save yourself some time. Leave proofreading to the proofreaders and focus on the science. Your reviewing mantra should be "Be fair, clear, and precise," but don't forget to champion manuscripts that you really like.
Good luck, and don't stop the presses!
Jeremy M. Boss, Ph.D.
Professor of Microbiology and Immunology, Emory University, School of Medicine
Susan H. Eckert, Ph.D.
Associate Dean for Administration, Emory University, School of Nursing