"You won't believe this new data my grad student presented at our lab meeting. It really is amazing; it opens a whole new area of research!" you say to your colleague with the curly red hair. As the weeks go by, a second set of experiments are also very compelling. But there is something fishy about the way the results are coming out. Everything seems nearly to add up to the conclusion you want, but not quite. While the initial experiments repeat successfully, others do not clearly support the conclusions. This goes on for 6 months until you say, "What is going on? Should I let the student continue this project or is it leading him--and my lab--into a black hole?"
Welcome to the world of the scientific red herring: a set of experiments and results that lead you down a path that ultimately offers no useful conclusion. Biological systems are by their very nature complex, and sometimes all the pieces of the puzzle are not accounted for by the experimental paradigm. Other times, an early, provocative experiment is over interpreted or inadequately controlled.
To be a successful scientist and a successful mentor, one needs to know when a system is not working out and when it is time to change direction or punt. This article will discuss how to avoid the red herring--staying with false leads too long--and when to call it quits on a line of investigation that is simply not going anywhere.
Recognizing and Avoiding Dead End Projects
How strong are the data? When you are doing the experiments yourself, you know how clean the data are and that the protocols are being followed correctly. As you begin to supervise the experiments of others, you will find that the quality of the data can vary widely. If the data are not clear, perhaps the interpretation has more hope than substance.
So what's the solution? Make sure your student's (or tech's, or postdoc's) data are of publication quality. In the ideal world, all experiments would be publication quality; this, indeed, is what we should all aspire to. The adage that we can always do it again is simply inefficient and may lead to wrong interpretations and wasted time. Adopting "Do it right the first time" as a lab motto may save you months on each project, adding up to years of savings over the course of your scientific career. That could make the difference between tenure and no tenure, and between an illustrious career and a pedestrian one.
Protocol drift? As protocols in the laboratory get passed down from one individual to another, changes of procedure may accumulate. Sometimes these changes are for the better, sometimes not. The issue of "protocol drift" is serious because as protocols change--usually because someone in the lab thinks a modification will save time--the parameter being assayed may also change. The larger the research group, the more rapid the drift. In the worst scenario, you won't know that protocol drift has occurred, and huge amounts of time and money will be wasted on wrong conclusions.
Establishing a protocol-alteration policy for your lab is a good idea. One approach: Changes to lab protocols should be proposed at lab meetings for everyone to hear and discuss, and then approved by you.
Do you have a proof-of-principle experiment? Exciting new data and paradigms are alluring, and great fun to follow up with experimentation. But sometimes we--and especially our enthusiastic but inexperienced grad students and postdocs--get caught up in "Wow, what if this is really true?" Good scientific practice with clear hypotheses--including experimental designs with solid positive and negative controls--should eliminate most errors that might send your research group down a wrong path.
But sometimes that single "proof-of-principle" experiment is missing. Such experiments let you know that your assay system, reagents, or model system is likely to give you clear interpretations of your data. Prevention point 1, therefore, is to make sure that all initial experimental systems that you pursue have a single set of experiments that prove that the assay system is accurate and faithful and that the reagents that are being used are precisely what you think they are.
Once the assay system and the reagents are accurate, point 2 is to design an experiment that produces unambiguous results and lets you know that the hypothesis can be proven. In point of fact, breaking down your experimental designs into smaller steps with yes/no outcomes will strengthen most of your experimental paradigms and allow you to interpret your data accurately. The sooner the yes/no experiment can be undertaken, the better.
When to change direction
The above discussions deal with adhering to fundamental and irrefutable scientific practices, but even when these are followed projects sometimes lead to a dead end. This can be especially true of lead-in projects that are given to incoming students and fellows, where the hope is that they will be able to develop it into a hot paper or dissertation. It is critical to a young investigator's career to know when to rethink and redirect a project, and when to stick with it. Below is a list of questions that may help you to make the decision to change direction or continue with the current plan. Your decision to change direction should be influenced to some degree by your tenure status and productivity. The more productive you have been, the easier it will be to overcome time wasted on projects that fail.
How long have you been working on this project, and how many months will it take to get to the key experiments that address your central hypothesis? The exact answer to this question depends on the quality of the data being produced and the time it takes or should take to do the necessary experiments. Another factor in the time it takes is the level of experience of the person doing the hands-on work; see the next question. If experiments to address the central hypothesis seem always to be in the distant future, then it may be time to redirect. Certainly, six months of serious effort by you or your staff should be more than sufficient time to decide if a project is worth the investment of more time. Of course, as you become more senior and have accumulated success, investing significant time in exploratory goals and out-of-the-box ideas may lead you to the next great breakthrough in your field.
What is the technical skill level of the person working on the project? Over your career, you will find that all students/fellows/techs are not equivalent in their ability to perform experiments, and that while someone will be good at performing one protocol, he may not be capable of another that someone else in the lab finds routine. This describes one of the basic difficulties in supervising students and fellows. It's not just the quality of the project and the student that matters; it's also the fit. You will have to match the student/fellow with the project to be able to achieve the highest level of productivity. If the person is relatively junior, still developing his skill set, and seems to be stymied by a critical step, you may want to assign this step to someone else in the lab or find someone to collaborate with, or outsource the step to a company.
If a senior, highly skilled person does not appear to be making progress, then perhaps it is due to the initial hypothesis or a basic problem with the power of the technology/protocol to answer the question. If this is the case, there are two choices: 1) approach the question from different angles; or 2) scrap the project. If you opt for the first choice, but still have trouble resolving the question, cut your losses and scrap it.
How risky was the initial experimental goal? Most high-risk experimental pursuits will not end up on the cover of Science; that, after all, is the very nature of risk. If you started a conversation with your new student with, "Let's see what will happen if we do," then you probably assigned that student a high-risk project. This is a fine approach. But give your student and the project a time limit and make sure that a proof-of-principle experiment is on the short list.
But the project is listed in a grant as a specific aim!
The necessity to complete a grant-specific aim when the experiments are not working can sometimes lead to a huge time investment but only a minor publication or none at all. Grant proposals are just that: proposals, documentation of proposed scientific work. Not every proposed experiment will work as predicted. If you intend to compete for a renewal of your project, you will need to demonstrate progress, and perhaps even demonstrate that the reason that you did not complete a specific aim is that you proved the hypothesis false. Many grants take this into consideration during the annual progress reports, where you can redirect your plans and pursue related questions and hypotheses.
Help from colleagues
Your colleagues are valuable resources. You can get the most out of them in two ways. First, if you have graduate students, you should make sure that your students have regular committee meetings attended by all of their faculty committee members. There is a national trend to have at least two such meetings per year. If a student's project is not progressing, ask her to schedule a meeting as soon as possible. The second way to tap into your colleagues' wisdom is to invite a few of them--colleagues with expertise in your area--to discuss your project over lunch, your treat. Approach this as a committee meeting for you. Providing your colleagues with an outline/aim handout may keep the meeting focused.
The core advice here is to follow good scientific practices. Plan proof-of-principle and yes/no experiments early in the process to be sure that the main hypothesis is correct and that your goals are attainable. Be conscious of the time (in calendar months) being spent on moving--or failing to move--the project forward. Importantly, if a project is not going well, put a time limit on when a specific goal or proof must be attained. If it is not attained at that date, change direction and move your other science forward.
While it is fun to have high-risk projects going on, most of them fail. It is therefore important to make sure that you have doable and important science always going on in your lab. Don't put all your eggs in one basket unless you're sure it can hold them. Good luck!
Jeremy M. Boss, Ph.D.
Professor of Microbiology and Immunology, Emory University School of Medicine
Susan H. Eckert, Ph.D.
Associate Dean for Administration, Emory University School of Nursing