Reposted with permission from Science News, 20 January 2006

After more than 3 decades of trying to increase the number of minority biomedical researchers, officials at the National Institutes of Health (NIH) have a raft of anecdotal evidence that the programs are working. One favorite success story is Erich Jarvis, a neuroscientist at Duke University in Durham, North Carolina, who, as an African-American student from Harlem, was supported by some of those programs and who in 2002 was named the nation’s top young scientist. The program managers themselves exemplify NIH’s goal of diversifying the nation’s biomedical research workforce. Clifton Poodry, head of the division of Minority Opportunities in Research at the National Institute of General Medical Sciences (NIGMS), was born on the Tonawanda Seneca Indian Reservation in Buffalo, New York, for example, and spent 2 decades as a successful academic researcher before joining NIH.

But personal success stories aren’t the same as hard data. And good data, says a recent report* by a National Academies’ National Research Council (NRC) panel, don’t exist. Asked to assess the programs, the panel threw up its hands. It is devilishly difficult to track participants through their training and into the workforce to find out if they have indeed achieved the gold standard of becoming biomedical researchers, the panel concluded. NIH hasn’t invested the time, money, and high-level interest needed for a proper evaluation, it added, nor shared what data do exist.

Hard to calibrate. NIH’s “Bridges” program helps minority students transfer from community colleges to research universities, but there are few data on what happens to them later.

“There’s no good way to track the success of these programs,” says John Bailar, emeritus University of Chicago statistician and co-chair of the panel, part of NRC’s Board on Higher Education and Workforce. “We were asked to find out what works, and we couldn’t do it because of serious problems with the data.” NIH can’t even say how many participants are actually minorities, it noted, much less how well its programs are doing in churning out minority scientists.

The report’s list of flaws is long and damning. Panel members deplored the lack of coordination among the programs, which are run by one or more of NIH’s 27 institutes and centers. They questioned NIH’s definition of success—the production of Ph.D. biomedical researchers good enough to win NIH funding—given the considerable opportunities open to those with less training and the importance of raising the level of public scientific literacy. They also pointed to a lack of commitment from the top. A meeting of minority-training coordinators convened by the panel, Bailar noted, was the first time all had been together in the same room. And even the report’s most basic recommendation—that NIH convene such a group and have it draw up guidelines for a thorough evaluation—has yet to be implemented more than 6 months after NIH officials were briefed on the report.

“The root problem is that these programs have suffered from a lack of sustained high-level interest,” Bailar asserts. Despite dedicated administrators such as NIGMS’s Poodry and John Ruffin, who heads NIH’s National Center on Minority Health and Health Disparities, Bailar says that “a lot of senior managers view these programs as an obligation and don’t give them the attention they deserve.”

Even so, the paucity of good data didn’t prevent the panel from concluding that the programs are essential. Indeed, it recommended that “NIH should commit to the continued funding of minority-targeted research training programs.” At a time when programs that favor members of a particular race or gender are under assault ( Science, 25 July 2003, p. 455), supporters worry that the lack of an adequate assessment could be a serious problem.

NIH has a long history of addressing the serious underrepresentation of African Americans, Hispanics, Native Americans, and Pacific Islanders in biomedical research. Some 79 programs serve populations from community college students (see sidebar) to postdoctoral fellows. Despite some gains, the current output is tiny—108 blacks, 175 Hispanics, and 11 Native Americans earned biological science Ph.D.s in 2003, for example—and their 7.3% share of the total number of degrees awarded is a far cry from the group’s 25% presence in the general population (see graphic).

The NRC panel examined 49 programs over a period from 1970 to 1999. After spending more than 4 years and $1.5 million, the panel delivered a stinging rebuke of NIH management practices.

One problem, according to the panel, is NIH’s narrow definition of success. What the agency wants, in the words of veteran NIH training administrator Walter Schaffer, is “people who can do research and sit on our review panels and advisory boards.” But although few participants make it through the doctorate—nobody has a clue what percentage—many still contribute to the biomedical sciences after earning a lesser degree. “Many of these programs serve quite a different population than the typical NIH training program, so a lower rate of success isn’t very surprising,” says Bailar. As a result, the panel concluded, NIH should consider a “broader definition of success.”

Another sticking point is the inaccessibility of relevant data. Project directors submit annual progress reports, but the data generally do not include longitudinal information on a student’s entire academic career. “Once a student leaves, what’s the motivation for an institution to track them?” asks Ruffin. And even when grantees dig out that information and submit it to NIH in their annual reports and renewal applications, the agency hasn’t found a way to compile it and use it effectively. “We had an electronic database that was also supposed to serve as a tracking mechanism,” says Adolphus Toliver, who runs both the Bridges to the Baccalaureate Degree and the Minority Access to Research Careers program for upper-level honors students. “But it didn’t work, so we stopped using it.”

Will This Bridge Take Me to the Lab?

The National Institutes of Health's (NIH's) "Bridges to the Baccalaureate Degree" tries to help underrepresented minority students become scientists. But measuring its impact--and, by extension, all of NIH's minority-training programs--may be a bridge too far.

The Bridges program links community colleges with research universities at dozens of sites around the country. Working with students more likely to have grown up on an Indian reservation, in an urban ghetto, or with parents speaking a language other than English than are their undergraduate peers attending research-intensive universities, the Bridges program supplements coursework with academic assistance, career counseling, paid lab jobs, lectures, and other activities.

It's an excellent way to expose them to a life in research. But it's a long way from fulfilling NIH's dream of turning them into biomedical researchers capable of winning federal grants. Even following what happens to the thousands of students who try to cross that bridge has so far proven impractical.

Why are the students so hard to track? The main reason is their educational peregrinations. Even if students earn their associate's degrees, they may not head to a 4-year school. If they do, they may not win a spot in another NIH-sponsored minority program, if one exists on campus. They may not major in science. Even if they graduate, they may not attend graduate school. And so on.

The available data are both impressive and sobering. California State University, Los Angeles (CSU-LA), has had a Bridges grant since the program's inception, giving project director Linda Tunstad an unusually long perspective on what happens to her students. From a 12-year pool of 148 students, she says, some 76% continued their education at a 4-year school, two-thirds of them at CSU-LA. At least 37%--55 and counting--have earned bachelor's degrees. Of the 75 who attended CSU-LA, 39 have earned bachelor's degrees, mostly in the biological and chemical sciences, and 22 have gone on to graduate programs.

Even if institutions take the trouble to follow their students, however, the significance of the journey may not be clear. When NIH's Adolphus Toliver told Bridges directors recently that the program has a 23% transfer rate--the share of community college students who advance to a 4-year school--the group's first question was: "Is that good or bad?" recalls biologist Thomas Landefeld of CSU Dominguez Hills, past president of the project directors' group. Toliver's answer? "We don't know."

The group would also like information on how their participants stack up against the general student population in terms of completing a bachelor's degree, entering graduate school, and earning a Ph.D. "We think that our programs are adding value," says Landefeld. "But without comparative data, it's hard to know for sure."

–Jeffrey Mervis

A third problem is that students don’t necessarily remain in a minority-training program throughout their education. Even programs that link different types of institutions—such as the Bridges to the Baccalaureate Degree from community colleges to 4-year institutions, and the Bridges to the Doctoral Degree from master’s to doctoral programs—don’t promise students a slot as they advance. That undermines the program’s effectiveness, not to mention making it harder to track students.

One of the biggest complaints from the NRC panel is that NIH officials were unwilling or unable to make program data available for a rigorous analysis. NIH shared the data with a private contractor, who surveyed participants and project directors. The response rates were as low as 8%, however. “The NIH data contract achieved a very low response rate,” the panel concluded. “As a result, there is a high likelihood of bias among the survey results.” The restriction also prevented the NRC panel from doing its own analyses, Bailar adds.

The failure to support claims of success could in the long run hurt these programs, which are already under threat from those who disapprove of race-based preferences. In November, for example, the U.S. Justice Department threatened to sue Southern Illinois University for running three such programs, one a “bridges” effort for underrepresented minorities pursuing science degrees, funded by the National Science Foundation ( Science, 25 November 2005, p. 1263). Even agencies that support minority preference may balk because the evidence for these programs is weak. “When I briefed NIH on the report,” says Bailar, “some of the institute directors said: ‘You haven't shown any evidence that it produces a lot of Ph.D.s. So why should I bother to fund it?’ ”

Jarvis and Poodry harbor no doubts about the value of the training programs. “I wouldn’t be where I am today without these programs,” says Jarvis, who this fall won a prestigious NIH Director’s Pioneer Award.

Poodry feels likewise. But he also thinks that the NRC panel is right in calling for clear and measurable outcomes. “We need a doubling [of minority Ph.D.s] every 8 years to shift the [participation] curve to where it should be,” he says. “If that doesn’t happen, then 20 years from now we’ll probably be looking at the same results and wondering why things haven’t improved.”

* Assessment of NIH Minority Research and Training Programs, Phase 3, NRC (2005).

Jeffrey Mervis is a senior correspondent for Science magazine.