Bias Indicated in NIH Grant Application Reviews. (July 29, 2008)

Valen Johnson

Valen Johnson, a biostatistician at the University of Texas, M.D. Anderson Cancer Center in Houston, reports in Proc. Natl Acad. Sci. USA doi:10.1073/pnas.0804538105 (2008) that his analysis of 19,000 NIH grant applications reviewed by more than 14,000 reviewers indicates that the NIH needs to consider correcting its peer review system to overcome bias which occur with those applications that are on the borderline of acceptance or rejection.

Dr Johnson estimates that some 25% of applications fall into the borderline classification and it's this area which is subject to bias.

It will be recalled that in 2007, NIH director Elias Zerhouni launched an initiative to examine the NIH peer-review process. The agency has since proposed several changes, including shortening grant applications and comparing young investigators' proposals separately. However, the review panels did not address the possibility of review bias in the assessment of grant applications.

While grants are assigned to study sections of about 30 members, each application is actually read by just two to five reviewers, (the mean is 2.8) and then favoured proposals discussed when the section meets. The application is then assigned scores by all of those present, and the final score is the mean.

Dr Johnson has served on several NIH grant-review committees and while sitting on one five years ago began to suspect the rankings were not as objective as they ought to be. In essence he found the top scoring grants were largely unaffected by reader bias, but that such bias did impact grants closer to the funding cut-off line: "With only two to three people, on average, reading the proposals, the particular individuals that happen to read it have a major impact on its final score."

NatureNews, quotes pathologist David Kaplan of Case Western Reserve University in Cleveland, Ohio, who has performed his own analyses of the NIH peer-review system as saying: "The statistical analysis that the NIH has done is pretty elementary; Johnson's approach basically says we can do better."

Toni Scarpa, director of the NIH Center for Scientific Review is not enamoured of Dr Johnson's analysis and according to NatureNews:

...says that the CSR had heard Johnson present preliminary results and was not interested in pursuing the project further. Although the center is interested in revising its scoring procedures, "there was relatively little enthusiasm" about Johnson's analysis, says Scarpa. Instead, the center is considering other changes, such as varying the weight given to different criteria such as innovation or the strength of preliminary data. It is also considering shortening applications and increasing the number of reviewers to about ten, he says.

One thing is clear there is still room for improvement as regards peer review and it would be surprising if Dr Zerhouni didn't press for it.