Scrutiny of NIH Peer Review Fingers Possible Shortcomings in Awarding of Grants. (December 6, 2012)

A "Comment" article by Joshua M. Nicholson and John P. A. Ioannidis in the December 6, 2012 issue of Nature, "Research grants: Conform and be funded" contends that: "Too many US authors of the most innovative and influential papers in the life sciences do not receive NIH funding".

The graphic below sums up their findings:

SOURCE: SCOPUS/NIH

Summarising their salient points:

Of the more than 20 million papers published worldwide between 2001 and 2012 catalogued by Scopus, 1,380 had received 1,000 citations or more as of April 2012.

700 of these were biomedical papers with a total of 1,172 discrete single, first or last authors, i.e. "primary" authors.

Nicholson and Ioannidis make the following points:

• Only 72 US-based authors who between them had published 84 eligible articles with 1,000 or more citations each and who were current members of an NIH study section, i.e. 0.8% of the 8,517 study-section members. Most of the 72 (n = 64, 88.9%) currently received NIH funding.

• A random sample of 200 eligible life- and health-science papers with 1,000 or more citations yielded a group of 158 articles with 262 eligible US authors who did not participate in NIH study sections. Only the minority (n = 104, 39.7%) of these 262 authors received current NIH funding.

• Among authors of extremely highly cited papers, study-section members and non-members showed no significant difference in their total number of highly cited papers, despite the fact that members of study sections were significantly more likely than non-members to have current NIH funding. For those with a single eligible highly cited paper (51/59 versus 91/243, p < 0.0001) and overall in a stratified analysis (p < 0.0001).

• A random sample of 100 NIH study-section members was chosen. Not surprisingly, 83% were currently funded by the NIH. The citation impact of the 100 NIH study-section members was usually good or very good, but not exceptional:... Only 1 of the 100 had ever published a paper with 1,000 or more citations as single, first or last author. [Nicholson and Ioannidis state: "[I]t is worrying that the majority of highly cited investigators do not have current NIH funding as principal investigators.]

The authors agree that there may be a number of reasons why authors of the very highly cited papers may not have go on to receive NIH funding,  e.g. they had applied for NIH grants but had been unsuccessful, the researchers were graduate students at the time of the high-impact publication, so had just started their own labs (the average age for being awarded a first NIH grant as primary investigator is 44 years), they had moved into industry or changed fields entirely, had retired or was receiving some NIH funding as a co-investigator but was not listed as a principal investigator.

Nevertheless, they maintain: "The mission of the NIH is to support the best scientists, regardless of whether they are young, old or in industry. If they have left academia — or moved out of research entirely — it could suggest that these authors of high-impact work did not want to continue the struggle for federal funding. Such innovative thinkers should not have so much trouble obtaining funding as principal investigators. One cannot assume that investigators who have authored highly cited papers will continue to do equally influential work in the future. However, a record of excellence may be the best predictor of future quality, and it would seem appropriate to give these scientists the opportunity of funding for their projects. We feel that by allowing grant holders to serve as grant reviewers, a conflict of interest becomes inescapable. Exceptional creative ideas may have difficulty surviving in such a networked system... there are many examples of major scientific discoveries, including Nobel prizes, that emerged from unfunded work.

    "Serious consideration should be given to increasing funding for investigators of outstanding ability who have already proved that they can accomplish something major... Such investigators could be funded ...without having to submit grant proposals or, as is currently done, by calling for proposals that present only broad goals."

All of which brings to mind a point made some years ago by Keith Yamamoto in dealing with study-section panels which he phrased as: "If you think like I think, then I think you're smart."