Toni Scarpa has spent the last six years at the US National Institutes of Health (NIH) as its director of the Center for Scientific Review, having been recruited in 2005 from Cleveland Ohio's Case Western Reserve University by the then NIH director Elias Zerhouni who instructed him to overhaul NIH's system of peer review.

In an interview with Nature's Meredith Wadman Dr Scarpa told her that in his view the most significant achievement during his 6-year tenure was: "Making the biggest change in peer review in 65 years. Elias [Zerhouni] was passionate about peer review. And he brought me in as the agent of change in an institution where changes are resisted."

And the matter  of greatest consequence?: "We started by trying to get the best reviewers, and by making it so that a lot of clinicians could review without coming here... And we brought in affirmative action for new investigators: now, each institute has to find a sufficient number of new investigators to fund, about 30% of its grants" (TFW's emphasis).

Dr Scarpa then said: "peer review is simple. I think it should ask only two questions. First: Is it worth doing? That is impact and significance. If the answer is yes, then you ask the second question: Can they do it? In the past we were asking those questions in reverse."

NIH Notice Number: NOT-OD-09-025 sets out just what is required of grant applicants where the following are the:

Core Review Criteria. Reviewers will consider each of the five review criteria below in the determination of scientific and technical merit, and give a separate score for each. An application does not need to be strong in all categories to be judged likely to have major scientific impact. For example, a project that by its nature is not innovative may be essential to advance a field.

Significance. Does the project address an important problem or a critical barrier to progress in the field? If the aims of the project are achieved, how will scientific knowledge, technical capability, and/or clinical practice be improved? How will successful completion of the aims change the concepts, methods, technologies, treatments, services, or preventative interventions that drive this field?

Investigator(s). Are the PD/PIs, collaborators, and other researchers well suited to the project? If Early Stage Investigators or New Investigators, do they have appropriate experience and training? If established, have they demonstrated an ongoing record of accomplishments that have advanced their field(s)? If the project is collaborative or multi-PD/PI, do the investigators have complementary and integrated expertise; are their leadership approach, governance and organizational structure appropriate for the project

Innovation. Does the application challenge and seek to shift current research or clinical practice paradigms by utilizing novel theoretical concepts, approaches or methodologies, instrumentation, or interventions? Are the concepts, approaches or methodologies, instrumentation, or interventions novel to one field of research or novel in a broad sense? Is a refinement, improvement, or new application of theoretical concepts, approaches or methodologies, instrumentation, or interventions propose.

Approach. Are the overall strategy, methodology, and analyses well-reasoned and appropriate to accomplish the specific aims of the project? Are potential problems, alternative strategies, and benchmarks for success presented? If the project is in the early stages of development, will the strategy establish feasibility and will particularly risky aspects be managed?
If the project involves clinical research, are the plans for 1) protection of human subjects from research risks, and 2) inclusion of minorities and members of both sexes/genders, as well as the inclusion of children, justified in terms of the scientific goals and research strategy proposed?

Environment. Will the scientific environment in which the work will be done contribute to the probability of success? Are the institutional support, equipment and other physical resources available to the investigators adequate for the project proposed? Will the project benefit from unique features of the scientific environment, subject populations, or collaborative arrangements.

To Ms Wadman's comment that "The peer-review process is subject to continuous evaluation. What metrics have you used to assess whether the changes [you've introduced] are working?" Dr Scarpa replied: "Sally Rockey, NIH's director of extramural research, has done a major [42-page] review of those changes," which was released in December last year.

The final paragraph of the reports executive summary indicates the difficulties in developing an outstanding mechanism for peer review and unfortunately throughout the whole of the 42-page report there is no reference to any attempts made to improve the quality of reviewers.

That final paragraph of the executive summary:

NIH has been continuously monitoring feedback on the peer review changes since their inception and has already implemented a number of refinements to the new peer review system in response to this information. Most notably, in January 2010, additional guidance was issued to clarify distinctions between Overall Impact and the review criterion Significance; in September 2010 NIH began requiring reviewers to include a narrative statement to explain the overall impact score. NIH will reassess stakeholder opinions of the scoring system, the critique format and other peer review processes at a later date. In addition, NIH’s ongoing review of the NIH peer review system will also examine the shortened applications and alignment of research plans with the NIH review criteria.

____________________________________________________

On July 15, 2011 Professor Keith Yamamoto, vice-chancellor for research at the University of California, San Francisco, spoke to the the US President's Council of Advisors on Science and Technology (PCAST) on the Impacts that "Federal and Industry Funding for Science, Engineering, and Medicine has on American Universities", while on July 26, 2011 he fronted the US House of Representatives Subcommittee on Research and Science Education. He emphasized the importance of using the merit review committees to judge proposals solely on their scientific merit, and said that it was inappropriate for reviewers to be asked to "step outside of their areas of expertise" and to "make guesses" if proposals are meeting national goals. Furthermore he believes it is impossible to make such assessments at the level of individual projects. In short that should be the job of governmental funding mechanisms and the mission-driven agencies that support research in areas such as health, environment, or national security -- not the NSF.

By clicking on the image below you can watch  astrophysicist, theoretical physicist, and computational biologist William Press ask Professor Yamamoto for his views of how the NIH is handling the matter of improving its system of peer review, and judge just what effect the proposed system of Australia's ERA would have on advancing federally funded research in the light of what Professor Yamamoto has to say (~7 min. -- 16.3MB).

The clip downloads to be played

Alex Reisner

The Funneled Web