NSF Continues to Fiddle With the Meaning / Utility of "Impact" for the Assessment of Grants. (January 10, 2012)

The overseer of the US National Science Foundation (NSF), the National Science Board (NSB) has come up with yet another couple of modifications in order to usefully utilise the concept of "broader impacts"* in the implementation of its merit review criteria.

Over the years the NSF has had great difficulty in determining how to evaluate "impact" and its latest effort in "tweaking the merit review rules seems to moving toward eliminating the concept without eliminating the term and admitting  defeat. The NSB's 300-page report is available online.

As Jeffrey Mervis notes in ScienceInsider: "NSF guidelines currently provide eight examples of possible outcomes [with regard to broader impacts]. They range from attracting more women and minorities into science to fostering ties between academia and industry. The list has become a de facto definition of broader impacts, in other words, a blueprint of the ideas investigators believe NSF is most likely to fund."

But that wasn't the intention, the NSB contends, and the report, published on December 14, 2011, states:

In the final analysis, NSB believes that the Intellectual Merit and Broader Impacts review criteria together capture the important elements that should guide the evaluation of NSF proposals. Because of the great breadth and diversity of research and education activities that are supported by NSF, the Board has decided not to recommend a specific set of activities related to Broader Impacts, just as it would not recommend particular types of research. Those decisions are best left to the PIs [principal investigators] to describe and to the NSF to evaluate.

The report goes on to suggest that some sort of aggregated assessment should be used and as Mr Mervis interprets the NSB's language: "it may make more sense to evaluate the success of a cluster of grants given out by an entire NSF program, or to examine the combined efforts of multiple grants at a particular university."

As the report puts it:

NSB notes that assessing the effectiveness and impact of outcomes of these activities one project at a time may not be meaningful, particularly if the size of the activity is limited. Thus, assessing the effectiveness of activities designed to advance broader societal goals may best be done at a higher, more aggregated, level than the individual project. Large, campus-wide activities or aggregated activities of multiple PIs could lend themselves to assessment, which should be supported by NSF.

Just how this aggregated assessment is supposed to work -- well the NSB may be in the business of synthesising camels as well.

_____________________________________

*On July 26, 2011  Professor Keith Yamamoto, vice-chancellor for research at the University of California, San Francisco fronted the US House of Representatives Subcommittee on Research and Science Education. He emphasized the importance of using the merit review committees to judge proposals solely on their scientific merit, and said that it was inappropriate for reviewers to be asked to "step outside of their areas of expertise" and to "make guesses" if proposals are meeting national goals. Furthermore, he believes it is impossible to make such assessments at the level of individual projects. In short that should be the job of governmental funding mechanisms and the mission-driven agencies that support research in areas such as health, environment, or national security -- not the NSF.