NSF and the Conundrum of Impact. (October 14, 2011)

Lord Sainsbury in a recent interview given to Times Higher Education was adamant in stating:

I think the coalition is making a mistake  -- which I have to say tended to be made by the Labour Party -- which is to think the way you get innovation is [by] putting a heavy emphasis on impact. this is totally to misunderstand what you should be doing. because it makes no sense to ask people doing basic research to say exactly what would be the impact of their work "because usually you don't know".

On July 26, 2011  Professor Keith Yamamoto, vice-chancellor for research at the University of California, San Francisco fronted the US House of Representatives Subcommittee on Research and Science Education. He emphasized the importance of using the merit review committees to judge proposals solely on their scientific merit, and said that it was inappropriate for reviewers to be asked to "step outside of their areas of expertise" and to "make guesses" if proposals are meeting national goals. Furthermore, he believes it is impossible to make such assessments at the level of individual projects. In short that should be the job of governmental funding mechanisms and the mission-driven agencies that support research in areas such as health, environment, or national security -- not the NSF.

Nevertheless, the National Science Foundation (NSF) since 1997 continue to use two criteria to judge the ~55,000 grant proposals it receives annually. 1) intellectual merit, 2) "broader impacts" which has come to be seen by researchers to require expert tutoring in order to avoid self harm.

According to Science's Jeffrey Mervis reporting in this week's issue: "For only $79—marked down from $197, according to one recent e-mail—a company in Florida offers a CD that teaches applicants how 'to successfully identify, distill, and communicate your project's broader impacts to NSF reviewers, improving your chances of funding'". Mr Mervis goes on to add: "For those scientists on a tight budget, there are plenty of free tips in the open literature."

That decision taken 14-years ago by the NSF was intended to deflect criticism that it "cared more about the interests of the academic community it serves than the needs of the taxpayers who ultimately foot the bill for its programs".

This past June the NSF in proposing a set of draft guidelines to clarify its definition(s) of impact drew some 265 emails mostly critical and as a result the formulating taskforce is back at work: "It's our attempt to be frighteningly clear," task force member Alan Leshner (CEO of AAAS) told the task force. "But if we're not, we need to know that."

Cora Marrett, deputy NSF director and former head of NSF's education directorate told Mr Mervis: "So many people have interpreted the broader-impacts criterion as requiring something involving K-12 education, but that's not our position at all. Efforts to find activities that are universal may be totally inappropriate for a particular field, or project, or individual PI [principal investigator] because people may not have the special expertise needed." And there appears to be a consensus among scientists that no two review panels follow exactly the same definition of broader impacts.

The taskforce noting the concerns voiced is in the throes of a redraft and Mr Mervis writes: "Although the task force has not made public a copy of its latest draft, John Bruer, president of the James S. McDonnell Foundation and chair of the task force, agreed to talk with Science about its key provisions. [T]here are three governing ideas... The first declares that all NSF projects 'should be of the highest quality.' The second says that, in the aggregate, projects should contribute more broadly to advancing societal goals. The third states that any assessment of a project's broader impacts should be 'scaled' to the size of the activity... The emphasis on top-quality research in the first principle, Bruer says, should assuage those who fear that the broader-impacts criterion 'would dilute NSF's commitment to excellence in research.' The second principle, he adds, explains 'how broader impacts might be achieved. We want to make clear that it's not seen simply as an additional activity on a grant but rather that it is an essential component of the research activities.'"

If by this time you're becoming bemused at the semantic contortions Dr Bruer -- who holds degrees in philosophy from the University of Wisconsin (B.A., 1971), University of Oxford (B. Phil., 1974), and the Rockefeller University (Ph. D., 1978, Thesis: Classical Mechanics as a Limiting Case of Quantum Theory) -- "said it may turn out that broader impacts are best measured 'in the aggregate.' Some types of NSF-funded activities and programs, such as large centers or collaborations with industry, 'are explicit about how broader impacts will be evaluated.' But that's not the case for standard grants to individuals. 'Principal investigators have tried to respond in good faith and devote appropriate resources to these activities,' he says. 'But it may be more appropriate to measure them at the portfolio level of an NSF directorate, or perhaps across an entire university department or institution.'"

This really is past approaching nonsense if what is under review by the NSF are proposals of basic research and if the proposal is of sufficient merit to be worthy of funding, the proposer is competent to undertake it, and the means to do so are available that should be sufficient because that encompasses potential impact.

To site just one classic example: Reed–Solomon codes are non-binary cyclic error-correcting codes developed by Irving  Reed and Gus Solomon when staff members of MIT's Lincoln Laboratory. At the time of their development, (1960) they were of no practical value, indeed an efficient decoding algorithm was not known. The first commercial application in mass-produced consumer products appeared in the compact disc some 22-years later in 1982.

Whether or not Lord Sainsbury is voted in as Chancellor of Cambridge this weekend, he still knows what he's talking about.