News & Views item - November 2010

 

 

 UK's Research Excellence Framework to Have Peer Reviewed Impact Component. (November 15, 2010)

Australia's political masters through the offices of Senator Carr, Minister for Research, have decided that the impact of research on the nation's wellbeing could not be meaningfully assessed  in its Excellence for Research in Australia (ERA) exercise, and overall peer review should be minimised in the ERA's scoring because of expense.

 

Now after more than a year of working on finding a solution to assess impact for its Research Excellence Framework (REF) the Education Funding Council for England (HEFCE), reckons its got the answer -- using peer-review panels to judge the 'impact' of research is "workable" and "robust". Nature's Natasha Gilberts reports that: "In [an] impact pilot study, university departments were asked to submit case studies — one for every ten of their academics — describing the impact of the research that academics had conducted during the past 17 years. These were reviewed by academics and industry scientists on subject-specific panels, and awarded rankings ranging from 4* (the best) to unclassified."

 

The peer-review panels are quoted as believing that, "expert review of case studies is an appropriate means for assessing impact," while Jonathan Grant, president of RAND Europe, a research consultancy based in Cambridge, who wrote a report last year criticizing the REF, now told Ms Gilbert, "If you are going to measure impact, this is the way to do it."

 

Nevertheless, concerns remain: difficulties may arise over describing impacts in which universities have collaborated with industry. Anna Grey, research manager at the University of York, says:
"Unless we can prove to the companies that the information will remain confidential, we will struggle to get hard evidence of impact."

 

And then there is the matter of the historical assessment of impact. The pilot studies allowed submissions of research  back 15 and 25 years ago. For physics, Ms Grey wants the full 25-year window. How relevant to the impact potential of the departments current research that is would seem, at the very least, to be a moot question. Furthermore, the relative importance of impact to the final REF scoring is the subject of debate -- 10%, 20%, 25%?

 

As to cost of the exercise -- the last RAE cost £60million (A$98 million), that remains a mystery while Rodney Phillips, an immunologist at the University of Oxford, UK who led his university's participation in the clinical-medicine pilot study and helped to judge submissions to the scheme informed Nature that rather than reducing the workload for academics — the government's stated motivation for the reforms — he says that the scheme "will be more work than the RAE [the previous Research Assessment Exercise]".

 

Finally, Ms Gilbert was told by Peter Main, director of education and science at the Institute of Physics in London,  that the general concerns about the procedural changes caused by the introduction of an impact measure to the REF have not been resolved. "Universities take research funding very seriously, and could put pressure on departments to continue research in a specific area to reap benefits from advances made a long time before, even when more future impact might be generated from new directions."