UK Government Reveals Plans for its Research Excellence Framework. (September 24, 2009)

David Sweeney, HEFCE director of research

The Higher Education Funding Council for England (HEFCE) yesterday admitted that its inquiries found that bibliometrics would not provide a "sufficiently robust" assessment of research quality in its proposed replacement for the Research Assessment Exercise (RAE), the Research Excellence Framework (REF). Nevertheless, Australia's Minister for Innovation, Industry, Science and Research, Senator Kim Carr, continues to extol the virtues of his replacement for the previous government's attempt to develop a Research Quality Framework (RQF), the Excellence in Research for Australia (ERA) because it will make heavy use of metrics and only use peer review as an adjunct thereby markedly reducing the burden of assessment without reducing its quality.

In the meantime in a 56-page consultation document the HEFCE's states: "While we remain concerned to reduce the burden of the assessment, we believe we have exhausted the main options for any radically different alternative approach (to the RAE)."

If there were any question as to the quagmire of these attempts at overall research assessment, witness that on the one hand Senator Carr spruiks that one of the virtues if his ERA is that it eschews the determination of impact while the HEFCE in its consultation paper now proposes an assessment regime based on three elements: "output quality" (weighted at 60%); "impact" (25%); and "environment" (15%) where "output quality" will continue to be based on peer review, termed "expert review" in the REF proposal and citation data will be available to "supplement and inform" the assessment of various science subjects.

According to Zoë Corbyn of The Times: "Impact must be evident during the REF assessment period, although the research could have been carried out ten to 15 years before. It will be assessed using departmental case studies - one for every five to ten staff submitted - and an impact statement from departments [while] the 'environment' measure will consider factors including research income, the number of postgraduate research students and their completion rates. Panels will decide whether bigger departments with "critical mass" should receive special recognition." In addition departmental infrastructure will be considered.

The consultation paper proposes a two-tier structure, with 30 sub-panels (one for each unit of assessment) working under the guidance of four main panels.

Nature reports: "The first REF exercise is due to run in 2013. From 2014, the results will be used to carve up the £1.76 (A$3.3)-billion funding pot for UK university research. HEFCE says that it will run the assessment 'periodically', possibly every four to six years."

Bahram Bekhradnia, director of the Higher Education Policy Institute, said: "Although the proposals have a new name, they are a development of the RAE," the added that it was "surprising and perhaps unwise [that impact be weighted at 25%]. It would have been wiser not to give so much weight to what is effectively an experiment."

Research Councils UK in a statement said: "We support Hefce's position that there is a role for quantitative evidence, including citation information ... but believe that peer review should still form the backbone of any assessment of research excellence."

But Sally Hunt, the general secretary of the University and College Union, criticised the plan, telling The Guardian: "Academic research should never be at the behest of market forces. History has taught us that some of the biggest breakthroughs have come from speculative research and it is wrong to try and measure projects purely on their economic potential."

Finally David Sweeney, the director of research at HEFCE, told a press conference that the REF is "not the same" as the RAE, the last of which ran in 2008. "There is a greater use of metrics and there will be an assessment of the impact arising from excellent research."

Is it really asking too much of the politicians and bureaucrats that they rethink and put their efforts into developing improved primary peer review systems of governmental funding bodies who support university investigators? Sufficient on costs should be provided while block funding for universities ought to be held to reasonable running costs to provide appropriate facilities for students and staff; if properly undertaken, an REF, or ERA is superfluous.

______________________________________________

According to the HEFCE:   Assessing research quality --

The REF will focus on three elements, which together reflect the key characteristics of research excellence. These are:

• Outputs
  The primary focus of the REF will be to identify excellent research of all kinds. This will be assessed through a process of expert review, informed by citation information in subjects where robust data are available (for example, in medicine and science).

• Impact
  Significant additional recognition will be given where researchers build on excellent research to deliver demonstrable benefits to the economy, society, public policy, culture and quality of life. Impacts will be assessed through a case-study approach that will be tested in a pilot exercise.

• Environment
  The REF will take account of the quality of the research environment in supporting a continuing flow of excellent research and its effective dissemination and application.