Editorial-26 June 2006

 

 

   


Minister for Education, Science and Training - Julie Bishop

The Continuing Heartburn Over the UK's Research Assessment Exercise

 

 
 
 
 

The Minister for Education, Science and Training, Julie Bishop, in the middle of this month signalled that although the government had put back the introduction of the Research Quality Framework (RQF) by twelve months, she remains committed to developing an instrument -- more or less modelled on the UK's Research Assessment Exercise (RAE) -- for its realisation.

 

In so far as her public pronouncements are concerned she appears oblivious to what Britain's researchers had been hoping for since March when the Chancellor of the Exchequer, Gordon Brown, announced his intention to scrap the much-criticized RAE (UK universities have devoted huge amounts of time and money to the exercise) with a useful reform. He signalled that he was looking to develop a system usefully emphasising metrics.

 

Now at a June 21 conference of university heads hosted by the UK's Higher Education Policy Institute (Hepi) strong criticisms were voiced of the government's 27-page discussion paper which suggests various models as possible replacements for the RAE.

 

According to The Guardian, "The director of Hepi, Bahram Bekhradnia, said the consultation paper was so brief that it contained 'just seven or eight pages of discussion and analysis, with no attempt to show that metrics would be a better system'". A number of the conferees concurred.

 

Also present at the conference was Sir Gareth Roberts, president of Oxford's Wolfson College, who reviewed the RAE for funding councils in 2004, and in Australia chaired the Expert Advisory Group which was responsible for the development of the stillborn preferred model for Australia's  RQF.

 

He told the June 21st conference that the scrapping of the RAE coincided with Australia's introduction of its own system of rating university research, modelled on the British version.

 

We don't know whether he has obtained confidential intelligence from the Minister or is just blessed with  precognition, but as one well placed academic put it, either way, "It's a chilling thought."

  From Nature 441:917 (2006)

 

The Guarding reported that Professor Roberts went on to say that a metrics system should have expert panels for all disciplines and funding councils should use financial allocations from the 2008 RAE as the "baseline" for future awards.  "He said international benchmarks could be used to compare research in universities around the world and he proposed the introduction of an 'alternative research assessment for less research-intensive universities'", which doesn't exactly suggest a significant simplification of the RAE but rather the transformation of a sow's ear into a dog's breakfast.

 

The UK government's consultation paper claims that were money allocated to universities based on how much income they already attract from competitive grants, charities and industrial contracts, the income would correlate well with the results of RAE peer review across the whole university sector.

 

However, as Jim Giles points out in his assessment in the June 22 Nature any of the five different models examined in the consultation paper, "...have roughly the same effect of boosting universities that are good at bringing in money, such as those with the research capacity to attract significant funding from medical charities (for example, University College London) or industry (such as Cranfield University), [while] taking money from those for whom such activity is a smaller component of their total income (see table).

 

And while proponents of the RAE claim it has boosted the quality of British science since it was introduced, it has never been determined just how effective of itself it has been, because it coincided with large injections of resources.

 

Tom Sastry, a Bristol-based senior researcher at the Higher Education Policy Institute, an independent think-tank, told Giles if the model(s) proposed in the government's paper were followed it would reduce "universities [to doing] research for other people", and he is not the only one suggesting that the Chancellor's intention is "to push universities towards application-driven projects... Fundamental research, which does not attract so much external funding, could then suffer. 'Many theoretical fields are going to find it hard,' agrees Anthony Van Raan, an expert in research assessment methods at Leiden University in the Netherlands."

 

Van Raan says universities could become overly focused on short-term pay-offs: "This might stop research that could be interesting in ten years or so."

 

He does agree that it is sensible to employ metrics in the assessment but they must be sufficiently broad in scope in order to obtain a plausible balance.

 

However, according to Giles, "The Treasury seems... to have limited interest in expanding the range of metrics used. [A] Treasury spokesman, while noting that the consultation asked for views on the possibility of including other metrics, said that "bibliometrics are not sufficiently developed to have been incorporated into the modelling" and described PhD numbers as a 'less robust measure'."

 

It is difficult not to come away with the opinion that the UK government is unwilling to develop or revert to more direct funding assessments based on individuals and well led research groups as determined by critical peer review.

 

Is in fact the suggestion that Gordon Brown's intention is to push universities towards application-driven projects because fundamental research doesn't attract so much external funding close to the mark? If so, the lack of understanding of the importance of fundamental scholarship and basic research to universities' infrastructure is something he shares with his Australian counterpart among other members of the Cabinet.

 

Certainly in Australia, because of its extended track record for prescriptive micromanagement, there are serious questions as to the motives of the Federal Government for wishing to introduce an RQF with an additional bureaucratic layer to determine the distribution of ~$600m of annual funding for research in the nation's publicly funded universities and research organisations.

 


 

Alex Reisner

The Funneled Web