Frank Larkins Estimates the Costs of the ERA Process So Far To Be $100 million. (August 24, 2011)

 Frank Larkins is Professor Emeritus in the School of Chemistry at the University of Melbourne. He is a former Deputy Vice Chancellor at that university, Dean of Science and Dean of Land and Food Resources. He has published more than 200 scientific papers. His current interests are in research, education and energy policy developments. He has recently published a book, Australian Higher Education Research Policies and Performance 1987-2010 (MUP 2011).His scientific publications have been in theoretical chemical physics, catalysis, coal chemistry, materials and fuel science.

In the August 24, 2011 LH Martin Institute online newsletter Professor Larkins contributed: 2010 Excellence in Research for Australia Exercise - What has been learned about discipline quality and diversity?

In his abstract Professor Larkins writes:

Australia has considerable diversity in its research activities as measured by the contributions of universities to the disciplines assessed in the ERA exercise. In terms of quality of performance, for 18 of the 25 disciplines evaluated at least half of Australia’s universities submitting research outputs were deemed to be at or above world standard. However, in some of the remaining 7 disciplines less than one third of the universities assessed were at world standard. Overall science-related disciplines are rated more highly against world standards than humanities and social sciences disciplines. The impact of these findings on the quality of Australian Universities teaching and research programs and on the government’s proposal to reform higher education requires much more debate.

In criticism of the Australian Research Council Professor Larkins cites the ARC's failure to publish world benchmarks against which the Australian  research was measured for the ERA. In his view this has compromised the ERA's audit of research quality.

In defence the ARC's chief executive Margaret Sheil told The Australian's Jill Rowbotham a copy of the data was kept in the office of every deputy vice-chancellor, research, in the country. "We used citation benchmarks from [bibliometrics company] Scopus, but to publish them would have cost us extra money." Furthermore, she said the benchmarks were not the only piece of information considered by the research evaluation committees and, had the ARC released the citation data, people would "not [have looked] at anything else".

Ms Rowbotham concludes: "As such, the strategy not to publish was part of maintaining the integrity of the ERA process. Further, citation benchmarks were not used in every discipline and, where they were, they were moderated by expert panels."

Just what that's supposed to mean remains a mystery.

Now the federal government in the 2011 budget papers list the over cost of the ERA at $35 million and Professor Shiel in an interview with the Times Higher Education's Paul Jump justified the spending of $35 million as worthwhile.

However, Professor Larkins writes in his article: "The responsible minister, Senator Kim Carr, Minister for Innovation, Industry, Science and Research has described the ERA exercise as ‘a key element of the government’s agenda for the reform of Australia’s higher education system’. The cumulative cost of the 2010 exercise to all parties involved was at least $100 million for data collection, reporting and evaluation. A further evaluation exercise will be conducted in 2012 to refine the ERA methodology."

Well which is it?

And then the former deputy vice-chancellor lists four matters of particular pertinence with regard to the ERA:

• What has been learnt by all parties from the 2010 exercise?

• How should the information be used within universities to manage the diversity and quality of an institution’s future research profile and its impact on teaching programs?

• How will the Federal Government use the ERA discipline quality information to allocate funding on a performance basis to universities in the future?

• How should a university prepare for the 2012 exercise in view of the 2010 outcomes?

But for all the statistics and five charts Professor Larkins produces in his paper does it really matter?

The whole $100 million ₊ exercise is geared to distribute research funding among university administrations using proxy methodology to determine the efficacy of past peer review by the ARC and NHMRC. 

Surely, if you are really interested in improving the research quality of the university sector you should improve the processes of competitive-funding assessment. And one of the most powerful tools to accomplish that is the full utilisation of modern communications technology by the funding bodies in order to improve the quality of those peers who review funding applications in conjunction with the principle of funding the individual(s) undertaking the research.

Toni Scarpa has spent the last six years at the US National Institutes of Health (NIH) as its director of the Center for Scientific Review. In his view:  "peer review is simple. I think it should ask only two questions. First: Is it worth doing? That is impact and significance. If the answer is yes, then you ask the second question: Can they do it? In the past we were asking those questions in reverse."

But it is vitally important to employ the best individuals to ask those questions and evaluate the answers.

Spending over $100 million on second guessing peer review through a rearward focusing ERA which will be used to determine the distribution of research funding through university administrations is not the way to do it.