The Development of Indicators for the ERA. (January 5, 2009)

It's reminiscent of the inexorable evolution of a gaseous cloud of certain mass as it condenses in space to form a star and with time expands into a red giant engulfing and incinerating its planetary satellites and then ends a white dwarf.

On Friday the 19^th of December the Australian Research Council announced in a media release:

The Australian Research Council (ARC) CEO, Professor Margaret Sheil, today released two key documents that will guide the Excellence in Research for Australia (ERA) initiative.

"ERA will use a range of indicators and peer review to support the evaluation of research excellence," Professor Sheil said.

"Today I release the ERA Indicator Principles and ERA Indicator Descriptors — two key documents that will assist institutions as they prepare for a trial of ERA in 2009.

"The ERA Indicator Principles provide the overall context and application of the indicator approach and the ERA Indicator Descriptors provide the detail of each indicator type."

The Minister for Innovation, Industry, Science and Research, Senator Kim Carr [said]: "This work will substantially assist ERA in demonstrating the quality and diversity of research undertaken at Australian universities."

Keeping in mind that the object is to evaluate research, which for the most part has been peer reviewed and supported before being undertaken, what is the point of effectively reviewing the research up to eight years since it was performed (i.e. 6 years post publication and perhaps 2 years following submission for publication)?

It would be interesting to hold a public debate:

Resolved that there is insufficient credible evidence that a Research Assessment-like Exercise improves the quality of a nation's universities' research.

Or if you prefer:

Resolved that there is credible evidence that a Research Assessment-like Exercise improves the quality of a nation's universities' research.

The approach that the ARC is suggesting for the ERA is outlined as follows:

The ARC will use a number of Research Evaluation Committees (RECs) to undertake expert review of the relevant disciplines.

Each REC will include members international-recognised members with expertise in research evaluation and broad discipline expertise.

Where there is an insufficient number of valid quantitative indicators to provide a reliable evaluation of research quality, the ARC will undertake peer review of a sufficient selection…

And then the pièce de résistance

The ARC acknowledges that peer review will have already occurred on a significant majority or (sic) research outputs and is already a condition of a research output being considered as one of the four major research output types collected for the HERDC (i.e. book, book chapter, journal article, refereed conference publications). Similarly, competitive grant income submitted for ERA will have been awarded on the basis of peer review. Any peer review process conducted in ERA will therefore be designed to avoid unnecessary duplication of effort.

As a TFW reader (admittedly a member of the hard science brigade) put it: "Sir Humphrey Appleby couldn't have expressed it more adroitly."

This is going to be a very, very expensive ERA of no clothes.