Peter Doherty

Second Guessing Peer Review Though the ERA - of What Value - at What Cost

Elizabeth Blackburn

In February 2008 the new federal  Minister for Innovation, Industry, Science and Research, Senator Kim Carr, announced that he would replace the Coalition’s proposed Research Quality Framework (RQF), which in turn was modelled on Britain’s Research Assessment Exercise (RAE, instituted in 1986) with a new quality assurance oversight of research in Australia’s universities which came to be known as Excellence in Research for Australia (ERA) with its first national report released on January 31, 2011.  For all three of these exercises the rationale was/is based on the assumption that the implementation of retrospective evaluation of university research will lead to more cost-effective support.

It is undeniable that following the introduction of the RAE, the quality of Britain’s university research improved significantly. The argument is, however, one of post hoc ergo propter hoc, but concurrent with the funding allocations derived from the RAE was a marked and continued increase in the allocation of resources to foster university research. To date no proponent of retrospective evaluation has made any attempt to analyse their relative importance.

On the other hand as far back as March 2004 Australia’s oldest living Nobel Laureate, Peter Doherty, noted for The Australian’s Dorothy Illing a fundamental difference from the US system. There, a $1 million grant for five years would attract a 42% payment ˗ or $420,000 ˗ to the host institution for infrastructure support. True, Australia’s Labor government has moved to increase defraying oncosts, but it has a long way to go before meeting them.

Then in June 2005 the ABC’s Ticky Fullerton interviewed Professor Doherty in preparation for the Four Corners’ “The Degree Factories”:

[T]he problem comes with the way that research support funds outside the competitive system are allocated. [T]he Australia Research Council and the National Health and Medical Research Council, distribute their resources on the basis of …peer review, …it’s equivalent to the process in the United States used by [the] National Institutes of Health, or in Britain, used by [the] Medical Research Council or [the US] National Science Foundation and so forth… you don’t really need a Research Assessment Exercise in this country. All you have to do is to look at where the research grant money goes. Now the Americans haven’t had a research assessment exercise, and the reason they haven’t had is that they simply pay indirect costs on the grants. As far as I’m concerned you don’t allocate research resources to an institution. You allocate them to individuals. Research is about the activities of individual human beings and the groups that they build in competing for the best outcomes, okay. So you don’t fund universities to do research; you fund people to do research and you then incidentally fund the institutions that support those people, and the way you do that is through an indirect cost rate mechanism.

Agreed that the system of peer review is imperfect, but introducing bureaucratic hindsight, which by its very nature is prejudiced against promising new researchers, is patently counterproductive. What is urgently required is updating the peer review systems of the ARC and the NHMRC, and allocating an increasing proportion of funding through competitive grants with adequate oncosts.

A crucial improvement of peer review would be to engage a significant proportion of overseas reviewers but without the inconvenience and expense of long-distance travel. As an example, William Sims Bainbridge, a sociologist and program director at the U.S. National Science Foundation (NSF) is developing the use of Second Life where the avatars of a group of peer reviewers for the NSF examine the question: “Can the hard work of grant review be done without face-to-face meetings?”

According to Dr Bainbridge: “the format of the meetings followed a traditional schedule, and all of the work was completed on time”. Were only a fraction of the $35.8 millon that Senator Carr so far admits to investing in his ERA to be directed toward improving the system of peer review and rebalancing the proportion of funding between block grants and competitive grants with adequate oncosts, a significantly greater return on research investment would be realised.

And the admonishment by Australia’s most recent Nobel Laureate, Elizabeth Blackburn, rebutted:

“I think there are tremendously good scientists in Australia but sometimes [I wonder] are they really being able to run with it in the way they are capable of?”  In the US, she said she benefited from a five-year grant that allowed her to follow her nose without having to write up “damn little” reports and catalogue milestones on a regular basis. “This was the perfect setting and I'm not aware that I would have been able to do that [here]. Short dribs and drabs” of money with tight constraints on basic research are in her opinion wasteful.

Alex Reisner

The Funneled Web