Some Observations on Citation Metrics. (January 18, 2007)

It's the 200 kg. gorilla lurking in the room which shows little signs of being moved on by the Minister for Innovation, Industry, Science and Research (DIISR), the additional layer of bureaucracy which Kim Carr wants to use to determine university research funding beyond that allocated through peer review by the research councils (ARC, NHMRC) -- the Research Quality Framework.

Interestingly, Senator Carr while avowing his admiration for the peer review system utilised by the Australian Research Council has give no indication of demonstrating it by allowing it the additional funds so that its grants would include appropriate on costs. Rather he has repeatedly harked back to the use of metric based evaluation of institutions (i.e. citation evaluation of some sort of its researchers coupled perhaps with other as yet to be described parameters).

The universities have repeatedly pointed out that the institutional research grants they receive are used to a marked extent to supplement the inadequate on costs provided by the commonwealth research grants.

Senator Carr has the opportunity to revise the research grants for which his department is responsible in a way which directly rewards the "guys" that show promise of doing good research and/or continuing to do so.

To ignore the contortions the UK government is going through to try to justify the continuation of the Research Assessment Exercise in some form or other defies understanding.

A letter in this week's Nature is just the most recent example of some of the pitfalls of relying on citation metrics in a simplistic fashion, and is it realistic to assume that what would evolve as the DIISR's methodology would be an improvement on allowing successful peer reviewed proposals to be adequately funded -- including on costs.

Certainly large infrastructure proposals submitted by institutions are and should continue to be peer reviewed apart from grants applied for by principal investigators, but that's a separate issue.

What is required are not rote applied formulae in the hands of government apparatchiks but the development of a peer review system of outstanding competence.

If Senator Carr thinks the ARC's system is good now why not use it as the basis for developing an outstanding one. Currently the NHMRC is in consultation with the US' NIH working on how to accomplish just that.