Research Quality and the Rating Game. (August 18, 2005)

    With the meeting tomorrow of the Expert Advisory Group working on models for an Australian Research Quality Framework (see the immediately preceding News and Views) Nature's editorial on "Ratings Games" appears timely.

First off they call attention to a newly suggested metric, the h-index, devised by physicist Jorge Hirsch, at the University of California, San Diego. Simply stated it's the highest number of papers that a scientist has written that have each received at least that number of citations, e.g. an h-index of 50 means someone has written 50 papers that have each had at least 50 citations.

What Hirsch finds is that on using his index the top ten physicists have  h values above 70, while the h-index for the top ten biologists equals 120 or above so disciplinary factors seem necessary for starters.

Having placed his preprint on www.arxiv.org/abs/physics/, Hirsch has called for comment before proceeding to publish. Of course once set up, obtaining an h-value requires little effort and perhaps even less thought on the part of funding agents but clearly has built in bias for popularity of research area within a discipline. In short to be useful it would seem to require significant fine tuning which is not to say that it wouldn't be useful as an adjunct.

As Nature's editorial puts it, "Whatever its virtues, any citation analysis raises as many questions as it answers and also tracks just one dimension of scientific outputs."

Secondly, Nature refers to the newly amended Research Assessment Exercise (RAE) which will be implemented in the UK beginning in 2008. The journal applauds the fact that the RAE will prohibit assessment panels from judging papers by the impact factors of the journals in which they will have appeared.

The common approach of the RAE's disciplinary panels is to assess up to four submitted outputs (typically research papers or patents) per researcher, of which a proportion will be assessed in some detail (25% for the biologists, 50% for the physicists). There will no doubt be something of a challenge in taking into account the fact that a typical publication has several co-authors.

    These outputs will sit alongside indicators of the research environment such as funds and infrastructure, and of esteem, such as personal awards and prestige lectures.

Whether or not the RAE since its inception in 1986 has per se significantly affected the quality of research in the UK is debatable because not until worthwhile increases in resources for research were made available did UK research benefit.

In Australia unfortunately the machinations of the Coalition Government as regards research funding seem designed to make available as little public funding as possible and simply reapportioning existing resources will prove all but futile as regards fostering a marked increase in the quality of the nation's research.