Citation Statistics Brought to Heel?  Italian Academics Say So. (October 21, 2008)

Question: Can you meaningfully compare citation statistics between disciplines?

Claudio Castellano at Rome's Sapienza University together with his colleagues say they can¹, at least within the scope of science, mathematics and engineering, even though the raw numbers are not comparable. For example, the authors note that for papers published in 1999, articles with 100 citations are 50 times more common in developmental biology than in aerospace engineering.

Their approach is remarkably simple, although the execution is tedious and time consuming: the authors claim that indices of scientific performance, if based on citations, can be normalized to account for differences in the overall citation rates of different disciplines, allowing for more meaningful comparisons. Furthermore, the results are claimed to be valid to grade individuals, departments, universities or even nations and as Philip Ball writes in NatureNews "in principle, the technique could be used to draw up a league table of the most influential scientists of all time, based on how much their work has been cited".

According to second author Santo Fortunato of the Institute for Scientific Interchange in Turin the approach: "would give you so much more [than previous metrics]."

In collecting a large number of papers, the researchers found that while the raw numbers of citations varied widely from one discipline to another if "the citation counts are divided by the average number of citations per paper for the discipline in that year, the resulting statistical distributions" fit closely to a single curve, corresponding to a log-normal distribution.

As an example of the strength of the approach Forunato says the method allows them to conclude that an article published in aerospace engineering in 1999 that gained 20 citations had more impact in its field than an article with 100 citations in developmental biology. And the relation holds between years. For example, Philip Ball reports "the statistical distributions of citations for 1990 and 2004 are indistinguishable (in every field) once normalized in this way to the averages for the respective year".

However, Sidney Redner, a specialist in citation statistics at Boston University cautions not all academic disciplines might match the 'universal' citation curve that the team has found. "In the humanities," Redner says, "monographs seem to be the primary intellectual expression and I would not be surprised if their citation statistics can't be scaled in the same ways as articles in archival journals."

If the claims for the approach hold up, it will be more powerful for the comparisons of citations than the h-index, devised in 2005 by physicist Jorge Hirsch² which is unable to be used across disciplines.

Finally, Mr Ball concludes: "All the same, it is widely acknowledged that automatically generated indices such as this should not be the sole judge of academic achievement — not least because they tend to measure the popularity of a paper rather than its intrinsic intellectual value, says Redner."

Now this ought to give Senator Carr and his press ganged minions battling with mechanisms for ERA analyses something to occupy their idle hours.

But the really interesting question is, if the finding is valid, why does the log-normal distribution describe citation frequencies.

___________________________

1. Radicchi, F. , Fortunato, S. & Castellano, C. Proc. Natl Acad. Sci. USA doi:10.1073/pnas.0806977105 (in the press).

2. Hirsch, J. E. Proc. Natl Acad. Sci. USA 102, 16569–16572 (2005).