An Academic Researcher's View Regarding the Use of Journal Impact Factors
Professor Paul Fisher
The comment by Professor Warwick Anderson, CEO of the
National Health and Medical Research Council (NHMRC) reflects a valid
realization that journal impact factor is not, in and of itself, a measure of
the quality of each of the papers within it. However, what he actually meant by
this was that the impact of the journal is not a measure of the impact of every
individual paper and that is of course also true. Unfortunately, like so many
before him, Professor Anderson then goes on to equate impact with quality at the
level of the individual paper. This is done, as so often, by a linguistic
gloss-over ‒ you simply substitute one term for
the other ‒ and before you know it you
are measuring quality once again by citation counts, this time of individual
papers. Of course citation counts measure no such thing since, if anything, they
measure the size of the scientific readership, which varies hugely depending on
What to do? Well it is true of course that there is no substitute for having experts in the discipline actually read the papers. However that is an enormous undertaking and has in fact already been done by the referees of the journals in which the papers are published. Can one then assume that journals with higher impact factors must be publishing only better quality papers? No. In fact the primary hurdle for publishing in the very highest impact journals like Nature or Science is not the quality of the paper but whether the topic is considered by the editor to be of broad interest, i.e. potential high impact. Such journals frequently reject the great majority of publications submitted to them without ever subjecting them to the quality test of review by referees.
One, albeit also imperfect, way to resolve the conundrum is to assume that the authors of a high quality paper may not be able to get it published in Nature or Science, but they will most of the time be able to find a journal whose impact factor is above the median for their field (i.e. is a respected journal) and which will send their paper out to review for a quality test. Having met that test, such a paper can be accepted on the face of it as being of international standard.
An additional refinement might be to rate papers published in journals in the top quartile (by impact factor) for their fields more highly than those in the 2nd quartile. However, making finer distinctions than this between journals would lead to the same problems as simple, unthinking application of raw journal impact factors. Counting publications that meet these criteria would then provide an indicator of the quantity of international quality research being published by an individual or a Department. Funding agencies can then also determine whether they are getting "bang" for their "scientific buck".'
Professor Paul Fisher
(Chair in Microbiology and
Dep. Exec Dean of Science,
Technology and Engineering,
La Trobe University)