October 10, 2013
An Academic Researcher's View Regarding the use of Journal Impact Factors
Paul Fisher
LaTrobe University
The comment by Professor Warwick Anderson, CEO of the
National Health and Medical Research Council (NHMRC) reflects a valid
realization that journal impact factor is not, in and of itself, a measure of
the quality of each of the papers within it. However, what he actually meant by
this was that the impact of the journal is not a measure of the impact of every
individual paper and that is of course also true. Unfortunately, like so many
before him, Professor Anderson then goes on to equate impact with quality at the
level of the individual paper. This is done, as so often, by a linguistic
gloss-over ‒ you simply substitute one term for
the other ‒ and before you know it you are
measuring quality once again by citation counts, this time of individual papers.
Of course citation counts measure no such thing since, if anything, they measure
the size of the scientific readership, which varies hugely depending on the
topic.
What to do? Well it is true of course that there is no substitute for having
experts in the discipline actually read the papers. However that is an enormous
undertaking and has in fact already been done by the referees of the journals in
which the papers are published. Can one then assume that journals with higher
impact factors must be publishing only better quality papers? No. In fact the
primary hurdle for publishing in the very highest impact journals like
Nature or Science is not the quality of the paper but whether the
topic is considered by the editor to be of broad interest, i.e.
potential high impact. Such journals frequently reject the great majority of
publications submitted to them without ever subjecting them to the quality test
of review by referees.
One, albeit also imperfect, way to resolve the conundrum is to assume that the
authors of a high quality paper may not be able to get it published in
Nature or Science, but they will most of the time be able to find
a journal whose impact factor is above the median for their field (i.e.
is a respected journal) and which will send their paper out to review for a
quality test. Having met that test, such a paper can be accepted on the face of
it as being of international standard.
An additional refinement might be to rate papers published in journals in the
top quartile (by impact factor) for their fields more highly than those in the 2nd
quartile. However, making finer distinctions than this between journals would
lead to the same problems as simple, unthinking application of raw journal
impact factors. Counting publications that meet these criteria would then
provide an indicator of the quantity of international quality research being
published by an individual or a Department. Funding agencies can then also
determine whether they are getting "bang" for their "scientific buck".'
____________________________________
Professor Paul Fisher
(Chair in Microbiology and
Dep. Exec Dean of Science,
Technology and
Engineering,
La Trobe University)