Beware the Impact Factor, My Son! The Jaws That Bite, The Claws That Catch! (February 18, 2013)

With apologies to Lewis Carroll, but the editorial in volume 12, February 2013 of Nature Materials (doi:10.1038/nmat3566) seems as wise to heed as the warning issued by the author of Through the Looking-Glass and What Alice Found There.

Here we reprint the final two paragraphs of the editorial:

[T]he impact factor is an appropriate measure of journal quality according to citations. And it is also beyond question that the impact factor does not generally correlate to the performance of individual researchers or to citations to individual papers^{2, 8, 9.} As with any statistical measure, it is unsafe to use it as a proxy for an unrepresentative subset of the original sample. It would thus be unwise, for instance, to rate scientists on the basis of the total number of papers weighted according to the impact factor of the journal where they have been published. A simple exercise proves the point: pick a few scientists and rank the papers they published five years ago in decreasing order of citations alongside the impact factor of the corresponding journal in that year. The odds are that, if there is any correlation at all, this is weak or the outliers are plentiful.

As Fig. 1 shows, half of the papers published by Nature Materials in the past five years have received more citations than at least half the papers published in most other journals (that is, any journal with a lower impact factor). The median and its predictor the impact factor are therefore quality signals that are valid for comparisons between journals publishing on similar scientific topics. Yet beware of those who use them instead of article-level metrics¹⁰ when assessing a small subgroup of papers or authors. Impact factors should have no place in grant-giving, tenure or appointment committees.

_______________________________________________________________

References

  2. Cameron, B. D. Portal Libraries Acad. 5, 105–125 (2005).

  8. Rostami-Hodjegan, A. & Tucker, G. T. Br. J. Clin. Pharmacol. 51, 111–117 (2001).

  9. Epstein, R. J. J. Invest. Med. 52, 531–536 (2004).

10.. Nature Mater. 11, 907 (2012).