The Performance of Scientific Peer Reviewers. (December 13, 2010)

Michael Callaham, MD and Charles McCulloch, PhD in the Department of Epidemiology and Biostatistics, University of California, San Francisco, recently published a paper they entitled Longitudinal Trends in the Performance of Scientific Peer Reviewers in which they "characterize changes in review quality by individual peer reviewers over time".

They analysed 14,808 reviews performed by 1,499 reviewers the quality of which were rated by 84 editors of the Annals of Emergency Medicine over a 14-year period. 92% of reviewers demonstrated very slow but steady deterioration in their scores. The rate of deterioration was unrelated to duration of reviewing but moderately correlated with the "mean reviewer quality score (R=0.52)". They also found that the "mean score of each reviewer's first 4 reviews predicted subsequent performance with a sensitivity of 75% and specificity of 47%", however, they noted that over time overall scoring of the reviewing group remained constant indicating that newly recruited reviewers initially had higher mean quality scores than those they replaced.

The authors overall conclusion?

This study, one of few tracking expert performance longitudinally, demonstrates that most journal peer reviewers received lower quality scores for article assessment over the years. This could be due to deteriorating performance (caused by either cognitive changes or competing priorities) or, to a partial degree, escalating expectations; other explanations were ruled out. This makes monitoring reviewer quality even more crucial to maintain the mission of scientific journals.

In addition it is not an unheard of practice for senior researchers to allocate papers given for them to review, to individuals within their groups who may be less qualified.

And as a member of several scientific editorial boards pointed out to TFW: if the trends identified in the article are repeated in other journals then a paper that has a predominance of senior reviewers will not be, statistically speaking, as satisfactorily reviewed as a paper where younger reviewers predominate.