ERA Journal Rankings Still in the News, Still Controversial. (May 14, 2011)

Jennifer Howard writing in the May 8, 2011 issue of The Chronicle of Higher Education writes: "In many fields, especially the sciences, the journal rankings have not been a big source of complaint. In the humanities and social sciences, though, the culture of audit has prompted anxiety, at least in some disciplines." Of course, as she notes, journal rankings aren't unique to Australia, but interestingly the US doesn't use them although, of course, U.S.-based journals figure prominently in the listings.

Indicating a degree of arbitrariness Ms Howard reports the example of the quarterly journal Biography. Craig Howes, a professor of English at the University of Hawaii-Manoa, is an editor. In the ERA rankings' first iteration, Biography was assigned a B. He and his fellow editors rallied Australian colleagues to petition the council for a revision. In the second version, the journal scored an A*. But in February 2010 the final version of the lists dropped Biography to a C. Professor Howes told Ms Howard: "We got dropped below a journal in our field that hasn't published an issue in two years," even though  Biography had been publishing quarterly, including a special issue in 2010 that was guest-edited by Sidonie Smith, who that year was the president of the Modern Language Association. "[One thing] I'm sure of," Mr. Howes wrote, "is that no one who knows anything about the journals in the field of life writing had anything to do with our current ranking, all institutional claims about panels of experts to the contrary."

While the CEO of the Australian Research Council, Margaret Sheil, contends "I think we got most of it right. But there's 200-odd [of the 22,000 journals] that we didn't necessarily get right in the first round. The numbers are small, but the noise is loud."

Nevertheless, according to Ms Howard: "She does acknowledge that using an A-B-C scheme was probably a mistake, and that it may be dropped in future rankings. 'The list that we had was the list for 2010, and whatever we do moving forward, it won't be the same,' she says."

That comment doesn't really instil great confidence in either the approach or its perpetrators.

Anna Poletti, a lecturer at Monash University has published in Biography and was offered a co-editorship of an issue.  She saw it at first as "an amazing career opportunity for me, but the way the rankings were shaping up, it looked like a problem rather than the fantastic opportunity it was." She told Ms Howard that the ERA evaluations have created particular anxiety for early-career researchers like herself: "Certainly for us it's a very confusing and worrisome process. And that's partly because our senior colleagues, whom we have traditionally been able to look to for advice, are unsure about the ramifications of the ERA."

Ms Howard goes on to report: "When  Ms Poletti and other life-writing scholars asked the research council to explain the journal's C grade, they were told that the council doesn't discuss individual rankings. 'I just thought that was astounding,' she says."

And then we have the quote of the month: In an interview, Ms. Poletti wonders whether the whole research-evaluation exercise, with its emphasis on factors such as prestigious editorial boards, isn't missing the essential point. "There are a lot of scholars in Australia saying, 'Why this emphasis on where work appears rather than the work itself?'"

And mightn't we wonder what the peer-review system of the ARC and NHMRC has been doing all this time.  Yes, it needs improvement. Layering a dubious and retrospective evaluation over it is not an improvement, if improving the quality of Australia's research is the goal.

And then we have the ARC's CEO saying with consummate naïveté that she is concerned about the unintended consequences of the research-evaluation system, particularly how it is being used as a personnel-management tool by universities. The council "didn't want the federal government to be doing evaluations of individuals," she says. But, she adds, "I don't know what else I can do other than say, 'Don't do this.' That wasn't what we designed it for." As one reader of The Chronicle commented: "It shows a total lack of  [a] basic [understanding of the] sociology of behaviour and amounts to saying: 'Well, I put a gun in the middle of the public place' but INSISTS that 'nobody should use it to do harm!'. With a CEO like that in a major Research organization, no wonder that the research system is going the wrong way."

But it's appropriate to let Dr Polrtti have the last word: "The desire to evaluate quality is not a desire that is antithetical to scholarship, because scholarship is about good-quality work. It's whether or not a kind of bureaucratic mode of evaluating quality actually allows the real quality to become visible."