Ian Dobson

A New ERA? Or a return to the dark ages?

Detail: Parable of the Blind

Ian R Dobson

Editor

Australian Universities’ Review

The latest issue of Australian Universities’ Review (AUR, vol. 53, no. 1)  includes a paper by Simon Cooper and Anna Poletti of Monash University that examines the Australian Research Council’s (ARC) current exercise of ranking scholarly journals. They say ‘... this process is not only a flawed system of measurement, but more significantly it erodes the very contexts that produce ‘quality’ research’. This paper is a ‘must read’ for everyone, because it highlights yet another time wasting and energy-sapping scheme foisted on the higher education sector by governments and government agencies that ought to think things through a bit more.

Arbitrary assessment of journals

Cooper and Poletti note that ‘the ERA represents a full-scale transformation of Australian universities into a culture of audit.... Instead we suggest the need to return to ‘basics’ and discuss how any comprehensive auditing regime threatens to alter and undermine the capacity for universities to produce innovative research and critical thought’. They go on to point out that any attempt to rank journals is at best arbitrary.

That is the problem in a nutshell! These attempts are arbitrary, illogical, random, unreasoned, unsupported and whimsical, to list just a few of the synonyms in my Thesaurus. Whatever good might come out of other aspects of the Excellence in Research Australia exercise, its system for ranking journals is at best the result of extremely muddled thinking.

Journals have been accorded one of four ranks: A*, A, B and C, comprising 5%, 15%, 30% and 50% of journals respectively. Perhaps the first question ought to be ‘why not A, B, C and D’? What’s wrong with using the normal alphabet? One presumes that we have followed the Poms with the terminology they employed in their Research Assessment Exercise, but why?

Within this schema, AUR is ranked ‘B’. Why B, and not A*, A or C? Well, we don’t know. The ARC won’t tell us. According to their website, ‘a journal’s quality rating represents the overall quality of the journal. This is defined in terms of how it compares with other journals and should not be confused with its relevance or importance to a particular discipline’. What doesn’t seem to be on the website is a definition of ‘overall quality’. Who decided what it is? How did they do so?

AUR is listed in field of research (FoR) 1301 Education Systems (probably 130103 Higher Education). An examination of the list of journals in the 1301 field reveals that it contains 184 journals, seven of which aren’t ranked. Only three journals in this field of research are ranked A* (1.7%), 18 are ranked A (10%), 46 are ranked B (26%) and 110 are ranked C (62%). However, the ARC website says that AUR is not being compared with journals it is like (of which there are but a handful), but rather a Pandora’s Box of all journals from all fields of research.

The FoR a journal is linked to is another mystery. Journals can be linked to up to three research fields, but most are linked only to one. Who picked these, and how were they picked? Other journals in FoR 1301 cover primary, tertiary, vocational and adult education, and a number of other journals appear to be ring-ins. Why is the Australasian Journal of Engineering Education listed in this category, when its European equivalent isn’t there? Come to think of it, if the Australasian Journal of Engineering Education is there, why aren’t Australian journals such as World Transactions on Engineering and Technology Education and the Global Journal in Engineering Education there as well? It’s a mystery!

Among the journals that AUR is arguably similar to are the Journal of Higher Education Policy and Management (ranked B), Higher Education Management and Policy (ranked C), and Higher Education Policy (not listed at all). The first two of these journals are ranked in a completely different FoR, to wit 1303 Specialist Topics in Education, and within that FoR, one can only presume that they are there under category 130304 Educational Administration, Management and Leadership. Apart from the failure to list the journal Higher Education Policy anywhere, why isn’t AUR also classified in this FoR? It is clear that it would sit quite well there. At the same time, why aren’t these other journals also categorised where AUR and the other higher education journals mentioned before are, under FoR 130103 Higher Education? Surely their titles are a dead give away!

Another problem is the assumption of homogeneity, and this is a problem on two fronts. First, some journals are niche journals, and these will be targeted by writers with an interest in that specific area. Such journals have natural constituencies, and therefore in the non-transparent ERA journal ranking system, authors who seek to publish in such areas are likely to rate such journals highly. Why would an author tell anyone that they publish in poor journals?

What about generalist journals, such as AUR? Where is its natural constituency? It doesn’t have one. There must be quite a few journals that suffer under this ranking regime not because someone has said they are a poor journal, but because they weren’t mentioned at all. However, AUR publishes papers across a wide range of higher education areas, and scanning its tables of contents over the past several years demonstrates just how broad its coverage is.

The other area where homogeneity is presumed is in the rankings themselves. By definition, 50 per cent of journals have been ranked ‘C’, and some of these journals will be better than others. But wouldn’t it be better to rank the papers, rather than the journals? Surely some of the papers published in C journals are excellent papers, perhaps written by new researchers, just as some papers published in A* journals written by more experienced writers are dross. Let us hope the ERA people at the ARC never become restaurant reviewers, because following their current methodology for ranking journals, they would probably rank restaurants according to the photographs on the menu, rather than by assessing either the restaurant’s food or the service.

It’s all fun and games until someone gets hurts

This would all be humorous if it weren’t so damaging to scholarship. Readers will have perhaps seen recent press reports about the demise of an Australian journal called People and Place, at the end of its 19th year. This journal, published via Monash’s Centre for Population and Urban Research, punched well above its weight right from the start, and has been responsible for many changes in Australian social policy.

It was ranked ‘B’ for the initial ERA journal ranking exercise, but it was subsequently demoted to ‘C’, something that has sounded its death knell. It is apparent that Monash University no longer values a journal that has been responsible for so much social policy change in Australia. Part of the problem for People and Place is that it is a local and national journal, rather than an international journal. More government- sponsored cultural cringe!

Which overseas journals could be expected to publish papers on topics such as Austudy and Youth Allowance, overseas students’ English language standards, and the myths behind the value of education exports or the effect of equity policy? People and Place was a major player in these areas, as well as in welfare policy, health policy and immigration.

The editorial from the last issue of People and Place, written by editors Katharine Betts and Bob Birrell makes for salutary reading. If it isn’t bad enough having ministers, government departments and their agencies producing dud policy, it’s a pity that our universities and their mouth pieces don’t have the intestinal fortitude to stand up defend the staff who work at their institutions. What have we heard from Universities’ Australia or university blocs such as the Go8? Perhaps their silence reflects the fact that they see the ERA as a power management tool which will allow them to take care of some difficult cases.

In this zero-sum game, by definition half of the listed scholarly journals have to be ranked C. If a writer has a paper published in a C Journal, it will actually diminish the chance of their university to rank highly. Therefore, more journals than People and Place are likely to close their doors, because which Australian authors will want to be published in a C journal? Of course, if we follow this process through to its logical conclusion, there will eventually be only one journal for each field of research, as the ‘worst’ in each category will no longer be good enough for ERA metrics. I don’t suppose it will go that far, but the whole logic behind this smacks of former US President Bush’s desire for all US schools to be above average.

The known unknowns of journal ranking

The overall problem with the ERA journal ranking exercise isn’t so much what we know about it, but what we don’t know about it. How was the ranking for each journal arrived at? Who did the ranking? What is ‘overall quality’, as NOT defined on the ARC website? Why compare journals in one discipline with journals in another?

Given the unending government rhetoric about transparency and accountability, why are these attributes always absent when the government (or one of its agencies) does things?

Clear answers are needed on all these questions. The so-called peak bodies and their role in ranking journals also needs to be looked at. The process doesn’t seem to have included any requirement for the disclosure of potential conflicts of interest. How many of the peak bodies are giving their own preferred journals a helpful plug? How much self-serving is there in this exercise? Again, we don’t know, and this is the problem.

Perhaps the final insult with ERA is some of its use of English. What we are going through at the moment is called ‘the ERA 2012 Ranked Outlets Consultation’. According to the Oxford Dictionary of English (2005), an outlet is ‘a pipe or hole through which water or gas may escape’. It is also ‘a point from which goods are sold or distributed’. Quite what an outlet (or an ‘oultet’ as it is written in one place on the ARC website) is in the ERA context, adds further to the mystery of this whole exercise.

In the future, the right thing would be for everyone to remember the Minister/s and the ARC leadership that we should blame for the ERA journal ranking exercise, but I don’t expect these things to be recorded in Derryn Hinch’s ‘Shame File’. Universities and individuals have to move on; they can’t dwell on the past or even the present. In any case, they don’t have access to the endless funds that government departments and agencies seem to have to produce poor policy.