Opinion

 

September 15, 2005

 

Peter Hall Discusses The 'Preferred Model' for Australia's Research Quality Framework

 

This article is made available on TFW with the permission of the Australian Mathematical Society Gazette

On 9th September the Australian Government released its 'Preferred Model' for the Research Quality Framework, or RQF. The discussion paper can be downloaded from the web page containing the Minister's announcement:
 

http://www.dest.gov.au/sectors/research_sector/policies_issues_reviews/key_issues/research_quality_framework/rqf_preferred_model.htm

 

In a nutshell, the Preferred Model would lead to a methodology for research assessment that has many similarities to the UK Research Assessment Exercise, or RAE.  The latter has been in operation since 1986; the sixth RAE will take place during 2008. 

The development of an RQF was signalled by the Prime Minister back in May 2004, in connection with the second tier of the government's  `Backing Australia's Ability' program.  In principle the RQF applies to all federally funded research agencies, and in theory it could be used to assess the work of CSIRO or ANSTO, for example.  However, no attempt is made in the discussion paper to connect the Preferred Model to either of these bodies.  However, the linkage to universities is emphasised throughout the paper.  The Minister has even implied that the RQF might be play a role in the review process for regular ARC grants; we shall return to this topic towards the end of this article. 

The RQF would use discipline panels, each with 12 to 15 members, to assess research performance.  Neither the number of panels nor their discipline areas has been given at this stage, so we can only guess at the level of specialisation.  For the 2008 UK RAE there will be a single `mathematics' panel (Panel F), broken up into sub-panels representing, respectively, Pure Mathematics, Applied Mathematics, Statistics & Operational Research, and Computer Science & Informatics.  It is unclear whether the the RQF panel for mathematics and statistics would reflect this level of generality, or be still more diverse.  In the smaller scientific environment of Australia, the panel is unlikely to be more narrow. 

Of course, the RAE sub-panels have specialist representation in their respective discipline areas.  The sub-panels do much of their work autonomously, and UK mathematical scientists tend to feel that their research is being assessed by people who are expert in fields that are not especially distant from their own.  It is not clear that this would be the case in Australia; no sub-panel structure is foreshadowed in the government's Preferred Model for the RQF. 

Similarities to the UK RAE would include the requirement that each person considered by the RQF have produced at least four research outputs (usually research papers and books, in the mathematical sciences) over the assessment period, and submit no more than four research outputs from that time-frame.  The assessment period would generally be six or seven years long.  Some flexibility in the number of research outputs would be allowed, for example in the case of early-career researchers.  

One major difference from the UK model would be in terms of the involvement of `at least 50 per cent international experts, primarily international experts who are resident overseas as well as some international experts resident in Australia.'  In another departure, apparently reflecting the government's desire to ensure specific practical outcomes from research funding, each panel would include `at least two experts, as deemed appropriate to the panel structure, who could represent the views of users of the research under assessment.' 

Thus, at least one quarter, and probably close to one half, of all panel members would be from abroad, and two panel members would vet the practical credentials of research.  The latter aspect is troubling, particularly since a great deal of the research conducted in Australian universities is of a basic or strategic nature. 

The likely backgrounds of the remaining panel members are not clear from the Preferred Model paper.  However, panel members would be chosen in a way which, in principle, is similar to that currently used to select members of the ARC College of Experts -- through `targeted invitations to relevant organisations and representative groups, including universities, the Learned Academies, PFRAs, business and community groups, etc.'  It is quite unclear how this process could be extended to international panelists; only with difficulty, one suspects.  The mathematical sciences community has been quite unhappy with its representation on the ARC College of Experts, and so we should pay particular attention to this aspect of the process. 

The matter of `metrics,' including bibliometric information such as citation data, is still on the table.  However, an attempt has been made to relate metrics to subject areas.  For example, it is stated in the Preferred Model paper that the RQF `will include metrics where appropriate to discipline areas, such as bibliometrics, industry research grants and short supporting statements of research impact by relevant end-users.' 

In the mathematical sciences, any meaningful use of bibliometric data would have to involve a time-scale much longer than the six or seven years used for RQF reporting.  Therefore, it would be particularly difficult to combine bibliometric data with reporting data in a meaningful way.  It is not clear that these issues are appreciated by the committee that wrote the Preferred Model paper. 

One of the major criticisms of the UK RAE has been the amount of academic-staff time it absorbs.  The RQF would be very similar in this respect.  Australian universities would be required to `nominate research groupings, provide researchers' evidence portfolios for inclusion in research groupings packages, [and] provide context statements which will include supporting evidence, e.g. bibliometrics, measures of peer esteem, etc, for research groupings.'  The British International Studies Association, in a document submitted to the joint UK funding bodies, noted that `The cost of the RAE is not negligible, but the bulk of the cost is borne by the academics that carry out the exercise.'

In addition to the expenses incurred by universities in operating the RQF, there would be costs that would have to be borne by the government.  In an environment where the Treasury's economic models cause it to question the need to spend money on peer review of grant proposals, would the RQF be paid for by new funds, or would the expense be taken out of other projects, for example the money already set aside to support research?  Will universities be compensated for the real cost of their work for the RQF? 

The RQF would also inherit other problems directly from its UK ancestor.  Among these would be problems assessing multi-disciplinary research.  In fact, the Preferred Model paper acknowledges that this is a potential weakness of the proposal: `Assessing cross-disciplinary work may be difficult.'  In a press release issued on 9th September, the Council for the Humanities, Arts and Social Sciences (CHASS) took the government to task on this issue, noting that the committee that produced the Preferred Model `has not found an adequate way to measure the increasingly-significant area of cross-disciplinary research.'  These remarks echo sentiments expressed five years ago by the UK Arts and Humanities Research Board, which observed that the RAE `militates against interdisciplinarity.' 

CHASS is also concerned that the RQF would place `undue emphasis on the `quality' of research rather than the `impact',' and comments that `the model proposed by the Advisory Group does need further work.  The approach it takes is too conservative, on nearly every substantive issue.'  Nevertheless, CHASS pronounced itself pleased that the government has made `a genuine attempt to work across the [university] sector and to include different kinds of measures for different research fields.'  At the time of writing, the Australian Academy of Science had not produced a response to the Preferred Model. 

The International Review of Mathematics (IRM), which reported on mathematics research in the UK to the UK Physical Sciences Research Council and to the Council for the Mathematical Sciences, discussed the UK RAE at some length.  Although the IRM found that the RAE had `enhanced the importance given to research in departmental and university policies,' it also pointed to a number of causes for concern.  These included the heavy administrative burden placed on staff preparing for an RAE (see above), discouragement of risk-taking in research, and the encouragement of `incremental research' (that is, the slicing of research results into relatively small, publishable parts so that the researcher would have enough papers to submit to the RAE panel).  The IRM observed that that RAE led to personnel decisions that created tensions within departments, and which ran counter to collegiality.  In some instances these effects of the RAE reduced research productivity.  

Most importantly, the IRM found that the RAE did not adequately support isolated pockets of excellence, that it produced a concentration of mathematics research resources into a small number of departments, and that it led those strong departments to concentrate on a relatively small number of fields, where their academic staff had specialised expertise.  In particular, the IRM lamented the decline in UK expertise in areas such as modern analysis, related areas of PDEs, and statistics.  Some of the pockets of strength noted by the IRM were quite fragile, since they existed only in one or two universities in the country, and were based on the activities of only a few people.  Commented the IRM, `The UK cannot afford to have its high quality research concentrated in too few leading departments that are competitive at an international level.' 

The tendency of the RAE to result in narrow and fragile concentrations of research resources is felt right across the spectrum of fields in which research is conducted in UK universities.  For example, in a very different research field, the British Society of Criminology reported on the impact of the RAE on research criminologists, and expressed three major concerns: `The first is the cost of the RAE; the second is its centralising tendency; the third is its failure to support small groups of excellent researchers working in larger departments.' 

These problems are likely to be significantly more serious in the Australian context than they are in the UK.  In a country which has only a modest scientific and scholarly culture, and only a small number of universities that can field mathematics departments of very high calibre, any move which reduces diversity and concentrates strength can create potentially serious problems.  It could lead to an unstable national research environment, where the loss of one or two key people could have a calamitous effect. 

In terms of hiring mathematical scientists, Australia has an almost closed cycle of employment.  Our special circumstances, including the distance from here to Europe or North America, mean that we seldom attract strong mathematicians from abroad to make their careers in Australia.  The UK can, and does, hire mathematicians from the Continent (e.g Germany, Scandinavia and Eastern Europe), but the difficulties experienced in the UK, arising from a reduction in diversity attributable at least in part to the RAE, are nevertheless pronounced. 

The intense competition engendered in Australian universities, by government policies over the last decade, has already greatly reduced inter-university cooperation in research and teaching.  The new contests that would be produced by an RQF, when magnified by the small size of the Australian research enterprise, might carry competition to a still more detrimental level.  In the UK the Royal Society of Edinburgh has noted that, `in a small country like Scotland,' the fact that `there has not been sufficient stimulus to inter-institution collaboration' resulting from the RAE `has been counter productive.' 

On the same theme, the UK's National Academies Policy Advisory Group has remarked that, while the RAE encourages inter-university competition, it fails to promote the benefits of collaboration and networking.  NAPAG has also noted that the substantial funding and performance pressures flowing from the RAE make it relatively difficult for early-career researchers to establish themselves.  Attracting, and finding career paths for, young Australian mathematical scientists is one of the most serious challenges we face. 

The Australian government appears to have a hidden, although somewhat distant, agenda for the RQF.  In an interview reported in the Financial Review on 10th September, the Minister remarked: `It is my clear intention that we will also be building a model to apply the RQF, or a variation of it, to the ARC competitive funding.'  An earlier comment by Dr Nelson, in a press release, had been less specific: `My Department will conduct a process...to develop the way in which the outcomes of the RQF will impact on the funding distributed by the research councils.'

Understandably, the Minister's remarks drew immediate fire from university researchers.  A linkage between the RQF and specific research grants would lead universities to tie RQF results, and both contested and block-grant research funding, very closely to individual scientists and scholars.  The linkage would also greatly reduce the `diversity of perception' in judging research, by narrowing the range of outlooks of the people who assess Australian research.  Diversity is crucial to making wise decisions about research funding.  More insidiously, Dr Nelson might be considering using the RQF to determine those universities or departments that are suitable for research-grant funding, and might refuse access to the grant competition to those groups that do not score sufficiently highly in RQF assessments. 

To be fair to the Minister, it should be noted that he qualified his remarks with the observation that his plans `probably won't eventuate in my time in the portfolio.'  So, we probably have a while in which to develop our arguments against his proposal. 

The Minister's comments pull into sharp focus the intrinsic redundancy of a research assessment program which both assesses research proposals by peer review (for example, reviews of ARC and NH&MRC grants) and subsequently, at great cost in time and money, dissects the entrails of the research to determine, in effect, whether the original decisions were correct.  Bob May, in his President's Anniversary Address to the Royal Society in 2003, commented that he was `unaware of any other country [besides the UK] which parallels the usual peer-reviewed competitive process for specific research projects with a second, and hugely elaborate, process of frequent review and grading of essentially every department in every university in order separately to allocate infrastructure costs.'  The Preferred Model paper argues that Australia should join the UK in this category. 

Under the Preferred Model the first RQF would be implemented in 2007, from which point it would occur on a cycle of approximately six years, except that there would be a gap of only three years between the first and second RQFs.  The first RQF would assess work in calendar years 2001 to 2006, and would judge institutions on the basis of those researchers in their employ on 1st January 2007. 

In the meantime, comments on the `Preferred Model' paper should be submitted to the government by 4th October 2005.  They can be sent to the email address, rqf@dest.gov.au.  I suggest that the Mathematical and Statistical Societies quickly develop responses to the paper.  Recommendations and suggestions might include one or more of the following:

Comments and other assistance from Tony Guttmann, Alex Reisner and Jan Thomas were very helpful in the preparation of this article.
 

Professor Peter Hall is at the Mathematical Sciences Institute, The Australian National University, Canberra.