Opinion- 15 September 2005 |
This article is made available on TFW with the permission of the Australian Mathematical Society Gazette |
On
9th September the Australian Government released its 'Preferred Model' for the
Research Quality Framework, or RQF. The discussion paper can be downloaded from
the web page containing the Minister's announcement:
In a nutshell,
the Preferred Model would lead to a methodology for research assessment that has
many similarities to the UK Research Assessment Exercise, or RAE. The latter
has been in operation since 1986; the sixth RAE will take place during 2008.
The development of an RQF was signalled by the Prime Minister back in May 2004,
in connection with the second tier of the government's `Backing Australia's
Ability' program. In principle the RQF applies to all federally funded research
agencies, and in theory it could be used to assess the work of CSIRO or ANSTO,
for example. However, no attempt is made in the discussion paper to connect the
Preferred Model to either of these bodies. However, the linkage to universities
is emphasised throughout the paper. The Minister has even implied that the RQF
might be play a role in the review process for regular ARC grants; we shall
return to this topic towards the end of this article.
The RQF would use discipline panels, each with 12 to 15 members, to assess
research performance. Neither the number of panels nor their discipline areas
has been given at this stage, so we can only guess at the level of
specialisation. For the 2008 UK RAE there will be a single `mathematics' panel
(Panel F), broken up into sub-panels representing, respectively, Pure
Mathematics, Applied Mathematics, Statistics & Operational Research, and
Computer Science & Informatics. It is unclear whether the the RQF panel for
mathematics and statistics would reflect this level of generality, or be still
more diverse. In the smaller scientific environment of Australia, the panel is
unlikely to be more narrow.
Of course, the RAE sub-panels have specialist representation in their respective
discipline areas. The sub-panels do much of their work autonomously, and UK
mathematical scientists tend to feel that their research is being assessed by
people who are expert in fields that are not especially distant from their own.
It is not clear that this would be the case in Australia; no sub-panel structure
is foreshadowed in the government's Preferred Model for the RQF.
Similarities to the UK RAE would include the requirement that each person
considered by the RQF have produced at least four research outputs (usually
research papers and books, in the mathematical sciences) over the assessment
period, and submit no more than four research outputs from that time-frame. The
assessment period would generally be six or seven years long. Some flexibility
in the number of research outputs would be allowed, for example in the case of
early-career researchers.
One major difference from the UK model would be in terms of the involvement of
`at least 50 per cent international experts, primarily international experts who
are resident overseas as well as some international experts resident in
Australia.' In another departure, apparently reflecting the government's desire
to ensure specific practical outcomes from research funding, each panel would
include `at least two experts, as deemed appropriate to the panel structure, who
could represent the views of users of the research under assessment.'
Thus, at least one quarter, and probably close to one half, of all panel members
would be from abroad, and two panel members would vet the practical credentials
of research. The latter aspect is troubling, particularly since a great deal of
the research conducted in Australian universities is of a basic or strategic
nature.
The likely backgrounds of the remaining panel members are not clear from the
Preferred Model paper. However, panel members would be chosen in a way which,
in principle, is similar to that currently used to select members of the ARC
College of Experts -- through `targeted invitations to relevant organisations
and representative groups, including universities, the Learned Academies, PFRAs,
business and community groups, etc.' It is quite unclear how this process could
be extended to international panelists; only with difficulty, one suspects. The
mathematical sciences community has been quite unhappy with its representation
on the ARC College of Experts, and so we should pay particular attention to this
aspect of the process.
The matter of `metrics,' including bibliometric information such as citation
data, is still on the table. However, an attempt has been made to relate
metrics to subject areas. For example, it is stated in the Preferred Model
paper that the RQF `will include metrics where appropriate to discipline areas,
such as bibliometrics, industry research grants and short supporting statements
of research impact by relevant end-users.'
In the mathematical sciences, any meaningful use of bibliometric data would have
to involve a time-scale much longer than the six or seven years used for RQF
reporting. Therefore, it would be particularly difficult to combine
bibliometric data with reporting data in a meaningful way. It is not clear that
these issues are appreciated by the committee that wrote the Preferred Model
paper.
One of the major criticisms of the UK RAE has been the amount of academic-staff
time it absorbs. The RQF would be very similar in this respect. Australian
universities would be required to `nominate research groupings, provide
researchers' evidence portfolios for inclusion in research groupings packages,
[and] provide context statements which will include supporting evidence, e.g.
bibliometrics, measures of peer esteem, etc, for research groupings.' The
British International Studies Association, in a document submitted to the joint
UK funding bodies, noted that `The cost of the RAE is not negligible, but the
bulk of the cost is borne by the academics that carry out the exercise.'
In addition to the expenses incurred by universities in operating the RQF, there
would be costs that would have to be borne by the government. In an environment
where the Treasury's economic models cause it to question the need to spend
money on peer review of grant proposals, would the RQF be paid for by new funds,
or would the expense be taken out of other projects, for example the money
already set aside to support research? Will universities be compensated for the
real cost of their work for the RQF?
The RQF would also inherit other problems directly from its UK ancestor. Among
these would be problems assessing multi-disciplinary research. In fact, the
Preferred Model paper acknowledges that this is a potential weakness of the
proposal: `Assessing cross-disciplinary work may be difficult.' In a press
release issued on 9th September, the Council for the Humanities, Arts and Social
Sciences (CHASS) took the government to task on this issue, noting that the
committee that produced the Preferred Model `has not found an adequate way to
measure the increasingly-significant area of cross-disciplinary research.'
These remarks echo sentiments expressed five years ago by the UK Arts and
Humanities Research Board, which observed that the RAE `militates against
interdisciplinarity.'
CHASS is also concerned that the RQF would place `undue emphasis on the
`quality' of research rather than the `impact',' and comments that `the model
proposed by the Advisory Group does need further work. The approach it takes is
too conservative, on nearly every substantive issue.' Nevertheless, CHASS
pronounced itself pleased that the government has made `a genuine attempt to
work across the [university] sector and to include different kinds of measures
for different research fields.' At the time of writing, the Australian Academy
of Science had not produced a response to the Preferred Model.
The International Review of Mathematics (IRM), which reported on mathematics
research in the UK to the UK Physical Sciences Research Council and to the
Council for the Mathematical Sciences, discussed the UK RAE at some length.
Although the IRM found that the RAE had `enhanced the importance given to
research in departmental and university policies,' it also pointed to a number
of causes for concern. These included the heavy administrative burden placed on
staff preparing for an RAE (see above), discouragement of risk-taking in
research, and the encouragement of `incremental research' (that is, the slicing
of research results into relatively small, publishable parts so that the
researcher would have enough papers to submit to the RAE panel). The IRM
observed that that RAE led to personnel decisions that created tensions within
departments, and which ran counter to collegiality. In some instances these
effects of the RAE reduced research productivity.
Most importantly, the IRM found that the RAE did not adequately support isolated
pockets of excellence, that it produced a concentration of mathematics research
resources into a small number of departments, and that it led those strong
departments to concentrate on a relatively small number of fields, where their
academic staff had specialised expertise. In particular, the IRM lamented the
decline in UK expertise in areas such as modern analysis, related areas of PDEs,
and statistics. Some of the pockets of strength noted by the IRM were quite
fragile, since they existed only in one or two universities in the country, and
were based on the activities of only a few people. Commented the IRM, `The UK
cannot afford to have its high quality research concentrated in too few leading
departments that are competitive at an international level.'
The tendency of the RAE to result in narrow and fragile concentrations of
research resources is felt right across the spectrum of fields in which research
is conducted in UK universities. For example, in a very different research
field, the British Society of Criminology reported on the impact of the RAE on
research criminologists, and expressed three major concerns: `The first is the
cost of the RAE; the second is its centralising tendency; the third is its
failure to support small groups of excellent researchers working in larger
departments.'
These problems are likely to be significantly more serious in the Australian
context than they are in the UK. In a country which has only a modest
scientific and scholarly culture, and only a small number of universities that
can field mathematics departments of very high calibre, any move which reduces
diversity and concentrates strength can create potentially serious problems. It
could lead to an unstable national research environment, where the loss of one
or two key people could have a calamitous effect.
In terms of hiring mathematical scientists, Australia has an almost closed cycle
of employment. Our special circumstances, including the distance from here to
Europe or North America, mean that we seldom attract strong mathematicians from
abroad to make their careers in Australia. The UK can, and does, hire
mathematicians from the Continent (e.g Germany, Scandinavia and Eastern Europe),
but the difficulties experienced in the UK, arising from a reduction in
diversity attributable at least in part to the RAE, are nevertheless
pronounced.
The intense competition engendered in Australian universities, by government
policies over the last decade, has already greatly reduced inter-university
cooperation in research and teaching. The new contests that would be produced
by an RQF, when magnified by the small size of the Australian research
enterprise, might carry competition to a still more detrimental level. In the
UK the Royal Society of Edinburgh has noted that, `in a small country like
Scotland,' the fact that `there has not been sufficient stimulus to
inter-institution collaboration' resulting from the RAE `has been counter
productive.'
On the same theme, the UK's National Academies Policy Advisory Group has
remarked that, while the RAE encourages inter-university competition, it fails
to promote the benefits of collaboration and networking. NAPAG has also noted
that the substantial funding and performance pressures flowing from the RAE make
it relatively difficult for early-career researchers to establish themselves.
Attracting, and finding career paths for, young Australian mathematical
scientists is one of the most serious challenges we face.
The Australian government appears to have a hidden, although somewhat distant,
agenda for the RQF. In an interview reported in the Financial Review on 10th
September, the Minister remarked: `It is my clear intention that we will also be
building a model to apply the RQF, or a variation of it, to the ARC competitive
funding.' An earlier comment by Dr Nelson, in a press release, had been less
specific: `My Department will conduct a process...to develop the way in which
the outcomes of the RQF will impact on the funding distributed by the research
councils.'
Understandably, the Minister's remarks drew immediate fire from university
researchers. A linkage between the RQF and specific research grants would lead
universities to tie RQF results, and both contested and block-grant research
funding, very closely to individual scientists and scholars. The linkage would
also greatly reduce the `diversity of perception' in judging research, by
narrowing the range of outlooks of the people who assess Australian research.
Diversity is crucial to making wise decisions about research funding. More
insidiously, Dr Nelson might be considering using the RQF to determine those
universities or departments that are suitable for research-grant funding, and
might refuse access to the grant competition to those groups that do not score
sufficiently highly in RQF assessments.
To be fair to the Minister, it should be noted that he qualified his remarks
with the observation that his plans `probably won't eventuate in my time in the
portfolio.' So, we probably have a while in which to develop our arguments
against his proposal.
The Minister's comments pull into sharp focus the intrinsic redundancy of a
research assessment program which both assesses research proposals by peer
review (for example, reviews of ARC and NH&MRC grants) and subsequently, at
great cost in time and money, dissects the entrails of the research to
determine, in effect, whether the original decisions were correct. Bob May, in
his President's Anniversary Address to the Royal Society in 2003, commented that
he was `unaware of any other country [besides the UK] which parallels the usual
peer-reviewed competitive process for specific research projects with a second,
and hugely elaborate, process of frequent review and grading of essentially
every department in every university in order separately to allocate
infrastructure costs.' The Preferred Model paper argues that Australia should
join the UK in this category.
Under the Preferred Model the first RQF would be implemented in 2007, from which
point it would occur on a cycle of approximately six years, except that there
would be a gap of only three years between the first and second RQFs. The first
RQF would assess work in calendar years 2001 to 2006, and would judge
institutions on the basis of those researchers in their employ on 1st January
2007.
In the meantime, comments on the `Preferred Model' paper should be submitted to
the government by 4th October 2005. They can be sent to the email address, rqf@dest.gov.au.
I suggest that the Mathematical and Statistical Societies quickly develop
responses to the paper. Recommendations and suggestions might include one or
more of the following:
Rules and guidelines for operation of the RQF must safeguard career paths for early-career researchers, and in particular must ensure that these people are encouraged, and not penalised, by the research assessment process;
The RQF must have the ability (which it would not enjoy under the Preferred Model) to assess effectively cross-disciplinary work, for example mathematical work in finance, biology and bioinformatics;
The representation of fields by panels and sub-panels in the RQF should assure scientists and scholars that their work is being carefully and knowledgeably assessed by their peers;
Measures should be put in place to ensure that the RQF does not make the Australian research enterprise even more fragile than it already is, through concentrating excellence in too few institutions and too few fields, and failing to support pockets of excellence in larger groups;
In fields where quality and impact are not closely linked to short-term practical outcomes, this fact should be fully recognised by the RQF assessment process;
When metrics and other sources of information are used as part of the assessment process, care must be taken to incorporate them in an appropriate manner (note, for example, that appropriate time-scales for bibliometric data in mathematics are usually much longer than the formal `assessment period' foreshadowed in the Preferred Model);
The potential advantages of cooperation by institutions in research should be recognised by the RQF, and cooperation should not be sacrificed in favour of competition; and
The work-load on universities for implementing the RQF should not be any greater than that for current research assessment procedures, without adequate financial support through new funding.
Comments and
other assistance from Tony Guttmann, Alex Reisner and Jan Thomas were very
helpful in the preparation of this article.
Professor Peter Hall is at the Mathematical Sciences Institute, The Australian National University, Canberra.