Group of Eight Suggests to DEST and the RQFDAG What is to be Done For and With Assessing Research Impact. (July 5, 2006)

    On June 16 the Branch Manager, Research Systems Branch, Innovation and Research Systems Group, Department of Education, Science and Training sought input from the Group of Eight (and a number of others no doubt) on the development of a methodology for assessing research impact as part of the proposed Research Quality Framework (RQF).

In a carefully crafted eight page submission of July 3 Virginia Walsh, Executive Director of the Go8, replied.

The Go8 document makes interesting reading, especially so when viewed against the Nature "Commentary", Europe pays the price for spending less, by Lord Patten of Barnes (Chris Patten, Chancellor of Oxford and Newcastle Universities) published on June 8.

Lord Patten read Modern History at Oxford, was Conservative MP for Bath from May 1979, until April 1992, and wrote (in 1983) The Tory Case, a study of Conservatism.  During the Thatcher Prime Ministership, he served as Parliamentary Under Secretary of State, Northern Ireland Office and in September 1985 became Minister of State at the Department of Education and Science. A year later he became Minister for Overseas Development at the Foreign & Commonwealth Office.

He was appointed Governor of Hong Kong in April 1992 by Prime Minister John Major, a position he held until 1997 when the Crown Colony was ceded back to China.

Chris Pattern is hardly a one-eyed apologist for science so when in his commentary he argues "Europe's contribution to the global advancement of science and the promotion of learning is in decline; better funding of universities and research institutions is needed to reverse this trend," it could be expected that governments might heed his warning that  "Europe's political leaders [who] bemoan the European Union's growing problem of international competitiveness... have difficulty in relating cause to effect..., I suppose, [it's] an example of their difficulty in grasping the scientific method."

But to return to the Go8 submission on assessing research impact, it opens with the caution in regard to "assessing research impact".

The Go8 supports the Government’s efforts to incorporate the assessment of the social, cultural, environmental and economic impact of publicly funded research in the RQF. This is pioneering work, never attempted elsewhere. If managed well, impact assessment could significantly enhance Australia’s research effort and international reputation. Conversely, if mishandled, the process could undermine the central objective of the RQF: to identify and reward research of the highest quality wherever it occurs.

Lord Patten points out, "The international league table of university performance of London's Times Higher Education Supplement does not reflect perfect methodology (how could it?), but uses various weighted factors covering peer review, recruiter review, percentage of international faculty, percentage of international students, faculty-to-student ratio and research impact measured by citations per faculty member." His use of "research impact" doesn't appear to be quite the same as that of the Department of Education, Science and Training.

In reading the Go8 submission one is led to ask, is the matter of assessing the social, cultural, environmental and economic impact as a separate exercise a sensible undertaking? The Go8, at least to this reader, appears to question it. Some excerpts from their submission:

There should be explicit recognition that it is the research outputs of individual researchers and the research groupings they belong to that are the subject of assessment for both quality and impact.

The indicators should encourage positive behaviours and in particular must not reward the pursuit of short-term outcomes at the expense of long-term research of great value.

Academic quality (measured as part of the quality assessment under the RQF) is often, if not invariably, a sound indicator of the likely broader impact of research.

...the work undertaken by Peter Doherty and Rolf Zinkernagel at the ANU between 1973 and 1975 on how the immune system recognises virus-infected cells... may have received a low or no research impact rating under the model as currently outlined.

The proposed six year window for measuring research impact risks encouraging the pursuit of short-term research at the expense of more ambitious projects... Further, the proposal that institutions may elect not to have the work of some Research Groups assessed for impact on the grounds that it is too early or otherwise inappropriate to do so, risks a large body of the most important research undertaken in Australia being perceived as lacking relevance.

Under such a proposal, it [is] arguable that research such as that undertaken by Peter Doherty at ANU in the 1970s, Barry Marshall and Robin Warren at UWA in the 1980s and Ian Frazer in the early 1990s would have received no impact rating. Such high impact research could be marginalised if the assessment and funding drivers built into the RQF devalue such research.

It is essential that those research groupings doing internationally recognised excellent work, that is not easily assessed by impact measures, are not disadvantaged by the RQF assessment process; such research must not be devalued by the RQF exercise.

Conclusion
The measurement of research impact is challenging. As the results of the RQF trials become available the magnitude of this challenge is becoming apparent. More work needs to be done to ensure that the assessment of research impact is objective, rigorous, relevant to individual fields of study and integrated with the assessment of quality.

And what of Chris Patten? He tells his readers, "I am not myself an academic, nor an administrator of educational or research programmes. Nevertheless, I believe my views reflect a growing opinion in Europe, that much of its higher-education system is in severe difficulties; the research base is threatened; many of the best researchers are being lost; and there is more competition in the knowledge business. There are serious consequences for Europe's future as an economy and a civilization.... [And] what is the problem with investing more in knowledge? The biggest problem by far is simply a lack of money from governments (or other sources), together with inadequate corporate investment in R&D.

    "I do not seek to argue that there is a direct correlation between spending on higher education and economic growth, nor between investment in R&D and successful innovation. Nevertheless, it is surely unarguable that science is a principal determinant of the wealth, creativity and well-being of society. Yesterday's good science — answering the 'whys' — added to today's technology — working out the 'hows' — has provided much of the quality and standard of our lives today. We are now living off what has been achieved in the past by researchers, technologists and entrepreneurs. What will Europe hand to the next generation?"

It's also interesting to note the statistics that Lord Patten quotes:

[S]pending on tertiary education per student (including R&D) is just over US$9,000 for France, just under US$11,000 for Germany and just under US$12,000 for the United Kingdom (OECD 2005 indicators). Austria, Belgium, Denmark, the Netherlands and Sweden do better, yet by comparison, the figure for the United States is heading towards US$26,000. The percentage of gross domestic product (GDP) spent on tertiary education (2002 figures) is 1.1% in France, Germany and the United Kingdom, and 2.6% for the United States (1.2% from public funds and 1.4% from private).

And as seen from the table below of the 23 nations, five showed a decrease in "expenditure per student" while 18 increased their expenditure per student between 1995 -2002. Australia is one of the five that moved into negative territory.

Patten then asks, "How much does all this matter to performance in research, development and innovation? First, it would be more than surprising if the condition of the principal agents for promoting both advanced learning and pioneering research did not have an impact on the broader research scene. Second, underinvestment in research in universities is matched by underinvestment in business. Corporate Europe does much less than corporate America and Japan."

And the non-scientist, non-academician than says, "High-quality, curiosity-driven research can be even more commercially significant than proprietary science — think of lasers and DNA. Knowledge is not simply another commodity in the market place. Yet while encouraging greater cooperation between industry and universities, research must not be skewed almost exclusively to subjects that interest industry rather than the public. The European Research Council is on its way, with the only major matter still to be resolved being the size of its budget."

To waste the money of Australia's citizens and the time and energy of its academicians on an ill conceived Research Quality Framework with or without an impossible effort to assess directly and effectively worthwhile economic impact in the pursuit of micromanaging the tertiary educational/research sector is at best disingenuous.

Were the Coalition Government genuine in its pursuit to improve the nation's research and innovation we would see an appropriate injection of public resources, the institution of significant incentives for the private sector to raise it's input into research and development, and the developing with academicians, public research agencies and the public granting bodies of an outstanding peer review system allowed to operate at arms length from ministerial control.

Returning to the Go8's caveat,  If managed well, impact assessment could significantly enhance Australia’s research effort and international reputation. Conversely, if mishandled, the process could undermine the central objective of the RQF: to identify and reward research of the highest quality wherever it occurs, can anyone seriously believe in the light of how the nation's tertiary institutions have fared at the hands of the Coalition Government over the past decade that research impact assessment as contemplated by that Government, will be "well managed" in the sense of enhancing research and development to the benefit of the commonweal?