Is New Zealand's Performance Based Research Funding (PBRF) Detrimental to Research Quality? (January, 9, 2009)

According to its website:

The primary goal of the Performance-Based Research Fund (PBRF) is to ensure that excellent research in the tertiary education sector is encouraged and rewarded. This entails assessing the research performance of Tertiary Education Organisations (TEOs) and then funding them on the basis of their performance.
Between 2004 and 2007 the PBRF progressively replaced the current EFTS (equivalent full-time student) 'top-up' funding for research. The PBRF model has three elements:

* to reward and encourage the quality of researchers - 60 percent of the fund
* to reflect research degree completions - 25 percent of the fund
* to reflect external research income - 15 percent of fund

The major element, the Quality Evaluation, is held periodically. The first was completed in 2003 and the second, a partial round, was held in 2006. The next full round of the Quality Evaluation will take place in 2012.
The PBRF contributes to the success of the Tertiary Education Strategy, supporting its goals of encouraging an integrated, specialised tertiary education sector, developing Maori and Pacific research capability and improving linkages with relevant communities.

New Zealand's PBRF is patterned on the UK's Research Assessment Exercise. In 2006 it utilised 12 panels involving some 175 peer reviewers.

Yesterday the University of Waikato provided a media release which is reprinted here in full:

Research system 'stifling quality'

8 January 2009

A study of New Zealand economists suggests that the flagship university research quality assessment system isn't yet bringing the expected increase in research quality, and may be weakening incentives in the academic labour market for economists to publish higher quality articles.

Professor John Gibson and Professor John Tressler at the University of Waikato Management School, and Professor David Anderson of Queen's University in Ontario, Canada, compared the ranks and publication records of all New Zealand-based academic economists to see how the Performance Based Research Fund (PBRF) had changed the relationship between research productivity and academic rank. The Tertiary Education Commission funds universities on the basis of their performance.

The first data set was from 1999 before the introduction of the PBRF; the second data set was from 2007 after two of the PBRF research quality assessments, in 2003 and in 2006.

"Assessments like the PBRF only make economic sense if the academics respond to the market signals," says Prof Gibson. "The aim of the PBRF is to raise research quality, so we would expect it to cause academics to concentrate more on quality publications and that should show up in a stronger link between research quality and rank, especially because rank is a good proxy for pay at New Zealand universities."

In fact, says Prof Gibson, contrary to what might be expected from the PBRF, the relative returns to quality seem to have gone down rather than up since its introduction.

"While the total volume of research output increased by one-third, average quality declined according to three of our five quality measures. Overall, the returns to quantity relative to quality increases from 1999 to 2007, particularly in the case of new appointments.

"Our findings suggest that university economists are more likely to gain promotion by increasing their research output rather than raising the quality of their work."

Prof Gibson says if the same findings hold true for other academic disciplines, this will have serious implications for the PBRF. "If individual academics find they gain no direct benefit from improving the quality of their research, then we cannot assume the gains from the PBRF will outweigh the very high direct and indirect costs of such an assessment."

For several years now TFW has been asking what data support the contention that Britain's RAE has improved the quality of university research, keeping in mind that marked increases in resources for research have been provided concurrently.

Considering the findings reported by Waikato University the question arises would overall research quality in Great Britain have been not only as good as but better than with the imposition of the RAE had assessment been left to peer review of grant applications and supplying full funding while providing basic block funding to the institutions housing the researchers?

In the current circumstances mightn't Prime Minister Rudd's call for fiscal restraint by his governmental departments include a critical look at the shallowly conceived  Excellence in Research for Australia?

Ah well, it was just a thought.