Labor Promises to Reinvent the Government's Research Quality Framework Proposal,  If It Wins Government. (February 21, 2007)

    When Federal Labor Senator Kim Carr, shadow minister for Innovation, Industry, Science & Research addressed the forum on An Investment In Australia's Future: Why the mathematical sciences matter on February 7 he told his audience that were Labor to ascend to government it would replace the government's suggested Research Quality Framework. He ridiculed the government's unworkable proposal to include an assessment of the impact of each research program on national productivity, and was damning of the government's allocation of some $85M to fund an RQF that would be allocating some $500M. He did, however, leave the distinct impression that Labor would introduce some form of RQF rather than scrapping a concept that no one has demonstrated to have merit.

Now having sat through Senate Estimates, he has told The Australian's Dorothy Illing, "As far as I'm concerned the Government's RQF has been badly designed and is fundamentally flawed." His statement coincided with Julie Bishop's (Federal Minister for Education, Science and Training) announcement of yet another committee to deal with the RQF, this time it's headed by two members of the DEST bureaucracy and is called the Research Quality Framework Reference Committee.

Senator Carr said, "There are different applications needed for the sciences, social sciences and the humanities and the arts, and it's particularly difficult in the arts and the humanities where we don't have the same level of internationally recognised journals. Clearly there needs to be a different approach taken in the performing arts."

And he suggested that changes to the metrics system to better reflect the performance of different disciplines where refereed articles were not always the best measure.

Clearly, Labor has considerable work left to do in order to present a plan that can be implemented.

That said the exchanges between Senator Carr and representatives first from the Australian Research Council and later from the Department of Education, Science and Training (DEST) revealed just how under prepared Ms Bishop's department is to undertake what they propose. 

    STANDING COMMITTEE ON EMPLOYMENT, WORKPLACE RELATIONS AND EDUCATION

WEDNESDAY, 14 FEBRUARY 2007

EWRE pages 10 - 15

Senator CARR (Kim Carr Labor, Victoria)—I turn now to the RQF panel structure. The ARC has made some comments in the past about these matters. I wonder whether you could provide the committee with your advice. What is your position at the moment as to the proposed panel structure of the RQF?
 

Mr Harper (Mr Greg Harper, Deputy Chief Executive Officer)—In the absence of our CEO, who has been more involved with that than those of us at the table, it is difficult for us to provide an answer on that. I note that the carriage of the RQF process is with DEST.
 

Senator CARR—I understand that—and, obviously, on behalf of the committee, send my best regards to Professor Hoj, given his accident last night, as you have advised me. Are you saying to me that there is no-one else here from the ARC who can deal with questions relating to the RQF, the Research Quality Framework, given the numbers of papers and positions you have released on this question?
 

Senator BRANDIS (George Brandis, Liberal Queensland and representing the Minister Julie Bishop, MP) —With respect, you know the circumstances in which the chief executive is not here. He is plainly the best officer to answer those questions. In his absence, if I may say so, through you, Chair, it seems a reasonable and helpful thing for the officers at the table to take the questions on notice so that the answers can be provided by the person best placed to answer the question.
 

Senator CARR—The question, though, is: is there only one person in the ARC who has been dealing with this matter?
 

Senator Brandis —I do not think that is the point. What the officers are obliged to provide to the Senate is the best answers they can, and, if in their professional judgement your questions are best answered by reference to an officer who cannot, for reasons you will understand, be available and therefore you will get a better and full answer by the question being taken on notice, I think not only is that appropriate but it is a proper courtesy to you.
 

Senator CARR—I appreciate the point you make. However, the question of what would be the best answers to questions I have yet to ask is, I think, somewhat presumptuous.
 

Senator Brandis—You have foreshadowed where you are going.
 

Senator CARR—I have asked one question in relation to the panel structure. I now ask: is it the case that only one officer in the ARC has been dealing with the RQF?
 

CHAIR (Senator Judith Troeth Liberal, Victoria) —I think the officers have indicated that Professor Hoj is the best person to answer that. He is not here.
 

Senator CARR—The best person is a separate question. I would like to know: has there only been one officer working on this issue? That is a question that surely the officers at the table can answer.
 

Mr Marsden (Mr Len Marsden, Chief Operating Officer)—The RQF covers quite a large area, as you can appreciate, so yes, there has been more than one officer involved in the RQF. But, in relation to the question you ask, Professor Hoj is the best placed for that particular question.
 

Senator CARR—On the question of the structure of the panels?
 

Mr Marsden—Yes, he is best placed on it. We are only a small organisation, as you can appreciate.
 

Senator CARR—I do.
 

Mr Marsden—So we tend to specialise just because of the nature of our size.
 

Senator CARR—I can understand that. Can you tell me this: has the ARC expressed a view with regard to the number of panels that is being proposed? That is not technical question; that is a straight question seeking factual information.
 

Mr Harper—The ARC has been involved in discussions concerning the Research Quality Framework and, in the course of those discussions, may or may not have expressed views.
 

Senator CARR—The point is that you do not know. Is that what you are telling me?
 

Mr Harper—I do not know the totality of the views expressed by the ARC on this matter, no.
 

Senator CARR—I appreciate it was a bit hard to predict that the CEO would not be available, and it was certainly beyond your control—although I do note that, in the case of CSIRO, the CEO indicated to me that he was not going to be here. Are there any other CEOs who have indicated to the committee that they will not be here?
 

CHAIR—Not so far as I know. They are the only two: one by circumstance and one by other plans.
 

Senator CARR—Has the AIMS CEO indicated he is coming?
 

CHAIR—We have not been advised that he is not.

:

:

:

Senator CARR—Did you participate in the drafting of the papers for the RQF?
 

Prof. Papadakis (Professor Elim Papadakis, Executive Director for Social, Behavioural and Economic Sciences)—For the Productivity Commission report?
 

Senator CARR—No, for the RQF more generally.
 

Prof. Papadakis—No. The CEO played the central role.
 

Senator CARR—Is he the only one who drafted those papers?
 

Prof. Papadakis—No. There was discussion among colleagues about—
 

Senator CARR—I just wondered if the CEO, as a rule, drafted his own papers.
 

Prof. Papadakis—The CEO has a very hands-on and effective role in the organisation and certainly his imprint would be on them.
 

Senator CARR—So none of the officers who helped draft the papers are available?
 

Prof. Papadakis—Which papers?
 

Senator CARR—With regard to the RQF.
 

Prof. Papadakis—I am not sure which papers in particular you are referring to.
 

Senator CARR—The responses to the government’s positions. It struck me that, given your role in the Productivity Commission submission, you might have had some role in the ARC support paper: a proposed panel structure for the RQF, dated August 2005.
 

Prof. Papadakis—That was primarily the CEO and our research evaluation—
 

Mr Marsden—And the director of our research evaluation section. Those were the two officers.
 

Senator CARR—Is she here?
 

Mr Marsden—No, she is not.
 

Senator CARR—Who helped with the drafting of the paper on the Research Quality Framework response to the preferred model, dated October 2005?
 

Mr Marsden—The same two officers.
 

Senator CARR—Only two officers?
 

Mr Marsden—Yes.
 

Senator CARR—And she is not here?
 

Mr Marsden—No.
 

Senator CARR—Is it the case that the paper assessing the research impact, dated 28 June, was drafted by the same two officers?
 

Prof. Papadakis—Which year?
 

Senator CARR—2005. It was actually July 2005 by the time it was published.
 

Mr Marsden—Yes.
 

:

:

:
 

Senator CARR—With regard to the RQF, which is discussed in this paper, you say: The incentives delivered by the Framework, and its potential to drive improvements in quality and impact of Australian research, will depend crucially on the nature of the link between funding and RQF ratings. You go on to talk about: … the comparability across disciplines of measures of quality and impact. Could you explain to the committee what you mean by that.
 

Dr Mackinnon—Could you repeat the last sentence you quoted?
 

Senator CARR—You mentioned the issue of ‘the comparability across disciplines of measures of quality and impact’. If you look at page 12 of your submission, there is a whole section on the RQF.
 

Dr Mackinnon—Yes, I am there. I am looking for the sentence.
 

Senator CARR—It is in the second last paragraph on page 12. It is: The incentives delivered by the Framework, and its potential to drive improvements in quality and impact of Australian research, will depend crucially on the nature of the link between funding and RQF ratings. The latter will in turn be influenced by such details as the configuration of research grouping -- and so on. You then go on to talk about the cross-discipline measures of quality and impact.
 

Mr Harper—I think we are saying there that all of those factors, which, as I understand it, are not yet finalised, will have an impact on the incentives delivered by the framework.
 

Senator CARR—Thank you, I can read that. I am just wondering if you could explain to me why the ARC believes that to be the case. Who drafted that section?
 

Mr Harper—It was not me, and I think it is much the same answer as we have given you several times before.
 

Senator CARR—Yes. The submission also says:
The ARC is, nevertheless, concerned that the operation of the RQF, and in particular the demands it will place on assessment capability in 2008, may adversely affect the availability of assessors and hence the quality of the assessments which the ARC will be able to undertake itself in that year. Could you explain to me what you mean by that.
 

Mr Harper—The ARC has been receiving an increasing number of applications, as we noted earlier today. Those applications are peer reviewed and the load on the peer reviewers, the assessing community, has been increasing over time. Clearly the RQF itself also has an assessing load. So what we are doing there is raising the concern that having both assessing loads in 2008 is something which needs to be managed carefully.
 

Senator CARR—I appreciate that. You go on to say: It would be unfortunate if an attempt to measure and ultimately improve the quality and impact of Australian research were to have the effect of limiting the ability of funding agencies such as the ARC to guarantee the quality of their own processes ... That is a pretty strong statement for a program that the government has just provided $87 million for. How are you going to manage the concern that you have raised?
 

Mr Harper—We will be engaged with DEST and other stakeholders to try to ensure that we conduct as good a peer review process as we can under those circumstances. It is hard to be specific in response to your question.
 

Senator CARR—Your concern remains, though?
 

Mr Harper—Yes.
 

Senator CARR—There is nothing that you have seen between December 2006 and now to satisfy you that your concerns were unfounded?
 

Mr Harper—The assessing load will, prima facie, increase. The assessing community is under a reasonable amount of pressure anyway, and that pressure is increasing with increasing numbers of applications to the ARC. We continue to be concerned to ensure that at least our processes are run as well as we can, and obviously the people with whom we are speaking on the RQF will be concerned to ensure that their process runs as well as it can.
 

Senator CARR—How much extra money have you been allocated to cope with the RQF?
 

Mr Marsden—Nothing.
 

Senator CARR—What will the additional cost be to the ARC to implement the RQF?
 

Mr Marsden—Can you expand on that question? DEST is implementing the RQF. We are indicating in what you have just read back to us that we think there are going to be assessor issues. We are not yet sure of the extent of those other than that we have communicated those concerns to the department itself. We will engage with the department as often as we can to manage, hopefully, that concern so it does not become a real issue.
 

Senator CARR—I am surprised that you have not been allocated any money out of the $87 million for implementation.
 

Mr Marsden—There has been no allocation to the ARC.
 

Senator CARR—Thank you. I will put the rest of my questions on notice.
 

______________________________[Ended10.04 am]_________________________________
 

EWRE pages 98 - 101

Senator CARR—Dr Arthur, I might ask you some questions regarding the RQF. The ARC have made a number of statements concerning the RQF. I wonder if you could respond to those matters that have been raised by the ARC. I might go through these questions. They were particularly concerned about the impact of the RQF in terms of the eight elements which ought to be included. Have you had an opportunity to review those papers?
 

Dr Arthur (Dr Evan Arthur, Group Manager, Innovation and Research Systems Group) —I did not quite catch the full detail of that question.
 

Senator CARR—Let us just go through it. My questions regard the paper dated October 2005. I might go to that one.
 

Dr Arthur—Indeed.
 

Senator CARR—What do you say to their concerns, for instance, about the matrix being used? They say: …this matrix will deliver inadequate and inaccurate information about the overall quality of research in Australian universities.
 

Dr Arthur—If I hear you correctly, they are comments made in 2005 at the stage when the process was under the guidance of the expert advisory group. There was a large amount of detailed work done addressing the development of the preferred model for the research quality framework in the course of 2006, leading obviously to the eventual government decision to proceed with the model, including on those questions. The ARC was involved in those processes. There were specific working groups looking at a number of those questions. I and Ms Harvey can probably go through a number of the tos and fros on that. All I can say is that they were comments made at a particular time. They were very helpful, as all the comments from the ARC were, in bringing forth matters to be considered, which were considered and dealt with in detail in the further processes that occurred.
 

Senator CARR—I thought you might say that. The criticism was based on an old paper. More recently, though, the question of the panel structure has been raised by the ARC. Do you agree that there is still a problem there as far as the ARC is concerned?
 

Dr Arthur—I cannot speak for the ARC. The panel structure, with one exception, has not changed greatly from the time of the expert advisory group. The ARC, in particular Professor Hoj, along with the NHMRC were in fact the major people who provided us with the recommendations which led to the panel structure which was eventually recommended by the expert advisory group. The major change I referred to which has occurred since then is that one of the panels—there were 12—has been divided into two after further commentary provided by the stakeholders. As I say, I cannot speak specifically for the ARC. Perhaps if you were to cite exactly their concern and the date of that, I could be more helpful.
 

Senator CARR—It is a whole series of things which obviously we have to put to the ARC, given that their chief executive was not here today.
 

Dr Arthur—Indeed.

Senator CARR—It is probably more fruitful if I go, then, to some of the comments that are reflected in more contemporary work by the Group of Eight along similar lines. Dr Arthur, you would agree that the Group of Eight have been vociferous in their criticism?
 

Dr Arthur—The Group of Eight have, like most stakeholders in the process, been very helpful and very constructive in trying to put forward comments which will lead to a good model.
 

Senator CARR—They argue the case in the document outlined in the 2006 paper with regard to the difference between how you measure the excellence pool and the impact pool. How will they be separated?
 

Dr Arthur—The government has not yet made a decision on that issue. They have made a decision that there will be separate scores for quality and impact. Those scores will remain separate and both will have an impact on the eventual result for universities in terms of the status and the view taken of the quality and impact of their research and on the funding that will be applied for, to which the RQF will be applied. However, the government has not made a decision about the specifics of how that will apply. It has not made any decision about the specifics about how the RQF will affect funding for universities.
 

Senator CARR—So how far will the RTS funding be affected or driven by the so-called impact measure?

Dr Arthur—Overall, the government has decided that the RQF will in general terms influence 50 per cent at least of the RTS funding. But, beyond that, on the question you specifically ask about the relative impact of quality and impact—excuse the circularity there—no decision has been made.
 

Senator CARR—So what is the $87 million the government recently announced to be used for?

Dr Arthur—The $87 million reflects the costs of developing and implementing the RQF both in terms of costs to the department and costs to the higher education sector.
 

Senator CARR—Over what length of time will that money be spent?
 

Ms Harvey (Ms Leanne Harvey, Branch Manager, Research Quality and Coordination Branch, Innovation and Research Systems Group) —Over four years: over the forward estimate years.
 

Senator CARR—Is it anticipated that these assessments will be undertaken every four years? How often will we be going through this exercise?
 

Ms Harvey—I think at the moment the government has indicated that they would be looking at doing an assessment every six years.
 

Senator CARR—Six years. So there will be a need for another allocation of money, presumably, as we run into the second round?
 

Dr Arthur—The allocation of funds in the future will not be of the scale required for the first iteration of the exercise. The government has made a decision to apply, as far as it is going to be possible, a technological solution to the data gathering and assessment. A large element of the costs in the department, and particularly costs in universities, is to establish an electronic infrastructure which will be available in the future and, indeed, which will greatly benefit a number of other issues in research. The particular part of funding that is irrelevant there is a significant amount of funding—Ms Harvey can provide the detail—for establishing a network of digital repositories in universities which will enable the electronic capture of research and, indeed, the electronic provision of that research to DEST.
 

Senator CARR—So will each university be required to maintain its own repository or will there be a centralised repository? How will that work?
 

Dr Arthur—That is yet to be determined. There will be detailed consultation with the sector on exactly what would be the most efficient model. But funding has been provided which will enable that to occur.
 

Senator CARR—So what are the list of issues yet to be resolved with regard to the implementation of this RQF?
 

Dr Arthur—There certainly are a whole range of issues.
 

Senator CARR—Could you tell me what they would be?
 

Dr Arthur—I can give you some of the more important ones, but it will not be an exhaustive list. The government has made decisions in terms of the broad criteria which will apply to the rankings of quality and impact. However, there is a lot of detailed work to be done to make those appropriate for the various discipline clusters that the panel will address. One of the most immediate steps we are now undertaking is a process of appointing the various discipline panels that will do the detailed work on the criteria both for the evidence portfolios and the criteria for reaching judgements linking to scores across the various discipline clusters. That is one large set of work. Another very large set of work is, as I indicated, the process work of establishing IT systems within the departments that can support the decision-making processes that will be required and that can work within the universities to both gather the data and establish, where appropriate, electronic infrastructure to manipulate those pieces of information.
 

Senator CARR—There are only two issues outstanding.
 

Dr Arthur—They are the two most important ones. There are a whole range of other ones. I can go on at some length.
 

Senator CARR—I am very interested in this.
 

CHAIR—Dr Arthur is trying to give you concise, succinct answers.
 

Senator CARR—Dr Arthur is always concise.
 

CHAIR—And succinct, I am sure.
 

Senator CARR—And succinct. But if you would like to take that on notice, I would be delighted to have the full and comprehensive list of unresolved issues with regard to the RQF. It may be that you will need until 31 March to complete that.

Dr Arthur—It is quite possible. What we can certainly provide you with is issues that we have identified now. As with any complex implementation task, there will be issues that will arise as we go through that we have not been able to specify now. We can certainly provide you with an indication of the significant issues that are there.
 

Senator CARR—But it will not be a comprehensive list?
 

Dr Arthur—At the moment we have, obviously, implementation plans. We can certainly, without going into the detail of that, provide you with the key issues that we have. It is normal planning processes identified that will need to be addressed to appropriately carry forward this very large project.
 

Senator CARR—Dr Arthur, can you indicate to me where I might find any international evidence about the way in which this impact proposal will work?
 

Dr Arthur—You will not find international evidence on the way we are doing this particular element of it because what we are doing with impact, as the minister has made quite clear in her announcements, is new. It is not new to try to assess the impact of research in these sorts of exercises. The two major exercises that occurred to date in this area in the UK and New Zealand certainly considered the impact of research in arriving at judgements of the quality of research. What is new in this exercise is the attempt to reduce the judgements of the impact of research to a separate uniform scale from the judgement of quality. That is technically a new task. Certainly it is technically going to be a challenging task.
 

Senator CARR—Yes. Dr Arthur, the technical challenge is different for the sciences and for the social sciences and for the humanities. Would that be true?
 

Dr Arthur—No. I do not think I would agree with that. It is essentially the same challenge. Both for quality and for impact there are going to be differences in the evidence portfolios that will be relevant and in the fine detail of criteria for that. As in any such assessment exercise—and at a technical level this is just another large assessment exercise similar to that which is carried out in a whole range of activities around the world—the technical processes that are involved are essentially similar across the subject matters.
 

Senator CARR—Who designed this scheme?
 

Ms Paul (Ms Lisa Paul, Secretary) —It has been designed with the help of these experts and consultative processes over a long period of time.
 

Dr Arthur—With a very, very large number of inputs. We can provide you with the detail, but it is all on the website. If you look at the numbers of committees, the members of those committees and the permission processes, you will see it has been an extremely iterative, collaborative process.
 

Senator CARR—I see. Of doubtful parentage.
 

Dr Arthur—I do not think that is in any way what I said.
 

Ms Paul—I was going to add that we have had, of course, very close regard to what has happened overseas. Through close regard to the 20 years of experience in the UK in particular and experience in New Zealand and then working with this range of experts through the expert advisory group and then the development advisory group and consultations much broader than that, I believe we have come to an approach to measuring excellence in research which is as streamlined and as collaborative as it possibly could be.
 

Senator CARR—So you do not agree with the criticism that it would be labour intense?
 

Ms Paul—It will involve work by universities, particularly in the first instance, which is what the funding is for. But we believe the system that, through all these processes, we have come to in this country will actually be more streamlined than in other places. That is probably all I need to say. There has been an enormous amount of work put into—and will continue to be, as Dr Arthur says—making it as straightforward as possible. The use of the digital repositories, for example, is a really good example of that. The use of digital sharing of research papers and so on will benefit universities in a whole range of ways beyond just the RQF.
 

Dr Arthur—I might add to your comment about doubtful parentage. It is important to note that all elements of this process have been extremely public. The members of the committees that have assisted in this, the committees that made the formal recommendations to the minister—the expert advisory group and the development advisory group—are also public. The actual recommendations they made have been made public, as have the government responses to that. So at every step, if you wish to look at it in detail of where the particular issues came from, we have been careful in the extreme to make sure that is all fully transparent.

Ms Paul—Perhaps it is not doubtful parentage. It is broad parentage, because so many have had so much input over a period of time.
 

Senator CARR—I see. Broad parentage. And the criticisms have also been quite broad, haven’t they?
 

Dr Arthur—This is a very important exercise affecting potentially large amounts of university funding. The universities should take this exercise very seriously and should engage, as they have done, robustly in putting their views forward to make sure that we get the best possible result.
 

Senator CARR—I have on page 15 of the additional estimates statements an indication of the line item research quality implementation. I wonder if you could give me on notice a breakdown of what these figures will be spent on across the forward estimates.
 

Dr Arthur—Certainly.
 

Senator CARR—Ms Paul, on that other question, I take it you will be able to tell me how much of the money has actually been committed.
 

Dr Arthur—If I could make a comment: the money that was announced by the minister was all, as I said, in the forward process. None of that money is currently flowing. The money that has been spent to date was two separate amounts of money provided, first of all, in the announcements for Backing Australia’s Ability. There was an amount of $2.8 million. Am I right?
 

Ms Harvey—Yes.
 

Dr Arthur—That was for the initial development of the RQF.
 

Ms Harvey—And the accessibility framework.
 

Dr Arthur—And the accessibility framework. There was a further amount of funding in the last budget for further development of that of $3 million. So it is $5.8 million to date. The money the minister has announced is for the full implementation and will start to flow from 1 July.
 

Senator CARR—I see. So none of that money has been allocated?
 

Dr Arthur—No.
 

Senator CARR—Has all of the money from the last two tranches been spent?
 

Dr Arthur—Not yet. It will be before we come to the end of this financial year. But not as yet.
 

Senator CARR—How much remains?
 

Dr Arthur—We will take that on notice. We will find a date, obviously, and report to you from that date.