|
|
|
|
News & Views item -
|
Labor Promises to Reinvent the Government's Research Quality Framework Proposal, If It Wins Government. (February 21, 2007)
When Federal Labor Senator Kim Carr, shadow minister for Innovation, Industry, Science & Research addressed the forum on An Investment In Australia's Future: Why the mathematical sciences matter on February 7 he told his audience that were Labor to ascend to government it would replace the government's suggested Research Quality Framework. He ridiculed the government's unworkable proposal to include an assessment of the impact of each research program on national productivity, and was damning of the government's allocation of some $85M to fund an RQF that would be allocating some $500M. He did, however, leave the distinct impression that Labor would introduce some form of RQF rather than scrapping a concept that no one has demonstrated to have merit.
Now having sat through Senate Estimates, he has told The Australian's Dorothy Illing, "As far as I'm concerned the Government's RQF has been badly designed and is fundamentally flawed." His statement coincided with Julie Bishop's (Federal Minister for Education, Science and Training) announcement of yet another committee to deal with the RQF, this time it's headed by two members of the DEST bureaucracy and is called the Research Quality Framework Reference Committee.
Senator Carr said, "There are different applications needed for the sciences, social sciences and the humanities and the arts, and it's particularly difficult in the arts and the humanities where we don't have the same level of internationally recognised journals. Clearly there needs to be a different approach taken in the performing arts."
And he suggested that changes to the metrics system to better reflect the performance of different disciplines where refereed articles were not always the best measure.
Clearly, Labor has considerable work left to do in order to present a plan that can be implemented.
That said the exchanges between Senator Carr and representatives first from the Australian Research Council and later from the Department of Education, Science and Training (DEST) revealed just how under prepared Ms Bishop's department is to undertake what they propose.
STANDING COMMITTEE ON EMPLOYMENT, WORKPLACE RELATIONS AND
EDUCATION
WEDNESDAY, 14 FEBRUARY 2007
EWRE pages 10 - 15
Senator CARR (Kim Carr Labor, Victoria)—I turn now to the RQF panel structure.
The ARC has made some comments in the past about these matters. I wonder whether
you could provide the committee with your advice. What is your position at the
moment as to the proposed panel structure of the RQF?
Mr Harper (Mr Greg Harper, Deputy Chief Executive Officer)—In the absence of our CEO, who has been more involved with that than those of us at the table, it is
difficult for us to provide an answer on that. I note that the carriage of the
RQF process is with DEST.
Senator CARR—I understand that—and, obviously, on behalf of the committee, send
my best regards to Professor Hoj, given his accident last night, as you have
advised me. Are you saying to me that there is no-one else here from the ARC who
can deal with questions relating to the RQF, the Research Quality Framework,
given the numbers of papers and positions you have released on this question?
Senator BRANDIS (George Brandis, Liberal Queensland and representing the
Minister Julie Bishop, MP) —With respect, you know the circumstances in which the chief
executive is not here.
He is plainly the best officer to answer those questions. In his absence, if I
may say so, through you, Chair, it seems a reasonable and helpful thing for the
officers at the table to take the questions on notice so that the answers can be
provided by the person best placed to answer the question.
Senator CARR—The question, though, is: is there only one person in the ARC who
has been dealing with this matter?
Senator Brandis —I do not think that is the
point. What the officers are obliged to provide to the Senate is the best
answers they can, and, if in their professional judgement your questions are
best answered by reference to an officer who cannot, for reasons you will
understand, be available and therefore you will get a better and full answer by
the question being taken on notice, I think not only is that appropriate but it
is a proper courtesy to you.
Senator CARR—I appreciate the point you make. However, the question of what
would be the best answers to questions I have yet to ask is, I think, somewhat
presumptuous.
Senator Brandis—You have foreshadowed where you are going.
Senator CARR—I have asked one question in relation to the panel structure. I now
ask: is it the case that only one officer in the ARC has been dealing with the
RQF?
CHAIR (Senator Judith Troeth
Liberal, Victoria) —I think the officers have indicated that Professor Hoj is
the best person to answer that. He is not here.
Senator CARR—The best person is a separate question. I would like to know: has
there only been one officer working on this issue? That is a question that
surely the officers at the table can answer.
Mr Marsden (Mr Len Marsden, Chief Operating Officer)—The RQF covers quite a large area, as you can
appreciate, so yes, there has been more than one officer involved in the RQF.
But, in relation to the question you ask, Professor Hoj is the best placed for
that particular question.
Senator CARR—On the question of the structure of the panels?
Mr Marsden—Yes, he is best placed on it. We are only a small organisation, as
you can appreciate.
Senator CARR—I do.
Mr Marsden—So we tend to specialise just because of the nature of our size.
Senator CARR—I can understand that. Can you tell me this: has the ARC expressed
a view with regard to
the number of panels that is being proposed? That is not technical question;
that is a straight question seeking factual information.
Mr Harper—The ARC has been involved in discussions concerning the Research
Quality Framework and,
in the course of those discussions, may or may not have expressed views.
Senator CARR—The point is that you do not know. Is that what you are telling me?
Mr Harper—I do not know the totality of the views expressed by the ARC on this
matter, no.
Senator CARR—I appreciate it was a bit hard to predict that the CEO would not be
available, and it was certainly beyond your control—although I do note that, in
the case of CSIRO, the CEO indicated to me that he was not going to be here. Are
there any other CEOs who have indicated to the committee that they will not be
here?
CHAIR—Not so far as I know. They are the only two: one by circumstance and one
by other plans.
Senator CARR—Has the AIMS CEO indicated he is coming?
CHAIR—We have not been advised that he is not.
:
:
:
Senator CARR—Did you participate in the drafting of the papers for the RQF?
Prof. Papadakis (Professor Elim Papadakis, Executive Director for Social,
Behavioural and Economic Sciences)—For the Productivity Commission report?
Senator CARR—No, for the RQF more generally.
Prof. Papadakis—No. The CEO played the central role.
Senator CARR—Is he the only one who drafted those papers?
Prof. Papadakis—No. There was discussion among colleagues about—
Senator CARR—I just wondered if the CEO, as a rule, drafted his own papers.
Prof. Papadakis—The CEO has a very hands-on and effective role in the
organisation and certainly his imprint would be on them.
Senator CARR—So none of the officers who helped draft the papers are available?
Prof. Papadakis—Which papers?
Senator CARR—With regard to the RQF.
Prof. Papadakis—I am not sure which papers in particular you are referring to.
Senator CARR—The responses to the government’s positions. It struck me that,
given your role in the Productivity Commission submission, you might have had
some role in the ARC support paper: a proposed panel structure for the RQF,
dated August 2005.
Prof. Papadakis—That was primarily the CEO and our research evaluation—
Mr Marsden—And the director of our research evaluation section. Those were the
two officers.
Senator CARR—Is she here?
Mr Marsden—No, she is not.
Senator CARR—Who helped with the drafting of the paper on the Research Quality
Framework response
to the preferred model, dated October 2005?
Mr Marsden—The same two officers.
Senator CARR—Only two officers?
Mr Marsden—Yes.
Senator CARR—And she is not here?
Mr Marsden—No.
Senator CARR—Is it the case that the paper assessing the research impact, dated
28 June, was drafted by the same two officers?
Prof. Papadakis—Which year?
Senator CARR—2005. It was actually July 2005 by the time it was published.
Mr Marsden—Yes.
:
:
:
Senator CARR—With regard to the RQF, which is discussed in this paper, you say:
The incentives delivered by the Framework, and its potential to drive
improvements in quality and impact of Australian research, will depend crucially
on the nature of the link between funding and RQF ratings. You go on to talk
about: … the comparability across disciplines of measures of quality and impact.
Could you explain to the committee what you mean by that.
Dr Mackinnon—Could you repeat the last sentence you quoted?
Senator CARR—You mentioned the issue of ‘the comparability across disciplines of
measures of quality and impact’. If you look at page 12 of your submission,
there is a whole section on the RQF.
Dr Mackinnon—Yes, I am there. I am looking for the sentence.
Senator CARR—It is in the second last paragraph on page 12. It is: The
incentives delivered by the Framework, and its potential to drive improvements
in quality and impact of Australian research, will depend crucially on the
nature of the link between funding and RQF ratings. The latter will in turn be
influenced by such details as the configuration of research grouping -- and so
on. You then go on to talk about the cross-discipline measures of quality and
impact.
Mr Harper—I think we are saying there that all of those factors, which, as I
understand it, are not yet finalised, will have an impact on the incentives
delivered by the framework.
Senator CARR—Thank you, I can read that. I am just wondering if you could
explain to me why the ARC believes that to be the case. Who drafted that
section?
Mr Harper—It was not me, and I think it is much the same answer as we have given
you several times before.
Senator CARR—Yes. The submission also says:
The ARC is, nevertheless, concerned that the operation of the RQF, and in
particular the demands it will place on assessment capability in 2008, may
adversely affect the availability of assessors and hence the quality of the
assessments which the ARC will be able to undertake itself in that year. Could
you explain to me what you mean by that.
Mr Harper—The ARC has been receiving an increasing number of applications, as we
noted earlier today. Those applications are peer reviewed and the load on the
peer reviewers, the assessing community, has been increasing over time. Clearly
the RQF itself also has an assessing load. So what we are doing there is raising
the concern that having both assessing loads in 2008 is something which needs to
be managed carefully.
Senator CARR—I appreciate that. You go on to say: It would be unfortunate if an
attempt to measure and ultimately improve the quality and impact of Australian
research were to have the effect of limiting the ability of funding agencies
such as the ARC to guarantee the quality of their own processes ... That is a
pretty strong statement for a program that the government has just provided $87
million for. How are you going to manage the concern that you have raised?
Mr Harper—We will be engaged with DEST and other stakeholders to try to ensure
that we conduct as good a peer review process as we can under those
circumstances. It is hard to be specific in response to your question.
Senator CARR—Your concern remains, though?
Mr Harper—Yes.
Senator CARR—There is nothing that you have seen between December 2006 and now
to satisfy you that your concerns were unfounded?
Mr Harper—The assessing load will, prima facie, increase. The assessing
community is under a reasonable amount of pressure anyway, and that pressure is
increasing with increasing numbers of applications to the ARC. We continue to be
concerned to ensure that at least our processes are run as well as we can, and
obviously the people with whom we are speaking on the RQF will be concerned to
ensure that their process runs as well as it can.
Senator CARR—How much extra money have you been allocated to cope with the RQF?
Mr Marsden—Nothing.
Senator CARR—What will the additional cost be to the ARC to implement the RQF?
Mr Marsden—Can you expand on that question? DEST is implementing the RQF. We are
indicating in what you have just read back to us that we think there are going
to be assessor issues. We are not yet sure of the extent of those other than
that we have communicated those concerns to the department itself. We will
engage with the department as often as we can to manage, hopefully, that concern
so it does not become a real issue.
Senator CARR—I am surprised that you have not been allocated any money out of
the $87 million for implementation.
Mr Marsden—There has been no allocation to the ARC.
Senator CARR—Thank you. I will put the rest of my questions on notice.
______________________________[Ended10.04 am]_________________________________
EWRE pages 98 - 101
Senator CARR—Dr Arthur, I might ask you some questions regarding the RQF. The
ARC have made a number of statements concerning the RQF. I wonder if you could
respond to those matters that have been raised by the ARC. I might go through
these questions. They were particularly concerned about the impact of the RQF in
terms of the eight elements which ought to be included. Have you had an
opportunity to review those papers?
Dr Arthur (Dr Evan Arthur, Group Manager, Innovation and Research Systems
Group) —I did not quite catch the full detail of that question.
Senator CARR—Let us just go through it. My questions regard the paper dated
October 2005. I might go to that one.
Dr Arthur—Indeed.
Senator CARR—What do you say to their concerns, for instance, about the matrix
being used? They say: …this matrix will deliver inadequate and inaccurate
information about the overall quality of research in Australian universities.
Dr Arthur—If I hear you correctly, they are comments made in 2005 at the stage
when the process was under the guidance of the expert advisory group. There was
a large amount of detailed work done addressing the development of the preferred
model for the research quality framework in the course of 2006, leading
obviously to the eventual government decision to proceed with the model,
including on those questions. The ARC was involved in those processes. There
were specific working groups looking at a number of those questions. I and Ms
Harvey can probably go through a number of the tos and fros on that. All I can
say is that they were comments made at a particular time. They were very
helpful, as all the comments from the ARC were, in bringing forth matters to be
considered, which were considered and dealt with in detail in the further
processes that occurred.
Senator CARR—I thought you might say that. The criticism was based on an old
paper. More recently, though, the question of the panel structure has been
raised by the ARC. Do you agree that there is still a problem there as far as
the ARC is concerned?
Dr Arthur—I cannot speak for the ARC. The panel structure, with one exception,
has not changed greatly from the time of the expert advisory group. The ARC, in
particular Professor Hoj, along with the NHMRC were in fact the major people who
provided us with the recommendations which led to the panel structure which was
eventually recommended by the expert advisory group. The major change I referred
to which has occurred since then is that one of the panels—there were 12—has
been divided into two after further commentary provided by the stakeholders. As
I say, I cannot speak specifically for the ARC. Perhaps if you were to cite
exactly their concern and the date of that, I could be more helpful.
Senator CARR—It is a whole series of things which obviously we have to put to
the ARC, given that their chief executive was not here today.
Dr Arthur—Indeed.
Senator CARR—It is probably more fruitful if I go, then, to some of the comments
that are reflected in more contemporary work by the Group of Eight along similar
lines. Dr Arthur, you would agree that the Group of Eight have been vociferous
in their criticism?
Dr Arthur—The Group of Eight have, like most stakeholders in the process, been
very helpful and very constructive in trying to put forward comments which will
lead to a good model.
Senator CARR—They argue the case in the document outlined in the 2006 paper with
regard to the difference between how you measure the excellence pool and the
impact pool. How will they be separated?
Dr Arthur—The government has not yet made a decision on that issue. They have
made a decision that there will be separate scores for quality and impact. Those
scores will remain separate and both will have an impact on the eventual result
for universities in terms of the status and the view taken of the quality and
impact of their research and on the funding that will be applied for, to which
the RQF will be applied. However, the government has not made a decision about
the specifics of how that will apply. It has not made any decision about the
specifics about how the RQF will affect funding for universities.
Senator CARR—So how far will the RTS funding be affected or driven by the so-called impact measure?
Dr Arthur—Overall, the government has decided that the RQF will in general terms
influence 50 per cent at least of the RTS funding. But, beyond that, on the
question you specifically ask about the relative impact of quality and
impact—excuse the circularity there—no decision has been made.
Senator CARR—So what is the $87 million the government recently announced to be used for?
Dr Arthur—The $87 million reflects the costs of developing and implementing the
RQF both in terms of costs to the department and costs to the higher education
sector.
Senator CARR—Over what length of time will that money be spent?
Ms Harvey (Ms Leanne Harvey, Branch Manager, Research Quality and
Coordination Branch, Innovation and Research Systems Group) —Over four years: over the forward estimate years.
Senator CARR—Is it anticipated that these assessments will be undertaken every
four years? How often will we be going through this exercise?
Ms Harvey—I think at the moment the government has indicated that they would be
looking at doing an assessment every six years.
Senator CARR—Six years. So there will be a need for another allocation of money,
presumably, as we run into the second round?
Dr Arthur—The allocation of funds in the future will not be of the scale
required for the first iteration of the exercise. The government has made a
decision to apply, as far as it is going to be possible, a technological
solution to the data gathering and assessment. A large element of the costs in
the department, and particularly costs in universities, is to establish an
electronic infrastructure which will be available in the future and, indeed,
which will greatly benefit a number of other issues in research. The particular
part of funding that is irrelevant there is a significant amount of funding—Ms
Harvey can provide the detail—for establishing a network of digital repositories
in universities which will enable the electronic capture of research and,
indeed, the electronic provision of that research to DEST.
Senator CARR—So will each university be required to maintain its own repository
or will there be a centralised repository? How will that work?
Dr Arthur—That is yet to be determined. There will be detailed consultation with
the sector on exactly what would be the most efficient model. But funding has
been provided which will enable that to occur.
Senator CARR—So what are the list of issues yet to be resolved with regard to
the implementation of this RQF?
Dr Arthur—There certainly are a whole range of issues.
Senator CARR—Could you tell me what they would be?
Dr Arthur—I can give you some of the more important ones, but it will not be an
exhaustive list. The government has made decisions in terms of the broad
criteria which will apply to the rankings of quality and impact. However, there
is a lot of detailed work to be done to make those appropriate for the various
discipline clusters that the panel will address. One of the most immediate steps
we are now undertaking is a process of appointing the various discipline panels
that will do the detailed work on the criteria both for the evidence portfolios
and the criteria for reaching judgements linking to scores across the various
discipline clusters. That is one large set of work. Another very large set of
work is, as I indicated, the process work of establishing IT systems within the
departments that can support the decision-making processes that will be required
and that can work within the universities to both gather the data and establish,
where appropriate, electronic infrastructure to manipulate those pieces of
information.
Senator CARR—There are only two issues outstanding.
Dr Arthur—They are the two most important ones. There are a whole range of other
ones. I can go on at some length.
Senator CARR—I am very interested in this.
CHAIR—Dr Arthur is trying to give you concise, succinct answers.
Senator CARR—Dr Arthur is always concise.
CHAIR—And succinct, I am sure.
Senator CARR—And succinct. But if you would like to take that on notice, I would be delighted to have the full and comprehensive list of unresolved issues with regard to the RQF. It may be that you will need until 31 March to complete that.
Dr Arthur—It is quite possible. What we can certainly provide you with is issues
that we have identified now. As with any complex implementation task, there will
be issues that will arise as we go through that we have not been able to specify
now. We can certainly provide you with an indication of the significant issues
that are there.
Senator CARR—But it will not be a comprehensive list?
Dr Arthur—At the moment we have, obviously, implementation plans. We can
certainly, without going into the detail of that, provide you with the key
issues that we have. It is normal planning processes identified that will need
to be addressed to appropriately carry forward this very large project.
Senator CARR—Dr Arthur, can you indicate to me where I might find any
international evidence about the way in which this impact proposal will work?
Dr Arthur—You will not find international evidence on the way we are doing this
particular element of it because what we are doing with impact, as the minister
has made quite clear in her announcements, is new. It is not new to try to
assess the impact of research in these sorts of exercises. The two major
exercises that occurred to date in this area in the UK and New Zealand certainly
considered the impact of research in arriving at judgements of the quality of
research. What is new in this exercise is the attempt to reduce the judgements
of the impact of research to a separate uniform scale from the judgement of
quality. That is technically a new task. Certainly it is technically going to be
a challenging task.
Senator CARR—Yes. Dr Arthur, the technical challenge is different for the
sciences and for the social sciences and for the humanities. Would that be true?
Dr Arthur—No. I do not think I would agree with that. It is essentially the same
challenge. Both for quality and for impact there are going to be differences in
the evidence portfolios that will be relevant and in the fine detail of criteria
for that. As in any such assessment exercise—and at a technical level this is
just another large assessment exercise similar to that which is carried out in a
whole range of activities around the world—the technical processes that are
involved are essentially similar across the subject matters.
Senator CARR—Who designed this scheme?
Ms Paul (Ms Lisa Paul, Secretary) —It has been designed with the help of these experts and consultative
processes over a long period of time.
Dr Arthur—With a very, very large number of inputs. We can provide you with the
detail, but it is all on the website. If you look at the numbers of committees,
the members of those committees and the permission processes, you will see it
has been an extremely iterative, collaborative process.
Senator CARR—I see. Of doubtful parentage.
Dr Arthur—I do not think that is in any way what I said.
Ms Paul—I was going to add that we have had, of course, very close regard to
what has happened overseas. Through close regard to the 20 years of experience
in the UK in particular and experience in New Zealand and then working with this
range of experts through the expert advisory group and then the development
advisory group and consultations much broader than that, I believe we have come
to an approach to measuring excellence in research which is as streamlined and
as collaborative as it possibly could be.
Senator CARR—So you do not agree with the criticism that it would be labour
intense?
Ms Paul—It will involve work by universities, particularly in the first
instance, which is what the funding is for. But we believe the system that,
through all these processes, we have come to in this country will actually be
more streamlined than in other places. That is probably all I need to say. There
has been an enormous amount of work put into—and will continue to be, as Dr
Arthur says—making it as straightforward as possible. The use of the digital
repositories, for example, is a really good example of that. The use of digital
sharing of research papers and so on will benefit universities in a whole range
of ways beyond just the RQF.
Dr Arthur—I might add to your comment about doubtful parentage. It is important to note that all elements of this process have been extremely public. The members of the committees that have assisted in this, the committees that made the formal recommendations to the minister—the expert advisory group and the development advisory group—are also public. The actual recommendations they made have been made public, as have the government responses to that. So at every step, if you wish to look at it in detail of where the particular issues came from, we have been careful in the extreme to make sure that is all fully transparent.
Ms Paul—Perhaps it is not doubtful parentage. It is broad parentage, because so
many have had so much input over a period of time.
Senator CARR—I see. Broad parentage. And the criticisms have also been quite
broad, haven’t they?
Dr Arthur—This is a very important exercise affecting potentially large amounts
of university funding. The universities should take this exercise very seriously
and should engage, as they have done, robustly in putting their views forward to
make sure that we get the best possible result.
Senator CARR—I have on page 15 of the additional estimates statements an
indication of the line item research quality implementation. I wonder if you
could give me on notice a breakdown of what these figures will be spent on
across the forward estimates.
Dr Arthur—Certainly.
Senator CARR—Ms Paul, on that other question, I take it you will be able to tell
me how much of the money has actually been committed.
Dr Arthur—If I could make a comment: the money that was announced by the
minister was all, as I said, in the forward process. None of that money is
currently flowing. The money that has been spent to date was two separate
amounts of money provided, first of all, in the announcements for Backing
Australia’s Ability. There was an amount of $2.8 million. Am I right?
Ms Harvey—Yes.
Dr Arthur—That was for the initial development of the RQF.
Ms Harvey—And the accessibility framework.
Dr Arthur—And the accessibility framework. There was a further amount of funding
in the last budget for further development of that of $3 million. So it is $5.8
million to date. The money the minister has announced is for the full
implementation and will start to flow from 1 July.
Senator CARR—I see. So none of that money has been allocated?
Dr Arthur—No.
Senator CARR—Has all of the money from the last two tranches been spent?
Dr Arthur—Not yet. It will be before we come to the end of this financial year.
But not as yet.
Senator CARR—How much remains?
Dr Arthur—We will take that on notice. We will find a date, obviously, and report to you from that date.