Now It's the IDG to Fix the ERA. (August 19, 2008)

In a joint announcement the Minister for Innovation, Industry, Science and Research, Senator Kim Carr, and CEO of the Australian Research Council (ARC), Professor Margaret Sheil, proclaimed "the establishment of an Indicators Development Group (IDG) to advise on the application of discipline-specific indicators for the Excellence in Research for Australia (ERA) initiative".

And how will they go about doing this?

According to Senator Carr: The IDG will consider, test and recommend appropriate discipline-specific indicators, including metrics and other proxies of quality, applied research and activity.

That ought to keep 'em occupied for a couple of election periods. Of course if they succeed in not bringing forth a mouse, expect to see a new star rising in the East.

And who are the group of savants that will perform this labour of Hercules?

And is there really any evidence that all this expenditure of effort and resources will lead to improvement of Australian research?

Wouldn't it be better to begin with an overhaul of the system of peer review of the granting bodies together with devising methodology for properly resourcing those grants which are awarded?

That is if the purpose really is to improve Australian research.

[Note added August 20, 2008: Guy Healy reports in today's Australian: "Professor Nolan (IDG Chair) told the HES yesterday that his group's task was to 'essentially refine and develop a set of indicators to be used for the Excellence in Research for Australia (exercise).Our task is to produce a set of indicators that most validly reflect the quality of research undertaken but deal with the extraordinary diversity of research that's happening in Australian universities.'
     "Asked what he saw as the main challenge facing the group, Professor Nolan nominated the difficulty of refining the indicators for the humanities and social sciences so they could be assessed for quality. He hoped to achieve it 'in a way that will have a sensible transaction cost and not paralyse with process the evidence required to demonstrate research outputs and the quality of that research'."]

_________________________________

According to the ARC document:

Indicators Development Group membership

Professor Terry Nolan (Chair)

Mr Ken Richardson

Dr Marcus Nichol

Mr Tony Sheil

Ms Paula Callan

Professor Max King

Mr Tim Yapp

Dr Jonathan Adams

Professor Anthony F.J. van Raan

Professor Patty Solomon

Mr Paul Hubbard

Ms Linda Butler

And below we reprint their thumbnail biographies.

Professor Terry Nolan (Chair)

University of Melbourne

Professor Terry Nolan is Head of the School of Population Health at the University of Melbourne and Associate Dean in the Faculty of Medicine, Dentistry and Health Sciences. He is also Head of the Vaccine and Immunisation Research Group, a joint program of the University of Melbourne and Murdoch Childrens Research Institute, and Director of the Centre for Clinical Research Excellence in Child and Adolescent Immunisation.

Professor Nolan has a Bachelor of Medicine/Bachelor of Science from the University of Western Australia. He received first class honours for his Bachelor of Medical Science. Professor Nolan has also received a PhD in Epidemiology and Biostatistics from McGill University.

Currently, Professor Nolan’s research group conducts community trials of new vaccines, evaluates immunisation service delivery and undertakes epidemiological studies. The group also undertake mathematical modelling of the population impact of vaccine-preventable diseases.

Professor Nolan was a member of National Health and Medical Research Council’s Research Committee for nine years, where he contributed significantly to thinking on the measurement of research quality.

Mr Ken Richardson

University of Queensland

Ken Richardson is the Executive Officer for the Office of the Deputy Vice-Chancellor of Research (DVCR) at the University of Queensland. In this role, Mr Richardson provides advice and support to the DVCR on the strategic management of research at an institutional level, development of research policy, management of the research budget and monitoring research performance, including data collection and reporting. He also oversees work associated with major grants including the Australian Research Council Centres of Excellence.

From August 2008, Mr Richardson will take up a new role as Director, Planning for the University of Queensland.

Four of the Australian Research Council Centres and Centres of Excellence based at the University of Queensland were awarded extension funding in 2007. Two of these centres—the Centre in Bioinformatics and Centre for Functional Nanomaterials—were also renamed Centres of Excellence. The other two are the Centre of Excellence for Integrative Legume Research and the Centre for Complex Systems.

The University of Queensland attracted significant amounts of research funding in 2007. This includes more than $16.8 million in Linkage Projects, $26.8 million for 61 National Health and Medical Research Council (NHMRC) Project Grants and $1.5 million for an NHMRC Enabling Grant, awarded over five years to establish the Australian Zebrafish Phenomic Facility.

Dr Marcus Nichol

National Health and Medical Research Council

Dr Marcus Nichol is the Director of Evaluation and Reporting for the National Health and Medical Research Council (NHMRC). In this role, he is responsible for providing grant funding and performance reporting data to internal and external stakeholders. He is also the NHMRC co-ordinator of the Excellence in Research for Australia (ERA) initiative consultation process and has previously played a key role in co-ordinating the collection and analysis of publication data from NHMRC grant recipients for bibliometric analysis.

Prior to his permanent role at the NHMRC, Dr Nichol worked with the organisation for several years as an external contractor investigating a proposed tool for the measurement of research impact and achievement. Dr Nichol has a PhD in Neuroscience and a Masters of Public Health and has worked on clinical trials and epidemiology at the National Stroke Research Institute in Melbourne.

The NHMRC is Australia's peak body for supporting health and medical research for the Australian community and has commissioned independent bibliometric analysis of Australian health research publications. This has shown that Australia greatly exceeds the expected number in the top one per cent of citations internationally. Many Australian health research areas have up to two or three per cent of published papers cited in the top one per cent internationally.

Another recent analysis of the outcomes from over 1200 NHMRC grants found that the research funded attracted another 28 cents for every NHMRC dollar from overseas, and another 27 cents from Australian sources.

Mr Tony Sheil

Griffith University

Tony Sheil is Senior Manager, Research Policy at Griffith University. His career in university management spans over 20 years, working for three universities in Australia and the United Kingdom.

In 2007, prior to joining Griffith University, Mr Sheil was the Foundation Executive Officer of Innovative Research Universities Australia (IRU Australia). IRU Australia is a consortium of six research-intensive universities founded during the period of higher education expansion in the 1960s–1970s.

Mr Sheil has a strong interest in the use of metrics relating to research performance measurement and analysis. In 2006, he served as a member of the Research Quality Framework Metrics Working Group. More recently he attended, as an invited speaker, the 3rd Meeting of the International Rankings Expert Group and the 2nd International Conference on World-Class Universities at Shanghai Jiao Tong University in October–November 2007. His study into the implications of world university rankings for national and institutional research strategy of small nations will form a chapter in a book to be jointly published by Shanghai Jiao Tong University and UNESCO-CEPES. The book is due for release in August 2008.

In June 2008, Mr Sheil presented at a conference organised by the OECD Programme on Institutional Management in Higher Education and the Nordic Universities Association, at the University of Iceland. His paper was titled Strategic options for small nations and their institutions in response to world university rankings. Recent newspaper articles by Sheil, appearing in The Australian and Campus Review, cover topics such as the use of bibliometrics for research performance analysis, the performance of Australian universities in world rankings exercises and the role of small nations in the higher education research ‘ecosystem’.

Ms Paula Callan

Queensland University of Technology

Paula Callan is the eResearch Access Coordinator at the Queensland University of Technology (QUT). Ms Callan collaborates with staff from institutes, faculties and divisions to support researchers’ uptake of e-research opportunities. In addition, Ms Callan develops and manages digital repositories for research publications and investigates systems for the organisation and citation of research datasets.

QUT leads in many eResearch areas including world leading rate of deposition by authors in its open access repository of research literature, QUT ePrints; projects and outreach in grid computing portals and workflows; and its work in legal protocols for copyright management through the Open Access to Knowledge (OAK) Law project.

Professor Max King

Monash University

Professor King is Pro-Vice Chancellor (Research and Research Training) and also a Sir John Monash Distinguished Professor at Monash University. He is also Convenor of the Council of Deans and Directors of Graduate Studies (DDOGS) and an experienced econometrician. He co-founded the Australasian Meetings of the Econometrics Society and has extensive knowledge of quantitative and analytical approaches to measurement.

DDOGS is a forum of representatives of universities engaged in graduate studies. The Council acts as a convening body to bring together academic deans and directors with university-wide responsibilities for graduate education in seminars and working groups to facilitate the exchange of ideas and dissemination of information on graduate education.

Mr Tim Yapp

Commonwealth Scientific and Industrial Research Organisation

Tim Yapp manages the Operational Performance Group in the Australian Commonwealth Scientific and Industrial Research Organisation (CSIRO).

CSIRO is Australia’s national science agency and one of the largest and most diverse research agencies in the world.  CSIRO has developed a performance measurement framework that is designed to support achievement of the Organisation’s goals through explicit consideration of:

• strategy implementation 

• achievement of research progress and external impact goals, and

• organisational sustainability, including consideration of science quality, resourcing, and the strength of internal and external relationships.  

Mr Yapp has been with CSIRO since 1992, chiefly in roles supporting CSIRO’s strategic and operational planning, science investment, evaluation, performance measurement and reporting processes. In 2007–08 he represented CSIRO on the Australian Bureau of Statistics Technical Working Group for the revision of the Standard Research Classification.   

Mr Yapp has published two papers on the topic of performance. From opportunities to outcomes—evaluating research priorities and performance in CSIRO Australia which was published in material for the Asia-Pacific Economic Cooperation (APEC) Symposium on Evaluation of S&T Programmes, Wellington, 1998, and Understanding Stakeholder Needs and Evaluating Performance in a National Research Organisation, which Mr Yapp wrote with Tim Healy, and was published in material for the International Evaluation Conference 2001.

Prior to joining CSIRO, Tim was Senior Economist in the Department of Conservation and Land Management in New South Wales. He has an honours degree in Agricultural Economics from the University of New England, and post-graduate qualifications in Economics, Theology and Public Policy. 

Dr Jonathan Adams

Evidence Ltd, UK

Dr Adams is the Lead Founder and a Director of Evidence Ltd. Evidence specialises in research performance analysis and interpretation. They carry out consultancy work for United Kingdom (UK) and European research funding organisations and produce competitor analysis reports for UK universities and research laboratories, including the UK Higher Education Research Yearbook—now in its 7th edition.

Dr Adams has worked at four international universities. He worked at King's College London from 1979 to 1980, the University of Newcastle upon Tyne from 1980 to 1983, the University of Leeds from 1983 to 1989 and the Imperial College London from 1989 to 1992.

From 1989 to 1992, Dr Adams was Science Policy Adviser to the Advisory Board for the Research Councils, first in the Department of Education and Science and then at the Cabinet Office, where he was responsible for developing performance indicators for the UK science budget.

As well as overseeing Evidence’s reports to clients, Dr Adams has published over 100 articles in research journals and contributed to scholarly books on research policy. Dr Adams has presented at many conferences including, in 2006 and 2007, the Royal Society, London; Academie des Sciences, Paris; Rand Corporation think-tank meetings in Washington, United States; and academic and commercial conferences in the UK, France and Sweden.

Dr Adams recently led the New Zealand Government’s review of research evaluation and spent two months in New Zealand visiting universities and working with the Tertiary Education Commission. In 2004, he chaired the European Community Monitoring Committee for the Evaluation of Framework Programme VI and in 2006, he chaired the Monitoring Group of the European Research Fund for Coal & Steel.

Professor Anthony F.J. van Raan

University of Leiden, The Netherlands

Anthony van Raan is Professor of Quantitative Studies of Science and Director of the Centre for Science and Technology Studies (CWTS) at Leiden University, Leiden, The Netherlands.

Professor van Raan's main research area is the design, construction and application of quantitative indicators on important aspects of science and technology. His current work covers the development of information systems in Science and Technology, cognitive and socio-organisational processes in the development of scientific and technological fields, analysis of scientific progress and assessment of scientific performance. His main research themes include research performance assessment by advanced bibliometric methods, mapping of science and technology and science as a 'self-organising' cognitive ecosystem. In this work, Professor van Raan combines ideas from physics and mathematics, particularly networks and complex systems, philosophy of science and sociology of science.

Professor van Raan's advisory work is frequently used by member states of the European Union, particularly the Netherlands Government and European Commission, the business sector, OECD and national and international research organisations.

Professor van Raan has been appointed by the German Federal Minister of Science and Education as a member of a committee to evaluate the Fraunhofer Society for Applied Research. He is also a member of the German Federal Committee on Regions of Excellent Research and the Audit Committee for the Fraunhofer Institute for Systems and Innovation Research (Karlsruhe).

Recently, Professor van Raan was appointed by the Netherlands Minister of Education, Culture and Sciences as a member of a State Committee to evaluate the Netherlands Advisory Council for Science and Technology Policy. He is also a member of the Board of Science Alliance (University-Society 'Interface' Organisation).

Professor Patty Solomon

University of Adelaide

Patty Solomon is Professor of Statistical Bioinformatics in the School of Mathematical Sciences at the University of Adelaide. She was previously Head of Statistics.

Professor Solomon was responsible for establishing the Microarray Analysis Group (MAG) in the School of Mathematical Sciences. This was a joint venture with the Adelaide Microarray Consortium, and its key collaborating partners, notably the Institute of Medical and Veterinary Science/Hanson Institute, Bionomics, Child Health Research Institute, the Australian Research Council Special Research Centre for the Molecular Genetics of Development and the Faculties of Science and Health Science at the University of Adelaide. The Group’s primary aim was to conduct high-level research on the design and analysis of gene expression studies, statistical data mining, multivariate analysis, protein analysis and other major areas in bioinformatics.

Professor Solomon’s principal research interests are in methods and applications in biostatistics, statistical bioinformatics and components of variance. Her main contributions have been to the fields of survival analysis, epidemic modelling and prediction of HIV/AIDS, the design and analysis of microarray experiments and gene expression studies, and components of variance.

Mr Paul Hubbard

Higher Education Funding Council for England

Paul Hubbard is currently Head of Research Policy at the Higher Education Funding Council for England (HEFCE). HEFCE distributes public money for teaching and research, to universities and colleges. In doing so, it aims to promote high-quality education and research, within a financially healthy sector. The Council also plays a key role in ensuring accountability and promoting good practice.

HEFCE is working to develop new arrangements for the assessment and funding of research. The new arrangements—the Research Excellence Framework—will be introduced after the 2008 Research Assessment Exercise.

Ms Linda Butler

The Australian National University

Linda Butler is Head of the Research Evaluation and Policy Project (REPP) in the Research School of Social Sciences at the Australian National University. REPP is Australia's leading centre for the systematic evaluation and mapping of research across all fields of scholarship.

REPP was awarded the analytical support contract for the Excellence in Research for Australia (ERA) initiative and will undertake the analysis and empirical testing for ERA indicators and bibliometrics. Ms Butler sits on the Indicators Development Group as a technical adviser.

A major focus of REPP is research on the advanced quantitative analysis of scientific performance. Parallel to this, REPP also recognises the value of developing novel qualitative and quantitative approaches to research assessment, and the need to generate indicators sensitive to the research and dissemination practices of a variety of fields not well served by standard bibliometric approaches. This is particularly in the social sciences, humanities and arts, as well as disciplines in the applied sciences such as computing and engineering.

Ms Butler has also served for a number of years on a National Health and Medical Research Council Working Group which was established by its Research Committee to devise measures of research impact and achievement (MORIA) that can be applied across its funding portfolio.

Ms Butler is on the editorial committees of Scientometrics and Research Evaluation. She has an excellent understanding of bibliometrics and Australian-specific issues.