The Mismeasurement of Science and The Granting System. (September 17, 2009)

Peter Lawrence, FRS holds appointments in the Department of Zoology University of Cambridge and the Medical Research Council Laboratory of Molecular Biology, Cambridge.

Two years ago TFW called attention to a paper by Professor Lawrence entitled "The Mismeasurement of Science", and we wrote: "The three page article ought to be required reading for every individual who has been caught up in the fiasco of the Research Quality Framework (RQF)." While the RQF has acquired a name change to become the ERA and the Minister for Science a sex change, the fiasco shows now signs of being dispatched.

In explanation, the News and Views item noted that in a careful critique which damns Britain's Research Assessment Exercise Professor Lawrence writes: "Over recent years, within governments and outside them, people have lost sight of the primary purposes of institutions, and a growing obsession with internal processes has driven more and more bureaucracy — such as increasingly complex grant applications and baroque research assessment exercises — at the expense of research effort. This bureaucracy is placing heavy demands on scientists that lay waste their sense of purpose and attack their self-esteem. But scientists of all ranks, senior as well as junior, are also to blame as we have meekly allowed this to happen."

Now Professor Lawrence has published a four-page perspective in PLoS Biology: Real Lives and White Lies in the Funding of Scientific Research: The granting system turns young scientists into bureaucrats and then betrays them.

We have already referred to the paper and a Blog in Science in which it is discussed. Here we reprint Professor Lawrence's suggestions for improving the grant peer review system in the hope that not only the ARC but also the Minister for Innovation, Industry, Science and Research, Kim Carr, as well as those he has burdened with trying to shape the ERA will consider them seriously.

[Part of the problems are] due to the time it [takes] to write... applications and the long lag period between submission and decision. Drastic simplification of this grant-writing process would help scientists return to the business of doing science. Grant applications should be much shorter, and if so, scientists would spend less time writing them and less time reviewing other applications. Since the failure rate is high, these savings of time and effort would be very substantial—in Cambridge University, an average scientist writes about three applications to get one grant; similarly, in the United States ‘‘after multiple submissions and a protracted process, only about 20% of grants will ultimately be funded’’, and the success rate is dropping.

It would help free up time for research if scientists were not forced to dream up future plans but instead could opt to present their recent publications. Even some first-time applicants with a strong recent record... could take this option; others might prefer to write projects. Grants should last for longer; three years is too short, and having five-year renewals would itself reduce the burden of grant writing.

There should be a presumption against large groups, and people who aim to run them should demonstrate both efficiency and effectivity; they should show that they have enough time to run each of their grants and to care for each of their people. We need to reduce the bureaucracy as well as the delays it generates: the right approach to too much paperwork is not to appoint expensive offices to cope, but to cull it. Rationalisation would save a lot of money that could be put into the grants themselves rather than into their administration.

An important reform would be to set a maximum limit on the number of papers that could be offered as part of any application or any assessment. At the moment, evaluating individuals and departments rewards those who produce many articles, mainly because counting papers is so much simpler than reading them. Over 20 years, this mismeasurement of science has wrought a sea change in practice: no longer are communication and record the primary purposes of publishing; instead, we now use papers as tokens to get jobs and funding. This same sea change has fuelled a huge increase in the number of papers and journals and decimated their quality and utility. The solution is to allow, say, only three to five of the best papers of the group from over the previous five years to be offered for assessment (as Howard Hughes Medical Institute already do in the US). The evaluation of these papers should be corrected for the size of the group, i.e., productivity would be rated per person, not per group. If these reforms were enacted, the pressure to rush out many papers would be replaced by pressure to complete projects that report stories of value and present them well. In consequence, the literature would be transformed and improved, and we would all benefit. But this reform would create a problem. At the moment, young people need a paper as a ticket for the next step, and we should therefore give deserving, but unlucky, students another chance. One way would be to put more emphasis on open interviews (with presentation by the candidate and questions from the audience) and references. Not objective? No, but only false objectivity is offered by evaluating real people using unreal calculations with numbers of papers, citations, and journal impact factors. These calculations have not only demoralised and demotivated the scientific community, they have also redirected our research and vitiated its purpose.

In the quotations Professor Lawrence includes in his paper is the following:

‘‘People do not realize that when it comes to arguing their case for more funding,
scientists who do basic research are the least articulate, least organized, and least
temperamentally equipped to justify what they are doing. In a society where
selling is so important, where the medium is the message, these handicaps can
spell extinction.’’—Arthur Kornberg, Nobel Laureate in Physiology or Medicine
(deceased 2007)