z-Oped-090606

Editorial - 27 February 2013
To view previous Editorials click here

Over the years (decades?) there has been much discussion as regards the form and efficacy of peer review, but without significant improvement in its utility. Rather like discussing the weather.

The importance of peer review when it comes to awarding public funding for research proposals is inarguable while its importance in reviewing presentations for print or electronic media as regards the advancement of research per se is rapidly diminishing, whether or not open access, whether or not the standard is green or gold. The increasing importance of arXiv.org and its ilk as vehicles for research findings is testament to that -- while the importance of face to face interactions with one's research peers has in no way diminished especially as the means are increasingly augmented.

Which is not to say that the journalistic sector will not continue to serve a useful purpose as regards archiving research both good and bad.

Which leads us to Australian Chief Scientist Professor Ian Chubb's address to the "Peer Review Symposium"

of the National Health and Medical Research Council (NHMRC) on February 20, 2013:

[It remains to be seen what the Australian Research Council (ARC) intends to do.]

...our peer review needs to be of the highest international standards. And more importantly, it needs to be beyond reproach.

Only then can our assessment of the quality, in your case of our health and medical research, be as robust as we need it to be.

And raising public awareness of peer review, and what it means, might also have a role to play in lifting the general population’s trust in science.

Which is why the NHMRC’s draft Principles of Peer Review, the basis of your consultation process, are so important*.

Those principles – fairness, transparency, independence, appropriateness and balance, research community participation, confidentiality, impartiality and continuous improvement – will give our system of peer review acceptance, respect and trust.

I note the Australian Academy of Science talks about reducing the burden of peer review and remarks that the amount of time researchers are spending on peer review might be impacting the time available for research. Probably true; for the best/most experienced.

I don’t think that it is to dispense with, or diminish further, peer-review. But it is to understand it. And explain it, and its value.

I think the public needs to know the steps we take to assure that standards are met before publication, and that the peer community responds after publication. That robust debate is a part of science.

This need for understanding will increase as peer review evolves as the conduct of research changes.

But it does seem to me that we need to look at peer review in several parts. And I am sure that you will have considered these – but let me summarise in my terms.

So, what is a peer? How close do they have to be to the proposal, the manuscript, the application under review?

What are they looking for: that the research was (or will be) conducted diligently, ethically and that the conclusions are (or will likely be) justified by the observations that have been properly analysed?

Another part is the response of the peer community once it is out there. It is likely to be much larger – and some of them are more than likely to be much closer to the actual work in the actual field.

And I think I could argue that part of the process – call it peer or scientific – is that the work is accepted, or not, once it has been scrutinised, replicated or otherwise, and stood the test of time in that (critical) environment.

It is after all true, that good scientists take a while to be satisfied – or put another way – good scientists take little at face value but see data as an opportunity to have a robust debate about the value, the interpretation, and the conclusions that can, or should, be drawn.

I remember a celebrated case a few years ago in the UK. As I recall, it was the peer community that identified the flaws in a paper that changed the vaccinating habits of a fair number of parents.

It was not an op-ed writer in a newspaper who produced the evidence. It was peers who were sceptical and did something about it. Experts who did something about it – tested, analysed, interpreted and found the flaws,

It takes courage to stand up to be counted. But it is the standing up that is an important part of our peer system. The review, the analysis, the critique, the confirmation or the change, the replication or the failure to replicate can all bring you into conflict – within the peer group and outside.

The change to our thinking that comes from the process is a key to understanding and advancement. It can’t stop.

[Note: Public Consultation on the NHMRC Draft Principles of Peer Review Closed on Friday, 1 February 2013, 05:00pm (AEDST). Submissions for this public consultation are no longer being accepted.

However, not until the National Health and Medical Research Council develops the mechanisms for true international peer review of the research proposals submitted to it will it even begin to honour its stated "Principles of Peer Review" (see below).

A mediocre implementation of peer review for assessing research proposals is not merely useless, which would be bad enough, it is counterproductive.

___________________________________