Making scientific evaluations more transparent

Before being published, research work is evaluated by peers, experts who examine the rigour of the approach and the reliability of the results. However, this critical process has some serious flaws. It also completely bypasses the general public, which is deprived of the essential elements for understanding how science is developed.   

Alex Holcombe
Professor at the School of Psychology, University of Sydney, Australia.

A year and a half into the Covid-19 pandemic, science has saved many lives. Without basic biomedical research, the vaccines could not have been engineered and evaluated for safety and efficacy. But the scientific community has not been as clear on other critical issues related to the pandemic. The evidence on the usefulness of masks, for example, and on the reliability of models of viral transmission, has been tentative. This is partly because many of the relevant studies have been flawed.

Science is complex, and it is easy to conduct a scientific study badly. To effectively evaluate the implications of these studies, we need experts to review  them. This is called the peer review – the traditionally confidential process for evaluating new scientific findings. As a researcher, I know from experience that other researchers often see flaws in my work that have eluded me. But the peer review process itself can also be flawed. 

The most recent scientific findings are only trustworthy when their authors have had time to submit them for additional tests, and when multiple experts have had a chance to scrutinize them. When new research is published, scientists and others can only see the writings of the researchers who conducted the study, and nothing more.

Behind closed doors

Traditionally, the peer review process begins after a team of scientists submits a manuscript describing new results to a scientific journal. An editor, usually an academic from another university, reads it and decides whether the work meets the standard of the journal. If it does, the editor will recruit experts on the topic being evaluated.   

Often, one or more of the experts will then provide the journal with a long list of comments and criticisms, which are forwarded to the authors. The comments could push back against our conclusions and challenge us to make our case more strongly, or to soften some of our claims.

As an author, I don’t relish seeing criticism of my work, but I know it’s necessary for scientific progress. After correcting some of the flaws in our logic, or substantiating the assumptions that have not been justified, we can be more confident about our conclusions, and hope they will have a greater impact. 

The back-and-forth that follows between the authors, editor, and expert reviewers results in a final manuscript, which analyses the data more rigorously and usually offers a more cautious take on its implications – whether or not it is finally accepted for publication.  

Unfortunately, the public never discovers which elements of the study were called into question, or which aspects of the debate were the most controversial. This is because the peer review process happens behind closed doors – within the password-protected walls of the journal’s database. To know where different experts stand on the issues, it is imperative that the public is aware of the conflicts of opinion that arise during peer reviews.

Disputed studies 

In 2020, two studies of the impact of hydroxychloroquine and blood pressure medications on the progress of Covid-19 were retracted after publication and conventional peer review. The Lancet and The New England Journal of Medicine (NEJM), two of the world’s most respected medical journals, both had experts examine the studies in peer review. The NEJM accepted the article after receiving comments from four experts, and, as is standard, any uncertainty expressed or questions posed during the peer review were not made public. 

We still do not know what the experts said. What we do know is that many scientists who were not included in the peer review process immediately detected signs that the data might be questionable, and wrote critical letters to the journal within days of its publication. 

In this case, the huge interest in the subject motivated experts to review the article and write in with their concerns. Unfortunately, this is quite unusual.  

Away from the high stakes of pandemic-related studies, evaluations of research are rarely made public. In 2013, researchers believed that they had created the smallest-scale reconstruction yet of proteins on the surface of HIV, the virus that causes AIDS. Their article was published in one of the world’s most prestigious journals. But as we later discovered, at least four other journals  had previously rejected the paper – some of them because of strong criticism from their peer reviewers. None of these were made public. It was only because other experts were approached by journalists that the scientific community learned about the case.

In response to growing criticism, many journals are now opening up – giving outsiders a chance to view the live disagreements that are characteristic of cutting-edge science. And many researchers have begun posting their manuscripts on the internet before submitting them to journals. Websites for scientists to discuss and criticize new research now buzz with activity.

Editors at the journals are finding it increasingly difficult to find experts for the traditional review process. Many editors tend to rely on experts they already know, and on experienced researchers who are highly sought after, and therefore unable to keep up with the demand.  This slows down the peer review process, and does not reflect the changing demographics of the scientific community. 

Today, more women and minorities are involved in the sciences, and contributions from developing countries – such as China – are increasing rapidly. But many of these researchers remain off the radar of the senior scientists who edit journals that are located mostly in North America and Europe.

Women and researchers from developing countries remain off the radar of science editors

Over time, these problems may be resolved with the emergence of  new avenues for criticism and commentary. Some journals now invite experts to make their comments public on their associated websites. The development of this practice could promote diversity in peer reviews.

A better understanding of science  

It is the dialogue among experts within the research community that defines the frontier of knowledge – that conveys the uncertainty associated with the effectiveness of a new vaccine, the  forecast of more droughts in a country’s future, the credibility of dietary advice, or an estimate of the economic consequences of tariffs. Giving journalists and the public access to some of the debates among experts would mean that media reports are more accurate – leading to a better understanding of science in general. 

The experts will also benefit. Having access to some of the peer review comments, researchers would be less likely to assume that a finding is totally solid, and more likely to reconsider their conclusions. Researchers will benefit. So will science. 

Read more:

Research: “This epidemic will be a detonator”, The UNESCO Courier, July-September 2020
SESAME: Scientific excellence in the Middle East, The UNESCO Courier, October-December 2018
Ada E. Yonath: “The challenge of science is like climbing Mount Everest”, The UNESCO Courier, January-March 2018

Subscribe to The UNESCO Courier for thought-provoking articles on contemporary issues. The digital version is completely free.

Follow The UNESCO Courier on: Twitter, Facebook, Instagram