“Don’t Look Up” and rethinking the depiction of peer-review in pop-culture
By Maanasa Koripalli
Officially the movie “Don’t Look Up” is an allegory about climate change as it tells the story of a corrupt political environment that trivializes the discovery of a "planet-killing" comet and undervalues the advice of experts. Yet it was the emphasis placed on peer review that surprised me the most in the movie. In failing to use peer-review, it is implied BASH, the fictional largest tech company dedicated to stopping the comet from destroying life on Earth, is completely unreliable. But what exactly is the peer-review process, and to what extent is it an effective tool to judge research that can arguably have world-saving implications?
Researchers often refer to peer-review as "the gold standard" which is to say an article published in a peer-reviewed scientific journal has a kind of professional approval. In some respects, this is reasonable. Scientific publishing relies on reviewers and editors to eliminate the most uninteresting or least worthy articles to improve the quality of papers published or determine which research proposals should be funded.
But peer review, a practice dating to the 17th century, is neither golden nor standardized. The flawed system that has become synonymous with quality has many challenges. We shouldn’t take the peer review process as the gold standard as the movie makes it out to be.
Rethinking Peer-Review
A common misconception about peer-review is that it is highly objective, reliable, and consistent. In 1982, Douglas Peters and Stephen Ceci led a study to determine how unbiased and reliable this process is. They carefully selected twelve papers accepted by prestigious journals two to three years earlier and changed the names and affiliations of the authors. After resubmitting the papers to the same journals, only 8 percent of editors or reviewers noticed the duplication, and 89 percent of reviewers recommended rejection for the remaining papers.
Peer-review is also extremely subjective and influenced by individuals' motivations. Because journals receive a huge volume of manuscript submissions covering a wide variety of topics, different reviewers will inevitably take differing views on a manuscript's strengths, weaknesses, and importance. As a result, gatekeeping decisions tend to be influenced by the reputation of the university and the past publication history of authors. We see this in the movie when the President decides to only trust the findings once they have been confirmed by experts from Ivy League schools.
Perhaps the most powerful criticism of peer review is that it fails to achieve its core objective: quality control. The bedrock of all empirical research is the data itself. Going through someone else’s data is excruciating and time-consuming, which is often why it is skimmed over in the peer-review process. Therefore, the analytical methods and algorithms behind most papers are never put to the test. A famous example of this is in 1982 when NASA programmers for Mariner 1 made a typo that resulted in the thrusters firing too soon and exploding the rocket. Many other examples of data scandals and dodgy papers are available on websites like PubPeer and Retraction Watch.
The Future of Peer-Review
The traditional peer-review process has many shortcomings, but we should not completely disregard it; rather we should be innovative in our efforts to improve peer review. The internet has made it possible to add more verification steps in the process of publication. This includes pre-print commenting and post-publication peer-review, which allows more people to view the study to provide an additional level of scrutiny before it undergoes traditional peer-review.
During the COVID-19 pandemic, pre-print manuscripts have gained a new level of notoriety and importance. Within the first 10 months of the pandemic alone, two of these sites, bioRxiv and medRxiv, hosted 25 percent of the scientific research related to COVID-19. Ivan Oransky, co-founder of Retraction Watch said "Post-publication peer-review is nothing new, but in the past it's happened in private, with no feedback for the authors or larger scientific community." Sites like bioRxiv and medRxiv open up the process and make it more transparent and should therefore be strengthened.
As a public health student, I believe peer-review and the publication process is perhaps more pertinent than ever in this era of “fake news” – and not just for scientists and academics. Unless we have an adequate peer-review process, how can we be sure of the science that shapes government policies such as lockdowns and social distancing measures?
In the end, BASH's technology failed to stop the comet, but had it been peer-reviewed, could we have been certain that it would have worked? Theodore Fox, who was the editor-in-chief of the Lancet journal, once said, "If I stacked the week's contributions into two piles, one for publication and another to return, do you think it would make a difference?"
We must rethink a flawed system with potentially world-saving implications, and support ways to improve it.
Acknowledgments:
We want to thank Dr. Ananya Tina Banerjee for encouraging her students to write and sharing this opinion article with us.
About the Author:
Maanasa Koripalli is a first-year MSc Public Health student in the Department of Epidemiology, Biostatistics, and Occupational Health. While completing her bachelor's degree at McGill in Honours Biochemistry, research has been the most consequential and rewarding component of her experiences. She wanted to pursue public health because it bridges the gap between research and the world outside of academia and wishes to explore a career in health policy.