It can be hard to trust the science when scientists aren’t on the same page.
Last week, the Dana-Farber Cancer Institute, an affiliate of Harvard University, announced it would seek retractions for six papers and corrections for 31 others, after an independent blogger found errors in the research.
The retractions have aroused a longstanding debate about the effectiveness of the peer review process.
To have work published in an academic journal, researchers must first pass the test of peer review, where their peers — researchers in the same field who have earned a PhD — check their work.
However, passing a peer review does not necessarily mean the research is legitimate, the Dana-Farber retractions show.
The quality of academic journals and the rigour of the review process vary significantly, academics say. The process can also be time consuming, and reviewers are almost never paid for their work.
“The peer review system isn’t fool-proof. Reviewers are just like any other researchers and have their own biases,” says Mane Kara-Yakoubian, a PhD candidate in psychology at Toronto Metropolitan University.
“There are papers being retracted so we know that peer review lets bad stuff through. And it also prevents good stuff from getting through, in part because it takes so long,” said Jesse Velay-Vitow, who earned his PhD in climate physics at the University of Toronto last year.
Velay-Vitow waited more than 200 days before reviewers finally found time to review his latest paper.
The median time for a paper to be accepted for publication after peer review in the climate change portfolio of Nature, one of the most prestigious scientific journals, was 217 days in 2022.
“What we have is a crisis of validation. The demand for mechanisms to prevent the publication of junk research far exceeds the current model of anonymous peer review,” he said.
All peer review is anonymous so reviewers can honestly share their feedback.
Replication crisis
In 2015, a now infamous study published in the journal Science showed about one-third to one-half of results in a sample of supposedly high quality psychological research couldn’t be replicated.
Even “exemplary” research may have “irreproducible empirical findings” because of random or systematic error, the study reads.
“Replicability is at the heart of the scientific method. It is a consequence of the foundational assumption that the universe follows unchanging rules. Replicability is what separates science from coincidence and unverifiable assertion,” said Velay-Vitow.
Kara-Yakoubian, of Toronto Metropolitan University, has written on the “replication crisis” as it’s come to be known in psychology.
“It’s important to recognize that no field of study, be it physics, chemistry, biology or psychology, speaks the Word of God, holding absolute and unchallenged truths,” she said.
Psychology “continually identifies and rectifies its own research errors… Recognizing this progressive and self-correcting nature allows us to value its contributions while still maintaining a healthy scrutiny, as should be the case with any scientific field.”
She says psychology researchers have been working hard to address the replication crisis by adopting “open science” practices, such as sharing the hypothesis of the study with a registry before doing the work or by sharing the “null results” of a study.
Null results refer to findings that don’t support the hypothesis or show a lack of statistical significance. Publishing these results shows the researchers’ experiment didn’t work as intended. These results can still be useful despite showing a discovery was not made, says Maxwell Morgan, general counsel for the Structural Genomics Consortium, a research organization at the University of Toronto that is committed to open science principles.
For example, when conducting a “meta-analysis” — a study that combines results from multiple different studies to answer a broader research question — including only the studies with positive results will lead to skewed data.
But publishing null results is a rare practice in most sciences, says Velay-Vitow.
“The market is already flooded with publications,” said Alan Doucette, a professor of chemistry at Dalhousie University. “And I would have to assume there would be 10 times more if everyone published negative results. So how could anyone sift through that?”
Flawed but necessary
Velay-Vitow is a strong proponent of peer reviewers being paid for their work and says the lack of incentives allows bad science to elude peer reviewers.
“Almost all peer review is non-compensated. Some journals pay, but the big ones don’t… There’s also no real training or standards. So if you do a truly abysmal job, the editors won’t ask you to do it again.”
Currently, one of the motivators for peer reviewers is to stay up to date with the latest research, he explains. But, there’s much more need for review than there are people who really want to do it.
“The benefits of making reviewing a highly skilled, well-compensated and prestigious part of science outweighs the costs.”
However, Avinash Mukkala, a PhD candidate in biology at the University of Toronto disagrees. He believes a profit motive could corrupt the process.
“That could create a mechanism to introduce lobbyists and special interest groups to also pay reviewers to get certain work published first. Publishing first is a huge advantage in fast-moving knowledge fields,” he said.
Professors in the normal course of their duties are already paid for their review work, says Doucette, of Dalhousie University.
“Reviewing manuscripts is just part of the broader position that we hold as professors. My work wouldn’t be published if someone hadn’t taken the time to review it, so in turn, I have that role of reviewing other’s manuscripts,” he said.
If his plate is full, Doucette says he will sometimes refuse to review a paper, and there’s no guilt involved.
“The response back is always: we totally understand and hope we can call on you for the next one. So it is a volunteer responsibility, but with the right to refuse,” he said.
“When a reviewer asks if we’ve thought about this additional hypothesis or additional experiment, most of the time those are good suggestions. Almost always, the manuscript changes, and I feel like the changes are for the better.”
Doucette says the peer review process is deeply flawed, but it’s the best thing academics have. “There’s a purpose behind the madness.”
