Dozens of major cancer studies cannot be reproduced

Eight years later, a project to recreate the results of major cancer biology research has finally come to an end. And the findings suggest that cancer research, as well as social science research, has replication problems.

Researcher Reproducibility Project: Cancer Biology The aim was to reproduce 193 experiments from 53 top cancer papers published between 2010 and 2012. But only a quarter of those experiments. I was able to reproduce, The team reports in two papers published on December 7th eLife..

Researchers were unable to complete most of the experiment because the team did not come together Enough information from the original paper Or get the author about how to use it, or the materials needed to try to duplicate it.

In addition, of the 50 experiments from the 23 reproduced papers, the effect size was on average 85% smaller than that reported in the original experiment. Effect size indicates the magnitude of the effect found in the study. For example, if two studies found that a particular chemical kills cancer cells, that chemical kills 30% of the cells in one experiment and 80% of the cells in another. In the first experiment, it is less than half the effect size seen in the second experiment.

Sign up for the latest from Science news

Latest headlines and summaries Science news Articles delivered to your inbox

Thank you for signing up!

I had a problem signing up.

The team also used five criteria to measure successful replication. Four focused on effect size, and fifth was whether both the original and repeated experiments showed similarly positive or negative results, and whether both result sets were statistically significant. I checked. Researchers were able to apply those criteria to 112 tested effects from reproducible experiments. In the end, researchers report that only 46%, or 51%, met more criteria than failed.

Jonathan Kimmelmann, a bioethicist at McGill University in Montreal, said:He co-authored Commentary About a project that explores the ethical aspects of the findings.

Kimmelman says he is worried whether irreproducible experiments will be used to initiate clinical trials and drug development efforts. If the science underlying the drug turns out to be unreliable, “it means that patients are unnecessarily exposed to unsafe drugs and can’t even actually affect cancer,” he says. increase.

At the same time, Kimmelmann warns against over-interpreting the findings, as it suggests that the current cancer research system is broken. “In fact, I don’t know how well the system works,” he says. One of the many questions left unanswered by the project is what is the appropriate replication rate in cancer studies, as it is not possible to completely replicate all studies. “It’s a moral question,” he says. “It’s a policy issue. It’s not really a scientific question.”

The project’s comprehensive lessons suggest that the substantial inefficiencies of preclinical studies may be hampering the later drug development pipeline, said Tim Ellington, who led the project. say. He is the principal investigator and co-sponsor of the research at the Center for Open Science in Charlottesville, Virginia.

Much the same as the 19 of 20 anticancer drugs participating in clinical trials It is never approved by the US Food and Drug Administration. It may be due to the drug’s lack of commercial potential, but often because it does not indicate the level of safety and efficacy required for the license.

Many of the failures are expected. “We are human beings trying to understand complex illnesses and never understand them correctly,” says Ellington. However, given the findings of the Cancer Reproducibility Project, perhaps “we knew we had failed before, or maybe we didn’t really understand what was the cause. Maybe. [an] An exciting discovery, “he says.

Still, failure to duplicate does not mean that the study was wrong, and duplicating it does not mean that the results are correct, epidemists at Brigham and Women’s Hospital in Boston and Harvard Medical School. Shirley Wang says. “That means you can reproduce,” she says, also emphasizing the reproducibility project.

Scientists need to assess whether the method of study is unbiased and rigorous, Wang says. He wasn’t involved in the project, but reviewed the results. She adds that if the results of the original experiment differ from their reproductions, it will be a learning opportunity to find out why and what the impact is.

Ellington and his colleagues Subset of findings from previous cancer reproducibility projectsHowever, this is the first time an analysis of the entire effort has been released (SN: 1/18/17).

During the project, researchers faced many obstacles. In particular, none of the original experiments contained sufficient details in published studies on how to attempt reproduction. Therefore, reproducibility researchers contacted the study authors for additional information.

The project found that about one-quarter of the authors were helpful, while another one-third did not respond to requests for more information or otherwise did not help. For example, in one of the experiments that the group could not reproduce, it was necessary to use a mouse model that was specially bred for the original experiment. Ellington says the scientists who conducted the study refused to share some of these mice with the reproducibility project and could not replicate without their rodents.

Ellington says some researchers were completely hostile to the idea that independent scientists were trying to recreate their work. That attitude is a product of a research culture that emphasizes innovation over reproduction, and values ​​scholarly publishing or extinction systems over cooperation and data sharing. Open Science Center And co-author of both studies.

Some scientists may feel threatened by replication because it is rare. “If the replication is normal and routine, people don’t consider it a threat,” Nosek says. But even the livelihoods and identities of scientists are often deeply rooted in their discoveries, so duplication may also feel scary, he says. “Publishing is a currency of progress and an important reward that translates into funding opportunities, job opportunities, and opportunities to continue the job,” says Nosek. “Replication does not fit well into that reward system.”

For a variety of reasons, including lost hard drives, intellectual property restrictions, and data that only former graduate students had, even authors who wanted to help couldn’t always share the data.

Science “Replication crisisHas grown over the years and is probably the most notable In psychology (SN: August 27, 2018)... Then, in 2011 and 2012, pharmaceutical companies Bayer When Amgen Reported the difficulty of replicating findings from preclinical biomedical research.

But not everyone agrees on a solution that includes: Whether a copy of an important experiment is actually useful or possibleOr what exactly is wrong with the way science is done, or what needs to be improved (SN: 1/13/15).

Yvette Seger, director of science policy for the Federation of American Societies of Experimental Biology, said the new findings have led to at least one clear and practical conclusion. It is the need to provide scientists with as many opportunities as possible to explain exactly how they did their research.

“Scientists need to be eager to include as much information as possible about the method to ensure that they understand the results on the other side,” says Seger, who was not involved in the reproducibility project.

After all, for science to be a field of self-correction, not only do we need to make mistakes, but we also need many opportunities to discover those mistakes, including duplicating experiments, the project said. Researchers say.

“In general, the general public understands that science is difficult, and I think the general public understands that science makes mistakes,” says Nosek. “The concern is, is science efficient in catching that error, and should it be?” The findings of cancer projects do not always answer that question, but they are a challenge to try to find. To emboss.