The inability to reproduce research results is a frequent and costly stumbling block in drug discovery. Reproducibility failures risk the validity of evidence and conclusions and can waste time and resources intended to help potential therapeutics move from the lab to clinical development and commercialization. Christina Brideau, Vice President of Preclinical Pharmacology at Deerfield Discovery and Development (3DC), shared her insights with Cure on the enduring issue of research reproducibility and how to find solutions.
CURE: Research reproducibility is not something people speak a lot about. Can you put this problem in context?
Brideau: Drug discovery is an iterative cycle, and decisions need to be made at every point. You need to understand the data, analyze the data, design the molecules, make the molecules and test them. A huge team of scientists and experts is involved in making these decisions, which requires robust data. At the beginning, especially in industry, we're looking for new targets and novel therapeutics, and we read the literature.
Many of us have had the experience of not being able to reproduce the data. You tend to put it on yourself. You say, ‘I didn't do the work well enough.’ Then, about 10 years ago, publications about reproducibility issues came out from Bayer and Amgen. They were the first ones to publish and recognize that this is a problem that's pervasive. Then, Nature came out with a survey of 1,500 scientists. About 70 percent of the scientists said, “Yes, I've had that problem where I couldn't reproduce the data.”
Cure: Why is reproducibility so important?
Brideau: In industry, we spend a lot of time initiating projects and spending time and money to then realize there may not be a path forward. It ends up wasting time. In fact, there was a study about the economics of reproducibility. In the United States alone, in terms of the life sciences and drug discovery in general, which includes government and industry, they're saying about $50 billion is spent on this effort, and based on these publications, about 50 percent is not reproducible.
Cure: Are there steps you can take to make sure you are not losing time and wasting effort?
Brideau: Whenever we start something new, we try to replicate what's been done from the literature. We try to really understand where are the weaknesses, where are the strengths? Ideally, we shouldn't have to do so much.
Cure: In terms of reproducibility, are there any rules or systems in place about what has to be reproducible before you can actually take it to the next step beyond discovery and into preclinical and then eventually human studies?
Brideau: It's about having that rigor and discipline and being able to communicate the details of protocols. Biology is so complicated. The experiments that you set up are very complex, and you’re having to create a lot of reagents and technologies.
It’s frustrating because we all want to succeed. The academic labs have some really interesting findings and want to progress them. We're trying to use that learning. We all want to discover a drug for the patient. Where it gets challenging is little simple things.
I have an example from working with an academic lab, and it was a very compelling target. They had set up cell lines that we could use. We said, ahead of time, “Can you make sure that your cell lines are not contaminated with mycoplasma?” What do you know, they were contaminated. So, a postdoc spent two months cleaning up the cell line and, at the end of the day, could not reproduce their own data in the lab. Everything went to a grinding halt.
Cure: The FDA is advancing alternative testing methods that reduce or replace animal testing — things like digital twins, like the organoid on a chip. What are your thoughts on the reproducibility of that?
Brideau: It's like what we thought about genome sequencing early on. We thought you sequenced the genome, and that’s it, everything is done. High throughput screening was the same thing. This is the issue with every new technology. There's a big buzz at the beginning, there's a big hope. And then the reality sets in. I think there still needs to be a lot more validation.
Cure: So how do you make sure that you can mitigate as much as possible?
Brideau: One important thing is to have more detailed protocols. When you buy reagents off the shelf from vendors, they're fully validated, so trying to use validated reagents is another area that could be helpful. Then, perhaps more mentoring. There's a lot of expertise around, so seek out those people for more mentoring in the lab. I think training needs to be improved, with a more disciplined way of planning and executing experiments. Powering statistics is another area — statistical analysis and the design of experiments — [of importance].
Cure: At the end of the day, what I'm hearing you say is that science requires discipline, and it requires rigor. It doesn't matter whether you're doing it in a new system or an old system.
Brideau: It's also more training in understanding the intricacies and the technical challenges. You can write lots of notes, but if you don't understand fundamentally every aspect of what you're doing, it could also lead you down the wrong path. A lot of these things are just inadvertent or perhaps even ignorance. You really believe in your data, and you've done everything properly, but if you missed a really significant step or you don't understand a basic function, it's just going to carry through until someone else who's an expert sees that and figures that out.