As governments around the world grapple with an emerging pathogen that threatens to overwhelm healthcare systems and stall the global economy, experts have wondered about the quality of the information that’s available to inform those decisions.
In an editorial published in STAT, John Ioannidis, MD, a noted critic of medical research who co-directs the Meta-Research Innovation Center at Stanford University, wrote that COVID-19 has been called a once-in-a-century pandemic. “But it may also be a once-in-a-century evidence fiasco.” Ioannidis says we lack reliable information on how many people have been or will be infected. Yet governments have taken dramatic steps to curb the pandemic. While people and businesses may be able to tolerate lockdowns for a short time, we don’t know how long these measures should continue if the virus continues to spread. “How can policymakers tell if they are doing more good than harm?” he asks.
We talked to two experts about the challenge of acting when evidence is still developing. Here’s what they had to say about what we know, how firm that knowledge is, and the problem of knowing what to do with it. They are Isaac Bogoch, MD, an infectious disease specialist and clinical investigator at the Toronto General Hospital Research Institute in Canada, and Perry Wilson, MD, an associate professor and expert on evidence-based medicine at the Yale School of Medicine.
WebMD: How difficult is it to be in government right now, to have to make decisions about the COVID-19 pandemic?
Isaac Bogoch, MD: Ultimately, we have to make major decisions in real time without a robust evidence base, and those decisions are going to have significant impact on populations. There isn’t time to wait for a strong evidence base to guide these decisions, so we have to extrapolate from existing data and we also have to factor in common sense and humanity with the policy decisions that are being made.
We know for example, how flu pandemics have played out in the past. We’re drawing on some of that data, but it’s not going to be a perfect fit because this is a brand new virus. We don’t know everything we need to know about it.
There are lots of questions. How long can it live on surfaces? We didn’t know until about a week ago. Do flight restrictions work? We don’t know for this one, but there are data and models from other epidemics.
We have to appreciate the strengths and weaknesses of the data that we’re tapping into and that we’re completely transparent and acknowledge those limitations when we communicate the data.
It’s also reasonable to expect that policy will change significantly throughout the course of this pandemic because there are growing evidence bases that will help public health policy.
One of the issues is communicating to the general public that policy changes aren’t a failure but represent keeping abreast of the most current evidence base.
WebMD: What about modeling? There is some really good and very fast work being done in modeling right now, but it can also feel pretty scary to see those predictions:
Bogoch: Modeling is extraordinarily important, but we have to remember that those are models and they are used to inform policy but there are inherent weaknesses in any model system. They are not crystal balls, but we need them to help us make decisions.
WebMD: What about the evidence behind drugs and therapeutics to treat COVID-19?
Perry Wilson, MD: The evidence that we have right now is not of the quality that we’ve come to expect in the context of the best medical literature. It’s not gold-standard randomized, controlled trials. What the coronavirus epidemic has forced us to do is to come to grips with the idea that the perfect is the enemy of the good. Time is simply not a luxury we have right now. It would be unwise of us to throw out the evidence that’s coming in because it’s not top quality.
We have to take everything that’s coming in with a grain of salt. We have to realize that this data may be telling us something that isn’t true. That isn’t a medical reality. But that’s not a reason to act.
The key is going to be something called biological plausibility—whether or not an agent behaves in a way that we can understand based on what we already know about it.
In large, gold standard randomized, controlled trials, the data often stood for itself.
When we’re looking at much smaller studies in data that’s not collected in as rigorous a way, and we have to ask ourselves, does this data make sense?
One current example of this is chloroquine. The data is very limited on this drug and whether it can treat COVID-19. It’s based on small number of patients. It has not peer reviewed. It has not been randomized. In the old days we would have said ‘Well, come back later when you’ve conducted a rigorous study.’ Now we have to take what we have and look at the biological plausibility. Is it possible it could treat this infection? We have good cell culture data that the drug might have some impact based on how we know about how the virus infects cells and how it acts in the body. So we have to act on that.
Right now, available evidence is worth more than best evidence, because we can only use what’s available.
WebMD: What about the flood of preprints that are being published to open access sites like MedRxiv. There are some scientists who have shared new information via Twitter.
Wilson: I think it is inspiring to see all the work that’s being done on this right now. It feels like trying to go to the moon, this scramble for information. There are so many researchers and scientists who are working very, very hard to turn the tide, and I find a lot of optimism and hope in that. It won’t all turn out to be true, but I think it’s amazing to see the effort.