In December 2015, STAT published a widely discussed article and data set detailing how pharmaceutical companies and prestigious medical research institutions were routinely breaking the US legal requirement to publicly post clinical study results on ClinicalTrials.gov. In this interview, investigative journalist Charles Piller of STAT discusses how he and his colleagues extracted and analysed the data, and explains some of the reasons behind the failure of the National Institutes of Health (NIH) and the Food and Drug Administration (FDA) to enforce federal law.
How did you access the data on Clinical Trials?
Charles Piller: There are two ways to get all the data on clinical trials held by ClinicalTrials.gov. The first is through the Clinical Trials Transformation Initiative, a public-private consortium that periodically provides complete data sets. But these data can be nearly a year old. The second method is to download the data directly from ClinicalTrials.gov. The entire database contains more than 200,000 complex records, and a subset of the fields in each record can be downloaded pretty easily. But we wanted the complete dataset, so we wrote our own code to pull it down from the government site. Experts at Duke, who have been deeply involved in the Clinical Trials Transformation Initiative, provided helpful advice on this. We are happy to share our code with other researchers who want to do their own studies, with the caveat that ClinicalTrials.gov is a work in progress, so some additional coding might be needed.
How did you go about analysing the raw data?
Charles Piller: The next step was to exclude some of the information. We were only interested in clinical trials that were required to report results under Section 801 of the 2007 Food and Drug Administration Amendments Act – many types of trials are not. First we consulted with a great 2015 study in NEJM [the New England Journal of Medicine] that analysed the data in a different way. The authors of that study generously helped us understand their process. There was a substantial overlap between their approach and our own. One big difference was that we were dealing with current data, while their data was years old.
What problems did you encounter when trying to crunch clinical trials data?
Charles Piller: Here are some important challenges we encountered that required painstaking work to resolve: All data is dirty, meaning that errors are made in data entry, and these errors must be corrected where possible. For example, we had to standardize names for each company or university, because several might be used for a single entity. We also had to link all subsidiary companies with their parents. Those methods helped us to ensure fair and correct results. Another problem was that pharmaceutical and device companies often ask for temporary FDA exemptions from reporting their data, to protect intellectual property rights. We didn’t want to say a company improperly withheld data that it was not legally obligated to provide. The FDA tells NIH that an exemption has been requested but not whether the request was granted. This seemed ridiculous, but we decided to err on the side of interpreting the information conservatively – assuming that an exemption was granted even if we didn’t know for sure. NIH was cooperative, and gave us a list of all exemption requests, about 5,000 in total; we excluded those trials from our analysis except in cases where the company submitted results after the exemption expired. This was one of several ways we gave every institution the benefit of the doubt. Even so, we found that some of the world’s leading research entities routinely and flagrantly ignored their obligations to report trial results.
How long did it take you to put together all this data?
Charles Piller: In total it took us six weeks to collect and filter the data, then about two weeks to clean the data and one month to analyse the data. Several people, led by our data expert Natalia Bronshtein and myself, were involved.
You are a news outlet, not a research project. Who funded your research?
Charles Piller: As an independent news organization, STAT self-funds our work, including investigations. We saw an important story, and we allocated the necessary resources to report it.
What was the response from NIH to your report?
Charles Piller: You’d have to ask NIH, but I can say that many of those who work in the ClinicalTrials.gov administration are terrific public servants who care deeply about making the system work. NIH director Francis Collins was unhappy at our critique of his agency’s own staff researchers, because we showed that they were not following their own reporting rules most of the time.
What sanctions does the law envision?
Charles Piller: The FDA has the power to impose fines of up to $10,000 per day on pharmaceutical and medical device companies that fail to disclose trial results. Academics and their institutions could theoretically lose NIH funding if they fail to comply with the law. For big companies, the $10,000 a day fine is not so harsh. It’s more like the change they lost down the side of the couch. Even so, if the federal government had been enforcing the law rigorously, it could have collected a whopping $25 billion from companies through last year. But it has not levied a single fine.
Why are these rules not being enforced?
Charles Piller: The government is working to improve compliance through persuasion and education, but so far has not been willing to use financial penalties. NIH says that once the rules have been refined and finalized – a process that has been going on for years, and might be completed soon, though the deadline keeps slipping – it will begin to bring down the hammer. The idea, they said, is that companies and universities will no longer be able to say that ambiguities in the law are to blame for their failures to report results. Still, NIH is not keen to pick a fight with important academic institutions, some of which have a lot of clout, and naturally the agency doesn’t want to disrupt important research. Meanwhile, the FDA has generally taken a non-confrontational approach to powerful businesses, in part, I think, to avoid the time and cost of fighting protracted lawsuits that might occur if the agency did levy large penalties. We might see stronger enforcement after the final rules are issued, but I’m not counting on it.
Has anything changed since your report?
Charles Piller: One thing we have seen is that naming and shaming the institutions that have the worst records in reporting the results of clinical trials can have an impact. STAT’s report sparked an outcry from top officials of the World Health Organization, Vice President Joseph Biden, former FDA commissioners, and patient-advocate groups. NIH recently said that as a result of STAT’s article, as well as its own educational programs, they saw a 25% increase in results reporting to ClinicalTrials.gov. That’s not nearly enough improvement, but public attention to these issues can and does make a difference.
This interview was conducted on 1st August 2016 by Dr Till Bruckner, campaigns manager at Sense about Science, and written up by Beth Ellis, intern at Sense about Science.