A study published yesterday in the open access journal PLOS Medicine suggests that pharmaceutical companies often fail to tell doctors and patients the full story of the negative side effects of the drugs they sell. Comparing seven previously undisclosed clinical study reports from trials of the slimming drug orlistat (usually marketed as ‘Alli’) with publicly available information, the researchers found that data on adverse effects was neither thoroughly gathered nor fully reported, concluding that “[t]he reporting of trials of orlistat in the 1990s understated harms”.
In this interview, study co-author Jeppe Bennekou Schroll explains his team’s struggle to get hold of the data, and warns that “[p]oor recording, analysis, publication and interpretation of harms has probably caused suffering and death of millions of people throughout the world.”
How long did it take you to get the clinical study reports (CSRs) from the European Medicines Agency (EMA) through a freedom of information request? Were there any notable hurdles, delays and/or costs involved?
Jeppe Schroll: Our colleague Anders Jørgensen together with Peter Gøtzsche applied for the CSRs for orlistat in June 2007. The EMA denied us access with reference to commercial interests and we appealed the decision to the European ombudsman with reference to public interest. The ombudsman asked the EMA for convincing arguments that data should not be released, but the EMA repeated their argument several times and denied to release the data. In June 2010, the ombudsman issued a press release accusing the EMA of malpractice, and a few months later the EMA agreed to release the CSRs. We received them in February 2011, more than three and half years after our initial request.
In a brief period following 2011, access was quite easy to obtain, but it has since been halted.
Would it be feasible to make all old CSRs in the EMA archives public by filing a freedom of information request for each one and then putting them all online?
Jeppe Schroll: The data did not come with any conditions and we have released it together with our PLOS Medicine paper. Currently the process is very lengthy and it might become even lengthier if we request all drugs rather than targeting “troublesome” drugs.
How easy would your study be to replicate? How much time did it take your team to analyse the data and write your paper?
Jeppe Schroll: Our protocol is available but the data extraction takes time as the documents are huge and not searchable. For one of the trials we used text recognition software on roughly 100 pages to be able to reanalyse harms data. Cleaning up data took more than a week. This paper has been through a very rigorous peer review process including legal reviews which took almost a year.
In analysing the clinical trial data, did you experience any difficulties? Is there any data that you were not able to access or that is missing?
Jeppe Schroll: Some of the modules of the CSRs were in fact missing in our dataset. However, we had the table of contents for the missing data, and we were able to estimate that it would not affect our analysis. We did not get communications between EMA and Roche and we cannot know if data was shared that could be of interest for our project.
In your personal opinion, do you think the regulators at the EMA spotted the discrepancies that you observed themselves during the approval process for orlistat, or not?
Jeppe Schroll: It is very unlikely that the EMA discovered the difference in duration of adverse events in the orlistat and the placebo arm. The EMA relies on the analysis conducted by the sponsor and typically does not do their own statistical analysis. In some situations they might ask the sponsor to conduct additional analyses.
Your paper states that harms caused by orlistat were “understated”. Can you give some examples of these harms?
Jeppe Schroll: The harms of orlistat are primarily diarrhoea and incontinence. Diarrhoea was recoded to “increased defecation” which was not specified in the protocol. Another post hoc change was that gastrointestinal adverse events should only be coded if they were considered “bothersome to the patient”. A condition that was not required for any other adverse events.
In the summary reports harms were downplayed. It was for instance stated that adverse events were “related to the pharmacological effect of drug” which is obviously beside the point. The summaries also typically reported a number of patients with adverse events which was almost 100% in both groups due to many follow up visits. Over a year most people will experience headache, muscle pain and therefore the difference between placebo and orlistat disappears. Instead they should have reported total number of adverse events which would make it clear that orlistat is responsible for many adverse events. In our estimation an average patient receiving orlistat had adverse events almost every single day. This is in accordance with observational studies that show that only 2% still use orlistat after two years.
Orlistat can still be bought over the counter. Your paper warns that “[s]elective reporting of harms can have disastrous consequences”. How dangerous are the understated harms in this case, and do you feel that regulators should re-examine the decision to allow over the counter sales?
Jeppe Schroll: In our analysis we did not find clear signals of serious adverse events. However, the patients were followed very closely in the trials. The inclusion criteria were strict and abnormal blood tests, ECG [electrocardiogram], ultrasounds, vitamin levels would lead to exclusion and therefore discontinuation of the drug. Without convincing data that orlistat is safe in a less controlled environment, over the counter sale should be revoked.
Beyond orlistat, how widespread do you think this problem is?
Jeppe Schroll: The problem of downplaying harms is probably one of the worst problems in medicine today. Poor recording, analysis, publication and interpretation of harms has probably caused suffering and death of millions of people throughout the world.
Do you think that your research findings may expose some of the stakeholders in these trials to legal challenges in the future?
Jeppe Schroll: Unfortunately no. The guidances on how to collect, report and analyse harms are way too open to interpretation, and even though it seems as if the drug sponsor has systematically chosen the option that makes the drug look more safe, it is still within the wide boundaries set up in the template. The regulatory agencies have – in my opinion – failed to protect citizens by demanding that harms are prioritised when the companies fail to do so.
What changes to policy and/or legislation would you like to see to make sure that the side effects of drugs do not remain hidden?
Jeppe Schroll: Clinical study reports should be made public as should all communication between sponsor and regulators. This should also include drugs that are rejected or withdrawn. Regulatory agencies’ reports should also be made public and open hearing processes could give patients and citizens a chance to share their concerns before a drug is approved.
This interview was conducted on 16th August 2016 via email by Dr Till Bruckner, campaigns manager at Sense about Science, and Beth Ellis, intern at Sense about Science. The paper discussed was published in the open access journal PLOS Medicine on 16th August 2016 under the title “Assessment of Adverse Events in Protocols, Clinical Study Reports, and Published Papers of Trials of Orlistat: A Document Analysis” and can be accessed here. The original AllTrials blog post reporting the study and initial reactions is here.