Research
Andrew P Prayle, NIHR doctoral research fellow,
Matthew N Hurley, Wellcome Trust paediatric clinical research fellow,
Alan R Smyth, professor of child health
Abstract
Objective
To examine compliance with mandatory reporting of summary clinical
trial results (within one year of completion of trial) on
ClinicalTrials.gov for studies that fall under the recent Food and Drug
Administration Amendments Act (FDAAA) legislation.
Design Registry based study of clinical trial summaries.
Data sources
ClinicalTrials.gov, searched on 19 January 2011, with cross referencing
with Drugs@FDA to determine for which trials mandatory reporting was
required within one year.
Selection criteria Studies registered on ClinicalTrials.gov with US sites which completed between 1 January and 31 December 2009.
Main outcome measure Proportion of trials for which results had been reported.
Results
The ClinicalTrials.gov registry contained 83 579 entries for
interventional trials, of which 5642 were completed within the timescale
of interest. We identified trials as falling within the mandatory
reporting rules if they were covered by the FDAAA (trials of a drug,
device, or biological agent, which have at least one US site, and are of
phase II or later) and if they investigated a drug that already had
approval from the Food and Drug Administration. Of these, 163/738 (22%)
had reported results within one year of completion of the trial compared
with 76/727 (10%) trials that were not subject to mandatory reporting
(95% confidence interval for the difference in proportions 7.8% to
15.5%; χ2 test, P=2.6×10−9). Later phase trials were more likely to report results (P=4.4×10−11), as were industry funded trials (P=2.2×10−16).
Conclusion Most trials subject to mandatory reporting did not report results within a year of completion.
Introduction
A
key principle in the good conduct of clinical trials is that a summary
of the trial protocol should be freely available while the study is
ongoing and that, on completion of the study, the results should be
readily accessible to all in a timely fashion. In February 2000 the Food
and Drug Administration (FDA) Modernization Act (1997) prompted the
creation of a national clinical trials registry (ClinicalTrials.gov).1 2
Similar databases (such as the ISRCTN) have been established elsewhere.
From 2005 the International Committee of Medical Journal Editors
(ICMJE) required that clinical trials should be indexed in a clinical
trial registry to qualify for publication in a journal following the
uniform requirements for manuscripts.3
Subsequently, the FDA Amendments Act (FDAAA) of 2007 required
registration of summaries of trial protocols for “applicable clinical
trials” (trials that are covered by the FDAAA).4
These are trials that have at least one site in the United States; are
of a drug, device, or biological agent; and are “initiated or ongoing as
of September 2007,” excluding phase I studies and early feasibility
trials of devices.5 Clinical trials are registered with ClinicalTrials.gov (clinicaltrials.gov) by “responsible parties” and uploaded to the website by using the Protocol Registration System (http://prsinfo.clinicaltrials.gov).
The uploading of trial results is achieved in a similar fashion and
reviewed by a Protocol Registration System administrator before
publication on ClinicalTrials.gov. At present, clinical trials of drugs
that already have FDA approval are required to report results within one
year of completion of the trial (with some provisions for delayed
reporting), although in the future applicable clinical trials of
unapproved drugs or biological agents that are regulated by the FDA may
also be required to report results.6
These results are posted in the form of a table of values for each of
the pre-specified primary and secondary outcome measures for each arm of
the clinical trial, with associated statistical tests.
This new legislation should help to overcome the problem of trials that are done but not reported.7 Zarin et al have recently described the current activity of ClinicalTrials.gov,8 and Wood has given a thoughtful discussion of the current legislation,6
but no systematic evaluation of compliance with the FDAAA has been
published (to the best of our knowledge). Zarin et al recently commented
that the “usefulness [of ClinicalTrials.gov] depends upon the research
community to submit accurate, informative data.” While searching the
ClinicalTrials.gov database, we noted that studies under the
jurisdiction of the FDAAA had not yet reported basic results. We decided
to look into this further.
Studies of a drug, device,
or biological agent with a site in the United States that completed in
2009 are subject to the FDAAA regulations and are required to submit a
protocol summary to ClinicalTrials.gov. Of these, most trials of drugs
approved by the FDA are subject to mandatory reporting within a year.
Our aim was to systematically assess the compliance with mandatory
reporting on ClinicalTrials.gov of interventional clinical trials
falling under the FDAAA and to look for evidence of reporting bias by
using publicly available datasets.
We did a database
search of trials registered on ClinicalTrials.gov which completed in
2009 and are covered by the FDAAA. To identify the subgroup of these
studies for which mandatory reporting is required within one year of
study completion, we cross referenced our search results with Drugs@FDA
(the US database of FDA approved drugs).
Methods
Not
all trials registered with ClinicalTrials.gov are covered by the FDAAA,
and only a proportion of those covered by the FDAAA are required to
report results within one year (“trials subject to mandatory
reporting”). We identified this group of trials as follows. We searched
ClinicalTrials.gov (search date 19 January 2011) for all intervention
studies. We identified completed studies by using the “primary
completion date” field (or, where this was not recorded, the “completion
date”). The primary completion date, as defined by ClinicalTrials.gov,
is the date of collection of primary outcome data on the last patient to
be enrolled. The completion date is the date of completion of the
study, as defined by the individual trial’s investigators. We selected
trials subject to mandatory reporting (within one year) which were
completed between 1 January and 31 December 2009. We chose these dates
because all the studies completing within this timeframe would have had
at least one year in which to report results.
We
excluded phase 0 and I trials, as well as trials that did not report a
phase. We used a script to programmatically obtain the study site(s) for
each trial. We excluded studies that did not have a site in the United
States. Additionally, we excluded studies that were not clinical trials
of a drug, biological therapy, or device. The remaining studies
therefore consisted of trials that are covered by the FDAAA.
Trials
of drugs that are not approved by the FDA at the time of the 12 month
deadline are not yet required to be reported. We cross referenced the
investigational drug for each of the applicable clinical trials with the
FDA database of approved drugs, Drugs@FDA (www.accessdata.fda.gov/scripts/cder/drugsatfda/index.cfm), to identify trials of approved drugs (hereafter termed “trials subject to mandatory reporting”—see data supplement (http://dx.doi.org/10.5061/dryad.j512f21p)
for further description and examples of this classification). Owing to
the nature of the records available on Drugs@FDA, this removed devices
and some biological agents from the mandatory reporting dataset. For the
primary analysis, we categorised a drug as being FDA approved if the
active compound listed on ClinicalTrials.gov was listed as an approved
drug on Drugs@FDA. For example, if tobramycin was administered by
injection in the trial, we classified this as an approved drug, as
tobramycin products administered by injection are approved on Drugs@FDA.
However, for a trial in which tobramycin was administered as a dry
powder for inhalation, we did not classify this as an approved drug, as
this formulation was not approved on Drugs@FDA at the time of our
search. One investigator (APP) classified all drugs. Another
investigator (MNH) classified a 10% sample of these to measure
inter-rater agreement.
We occasionally had difficulty in
deciding whether a generic drug listed in a ClinicalTrials.gov record
was in fact a formulation approved by the FDA. For this reason, we did a
subgroup analysis in which only drugs approved by the FDA and
identified by brand name on ClinicalTrials.gov were considered approved
drugs.
We compared the reporting rate of trials subject
to mandatory reporting with trials that fall under the FDAAA but are not
subject to mandatory reporting. We examined the relation of phase of
study and funder of the trial with reporting of results for trials
subject to mandatory reporting.
We imported the dataset
from ClinicalTrials.gov into Microsoft Excel. We used R (R-Foundation
for Statistical Computing, Vienna, Austria; version 2.12.1) to
“webscrape” additional information from the ClinicalTrials.gov website.9
We analysed the final reported dataset by using R. Further details of
the categorisation of trials, the complete dataset, and R scripts to
produce the results that we report here are available as data
supplements (http://dx.doi.org/10.5061/dryad.j512f21p).
Results
At the time of our search, the ClinicalTrials.gov registry included 83 579 entries for intervention studies (figure⇓).
Of these, we identified 31 556 as having “completed,” and 5642 had
completion dates between 1 January and 31 December 2009. We excluded
phase 0 and I studies (n=1098), as well as trials that did not indicate a
phase of clinical study (1170), those that did not have a US site
(1752), and trials that were not of a device, drug, or biological agent
(157). This established a group of 1465 trials covered by the FDAAA, for
which more than a year had passed since completion in which to report
results.
Flow diagram to show selection of trials
Reporting
for applicable trials is (at present) mandatory only for drugs,
biological agents, or devices approved by the FDA. By cross referencing
with Drugs@FDA (www.accessdata.fda.gov/scripts/cder/drugsatfda/index.cfm),
we found that 738 trials covered by the FDAAA were subject to mandatory
reporting. APP categorised drugs from trials as previously approved by
the FDA. MNH repeated this classification for a sample of 150 (10%) of
the records. Disagreement resolved by discussion led to a change in
classification in 3/150 (2%) of records.
Of the 738
trials that were classified as subject to mandatory reporting, 163 (22%)
had reported results. In comparison, 76/727 (10%) trials covered by the
FDAAA but not subject to mandatory reporting had reported results (95%
confidence interval for the difference in proportions 7.8% to 15.5%; χ2 test, P=2.6×10−9).
The
proportion of trials subject to mandatory reporting that had reported
results was influenced by the phase of the study (table 1⇓). Fewer phase II trials had reported results compared with phase III and IV trials (χ2 test, P=4.4×10−11).
We categorised the funder of the trial as industry, mixed, National
Institutes of Health/government, or other and found that the funder of
the trial influenced the proportion that had reported results (Fisher’s
exact test, P=2.2×10−16) (table 2⇓).
Industry funded trials were more likely to report results. We therefore
grouped trials subject to mandatory reporting into “solely industry
funded” and “not solely industry funded.” More solely industry funded
studies (126/317; 40%) had reported results than had those not solely
industry funded (37/421 (9%); 95% confidence interval for the difference
24.7% to 37.3%; χ2 test, P=2.2×10−16).
View this table:
Table 1
Number of trials subject to mandatory reporting which had reported results, grouped by phase of study
View this table:
Table 2
Number of trials subject to mandatory reporting which had reported results, grouped by funder of study
We
did a subgroup analysis to determine the proportion of trials that gave
the brand name of the investigated drug on the ClinicalTrials.gov
record (for which we had more confidence in our categorisation of the
drug as being approved by the FDA) which reported results. Of the 738
drug trials that we had classified as being subject to mandatory
reporting, 347 trials gave the brand name for each and every drug in the
trial. Of these 347 trials, 96 (28%) had reported results.
Discussion
Reporting
of summary results on ClinicalTrials.gov is an important step forward
in reducing bias in the literature. We have cross referenced the
ClinicalTrials.gov and Drugs@FDA databases to produce a dataset of
trials that should have reported results at the time of our search, and
we found that only 22% of trials had done so. If the reporting rate does
not increase, the laudable FDAAA legislation will not achieve its goal
of improving the accessibility of trial results.
The
influence of funding body and sponsor seems to be considerable. Industry
funded trials subject to mandatory reporting were more likely to report
results compared with other funders. Phase III and IV studies seem more
likely to be reported than phase II studies.
Comparison with literature
Many potential sources of bias may skew the literature.10 11
Apart from citation bias, these biases can be ascribed to the
investigator, sponsor, and publisher. Although studies with positive
results are published more quickly,12 13 14
no significant relation seems to exist between trial results and time
between submission of a manuscript to a journal and publication.15
ClinicalTrials.gov allows dissemination of summary results independent
of a publisher. Our study supports the suggestion that study
investigators and sponsors act as the principal sources of reporting
bias; reporting of results to ClinicalTrials.gov is independent of peer
review, manuscript preparation, and editorial priorities.
Strengths and limitations of study
The
FDAAA requirements for mandatory reporting came into force during 2008.
By selecting studies that completed during 2009, and doing our search
in early 2011, we have ensured that the trials completed during the
period when the FDAAA was applicable and that at least one year had
elapsed for data to be submitted to ClinicalTrials.gov. We are reliant
on responsible parties from each study uploading accurate data to
ClinicalTrials.gov. We believe that our search in January 2011
represents the earliest reasonable time to do our study. We anticipate
(and hope) that as more investigators become familiar with the
legislation, reporting rates will increase. As we present a cross
sectional study from a single year, we cannot comment on trends in
reporting of results.
An important limitation of the
study is that we have identified trials subject to mandatory reporting
through publicly available summary data on the trials. We categorised a
trial as covered by the FDAAA if it had at least one site in the United
States, was Phase II or later, and investigated a device, drug, or
biological agent. The possibility remains of misclassification of trials
as either FDAAA applicable or FDAAA non-applicable, owing to the nature
of the information available to us on ClinicalTrials.gov.
We
identified a group of trials subject to mandatory reporting by cross
referencing the ClinicalTrials.gov record with Drugs@FDA (the online
database of FDA approved drugs). This was straightforward when the brand
name of the drug was listed in the ClinicalTrials.gov record and that
product had FDA approval. Similarly, categorising drugs as non-FDA
approved when the brand name drug was not listed on Drugs@FDA or when a
generic form with a different formulation or route of administration to
the FDA approved product was used in the trial was a simple process. We
had more difficulty in categorising a drug administered in a trial if
the generic name given in the ClinicalTrials.gov record was approved by
the FDA, as the approved drug may not necessarily have been the
formulation administered in the trial. In these cases, we took an
inclusive approach and included these trials in our mandatory reporting
required group. We did a subgroup analysis using only trials of drugs
for which the brand name of the drug was given in the trial (thus
eliminating uncertainty as to whether the formulation in the trial was
or was not FDA approved). In this analysis, 28% of trials had reported
results, broadly similar to the 22% that we found with the more
inclusive approach.
We have no data on the number of
studies that applied for exemptions from the requirement for reporting.
Industry funded studies are perhaps most likely to form part of an
application for licensing/marketing, and therefore most likely to apply
for an exemption, but we have found that industry funded studies are
more likely to report results than are studies funded by other means. We
also note that trials investigating new indications for drugs
previously approved by the FDA are not required to upload results until
two years after completion, but our methods did not allow us to identify
these trials.
Although these various effects could
increase our denominator and falsely decrease the overall reporting
effect, we believe that this is unlikely to account for approximately
78% of the trials not reporting. Phase IV trials are (by definition)
trials of drugs that have marketing approval, and so are of an FDA
approved drug, and the indication studied in the trial should be the
same as the approved indication. All of these studies should have
reported results within one year, and yet only 31.3% had done so (table 1⇑).
Although our denominator is necessarily somewhat uncertain, owing to
the nature of the data that are publicly available, we believe that many
clinical trials that should have reported basic results had not.
Conclusions and policy implications
Through
studying the ClinicalTrials.gov database, which mandates the reporting
of outcomes of completed but unpublished studies, we believe that an
understanding of the extent of bias in the literature due to unpublished
studies can be gained. This will be of great benefit to the clinicians
who prescribe new treatments, the patients who take them, healthcare
funders, and especially researchers doing systematic reviews of
treatment interventions which inform all of the above. The availability
of both the protocol and results in outline form helps to overcome both
outcome reporting bias and publication bias.
We report a
systematic approach to examining compliance with mandatory reporting.
In a study such as ours, which uses only publicly available data and
cross references one dataset to another, some debate will inevitably
take place about whether we have accurately identified trials that fall
under the FDAAA requirement for mandatory reporting. Within the
limitations of our study design, we have identified a group of trials of
which the vast majority should have reported summary results within a
year and had not done so.
We believe that the FDAAA is
an important milestone on the path to a future in which all the
available evidence can be used to make decisions about treatment, and we
applaud the farsighted nature of the legislation. We have found that
trials subject to mandatory reporting are more likely to have reported
results compared with other trials, illustrating the positive effect of
the legislation and efforts of ClinicalTrials.gov. Nevertheless, overall
reporting is poor.
What is already known on this topic
- Reporting bias prevents the dissemination of results of clinical trials
- Where trial data are not accessible to practitioners, clinical decisions cannot be based on the best evidence and may be flawed
- Recent US legislation mandates publication of a results summary on ClincalTrials.gov for recent trials of drugs approved by the Food and Drug Administration and should improve access to trial results
What this study adds
- At the time of the search, many trials that should have published data on ClincalTrials.gov had not done so
Notes
Cite this as: BMJ 2012;344:d7373
Footnotes
- Contributors: APP, MNH, and ARS designed the study. APP wrote the script to webscrape additional data from ClinicalTrials.gov. APP categorised the trials, MNH cross checked a subset, and ARS arbitrated. All authors had full access to the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. All authors wrote the manuscript. APP and MNH contributed equally to this work. ARS is the guarantor.
- Funding: The study was not externally funded.
- Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare that APP is supported by a National Institute for Health Research fellowship (DRF-2009-02-112) and MNH is supported by a Wellcome Trust fellowship (WT092295AIA). ARS declares relevant activities outside the submitted work of membership of a REMPEX steering committee, consultancies for Novartis and Biocontrol, and a lecture paid for by Chiesi Pharma. ARS has registered trials on ClinicalTrials.gov and other registries.
- Ethical approval: Not needed.
- Data sharing: Technical appendix, statistical code, and dataset available from the corresponding author (andrew.prayle@nottingham.ac.uk) or at http://dx.doi.org/10.5061/dryad.j512f21p. Participants’ consent for data sharing not required.
This
is an open-access article distributed under the terms of the Creative
Commons Attribution Non-commercial License, which permits use,
distribution, and reproduction in any medium, provided the original work
is properly cited, the use is non commercial and is otherwise in
compliance with the license. See: http://creativecommons.org/licenses/by-nc/2.0/ and http://creativecommons.org/licenses/by-nc/2.0/legalcode.
No comments:
Post a Comment