August 1995
By Spencer Cox
Edited by Mark Harrington & Bruce Schackman
From the Executive Summary
Over the period in which FDA has reviewed and regulated anti-HIV therapies, there has been a substantial decline in the quality and quantity of available information regarding the clinical utility of those drugs at the time of their approval. While the experience with accelerated approval remains limited, it seems that the decline in information rapidly increased following the implementation of those regulations. While on the surface this often seems to represent problems in basic trial design methodologies — such as improper controls, early termination, inadequate sample size and post-hoc adjustment and analysis — FDA’s willingness to accept fundamentally flawed studies as providing sufficient confirmatory evidence to validate approval may be reducing incentives to properly design and implement post-marketing studies.
Traditionally, regulation has provided incentives and disincentives in order to shape the kinds of safety and efficacy trials performed by pharmaceutical companies in developing new products. By setting “threshold” standards of safety and efficacy, FDA has helped to ensure that studies were designed to answer basic empirical questions about the safety and efficacy of therapies in use. Consequently, efforts to satisfy FDA standards at the end of the process have driven the design of clinical studies much earlier in the process.
In 1989, activists began to articulate the need for a new standard of efficacy, one which “takes account of the uniqueness and potential value of a drug and the urgency of the need for it.” In other words, data would be reviewed not based on efforts to answer certain primary questions, but would, in the words of the 1989 “Bush Initiative,”
“consider whether the benefits of the drug outweigh the known and potential risks of the drug and the need to answer remaining questions, about the risks and benefits of the drug, taking into consideration the severity of the disease and the absence of satisfactory alternative therapies.”
Without the knowledge that they would be required to provide specific information about the safety and efficacy of their products, manufacturers began to submit applications for approval based on unrandomized phase I data compared to historical controls, preliminary analyses from studies that were not intended to evaluate treatment effect on the endpoint of interest, and subgroup analyses purporting to demonstrate efficacy in tiny subsets of the original study.
While the FDA’s Antiviral Drug Products Advisory Committee has discounted claims based entirely on unrandomized data, it seems ultimately to have settled on an efficacy standard that permits accelerated approval based on one randomized study showing surrogate benefits, even in the presence of conflicting data, as was the case for ddC and d4T. For movement from accelerated approval, the standard is not yet clear. However the case of ddC demonstrates that the standard is not, as was recently suggested by FDA Commissioner David Kessler, identical to that imposed by pre-accelerated approval policies.
The concern regarding this flexibility is that it will offer incentives for sponsors to design trials that produce little or no reliable information, because these trials are believed to be faster and less expensive. With accelerated approval virtually guaranteed for antiretroviral drug products sponsored by large, multi-national pharmaceutical companies, the balance of incentives began to shift, producing a “mandate for ignorance”: the manufacturer’s financial interest is to avoid collecting reliable efficacy data (which would require longer or larger and more expensive trials), and to submit an application for full approval based on little more than time since initiation of product development. If all of the good news comes from small studies of treatment effect on surrogate markers, and all of the bad news comes from large-scale clinical endpoint studies, then industry has financial incentives to avoid the latter.
There is some evidence that such a shift is already occurring: the presumption seems to have favored approval in the absence of data suggesting lack of safety — a concept suggested by conservative policy analysts and some AIDS activists as early as 1988. Consequently, the question about ddC was not, “Is this product safe and effective for the treatment of HIV disease?” but “Can we conduct further studies of this drug, or should we just approve it now?” Given the unwillingness of many patients to enter properly controlled studies following approval, it may be impossible to conduct a simple, empiric evaluation of basic safety and efficacy in postmarketing studies. FDA’s Dr. Ellen Cooper warned of such concerns in 1990:
“For drugs to treat serious or life-threatening conditions, the first evidence of efficacy, particularly if it comes from a controlled trial, often makes it very difficult, if not impossible, to conduct additional controlled trials as the world makes up its mind based on the early evidence. Thus, it is very important that the first randomized, controlled trials of new drugs under investigation for the treatment of life-threatening diseases be as well- designed and conducted as possible, with the objective of providing the highest quality results that have adequate power to demonstrate efficacy or lack thereof with reasonable assurance.”
Too often, the response of patient advocates to preliminary data suggesting possible efficacy has assumed a greater reliability of such data than may actually be warranted. Such assumptions occur in the best of faith, from a real desire to offer patients hope. As one physician who was himself living with AIDS, noted after the approval of AZT, “We’re beginning to bias the process in favor of good news.”
Unfortunately, biasing the process in favor of good news is probably the wrong empirical approach: according to the Pharmaceutical and Research Manufacturers of America (PHaRMA), only one drug is approved for every five that enter human testing. In 1990, FDA’s Dr. James Bilstad informed the Antiviral Drug Products Advisory Committee that:
“Approximately 70 percent of the drugs for which INDs are submitted successfully complete phase-l studies…Approximately one-third of the drugs for which INDs are submitted to the Agency successfully complete phase II testing…There is not much of a drop-off from those that successfully complete phase II to phase III. Approximately 25 to 30 percent of those INDs submitted complete phase III testing.”
In other words, the vast majority of drugs entering large-scale human trials will be proven to be unsafe or ineffective. Our presumption must be that new drugs are either unsafe or ineffective until proven otherwise. We must temper our very real desire to rush new treatments to patients at greatest risk with skepticism about “possible” or even “probable” efficacy.
Ultimately, each of the agents currently available presents the very real risk that, measured by the standard of disease and death, risks may outweigh benefits. There are risks to premature approval — risks of acceleration of morbidity and mortality, risks of expenditure of vital health-care resources on therapies of negative or no utility, and risks of utilizing less-than-optimal treatments when more effective agents might be available but poorly characterized. So far, study of antiretroviral agents has been guided by fumbling mistakes, excessive therapeutic optimism and misplaced good intentions.
For example, ddl was considered “validated” by a study comparing two doses of the drug to continued AZT in patients who had already been extensively pre-treated with AZT. However the utility of the “control arm” was unclear given AZT’s well- characterized time-limited utility. As Dr. Donald Abrams told the FDA’s Antiviral Drug Products Advisory Committee, “I’m not sure if what I’m seeing is a benefit from switching to ddl, or from going off AZT.”
And, if the validation of ddl must be regarded as ambiguous at best, then that of ddC, which was based in a comparison to ddl, must also be called into question. The problems of validation multiply when further controlled studies are regarded as impossible.
One school of regulatory thought has held that:
Clinical studies can only predict how well a given therapy will work for a group of patients, but no matter how positive the outcome, they cannot predict whether it will work for individuals…The best situation is one which permits the patients and their physicians to choose freely among the alternative antiviral therapies, finding the ones which work best for the individual. Clinical trials need only tell patients and physicians that a drug has useful activity, what its side effects are, and the proper dose.
One advocate of this approach has proposed trials
which do not even try to prove whether a drug is effective…Instead of asking for statistical proof, the important question is whether [a study] provides information useful for making treatment decisions.
Such an approach seems initially attractive and reasonable, because we would all like to share the bias upon which the approach is predicated: we would like to believe that physicians can tell “which [drugs] work best for the individual.” Typically, this philosophy has embraced the use of “surrogate markers,” such as post-therapeutic changes in absolute CD4+ cell count and various markers of virologic activity, in order to determine whether or not a therapy is “working” in the individual. It is assumed that such changes in surrogate markers must predict rates of continued illness and death.
While the search for useful surrogate markers is rational, our experience both in AIDS and in other diseases suggests that we approach proposed markers with skepticism. The history of failed surrogates in the medical literature, including tumor size for cancer, and rate of arrhythmias in heart disease, is extensive. Dr. Tom Fleming of the University of Seattle recently reviewed data on changes in absolute CD4+ cells as a marker for reduction of AIDS-related morbidity and morality, and concluded that that the direction of surrogate change in previous therapeutic studies has been congruent with the direction of clinical effect approximately 51 percent of the time — in other words, the use of the surrogate for predicting clinical outcome in a population is about as reliable as flipping a coin.
As markers for evaluating clinical effects in individuals, we are forced to use what is available to us. However, clinical confirmation, as required by the accelerated approval regulations, remains integral to the interpretation of individual response. The supposed irrelevance of population-based outcomes to individual outcome is simply untenable. Population-based evaluation is never intended to define the necessary course of treatment for individual patients, but rather to direct bias; a therapy that is helpful “on average” is probably worth trying in the individual (although toxicity concerns may alter that equation).
Ultimately, recent approaches have been predicated on an assumption that it is possible to keep those who are currently desperately ill alive until a cure is found. Unfortunately, this assumption is no longer sustainable. Large treatment effects are easy to discern, but nothing currently in clinical development can be expected to provide miraculous cures. Many of those now ill will die. Barring unexpected advances, many of us who are now infected may be expected to grow ill and to die. However, by careful planning and rigorous evaluation, we can ensure that the available therapeutic arsenal is maximized, to provide persons with HIV/AIDS with the longest healthy survival possible.
When the accelerated approval regulations were established, they were intended to “merely change when some of the studies are done,” rather than to “lower the standards for drug approval”. As one of the leading advocates for accelerated approval noted, its success depends on FDA’s willingness to:
- Make sure drug companies keep their commitments to doing the long-term studies
- Remove from the market drugs approved in this manner if they later prove unsafe or ineffective
- Decide which drugs warrant expedited approval
- Guarantee adequate staff resources to implement expedited approvals
By building in new mechanisms to ensure that products granted accelerated approval are rigorously evaluated for both surrogate and clinical effects, it will become possible to provide the information needed to meet the access and informational needs of a variety of patients and patient populations.
For too long, activists, scientists and regulators have accepted as dogma that “acceleration comes at a price — less information about drugs, and even the risk that treatments may be useless.” We have believed without question that
statistical proof using clinical endpoints in early HIV disease requires long, large trials. Adding the time required to conduct and analyze the trial itself, the time for recruiting, and the time to build the required commercial and professional momentum to get a large trial going, it is likely to take several years to test each drug or combination. Such trials are not feasible for many reasons.
In cancer and heart disease, clinical research methodologies have been developed that can quickly and efficiently enroll the large numbers of patients needed to reliably answer important questions about the safety and efficacy of therapies. While early versions of these studies were difficult to initiate, subsequent studies utilizing the established networks of researcher/physicians have enrolled massive numbers of patients at lightning speed. Already, randomized expanded access programs have created such a network of physician/researchers in AIDS. We must fully utilize the potential of these networks to provide early, rapid access to the desperately ill, and to rigorously evaluate the effects of treatments in those who are less ill.
In addition, we must ask ourselves on whose behalf we advocate. While advocates are attempting to define access mechanisms for the latest “drug of the month,” Pneumocystis carinii pneumonia, a largely preventable disease, remains the leading cause of AIDS diagnosis, and of AIDS-related morbidity and mortality in New York City, and probably, in the United States. AIDS advocacy that reflects only the treatment priorities of the advocates themselves will be ineffectual at best, and a harmful diversion at worst.
The urgent need to plan for the future has only grown. As the epidemic continues to spread among intravenous drug users, their sexual partners and their children, it has re-emerged as a threat to gay men in the United States. A recent paper from the Multi- Center AIDS Cohort Study (MACS) begins to suggest how high the stakes may be. The authors estimated the lifetime risk of HIV infection for a twenty-year-old gay man today. According to their estimates, “such a man has a 20 percent chance of seroconverting before reaching the age of 25…The overall probability of seroconversion prior to age 55 is about 50, with some seroconversions continuing after that.” In other words, the majority of twenty-year-old gay men today can expect to become HIV-infected at some point in their lives, barring unforeseen improvements in prevention technology.
Evaluation of AIDS drugs today does not just affect those who are currently ill; our planning now will determine the quality of HIV treatment for the next generation of people with HIV/AIDS. If we fail to address the basic question of efficacy standards now — “At what point in the development process do we reliably determine that a treatment can extend health and life?” — then our drug development systems, and our standards of HIV treatment are destined to leave us with substandard therapy for most HIV- infected persons, serious vulnerability to fraudulent claims, and ultimately, to unnecessary sickness and death.
We must and can do better. In continually reviewing and improving our regulatory mechanisms, it may be possible to reconcile the seemingly opposed needs for access to new therapies and reliable information about their efficacy.