Image via WikipediaThe FDA has just released its new Draft guidance on clinical trial design for antibacterial drugs in the treatment of hospital-acquired pneumonia (HAP) and ventilator-associated pneumonia (VAP). As anticipated, they call for an endpoint of 28 day all cause mortality. But the devil is in the details. The FDA suggests that patients at high risk of dying (those with APACHE scores of 15 or greater) be enrolled to allow for a treated mortality rate of around 20%. That would allow for a non-inferiority margin of 10%. BUT – the analysis population is the microbiology intent to treat (MITT) population – that is, those patients enrolled and treated who have an identified bacterial pathogen at study entry.
Once again, in the appendix to the guidance, the FDA has gone to extraordinary and unscientific lengths to discount the treatment effect of antibiotics in VAP such that they arrive at an infeasible trial design. They demonstrate that inappropriate therapy for VAP is associated with a 62% mortality and that appropriate therapy is associated with a 20% mortality. The treatment effect of appropriate vs. inappropriate therapy is, therefore, 42%. Of course, inappropriate therapy is not no therapy, and 42% as a treatment effect is therefore already conservative. But, the FDA is not satisfied with that. They go on to apply their 95/95 rule using the upper bound of the 95% confidence interval for treatment effect of inappropriate therapy and the lower bound for appropriate therapy yielding a treatment effect of 29% instead of 42%. They then discount the treatment effect of 29% by an additional 30% for good measure. The FDA justifies their additional 30% discount by claiming that this is necessary to correct for “uncertainties” of the historical database. This brings the treatment effect or M1 down to 20%. How convenient! They now take 50% of that and call that a justified non-inferiority margin of 10%. (Is 10% starting to sound like a familiar number?) But since the margin simply has to be smaller than the treatment effect, the margin could just as easily have been 19%. But of course the entire discounting argument is overly conservative and irrational.
So, as an amateur developer, I’ve been trying to understand the practical consequences of this design. Lets take VAP since the FDA seems to focus on VAP in their guidance. In my calculations I have made the following assumptions:
Cure rate = 80% (20% mortality).
Evaluability – as far as I can tell, in modern trials, the MITT population is about 45% of the enrolled population.
NI margin = 10%
90% power (to exclude the chance of falsely concluding inferiority as much as possible).
I calculate that one would need to enroll over 1200 patients per trial. That would be at least 2400 patients in two trials.
I believe that the largest HAP/VAP trials ever performed were the ATTAIN-1 and -2 trials by Theravance/Astellas. They enrolled 1518 patients over about 28 months. I don’t know how many centers were used. They powered their trials to show a 20% non-inferiority for treatment of MRSA infection compared to vancomycin. Non-inferiority margins of 15-20% used to be common for VAP trials since these patients are so hard and so slow to enroll and since the trials are so expensive to run. Those were the days when we actually considered feasibility of trial designs before we issued guidance. Two trials at 1200 patients per trial in VAP is, once again, totally infeasible. Even just in terms of the time that would be required to complete such a trial.
I calculate, based on recent enrollment rates as cited by Steve Barriere of Theravance, that such a trial could take anywhere from 5 to 50 years to complete. In the current FDA guideline, the suggested severity of illness (APACHE score > 15) will slow enrollment. The number of centers participating in the trial and, as a corollary, the number of competing trials will all affect enrollment rates and costs. After even 5 years, the trial design and comparator might already be obsolete.
In terms of cost, VAP trials are probably the most expensive trials we currently undertake. I’m guess that costs now are in the order of $30-50,000 or more per enrolled patient. If the analysis population will now be the MITT population, these costs could be even higher. Frank Tally estimated that in Cubist’s failed (for lack of enrollment) trial of daptomycin in the treament of bacteremia caused by vancomycin-resistant Enterococcus, their costs were $250,000 per evaluable patient. They spent $20 million before jettisoning the trial. At $50,000 per enrolled patient, a 2400 patient set of trials in VAP would cost $120 million. No company, let me repeat that, NO COMPANY will take this on! The costs rapidly outstrip any return on investment.
What about superiority trials as a solution? We’ll address that in a future blog.
Have a great day!
Footnote: As far as I can tell, Theravance never published the full results of their trials. I can only find abstracts and a very brief and incomplete summary of their results on clintrials.gov. Now that it is clear what the FDA wants – will Theravance be able to provide it?
I have received a number of comments from PhRMA buddies. Since they, as usual, prefer to remain anonymous, or are at least too shy to post comments themselves, I have added them here.
ReplyDeleteOne PhRMA colleague wrote,
Actually, my calculations suggest it is worse than that. When I calculate the sample size for
80% success rate
45% evaluable
10% margin
95% confidence interval
90% power
= 747/arm or 1494 per study.
The same PhRMA colleague calculated what a non-inferiority study would be like if Pseudomonas aeruginosa was targeted. In this calculation, if 8% of subjects have P. aeruginosa as one of the causative pathogens, the sample size becomes 4151/arm or 8302 for the study.
These calculations ignore the loss to enrollment from the entry criteria requirements of high APACHE, no antibiotics in the previous 30 days, and a requirement for 15% bandemia.
Another PhRMA colleague noted that the rule that folks need to be sick enough to ensure a 20% overall mortality rate is inconsistent (again from a feasibility standpoint) with ALSO prohibiting prior antibiotic use.
One comment was, simply, “Scary!”
This is a comment from Brad Spellberg -
ReplyDeleteYour comments, as always, are to the heart of the matter. Your sample size calculations are indeed frightening. I believe these trials can be conducted if some modifications to the posted draft guidance are made. But no one can know for sure that these trials can be conducted because a trial to these specs has never before been conducted.
The choke points for sample size and feasibility are: 1) the margin size; 2) the success rate in the comparator arm; 3) the microbiology confirmation rate; and 4) the inability to prescribe even a single dose of antibiotics before randomization.
1) Margin size. We are stuck with all-cause mortality because there are no available datasets documenting antibacterial effect size for a clinical endpoint. Everyone I know believes that for a mortality endpoint, a margin of greater than 10% should not be allowed (in contrast to clinical endpoints, where margin sizes of 15% seem perfectly reasonable). So, the only way to get the margin size bigger is for companies or academic investigators to go back to, or create anew, datasets in which clinical endpoints are analyzed for success in patients receiving initially effective vs. initially ineffective antibacterial therapy. I urge anyone with access to such databases to analyze them ASAP, and all sponsors prospectively conducted trials should build analyses into the protocol to generate these data, to enable future primary clinical endpoints.
I agree with you David that discounting an effect size to determine a margin creates the facade of mathematical precision, but in the end it is nothing more than an arbitrary means to manipulate the numbers. When the effect size is markedly larger than the planned NI margin, it is far more rationale and logical to select a margin the balances statistical imprecision with trial feasibility and is based on preservation of a clinically meaningful effect. In this circumstance, the arbitrary manipulation of the effect size by discounting was done in a manner that ended up being concordant with the 10% margin that clinicians have agreed is acceptable as a trade-off between uncertainty and feasibility.
2) Comparator arm success rate. The draft guidance indicates a need to have at least a 20% mortality in the comparator arm. This is not commensurate with the discussions held at the 2009 HAP-VAP workshop. The data reviewed at that workshop indicated that studies upon which antibacterial efficacy were calculated had mortality rates ranging from 15-20%. Hence, based on the DATA, the mortality rate in the comparator arm should be 15-20%, not greater than 20%. This is a critical feasibility issue. Just by increasing the success rate from 80% to 85%, and leaving all other study elements the same, the sample size will shrink. David, are you up to the challenge of modeling this sample size for us?
3) Microbiology confirmation rate. I may be naive and overly optimistic, but I believe that with intense effort regarding bronchoscopy and endotracheal aspiration, in conjunction with rapid molecular diagnostic assays, that we can do better than a 45% microbiology confirmation rate for this disease. If we can get the micro confirmation rate up to 60%, or dare I say, even 70%, that should make the trials much easier to conduct.
4) Prior antibiotics. This one is really a problem. It is difficult to believe that a single dose of antibacterial could meaningfully improve VAP cure rates given the very nasty nature of the microbiology, the presence of the foreign body, the debilitated host, etc. Furthermore, in contrast to CAP, antibacterial therapy for HAP/VAP is almost never once daily. I can't think of a single once daily option for the treatment of this disease. Most are thrice daily, and some are four times daily drugs. With such short half lives, one dose of prior non-study therapy should be allowed, particularly given the critical need to start therapy immediately when a really sick, hospitalized patient develops a life-threatening infection.
This comment has been removed by a blog administrator.
ReplyDelete