Misreading a Flawed Study
Vaccination against UTIs is a novel idea that holds enormous promise, but clinical trials must be well-designed and carefully analysed.Â
In April 2024, a colleague and I published an article discussing the first vaccine designed to prevent recurrent urinary tract infections in women. UTIs are one of the most common infections, affecting roughly fifty percent of all women, with nearly half going on to experience a recurrence within six-to-twelve months. Recurrent UTIs carry a significant toll in morbidity and mortality, accounting for up to six percent of all healthcare visits (roughly seven million visits) in the US and are estimated to cost up to US$1.6 billion annually. Finally, recurrent UTIs have a profound effect on women’s activity, health-related quality of life, and productivity.
Antibiotics are the only effective treatment for recurrent UTIs. Physicians prescribe them but are very concerned about the increasing problem of antibiotic resistance. For this reason, there is a great interest in “antibiotic-sparing modalities” to prevent UTIs, particularly vaccines.
Our article reviewed the results of a randomised clinical trial of the vaccine known as MV140. MV140 is a sublingual vaccine with four inactivated bacteria types that cause most UTIs. It has been available under the commercial name Uromune in many countries in the European Union, the UK, Australia, and New Zealand under special-access programs and on a compassionate use basis since 2010. The paper was published in the New England Journal of Medicine Evidence in 2022. The study included 240 women aged 18 to 75 from Spain and the UK with recurrent UTIs. Subjects received either the vaccine for three or six months, or placebo for six months. The primary endpoint was the number of UTIs in the nine months following three months of intervention.
The results appeared to be impressive. Among women in the placebo group, the median number of UTIs was 3.0, compared to zero in both groups receiving the vaccine, a difference that was statistically significant. Among women who received the placebo, 25 percent were free of UTIs compared with 56 percent and 58 percent of those who received the oral vaccine for three and six months, respectively. The authors concluded that “In this controlled trial … MV140 showed promising clinical efficacy in reducing recurrent UTI in women suffering from this condition.”
After reading the NEJM Evidence paper, I contacted one of the authors (Dr Curtis Nickel) to find out where the vaccine stood with regard to approval by the Canadian government. In March 2022, the Canadian healthcare company Red Leaf Medical had submitted an application to Health Canada for the licensing of MV140. Dr Nickel told me about difficulties the company encountered from the committee conducting the review. After waiting several additional months for some word of the outcome of the review, I decided to take a closer look at the paper.
First, I was surprised to find that there were slightly more premenopausal than postmenopausal women in the sample (111 vs. 104). This is strange since recurrent UTIs are more common in postmenopausal women. Second, in an earlier paper from the same group in 2017 testing the efficacy of MV140 in 75 women with recurrent UTIs (but with no placebo group), it is reported that 59 women (78 percent) given the vaccine reported no subsequent UTIs during the treatment and subsequent follow-up period. Of the 16 women who experienced a UTI recurrence, fourteen (87 percent) were postmenopausal.
Geoffrey C. Kabat
These results are from two different studies. Nevertheless, if one considers the combined effects of the relatively small number of postmenopausal women in the clinical trial and the fact that the vaccine appeared to be less effective in postmenopausal women in the smaller study, this would suggest that the main driver of the result reported in the clinical trial was the apparent effect of the vaccine in premenopausal women.
This suggests that, if the authors of the clinical trial had stratified their results by menopausal status (i.e., repeated the analysis in premenopausal and postmenopausal women separately), the results would be stronger among premenopausal women and weaker in the postmenopausal group—the group most affected by recurrent UTIs.
Carrying out a stratified analysis is standard practice in epidemiology. In assessing a risk factor for breast cancer, one would never do an analysis combining both pre- and postmenopausal women. However, in this study, when one looks at the numbers of premenopausal women in each of the three treatment groups (placebo, three months on MV140, six months on MV140), they are quite small (31–41), leading to unstable estimates.
This example underscores why it is crucial to have an adequate sample size for each group you want to address and to analyse the data appropriately. In view of these weaknesses, the NEJM Evidence clinical trial does not appear to provide the kind of solid evidence for MV140 necessary to obtain regulatory approval. And it is not surprising that neither Health Canada nor the US FDA has approved the vaccine.
A recent review article in Nature Reviews in Microbiology includes mention of a second vaccine that is currently enrolling patients in a phase III clinical trial. The Janssen 9-valent Extraintestinal Pathogenic Escherichia Coli Vaccine (ExPEC9V) contains antigens of the most common types of E. coli. Eligible subjects are over sixty years old with a history of UTI. They will be followed for up to three years post-vaccination, evaluating a variety of primary and secondary endpoints, including time to first UTI, number of UTIs per participant, and medical resource utilisation owing to UTI. An earlier version of this vaccine, which included only four serotypes, was found to reduce the number of recurrent UTIs in a cohort of women with a history of recurrent UTIs in a phase 1b trial. According to the review in Nature Microbiology, “This study shows that a top UPEC [uropathogenic E. coli] vaccine candidate that is currently in phase III clinical trials reduces UTI incidence in a cohort of women with recurrent UTI.” Another study showed that EXPEC4V was well tolerated and elicited robust and functional antibody responses across all serotypes, doses, and age groups. The immune response persisted for one year.
The Janssen study has a target sample size of 18,556 participants. The lower age limit of sixty means that the study is restricted to postmenopausal women. According to the Nature review, “The results of the ExPEC9V phase III trial are not expected until at least 2025, but if effective, it would represent a significant advance in the prevention of UTIs.” This large and well-designed multi-centre study will have the statistical power to detect an effect of the vaccine if such an effect exists.
In retrospect, I should have noticed the problems with the MV140 clinical trial on the first reading—small numbers, inclusion of premenopausal women, and failure to stratify on menopausal status. I did realise that the findings were not robust, owing to small numbers and only a moderate effect of the vaccine. Nevertheless, I gave the study results qualified credence, reasoning that they might signal a real effect. No doubt, I was also influenced by the New England Journal of Medicine moniker, even though the paper was published in an annex of the prestigious journal.
This story provides a useful reminder that researchers will often publish results that, at first blush, may appear promising and possibly important, but which, on closer inspection, turn out to be based on small numbers, a poorly selected sample, and a failure to analyse the data properly. Having drawn attention to the NEJM Evidence paper in our article, I felt it was important to correct our initial reading.
The story also reminds us that coming up with treatments for difficult medical conditions is often a long road—a process in which early failures are analysed, assumptions are questioned, new hypotheses are formulated, and methods are recalibrated.
