When studies conflict: building a decision-support system for clinicians

One day you hear that red wine is good for your heart. The next day, it’s not. The same goes for chocolate. And coffee. The see-saw of contradicting information isn’t anything new, but what happens when clinicians hear conflicting studies about a medication they use for their patients? Researchers at the Perelman School of Medicine at the University of Pennsylvania are hoping to use, among other methods, a variety of artificial intelligence to help sort that out. 

Ellen Caniglia, ScD, an assistant professor of Epidemiology and Enrique Schisterman, PhD, a professor and the chair of the Department of Biostatistics, Epidemiology, and Informatics (DBEI), received a $1 million grant from the Patient-Centered Outcomes Research Institute (PCORI) to develop a system that would allow clinicians to weigh the results of two conflicting clinical trials and make an informed choice about how those findings should influence their patients’ treatment.  

“Clinicians and policymakers need user-friendly tools to understand why seemingly identical studies could produce conflicting results,” said Caniglia. “With a tool like that, we can be much more confident in what is being recommended and help patients make informed decisions, as well as identify those who may benefit or be harmed by a treatment being considered.” 

Conflicting information or guidance on treatments used for patients are not uncommon. A high-profile recent example is aducanumab, which had been used among those with Alzheimer’s disease with the idea of targeting plaque build-ups in the brain. It was given accelerated approval by the FDA in 2021, was embroiled in controversy, both in its trial results and over the mechanisms and process surrounding that approval. Eventually, it was discontinued by its maker.   

Comparing conflicting studies of a preventive drug for preterm birth 

Caniglia and Schisterman will partner with other Penn Medicine investigators in DBEI and the Department of Obstetrics and Gynecology to complete work on the decision-support system. Joining them will be Eric Tchetgen Tchetgen, PhD, a professor of Biostatistics; Lisa Levine, MD, MSCE, chief of Maternal Fetal Medicine; and Beth Pineles, MD, PhD, an associate professor of Obstetrics and Gynecology. 

Their work will begin by comparing a pair of studies that examined the use of a medication called 17-alpha-hydroxyprogesterone caproate (17P). A weekly injection, 17P was recommended for pregnant individuals who previously had a preterm birth to prevent it in their current pregnancy. One study from 2003 found that 17P gave significant protection against having a preterm birth, but a second study published in 2019 found no effect.  

As a result of the second study, the FDA and other national medical organizations rescinded their support for the use of 17P in preventing preterm births, leading to hospitals around the country to stop routinely administering it.  

But why did these studies differ? 

“We still have many residual questions as to why it worked in one large trial and not another,” Caniglia said. “So, we’re aiming to develop methods and software that could uncover how variations in study findings are explained by things like hidden differences in study populations, the extent to which study participants follow the protocol, and, ultimately, what harm or benefit may come to individuals receiving the treatment.” 

Deeper analysis to understand differences in studies 

Caniglia, Schisterman, and the research team will develop and apply novel causal inference—a discipline of investigation that considers assumptions and tries to drill deeper to the cause of something—and machine learning—a variety of artificial intelligence that “learns” by analyzing large datasets— in their effort to reconcile results of conflicting trials. They’ll then build their decision-making system for comparing studies using what they learned. 

The work is supported by PCORI’s Improving Methods for Conducting Patient-Centered Outcomes Research Cycle 2 2023 funding opportunity. It’s a funding program designed to “address important methodological gaps and lead to improvements in the strength and quality of evidence” used in research to generate better patient outcomes. 

Caniglia and her team will receive $1 million spread out over three years of work. 

“If we can create something that helps clinicians and policymakers faced with conflicting results from randomized trials better decide whether a treatment should be available—and to which populations—our efforts will be a success,” said Caniglia. 

The award has been approved pending completion of a business and programmatic review by PCORI staff and issuance of a formal award contract.