Does coffee improve memory? Do carrots boost vision? Does vitamin D deficiency increase the risk for COVID-19?

It depends.

The same research question can yield vastly different answers depending on how a study is designed, which variables are measured, and how results are analyzed.

Because of the hodgepodge of approaches used to decipher the interplay between variables, association studies—those that explore how one thing affects another—are notoriously prone to error or “bias.”

Finding a link where none exists or missing one if it does can thwart the pursuit of critical scientific questions and solutions, lead researchers down the wrong path and generate contradictory results that confuse peer scientists and the public alike.

To help remedy such problems, a team of computational scientists from Harvard Medical School has developed an auditing tool called vibration of effects (VoE).

The tool, first described in PLoS Biology in September 2021, has now been deployed to analyze reported links between various gut microbes and six diseases in 15 previously published studies comprising samples from 2,434 patients with colon cancer, type 1 diabetes, type 2 diabetes, cardiovascular disease, inflammatory bowel disease (IBD) and cirrhosis of the liver.

The new research, published March 2 in PLoS Biology, is the final installment in a three-paper series and represents the culmination of the team’s two-year journey undertaken at the start of the COVID-19 pandemic and conducted with collaborators working remotely across the country.

The results of the latest study reveal that a full one-third of 581 reported microbe-disease associations were inconsistent, with outcomes changing depending on how the design was tweaked and which other variables were included in the analysis.

A particularly striking finding, the team said, was that more than 90 percent of the research findings of studies exploring the link between gut microbes and type 1 and type 2 diabetes were inconsistent.

Studies exploring the link between cirrhosis of the liver and gut microbiome yielded the greatest consistency—60 percent of these analyses showed consistent results when run through different models.

Cardiovascular disease association studies showed nearly 50 percent consistency, as did one-third of IBD-microbiome association studies.

The tool uses a brute-force computational approach that tests the reliability of research findings and can be used by researchers to audit their own results before submitting them for publication. It is publicly accessible and available for free online.

“At its most basic, the vibration of effects model analyzes how the modeling choices a researcher makes can influence what they will discover,” said Braden Tierney, one of the tool’s chief architects.

A former doctoral student at HMS, Tierney is now a postdoctoral research fellow at Weill Cornell Medical College.

“This approach is one way to maximize researchers’ confidence in the results they are getting from their analyses before they even publish them.”

In the latest study, the team checked each of the reported associations by running millions of modeling strategies, including the addition and subtraction of different variables. The modeling demonstrated how results could shift dramatically depending on which variables were used and which questions were asked.

Overall, studies that scored high were less reliable because their results showed a great degree of variation when run through multiple models.

By contrast, studies that scored low on VoE were deemed robust because they pinpointed associations that remain consistent even when a different testing model is applied.

Going a step further, the team demonstrated how the VoE tool can be used to identify potential confounders—factors whose influence is not measured or accounted for in the study design and thus interfere with the reliability of the results.

To do so, the team ran more than 6 million statistical modeling strategies on the findings of previous studies, adding and subtracting variables and testing different combinations of variables.

For example, analyzing the role of the microbe F. prausnitzii in colonic disease, the researchers demonstrated how including factors such as a person’s blood sugar and cholesterol levels and body mass index can give seriously divergent results.

“The VoE tool can help researchers not only identify problems but also diagnose what may be causing them,” Tierney said. “It can help them understand why they may be getting conflicting findings on the same research question, and it can help them dig deeper and find links they may otherwise overlook.”

High stakes

When done well, association studies can become critical gateways to further research that builds on these initial findings. Identifying linkages between variables, such as coffee intake and memory, carrot consumption and eyesight, are important because they can inform hypotheses that scientists can test in the lab and in clinical trials to determine cause and effect and, eventually, the underlying mechanism of an observed effect.

“Understanding correlation is a prerequisite for understanding causality but is not enough,” said Chirag Patel, co-senior author on the trio of studies and associate professor of biomedical informatics in the Blavatnik Institute at HMS.

“So, it’s essential for researchers to have confidence in the robustness of the observed association before they engage in further, sometimes expensive, study.”

Read full article in HMS News