Improving machine learning models for microbiome analysis

and democratizing data science along the way

Kelly Sovacool

Jun 22, 2023

What is a microbiome?

The human gut microbiome changes during diseases

• Irritable bowel diseases
• Colorectal cancer
• Clostridioides difficile infection

Open problems and opportunities

• Understand the mechanistic role of the gut microbiome in gut diseases.
• Identify biomarkers to improve diagnostic and prognostic tools.

• Changes in the taxonomic composition and metabolic activity of human microbiomes have been observed in several diseases including colorectal cancer (CRC) and Clostridioides difficile infection (CDI).
• CRC - enrichment for bacteria that produce enterotoxins in CRC gut microbiomes.
• CDI - treatment with antibiotics alters the taxonomic composition of the gut microbiome, allowing C. diff to colonize the gut.

Possible causative role of the microbiome in disease etiology. mechanisms are not entirely understood. not always clear whether the disease causes the microbiome to change, the change in the microbiome causes the disease, or some combination.

• CDI: antibiotics disrupts the community and allows cdiff to colonize, but what causes some people to develop severe outcomes vs clear the infection easily?

  • Bile acids metabolized by the gut bacteria can inhibit C. difficile growth and affect toxin production (4, 10, 11). Bacteria in the gut also can compete more directly with C. difficile through antibiotic production or nutrient consumption

• CRC: some CRC patients have strains of E. coli that produce enterotoxins, but not all.

  • FIT cartridge sensitivity (true pos rate): 70% (early stage) to 80% (late stage)
  • colonoscopy sensitivity: 78% to 93%
  • detecting CRC early improves prognosis, but colonoscopies are invasive.

• Baxter et al: “The microbiota-based random forest model detected 91.7 % of cancers and 45.5 % of adenomas while FIT alone detected 75.0 % and 15.7 %, respectively. Of the colonic lesions missed by FIT, the model detected 70.0 % of cancers and 37.7 % of adenomas. We confirmed known associations of Porphyromonas assaccharolytica, Peptostreptococcus stomatis, Parvimonas micra, and Fusobacterium nucleatum with CRC. Yet, we found that the loss of potentially beneficial organisms, such as members of the Lachnospiraceae, was more predictive for identifying patients with adenomas when used in combination with FIT.”

• However, these changes are not universal across individuals or datasets.

• Mouse experiments indicate a possible causative role of the microbiome.

  • fecal matter transplant from susceptible to germ-free mice
  • predict colonization following Cdiff challenge

• Open problems, and opportunities:

  • Understand the mechanistic role of the gut microbiome in gut diseases.
  • Identify biomarkers to improve diagnostic and prognostic tools.

The human gut microbiome changes during C. difficile infection

• CDI - treatment with antibiotics alters the taxonomic composition of the gut microbiome, allowing C. diff to colonize the gut.
• mouse studies have found differences resistance to colonization, time to clearance, and severe outcomes based on the initial state of the gut microbiome.

How to study the gut microbiome

analyze microbial sequences to determine which microbes are there and how abundant they are.

Machine learning for science & health care

Supervised machine learning (ML) - computational techniques that identify patterns in large datasets to classify samples or predict outcomes.

Applications:

• Diagnose colorectal cancer early and less invasively than colonoscopy.
• Identify COVID-19 patients at risk of clinical deterioration.
• Predict severe outcomes of C. difficile infections.

Traditional microbial ecology methods aren’t sufficient to distinguish between disease states. individual microbes aren’t enough to explain it; something more complex is going on

• high dimensionality: 20k OTUs

Overview

1. Improve methods for processing microbiome data.
2. Predict severe C. difficile infections from gut microbiome composition.
3. Contribute to democratizing data science.

How to characterize microbiomes

to characterize the bacteria in a microbial community, the gene encoding the 16S ribosomal RNA subunit is commonly chosen as the target gene to amplify and sequence. after sequencing, the goal is to identify which bacteria are present and how abundant are they in each sample. then we can compare across samples to identify patterns that differentiate disease states or outcomes. ideally, for each sequence we would identify which species it came from, but that’s not possible with amplicon sequencing.

Difficulties in bacterial taxonomy

• Inconsistent species concept for microbes.
• Most microbes are not easy to culture, thus are not well-characterized.
• Many bacteria have multiple copies of target genes for amplicon sequencing.

why OTUs?

• Inconsistent species concept for bacteria.
  • Binary fission - bacteria reproduce by cloning themselves.
  • Horizontal gene transfer - bacteria can swap genes with each other.
  • Their phenotypic properties do not reveal common ancestry.
• Amplicon sequences do not contain enough information to identify bacteria at the species level.
  • 16S highly conserved; 16S rRNA trees informed classifications at and above the rank of genus (https://www.nature.com/articles/s41396-021-00941-x).
• Many bacteria have multiple copies of the 16S rRNA gene, the most popular marker gene for amplicon sequencing bacterial communities.
• Using every sequence (ASV level) as a feature would split them into separate taxonomic groups.
• Bacteria that are difficult to culture are not well-characterized.
  • So we have names assigned to many bacteria that grow easily in the lab, but we’re missing information for lots of bacteria that are hard to culture.
  • This leaves holes in public reference databases.
  • Database-independent clustering methods allow exploring microbial “dark matter”.

Solution: cluster the amplicon sequences into operational taxonomic units (OTU) based on how similar the sequences are to each other. OTUs can generally be identified down to the genus level.

Clustering amplicon sequences into Operational Taxonomic Units (OTUs)

De novo OptiClust algorithm

Clustering amplicon sequences into Operational Taxonomic Units (OTUs)

De novo OptiClust algorithm

Clustering amplicon sequences into Operational Taxonomic Units (OTUs)

De novo OptiClust algorithm

Clustering amplicon sequences into Operational Taxonomic Units (OTUs)

De novo OptiClust algorithm

TODO highlight limitations here

Clustering amplicon sequences into Operational Taxonomic Units (OTUs)

Reference-based algorithm

Clustering amplicon sequences into Operational Taxonomic Units (OTUs)

Reference-based algorithm

Clustering amplicon sequences into Operational Taxonomic Units (OTUs)

Reference-based algorithm

Limitations of existing clustering methods

• De novo clustering results in inconsistent OTUs when adding new data.
• Reference-based clustering results in lower quality OTUs than de novo.

OptiFit

reference-based clustering to de novo OTUs

OptiFit

reference-based clustering to de novo OTUs

OptiFit

reference-based clustering to de novo OTUs

OptiFit benchmarking results

so we benchmarked the optifit algorithm alongside a few other clustering strategies. We tested these algorithms with four public datasets from human gut, mouse gut, soil, and marine microbiome samples. For the sake of time I’m only showing the results from the human dataset here.

we used the Matthews Correlation Coefficient (MCC) to measure how well each algorithm adhered to the distance threshold. The MCC ranges from -1 to 1, with 1 being perfect quality where all pairs of sequences that are within the distance threshold are in the same OTU and all pairs of sequences that are more dissimilar than the threshold are in different OTUs.

on the y-axis we have each clustering strategy. first is de novo clustering with OptiClust and de novo clustering with another program called VSEARCH. we then tested OptiFit with a self-split strategy, where we clustered half of the sequences de novo then fit the remaining sequences to the de novo OTUs as a reference. the self-split strategy with OptiFit performed just as well as OptiClust in terms of MCC. Despite issues with public databases, traditional reference-based clustering to an external database is still fairly common, so we tested OptiFit’s ability to fit sequences to three databases: greengenes, silva, and the RDP. for database-depending clustering was highly dependent on the size of the database.

OptiFit benchmarking results

OptiFit improves reference-based clustering

• New method to use de novo OTUs as a reference for new sequences.
• Clusters sequences at the same quality as de novo methods.
• Outperforms other reference-based methods in terms of quality.
  • Trade-off: longer run time.

• Follow-up: OptiFit is suitable for machine learning.

So in conclusion for this project, we found that OptiFit is a new method that improves reference-based clustering.

In a follow-up study, Courtney Armour demonstrated that you can use OptiFit to fit new data to existing OTUs and machine learning models tested with the new data performed just as well as if all of the data had been clustered de novo.

Overview

1. Improve methods for processing microbiome data.
2. Predict severe C. difficile infections from gut microbiome composition.
3. Contribute to democratizing data science.

C. difficile infection (CDI)

Clostridioides difficile is a bacterium that causes infection.

CDI has a mortality rate of about 8% and that risk is higher in older patients. CDI is a significant burden on the US health care system.

Clinical data predict complicated CDI

median AUROC: 0.69

• Motivation: inform clinicians on which CDI patients may be most at risk of severe outcomes in order to tailor treatments.
• Electronic health record (EHR) data used as features to predict whether disease-related complications occurred.

Area under the receiver-operator characteristic curve (AUROC) - given a randomly selected severe case and a not severe case, what is the probability that the model correctly ranks the severe case as having a higher risk.

CDI treatment. Li et al. (2019). https://academic.oup.com/ofid/article/6/5/ofz186/5475497 “The development and validation of EHR-based risk stratification models for predicting complicated CDI could eventually help clinicians tailor treatments to individuals. On the day of CDI diagnosis, a patient’s estimated risk for complications could serve as an adjunct, easily obtainable resource for clinical decision support. Treatment decisions such as whether to use high-dose vancomycin or perform a loop ileostomy with antegrade vancomycin infusions [23] often do not occur until complicated CDI has already set in. In severe cases, early aggressive therapy can positively impact the course. However, invasive treatments such as enemas (fecal microbiota transplantation or vancomycin) and surgery are optimally used in only select patients, and such decisions lack the rigorous guidelines associated with initial treatment.”

Would help clinicians tailor treatment options early on. Whether to go with an aggressive treatment plan or not.

Can we use the composition of the gut microbiome to predict CDI severity?

Clinical factors have been used to predict CDI severity, and the initial state of the gut microbiome plays a role in colonization resistance and clearance.

Our central question is: Can we use the composition of the gut microbiome to predict CDI severity?

How to define CDI severity

1,257 CDI patient stool samples collected on the day of diagnosis

Severity	n	% severe
IDSA	1,072	34.2
All-cause	1,218	7.1
Attributable	1,178	2.2

Severity	n	% severe
IDSA	1,072	34.2
All-cause	1,218	7.1
Attributable	1,178	2.2
Pragmatic	1,218	5.4

First we need to talk about how to define CDI severity. There are a number of ways to do it. Following this branch to the left, you can consider serum biomarkers like creatinine level and white blood cell count. The Infectious Diseases Society of America uses elevated creatinine and wbc count as a measure of CDI severity. It effectively tells clinicians “how sick is this patient right now”. These data are also relatively easy to obtain.

On the other hand, we can look at the ultimate outcomes of the infection at 30 days since diagnosis.

So for each patient sample collected on the day of diagnosis, we performed amplicon sequencing and clustered sequences into operational taxonomic units. We used the relative abundances of our OTUs as features for training machine learning models to predict severe CDI outcomes.

Training machine learning models

Model performance

Random Forests trained on 100x train/test splits of each dataset

ROC = trade-off between correctly predicting positive vs negative samples. The baseline is 0.5 for a model that performs no better than random.

sensitivity/recall = \(TPR = \frac{TP}{TP + FN}\)

Out of all positive samples, how many correctly predicted positive.

specificity = \(TNR = \frac{TN}{TN + FP} = 1 - FPR\)

Out of all negative samples, how many correctly predicted negative.

\(FPR = \frac{FP}{FP + TN}\)

Out of all negative samples, how many incorrectly predicted positive.

Model performance

Random Forests trained on 100x train/test splits of each dataset

EHR-based models from Li et al.

Median AUROC: 0.69

We can also summarize ROC curves with the area under the roc curve (AUROC).

ROC = trade-off between correctly predicting positive vs negative samples.

sensitivity/recall = \(TPR = \frac{TP}{TP + FN}\)

Out of all positive samples, how many correctly predicted positive.

specificity = \(TNR = \frac{TN}{TN + FP} = 1 - FPR\)

Out of all negative samples, how many correctly predicted negative.

\(FPR = \frac{FP}{FP + TN}\)

Out of all negative samples, how many incorrectly predicted positive.

Model performance on severe cases

Random Forests trained on 100x train/test splits of each dataset

To ensure the AUROC isn’t artificially inflated due to the high proportion of negative samples in our dataset, we also plotted precision-recall curves.

ROC = trade-off between correctly predicting positive vs negative samples.

precision = \(PPV = \frac{TP}{TP + FP}\)

Out of the samples that were predicted positive, how many were correctly predicted.

sensitivity/recall = \(TPR = \frac{TP}{TP + FN}\)

Out of all positive samples, how many correctly predicted.

Top OTUs contributing to model performance

Each of the most important OTUs were more abundant in severe cases, and each have been associated with loss of colonization resistance or longer time to clearance in mouse studies of CDI. In particular, Enterococcus has been found to be associated with increased risk of severe outcomes in mouse studies.

Clinical value of prediction models

• The pragmatic severity model performed just as well as an EHR-based model.
• The most important OTUs for model performance concord with prior studies.
• Open question: would deploying EHR or OTU-based models improve clinical outcomes?

• The pragmatic severity model performed just as well as an EHR-based model.
• The most important OTUs for model performance concord with prior studies.
• However, decent model performance is not enough to justify model deployment.
• Open question: would deploying EHR or OTU-based models improve clinical outcomes?
  • A better comparison would extract EHRs from this same dataset and compare performance for models trained on EHRs, OTUs, or both.
  • Clinical value of prediction models is also highly dependent on treatment options available to help patients identified as high risk.
  • To assess clinical value, we need to consider the trade-offs of the resources required to train and deploy these models versus the potential benefit gained for patients.

Overview

1. Improve methods for processing microbiome data.
2. Predict severe C. difficile infections from gut microbiome composition.
3. Contribute to democratizing data science.

Democratizing data science

• Make data science tools more accessible to researchers from non-computational backgrounds.
• Disseminate user-friendly tools & curricula with open source licenses.
• Promote diversity, equity, and inclusion in data science.

To these ends, we developed resources targeting 3 key audiences. high schoolers, academic researchers, novice ML practitioners

Addressing the gender gap in STEM

• 18% of Computer Science degrees are awarded to women.
• U-M Bioinformatics PhD graduates: 25% women as of 2021.
• Most of us (women in U-M Bioinformatics) didn’t learn to code until college or later.

Solution: start a Girls Who Code club focused on teaching coding for data science to high school students.

Intro to Python for Data Science

for Girls Who Code at U-M DCMB

teach high schoolers the basics of coding in Python for data science. from knowing no programming to being able to analyze a data science and present findings at the end of the club.

live coding. hands on lessons and practice notebooks. pivot to flipped format for pandemic.

1. Each lesson begins with a recapping of the relevant core skills presented in the previous lessons.
2. All lessons are designed to be taught via 15-minute live-coding sessions. This method is used by The Carpentries and is demonstrated to be an effective method that engages learners (Nederbragt et al., 2020; Wilson, 2016) since learners must actively engage with the material and deal with errors and bugs as they arise.
3. Each lesson ends with a summary of core skills presented within the material.
4. Each short lesson is also accompanied by a subsequent 10-minute independent practice, providing further opportunity for practical experience implementing the coding skill at hand and testing learners’ understanding of the content.

Impact of Girls Who Code

• 135+ graduates of the club & summer camp.
• Graduates improve skills in coding, problem solving, collaboration, and self-confidence.

Because of GWC, I learned about bioinformatics. I was very interested in it, because it combined my interests in programming and research… I’m pretty certain I want to go into bioinformatics now.

I plan to go to college for Computer Science and get a robotics minor when my college offers it. GWC has inspired me to consider pursuing a Masters or PhD in CS as well as take some electives in Data Science.

Coding for Reproducible Research

• 34,000+ learners have attended Software Carpentry workshops worldwide since 2010.
• U-M instance of Software Carpentry active since 2016.

• Software Carpentry is an international non-profit that teaches grad students, postdocs, and other researchers how to perform reproducible computational science via interactive coding workshops.
• Over 34,000 learners have attended since 2010, ranging from biologists to physicists to engineers and economists.
• U-M instance of Software Carpentry active since 2016.

Integrated workshop curriculum

teaching R, the Unix shell, and Git

Impact of Software Carpentry

Pilot workshop pre & post survey

New curriculum used in 7 workshops to date.

Reproducible ML pipelines with mikropml

"meek-rope em el"

• Typical microbiologist lacks ML expertise
• Many microbiologists have enough computational skills to be “dangerous”
• Common pitfalls
  • Lack of transparency in method implementation
  • No separate held-out test data
  • Not reporting variation in the predictive performance on different folds of cross-validation, or cross-validation vs. testing performances

Goals

• Restructure our ML framework to improve reusability, scalability, & reproducibility.
• Strike a balance between easy-to-use interface for beginners with reasonable default options, with enough flexibility for advanced users to customize options.
• We implemented this as an R package and also included a snakemake workflow demonstrating how to use the package in a high performance computing environment.
  • We intend to use it alongside other software in the lab: mothur for processing amplicon sequences and schtools – schloss lab tools for reproducible microbiome research.
  • But mikropml is applicable outside just the microbiome field. So we published it in the Journal of Open Source Software

best practices in swe

• GitFlow, issue tracking, peer review
• Unit tests + CI
• Document functions & vignettes
• Semantic versioning, changelog

mikropml impact

CRAN:

conda-forge:

In the lab

• CRC classification across taxonimic levels
• Mouse C. diff colonization & clearance
• Mouse models of colitis

In the wild

• Diet and antimicrobial resistance
• Evolution of grasses
• Psychology research
• Political science

Summary

• Improved OTU clustering algorithms to enable machine learning for microbiome research.
• Applied machine learning to predict severe C. difficile infections, demonstrating similar performance as models trained on Electronic Health Records.
• Contributed to democratizing data science for three key audiences.

post-PhD plans

Bioinformatics Software Engineer for Frederick National Lab (NIH/NCI)

• Improved OTU clustering algorithms to enable ML for microbiome research.
• Applied ML to predict severe CDIs, demonstrating similar performance as EHR models.
• Contributed to democratizing data science for three key audiences: high schoolers, researchers, and novice ML practitioners.

Acknowledgements

Schloss Lab
Patrick Schloss
Megan Coden
Sarah Lucas
Courtney Armour
Allison Mason
Adena Collens

alumni
Nick Lesniak
Sarah Tomkovich
Sarah Westcott
Katie McBride
Jay Moltzau
Begüm Topçuoğlu
Joshua Stough
Will Close
Ada Hagan
Kaitlin Flynn

Curricula contributors
Marlena Duda
Zena Lapp
Brooke Wolford
Negar Farzaneh
Vy Nguyen
Sarah Haynes
Hayley Falk
Katie Furman
Logan Walker
Rucheng Diao
Morgan Oneka
Audrey Drotos
Gabrielle Dotson
April Kriebel
Lucy Meng
Stephanie Theide
Dana King
Catherine Barnier
Matthew Flickinger
Jule Krüger
Maya Lapp
Jason Tallant

Collaborators
& Committee
Krishna Rao
Vince Young
Jenna Wiens
Greg Dick

Funding
NIH T32 GM070449
NIH U01AI124255

I have one more plot to show you all. Early on in my PhD, my friends and I noticed that there was a particular phrase we kept saying fairly regularly. And not just us, but also colleagues and collaborators and others in our scientific circles.

So I decided to keep a tally.

Phrases included

• “Pat’s the best”
• “Pat is a delight.”
• “Wow this is incredible. Pat is literally a hero!”
• “Your PI is amazing”

That phrase is “Pat is the best”.

For the sake of completeness I also included similar phrases uttered such as “Pat is a delight”, “Pat is literally a hero”, and “Your PI is amazing”.

There were so many times throughout my PhD when Pat did or said something that elicited this response. From his response to COVID, to his support of my work with Girls Who Code and other volunteer efforts, all the ups and downs of grad school… Pat even made an educational YouTube channel to teach people how to code.

Phrases included

• “Pat’s the best”
• “Pat is a delight.”
• “Wow this is incredible. Pat is literally a hero!”
• “Your PI is amazing”

(This one is my favorite thumbnail from his videos.)

Finally, Pat has persistently done a great job of assembling a lab.

I thanked members of the Schloss Lab earlier for your scientific contributions but now I’m thanking you again for being wonderful people because you are all just that special and played such a huge role in shaping me as a scientist. Many lab mates have come and gone over the years and every one I’ve interacted with has individually contributed to an awesome positive and supportive lab culture that I’m proud to be a part of.

Next I’d like to thank my friends. You all helped keep me sane during grad school. We made sure to have lots of fun and maintain a healthy work-life balance in this crazy chapter of life.

Finally, I’d like to thank my family - my sisters Katie and Erika, my parents, and my boyfriend David. David has the uncanny ability to take any rainy day and turn it into a sunny one. My parents have been my cheerleaders from the very beginning, and they always encouraged me to pursue whatever I was interested in. Your support means everything to me.

Thank you all for your attention and I’ll take any questions now