Source-linked AI summary
Waste Not, Want Not: Why Rarefying Microbiome Data is Inadmissible
Paul J. McMurdie, Susan Holmes
TL;DR
Microbiome sequencing data have unequal library sizes and biological overdispersion, challenging simple normalization. The paper combines statistical theory, simulations, and empirical data to evaluate mixture-model normalization against proportions and rarefaction. Mixture models improved performance and supported avoiding rarefaction, despite threshold and data-loss limitations.
Problem
Microbiome sequencing counts vary widely in library size and are overdispersed relative to a Poisson model, requiring normalization that accounts for both sources of variability.
Method
The paper uses statistical theory, simulations, and empirical data to evaluate hierarchical mixture-model normalization for microbiome counts.
Results
Mixture-model normalization improved sensitivity and specificity, whereas proportions and rarefied counts produced unacceptably high false-positive rates.
Takeaways & Limitations
The authors recommend modeling microbiome count uncertainty with hierarchical mixtures and avoiding rarefying, particularly for differential-abundance analysis.
Takeaways & Limitations
Rarefying requires an arbitrary minimum library-size threshold whose optimal value cannot be known for new empirical data, and it discards data.
Abstract
from arXiv · showhide
The interpretation of count data originating from the current generation of DNA sequencing platforms requires special attention. In particular, the per-sample library sizes often vary by orders of magnitude from the same sequencing run, and the counts are overdispersed relative to a simple Poisson model These challenges can be addressed using an appropriate mixture model that simultaneously accounts for library size differences and biological variability. This approach is already well-characterized and implemented for RNA-Seq data in R packages such as edgeR and DESeq. We use statistical theory, extensive simulations, and empirical data to show that variance stabilizing normalization using a mixture model like the negative binomial is appropriate for microbiome count data. In simulations detecting differential abundance, normalization procedures based on a Gamma-Poisson mixture model provided systematic improvement in performance over crude proportions or rarefied counts -- both of which led to a high rate of false positives. In simulations evaluating clustering accuracy, we found that the rarefying procedure discarded samples that were nevertheless accurately clustered by alternative methods, and that the choice of minimum library size threshold was critical in some settings, but with an optimum that is unknown in practice. Techniques that use variance stabilizing transformations by modeling microbiome count data with a mixture distribution, such as those implemented in edgeR and DESeq, substantially improved upon techniques that attempt to normalize by rarefying or crude proportions. Based on these results and well-established statistical theory, we advocate that investigators avoid rarefying altogether. We have provided microbiome-specific extensions to these tools in the R package, phyloseq.
Author Summary
Microbiome count data require normalization because sequencing libraries differ greatly in size. Mixture-model methods outperform proportions and rarefaction for comparing species abundances.
- Sequencing samples from the same run can have vastly different total numbers of sequences, creating unequal library sizes.
- Common approaches either analyze proportional abundance or discard sequences from larger libraries through rarefaction.
- Both approaches can work acceptably for comparing entire microbiomes but perform poorly when comparing bacterial species proportions across samples.
- Statistical mixture-model methods perform very well for comparing microbial abundances.
Introduction
Microbiome sequencing counts vary greatly in library size and exhibit biological overdispersion, making simple scaling or Poisson-based analysis inadequate. The paper motivates mixture-model normalization and explains why rarefying discards information and reduces statistical power.
- Introduction: Library sizes can vary by orders of magnitude within a single sequencing run, and unequal denominators produce unequal uncertainty across feature proportions.Simple linear or logarithmic scaling does not account for the resulting sampling variance differences.
- Introduction: Biological replicate variation requires a mixture model, such as Gamma-Poisson, because Poisson models capture technical but not added biological uncertainty.The Gamma-Poisson framework has provided satisfactory fits for RNA-Seq data and supports inference with overdispersed counts.
- Introduction: Rarefying normalizes libraries by selecting a minimum size, discarding smaller libraries, and subsampling the remaining libraries without replacement.The minimum size is often set to the smallest retained library, making defective-sample identification potentially subjective and bias-prone.
- Statistical motivation: Microbiome OTU variances exceed Poisson expectations, and variance-stabilizing mixture-model normalization addresses both unequal library sizes and overdispersed proportions.The paper presents this approach as an alternative to statistically inadmissible rarefying.
- Statistical motivation: Reducing a 1000-read sample to 100 reads prevents formal tests from distinguishing it from a 100-read sample, despite differences visible in the original data.The loss of power results from reducing the larger sample tenfold and widening its proportion confidence intervals.
- Statistical motivation: Rarefying equalizes variances by inflating them to the largest value, while random subsampling adds uncertainty and reduces discriminating power.A deterministic common-scale transformation avoids the random step but still loses data and shares rarefying’s information-loss problem.
Materials and Methods
The study evaluates normalization and distance/modeling choices through two simulation frameworks: clustering whole microbiome samples and detecting differentially abundant OTUs. Simulations vary library and effect sizes, using empirical microbiome data and known answers to compare performance.
- Simulation design: The simulations repeatedly subsample empirical microbiome data, vary library and effect sizes, and evaluate methods against known answers.Library sizes were sampled from the Global Patterns dataset distribution and scaled to prescribed median library sizes.
- Simulation A: Simulation A distinguishes whole-microbiome sample relationships by normalizing counts, calculating sample-wise distances, and clustering with partitioning around medoids.Ocean and Feces templates with negligible OTU overlap were mixed to control classification difficulty from trivial to impossible.
- Simulation A: Simulation A compares DESeq variance stabilization, untransformed counts, proportions, rarefying, and UQ-logFC normalization before distance calculation.Rarefying uses the 15th-percentile library size within each simulated experiment.
- Simulation A: The clustering analysis evaluates Bray-Curtis, Euclidean, PoissonDistance, top-MSD, unweighted UniFrac, and weighted UniFrac distances.Distance matrices are calculated sample-wise after each applicable normalization.
- Simulation B: Simulation B applies defined perturbations to random OTU subsets in two equally sized classes, then tests differential abundance using parametric, rarefied, proportion, and multiple-testing workflows.Methods include Welch t-tests, edgeR, DESeq, DESeq2, and metagenomeSeq; tests use Benjamini-Hochberg correction.
Results and Discussion
Across simulations and empirical surveys, microbiome counts were overdispersed, while rarefying impaired clustering and differential-abundance inference. Mixture-model methods improved sensitivity and specificity, whereas rarefying introduced power loss, false positives, and threshold-dependent decisions.
- Count variability: Overdispersion was consistently observed among biological replicates, with variances exceeding Poisson expectations and a mean–variance trend supporting Negative Binomial modeling.This pattern held across surveyed microbiome datasets, including abundant OTUs.
- Clustering: Rarefying undermined clustering by discarding reads, adding sampling noise, and removing small-library samples that alternative methods clustered accurately.Its impact depended on effect size, typical library size, and the minimum-library threshold.
- Clustering: The optimal rarefying threshold hovered around the 15th percentile in most simulations but was not generalizable to new data.Lower thresholds retain more samples with fewer reads, whereas higher thresholds discard more samples but retain larger libraries.
- Differential abundance: Mixture-model normalization and testing improved sensitivity and specificity over rarefied counts or proportions in differential-abundance simulations.More sophisticated parametric methods showed large gains when rarefying was omitted.
- Differential abundance: Proportions and rarefied counts produced unacceptably high false-positive rates that increased with effect size, while Negative Binomial methods had near-zero false positives under most conditions.The false positives reflect underestimated uncertainty when overdispersion is not modeled.
- Implications: Rarefying combines data loss with persistent overdispersion, increasing Type-II error and Type-I error while making dispersion estimation more difficult.Random subsampling also adds artificial uncertainty and requires an arbitrary threshold whose optimum is unknown in practice.
- Implications: The authors advocate avoiding rarefying and recommend DESeq2 or edgeR for microbiome differential-abundance analysis.They note that alternative RNA-Seq methods may outperform DESeq under special conditions, while gains from modeling mixture noise are generally much larger than method-to-method differences.
Text S1: Mathematical Supplement
The supplement develops hierarchical mixture models for microbiome count data, especially the negative binomial and Gamma-Poisson formulations, to model biological and library-size variability. Variance-stabilizing transformations make variability more nearly constant and support more efficient comparisons than rarefying.
- The negative binomial can be parameterized by its mean m and overdispersion φ, with variance Var(X) = m + φm^2.When φ = 0, the negative binomial reduces to Poisson(m).
- Microbiome count data require mixture models because biological replicate variation and library-size differences add variability beyond technical Poisson variation.
- Variance-stabilizing transformations seek approximately constant variance across noise levels, enabling more suitable comparisons across samples with unequal variability.The square-root transformation approximately stabilizes variance for Poisson variables, while generalized logarithmic transformations are used in related microarray and RNA-seq normalization.
- The hierarchical microbiome model treats taxa proportions as Gamma-distributed and generates read counts through a Poisson model with sample-specific library sizes.Condition-specific parameters yield negative binomial counts for taxa across biological replicate conditions.
- Sharing information across OTUs improves estimates of mean and overdispersion, producing a variance-stabilizing transformation for statistically efficient comparisons between conditions.
- Rarefying is statistically inadmissible because a hierarchical Bayes procedure dominates it under mean squared error loss.The supplement presents variance-stabilizing transformations as yielding more efficient estimators and better decision rules than normalization through subsampling.