Source-linked AI summary
Stochastic Variational Inference
Matt Hoffman, David M. Blei, Chong Wang, John Paisley
TL;DR
Massive data sets challenge existing posterior-inference methods, motivating a general scalable approach. The paper develops stochastic variational inference by combining variational inference with stochastic optimization and subsampling, and demonstrates it on LDA and HDP topic models. It applies the method to collections containing millions of documents and argues that the approach generalizes broadly, while noting limitations for some nonconjugate models and reliance on predictive-distribution evaluation.
Problem
Existing MCMC and variational-inference methods do not scale easily to massive data sets, motivating a general scalable inference method.
Method
Stochastic variational inference uses stochastic optimization, repeatedly subsampling data to obtain noisy gradient estimates for variational updates.
Results
The method is demonstrated on LDA and HDP topic models and applied to collections containing millions of documents.
Takeaways & Limitations
Stochastic variational inference enables complex Bayesian models to be applied to massive data sets and generalizes beyond the demonstrated topic models.
Takeaways & Limitations
The presented method cannot be used for some nonconjugate models, and traditional held-out perplexity may fail to reveal overfitting.
Abstract
from arXiv · showhide
We develop stochastic variational inference, a scalable algorithm for approximating posterior distributions. We develop this technique for a large class of probabilistic models and we demonstrate it with two probabilistic topic models, latent Dirichlet allocation and the hierarchical Dirichlet process topic model. Using stochastic variational inference, we analyze several large collections of documents: 300K articles from Nature, 1.8M articles from The New York Times, and 3.8M articles from Wikipedia. Stochastic inference can easily handle data sets of this size and outperforms traditional variational inference, which can only handle a smaller subset. (We also show that the Bayesian nonparametric topic model outperforms its parametric counterpart.) Stochastic variational inference lets us apply complex Bayesian models to massive data sets.
1. Introduction
Modern data analysis needs inference methods that scale to massive, complex, and potentially streaming data. The paper develops stochastic variational inference, using stochastic optimization and subsampling to make graphical-model inference more efficient and broadly applicable.
- Motivation: Massive, complex, and potentially never-ending data sets create a scale problem for traditional inference algorithms.Existing methods cannot easily handle data volumes such as millions of books, users, photographs, or gene sequences.
- Empirical illustration: The approach supports posterior topic analysis of 1.8M New York Times articles and 350K Nature articles.Each topic is represented as a weighted distribution over vocabulary, visualized through its most frequent words.
- Contribution: The method is derived for a large class of graphical models and demonstrated on LDA and the more flexible hierarchical Dirichlet process topic model.The HDP application illustrates use in Bayesian nonparametric settings where the number of discovered topics grows with the data.
- Motivation: Neither MCMC nor traditional variational inference scales easily to massive data, while existing speed-ups are often model-specific or compromise correctness.The paper targets a general scalable variational method rather than a narrowly tailored acceleration.
- Method: Stochastic optimization makes variational inference scalable by estimating gradients from independently sampled data subsets.The variational objective decomposes across data points, so subsampling can produce unbiased noisy gradient estimates.
- Method: Stochastic variational inference repeatedly subsamples data, analyzes the subsample with current parameters, applies a closed-form update, and repeats.This procedure avoids reanalyzing the entire data set at every iteration.
2. Stochastic Variational Inference
Stochastic variational inference approximates posterior distributions for probabilistic models by combining mean-field variational inference with stochastic optimization. It uses subsampling and noisy natural gradients to update global parameters efficiently, enabling inference on massive data sets while retaining convergence guarantees under suitable step-size conditions.
- Stochastic variational inference approximates posterior distributions in probabilistic models with hidden variables and handles massive data sets of observations.
- It combines mean-field variational inference with stochastic optimization, repeatedly subsampling data to obtain noisy estimates of the natural gradient of the ELBO.
- The method applies to models with global and local hidden variables whose joint distribution factorizes into a global term and local terms.
- At each iteration, the algorithm computes local parameters for a sampled observation, forms an intermediate global update, and averages it with the current global parameters.
- The algorithm improves global parameters without analyzing the whole data set first, and each iteration requires computation about only one local context.
- With suitable step-size conditions, the stochastic procedure converges to a local optimum of the ELBO, while stochastic natural-gradient estimates remain unbiased in expectation.
3. Stochastic Variational Inference in Topic Models
The paper derives stochastic variational inference for LDA and HDP topic models, targeting posterior inference in massive document collections. Across these models, stochastic inference scales beyond traditional batch methods and achieves faster convergence to better models.
- Model-specific stochastic inference: The paper derives stochastic variational inference for LDA and its Bayesian nonparametric counterpart, the HDP topic model.The derivation proceeds from model assumptions through complete conditionals and variational families to stochastic algorithms.
- Model-specific stochastic inference: Topic models represent documents with shared topics and document-specific topic proportions, while topic assignments and topics remain latent.LDA uses a fixed number of topics; HDP allows the posterior to determine how many topics are needed.
- Empirical results: 350K Nature, 1.8M New York Times, and 3.8M Wikipedia documents were analyzed; stochastic inference converged faster and to a better model than batch inference on 100,000-document subsets.The comparison used 100-topic LDA and per-word predictive log likelihood.
- LDA inference: Traditional LDA variational inference is inefficient because updating each topic parameter requires summing over variational parameters for every word in the collection.This full-collection dependency is the root of its poor scaling on large document sets.
- LDA inference: Stochastic variational inference scales LDA inference by approximating global updates from sampled documents while retaining local variational parameters for documents and word assignments.Its global parameters are topic Dirichlet parameters, while local parameters describe document topic proportions and word-topic assignments.
- Empirical results: For HDP, stochastic variational inference also converged faster and to a better model, while the model's posterior determines the number of topics needed.The HDP construction separates global and local variables and supports an unbounded number of topics.
4. Empirical Study
The empirical study evaluates stochastic variational inference for LDA and HDP on three large document collections, comparing predictive performance and sensitivity to learning parameters. Stochastic inference scales to these collections, converges faster and to better models than batch inference on subsets, and HDP is consistently more robust than LDA.
- Experimental setup: The study evaluates LDA and HDP on three large document collections using held-out predictive likelihood to measure model fitness.The collections include 350,000 Nature documents, 1.8M New York Times documents, and 3.8M Wikipedia documents; 10,000 documents per collection were held out for testing.
- Experimental setup: The predictive evaluation estimates topic proportions from observed test words and scores the held-out words under the resulting predictive distribution.Training data estimate the topic posterior, while observed test words estimate document topic proportions before evaluating held-out-word log probabilities.
- Model comparison: The HDP gives consistently better predictive performance than LDA and remains robust to overfitting as the number of LDA topics increases.The authors attribute this robustness in part to HDP corpus-breaking proportions, which allow topics to have unequal prior prevalence.
- Evaluation caveat: The predictive distribution reveals LDA overfitting that held-out perplexity does not, although HDP still outperforms LDA under that traditional metric.The authors therefore regard predictive distribution as a better model-fitness metric for this comparison.
- Learning-parameter sensitivity: Higher forgetting rates improve HDP convergence, while larger mini-batches generally perform better and batch sizes of 500 and 1,000 show little difference.With batch size fixed at 500, values of κ closer to one converge to better local optima; with κ = 0.9, the New York Times corpus is most batch-size-sensitive and Wikipedia least sensitive.
5. Discussion
The discussion presents stochastic variational inference as a scalable framework for complex probabilistic models, while identifying extensions beyond its current assumptions and opportunities for improved stochastic inference.
- Contributions: Stochastic variational inference applies stochastic optimization to noisy natural-gradient estimates obtained by repeatedly subsampling data.The method was demonstrated with latent Dirichlet allocation and hierarchical Dirichlet process topic models.
- Extensions: Subsequent work extended stochastic inference to mixed-membership stochastic blockmodels, non-uniform data sampling, MCMC local updates, and adaptive learning rates.Adaptive learning rates were reported to outperform preset schedules and require less hand-tuning.
- Parameter settings: Slower forgetting rates are preferred when varying the forgetting-rate parameter with batch size held fixed.The parameter grid was simplified for presentation by holding one parameter fixed while varying the other.
- Limitations and future work: The presented general algorithm cannot be used for nonconjugate models, motivating stochastic methods beyond conjugate exponential-family settings.The paper also identifies moving beyond closed-form updates and fully factorized approximate posteriors as a promising direction.