PaperlayerView exact revision on arXiv

Source-linked AI summary

Autoencoding Variational Inference For Topic Models

Akash Srivastava, Charles Sutton

arXiv:1703.01488v1stat.ML

TL;DR

Topic-model changes traditionally require new inference derivations, while AEVB is difficult to apply because of the Dirichlet prior and component collapsing. The paper introduces AVITM, an effective black-box AEVB method, and uses it to build ProdLDA, which yields better topics with minimal implementation changes.

  • Problem

    Existing inference methods require re-derivation for changed topic models, while applying AEVB is difficult because of the Dirichlet prior and component collapsing.

  • Method

    The paper develops AVITM, an AEVB inference method for LDA that addresses Dirichlet-prior and component-collapsing problems, and applies it to ProdLDA.

  • Results

    ProdLDA consistently produces better topics than LDA, while AVITM matches standard mean-field topic quality and avoids variational optimization on test data.

  • Takeaways & Limitations

    AVITM is presented as a black-box inference method applicable to new topic models, with ProdLDA requiring only a one-line implementation change from LDA.

  • Takeaways & Limitations

    The reported experiments could not provide topic coherence for DMFVI on RCV1.

Abstract

from arXiv · show

Topic models are one of the most popular methods for learning representations of text, but a major challenge is that any change to the topic model requires mathematically deriving a new inference algorithm. A promising approach to address this problem is autoencoding variational Bayes (AEVB), but it has proven diffi- cult to apply to topic models in practice. We present what is to our knowledge the first effective AEVB based inference method for latent Dirichlet allocation (LDA), which we call Autoencoded Variational Inference For Topic Model (AVITM). This model tackles the problems caused for AEVB by the Dirichlet prior and by component collapsing. We find that AVITM matches traditional methods in accuracy with much better inference time. Indeed, because of the inference network, we find that it is unnecessary to pay the computational cost of running variational optimization on test data. Because AVITM is black box, it is readily applied to new topic models. As a dramatic illustration of this, we present a new topic model called ProdLDA, that replaces the mixture model in LDA with a product of experts. By changing only one line of code from LDA, we find that ProdLDA yields much more interpretable topics, even if LDA is trained via collapsed Gibbs sampling.

1 INTRODUCTION

Topic-model inference is computationally demanding and difficult to adapt when model assumptions change. AVITM addresses these challenges with black-box inference, enabling faster inference and the ProdLDA model.

  • Topic models support unsupervised text representations across many applications, but posterior computation remains a major computational challenge.
  • Existing mean-field and collapsed-Gibbs methods require mathematically arduous re-derivations when topic-model assumptions change.
  • AEVB uses an inference network to map documents directly to approximate posterior distributions, avoiding further variational updates on test data.
  • AVITM is the first effective AEVB inference method presented for topic models, matching standard mean-field topic quality while reducing training time.
  • ProdLDA replaces LDA’s topic mixture with a product of experts and consistently produces better topics than standard LDA.
  • AVITM is fast on new data, black-box, and readily applicable to a wide range of topic models.

2 BACKGROUND

LDA represents documents as mixtures of topics but has intractable posterior inference. Variational and autoencoding approaches approximate this posterior, with AEVB using a neural inference network to compute variational parameters from observed data.

  • 2.1 LATENT DIRICHLET ALLOCATION: In LDA, each document is represented as a mixture of topics, with each topic defined as a probability distribution over the vocabulary.
  • 2.1 LATENT DIRICHLET ALLOCATION: LDA generates each word by sampling a topic from document proportions and then sampling the word from that topic’s distribution.
  • 2.1 LATENT DIRICHLET ALLOCATION: Posterior inference over document topic proportions and word-topic assignments is intractable because of coupling under the multinomial assumption.
  • 2.2 VARIATIONAL INFERENCE: Mean-field variational inference breaks the coupling between topic proportions and assignments using free variational parameters for each document.
  • 2.2 VARIATIONAL INFERENCE: For LDA, mean-field inference has closed-form coordinate-descent updates, but applying it to new models requires deriving those updates.
  • 2.3 AUTOENCODING VARIATIONAL BAYES: AEVB rewrites the variational objective as a prior-matching term plus an expected data-reconstruction term.
  • 2.3 AUTOENCODING VARIATIONAL BAYES: Unlike mean-field inference, AEVB computes variational parameters with an inference network that takes observed data as input.
  • 2.3 AUTOENCODING VARIATIONAL BAYES: The reparameterization trick uses an auxiliary variate independent of variational parameters to estimate expectations during optimization.

3 AUTOENCODING VARIATIONAL BAYES IN LATENT DIRICHLET ALLOCATION

Applying AEVB to LDA requires addressing reparameterization of the Dirichlet topic proportions and component collapsing. The paper uses a logistic-normal approximation and optimization changes to make inference effective.

  • Challenges: AEVB is difficult to apply to LDA because Dirichlet distributions hinder reparameterization and component collapsing can make inferred topics identical.The collapsing arises when the inference network reaches a poor local optimum near the prior.
  • Collapsed variables: Collapsing the discrete topic assignments z leaves only θ to sample, simplifying reparameterized inference for LDA.The resulting word distribution is Multinomial(1, βθ).
  • Dirichlet approximation: The Dirichlet prior is approximated by a logistic normal in the softmax basis, enabling unconstrained optimization with diagonal covariance.The approximation uses a multivariate normal over softmax variables before expressing it in the simplex basis.
  • Variational inference: The inference network defines document-dependent logistic-normal means and diagonal covariances for the topic proportions.Separate feed-forward networks produce the variational mean and covariance from each document.
  • Training: High-momentum, higher-learning-rate ADAM training and batch normalization are used to avoid early collapsing and optimizer divergence.Batch normalization smooths the functional space and curbs sudden divergence.
  • Training: Dropout applied to θ increases performance by forcing the inference network to use more of its capacity.The reported improvement is attributed to applying dropout units to the topic-proportion representation.

4 PRODLDA: LATENT DIRICHLET ALLOCATION WITH PRODUCTS OF EXPERTS

ProdLDA replaces LDA’s word-level mixture of multinomials with a weighted product of experts. This change allows sharper predictions and produces more coherent topics.

  • Model: ProdLDA replaces LDA’s word-level mixture assumption with a weighted product of experts.The topic matrix is unnormalized, and word probabilities use σ(βθ).
  • Motivation: Because products of experts can make sharper predictions than individual experts, ProdLDA addresses a limitation of mixture-based word distributions.The paper links this modeling choice to improved topic coherence.
  • Model: The product-of-experts formulation corresponds to mixing natural parameters, which yields a weighted geometric average of multinomial mean parameters.This connects the natural-parameter representation to the product-of-experts interpretation.
  • Connections: ProdLDA is an instance of exponential-family PCA and relates to exponential-family harmoniums with non-Gaussian priors.These connections situate the model within broader exponential-family latent-variable models.

5 RELATED WORK

Prior work had explored neural topic models and neural variational inference, but the paper identifies a gap in generic AEVB application to analyst-specified topic models. It also distinguishes its approach from NVDM.

  • Prior neural methods: Existing work includes topic models based on neural networks and neural variational inference, but not generic AEVB for analyst-specified topic models.The paper also reports no successful AEVB application to LDA at the time described.
  • NVDM: NVDM uses a latent Gaussian topic distribution and averages topic-word distributions in logit space.The paper presents NVDM as closely related work.
  • Distinction from NVDM: The paper identifies explicit Gaussian approximation of the Dirichlet prior and high-momentum training as two aspects absent from NVDM.Its experiments associate these aspects with improved training and better topics.

6 EXPERIMENTS AND RESULTS

Experiments compare AVITM with standard inference methods across topic coherence, perplexity, training speed, and test-time inference, while also evaluating ProdLDA and prior choices. AVITM achieves similar quality to mean-field inference with substantially faster training, and ProdLDA produces better topics than LDA.

  • Evaluation metrics: NPMI is adopted as the primary automatic topic-coherence metric because it most closely matches human judgments among competing metrics.Perplexity is also reported, primarily to evaluate the capability of different optimizers.
  • AVITM versus standard inference: AVITM yields similar topic coherence and perplexity to standard mean-field inference while training substantially faster.Training takes 46 seconds on 20 Newsgroups versus 18 minutes for DMFVI; on RCV1, AVITM takes under 1.5 hours while DMFVI fails to return results after 24 hours.
  • ProdLDA evaluation: ProdLDA finds significantly better topics than LDA, including when LDA is trained with collapsed Gibbs sampling.The topics from ProdLDA also appear visually more coherent than those from NVDM or LDA.
  • Test-time inference: Inference-network-only test perplexity remains practically unchanged compared with adding variational optimization on the test set.Computing perplexity with the neural network takes well under a minute on both datasets, versus approximately 3 minutes for standard variational approximation even on 20 Newsgroups.
  • Prior effects: The Laplace approximation to Dirichlet priors significantly promotes sparsity in posterior topic proportions relative to the standard Gaussian prior used by NVDM.This supports the hypothesis that preserving the Dirichlet prior contributes to increased topic coherence.

7 DISCUSSION AND FUTURE WORK

The paper presents AVITM as an effective black-box inference method for LDA and illustrates its flexibility with ProdLDA. It concludes that ProdLDA produces better topics than LDA with minimal code changes.

  • Discussion: AVITM is presented as the first effective AEVB inference algorithm for latent Dirichlet allocation.The method addresses training difficulties associated with the Dirichlet prior and component collapsing.
  • Discussion: AVITM computes topic proportions for new documents without running further variational optimization.This advantage comes from using a neural-network inference method.
  • Discussion: Table 6 shows five randomly selected topics from all the models.The table is used to compare sampled topics across models.
  • Discussion: Table 7 reports that VAE-LDA fails to learn meaningful topics when component collapsing occurs.The displayed topics are five randomly sampled, essentially slight variants of one another, from training without batch normalization and high-momentum training.
  • Future Work: ProdLDA achieves significantly better topics than LDA while requiring only one line of code changed from AVITM for LDA.The paper presents this as an illustration of the advantages of black-box inference techniques.

A NETWORK ARCHITECTURE

Figure 2 presents the architecture of the inference network used in the experiments.

  • Network Architecture: Figure 2 depicts the inference network architecture used in the experiments.The passage identifies the figure's subject but does not describe its components or connections.
  • Network Architecture: The figure concerns the network used for the paper's experimental setup.No further architectural details are stated in the supplied passage.
  • Network Architecture: Figure 2 provides an architectural view of the experimental inference network.The supplied text does not specify the network's inputs, outputs, or layers.
Loading 1703.01488v1…