PaperlayerView exact revision on arXiv

Source-linked AI summary

WHAI: Weibull Hybrid Autoencoding Inference for Deep Topic Modeling

Hao Zhang, Bo Chen, Dandan Guo, Mingyuan Zhou

arXiv:1803.01328v2stat.MLstat.APstat.CO

TL;DR

Deep topic models may require iterative test-document inference, and common VAE latent distributions and parameter estimates have limitations for document representations. WHAI pairs stochastic-gradient MCMC with a Weibull upward-downward variational autoencoder; it reportedly performs comparably to Gibbs sampling while supporting scalable training and fast out-of-sample prediction.

  • Problem

    Deep topic models often use iterative inference for each test document, while Gaussian-based VAEs can poorly approximate sparse, nonnegative document representations and often use point estimates of global parameters.

    Source: §1 · pp. 1–2

  • Method

    WHAI combines stochastic-gradient MCMC for global parameters with a Weibull upward-downward variational autoencoder for local parameters in deep latent Dirichlet allocation.

    Source: §1 · p. 2

  • Results

    WHAI clearly outperforms GHAI, both outperform counterparts without top-down inference links, and WHAI is comparable to Gibbs sampling in performance.

    Source: §1 · p. 2

  • Takeaways & Limitations

    WHAI supports scalable mini-batch training and considerably fast out-of-sample prediction through an inference network.

    Source: §1 · p. 3

    For the gamma-based variation using rejection-sampling variational inference, ELBO gradients can have relatively high variance and require extra uniform random variables.

    Source: §2.4 · p. 7

Abstract

from arXiv · show

To train an inference network jointly with a deep generative topic model, making it both scalable to big corpora and fast in out-of-sample prediction, we develop Weibull hybrid autoencoding inference (WHAI) for deep latent Dirichlet allocation, which infers posterior samples via a hybrid of stochastic-gradient MCMC and autoencoding variational Bayes. The generative network of WHAI has a hierarchy of gamma distributions, while the inference network of WHAI is a Weibull upward-downward variational autoencoder, which integrates a deterministic-upward deep neural network, and a stochastic-downward deep generative model based on a hierarchy of Weibull distributions. The Weibull distribution can be used to well approximate a gamma distribution with an analytic Kullback-Leibler divergence, and has a simple reparameterization via the uniform noise, which help efficiently compute the gradients of the evidence lower bound with respect to the parameters of the inference network. The effectiveness and efficiency of WHAI are illustrated with experiments on big corpora.

1 INTRODUCTION

Existing deep topic models can require iterative inference for each test document, while common VAE latent distributions and global-parameter estimates have limitations for document representations. WHAI combines stochastic-gradient MCMC with a Weibull upward-downward variational autoencoder; the paper reports performance comparable to Gibbs sampling alongside scalable training and fast out-of-sample prediction.

1 INTRODUCTION

  • Prior deep document models include binary-hidden DBMs and Poisson factor models, while DEF and PGBN use gamma hidden units; PGBN couples layer shape parameters through weights and next-layer units.The paper describes that PGBN factorization as yielding strong nonlinearity and interpretable multilayer latent representations.Source: §1 · p. 1
  • Existing deep topic models often require iterative test-document inference, while Gaussian-based VAEs can poorly approximate sparse, nonnegative latent representations and often estimate global generative parameters with point estimates.The introduction also notes that AVITM uses a heuristic to encourage sparse document representations.Source: §1 · pp. 1–2
  • WHAI combines a DLDA gamma-belief-network generative model with Weibull upward-downward inference and stochastic-gradient MCMC for global and local posterior inference.Its inference network joins a deterministic bottom-up neural network with a probabilistic top-down generative model; Weibull variables approximate gamma conditional posteriors and permit uniform-noise reparameterization.Source: §1 · p. 2
  • Weibull distributions approximate gamma distributions, have an analytic KL divergence to gamma, and support uniform-noise reparameterization for efficient inference-network ELBO gradients.These properties motivate Weibull variables in WHAI's probabilistic top-down inference network.Source: §1 · p. 2
  • WHAI clearly outperforms GHAI, and both outperform versions without top-down inference links; WHAI is comparable to Gibbs sampling and supports scalable mini-batch training and fast out-of-sample prediction.The comparison and training and prediction properties are reported for deep topic modeling.Source: §1 · pp. 2–3

2 WHAI FOR MULTILAYER DOCUMENT REPRESENTATION

WHAI pairs a deep gamma-based topic-model decoder with a Weibull upward-downward inference network and hybrid MCMC/VAE training. Its design targets scalable training and fast inference for new documents while retaining posterior samples of model and inference-network parameters.

2.1 DOCUMENT DECODER: DEEP LATENT DIRICHLET ALLOCATION

  • DLDA models sparse document-count vectors with a multilayer Poisson generative model, and WHAI uses the Poisson gamma belief network as its generative network.With gamma latent variables marginalized, the PGBN can also be represented as deep LDA.Source: §2.1 · p. 3
  • DLDA Gibbs sampling processes all documents each iteration, while DLDA-TLASGR uses mini-batch stochastic-gradient MCMC for big corpora; test-time inference can still require many iterations.That iterative requirement motivates an inference network for fast out-of-sample prediction.Source: §2.1 · pp. 3–4

2.2 DOCUMENT ENCODER: WEIBULL UPWARD-DOWNWARD VARIATIONAL ENCODER

  • WHAI proposes WUDVE because Gaussian VAE inference does not generalize well to sparse, nonnegative, skewed document representations, while gamma inference gradients are difficult to compute efficiently.The proposed encoder uses Weibull distributions to approximate gamma distributions and support reparameterization.Source: §2.2 · p. 4

2.2.1 WEIBULL AND GAMMA DISTRIBUTIONS

  • Weibull distributions offer an analytic KL divergence to gamma distributions and a simple reparameterization; their approximation is generally accurate except for gamma shapes near zero or very large.The paper uses the analytic KL to choose Weibull parameters that approximate a given gamma distribution.Source: §2.2.1 · pp. 4–5

2.2.2 UPWARD-DOWNWARD INFORMATION PROPAGATION

  • WUDVE combines deterministic upward information propagation from the observation with stochastic downward layers that approximate gamma conditional posteriors using Weibull distributions.Unlike the DLDA Gibbs sampler, its upward path does not rely on local latent variables.Source: §2.2.2 · p. 5

2.3 HYBRID MCMC/VAE INFERENCE

  • WHAI combines TLASGR-MCMC for global topic parameters with WUDVE for local posteriors, jointly inferring posterior samples of the global parameters and inference-network parameters.Weibull reparameterization enables ELBO gradients for the inference network, and the network supplies local posteriors for new mini-batches.Source: §2.3 · pp. 5–6

2.4 VARIATIONS OF WHAI

  • The variations compare Weibull and gamma inference, with or without stochastic-downward connections, and compare TLASGR-MCMC with SGD for global parameters.For gamma inference, RSVI can yield relatively high-variance ELBO gradients and requires additional uniform random variables.Source: §2.4 · pp. 6–7

3 EXPERIMENTAL RESULTS

The experiments evaluate WHAI’s held-out prediction and testing speed against topic-modeling baselines, then inspect learned topic hierarchies and latent interpolations.

3.1 PER-HELDOUT-WORD PERPLEXITY

  • Evaluation uses 20News, RCV1, and Wiki, with 70% of each document’s word tokens for training and the remaining 30% held out for perplexity.Wiki contains 10 million documents, of which 100,000 are selected for testing; Table 1 compares per-heldout-word perplexity and average testing seconds per document.Source: §3 · p. 7, §3.1 · pp. 7–8, Table 1 · p. 8
  • DLDA-Gibbs has the best held-out-token prediction among the compared algorithms, while inference-network methods are faster at testing than methods using iterative inference.Among mini-batch methods, DLDA-WHAI slightly underperforms DLDA-TLASGR, but WHAI and its variations are considerably fast for processing testing documents.Source: Table 1 · p. 8, §3.1 · p. 8
  • On RCV1 and Wiki, WHAI converges slightly slower than TLASGR-MCMC but much faster than Gibbs sampling, and it consistently outperforms WAI during training.Across the three datasets, WHAI provides lower perplexity over time than DPFA and AVITM; RSVI-based GHAI converges more slowly than WHAI.Source: §3.1 · p. 9, Figure 3 · p. 9

3.2 TOPIC HIERARCHY AND MANIFOLD

  • On Wiki, WHAI’s learned topic hierarchy has interpretable topic meanings and connections; on MNIST, layers represent localized points, strokes, and digit components.The MNIST analysis uses a three-hidden-layer WHAI and a Poisson likelihood for nonnegative integer pixel values.Source: §3.2 · pp. 9–10, Figure 4 · p. 10, Figure 5 · p. 10
  • Removing stochastic-downward inference leaves some latent structures but causes relationships between adjacent layers to almost all disappear.The comparison is between WHAI’s deterministic-upward-stochastic-downward encoder and an encoder with the stochastic-downward part removed.Source: §3.2 · pp. 10–11, Figure 5 · p. 10
  • MNIST latent-space interpolations between test examples are smooth, and intermediary generated images remain interpretable, consistent with a learned latent manifold.The interpolated latent points are passed through the generative model to produce input-space images.Source: §3.2 · p. 11, Figure 6 · p. 11

4 CONCLUSION

WHAI combines stochastic-gradient MCMC for global parameters with a Weibull upward-downward variational autoencoder for local posterior inference. The authors report interpretable multilayer representations and scalability on large corpora, with fast out-of-sample prediction.

4 CONCLUSION

  • WHAI uses topic-layer-adaptive stochastic gradient Riemannian MCMC to update global parameters and a Weibull upward-downward variational autoencoder to infer local conditional posteriors.The variational autoencoder uses a Weibull distribution that resembles the gamma distribution and has a simple reparameterization, making one ELBO part analytic and gradient computation efficient for the non-analytic part.Source: §4 · p. 11
  • The method provides posterior samples for global generative-model parameters and inference-network parameters, and yields interpretable multilayer latent document representations.Source: §4 · p. 11
  • The authors report scalability to big training corpora through stochastic-gradient MCMC and fast out-of-sample prediction through an inference network.They report compelling experimental results on big text corpora in quantitative and qualitative analysis.Source: §4 · p. 11

ERRATUM

The arXiv update corrects two missing negative signs and adds the definition of γ as the Euler–Mascheroni constant.

ERRATUM

  • The update fixes missing negative signs in the Weibull density and Weibull-to-gamma KL definition, and defines γ as the Euler–Mascheroni constant.The paper identifies these as errors in its camera-ready copy.Source: ERRATUM · p. 12

A HIERARCHICAL TOPICS LEARNED FROM WIKI

The section presents example hierarchical topics learned from Wiki using WHAI models with three and four hidden layers.

A HIERARCHICAL TOPICS LEARNED FROM WIKI

  • The examples show Wiki hierarchical topics learned by three-hidden-layer 128-64-32 and four-hidden-layer 256-128-64-32 WHAI models.Source: Figure 7 · p. 14, Figure 8 · p. 14

B MANIFOLD ON DOCUMENTS

The section presents document-topic examples spanning document categories, including medicine-to-space, religion, and graphics-to-hardware examples.

B MANIFOLD ON DOCUMENTS

  • A document transition runs from sci.medicine to eci.space, with listed topic terms including medicine, health, space, launch, and satellite.Source: B MANIFOLD ON DOCUMENTS · p. 15
  • Religion-related examples list recurring terms such as god, Jesus, faith, Bible, church, and Christian.Source: B MANIFOLD ON DOCUMENTS · p. 15
  • A graphics-to-PC-hardware transition is accompanied by topic lists containing image, color, files, windows, card, and program.Source: B MANIFOLD ON DOCUMENTS · p. 15
Loading 1803.01328v2…