PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Variational Inference for Text Processing

Yishu Miao, Lei Yu, Phil Blunsom

arXiv:1511.06038v4cs.CLcs.LGstat.ML

TL;DR

Deep text models make exact Bayesian inference difficult because latent-variable distributions become intractable. The paper replaces analytic approximations with text-conditioned neural inference networks and achieves state-of-the-art performance on document modelling and question-answer selection.

  • Problem

    As text generative models become deeper and more complex, exact Bayesian inference becomes intractable because of high-dimensional integrals over latent variables.

  • Method

    The paper uses text-conditioned neural inference networks to approximate latent-variable distributions in generative and conditional text models.

  • Results

    The models achieve state-of-the-art performance on both document modelling and question-answer selection tasks.

  • Takeaways & Limitations

    Neural variational inference provides a simple, expressive, efficient framework applicable to both unsupervised and supervised text learning.

  • Takeaways & Limitations

    The presented treatment focuses on continuous latent variables with diagonal Gaussian variational distributions, while discrete variables require modified estimation methods.

Abstract

from arXiv · show

Recent advances in neural variational inference have spawned a renaissance in deep latent variable models. In this paper we introduce a generic variational inference framework for generative and conditional models of text. While traditional variational methods derive an analytic approximation for the intractable distributions over latent variables, here we construct an inference network conditioned on the discrete text input to provide the variational distribution. We validate this framework on two very different text modelling applications, generative document modelling and supervised question answering. Our neural variational document model combines a continuous stochastic document representation with a bag-of-words generative model and achieves the lowest reported perplexities on two standard test corpora. The neural answer selection model employs a stochastic representation layer within an attention mechanism to extract the semantics between a question and answer pair. On two question answering benchmarks this model exceeds all previous published benchmarks.

1. Introduction

The introduction motivates neural variational inference as an efficient alternative for complex text models and presents a framework using text-conditioned inference networks. It applies this framework to unsupervised document modeling and supervised question-answer modeling with stochastic latent representations.

  • Motivation: Deeper generative models make exact Bayesian inference intractable, while MCMC is impractical for large neural networks and conventional variational inference underestimates posterior variance.These limitations hinder the development of highly expressive text models.
  • Neural variational framework: The framework uses a deep neural inference network conditioned on text to approximate intractable latent-variable distributions instead of relying on analytic variational approximations.It is inspired by the variational autoencoder.
  • NVDM: NVDM is an unsupervised generative model that extracts a continuous semantic latent variable for each document and can be interpreted as a variational auto-encoder.Its architecture includes an MLP encoder.
  • NASM: NASM is a supervised conditional model that equips LSTMs with latent stochastic attention to model question-answer semantics and predict pair relatedness.The attention focuses on answer phrases connected to question semantics and uses a latent distribution to address task ambiguity.
  • Summary: The paper demonstrates neural variational inference across diverse unsupervised and supervised text tasks using simple, expressive models trained efficiently with scalable stochastic-gradient back-propagation.The framework can generalize to incorporate any type of neural network.

2. Neural Variational Inference Framework

The framework approximates intractable latent-variable posteriors with an inference network conditioned on observed text variables, using a diagonal Gaussian and stochastic back-propagation for optimization. It also covers unsupervised learning and can be adapted to discrete latent units by replacing the Gaussian with a softmax-parameterized multinomial.

  • Variational distribution: The variational distribution is qφ(h|x, y) = N(h|µ(x, y), diag(σ²(x, y))), designed to approach the posterior p(h|x, y).The generative distributions are parameterized by θ, while the inference distribution is parameterized by φ.
  • Inference network: The inference network encodes observed variables, combines their representations with an MLP, and maps the result to Gaussian parameters µ and log σ.The construction uses u = fx(x), v = fy(y), π = g(u, v), µ = l1(π), and log σ = l2(π).
  • Optimization: Sampling h ∼ qφ(h|x, y) enables stochastic back-propagation to optimize the lower bound, updating both generative parameters θ and inference parameters φ.For φ, reparameterization uses h = µ + σ · ϵ with ϵ(l) ∼ N(0, I) to reduce variance in stochastic estimation.
  • Special cases: Unsupervised learning is a special case in which h has no parent x and is drawn directly from the prior p(h) rather than pθ(h|x).In this case, the objective uses s(h) = log pθ(y|h)pθ(h) − log qφ(h|y).
  • Discrete variables: The framework extends to discrete latent units by replacing the Gaussian with a softmax-parameterized multinomial and using policy gradients instead of continuous reparameterization.The discussed formulation otherwise assumes continuous latent variables and a parameterized diagonal Gaussian.

3. Neural Variational Document Model

The Neural Variational Document Model introduces a continuous hidden variable to represent document semantics and generates words independently from it. An inference network approximates the latent posterior from each document, enabling stochastic-gradient optimization with an analytically computable Gaussian KL term.

  • Latent document representation: NVDM represents each document’s semantic content with a continuous hidden variable h ∈ R^K that generates all words independently.Documents use a bag-of-words representation X, while each word position uses a one-hot representation x_i.
  • Encoder-decoder model: An MLP encoder maps document representations X to continuous hidden vectors h, while a softmax decoder reconstructs documents by independently generating words.The decoder factorizes p(X|h) across the document’s N word positions.
  • Word-generation model: The decoder models pθ(x_i|h) with multinomial logistic regression shared across documents, learning semantic word embeddings R and word biases b_xi.The model treats N as observed and uses a Gaussian prior p(h).
  • Inference network: The inference network qφ(h|X) conditions only on the current document and produces document-specific µ and σ parameters for the latent semantic distribution.Samples h ∼ qφ(h|X) support lower-bound optimization by back-propagating stochastic gradients with respect to θ and φ.
  • Optimization: Because p(h) is standard Gaussian, the Gaussian KL divergence is computed analytically, reducing gradient variance and regularizing inference-network updates.The KL term is D_KL[qφ(h|X)∥p(h)].

4. Neural Answer Selection Model

NASM is a supervised answer-selection model that learns question–answer representations and predicts their relatedness. It combines separate LSTM encoders with stochastic question semantics and attention over answer tokens.

  • Task formulation: Answer sentence selection classifies candidate answers as correct or incorrect for a question.Each training example is a triple (q, a, y), where y indicates whether answer a is correct.
  • Neural Answer Selection Model: NASM uses two separate LSTMs to embed raw question and answer inputs for relatedness prediction.The encoders produce state outputs for question positions j and answer positions i.
  • Stochastic question representation: NASM represents question semantics with a diagonal-Gaussian latent distribution pθ(h|q) instead of a deterministic question vector.The distribution is parameterised by µ(q) and σ(q), and its samples guide answer attention.
  • Attention mechanism: The attention mechanism weights answer-token states to form a context vector, which is combined with the final answer state for relatedness prediction.The question representation zq(q) equals the final question-LSTM state, while za(a, h) combines the final answer state with c(a, h).
  • Variational training: An inference network models qφ(h|q, a, y) as a Gaussian approximation, and SGVB jointly updates model and inference-network parameters.The Gaussian KL divergence between qφ(h|q, a, y) and pθ(h|q) is computed analytically during training.

5. Experiments

The experiments evaluate NVDM for document modeling and NASM for answer selection across standard datasets, using neural baselines and established experimental setups. NVDM achieves the best document-modeling performance, while NASM further improves answer-selection results and its latent attention captures useful answer information.

  • Document modeling: NVDM is evaluated on 20NewsGroups2 and Reuters RCV1-v2, with vocabulary sizes of 2,000 and 10,000, respectively.The datasets contain 11,314 training and 7,531 test articles for 20NewsGroups2, and 794,414 training and 10,000 test cases for RCV1-v2.
  • Document modeling: The document-modeling experiments use 50- and 200-dimensional representations and a two-layer 500-dimensional rectifier-linear-unit inference network.The setup follows prior work, including the preprocessing and experimental procedures of Hinton & Salakhutdinov, Larochelle & Lauly, Srivastava et al., and Mnih & Gregor.
  • Document modeling: NVDM achieves the best performance on both document datasets, and its 50-dimensional RCV1-v2 model outperforms fDARN with 200 dimensions.Unlike the baselines’ discrete latent variables, NVDM uses a continuous stochastic document representation.
  • Document modeling: NVDM’s 50-dimensional latent space yields interpretable topics corresponding to Space, Religion, Encryption, Sport, and Policy.The topics are inferred from the ten words with the strongest positive connections for five randomly selected latent dimensions.
  • Answer selection: Latent and deterministic attention both identify crucial information in correct answer sentences, whereas neither reliably attends to useful words in most negative answers.NASM prediction uses 20 samples to calculate the expectation of the lower bound.

6. Discussion

Neural variational inference consistently improves both NLP tasks by summing over semantic possibilities and helping guard against overfitting. The discussion attributes these benefits to task-specific KL regularization, stochastic attention behavior, and efficient GPU-parallelizable training.

  • Benefits: Neural variational inference consistently improves performance on both NLP tasks by summing over semantic possibilities and guarding against overfitting.The discussion connects these gains to latent distributions and Bayesian learning.
  • Regularization: In NVDM, the analytically computed KL divergence acts as an L2 regularizer on µ and encourages qφ(h|X) to approach the Gaussian prior p(h).The prior p(h) is a standard Gaussian, and the KL term penalizes large µ.
  • Regularization: In NASM, the KL divergence minimizes distance between learned conditional distributions, with pθ(h|q) serving as a moving baseline that regularizes qφ(h|q, a, y).The baseline adapts for each question condition.
  • Stochastic attention: NASM learns more peaked attention scores than deterministic LSTM+Att, whose question-semantic gradients show relatively high variance on the small, noisy training set.LSTM+Att uses deterministic sq(|q|), whereas NASM uses stochastic h.
  • Efficiency: NVDM and NASM parallelize on GPUs, require only one training sample, and update all parameters by backpropagation, limiting stochastic-unit overhead to inference-network parameters.The stochastic units add computation through the added inference-network parameters.

7. Related Work

Related work develops neural variational inference through learned inference networks and variance-reduction techniques, with strong results in image processing. Related text-generation and question-answering models include autoregressive document models, probabilistic topic models, RBMs, knowledge-base mapping, and memory networks.

  • Neural variational inference: Learned inference networks were introduced for Helmholtz machines, while later methods reduced gradient-estimator variance through reparameterisation, control variates, or importance sampling.These approaches target continuous random variables or posterior approximation.
  • Neural variational inference: Neural variational inference instantiations and generative auto-encoders demonstrated strong or competitive performance on image-processing tasks.The cited instantiations include Gregor et al. (2015), Kingma et al. (2014), and Ba et al. (2015), while generative auto-encoders are described as very competitive.
  • Document modelling: Autoregressive neural generative models improved document generation relative to conventional probabilistic topic models and RBMs.The passage contrasts these models with topic-model references including Hofmann (1999) and Blei et al. (2003), and RBM references including Hinton & Salakhutdinov (2009) and Srivastava et al. (2013).
  • Question answering: Deep-neural question-answering work includes mapping factoid questions to knowledge-base answer triples and using memory networks as dynamic knowledge bases.The passage identifies these as major streams of question-answering research.

8. Conclusion

The paper introduced a deep neural variational inference framework for generative text models and demonstrated its effectiveness on document modelling and question answer selection. In both tasks, the models achieved state-of-the-art performance, while training with SGVB was described as simple, expressive, and efficient.

  • The paper introduced a deep neural variational inference framework for generative models of text.
  • The framework was evaluated on two diverse tasks: document modelling and question answer selection.
  • In both tasks, the models achieved state-of-the-art performance.
  • The model was described as simple, expressive, and efficient when trained with the SGVB algorithm.

A. t-SNE Visualisation of Document Representations

Figure 6 visualizes document representations from NVDM and SWV using t-SNE on the held-out 20NewsGroups test dataset. The visualization covers documents from 20 news groups, represented by differently colored points.

  • Model comparison: Figure 6 compares t-SNE visualizations of document representations produced by NVDM and SWV.The two visualizations correspond to panels (a) NVDM and (b) SWV.
  • Evaluation dataset: The visualizations use the held-out test dataset of 20NewsGroups.
  • Visualization encoding: The dataset contains documents from 20 different news groups, shown as points with different colours.

B. Details of the Deep Neural Network Structures · B.1. Neural Variational Document Model · B.2. Neural Answer Selection Model

The paper specifies separate neural variational document and answer-selection models, each organized around an inference network, a KL-divergence term, and an optimised variational lower bound.

  • B.1. Neural Variational Document Model: The document model defines an inference network qφ(h|X) for the text input X.
  • B.1. Neural Variational Document Model: Its generative component is represented as pθ(X|h).
  • B.1. Neural Variational Document Model: The document model includes the divergence DKL[qφ(h|X)||p(h)].
  • B.1. Neural Variational Document Model: Training optimises the document model’s variational lower bound.
  • B.2. Neural Answer Selection Model: The answer-selection model defines an inference network qφ(h|q, a, y) conditioned on question, answer, and label variables.
  • B.2. Neural Answer Selection Model: Its answer-selection formulation includes the expression q Mza + b.
  • B.2. Neural Answer Selection Model: The answer-selection model uses the divergence DKL[qφ(h|q, a, y)||pθ(h|q)].
  • B.2. Neural Answer Selection Model: Training optimises the answer-selection model’s variational lower bound.

C. Computational Complexity

The section characterizes NVDM and NASM computational costs through inference and reconstruction components. Their computations can be GPU-parallelized, requiring one training sample and matching deterministic counterparts up to an O(LK^2) increase for NASM.

  • NVDM: NVDM training complexity is Cφ + Cθ = O(LK^2 + KSV), combining inference-network sampling with document reconstruction.Cφ = O(LK^2), while Cθ = O(KSV), where L is inference-network depth, K layer dimension, S document length, and V vocabulary volume.
  • NASM: NASM training complexity is Cφ + Cθ = O((L+S)K^2 + SW), including inference, attention, and relatedness-prediction costs.The inference network costs O(LK^2 + SW), with 2SW + 2K producing the joint representation and LK^2 generating a sample.
  • Efficiency: GPU parallelization and one required training sample make neural variational inference efficient for both NVDM and NASM.The section states that both models’ computations can be parallelized on GPUs and that only one sample is required during training.
  • Comparison: NVDM matches deterministic auto-encoder complexity, while LSTM+Att matches NASM at O((L+S)K^2+SW) and NASM adds only O(LK^2) time.NVDM is an instantiation of a variational auto-encoder; LSTM+Att is NASM’s deterministic counterpart.
Loading 1511.06038v4…