PaperlayerView exact revision on arXiv

Source-linked AI summary

Topic-Guided Variational Autoencoders for Text Generation

Wenlin Wang, Zhe Gan, Hongteng Xu, Ruiyi Zhang, Guoyin Wang, Dinghan Shen, Changyou Chen, Lawrence Carin

arXiv:1903.07137v1cs.CL

TL;DR

Text VAEs commonly use simple Gaussian priors, while semantic information in documents spans different groups and strong decoders may ignore latent codes. TGVAE combines topic-guided mixture priors with Householder-flow inference, and the paper reports effectiveness across unconditional generation and conditional summarization. The authors conclude that the approach supports topic-guided generation across multiple NLP tasks.

  • Problem

    Simple Gaussian priors do not encode semantic structure among sentences, and strong autoregressive decoders may ignore latent codes during generation.

    Source: §1 · p. 1

  • Method

    TGVAE uses topic-guided latent generation and Householder flow to construct a flexible approximate posterior from a sequence of invertible transformations.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    The authors report that TGVAE is effective on both unconditional text generation and conditional summarization, outperforming alternative approaches across these settings.

    Source: §1 · p. 2, §5.1 · p. 6, §5.2 · p. 9

  • Takeaways & Limitations

    The authors conclude that learned topics and Householder-flow inference yield encouraging results across multiple NLP tasks, with topics extracted from data rather than predefined labels.

    Source: §6 · p. 9

Abstract

from arXiv · show

We propose a topic-guided variational autoencoder (TGVAE) model for text generation. Distinct from existing variational autoencoder (VAE) based approaches, which assume a simple Gaussian prior for the latent code, our model specifies the prior as a Gaussian mixture model (GMM) parametrized by a neural topic module. Each mixture component corresponds to a latent topic, which provides guidance to generate sentences under the topic. The neural topic module and the VAE-based neural sequence module in our model are learned jointly. In particular, a sequence of invertible Householder transformations is applied to endow the approximate posterior of the latent code with high flexibility during model inference. Experimental results show that our TGVAE outperforms alternative approaches on both unconditional and conditional text generation, which can generate semantically-meaningful sentences with various topics.

1 Introduction

Existing VAEs generate text from latent codes, but simple Gaussian priors do not represent semantic structure, and powerful decoders may ignore the codes. TGVAE uses topic-guided latent priors, flexible posterior inference, and an attention-based extension for summarization.

1 Introduction

  • A simple Gaussian prior may fail to indicate semantic structure among sentences, while strong autoregressive decoders can ignore latent codes and rely on previous tokens.Existing structured-prior variants may use predefined settings that omit semantic meaning and risk over-regularization.Source: §1 · p. 1
  • TGVAE uses a neural topic model to parameterize a Gaussian-mixture prior whose components correspond to topics, guiding text generation under designated topics.The model is proposed for both unconditional and conditional generation.Source: §1 · p. 2
  • Householder transformations are applied during inference to turn an initial mixture distribution into a more flexible approximate posterior.The contribution statement describes this inference as powerful approximate posterior inference.Source: §1 · p. 2
  • TGVAE extends to abstractive summarization by adding attention and injecting learned topic information into conditional generation.The paper presents this as an extension beyond unconditional text generation.Source: §1 · p. 2
  • Experiments are reported for both unconditional and conditional text generation as evidence of the proposed approach’s effectiveness.Source: §1 · p. 2

2 Model

The model combines a neural topic model, which captures document-level semantic meaning, with a sequence model that generates sentences under topic guidance. Topic distributions also parameterize a semantically meaningful Gaussian-mixture prior for the sequence model’s latent code.

2 Model

  • The model combines a neural topic model for long-range document semantics with a neural sequence model for generating sentences under designated topic guidance.Source: §2 · p. 2

2.1 Neural Topic Model

  • The neural topic model maps a Gaussian random vector through a softmax transformation to a topic embedding and uses topic distributions over words.The topic model represents documents using bag-of-words counts over a vocabulary.Source: §2.1 · pp. 2–3

2.2 Neural Sequence Model

  • The sequence model samples a continuous latent code from a prior and generates a text sequence from a decoder distribution conditioned on that code.The latent variable is intended to capture high-level variation across sentences.Source: §2.2 · p. 3
  • The topic-guided GMM gives each mixture component semantic meaning through topic information, making generation more interpretable and controllable than a standard GMM with identical Gaussian components.Topic usage and word distributions inform the component parameters.Source: §2.2 · p. 3

3 Inference

Inference jointly optimizes the neural topic and sequence models, while Householder flow enriches posterior inference. The framework also extends topic guidance to summarization and regularizes topic diversity.

3 Inference

  • The ELBO combines a neural topic-model term for reconstructing document words with a neural sequence-model term for generating text under the topic-dependent prior.The sequence term includes expected text log-likelihood and a KL penalty between the approximate posterior and the topic-dependent prior.Source: §3 · p. 4

3.1 Householder Flow for Approximate Posterior

  • Householder flow composes invertible mappings to transform an initial latent sample into a flexible approximate posterior.Each transformation is a reflection; its unit-magnitude Jacobian determinant simplifies the lower-bound computation.Source: §3.1 · pp. 4–5
  • The initial posterior and topic-dependent prior are both GMMs, so inference uses an explicit upper bound for their KL divergence.The bound is computable using closed-form KL divergences between Gaussian components.Source: §3.1 · p. 4

3.2 Extension to text summarization

  • For summarization, the model represents document-summary pairs and conditions both the sequence generator and latent-code posterior on the source document.The latent code provides high-level topic guidance alongside the attention-based sequence-to-sequence model.Source: §3.2 · p. 5

3.3 Diversity Regularizer for NTM

  • Topic-diversity regularization addresses redundancy by regularizing row-wise distances between paired topics during inference.The regularizer is defined using the mean and variance of pairwise topic cosine distances.Source: §3.3 · p. 5

4 Related Work

Prior work applies VAEs and learned topics to text generation and related NLP tasks. This paper distinguishes its approach by using topics to guide the VAE prior, and extends it to summarization.

4 Related Work

  • Text-generation VAEs span bag-of-words and RNN encoder-decoder representations, with later work exploring convolutional designs and control of sentiment or tense.VAE applications also include conditional tasks such as translation, image captioning, dialogue, and summarization.Source: §4 · p. 5
  • Earlier topic-informed NLP approaches combine topics with neural language models or word embeddings, whereas this work uses topics to guide the VAE prior rather than only its decoder.The authors state that this provides more flexibility in text modeling and can support latent-code posterior inference for visualization and downstream tasks.Source: §4 · p. 5
  • The popularity of attention-based encoder-decoder summarization, whose attention is described as a keyword detector, motivates extending the topic-guided VAE to summarization.The cited framework includes RNN encoders and decoders with attention.Source: §4 · pp. 5–6

5 Experiments

Experiments test text generation across three corpora and summarization on GIGAWORDS and DUC-2004; TGVAE improves reported benchmark performance and yields topic-structured qualitative outputs.

5.1 Text Generation

  • TGVAE achieves a lower perplexity upper bound with relatively larger UKL, while VAE and VAE+Householder models do not outperform a well-optimized language model.The authors attribute the gains to topical clustering in the latent space and improved posterior inference from Householder flow; improvements are consistent as topic count increases.Source: §5.1 · p. 6, Table 3 · p. 7
  • As topic count increases, TGVAE improves self-BLEU and boosts test-BLEU relative to standard VAEs, with generated sentences described as diverse under topic guidance.Test-BLEU measures generated-sentence quality against test-set references (higher is better), while self-BLEU mainly measures sample diversity (lower is better).Source: §5.1 · p. 6, Table 2 · p. 7
  • Topic coherence is measured with NPMI, averaging scores over the top 5, 10, 15, and 20 topic words and then across topics.The authors use this evaluation to assess the rationality of their joint learning framework, while noting that the paper focuses on coherent topic-specific sentences rather than the learned topics themselves.Source: §5.1 · pp. 6–7, Table 4 · p. 7

5.2 Text Summarization

  • On GIGAWORDS and DUC-2004, TGVAE outperforms strong summarization baselines; Householder flow and the topic-dependent prior each contribute further improvements.The comparison uses a relatively simple RNN architecture, and the authors attribute the gains mainly to topic-guided text generation.Source: Table 6 · p. 8, §5.2 · p. 9
  • Generated summaries can contain semantically meaningful words absent from both the reference and source, and can begin with meaningful words.Table 7 presents example generated summaries; its blue words are source-only topic words and its red words are absent from both source and reference.Source: Table 7 · p. 8, §5.2 · p. 9
  • The t-SNE visualization shows separable topic-dependent components, with semantically related stock and finance clusters near each other and finance and disease farther apart.The paper also presents top words for ten randomly chosen topics per dataset.Source: Figure 2 · p. 8, §5.2 · p. 9, Table 8 · p. 9

6 Conclusion

The paper combines a VAE-based neural sequence model with a neural topic model for text generation. It frames topic extraction as unsupervised clustering and uses Householder flow to encourage complexity and diversity in learned topics.

6 Conclusion

  • The proposed text generator combines a VAE-based neural sequence model with a neural topic model.Source: §6 · p. 9
  • Topics are extracted from data with clustering structure rather than predefined labels, extending conditional VAEs within unsupervised learning.Source: §6 · p. 9
  • Householder flow is designed to encourage complexity and diversity in the learned topics, and experiments are encouraging across multiple NLP tasks.Source: §6 · p. 9

Related papers

Loading 1903.07137v1…