PaperlayerView exact revision on arXiv

Source-linked AI summary

Diverse Text Generation via Variational Encoder-Decoder Models with Gaussian Process Priors

Wanyu Du, Jianqiao Zhao, Liwei Wang, Yangfeng Ji

arXiv:2204.01227v1cs.CL

TL;DR

Generating high-quality text with diversity remains a goal for text-generation applications, while encoder-side variation can be difficult to add without harming quality. This paper uses a GP-prior stochastic function to produce context-aware random variables. The authors report higher generation quality than deterministic encoder-decoder models and conditional variational auto-encoders, while supporting diverse outputs.

  • Problem

    Generating high-quality texts with high diversity is important for text-generation applications, but incorporating meaningful encoder-side variation without hurting quality is difficult.

    Source: §1 · p. 1

  • Method

    A GP-prior stochastic function maps deterministic encoder hidden states to dependent random context variables, with variational inference approximating the GP posterior.

    Source: §1 · p. 2, §2.2 · p. 4, §3 · p. 5

  • Results

    The method reports higher-quality texts than deterministic encoder-decoder models and conditional variational auto-encoders while supporting diverse generation from sampled context variables.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The approach is reported to adapt to pointer-generator and T5 architectures while enabling diverse generation through random context variables.

    Source: §4 · p. 6

    Training requires more computation to calculate the GP prior’s full context-variable covariance matrix, while inference has the same complexity as conditional variational auto-encoder baselines.

    Source: §4.5 · p. 8

Abstract

from arXiv · show

Generating high quality texts with high diversity is important for many NLG applications, but current methods mostly focus on building deterministic models to generate higher quality texts and do not provide many options for promoting diversity. In this work, we present a novel latent structured variable model to generate high quality texts by enriching contextual representation learning of encoder-decoder models. Specifically, we introduce a stochastic function to map deterministic encoder hidden states into random context variables. The proposed stochastic function is sampled from a Gaussian process prior to (1) provide infinite number of joint Gaussian distributions of random context variables (diversity-promoting) and (2) explicitly model dependency between context variables (accurate-encoding). To address the learning challenge of Gaussian processes, we propose an efficient variational inference approach to approximate the posterior distribution of random context variables. We evaluate our method in two typical text generation tasks: paraphrase generation and text style transfer. Experimental results on benchmark datasets demonstrate that our method improves the generation quality and diversity compared with other baselines.

1 Introduction

The introduction frames high-quality, diverse text generation as an important goal and presents a GP-based stochastic encoder function to enrich contextual representations while supporting diverse outputs. It also highlights an efficient variational inference method and reports improved generation quality alongside diverse generation.

1 Introduction

  • The introduction identifies a challenge in introducing encoder-side variation without hurting generated-text quality, while independent latent variables can lose contextual information.Decoder-side diversity strategies are also described as trading off quality against diversity.Source: §1 · pp. 1–2
  • A Gaussian process prior models dependencies among random context variables and provides infinitely many joint Gaussian distributions, unlike prior approaches with independent variables and fixed prior distributions.Figure 1 contrasts independent mappings from hidden states with GP-imposed dependencies among context variables.Source: Figure 1 · p. 1, §1 · p. 2
  • The method introduces context-aware variation into encoder-decoder models to learn richer contextual representations and promote diverse generation.A stochastic function maps deterministic encoder hidden states to random context variables, which the decoder can sample to generate different outputs.Source: §1 · p. 2
  • The authors propose efficient variational inference to approximate the joint distribution of fully connected random context variables.This is presented as one of the work’s major contributions for learning the proposed model.Source: §1 · p. 2
  • The proposed method reports higher-quality text than deterministic encoder-decoder models and conditional variational auto-encoders, while supporting diverse generation from sampled context variables.The introduction says the method was tested with LSTM-based and Transformer-based encoder-decoder models.Source: §1 · p. 2
  • The GP-prior approach is presented as a comparison with conditional variational auto-encoder priors under the variational encoder-decoder framework.Figure 2 identifies this comparison, while the introduction describes GP priors as offering infinitely many joint distributions rather than a fixed set.Source: §1 · p. 2, Figure 2 · p. 3

2 Model Description

The model maps deterministic encoder states to random context variables and uses them to condition text generation. A Gaussian-process prior supplies structured variation, and the model incorporates sampled variables into LSTM- and Transformer-based encoder-decoder systems.

2 Model Description

  • The model transforms deterministic encoder hidden states into random hidden states through stochastic functions to obtain richer context representations.Source: §2 · p. 3

2.1 Encoding with Stochastic Functions

  • Different random context-variable sets sampled from the encoder states condition the decoder, introducing context-aware variation into generation.The stochastic mapping is intended to preserve contextual information while adding variation.Source: §2.1 · p. 3

2.2 Gaussian Process Priors for Stochastic Functions

  • A Gaussian-process functional prior over the stochastic mapping supports infinitely many random functions and sets of context variables, unlike a fixed set of joint distributions.The prior uses mean and covariance functions; setting m(h)=h makes it semi-parametric and keeps the expected sampled state at h.Source: §2.2 · p. 4

2.3 Generation with Random Context Variables

  • The model applies random context variables in LSTM- and Transformer-based encoder-decoder systems, sampling them before generating the output sentence.For the LSTM system, attention is applied over the random context variables; the two model variants are evaluated in Section 4.Source: §2.3 · p. 4

3 Efficient Variational Inference

The method approximates the Gaussian-process posterior over stochastic functions conditioned on encoder hidden states. It uses variational inference for context variables and jointly learns other model parameters by maximum likelihood.

3 Efficient Variational Inference

  • Variational inference approximates the Gaussian-process posterior over functions conditioned on encoder hidden states, while other model parameters are learned jointly with maximum likelihood.The posterior is described as a Gaussian distribution with non-isotropic covariance.Source: §3 · p. 5
  • The variational posterior qφ(z | h, y) approximates the true posterior p(z | h, y) by maximizing the evidence lower bound of the marginal log-likelihood.Source: §3 · p. 5
  • During generation, the method approximates qφ(z | h, y) with qφ(z | h), then uses a mean-field amortized approximation with neural networks for its parameters.The networks estimate the mean and covariance.Source: §3 · p. 5

4 Experiments

Experiments compare the method with deterministic and variational encoder-decoder baselines on paraphrase generation and style transfer, evaluating text quality and diversity across LSTM and Transformer models.

4 Experiments

  • The experiments cover paraphrase generation and style transfer, comparing the proposed method with prior work on quality and diversity.The tested architectures include the pointer-generator network and T5.Source: §4 · pp. 5–6

4.1 Evaluation Methods

  • Quality evaluation uses METEOR and BLEU, while diversity evaluation uses self-BLEU, Div-4, and uniqueness.Lower self-BLEU indicates less overlap; higher Div-4 and uniqueness indicate more diverse outputs.Source: §4.1 · p. 6

4.2 Competitive Baselines

  • Baselines include deterministic PG and T5 models, prior variational models, and PG or T5 variants with Normal and GP priors.The compared variational systems include Variation Attention, Multi-Selectors, and T-CVAE.Source: §4.2 · p. 6

4.3 Generation Setups

  • For quality generation, the experiments use the mean of the variational posterior; for diversity, they sample 10 context-variable sets and generate 10 outputs.Beam search uses size 10, and covariance scaling is applied because original posterior samples tended to yield repeated sentences; the scaling factor varies by model and task.Source: §4.3 · pp. 6–7

4.4 Paraphrase Generation

  • For paraphrase quality, PG + GP prior outperforms the compared LSTM baselines, but T-CVAE produces better-quality outputs than the Transformer models.T5 + GP prior still outperforms T5 + Normal prior; the authors caution that T-CVAE's lowercase preprocessing may advantage its quality scores.Source: §4.4 · pp. 7–8, Table 2 · p. 7

4.5 Text Style Transfer

  • On style transfer, PG + GP prior is most accurate among the LSTM models, while T5 + GP prior performs best among the Transformer models and leads on diversity.The diversity comparison covers both token- and sentence-level measures; the authors also report that increasing diversity can degrade quality.Source: Table 2 · p. 7, §4.5 · p. 8, Table 3 · p. 8

5 Related Works

Related work promotes diverse generation through decoder-side decoding strategies and encoder-side latent representations or mixture-based approaches.

Diverse text generation

  • Decoder-side methods use diverse beam search, top-k sampling, or nucleus sampling, while mixture-of-decoder approaches generate outputs from different decoders.The paper describes its encoder-side approach as orthogonal to diverse decoding algorithms.Source: §5 · p. 9

Conditional variational autoencoders

  • Encoder-side approaches include latent attention alignments, conditional variational autoencoders for latent context, and mixture-of-selector models.The cited examples use latent variables or masks to alter which source information is represented or selected.Source: §5 · p. 9
  • Other related approaches learn pattern embeddings for diverse paraphrases or use group information to regularize latent variables in dialogue generation.The latter is reported to alleviate posterior collapse and improve performance on dialogue generation.Source: §5 · p. 9

6 Conclusion

The conclusion presents a GP-prior variational encoder-decoder approach that introduces context-aware variation into encoder states and reports higher generation quality than deterministic and conditional variational autoencoder baselines.

6 Conclusion

  • A stochastic function maps encoder hidden states to context-aware variations, giving the decoder more diverse contextual representations.The approach uses a Gaussian process prior to model dependencies between random context variables.Source: §6 · p. 9
  • An amortized variational inference method approximates the GP posterior for the stochastic function.Source: §6 · p. 9
  • Experiments report higher generation quality than deterministic encoder-decoder models and conditional variational autoencoders.Source: §6 · p. 9

A Gaussian Processes as Function Priors

The section models mappings from encoder hidden states to latent context variables as functions drawn from a Gaussian process. Conditioning on observed states constrains the function posterior, while different functions yield diverse context representations.

A Gaussian Processes as Function Priors

  • The Gaussian process places a distribution over nonlinear mapping functions from encoder hidden states to latent context variables.The prior allows uncertainty to account for noisy observations.Source: A Gaussian Processes as Function Priors · p. 12
  • Bayesian inference predicts a test context variable conditioned on observed training data points.The section denotes the observed hidden states and latent variables as h1:N and z1:N.Source: A Gaussian Processes as Function Priors · p. 12
  • Conditioning on training data constrains possible functions because their covariance becomes smaller.The paper illustrates this change in Figure 3(b).Source: A Gaussian Processes as Function Priors · p. 12
  • Sampling different functions from the prior yields different sets of latent context representations and thus diverse encoder-decoder contexts.The paper says the GP prior theoretically permits infinitely many mapping functions.Source: A Gaussian Processes as Function Priors · p. 12

B Derivations of ELBo

The derivation uses a conditional variational autoencoder setup in which latent context variables condition text generation. It relates minimizing the variational posterior's KL divergence to maximizing the evidence lower bound.

B Derivations of ELBo

  • The model assumes z is drawn from p(z | h) and generates y from p(y | h, z), learning the variational posterior by minimizing KL divergence.This objective is equivalent to maximizing the marginal log-likelihood evidence lower bound.Source: B Derivations of ELBo · p. 12
  • The non-negativity of KL divergence supplies the inequality used in the ELBo derivation.The supplied derivation passage states KL(qφ(z | h, y)∥p(z | h, y)) ≥ 0.Source: B Derivations of ELBo · p. 13
  • The variational inference networks are parameterized by φ.Source: B Derivations of ELBo · p. 13

C Experiment Setup Details

The experiments configure separate PG and T5 encoder-decoder models, with task-specific latent dimensions. Training uses Adam, early stopping, and validation-set grid search for kernel hyperparameters.

C Experiment Setup Details

  • PG uses a copying LSTM encoder-decoder with 300-dimensional embeddings, 512 hidden dimensions, and a 256-dimensional latent variable.Its encoder is a one-layer bidirectional LSTM and its decoder a one-layer unidirectional LSTM; posterior mean and covariance networks are single feed-forward networks.Source: C Experiment Setup Details · p. 13
  • T5 uses the pretrained T5-base configuration and a 512-dimensional latent variable with single feed-forward posterior mean and covariance networks.The model is fine-tuned on the target task datasets.Source: C Experiment Setup Details · p. 13
  • Training uses Adam at learning rate 0.0001, early stopping after 10 validation-loss epochs without improvement, and validation grid search for kernel hyperparameters.KL annealing is not applied and the KL coefficient remains 1.Source: C Experiment Setup Details · p. 13

Related papers

Loading 2204.01227v1…