PaperlayerView exact revision on arXiv

Source-linked AI summary

Generating Sentences from Disentangled Syntactic and Semantic Spaces

Yu Bao, Hao Zhou, Shujian Huang, Lei Li, Lili Mou, Olga Vechtomova, Xinyu Dai, Jiajun Chen

arXiv:1907.05789v1cs.CL

TL;DR

VAEs provide continuous latent spaces for sentence generation, but do not explicitly model syntax there, and prior approaches do not preserve VAE-style syntax sampling and manipulation. DSS-VAE separates syntax and semantics in the latent space; it outperforms a VAE baseline in reconstruction and unconditional generation and reports promising results for paraphrase and syntax transfer.

  • Problem

    VAEs enable sentence sampling from a continuous latent space but blend syntax and semantics without explicitly modeling syntax, while prior syntactic approaches do not provide syntax sampling and manipulation in that space.

    Source: §1 · p. 1

  • Method

    DSS-VAE uses separate continuous latent variables for syntax and semantics, with adversarial approaches and an adversarial reconstruction loss to encourage their separation.

    Source: §1 · p. 1

  • Results

    DSS-VAE outperforms the VAE baseline in reconstruction and unconditional generation, and achieves promising results in unsupervised paraphrase and syntax-transfer generation.

    Source: §5 · p. 9

  • Takeaways & Limitations

    Disentangled latent spaces support sentence generation with syntax manipulation, including unsupervised paraphrase and syntax transfer.

    Source: §1 · p. 2, §5 · p. 9

    Language models trained on real sentences outperform those trained on sampled sentences, indicating substantial room for improvement among current sentence generators.

    Source: §4.1 · p. 7

Abstract

from arXiv · show

Variational auto-encoders (VAEs) are widely used in natural language generation due to the regularization of the latent space. However, generating sentences from the continuous latent space does not explicitly model the syntactic information. In this paper, we propose to generate sentences from disentangled syntactic and semantic spaces. Our proposed method explicitly models syntactic information in the VAE's latent space by using the linearized tree sequence, leading to better performance of language generation. Additionally, the advantage of sampling in the disentangled syntactic and semantic latent spaces enables us to perform novel applications, such as the unsupervised paraphrase generation and syntax-transfer generation. Experimental results show that our proposed model achieves similar or better performance in various tasks, compared with state-of-the-art related work.

1 Introduction

VAEs support sentence sampling from a continuous latent space, but do not explicitly model syntax there. DSS-VAE separates syntactic and semantic representations to retain latent-space manipulation while modeling syntax, and reports gains in generation and novel applications.

1 Introduction

  • DSS-VAE models syntax explicitly in the VAE’s continuous latent space while separating syntactic and semantic information into two variables.Adversarial approaches and an adversarial reconstruction loss are used to encourage separation despite the close relationship between syntax and semantics.Source: §1 · p. 1
  • DSS-VAE generates more fluent sentences than a traditional VAE, with lower perplexity and higher reconstruction BLEU scores.The introduction also reports comparisons with the state-of-the-art syntactic language model of Shen et al. (2017).Source: §1 · p. 2
  • DSS-VAE enables unsupervised paraphrase generation by sampling syntax while using MAP inference for semantics, producing the same meaning with different syntax without parallel training data.It outperforms traditional VAE and a state-of-the-art Metropolis-Hastings sampling approach (Miao et al., 2019) on this task.Source: §1 · p. 2
  • DSS-VAE also transfers one sentence’s syntax to another, with qualitative and quantitative results showing grafting under certain circumstances.The approach uses manipulation of its disentangled syntactic and semantic latent spaces.Source: §1 · p. 2

2 Related Work

Prior work on syntactic language modeling includes parsing-based methods and PRPN, while text style-transfer methods generally use categorical features. This work instead represents syntax in a continuous space and extends style-transfer approaches to more complex syntactic structure.

2 Related Work

  • PRPN learns latent syntax with a language-modeling objective and reports state-of-the-art syntactic language-modeling results; DSS-VAE instead models syntax in a continuous space that supports sampling and manipulation.Earlier syntactic language-modeling work also used top-down parsing mechanisms.Source: §2 · p. 2
  • Unlike prior style-transfer work that usually defines style with categorical features, this approach handles sequence-level, non-categorical syntax.It uses adversarial reconstruction losses to encourage separation of syntax and semantics.Source: §2 · p. 2

3 Approach

The approach extends a VAE with separate semantic and syntactic latent spaces, modeling syntax through linearized parse trees and using auxiliary losses to encourage the spaces to capture different information.

3 Approach

  • The approach section introduces the VAE foundation, DSS-VAE architecture, and training procedure for generating from disentangled syntactic and semantic spaces.The stated training setup uses Adam and applies KL annealing and word dropout to avoid KL collapse.Source: §3 · p. 2, §3.3 · p. 5

3.1 Variational Autoencoder

  • A VAE encodes a sentence in a probabilistic latent variable and decodes it, with training maximizing the evidence lower bound (ELBO).The sentence probability is computed by integrating the decoder likelihood over the latent prior.Source: §3.1 · p. 2

3.2 Proposed Method: DSS-VAE

  • DSS-VAE extends the VAE with separate semantic and syntactic latent variables, which are sampled and concatenated as the decoder’s initial state during training.At test time, both spaces can be sampled for generation, or inference and sampling can be applied in the respective spaces to preserve or vary semantics and syntax.Source: §3.2 · pp. 2–3

3.2.1 Modeling Syntax by Predicting Linearized Tree Sequence

  • Syntax is modeled by predicting a linearized tree sequence, obtained through top-down traversal with backtracking nodes added after nonterminal children are traversed.Training uses parse trees from ZPar; external syntactic trees are not needed at test time.Source: §3.2.1 · p. 3

3.2.2 Disentangling Syntax and Semantics into Different Latent Spaces

  • Multi-task losses predict BoW from zsem and linearized parse-tree tokens from zsyn, while adversaries predict semantics from zsyn and syntax from zsem.The adversarial losses are trained separately from the VAE; the VAE then learns to fool the fixed adversaries by maximizing their losses.Source: §3.2.2 · p. 4

3.2.3 Adversarial Reconstruction Loss

  • Because a semantic space containing BoW can support word reordering during reconstruction, the authors add adversarial reconstruction losses to discourage either single space from predicting the sentence alone.A decoding adversary predicts the sentence from one latent space; the losses are applied to both syntactic and semantic spaces.Source: §3.2.3 · pp. 4–5

3.3 Training Details

  • The training objective combines the VAE loss with multi-task, adversarial, and adversarial-reconstruction losses for syntax and semantics.Hyperparameters are selected on validation ELBO; tuning is mostly limited to sensitive parameters and is described as empirical.Source: §3.3 · p. 5

4 Experiments

Across reconstruction and unconditional generation, unsupervised paraphrase generation, and syntax-transfer generation, the experiments evaluate DSS-VAE against VAE and other baselines. The reported results favor DSS-VAE across these tasks, while noting a remaining gap from real-data language modeling and better syntax transfer between compatible sentences.

4.1 Reconstruction and Unconditional Language Generation

  • DSS-VAE outperforms VAE in BLEU at controlled Forward PPL, and in Forward PPL at controlled BLEU; it also achieves Reverse PPL comparable to or slightly better than PRPN-LM.DSS-VAE and PRPN consistently outperform VAE and LSTM-LM in sentence generation.Source: §4.1 · pp. 6–7
  • Real-data language modeling outperforms models trained on generated samples, indicating substantial room for improvement in sentence generation.Source: §4.1 · p. 7

4.2 Unsupervised Paraphrase Generation

  • The method samples a syntactic vector while fixing the semantic vector to generate a paraphrase with different syntax but the same meaning.Source: §4.2 · p. 7
  • For unsupervised paraphrase generation, DSS-VAE outperforms CGMH and VAE in BLEU-ref, and its BLEU-ori/BLEU-ref trade-off dominates both methods.The authors report that DSS-VAE’s BLEU-ref is closer to supervised paraphrase methods.Source: §4.2 · pp. 7–8

4.3 Syntax-Transfer Generation

  • On syntax transfer, the full DSS-VAE achieves the best total difference; adding adversarial reconstruction losses to both spaces balances the word-BLEU and TED differences.The total difference increases by 12.09 over the traditional VAE, which the authors report cannot accomplish the task.Source: §4.3 · p. 8
  • Syntax transfer produces more promising results between compatible than incompatible sentences; a large length mismatch is one example of incompatibility.Source: §4.3 · p. 9

5 Conclusion

The conclusion presents DSS-VAE as a model that explicitly represents syntax in VAE’s latent space while retaining syntax sampling and manipulation. It reports better reconstruction and unconditional generation than the VAE baseline, plus promising results for unsupervised paraphrase and syntax-transfer generation.

5 Conclusion

  • DSS-VAE explicitly models syntax in VAE’s latent space while retaining sampling and manipulation of sentence syntax.The conclusion also reports better reconstruction and unconditional language generation than the VAE baseline.Source: §5 · p. 9
  • DSS-VAE applies its sampling and manipulation advantages to unsupervised paraphrase and syntax-transfer generation, with promising results in both applications.Source: §5 · p. 9

A Hyperparameter Details

The appendix lists hyperparameters for the PTB and Quora datasets and describes a validation-based model-saving rule. It also identifies a table of syntax-transfer case studies.

A Hyperparameter Details

B Case Study of Syntax Transfer

The case studies contrast plain VAE interpolation with DSS-VAE syntax transfer. DSS-VAE can transfer a target syntax while changing the sentence meaning little, though matching syntax between the references can yield the semantic reference itself.

B Case Study of Syntax Transfer

Related papers

Loading 1907.05789v1…