PaperlayerView exact revision on arXiv

Source-linked AI summary

Adversarial Text Generation via Feature-Mover's Distance

Liqun Chen, Shuyang Dai, Chenyang Tao, Dinghan Shen, Zhe Gan, Haichao Zhang, Yizhe Zhang, Lawrence Carin

arXiv:1809.06297v2cs.CL

TL;DR

Unsupervised text generation is difficult because text is discrete and existing GAN training can be unstable. FM-GAN uses optimal transport to match feature distributions of real and synthetic sentences, and the paper reports improved outcomes across several text-generation tasks.

  • Problem

    Unsupervised text generation remains challenging, while text GANs face discrete-data difficulties and existing approaches can have unstable training or poor generation quality.

    Source: §1 · pp. 1–2

  • Method

    FM-GAN replaces the usual critic objective with a tractable feature-mover’s distance between real and synthetic sentence-feature distributions.

    Source: §3 · p. 3

  • Results

    FM-GAN reports improved unconditional-generation quality and diversity, higher sentiment-transfer accuracy than comparison methods, and higher deciphering accuracy than CipherGAN.

    Source: §1 · p. 2, §5.2 · p. 8, §5.3 · p. 9

  • Takeaways & Limitations

    The reported evaluations demonstrate FM-GAN on unconditional generation, non-parallel style transfer, and unsupervised cipher cracking.

    Source: §1 · p. 2, §6 · p. 9

    The standard GAN objective has an unstably weak learning signal near a locally optimal discriminator because its JSD is discontinuous with respect to generator parameters.

    Source: §2.1 · p. 3

Abstract

from arXiv · show

Generative adversarial networks (GANs) have achieved significant success in generating real-valued data. However, the discrete nature of text hinders the application of GAN to text-generation tasks. Instead of using the standard GAN objective, we propose to improve text-generation GAN via a novel approach inspired by optimal transport. Specifically, we consider matching the latent feature distributions of real and synthetic sentences using a novel metric, termed the feature-mover's distance (FMD). This formulation leads to a highly discriminative critic and easy-to-optimize objective, overcoming the mode-collapsing and brittle-training problems in existing methods. Extensive experiments are conducted on a variety of tasks to evaluate the proposed model empirically, including unconditional text generation, style transfer from non-parallel text, and unsupervised cipher cracking. The proposed model yields superior performance, demonstrating wide applicability and effectiveness.

1 Introduction

Unsupervised text generation remains challenging, and text GANs face both discrete-data optimization difficulties and training instability. FM-GAN replaces the standard objective with feature-mover’s distance and is evaluated on unconditional generation and conditional text tasks.

1 Introduction

  • Unsupervised text generation estimates a corpus’s real-text distribution, but likelihood-based word prediction can suffer exposure bias between training on ground-truth prefixes and inference on generated prefixes.During inference, each word depends on previously generated words, whereas training uses ground-truth words at each timestep.Source: §1 · p. 1
  • RL-based text GANs estimate generator gradients with policy gradients, which typically have high variance, while RL-free approaches use differentiable soft-argmax or Gumbel-softmax approximations.The RL-free approaches provide low-variance gradients that can improve training stability and speed.Source: §1 · p. 2
  • TextGAN’s feature-matching remedy adds MMD, yet training remains difficult because RBF bandwidth is hard to choose, kernel methods scale poorly, and generated sentences tend to be short.Source: §1 · p. 2
  • FM-GAN uses optimal transport to match real and synthetic sentence-feature distributions with FMD, which is made differentiable and computed using the proximal point method.Its critic maximizes feature-distribution dissimilarity while the generator minimizes FMD.Source: §1 · p. 2
  • FM-GAN reports substantially improved unconditional-generation quality by BLEU statistics and human evaluation, with good diversity by self-BLEU, and extends to style transfer and cipher cracking.The paper describes evaluations on non-parallel text style transfer and unsupervised cipher cracking as demonstrations of the method’s versatility.Source: §1 · p. 2

2 Background

The background frames GAN training as adversarial distribution matching, then describes sentence-level feature extraction and optimal-transport distances. These ideas set up the feature-based transport objective used by the paper.

2.1 Adversarial training for distribution matching

  • Adversarial distribution matching learns to draw synthetic samples from a latent prior and uses a critic to match their ensemble with real-data samples, without specifying a likelihood function.The discrepancy is defined by maximizing the sample-based variational function over the critic.Source: §2.1 · p. 2

2.2 Sentence to feature

  • Text sentences are embedded word by word into a matrix, then a neural network such as an RNN, CNN, or SWEM extracts a feature vector.The discrete sentence is represented as one-hot word vectors before applying the embedding matrix.Source: §2.2 · p. 3

2.3 Optimal transport

  • Earth-Mover’s Distance compares distributions by assigning a ground cost between elements, with the distance given by the minimum transport cost over couplings having the required marginals.Source: §2.3 · p. 3

3 Feature Mover GAN

FM-GAN replaces the standard GAN critic with a feature-mover’s distance (FMD) between real and synthetic sentence features, using IPOT to compute the transport solution. The framework extends to style transfer and unsupervised cipher cracking.

3 Feature Mover GAN

  • FM-GAN replaces the original GAN critic with FMD between real and synthetic sentence features, using IPOT to compute the transport solution.FMD is a tractable variant of Earth-Mover’s Distance; IPOT computes the optimal transport matrix.Source: §3 · p. 3, §3.1 · p. 4

3.1 Feature-mover’s distance

  • FMD compares sentence-feature sets through a transport matrix whose cost is cosine distance, with equal set sizes during training.The training mini-batch sets m and n equal.Source: §3.1 · p. 4

3.2 Adversarial distribution matching with FMD

  • The adversarial distribution-matching objective applies FMD to features from real and generated sentences, using a CNN feature extractor and soft-argmax for differentiable generation.Multiple filters with different window sizes act as linguistic feature detectors; the LSTM generator uses soft-argmax as a continuous approximation to argmax.Source: §3.2 · pp. 4–5

3.3 Extensions to conditional text generation tasks

  • For non-parallel style transfer, the model combines reconstruction and adversarial losses; for cipher cracking, it combines feature-space cycle consistency and adversarial feature matching.The style-transfer objective balances its losses with λ; the decipher objective likewise balances cycle-consistency and adversarial losses.Source: §3.3 · pp. 5–6

4 Related work

Prior text GANs use reinforcement learning or continuous relaxations to handle discrete text, while OT-based approaches face challenges with vanishing gradients or regularization sensitivity. FM-GAN is distinguished by using IPOT for the original OT problem.

Related text GANs

  • RL-based text GANs use REINFORCE and discriminator rollouts, while LeakGAN uses hierarchical RL but is slow because of its complex design [60] [8] [37] [15] [24].RL-free GSGAN and TextGAN use Gumbel-softmax and soft-argmax, respectively, but retain the original GAN loss, which may produce discriminator gradient vanishing [31] [61].Source: §4 · p. 6

GAN with OT

  • For discrete text, WGAN’s EMD objective still has vanishing discriminator gradients after a few iterations, even with weight clipping or gradient penalty [23] [20].Source: §4 · p. 6
  • Sinkhorn-GM [19] and OT-GAN [50] optimize Sinkhorn divergence, an entropy-regularized EMD whose solution is sensitive to ε, trading computational efficiency against training stability; our method uses IPOT for original OT.IPOT is more efficient than Sinkhorn, and β only affects the convergence rate [59].Source: §4 · p. 6

5 Experiment

FM-GAN is evaluated on unconditional generation, non-parallel sentiment transfer, and unsupervised decipher. The reported results cover quality and diversity, transfer accuracy, and deciphering accuracy.

5 Experiment

  • The experiments cover unconditional generation on CUB, MS COCO, and WMT News, as well as non-parallel opposite-sentiment transfer and unsupervised decipher.CUB is used as a toy evaluation; the COCO sample contains 130,000 sentences, and WMT is a long-text collection.Source: §5 · pp. 6–7
  • Generation is evaluated with test-BLEU, self-BLEU, and human evaluation; style transfer uses a pretrained classifier and human evaluation, while deciphering uses word-mapping accuracy.For the WMT human Turing test, 10 judges rated over 100 randomly sampled sentences per model on a 0–5 scale.Source: §5 · p. 7, §5.1 · p. 8

5.1 Generic text generation

  • On CUB and MS COCO, FM-GAN achieves both high test-BLEU and low self-BLEU; on long-text WMT, it still produces realistic sentences.Figure 2 compares test-BLEU with self-BLEU, showing BLEU-3 above and BLEU-4 below.Source: §5.1 · pp. 7–8, Figure 2 · p. 7

5.2 Non-parallel text style transfer

  • On Yelp sentiment transfer, FM-GAN significantly exceeds CAE and also outperforms controllable text generation and BST in transfer accuracy.Table 4 presents transfer accuracy and human-evaluation results.Source: §5.2 · p. 8, Table 4 · p. 8

5.3 Unsupervised decipher

  • On Brown-W200 deciphering, replacing CipherGAN’s critic with the FMD objective yields higher accuracy than the original CipherGAN.The frequency-analysis baseline operates only when the cipher key is known; some CipherGAN setups were omitted because accuracy exceeded 99%.Source: §5.3 · p. 9, Table 6 · p. 9

6 Conclusion

FM-GAN applies feature-mover’s distance to text generation and performs well compared with existing approaches. The conclusion identifies several other tasks as potential future applications.

6 Conclusion

  • FM-GAN is a text-generation approach based on feature-mover’s distance (FMD).The conclusion reports good performance compared with existing text-generation approaches.Source: §6 · p. 9
  • The authors report good performance compared with existing text-generation approaches across several tasks.Source: §6 · p. 9
  • Potential future applications include image captioning, joint distribution matching, unsupervised sequence classification, and unsupervised machine translation.Source: §6 · p. 9

A Proof

The proof uses the Envelope theorem to establish that the transport-matrix gradient is zero. It frames the result through a constrained optimum and a value function, while noting that the derivative over the flow matrix is not computed.

A Proof

  • The proof uses the Envelope theorem to show that the gradient for the transport matrix is zero.Source: A Proof · p. 12
  • The setup assumes x* is an optimal solution of g with fixed θ under the constraint ξ(x) = 0, and defines V(ψ) from g evaluated at x*(ψ).Source: A Proof · p. 12
  • The derivation considers gradients with respect to θ and φ, while stating that the derivative over the flow matrix T is not computed.Source: A Proof · p. 12

B.1 Quantitative results

The appendix directs readers to quantitative results in Tables 7–12, covering test BLEU and self-BLEU results on CUB, MS COCO, and EMNLP WMT.

B.1 Quantitative results

B.2 Qualitative results

The appendix points to sentiment-transfer examples and generated-text samples from three datasets; these passages identify the relevant tables but do not state sample outcomes.

B.2 Qualitative results

Loading 1809.06297v2…