Evaluating Text GANs as Language Models

1 Introduction

• LM-based text generation trains on gold-token histories but predicts from its own outputs at test time, creating exposure bias and difficulty recovering from prior errors.Source: §1 · p. 1
• GANs are difficult to evaluate with LM log-probability because they output discrete tokens rather than a probability distribution, and no single evaluation metric is established.Existing n-gram-overlap metrics lack robustness and correlate weakly with semantic coherence.Source: §1 · p. 1
• The paper proposes approximating a GAN generator's expected prediction as a language model, enabling evaluation with probability-based metrics such as perplexity or BPC.The authors apply the procedure to several RNN-based GANs.Source: §1 · p. 1
• RNN-based GANs are reported to have substantially lower BPC than state-of-the-art LMs, and concurrent work reached a similar conclusion using metrics proposed for GAN evaluation.The authors present direct comparison with LMs as a way to put current GAN text-generation performance in perspective.Source: §1 · pp. 1–2

2 Background

• Text GANs use either recurrent neural networks or convolutional neural networks as generators.RNNs can generate variable-length sequences, conditioning each token on tokens generated earlier.Source: §2 · p. 2
• Generating discrete symbols is nondifferentiable; continuous relaxation keeps generators differentiable, while SeqGAN and LeakGAN use policy gradients instead.The paper applies its approximation to both model types.Source: §2 · p. 2

3 Evaluating GANs and LMs

• For language models, average cross-entropy is computed from the log-probabilities of test tokens; perplexity is used for word-based models and BPC for character-based models.Source: §3 · p. 2
• Because text GANs output discrete sequences rather than probability distributions, standard LM metrics cannot directly evaluate them.Source: §3 · p. 2
• Existing GAN metrics include n-gram overlap, LM scores, proxy-LM perplexity, feature-distribution distance, grammar likelihood, and combined measures, each with stated drawbacks or trade-offs.The paper argues current methods cannot fully capture GAN performance and that a single measure is preferable when comparing text-generation models.Source: §3 · pp. 2–3

4 Proposed Method

• The proposed method approximates a GAN's distribution over tokens and evaluates the resulting model with standard LM metrics.The approach is described for RNN-based language models and can also apply to other autoregressive models.Source: §4 · p. 3

4.1 Language Model Approximation

• At evaluation time, gold tokens are supplied as input, unlike inference where the generator's previous output is fed back; the resulting approximate LM can be scored with BPC or perplexity.Source: §4.1 · p. 3
• The expected generator output defines a distribution over the next token, which the method estimates by averaging independent samples from the generator.The generator's stochasticity can come from its initial state or its internal distribution over output tokens.Source: §4.1 · p. 3

4.2 Approximation Bound

• A theoretical bound guides the sample count by bounding the probability that any vocabulary token's estimated probability differs from its expected probability by more than γ.The method defines this discrepancy as a bad event whose probability can be bounded by ε.Source: §4.2 · p. 3, §1 · p. 3

1. The probability of a token at to be v is pv

• The Chernoff–Hoeffding bound controls each coordinate's estimation error, and a union bound over V yields a lower bound on N.The argument assumes the samples indexed by n are i.i.d.Source: §1 · p. 4
• For a character-based LM on text8 with |V| = 27 and ϵ = 10^-2, the bound is N > 4.3 · 10^6; with the same γ and ϵ, a typical word-based LM with |V| = 50,000 requires N > 8.1 · 10^6.Source: §1 · p. 4
• Because LM probability vectors tend to be sparse and their sparsity is difficult to bound across models, the authors suggest empirically choosing N.They note that sparsity may allow a much smaller N for a good approximation.Source: §1 · p. 4

5.1 Models

• The evaluation tests two character-based RNN GANs initialized with gold text: continuous Recurrent GAN and discrete SeqGAN, which is pre-trained as a language model.Source: §5.1 · p. 4
• LeakGAN was also evaluated but omitted because it is similar to SeqGAN and performed worse.Source: §5.1 · p. 4

5.2 Evaluation Settings

• The models are evaluated on text8, using its standard 90/5/5 train/validation/test split and BPC.Source: §5.2 · p. 4

5.3 Results

• GAN-based models trail state-of-the-art LMs by a large margin; SeqGAN’s approximate BPC is 1.95 for the pre-trained LM and 2.06 for the fully trained model, worsening with continued adversarial training.Source: Table 1 · p. 5, §5.3 · p. 5
• Random 100-character samples indicate adversarial training slightly reduces SeqGAN text quality, and the quality of Press et al.’s 100-character sequences is low.Source: Table 2 · p. 5, §5.3 · p. 5

5.2 Evaluation Settings

• Figure 2 plots approximation error against sample count; N > 1600 meets the criterion, so the evaluation uses N = 2000.Source: §5.2 · p. 4, Figure 2 · p. 5

5.3 Results

• Approximate BPC is only slightly higher than true BPC for SeqGAN, whose token distributions permit measuring the true score.Source: §5.3 · p. 5
• Generating Press et al.’s sequences beyond 7 characters hurts BPC, while such short sequences make generation quality difficult to assess.Source: §5.3 · p. 5

• The procedure approximates GAN output distributions and applies standard LM metrics, with a bound on required samples and an empirical finding that as few as 2000 samples per time-step suffice.Source: §6 · p. 5
• Character-based GAN models evaluated with the procedure perform substantially worse than state-of-the-art LMs.Source: §6 · p. 5
• The authors hope the method will clarify text GAN quality and lead to progress in GAN-based text generation.Source: §6 · p. 5