Source-linked AI summary
Long Text Generation via Adversarial Training with Leaked Information
Jiaxian Guo, Sidi Lu, Han Cai, Weinan Zhang, Yong Yu, Jun Wang
TL;DR
Long-text adversarial generation is hindered because discriminator feedback arrives only after completion and lacks intermediate structural information. LeakGAN leaks discriminator features into a MANAGER–WORKER hierarchical generator, and experiments report strong improvements across text lengths and tasks, with unsupervised sentence-structure learning.
Problem
Adversarial text generation has sparse, non-informative scalar guidance because discriminator feedback is available only after the entire text is generated, limiting long-sequence generation.
Method
LeakGAN uses a hierarchical MANAGER–WORKER generator whose MANAGER receives high-level discriminator features and produces goals for the WORKER during generation.
Results
LeakGAN significantly improves BLEU statistics and human Turing-test performance across synthetic and real datasets, while implicitly learning sentence structures without supervision.
Takeaways & Limitations
Leaked discriminator information provides guidance throughout generation, and MANAGER–WORKER interaction can implicitly learn sentence structures such as punctuation, clauses, and long suffixes.
Takeaways & Limitations
Feature-matching guidance in prior adversarial methods remains sparse because it is applied only after the whole text sample is generated.
Abstract
from arXiv · showhide
Automatically generating coherent and semantically meaningful text has many applications in machine translation, dialogue systems, image captioning, etc. Recently, by combining with policy gradient, Generative Adversarial Nets (GAN) that use a discriminative model to guide the training of the generative model as a reinforcement learning policy has shown promising results in text generation. However, the scalar guiding signal is only available after the entire text has been generated and lacks intermediate information about text structure during the generative process. As such, it limits its success when the length of the generated text samples is long (more than 20 words). In this paper, we propose a new framework, called LeakGAN, to address the problem for long text generation. We allow the discriminative net to leak its own high-level extracted features to the generative net to further help the guidance. The generator incorporates such informative signals into all generation steps through an additional Manager module, which takes the extracted features of current generated words and outputs a latent vector to guide the Worker module for next-word generation. Our extensive experiments on synthetic data and various real-world tasks with Turing test demonstrate that LeakGAN is highly effective in long text generation and also improves the performance in short text generation scenarios. More importantly, without any supervision, LeakGAN would be able to implicitly learn sentence structures only through the interaction between Manager and Worker.
Introduction
LeakGAN addresses the sparse, non-informative guidance that limits adversarial text generation on long sequences by leaking discriminator features into a hierarchical generator. Experiments report improvements across synthetic and real text tasks, with unsupervised learning of sentence structures.
- Existing adversarial text-generation methods mainly report results on short samples, while long generation is needed for news articles and product descriptions.
- Final binary or scalar discriminator feedback is sparse and does not preserve intermediate syntactic or semantic structure during generation.
- LeakGAN leaks discriminator high-level features to a hierarchical generator with MANAGER and WORKER modules.The MANAGER receives discriminator features and forms timestep-specific goals for the WORKER.
- The WORKER combines current-word representations with the MANAGER’s goal embedding, making discriminator guidance available throughout generation as well as at completion.
- LeakGAN reports lower negative log-likelihood at sequence lengths 20 and 40 and significant BLEU and human Turing-test improvements across synthetic, news, caption, and poem data.
- MANAGER–WORKER interaction indicates that LeakGAN implicitly learns punctuation, clause structure, and long suffixes without supervision.
Related Work
Prior work used adversarial training, feature matching, and hierarchical reinforcement learning to address weaknesses in text generation. LeakGAN combines leaked discriminator features with hierarchical reinforcement learning to provide denser guidance for long sequences.
- Supervised generative models face difficulty designing an appropriate differentiable, low-bias evaluation metric, motivating adversarial training.
- Feature Matching reduces weak guidance and mode collapse by matching real and generated latent feature distributions, but only after the complete text is generated.
- Hierarchical reinforcement learning addresses sparse rewards by learning high-level goals and low-level policies, with some frameworks learning sub-tasks implicitly rather than manually.
- LeakGAN sends discriminator features from the current generated sequence to a MANAGER that guides a WORKER, combining information leakage with hierarchical reinforcement learning.
Methodology
LeakGAN addresses sparse, uninformative adversarial guidance by leaking discriminator features into a hierarchical generator. A MANAGER converts these features into goals for a WORKER, with additional training techniques stabilizing adversarial learning and reducing mode collapse.
- Leaked Features: LeakGAN sends discriminator feature vectors from the current sentence to the generator as richer guidance than the scalar reward.The feature vector indicates the current sentence’s position in the discriminator’s extracted feature space.
- Hierarchical Generator: The MANAGER LSTM maps leaked features to goal vectors, while the WORKER uses recent goals to guide next-word generation.A linear transformation combines recent goals into an embedding that determines the WORKER’s vocabulary distribution.
- Generation Process: During generation, the MANAGER receives discriminator features at each step and produces goals that direct the WORKER toward higher-reward feature regions.Both modules begin from all-zero hidden states, and the MANAGER updates its recurrent state using the leaked features.
- Adversarial Training: The MANAGER is trained to align its goals with advantageous directions in discriminator feature space, while the WORKER receives reward-based policy-gradient training.The alignment uses cosine similarity between feature changes after c steps and the MANAGER’s goal, weighted by an expected reward estimated with Monte Carlo search.
- Training Stabilization: Bootstrapped rescaled activation ranks rewards within each mini-batch to stabilize their expectation and variance and reduce gradient vanishing.The method re-projects rank-based scores through an activation function; δ controls the smoothness of the rescaling.
- Training Stabilization: Interleaved supervised and adversarial training is used after pre-training to alleviate mode collapse and regularize the generator toward the maximum-likelihood solution.The example schedule performs one supervised-learning epoch after 15 adversarial-training epochs.
Experiment
Experiments evaluate LeakGAN on synthetic sequences and real text across long, mid-length, and short-generation settings using likelihood, BLEU, and human ratings.
- Experimental Setup: LeakGAN is compared mainly with an MLE-trained LSTM, SeqGAN, and RankGAN, with additional variants and real data used as an upper bound.
- Evaluation: Evaluation uses negative log-likelihood for synthetic data and BLEU statistics plus human Turing-test ratings for real-world data.The study also applies a t-test to compare LeakGAN with the second-highest performance and reports the p-value.
- Long Text Generation: EMNLP2017 WMT News: On WMT News, LeakGAN shows significant gains over baseline models in all measured BLEU metrics, indicating higher local-feature similarity to real text.
- Mid-Length and Short Text Generation: LeakGAN significantly outperforms baseline models on COCO captions and successfully handles Chinese Poems in short-text generation.The COCO result concerns mid-length text, while the Chinese Poems evaluation uses BLEU-2.
Short Text Generation: Chinese Poems
LeakGAN is evaluated on short Chinese-poem generation using BLEU-2, alongside analyses of long-text performance, feature traces, and Manager–Worker interactions.
- Short Text Generation: Chinese Poems: Chinese Poems contains 4-line 5-character poems, and BLEU-2 is used to evaluate short-text generation.The reported results indicate that LeakGAN successfully handles this short-text generation task.
- Long Text Generation: LeakGAN yields larger performance gains over SeqGAN and RankGAN as generated WMT News sentences become longer.This supports the framework’s robustness for long-text generation.
- Feature Trace: In WMT News feature traces, LeakGAN’s generated representations gradually approach the real-data feature region, unlike SeqGAN and RankGAN.The comparison is based on discriminator features visualized with two-dimensional PCA.
- Behaviors of Worker and Manager: The dimension-wise Worker–Manager interaction exhibits interpretable signals related to token divergence, suffix length, and sentence substructure.The fifth, sixth, and seventh dimensions are associated with structural tokens, long suffix triggers, and punctuation or conjunctions, respectively.
Conclusion and Future work
The paper concludes that LeakGAN improves adversarial text generation across long, mid-length, and short texts, with larger gains for longer sentences, while identifying future application and discriminator directions.
- Conclusion and Future work: LeakGAN leaks discriminator features as step-by-step guidance to address the non-informativeness and sparsity of scalar rewards in long-text generation.The framework uses adversarial training with a hierarchical generator.
- Conclusion and Future work: LeakGAN significantly improves BLEU scores and human ratings over previous solutions across long, mid-length, and short text.The conclusion covers synthetic and real-world experiments, including multiple text-length settings.
- Future work: The authors plan to apply LeakGAN to dialogue systems and image captioning with task-specific guiding information, and to improve global sentence-consistency checking.These are identified as future directions.
Appendix Formulas for Reference
The appendix defines the discriminator’s feature extraction and scalar-output computation.
- Appendix Formulas for Reference: The feature extractor maps a sequence s to a feature vector f, which the discriminator converts into a sigmoid score using its output layer.The formulas are f = F(s; φf) and Dφ(s) = sigmoid(φl · f).
- Appendix Formulas for Reference: The discriminator score is computed by applying sigmoid to the output-layer parameters dotted with the extracted feature vector.The displayed notation writes the score as sigmoid(φl · F(s; φf)).
- Appendix Formulas for Reference: The symbols φf and φl denote the feature-extractor and output-layer parameters in the discriminator equations.This distinction follows directly from the formula notation and architecture description in the passage.
Pseudo Code
The pseudocode alternates discriminator and hierarchical-generator pre-training before adversarial training, where generated sequences provide leaked features for guidance.
- Pseudo Code: Algorithm 1 initializes the hierarchical policy generator, discriminator, and their parameters with random weights.The generator uses Manager and Worker parameter sets θm and θw.
- Pseudo Code: The discriminator is pre-trained with dataset sequences as positive samples and generator outputs as negative samples.Its feature extractor and sigmoid output layer are trained together.
- Pseudo Code: The hierarchical generator is pre-trained using leaked discriminator information, with discriminator and generator pre-training interleaved until convergence.This stage precedes the repeated adversarial-training loop.
- Pseudo Code: During adversarial training, the generator samples a sequence and stores discriminator features f_t at each generation step.The stored features are then used in the subsequent policy-guidance computations.
Experiment Settings
The experiments use separate settings for synthetic sequences of lengths 20 and 40, with convolutional discriminator structures specified by sequence length, window size, and kernel count.
- Synthetic sequences of length 20 use learning rate 0.001 for both MANAGER and WORKER.
- Both synthetic settings use goal dimension k = 16 and LSTM-RNN embedding size 32.
- Synthetic sequences of length 40 use learning rate 0.0005 for both MANAGER and WORKER.
- The discriminator uses convolutional layer structures organized by sequence length, window size, and kernel numbers.
Discussions
The discussion treats hierarchical reinforcement learning as a useful design choice for incorporating leaked discriminator information, not as a necessary architecture.
- Hierarchical reinforcement learning is described as a good choice for sequence decision scenarios rather than a mandatory solution.
- Replacing the hierarchical architecture with a fully connected layer produced numerically sensitive, unstable training under the original settings.
- The authors suggest rapidly changing CNN feature spaces and unnormalized linear transformation may explain the instability.
Illustration of WORKER and MANAGER’s Behaviors
LeakGAN’s MANAGER and WORKER interact through token-level feature scores: the MANAGER supplies guidance, while the WORKER uses current words and that goal to choose actions.
- During generation, the WORKER combines its encoded current words with the MANAGER’s goal embedding to select the next action.
- Each curve shows a token subscore at a time step, with each dimension measuring a contextual feature of the token.
- The 5th dimension measures divergence from an entity token, with high values associated with structural tokens such as articles or prepositions.
- The 6th dimension indicates the expected suffix length, with peaks marking tokens that trigger long suffixes.
- The 7th dimension fluctuates near sub-sentence boundaries, where tokens are likely punctuation or conjunctions.
Illustration of Feature Trace
The feature-trace analysis compares generated and real-data representations, showing LeakGAN’s traces moving toward the real-data feature region during generation.
- Feature traces for SeqGAN, RankGAN, LeakGAN, and real data are compressed to two dimensions using PCA on WMT News.
- LeakGAN’s feature vector gradually approaches the real-data feature-vector region during generation.
- SeqGAN and RankGAN fail to match the real-data features even when generation is complete.