Source-linked AI summary
Adversarial Ranking for Language Generation
Kevin Lin, Dianqi Li, Xiaodong He, Zhengyou Zhang, Ming-Ting Sun
TL;DR
GANs struggle with discrete language and binary discrimination that inadequately represents rich sentence quality. RankGAN instead ranks machine- and human-written sentences relative to human references, trains the generator with policy gradients, and reports favorable results across multiple public datasets. The paper also notes that convergence to the intended Nash equilibrium remains an open problem for non-Bernoulli GANs.
Problem
GANs face non-differentiable discrete text and binary discriminator outputs that are too restrictive for diverse natural-language expressions.
Method
RankGAN replaces binary discrimination with relative ranking of machine- and human-written sentences against human references and uses policy gradients during generator training.
Results
Experimental results on multiple public datasets report favorable performance for RankGAN, including higher BLEU scores and better human-evaluation scores than compared methods.
Takeaways & Limitations
Collective relative ranking provides the discriminator with richer information for assessing sentence quality and supports high-quality natural-language generation.
Takeaways & Limitations
How a non-Bernoulli GAN converges to the intended Nash equilibrium remains an open problem.
Abstract
from arXiv · showhide
Generative adversarial networks (GANs) have great successes on synthesizing data. However, the existing GANs restrict the discriminator to be a binary classifier, and thus limit their learning capacity for tasks that need to synthesize output with rich structures such as natural language descriptions. In this paper, we propose a novel generative adversarial network, RankGAN, for generating high-quality language descriptions. Rather than training the discriminator to learn and assign absolute binary predicate for individual data sample, the proposed RankGAN is able to analyze and rank a collection of human-written and machine-written sentences by giving a reference group. By viewing a set of data samples collectively and evaluating their quality through relative ranking scores, the discriminator is able to make better assessment which in turn helps to learn a better generator. The proposed RankGAN is optimized through the policy gradient technique. Experimental results on multiple public datasets clearly demonstrate the effectiveness of the proposed approach.
1 Introduction
Language generation is important across NLP applications, but GANs face discrete-token optimization challenges and overly restrictive binary discrimination for rich language outputs.
- Language generation supports applications including machine translation, image captioning, and dialogue systems.
- GANs train a discriminator and generator adversarially, with the generator seeking to confuse the discriminator using its training signal.
- Discrete text tokens are non-differentiable, making GAN optimization difficult for natural language sequences.
- Existing language GANs commonly use binary human-versus-machine predicates, which are too restrictive for diverse and rich natural-language expressions.
- RankGAN replaces binary discrimination with relative ranking against human-written references and uses policy gradients to address non-differentiability.
2 Related works
Prior work applies GANs and learning-to-rank methods to language and retrieval, while RankGAN uses ranking scores as rewards for language generation.
- GANs: Prior GAN research in NLP uses policy gradients to handle discrete sequences, including sequence generation, dialogue, and image-caption diversity.
- Learning to rank: Learning-to-rank methods use reference information to return relevant documents early, whereas RankGAN uses ranking scores as rewards for language generation.
- RankGAN: RankGAN’s Figure 1 depicts a ranker comparing one synthetic sequence with multiple human-written sentences relative to a human-written reference.
3 Method
RankGAN replaces binary discrimination with a reference-based ranking system that compares human- and machine-written sentences. It trains the generator with policy gradients using ranking rewards, while the ranker evaluates relative quality across sentence sets.
- Overall architecture: RankGAN uses a sequence generator and ranker, training the ranker to assign relative scores rather than binary real-or-fake labels.The ranker compares machine-written and human-written sentences using reference groups.
- Model components: The generator uses LSTMs to produce word sequences, while the ranker maps sequence representations into feature vectors before scoring them against references.The ranker uses a convolutional architecture for sequence feature extraction.
- Rank score: The ranker computes sentence relevance from cosine similarity between input and reference embeddings, then converts similarities into comparison-set ranking scores.A temperature-like parameter γ controls how strongly higher similarities bias the scores.
- Rank score: Reference sentences are sampled from human-written data, while comparison sets contain human-written or machine-generated sentences according to the input sentence type.The resulting expected ranking score averages scores across sampled references and supplies objectives for both models.
- Training: Because text tokens are discrete, RankGAN trains its generator with policy gradients rather than ordinary back-propagation.A partial sequence is treated as a state, the next token as an action, and the completed sentence receives a ranking reward.
- Training: Monte Carlo rollouts estimate future ranking rewards for incomplete sequences, allowing intermediate rewards to guide generator updates.The average ranking score over sampled completion paths approximates the expected future reward.
4 Experimental results
Experiments on synthetic sequences, Chinese poems, COCO captions, and Shakespeare demonstrate that RankGAN performs favorably across automatic and human evaluations. The method also shows continued improvement during synthetic-data training and generates fluent, novel sentences on COCO captions.
- Experimental setup: RankGAN is evaluated against MLE, PG-BLEU, and SeqGAN on synthetic data and public datasets including Chinese poems, COCO captions, and Shakespeare’s plays.The experiments use NLL, BLEU, and human evaluation protocols across these settings.
- 4.1 Simulation on synthetic data: RankGAN consistently performs more favorably than the compared methods on the synthetic-data evaluation in terms of NLL.Its learning curve continues improving while MLE, PG-BLEU, and SeqGAN tend to converge after 200 training epochs.
- Chinese poems: RankGAN achieves higher BLEU-2 and human evaluation scores than the compared methods for Chinese poem generation, although human-written poems score highest.The human study involved 57 native Mandarin Chinese speakers who graded poems from 1 to 10.
- COCO captions: RankGAN obtains better BLEU scores than other methods on COCO captions, and examples show fluent, novel sentences absent from the training set.The COCO language models were trained on 80,000 captions, with 5,000 validation captions.
- COCO captions: RankGAN receives a better human evaluation score than SeqGAN among GAN approaches on COCO captions.Twenty-eight native or proficient English speakers graded sampled sentences from 1 to 10, while human-written sentences received the highest score overall.
- Shakespeare’s plays: RankGAN achieves consistently higher BLEU scores than MLE and SeqGAN across different n-gram criteria on Romeo and Juliet.The results indicate that it captures word-transition patterns in novel, delicate, and complicated training sentences.
5 Conclusion
RankGAN replaces binary discrimination with relative ranking between human-written and synthesized sentences, training the generator to produce sentences ranked higher by the learned ranker. Experiments on multiple public datasets report significantly better performance than previous state-of-the-art language generators.
- 5 Conclusion: RankGAN trains a ranker to place human-written sentences above synthesized sentences rather than assigning absolute binary predicates.The generator is then trained to produce sentences that receive higher ranking scores than human-written references.
- 5 Conclusion: The relative-ranking objective provides the discriminator with richer information for evaluating language sentences.The paper describes this relaxation of binary classification as favorable for synthesizing high-quality natural language.
- 5 Conclusion: Experimental results on multiple public datasets demonstrate significantly better performance than previous state-of-the-art language generators.
- 5 Conclusion: The authors plan to extend RankGAN to image synthesis and conditional GANs for image captioning.