Source-linked AI summary
Texygen: A Benchmarking Platform for Text Generation Models
Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, Yong Yu
TL;DR
Text generation research lacks comprehensive evaluation, reproducible implementations, and reliable diversity assessment. Texygen responds with an open-source benchmarking platform that combines baseline models with metrics covering diversity, quality, and consistency. The platform reports that some NLP metrics, such as CFG, can favor models with more severe mode collapse.
Problem
Text generation evaluation lacks a comprehensive metric, reproducible implementations are difficult to obtain, and diversity assessment remains inadequate.
Method
Texygen combines open-source baseline text-generation models with automatically computable metrics evaluating generated-text diversity, quality, and consistency.
Results
Texygen finds that CFG cannot distinguish different models and can favor models with more severe mode collapse.
Takeaways & Limitations
Texygen provides a common benchmarking framework for evaluating models and comparing them with existing baselines from multiple perspectives.
Takeaways & Limitations
In one experiment, GSGAN failed to generate any semantically meaningful sentences.
Abstract
from arXiv · showhide
We introduce Texygen, a benchmarking platform to support research on open-domain text generation models. Texygen has not only implemented a majority of text generation models, but also covered a set of metrics that evaluate the diversity, the quality and the consistency of the generated texts. The Texygen platform could help standardize the research on text generation and facilitate the sharing of fine-tuned open-source implementations among researchers for their work. As a consequence, this would help in improving the reproductivity and reliability of future research work in text generation.
1 INTRODUCTION
Open-domain text generation lacks comprehensive evaluation, reproducible implementations, and reliable diversity assessment. Texygen addresses these gaps with an open-source platform combining baseline models and metrics for diversity, quality, and consistency.
- Evaluation lacks a single comprehensive metric, so multiple metrics are required to assess text generation models.Existing measures include perplexity, oracle generated log-likelihood, human scores, and BLEU.
- Researchers often do not publicly release source code, making reported experimental results difficult to reproduce.
- Text generation faces a quality-diversity tradeoff in which restricted output patterns and mode collapse affect released models.
- There is a need for a reliable common framework that thoroughly evaluates existing models and facilitates development of new ones.
- Texygen is a fully open-sourced benchmarking platform containing baseline models and metrics for generated-text diversity, quality, and consistency.
2 THE TEXYGEN PLATFORM
Texygen combines open-source baseline text-generation models with decoupled, automatically computable metrics for evaluating quality, similarity, likelihood, and diversity. Its metrics include newly proposed measures such as EmbSim, NLLtest, and Self-BLEU, while the platform supports both synthetic-data and real-data training workflows.
- Platform components: Texygen provides well-trained baseline models, automatically computable evaluation metrics, and APIs for researchers to evaluate their own models and generated text.The platform is open source and designed for customization through decoupled system components.
- Baseline models: The implemented baseline collection spans likelihood-based, adversarial, and hierarchical methods, including MLE, SeqGAN, MaliGAN, RankGAN, TextGAN, GSGAN, and LeakGAN.SeqGAN uses a discriminator and REINFORCE with Monte Carlo Q-value estimation; GSGAN replaces multinomial sampling with a differentiable Gumbel Softmax reparameterization.
- Document similarity and likelihood metrics: The quality and similarity metrics compare generated text with natural language or training data using BLEU, EmbSim, and likelihood-based measures.EmbSim compares word-embedding similarity matrices, while NLLoracle evaluates fit to an oracle LSTM and NLLtest evaluates capacity to fit real test data.
- Likelihood-based metrics: NLLtest evaluates a model’s capacity to fit real test data and is dual to NLLoracle, but it applies only to autoregressive generators such as RNNs.Its likelihood calculation uses the generator’s conditional probability for each word given preceding tokens.
- Divergence-based metrics: Self-BLEU measures generated-data diversity by averaging BLEU scores across generated sentences, with higher scores indicating lower diversity and more serious mode collapse.This metric is motivated by the tendency of GANs to collapse onto a single sample or a small family of similar samples.
3 PLATFORM ARCHITECTURE
Texygen separates reusable utilities from model training through decoupled system components. Users interact primarily with shared metric, oracle, and GAN interfaces while choosing synthetic or real-data training.
- System architecture: Texygen is implemented over TensorFlow as two decoupled system parts containing three major classes.The architecture is designed for customization.
- Utilities: The utils part provides Metrics and Oracle classes for evaluation and oracle initialization.Metrics supports BLEU, NLL loss, and EmbSim; Oracle supports LSTM-, GRU-, and SRU-based oracles, with LSTM as default.
- Model interface: The model part exposes a GAN class that hides generator, discriminator, and reward-class details during training.This interface supports both synthetic-data and real-data training processes.
- Training modes: Synthetic-data training uses an oracle LSTM, whereas real-data training uses real-world datasets such as COCO image captions.
4 EXPERIMENT
Texygen evaluates text-generation models across synthetic and real data using likelihood, semantic similarity, BLEU, and diversity-oriented metrics. The experiments report distinct trade-offs: LeakGAN performs strongly across several quality measures, while MLE and MaliGAN retain greater diversity.
- Training setting: Synthetic training uses 5,000 words, sentence length 20, and 10,000 oracle-generated sentences; real training uses 20,000 COCO captions split evenly into training and test sets.
- Evaluation metrics: NLLoracle and NLLtest evaluate synthetic-data training, while BLEU, Self-BLEU, and EmbSim evaluate real-data training.BLEU and EmbSim are not calculated for synthetic data because the oracle LSTM cannot generate semantically meaningful words.
- Synthetic-data results: LeakGAN converges more quickly and performs well on NLLoracle and NLLtest, whereas TextGAN has the best NLLoracle but the worst NLLtest during pretraining.SeqGAN, MaliGAN, and RankGAN have nearly identical curves until adversarial training, after which MaliGAN becomes less competitive.
- Real-data results: During adversarial training, only LeakGAN maintains its EmbSim score; other baseline models decline relative to pretraining.LeakGAN begins with very high similarity, while TextGAN improves slowly.
- Experimental scope: GSGAN is excluded from this experiment because it fails to generate semantically meaningful sentences.
- Real-data results: LeakGAN outperforms other baselines on BLEU and shows strong test performance, while MaliGAN has the lowest BLEU score.Training- and test-data BLEU scores are reported separately in Tables 1 and 2.
- Real-data results: All models are less diverse than the original training data; LeakGAN and TextGAN show more mode collapse, while MLE and MaliGAN produce the highest diversity.
- Additional observations: LeakGAN tends to generate longer sentences, whereas TextGAN tends to generate shorter sentences.The paper identifies this as an example of a more detailed empirical study available through the Texygen project webpage.
5 CONCLUSION AND FUTURE WORK
Texygen provides a benchmarking platform for fair, multi-perspective comparisons between researchers’ models and existing baselines. The authors also find that some NLP metrics are unsuitable for text generation and plan to develop better metrics.
- Texygen enables fair and convenient comparisons between researchers’ models and existing baseline models from different perspectives.
- Its varied evaluation metrics support a more comprehensive benchmark of text generation models.
- Context-free grammar cannot distinguish different models and may favor models with more severe mode collapse.
- Future work will add models and design novel metrics for improved text-generation benchmarking.