PaperlayerView exact revision on arXiv

Source-linked AI summary

A Contrastive Framework for Neural Text Generation

Yixuan Su, Tian Lan, Yan Wang, Dani Yogatama, Lingpeng Kong, Nigel Collier

arXiv:2202.06417v3cs.CL

TL;DR

Neural text generation can degenerate into repetitive or semantically inconsistent text because maximization-based decoding and sampling have complementary weaknesses. The paper introduces SimCTG and contrastive search to shape token representations and select coherent, diverse continuations, substantially outperforming current state-of-the-art approaches in human and automatic evaluations.

  • Problem

    Neural language models often generate repetitive text under maximization-based decoding, while sampling can produce continuations that diverge from the prefix semantics.

  • Method

    SimCTG uses contrastive training to learn isotropic, discriminative token representations, and contrastive search selects probable candidates while penalizing similarity to prior context.

  • Results

    The approach substantially reduces model degeneration and significantly outperforms current state-of-the-art text-generation methods in human and automatic evaluations across three benchmarks and two languages.

  • Takeaways & Limitations

    Contrastive training and decoding jointly target diversity and semantic coherence in open-ended neural text generation.

  • Takeaways & Limitations

    The approach is not tested on other GPT-like models designed for dialogue generation, which the authors leave for future work.

Abstract

from arXiv · show

Text generation is of great importance to many natural language processing applications. However, maximization-based decoding methods (e.g. beam search) of neural language models often lead to degenerate solutions -- the generated text is unnatural and contains undesirable repetitions. Existing approaches introduce stochasticity via sampling or modify training objectives to decrease probabilities of certain tokens (e.g., unlikelihood training). However, they often lead to solutions that lack coherence. In this work, we show that an underlying reason for model degeneration is the anisotropic distribution of token representations. We present a contrastive solution: (i) SimCTG, a contrastive training objective to calibrate the model's representation space, and (ii) a decoding method -- contrastive search -- to encourage diversity while maintaining coherence in the generated text. Extensive experiments and analyses on three benchmarks from two languages demonstrate that our proposed approach significantly outperforms current state-of-the-art text generation methods as evaluated by both human and automatic metrics.

1 Introduction

The paper attributes neural text degeneration to anisotropic token representations and proposes SimCTG with contrastive search to improve diversity while preserving coherence. Experiments across three benchmarks show strong gains over prior generation methods.

  • Motivation: MLE-based decoding often produces dull text with undesirable token-, phrase-, and sentence-level repetitions.Sampling can reduce repetition but may cause semantic inconsistency with the human-written prefix.
  • Motivation: GPT-2 token representations can have within-sentence cosine similarities above 0.95, reflecting an overly narrow representation space.The paper argues that such high similarity can cause repetitive token generation.
  • Approach: SimCTG trains discriminative and isotropic token representations, while contrastive search selects probable candidates and preserves similarity-matrix sparseness.These choices target semantic coherence and reduced degeneration simultaneously.
  • Results: Contrastive search with SimCTG significantly outperforms prior state-of-the-art decoding methods in human and automatic evaluations.The experiments cover three benchmarks, two languages, multiple tasks, and different model sizes.

2 Background

The background frames open-ended generation as continuation from a human-written prefix and distinguishes deterministic decoding from stochastic sampling. Both families have recognized weaknesses: probability maximization causes degeneration, while sampling can undermine semantic consistency.

  • Motivation: Likelihood maximization can yield anisotropic model representations that undermine the model’s capacity.This provides a representation-level motivation for the paper’s later contrastive objective.
  • Task and decoding: Open-ended generation decodes a continuation conditioned on a human-written prefix and uses either deterministic or stochastic methods.The resulting text combines the prefix with the decoded continuation.
  • Deterministic methods: Greedy and beam search select high-probability continuations, but maximizing output probability often produces dull and degenerate text.These methods are presented as widely used deterministic approaches.
  • Stochastic methods: Top-k and nucleus sampling restrict sampling to selected vocabulary subsets to avoid unreliable distribution tails.Top-k uses a fixed-size subset, whereas nucleus sampling uses a probability-mass threshold.
  • Stochastic methods: Sampling alleviates degeneration but can make generated semantics diverge from or contradict the human-written prefix.This semantic inconsistency is attributed to the intrinsic stochasticity of sampling.

3 Methodology

The method combines contrastive training, which shapes token representations toward discriminative and isotropic geometry, with contrastive search, which balances model confidence against contextual similarity. The decoder operates over top-k candidates and penalizes candidates resembling the prior context.

  • Contrastive Training: SimCTG adds a contrastive objective LCL to language-model training to encourage discriminative and isotropic token representations.The objective is defined over token representations in a variable-length sequence.
  • Contrastive Training: LCL pulls apart representations of distinct tokens, while the combined SimCTG objective includes the standard MLE objective.When the margin ρ equals 0, the combined objective reduces to vanilla MLE.
  • Contrastive Search: Contrastive search selects each output from the model’s most probable candidates while requiring sufficient discrimination from the previous context.This is designed to preserve prefix coherence while avoiding degeneration.
  • Contrastive Search: The decoder’s degeneration penalty is the maximum cosine similarity between a candidate representation and representations of tokens in the previous context.A larger penalty indicates greater similarity to the context and a greater likelihood of degeneration.
  • Contrastive Search: Contrastive search typically evaluates a top-k set with k=3∼10 and weights model confidence against the degeneration penalty using α.Setting α=0 makes the method equivalent to greedy search.

4 Document Generation

The document-generation experiments evaluate SimCTG and contrastive search on Wikitext-103 using language-modeling and generation-quality metrics, plus human judgments. Results indicate that SimCTG improves intrinsic modeling quality, while combining it with contrastive search reduces degeneration and improves human-likeness and coherence.

  • Experimental Setup: Experiments use GPT-2 fine-tuned on Wikitext-103, comparing SimCTG and baselines across greedy, beam, nucleus-sampling, and contrastive-search decoding.The setup uses equal overall training steps across compared training objectives and generates 128-token continuations from 32-token prefixes.
  • Evaluation: Evaluation separates language-modeling quality from generation quality, using perplexity, next-token accuracy, repetition, diversity, MAUVE, coherence, and generation perplexity.Human evaluation additionally measures coherence, fluency, and informativeness.
  • Language Modelling Quality: SimCTG achieves the best perplexity and next-token accuracy, indicating improved intrinsic language-modeling quality.The paper attributes this result to more discriminative representations that make next-token prediction less confusing.
  • Language Modelling Quality: Unlikelihood training obtains the best repetition metrics but causes unfavorable drops in perplexity and next-token accuracy.This result illustrates a trade-off between reducing repetition and preserving language-modeling quality.
  • Generation Quality: SimCTG with contrastive search performs best on repetition and diversity, produces the closest token distribution to human text, exceeds 0.6 coherence, and obtains better generation perplexity than other approaches.The results identify the combined training-and-decoding approach as the strongest generation-quality configuration on these metrics.
  • Generation Quality: Contrastive search improves MLE and unlikelihood baselines over greedy and beam search, but its gains are smaller without contrastive training because candidate degeneration penalties are less distinguishable.The paper links this limitation to less discriminative token representations under MLE or unlikelihood training.
  • Human Evaluation: Human evaluations show that SimCTG with contrastive search significantly outperforms nucleus sampling in coherence and fluency, while SimCTG-large performs best overall and comparably to human text in fluency.The reported significance test gives p-value < 0.05 for the coherence and fluency comparison and p-value > 0.4 for the large-model fluency comparison.

5 Open-domain Dialogue Generation

The authors evaluate SimCTG and four decoding methods for open-domain dialogue generation in Chinese and English. SimCTG with contrastive search significantly outperforms alternatives across multiple human-evaluated dimensions, while contrastive search also benefits MLE models.

  • Experimental Setup: The evaluation uses LCCC Chinese and DailyDialog English, comparing MLE and SimCTG models with greedy, beam, nucleus, and contrastive search.Human evaluation covers coherence, fluency, and informativeness on 200 dialogue contexts rated by five annotators using a 5-point Likert scale.
  • Results: SimCTG with contrastive search significantly outperforms other methods on multiple metrics across both dialogue datasets.The results support generalization across the evaluated languages and task setting.
  • Results: On LCCC, SimCTG with contrastive search exceeds human fluency performance while performing comparably to humans on coherence and informativeness.The reported coherence and informativeness comparison has Sign Test p-value > 0.4.
  • Results: Contrastive search also significantly improves the MLE model without contrastive training on the evaluated dialogue task.The authors attribute this result to the Chinese language model’s intrinsically isotropic representation space.
  • Scope: The authors leave evaluation on other GPT-like dialogue models for future work.The study uses GPT-2 and acknowledges other dialogue-oriented GPT-like models as an untested scope boundary.

6 Further Analysis

Further analyses examine representation self-similarity, contrastive-margin effects, diversity–perplexity trade-offs, decoding latency, and qualitative generation behavior. They show that SimCTG improves representation discrimination, contrastive search balances diversity with quality, and its latency remains practical relative to beam search.

  • Token Representation Self-similarity: SimCTG produces notably lower output-layer self-similarity than MLE and unlikelihood baselines, indicating more discriminative token representations.Intermediate-layer self-similarity is relatively similar across models, while the strongest difference appears at the output layer.
  • Contrastive Margin: Contrastive training improves perplexity over MLE across tested margins, although margins that are too small or large degrade the learned representation space.The analysis varies ρ from 0.1 to 1.0 and notes that ρ = 0 makes SimCTG equivalent to MLE.
  • Contrastive Search versus Nucleus Sampling: For nucleus sampling, small p yields human-like perplexity but low diversity, whereas large p yields human-like diversity but substantially higher perplexity.These settings respectively correspond to repetition loops and unlikely, low-quality text.
  • Contrastive Search versus Nucleus Sampling: When α ∈[0.5, 0.8], contrastive search achieves diversity and generation perplexity comparable to human performance.The result indicates a better balance between diversity and perplexity than the tested nucleus-sampling settings.
  • Decoding Latency Comparison: Contrastive search and beam search have nearly identical latency when b and k are at most 6, while contrastive search becomes faster when they exceed 6.Greedy search remains the fastest method in the fixed-length, batch-size-one comparison.
  • Case Study: Beam search generates sequence-level repetitions, while nucleus sampling can contradict the prefix semantics; contrastive search produces more reasonable repetitions and coherence.The case study reports low diversity for beam search and low coherence for the contradictory nucleus-sampling output.
  • Token Similarity Analysis: Similarity matrices are used to compare token-representation patterns for MLE with beam search, SimCTG with beam search, and SimCTG with contrastive search.The figure highlights prefix and generated-text similarity regions in red and yellow.

7 Conclusion

The paper attributes neural text degeneration to anisotropic token representations and proposes SimCTG with contrastive search to obtain more isotropic, discriminative representations and coherent generation. Experiments across three benchmarks and two languages show reduced degeneration and strong performance against existing approaches.

  • Conclusion: The conclusion identifies anisotropic token representations as the source of neural language-model degeneration.The proposed response is to train an isotropic, discriminative representation space with SimCTG and decode with contrastive search.
  • Conclusion: Automatic and human evaluations show that the approach substantially reduces model degeneration and significantly outperforms current state-of-the-art text-generation methods.The reported evidence spans three benchmarks and two languages.

Checklist

The paper reports checklist compliance, situates its approach within neural text generation and contrastive learning, and identifies future directions and limitations.

  • Checklist: The paper states that its main claims, limitations, code, training details, and experimental resources are documented.It also reports using publicly available datasets and provides human-evaluation guidelines and compensation details.
  • Future Work: The contrastive loss treats all other tokens in a sequence as negative samples, although some may share similar meanings despite different surface forms.The authors leave incorporating such semantically similar tokens to future work.
  • Future Work: The approach is architecture agnostic, but adapting it beyond open-ended generation to tasks such as translation and summarization remains future work.The paper also describes an accompanying Python package and instructions for reproducing results.
  • Background: Open-ended generation seeks natural, coherent, and informative continuations, whereas constrained generation is more tightly scoped by its input.The paper distinguishes deterministic and stochastic decoding methods for open-ended generation.
  • Related Work: Contrastive learning has been applied to token-, sentence-, and discourse-level representations and to several NLP applications.The paper positions its work as applying contrastive learning to neural text degeneration.
  • Novelty: The authors describe their work as the first effort, to their knowledge, to apply contrastive learning to neural text degeneration.They present this as a direction intended to facilitate future research.

D Experiments on Different Language Models

Experiments on Wikitext-103 evaluate SimCTG and contrastive search across language-model architectures and sizes, finding strong generalization across the evaluated models.

  • D Experiments on Different Language Models: Lower conicity or self-similarity indicates that a language model’s representation space more closely follows an isotropic distribution.The study uses both metrics to measure isotropy.
  • D Experiments on Different Language Models: SimCTG plus contrastive search performs best on all evaluated language models, indicating clear generalization across the tested models.The evaluation includes vanilla Transformers, GPT-2-small, and GPT-2-large.

E Ablation Study on the Hyperparameters of Contrastive Search

The hyperparameter ablation varies contrastive-search k and α on Wikitext-103 and examines their effects on generation diversity and perplexity.

  • E Ablation Study on the Hyperparameters of Contrastive Search: The ablation simultaneously varies k over {5, 8, 10} and α over {0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0}.Results are plotted for generation diversity and generation perplexity on the Wikitext-103 test set.

F Gen-ppl Results Measured by Different Models

The paper compares generation perplexity across evaluation models and reports that SimCTG with contrastive search is closest to human-written text, while noting additional language-specific and efficiency findings.

  • F Gen-ppl Results Measured by Different Models: SimCTG plus contrastive search achieves the generation perplexity closest to human-written text under both MLE- and Unlikelihood-trained evaluators.The two tables use different evaluator models, so their values differ slightly from Table 1 while supporting the same conclusion.
  • Human Evaluation: The human evaluation assesses coherence, fluency, and informativeness using five-point rating guidelines.Five annotators are hired for each experiment, and each annotator is paid $400 per task.
  • H Self-similarity of Chinese Language Models: For Chinese language models, MLE and SimCTG show similar layer-wise token self-similarity across all layers, including the final layer.This differs from the reported English-language pattern, where final-layer self-similarities are notably different.
  • H Self-similarity of Chinese Language Models: Contrastive search can be applied directly to the Chinese language model without contrastive training, although extending this finding to other character-tokenized languages remains untested.The paper specifically mentions Korean and Japanese as possible future settings.

I Training Efficiency Comparison

SimCTG is reported as more training-efficient than unlikelihood, adding only 1.48% computational overhead relative to MLE. Generated examples also show fluent, context-consistent responses across languages, while a small amount of initial sampling yields diverse outputs.

  • Training efficiency: 1.48% extra computational overhead is introduced by SimCTG compared with MLE on Wikitext-103 training.The comparison measures total training FLOPs across MLE, Unlikelihood, and SimCTG.
  • Training efficiency: SimCTG is reported as more training-efficient than the unlikelihood method.
  • Scope and limitations: The isotropy conjecture does not hold for larger English models such as GPT-2-large, motivating further investigation across model sizes and languages.
  • Generated examples: SimCTG plus contrastive search generates responses that are grammatically fluent and semantically consistent with dialogue context on Chinese LCCC examples.The case study is presented as evidence of generality across languages and tasks.
  • Diverse contrastive search: Sampling only 2 initial tokens is reported to produce a diverse set of results with diverse contrastive search.The procedure then uses contrastive search for the remaining 126 of 128 generated tokens.
Loading 2202.06417v3…