PaperlayerView exact revision on arXiv

Source-linked AI summary

Residual Energy-Based Models for Text Generation

Deng, Yuntian, Bakhtin, Anton, Ott, Myle, Szlam, Arthur, Ranzato, Marc'Aurelio

arXiv:2004.11714v1cs.CLcs.LG

TL;DR

Locally normalized autoregressive models generate efficiently but face exposure bias and limited sequence-level coherence. The paper adds a globally normalized residual energy model over a pretrained language model and trains it with noise contrastive estimation. Across two large datasets, the approach lowers perplexity and produces generations preferred by human evaluators, while relying on the quality of its base language model and practical sampling constraints.

  • Problem

    Locally normalized token-by-token generation suffers from exposure bias and can lack long-range coherence.

  • Method

    The paper trains a sequence-level residual EBM over a fixed pretrained language model using conditional noise contrastive estimation and importance sampling.

  • Results

    Residual EBMs achieve lower perplexity than autoregressive baselines on two large datasets and receive 56% to almost 60% human preference in reported comparisons.

  • Takeaways & Limitations

    Residual EBMs provide an efficient way to combine sequence-level scoring with pretrained bidirectional representations for text generation.

  • Takeaways & Limitations

    Performance and training depend strongly on the base language model, while naive autoregressive sampling from the joint model is computationally expensive and impractical.

Abstract

from arXiv · show

Text generation is ubiquitous in many NLP tasks, from summarization, to dialogue and machine translation. The dominant parametric approach is based on locally normalized models which predict one word at a time. While these work remarkably well, they are plagued by exposure bias due to the greedy nature of the generation process. In this work, we investigate un-normalized energy-based models (EBMs) which operate not at the token but at the sequence level. In order to make training tractable, we first work in the residual of a pretrained locally normalized language model and second we train using noise contrastive estimation. Furthermore, since the EBM works at the sequence level, we can leverage pretrained bi-directional contextual representations, such as BERT and RoBERTa. Our experiments on two large language modeling datasets show that residual EBMs yield lower perplexity compared to locally normalized baselines. Moreover, generation via importance sampling is very efficient and of higher quality than the baseline models according to human evaluation.

1 INTRODUCTION

Locally normalized autoregressive models generate efficiently but suffer from exposure bias and limited sequence-level coherence. Residual EBMs address these issues by combining a fixed language model with a globally normalized sequence-level energy function, enabling efficient training, evaluation, and generation.

  • 1 INTRODUCTION: Locally normalized generation suffers from exposure bias and can lack long-range coherence because it selects tokens greedily without lookahead.Training uses ground-truth context, whereas testing uses the model’s own generated context.
  • 1 INTRODUCTION: Residual EBMs score complete sequences with a globally normalized energy function layered over a fixed locally normalized language model.The energy term provides sequence-level modeling while the pretrained language model supplies the base distribution.
  • 1 INTRODUCTION: Conditional noise contrastive estimation trains the residual energy function by distinguishing the joint model from negatives sampled from the pretrained language model.Because the joint model differs from the language model by the energy term, the classifier score reduces to -Eθ.
  • 1 INTRODUCTION: Importance sampling makes residual-EBM evaluation and generation efficient while allowing perplexity comparisons with standard autoregressive models.The pretrained language model acts as a natural proposal distribution for sampling.
  • 1 INTRODUCTION: Residual EBMs lower perplexity on two large datasets and receive significantly higher human preference for generated text than strong autoregressive baselines.The paper presents this as the first demonstration of improved EBM generation against strong autoregressive baselines using both estimated perplexity and human evaluation.

2 RELATED WORK

Prior work connects energy-based sequence modeling with discriminative reranking, adversarial generation, and sequence-level training. This paper extends residual energy modeling with tractable perplexity bounds, efficient importance-weighted generation, and bidirectional-transformer language modeling.

  • 2 RELATED WORK: The approach resembles discriminative reranking but directly tunes a generative model toward the data distribution rather than optimizing a surrogate ranking loss.The paper characterizes this as a generative model rather than a reranker.
  • 2 RELATED WORK: Unlike sequence-level objectives whose test-time generation remains greedy, the proposed generative model samples from a globally defined joint distribution.The distinction is between improving a baseline during training and changing the generation procedure at test time.
  • 2 RELATED WORK: Earlier energy-network work used residual modeling and related training procedures, but this paper develops bounds that enable perplexity evaluation under the joint model.The cited prior work used an LSTM generator and CNN-LSTM energy function for speech recognition.
  • 2 RELATED WORK: The residual formulation makes it natural to use BERT for language modeling, with reported gains over modern baselines in perplexity and human evaluation.The paper positions bidirectional contextual modeling as a key benefit of the residual EBM framework.
  • 2 RELATED WORK: Related discriminator-based methods use fixed or non-locally normalized generators, whereas this work uses a pretrained locally normalized language model as the fixed generator.The comparison includes adversarial and importance-sampling approaches.

3 RESIDUAL ENERGY-BASED MODELS

The model combines a fixed locally normalized language model with a globally normalized residual energy function, trained using conditional noise contrastive estimation. Evaluation estimates perplexity through asymptotic partition-function bounds, while efficient generation uses importance sampling rather than impractical autoregressive marginalization.

  • 3 RESIDUAL ENERGY-BASED MODELS: The joint model multiplies a fixed pretrained language model by an exponential residual energy term, producing a globally normalized sequence model.The residual energy function is trained while the pretrained language model remains fixed.
  • 3 RESIDUAL ENERGY-BASED MODELS: The partition function is intractable because exact computation sums over |V|^(T−p) continuations, making conventional likelihood sampling too expensive for text.The experiments use a vocabulary of 50,096 and generation length 40 tokens.
  • 3 RESIDUAL ENERGY-BASED MODELS: Conditional noise contrastive estimation trains the energy function by distinguishing human-generated positive sequences from language-model-generated negative sequences.The objective assigns lower energy to real data and higher energy to generated data.
  • 3 RESIDUAL ENERGY-BASED MODELS: Perplexity estimation uses lower and upper partition-function estimators whose bounds hold asymptotically when the sample count is sufficiently large.A leave-one-out strategy is used to reduce the variance of the upper estimator.
  • 3 RESIDUAL ENERGY-BASED MODELS: Autoregressive generation by marginalizing future tokens is computationally expensive and impractical, so it is used only for qualitative analysis.The paper instead relies on self-normalizing importance sampling for efficient generation.
  • 3 RESIDUAL ENERGY-BASED MODELS: Efficient generation samples sequences from the pretrained language model and resamples them according to their energy-based weights, optionally applying top-k sampling.With infinitely many proposal samples, this procedure recovers exact samples from the joint model distribution.

4 EXPERIMENTS

Experiments evaluate residual EBMs across two datasets, matched autoregressive baselines, perplexity estimation, human preferences, and distributional analyses. Bidirectional residual models achieve the strongest reported performance, while estimates and generation quality are examined under sampling-based evaluation.

  • Automatic Evaluation: JOINT BIT-BASE* and JOINT BIT-LARGE* outperform JOINT BIT-BASE, while JOINT BIT-BASE already exceeds JOINT UNIT with fewer parameters.The results compare residual models initialized from RoBERTa with causal and bidirectional variants.
  • Automatic Evaluation: JOINT BIT-BASE on a deeper BASE LM-24L achieves lower perplexity than BALM-24L despite fewer parameters, while JOINT BIT-MED lowers perplexity further at matched size.The comparison uses the lower part of Table 1 and includes deeper autoregressive baselines.
  • Human Evaluation: Humans preferred JOINT BIT-BASE and JOINT BIT-LARGE* over both language-model baselines in 56% to almost 60% of comparisons.The evaluation presents one completion per model for each prefix and reports preferences on CC-News.
  • Analyses: Beyond 20,000 samples, the upper perplexity estimate becomes stable, although the estimates may remain biased.The lower and upper bounds converge as sample counts increase, but the bounds are asymptotic estimates.
  • Analyses: The joint model produces slightly more unique 2-, 3-, and 4-grams than the baseline language model, but fewer than human text.This analysis tests whether residual modeling alleviates repetition artifacts.
  • Analyses: The joint model’s log-probability density over its samples more closely matches the real-data distribution than the baseline language model’s density.Figure 3 compares real samples with samples from BASE LM and BIT-BASE.

5 LIMITATIONS

Residual EBMs depend strongly on the quality and coverage of the pretrained autoregressive language model used to generate negatives. This base-model dependence constrains when the approach succeeds and can make search inefficient.

  • 5 LIMITATIONS: Training uses negatives from the pretrained language model, so success requires plausible ground-truth and alternative sequences to be reachable by that model.
  • 5 LIMITATIONS: Poor base-model quality makes joint-model generation poor because the joint model resamples generations from the original language model.
  • 5 LIMITATIONS: Poor base models also make search catastrophically inefficient because many negatives may be needed to find samples near the true data manifold.
  • 5 LIMITATIONS: Residual EBMs are expected to work well only when the pretrained base language model is strong.The paper describes this as a strong implicit assumption and notes that residual learning carries liability to its base model.

6 CONCLUSIONS AND FUTURE WORK

The paper combines a residual EBM with a pretrained autoregressive language model to score sequences holistically while keeping training and generation efficient. It reports lower perplexity than the base model and identifies improved negative generation and longer-text modeling as future directions.

  • 6 CONCLUSIONS AND FUTURE WORK: The residual EBM scores sequences holistically while training through binary classification and generating by resampling base-model negatives.
  • 6 CONCLUSIONS AND FUTURE WORK: The resulting model has lower perplexity than the pretrained base language model.
  • 6 CONCLUSIONS AND FUTURE WORK: Future work includes generating better negatives, exploring other loss functions, and modeling longer text autoregressively at the chunk level.

A.1 TOP-K AUTO-REGRESSIVE SAMPLING

The appendix approximates joint-model next-token probabilities by sampling completions from top BASE LM words, then compares the highest-probability entries. BIT-BASE generally lowers probabilities assigned to repetitions while preserving the same top-k words.

  • A.1 TOP-K AUTO-REGRESSIVE SAMPLING: The approximation considers the top 128 BASE LM words, samples 10,000 completions for each, and retains the top 10 renormalized probabilities.
  • A.1 TOP-K AUTO-REGRESSIVE SAMPLING: BIT-BASE usually assigns lower probabilities to repetitions than BASE LM, although the top-k words remain the same.The comparison uses sampled completions for the top 128 BASE LM words and reports the top 10 entries after renormalization.

A.2 PROOF OF THEOREM 2

Theorem 2 establishes that the empirical log-normalizer estimate approaches the true log-normalizer, while a bias-corrected combination approaches it from above. The proof uses asymptotic expansions of the estimator’s expectation.

  • A.2 PROOF OF THEOREM 2: Theorem 2 bounds the expected empirical estimate below the true log-normalizer and a bias-corrected estimate above it for sufficiently large n.
  • A.2 PROOF OF THEOREM 2: The empirical estimate converges in expectation to the true log-normalizer as n approaches infinity.
  • A.2 PROOF OF THEOREM 2: The proof derives the lower-side behavior from an asymptotic expansion involving central moments of the log-weight estimator.
  • A.2 PROOF OF THEOREM 2: The corrected estimator’s positive asymptotic offset establishes its eventual upper-bound behavior.

A.3 OPTIMIZATION SETTINGS

The optimization settings are documented in Table 4, which uses the same setting for CC-News and the Toronto Book Corpus.

  • Table 4 lists the optimization settings used in the experiments.
  • The same optimization setting is used for CC-News and the Toronto Book Corpus.
  • The section includes a reference to the optimization-settings table but provides no additional settings values here.

A.4 HUMAN EVALUATION

The human evaluation compares model continuations through controlled voting by qualified crowd workers. The paper also presents examples where the proposed model either outperforms or underperforms BALM.

  • 333 CC-News test sentences were rated by three Amazon Mechanical Turk workers, with the continuation receiving more votes judged better.Workers were screened using a qualification task, and the top half were retained.
  • The evaluation presents examples where BIT-BASE outperforms BALM and examples where it underperforms BALM.
  • Table 5 contains examples of BIT-BASE wins over BALM, while Table 6 contains examples of BIT-BASE losses.BIT-BASE-WORST identifies the language-model sample with the highest energy score in both tables.
Loading 2004.11714v1…