Source-linked AI summary
COLD Decoding: Energy-based Constrained Text Generation with Langevin Dynamics
Lianhui Qin, Sean Welleck, Daniel Khashabi, Yejin Choi
TL;DR
Constrained text generation must satisfy varied hard and soft semantic or stylistic requirements, but existing approaches may require costly task-specific data or model training. COLD unifies these constraints in an energy function and samples with Langevin dynamics over continuous text relaxations. Across lexical, abductive, and counterfactual generation, it reports stronger lexical coverage or text quality than competing methods and supports automatic and human evaluation.
Problem
Constrained generation involves varied hard and soft semantic or stylistic requirements, while task-specific supervision and model training can be costly or infeasible for very large language models.
Method
COLD specifies an energy function from task-relevant constraints, performs Langevin sampling over continuous token representations, and guidedly discretizes the samples into text.
Results
COLD supports lexical, abductive, and counterfactual constrained generation, achieving better lexical coverage than NEUROLOGIC and more coherent, higher-quality text than DELOREAN.
Takeaways & Limitations
COLD provides a unified decoding framework that applies directly to off-the-shelf language models and performs strongly under automatic and human evaluation.
Takeaways & Limitations
Langevin dynamics is applied to continuous token vectors because gradients are not well-defined for discrete text sequences.
Abstract
from arXiv · showhide
Many applications of text generation require incorporating different constraints to control the semantics or style of generated text. These constraints can be hard (e.g., ensuring certain keywords are included in the output) and soft (e.g., contextualizing the output with the left- or right-hand context). In this paper, we present Energy-based Constrained Decoding with Langevin Dynamics (COLD), a decoding framework which unifies constrained generation as specifying constraints through an energy function, then performing efficient differentiable reasoning over the constraints through gradient-based sampling. COLD decoding is a flexible framework that can be applied directly to off-the-shelf left-to-right language models without the need for any task-specific fine-tuning, as demonstrated through three challenging text generation applications: lexically-constrained generation, abductive reasoning, and counterfactual reasoning. Our experiments on these constrained generation tasks point to the effectiveness of our approach, both in terms of automatic and human evaluation.
1 Introduction
COLD addresses constrained text generation by representing diverse hard and soft constraints as energy functions and sampling from them with Langevin dynamics. It applies this unified decoding framework directly to pretrained language models and performs strongly across three constrained-generation tasks.
- Motivation: Task-specific supervised data is costly because constrained-generation applications involve varied and evolving constraints.Prior energy-based approaches reduce supervised-data needs but still require training task-specific generation models, which may be infeasible for extremely large language models.
- Approach: COLD formulates constrained decoding as sampling from an energy-based model whose energy combines arbitrary task-relevant constraint functions.The framework unifies hard lexical constraints and soft contextual constraints within a single energy-based formulation.
- Approach: Langevin dynamics iteratively updates a continuous relaxation of text using energy gradients before guided discretization produces fluent constrained sequences.This addresses the difficulty of sampling discrete text from energy-based models through gradient-based sampling in a continuous space.
- Experiments: COLD evaluates lexically constrained generation, abductive reasoning, and counterfactual story generation under one decoding framework.Each setting uses an energy function composed of fluency and task-specific constraints.
- Results: COLD achieves better lexical coverage than NEUROLOGIC and more coherent, higher-quality text than DELOREAN across the evaluated constrained-generation settings.The paper reports strong performance on both automatic and human evaluation.
2 Background
Constrained text generation requires decoding beyond standard left-to-right token selection, which struggles with diverse lexical, coherence, and similarity constraints. COLD uses energy-based modeling and continuous Langevin sampling to make such constraints differentiable and applicable during decoding.
- Neural text generation: Neural language models factorize sequence probabilities into left-to-right per-token conditionals, and decoding algorithms use these distributions to generate text.Common methods include beam search, greedy decoding, top-k sampling, and nucleus sampling.
- Constrained text generation: Constrained generation seeks sequences satisfying collections of soft and hard requirements, including fluency, coherence, lexical inclusion, and similarity.Examples include text infilling, keyword-constrained generation, and constraints extending beyond a left-hand prefix.
- COLD decoding: COLD performs Langevin iterations on a soft sequence and then applies top-k filtering to obtain discrete text.The resulting sequence approximately samples from the desired constrained distribution after the specified number of iterations.
- Constrained text generation: Monotonic left-to-right decoding makes diverse constraints difficult to enforce because each next-token decision depends on the preceding prefix.This limitation is especially relevant when constraints involve broader context or similarity to another sequence.
- Energy-based models: Energy-based models represent distributions with energies and can incorporate arbitrary constraint functions into the energy.Unlike prior task-specific text-EBM training approaches, COLD uses the formulation for inference with pretrained language models without fine-tuning.
- Energy-based models: Sampling from text energy-based models is difficult because normalization is intractable and text is discrete, making energy gradients undefined.COLD bridges this gap using continuous relaxation, differentiable constraints, and guided discretization.
3 COLD Decoding with Langevin Dynamics
COLD treats constrained generation as sampling from an energy-based distribution over soft token sequences, using Langevin dynamics to combine differentiable constraints before producing discrete text.
- 3.1 Energy-based Decoding: COLD defines constrained decoding as sampling from an energy-based distribution whose energy combines weighted constraint functions.The framework supports arbitrary task-specific constraint functions and can generate multiple samples for selection.
- 3.2 A Collection of COLD Constraints: COLD supports differentiable fluency, future-context, lexical, and n-gram similarity constraints within one energy-based decoding procedure.N-gram similarity can encourage keywords when n = 1 or resemblance to a reference sequence when n is larger.
- 3.1 Energy-based Decoding: Because text is discrete and lacks a usable gradient, COLD applies Langevin dynamics to continuous soft token vectors.Each soft-token position contains vocabulary logits; softmax yields a vocabulary distribution that can approach a one-hot token.
- 3.3 From Soft to Discrete and Fluent Text: Langevin iterations update soft sequences using the energy gradient and noise, after which top-k filtering maps the resulting sample to discrete text.Top-k filtering selects among language-model candidates using the soft sample, preserving fluency while accommodating constraints.
- 3.2 A Collection of COLD Constraints: The soft fluency constraint matches token distributions to predictions from an underlying language model, optionally using both left and right language-model contexts.The future-token constraint adjusts soft tokens so the language model predicts fixed tokens on the right, supporting coherent infilling.
4 Experiments
COLD is evaluated as a fine-tuning-free constrained decoding method across abductive reasoning, counterfactual story rewriting, and lexical constraint satisfaction. It generally improves coherence, coverage, or both against task-specific decoding baselines, while exposing trade-offs between competing constraints and sampling efficiency.
- Experimental Setup: COLD uses off-the-shelf language models without fine-tuning and is compared primarily with alternative decoding methods across three constrained generation tasks.The tasks are abductive reasoning, counterfactual story rewriting, and lexically constrained generation, using GPT2-XL as the base language model.
- Abductive Reasoning: For abductive reasoning, COLD combines left-context fluency, right-context coherence, future-token prediction, and keyword similarity constraints in one energy function.The method targets a bridge sentence that must fit both the beginning and ending of a story.
- Abductive Reasoning: COLD consistently outperforms DELOREAN and LEFT-ONLY on automatic abductive-reasoning metrics, including BLEU, ROUGE, CIDEr, and BERTScore.Human evaluation also gives COLD the best overall coherence and high grammaticality, whereas DELOREAN has weaker right-side coherence and LEFT-ONLY ignores the right context.
- Counterfactual Story Rewriting: In counterfactual story rewriting, COLD achieves substantially improved human-evaluated coherence and comparable minimal-edit performance relative to DELOREAN.Its constraints promote coherence with the new context while preserving similarity to the original ending; LEFT-ONLY is coherent but fails minimal-edit.
- Lexically Constrained Generation: For lexical constraints, COLD is comparable to NEUROLOGIC, achieves higher keyword coverage at slightly lower fluency, and substantially outperforms TSMH on coverage and fluency.Keyword coverage is measured by the average number and percentage of constraint words present, while fluency uses perplexity and human judgment.
- Additional Analysis: Removing constraints improves grammaticality but reduces coherence, while gradient-based COLD sampling is 30% faster than discrete Mix-and-Match with similarly sized base language models.The ablation indicates that all constraints together achieve the best overall coherence, and constraint weights provide a way to balance competing objectives.
5 Related Work
Prior constrained decoding methods enforce constraints discretely or adjust vocabulary distributions, while energy-based approaches have also been used for text generation. COLD instead targets constrained decoding directly and applies Langevin dynamics to pretrained language models without fine-tuning.
- The related efficiency comparison reports runtime in seconds per sample for COLD and gradient-free Mix-and-Match on Counterfactual Story Rewriting.
- Beam-search variants enforce lexical constraints in discrete space, while other methods adjust vocabulary distributions with an additional discriminator or language model.
- Earlier energy-based methods incorporate additional information during training, whereas COLD focuses on inference-time constrained decoding.COLD is designed to work directly with pretrained language models without task-specific fine-tuning.
6 Conclusion
COLD is an energy-based constrained decoding framework that represents soft and hard constraints with an energy function and samples using Langevin dynamics. It applies directly to off-the-shelf language models and performs strongly across three constrained-generation applications.
- COLD expresses soft and hard constraints through an energy function and samples from it using Langevin dynamics.
- The framework applies directly to off-the-shelf language models without task-specific fine-tuning.
- COLD is evaluated on three distinct constrained-generation applications.
Checklist
The checklist reports documentation of the paper’s claims, limitations, experiments, resources, assets, and ethical considerations. It also records that stable confidence intervals were not obtained because computational resources limited repeated evaluation.
- The paper reports that its main claims accurately reflect its contributions and that the work’s limitations and scope are described.
- The paper discusses potential negative societal impacts, including malicious uses of generated toxic, biased, offensive, or fake information.
- The experiments provide reproducibility materials and specify training details, while compute-resource information is available in the appendix.
- The paper reports no stable confidence intervals because computational constraints prevented enough cross-validation splits or random seeds.
- The paper states that it uses existing assets without introducing new datasets and refers readers to original releases for license information.
- The experiments use public data, and the paper states that the corpora do not contain personally identifying information to its knowledge and experience.
B Experimental Configurations
The experiments select constraint weights on development data and use task-specific decoding configurations. These include fixed Langevin dynamics settings, top-k filtering, sample counts, and task-dependent sequence lengths.
- Abductive Reasoning: Abductive reasoning uses a total fluency-constraint weight of 0.5, with language-model constraints balanced at a 6:4 ratio.
- Abductive Reasoning: Abductive reasoning uses N = 2000 Langevin steps, step size η = 0.1, decoded length 10, k = 2 filtering, and 16 samples per example.
- Counterfactual Story Rewriting: Counterfactual story rewriting sets λlr_a = 0.16, uses length 20, k = 5 filtering, and generates 32 samples per test example.
- Lexically Constrained Decoding: Lexically constrained decoding increases the n-gram similarity ratio to 1:0.1 and uses k = 5 top-k filtering.
- Language Models: The right-to-left language model was trained following GPT-2 on the OpenWebText corpus and publicly released by West et al.
- Computing: All experiments ran on a server with 8 NVIDIA V100 GPUs.
C Human Evaluation Details
Human evaluation covered lexically constrained generation, abductive reasoning, and counterfactual reasoning using randomly sampled prompts and shuffled HITs.
- Human evaluation was conducted for lexically constrained generation, abductive reasoning, and counterfactual reasoning.
- For each task, evaluators judged 200 randomly sampled prompts from the corpus.
- HITs were shuffled to reduce systematic bias from rater availability over time.
C.2 Human Evaluation Payment
Mean hourly payment for human evaluation was set at $15 per hour using a JavaScript timing tool.
- Mean hourly pay was determined using a JavaScript timing tool to be $15 per hour.
D Ablation Study: Top-k Filtering
The top-k filtering ablation examines how k changes the balance between fluency and constraint satisfaction. Larger k improves right and overall coherence initially, but excessive flexibility can reduce fluency and limit further gains.
- Table 6 reports an ablation of the top-k filtering parameter k under the same setting as Table 5.
- As k increases, grammar score and left coherence tend to decrease because broader candidate flexibility can weaken fluency.
- Right and overall coherence generally improve with larger k, but stop improving at k values of 50 or 100 as inferior fluency harms coherence.
E Generated Samples
The paper provides generated examples for all three constrained generation tasks: abductive reasoning, counterfactual reasoning, and lexical constraint satisfaction.
- Generated samples for abductive reasoning are presented in Table 7.
- Generated samples for counterfactual reasoning are presented in Table 8.
- Generated samples for lexically constrained generation are presented in Table 9.