Source-linked AI summary
FUDGE: Controlled Text Generation With Future Discriminators
Kevin Yang, Dan Klein
TL;DR
Controlled text generation needs to impose new attributes on an existing generator while preserving its original distribution. FUDGE uses a future-attribute predictor with the generator’s output logits, and shows strong performance across poetry, topic control, and formality translation while supporting modular attribute composition.
Problem
Controlled generation seeks P(X|a) for attributes not already built into an existing P(X), without sacrificing the original distribution.
Method
FUDGE learns a predictor of future attribute satisfaction from prefixes and combines its probabilities with the generator’s output probabilities through Bayesian factorization.
Results
FUDGE performs strongly across poetry couplet completion, topic control, and informal-to-formal machine translation, outperforming direct fine-tuning and PPLM.
Takeaways & Limitations
FUDGE supports modular composition of multiple attributes and can be applied to controlled generation tasks with trainable attribute discriminators.
Takeaways & Limitations
FUDGE cannot fully guarantee the desired attribute because future-attribute modeling is approximate and decoding considers only the top 200 token candidates.
Abstract
from arXiv · showhide
We propose Future Discriminators for Generation (FUDGE), a flexible and modular method for controlled text generation. Given a pre-existing model G for generating text from a distribution of interest, FUDGE enables conditioning on a desired attribute a (for example, formality) while requiring access only to G's output logits. FUDGE learns an attribute predictor operating on a partial sequence, and uses this predictor's outputs to adjust G's original probabilities. We show that FUDGE models terms corresponding to a Bayesian decomposition of the conditional distribution of G given attribute a. Moreover, FUDGE can easily compose predictors for multiple desired attributes. We evaluate FUDGE on three tasks -- couplet completion in poetry, topic control in language generation, and formality change in machine translation -- and observe gains in all three tasks.
1 Introduction
Controlled text generation seeks to add a desired attribute to an existing distribution without rebuilding the generator or sacrificing its original behavior. FUDGE addresses this by conditioning through the generator’s output probabilities and demonstrates broad effectiveness across three tasks.
- Controlled generation models text conditioned on an additional attribute a that is not already built into P(X).
- Adding a new attribute to an existing P(X) is difficult without retraining or fine-tuning, while rejection sampling can be inefficient.
- FUDGE accesses only G’s output probabilities, predicts whether a partial sequence will eventually satisfy attribute a, and combines those predictions with G’s probabilities.
- FUDGE evaluates couplet completion, topic control, and formality change in machine translation, and outperforms direct fine-tuning and PPLM.
2 Related Work
Prior approaches control attributes by fine-tuning models, modifying generation through gradients, or directly reweighting output probabilities. FUDGE distinguishes itself among weighted-decoding methods by predicting future attribute satisfaction and reports stronger performance than previous weighted-decoding approaches.
- Fine-tuning can make it difficult to separate the desired attribute from distribution shift and may require separate training for each attribute.
- Gradient-based methods modify model activations or trigger strings and can achieve strong attribute control, including uses in adversarial attacks.
- Weighted-decoding methods operate directly on G’s output logits and are relatively interpretable, but prior implementations have performed poorly in controlled generation.
- FUDGE follows the Bayesian factorization more closely than prior weighted-decoding methods by modeling whether the attribute will hold in the future.
- FUDGE substantially outperforms previous weighted-decoding approaches in the reported experiments.
3 Future Discriminators for Generation
FUDGE conditions an autoregressive generator by modeling future attribute satisfaction and combining that predictor with the generator’s next-token probabilities. Its modular design supports model swapping and multiple attributes, but approximation and top-200 pruning prevent a full guarantee of attribute satisfaction.
- 3 Future Discriminators for Generation: FUDGE models the desired conditional distribution P(X|a) by replacing direct conditional modeling with a Bayesian factorization.
- 3 Future Discriminators for Generation: The base generator supplies P(x_i|x_1:i−1), while a binary classifier models P(a|x_1:i) from a sequence prefix.
- 3 Future Discriminators for Generation: Unlike methods that optimize the immediate next extension, FUDGE predicts whether the completed future sequence will satisfy attribute a.
- 3 Future Discriminators for Generation: At each decoding step, FUDGE multiplies the predictor’s probabilities with G’s probabilities and renormalizes over candidate tokens to obtain the conditioned distribution.
- 3 Future Discriminators for Generation: For efficiency, FUDGE uses a lightweight predictor and considers only G’s top 200 token possibilities at each step.
- 3.1 Advantages and Limitations: FUDGE requires only G’s output logits, permits swapping in another model with shared tokenization, and composes conditionally independent attributes by summing log-probabilities.
- 3.1 Advantages and Limitations: FUDGE cannot fully guarantee the desired attribute because modeling P(a|x_1:i) is approximate and decoding prunes candidates to the top 200.
4 Experiments
FUDGE is evaluated across poetry couplet completion, topic-controlled generation, and machine-translation formality change. Across these tasks, it controls attributes while preserving useful generation quality and outperforming relevant baselines.
- Experimental scope: Experiments cover poetry couplet completion, topic-controlled language generation, and formality change in machine translation.The evaluation includes distinct controlled-generation settings with task-specific setups and baselines.
- Poetry couplet completion: FUDGE combines separate predictors for meter, rhyme, and sentence-ending in couplet completion, reflecting an assumed conditional independence among these attributes.Each predictor evaluates whether the completed sequence will satisfy its corresponding property.
- Poetry couplet completion: FUDGE substantially outperforms automated baselines on couplet success, even with training data from a different domain containing essentially no correct couplets.It also maintains high diversity, although quality is lower than the base model under the difficult constraint.
- Topic control: FUDGE achieves the highest topic-control success by a substantial margin and outperforms baselines in human-rated topic relevance and fluency.It preserves high quality and diversity, whereas FINETUNE and WDEC often repeat the supplied topic words.
- Efficiency and access: FUDGE outperforms PPLM using only G’s output logits and runs the full set of 420 generations in 15 minutes versus a few hours for PPLM.Its non-reliance on gradients also supports use when gradients are unavailable through an API.
- Formality change in machine translation: FUDGE increases formality in translation while retaining stronger BLEU than the style-transfer baseline and improving BLEU over unfine-tuned G.The formality predictor is trained on Entertainment/Music but tested on Family/Relationships, and substantial room for improvement remains.
5 Discussion
FUDGE provides a principled approach to controlled text generation that follows a Bayesian factorization while preserving the base distribution as much as possible. It performs strongly across poetry, topic control, and informal-to-formal machine translation, while supporting modular attribute composition.
- FUDGE models P(X|a) through a Bayesian factorization designed to preserve the base P(X) as much as possible.The approach is described as principled controlled text generation.
- FUDGE achieves strong performance on poetry couplet completion, topic control, and informal-to-formal machine translation.
- FUDGE composes multiple attributes modularly, including meter, rhyme, and end-of-sentence constraints for couplet completion.
6 Ethics of Controlled Text Generation
The paper notes that controlled generation can be misused for harmful outputs or misinformation, while also supporting bias mitigation such as detoxifying language. Its couplet evaluation uses strict, hand-designed well-formedness checks.
- Controlled generation methods can produce harmful outputs or misinformation when used adversarially.
- Controlled generation can also help mitigate harmful biases learned by large pretrained language models, including through detoxifying language.
- The couplet task uses F to check iambic pentameter, rhyme, and sentence-ending constraints.
- Iambic pentameter is treated as iambic meter plus exactly ten syllables.
- FUDGE and FINETUNE fit whatever output F provides, so the couplet task tests their ability to satisfy a difficult well-formedness constraint.
A.1 Shakespeare Evaluation
The Shakespeare reference is evaluated against narrow automated definitions that do not accommodate several acceptable poetic deviations. Dictionary coverage and pronunciation differences therefore affect the metrics.
- Shakespeare performs somewhat poorly under the paper’s automated formality and evaluation metrics designed for the baselines.
- Out-of-vocabulary words appear in a fifth of Shakespeare’s lines and cause failures on the iambic meter and syllable checks.
- Shakespeare often elides syllables and uses loose rhymes, deviations that the strict automatic metric does not allow.
- Differences between archaic and modern pronunciations likely exacerbate these metric errors.
- Table 10 illustrates both a syllable-count deviation and a loose non-perfect rhyme under the paper’s narrow correctness definition.
B PPLM Baseline in Machine Translation
The PPLM comparison requires backtranslation to obtain training states in the Spanish-English translation setting. PPLM makes some useful formality changes but is less formal than FUDGE despite similar BLEU.
- PPLM learns P(a|X) from an English formality dataset without parallel Spanish, requiring English backtranslation to obtain training hidden states.
- PPLM makes reasonable formality modifications such as changing “hard” to “difficult,” but also produces occasional disfluencies or repetitions.
- PPLM achieves similar BLEU to FUDGE but is substantially less formal.
- Topic-control FUDGE uses λ to control conditioning strength and the number of future bag-of-words targets.
- The topic-control validation selected λ = 4 from values ranging from 1 to 6, although the result may be somewhat noisy.
- WDEC selected 4 as its only reasonable conditioning strength without total diversity collapse, but diversity still collapsed on the seven main test bags.
D Ablations on Predictor Architectures
FUDGE uses task-specific predictor architectures when tasks require different auxiliary inputs or families of related predictors. Ablations with more homogenized predictors show relatively small differences on couplet completion and topic control.
- Architecture choices: Task-specific architectures are necessary because the controlled-generation tasks differ and PPLM is difficult to adapt.The core architecture uses word embeddings, an LSTM, and an output layer, but predictor variants address task diversity.
- Architecture choices: Some predictors are families of related predictors represented as one predictor with additional inputs such as rhyme sound.This design is used in the poetry and topic tasks.
- Ablation: Modified predictors embed additional inputs and concatenate them to each input word embedding in the ablation.FUDGEMOD denotes this ablated version of FUDGE.
- Ablation: The difference between the original and homogenized predictors is relatively small on both couplet completion and topic control.The comparison is reported in Tables 14 and 15.
E Alternative Perplexity Measurements
The appendix reports alternative perplexity measurements and related validation or ablation tables. Perplexity rankings are generally stable across GPT and Transformer-XL, with some exceptions.
- Alternative measurements: Perplexity is measured with Transformer-XL for couplet completion and topic control, alongside GPT-based measurements.Couplet completion also uses a GPT model fine-tuned on Shakespearean language.
- Alternative measurements: Relative perplexities among most models remain largely similar when switching between GPT and Transformer-XL, with a few exceptions.Compared with base GPT, Shakespeare’s perplexity naturally decreases under the Shakespeare-fine-tuned model.
- Validation tables: Validation tables report FUDGE and WDEC results under different conditioning strengths for a fantasy-topic bag of words.During validation, success directly measures use of words in the given bag rather than a heldout bag.
- Ablations: Predictor-architecture ablations for couplet completion and topic control are reported in Tables 14 and 15.These tables correspond to the modified predictor architecture described for FUDGEMOD.
- Alternative measurements: Tables 16 and 17 compare perplexity measurements using GPT and Transformer-XL for couplet completion and topic control.Table 16 additionally includes GPT-Shakespeare; the main-paper results use GPT.
F Statistical Significance
FUDGE’s comparisons against automated baselines are statistically significant across couplet completion, topic control, and translation formality. Additional experiments show stronger conditioning improves control while increasing perplexity.
- Statistical significance: p < 0.0001: FUDGE outperforms FINETUNE on couplet-completion success rate under a paired McNemar test.Generations are paired for each Shakespeare prefix.
- Statistical significance: p = 0.04: FUDGE outperforms PPLM in topic control under a paired Wilcoxon matched-pairs test.Generations are paired for topic-prefix combinations.
- Statistical significance: p < 0.0001: FUDGE generations are more formal than base-G generations in translation under a paired t-test.The comparison concerns translation formality.
- Conditioning strength: As λ increases from 2 to 8, conditioning becomes stronger while perplexity also increases in topic control.The main topic-control experiments use λ = 4.
- Conditioning strength: λ = 8 generations remain mostly fluent and interesting despite worse grammaticality and perplexity.The examples use the same prompts and topics as the λ = 4 main-text examples.
- Computational setting: FUDGE’s predictor receives only the top 200 candidates from G for each next-token prediction, with ablations testing 100 and 400 candidates.This candidate restriction is introduced for computational efficiency.
J Additional Topic Control Examples
Additional topic-control examples show that FUDGE generally stays on topic with λ = 4, while the appendix also documents conditioning-strength trade-offs, candidate-count settings, and baseline examples.
- FUDGE examples: Virtually all λ = 4 FUDGE examples across seven topics are clearly on topic while avoiding repetitiveness.The examples use three prefixes and three separate generations per topic setting.
- Failure cases: FUDGE sometimes fails because of rhyme or ten-syllable constraints, missing pronunciation-dictionary entries, or excess sentence-ending punctuation.These errors are reported for couplet completion rather than topic control.
- FUDGE examples: At λ = 4, the space example is somewhat tangential for “More importantly,” while the other six topic generations are on topic.The table selects the first generation for each prefix.
- FUDGE examples: For “It has been shown,” the space example seems unrelated and the military example is somewhat tangential, while the other five are on topic.These are first generations selected for each topic.
- FUDGE examples: For “To review,” the politics example is somewhat tangential, while the other six topic generations are on topic.The table reports the first generation selected for each prefix.
- Baseline comparisons: Baseline examples show G ignores conditioning information, FINETUNE is often repetitive and off topic, and WDEC frequently repeats topic-bag words.These descriptions accompany Tables 27–29.