Source-linked AI summary
Plug and Play Language Models: A Simple Approach to Controlled Text Generation
Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, Rosanne Liu
TL;DR
Controlling attributes in pretrained language models ordinarily requires architectural changes or attribute-specific fine-tuning. PPLM instead steers a pretrained LM during sampling with lightweight attribute models, and evaluations report controlled, fluent text across topics and sentiment while supporting applications such as detoxification. The paper also notes that controllability varies across topics and that the mechanism can be misused to create more toxic language.
Problem
Controlling generated text attributes is difficult after training without modifying the model architecture or fine-tuning on attribute-specific data.
Method
PPLM combines a pretrained language model with bag-of-words or small discriminator controllers whose gradients steer hidden activations during sampling without LM retraining.
Results
PPLM generates attribute-controlled, fluent text across topics and sentiment, performing on par with or often outperforming CTRL and GPT-2 fine-tuned for positivity on attribute relevance and fluency.
Takeaways & Limitations
PPLM provides fine-grained, flexible control by combining a large pretrained LM with lightweight attribute models during generation.
Takeaways & Limitations
Controllability varies substantially across topics, with religion, science, and politics easier to control than computers and space; the mechanism can also be exploited to create more toxic language.
Abstract
from arXiv · showhide
Large transformer-based language models (LMs) trained on huge text corpora have shown unparalleled generation capabilities. However, controlling attributes of the generated language (e.g. switching topic or sentiment) is difficult without modifying the model architecture or fine-tuning on attribute-specific data and entailing the significant cost of retraining. We propose a simple alternative: the Plug and Play Language Model (PPLM) for controllable language generation, which combines a pretrained LM with one or more simple attribute classifiers that guide text generation without any further training of the LM. In the canonical scenario we present, the attribute models are simple classifiers consisting of a user-specified bag of words or a single learned layer with 100,000 times fewer parameters than the LM. Sampling entails a forward and backward pass in which gradients from the attribute model push the LM's hidden activations and thus guide the generation. Model samples demonstrate control over a range of topics and sentiment styles, and extensive automated and human annotated evaluations show attribute alignment and fluency. PPLMs are flexible in that any combination of differentiable attribute models may be used to steer text generation, which will allow for diverse and creative applications beyond the examples given in this paper.
1 INTRODUCTION
PPLM enables controlled language generation by steering a pretrained language model with lightweight attribute models, avoiding model retraining or fine-tuning. Evaluations report controlled, fluent text across topics and sentiment, with applications including detoxification and structured story writing.
- Method: Activation-space optimization guides generation ex post facto, so the language model requires no retraining or fine-tuning.Attribute influence is adjusted during sampling through controller strength, allowing the original model to be recovered at strength 0.
- Method: PPLM introduces plug-and-play controlled generation by combining a pretrained language model with user-specified or learned attribute controllers.The approach supports bag-of-words topic controllers and a learned sentiment discriminator without changing the LM parameters.
- Experiments: PPLM controls 7 topics defined by bags of words and 1 sentiment attribute defined by a discriminator, evaluated with automated and human measures.The evaluations assess perplexity, sentiment, attribute relevance, and fluency.
- Results: PPLM is reported to perform on par with or often outperform CTRL and GPT-2 fine-tuned for positivity on attribute relevance and fluency.This comparison is made without LM training.
- Applications: The method is demonstrated for detoxifying likely toxic generations and for structurally constrained story writing.Detoxification follows the negative gradient of a toxicity detector.
- Related work: Unlike approaches that separately fine-tune an entire model for each attribute, PPLM combines small custom attribute models with a general pretrained language model.This plug-and-play design distinguishes it from several existing controlled-generation methods.
3 PLUG AND PLAY LANGUAGE MODELS
PPLM steers a pretrained transformer during generation by updating cached hidden representations with attribute and language-model gradients, then sampling from the resulting distribution. Repeated latent updates guide text toward the desired attribute while fluency-preserving mechanisms keep generation near the unconditional model.
- Steering generation: At each time step, PPLM shifts the history H_t using gradients that increase attribute likelihood and preserve language-model likelihood.The update is restricted to H_t because future predictions depend on the past through this cached history.
- Steering generation: Generation proceeds through an attribute forward pass, latent-space gradient update, and sampling from a distribution produced by the updated latents.The three-phase cycle repeats at each time step, producing a gradual transition toward the desired attribute.
- Steering generation: 3 to 10 gradient updates are used in practice before a forward pass with updated key-value pairs produces the next-token distribution.The update uses step size α and normalization coefficient γ.
- Ensuring fluency: Unchecked attribute optimization can produce unrealistic low-probability text, so PPLM adds mechanisms that keep fluency near the unconditional language model.The method minimizes KL divergence between modified and unmodified output distributions and also applies post-norm fusion.
- Sampling and ranking: The attribute model both ranks samples by attribute likelihood and supplies the gradient direction for latent updates.PPLM can additionally discard repetitive samples using a threshold on mean Dist-1, Dist-2, and Dist-3 scores.
4 EXPERIMENTS, RESULTS, AND EVALUATION
PPLM is evaluated for topic, sentiment, and toxicity control using automated and human measures of attribute alignment, fluency, and diversity. Across these settings, latent manipulation generally improves control while preserving fluency, though controllability varies substantially by topic.
- Evaluation methodology: PPLM evaluations measure target-attribute satisfaction and whether stronger control degrades text quality, using automated and human assessments.Automated measures include perplexity and distinct n-gram diversity; human measures include fluency and attribute relevance.
- Evaluation methodology: The ablation compares an unchanged GPT-2 baseline, repeated sampling with ranking, latent updating, and latent updating combined with ranking.These variants isolate the effects of sampling, latent manipulation, and their combination.
- BoW attribute models: 51.7% of BCR samples were judged on topic versus 15.8% for B, while BCR retained similar fluency and diversity with minimal perplexity degradation.BCR also slightly outperformed CTRL at 51.7% versus 50.0% and significantly outperformed WD at 36%.
- BoW attribute models: Latent updates exerted greater influence on topic relevance than output-distribution reranking, indicating that shifting latent meaning was more effective than direct reweighting.The comparison includes variants with and without ranking and latent manipulation, while WD performed relatively worse.
- BoW attribute models: Controllability varies widely across topics: religion, science, and politics are easier to control than computers and space.The paper also reports successful control for unusual prefix–attribute combinations.
- Discriminator attribute models: 73.7% sentiment accuracy for BCR substantially exceeded the baseline's 19.3% while retaining fluency similar to B and BR.BCR also performed comparably to CTRL at 73.7% versus 80.0%, and outperformed GPT2-FT-RL and WD in the reported comparisons.
- Language detoxification: A toxicity classifier reduced toxicity to 4.6% on average for adversarial triggers and produced 6%, 4%, and 10% toxicity rates for natural prompts.The paper notes that the same detoxification capability could potentially be misused to generate toxic language.
5 CONCLUSION
PPLM combines a large pretrained language model with bag-of-words or small discriminator controllers and uses gradient-based sampling for fine-grained attribute control while maintaining fluency.
- PPLM flexibly combines a large pretrained LM with a BoW or small discriminator controller.
- Gradient-based sampling provides fine-grained control over generated attributes.
- PPLM can control generation while maintaining fluency.
S6 ETHICS OF CONTROLLED LANGUAGE MODELS
The section discusses ethical risks of controlled language models, including bias, disinformation, and the possibility that PPLM’s detoxification mechanism could also generate more toxic language.
- Controlled language models may recapitulate problematic social biases or be abused to generate disinformation.
- PPLM can detoxify likely-toxic generations by using a toxicity detector’s negative gradient.
- The same mechanism could instead be exploited to create more toxic language.
S8 FURTHER DETAILS ON HUMAN AND AUTOMATED EVALUATION
The evaluation combines human and automated measures of attribute relevance and fluency across topic, sentiment, and style controls, with detailed results reported for PPLM and baselines.
- Topic relevance is evaluated entirely by humans, while sentiment relevance uses human annotation and a separately trained sentiment classifier.
- The PPLM-BoW evaluation includes 7560 ablation annotations and 1560 baseline-comparison annotations.
- The PPLM-Discrim evaluation includes 1620 sentiment-ablation annotations, 495 sentiment-baseline annotations, and 810 clickbait annotations.
- Fluency-score distributions are similar across B, BC, BR, and BCR methods for both BoW and discriminator evaluations.
- Figure S3 shows that taking a PPLM gradient step makes a large difference in human-rated topic relevance, while reranking is mostly helpful.
S9 ODD COMBINATION OF TOPICS AND PREFIXES
PPLM is tested on deliberately mismatched topic-prefix and sentiment-prefix combinations, where control remains effective and generated examples are reported as fluent and sensible.
- The study tests odd pairings such as ordinary prefixes controlled toward military, legal, computers, politics, or religion topics.
- The evaluation summaries include human and automated measures for PPLM-BoW and PPLM-Discrim, with Figure S4 visualizing discriminator relevance.
- It also tests positive control on negative-sentiment prefixes and negative control on a positive-sentiment prefix.
- PPLM control remains easy in these scenarios, but the attribute strength α is increased from 0.01 to 0.02 or 0.03.
S10 FINE-GRAINED CONTROL WITH PPLM-BOW
Fine-grained PPLM-BoW control depends on tuning latent-update behavior and the update window. Large step sizes can cause repetition, while early stopping and recent-token updates improve control quality or fluency.
- Strength factors: Table S17 compares unconditioned GPT-2 with PPLM-BoW generations using the same word bags but different strength factors.
- Experimental settings: The experiments use task-specific hyperparameters, including selecting three highest-scoring samples from a batch of r = 10 for PPLM-BoW.
- Early stopping of latent updates: Large α can make PPLM-BoW repeat targeted keywords, producing degeneration in generated text.Reducing α or stopping latent updates early mitigates this behavior.
- Early stopping of latent updates: Stopping latent updates after 20 time steps produces samples with much less degeneration under the same setting.
- Latent-update window: Restricting updates to the most recent w tokens reduces computation, and w = 5 produces more fluent PPLM-BoW passages.Each hidden-state entry is modified at most w times rather than updating the entire past.
- Gradient normalization: Gradient normalization for bag-of-words control uses the maximum gradient norm over time, making updates smaller when a target word is less likely to appear.
S12 ADDITIONAL DETAILS AND SAMPLES OF CONTROLLED STORY TELLING
Additional examples extend controlled storytelling to multiple attribute models and language detoxification. The examples combine different controllers and compare toxicity before and after control under natural and adversarial prompts.
- Multiple attribute models: Multiple knobs combine three to four attribute models, including PPLM-BoW and PPLM-Discrim, across different controller types.
- Language detoxification: Toxicity evaluation compares original and toxicity-controlled GPT-2 outputs under natural prompts and adversarial triggers using binary volunteer annotations.
S14 MORE EXAMPLES
The supplementary examples cover additional controlled generations, prefix variation, skeleton storytelling, mixed attribute control, and fluency distributions for BoW and discriminator controllers.
- Additional examples: The paper provides additional PPLM-controlled generation examples in Tables S24–S27.
- Skeleton storytelling: Tables S20 and S21 present skeleton stories generated with fixed skeletons and one or multiple attribute models, with controlled keywords highlighted.
- Mixed attribute control: BoW and discriminator controllers can be mixed to steer generation toward combined topics and sentiment, such as WINTER, POLITICS, KITCHEN, and POSITIVE.
- Ablation fluency: Fluency-score histograms report similarly distributed scores across the four ablation approaches for both PPLM-BoW and PPLM-Discrim.
S17 WORD LISTS FOR BAG OF WORDS APPROACHES
The supplementary materials document word-list resources and topic vocabularies used for PPLM-BoW, alongside examples of controlled outputs and toxicity evaluation.
- Word-list construction: PPLM-BoW word lists are curated from EnchantedLearning word lists.
- Topic vocabularies: The listed topic vocabularies include Science, Fantasy/Magic, Space, Politics, Military, Religion, and Computers.
- Religion vocabulary: The Religion vocabulary includes terms associated with religious institutions, beliefs, figures, practices, and concepts.
- Computers vocabulary: The Computers vocabulary contains terms spanning software, hardware, networking, files, and online services.
- Multiple-knob examples: Table S22 provides examples of attribute-controlled text generation with multiple knobs and includes a clickbait discriminator trained on Potthast et al. (2018).
- Toxicity evaluation: Table S23 reports toxicity counts, sample totals, percentages, and p-values comparing Before and After detoxification.