PaperlayerView exact revision on arXiv

Source-linked AI summary

Plug and Play Language Models: A Simple Approach to Controlled Text Generation

Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, Rosanne Liu

arXiv:1912.02164v4cs.CLcs.AIcs.LG

TL;DR

Controlling attributes in pretrained language models ordinarily requires architectural changes or attribute-specific fine-tuning. PPLM instead steers a pretrained LM during sampling with lightweight attribute models, and evaluations report controlled, fluent text across topics and sentiment while supporting applications such as detoxification. The paper also notes that controllability varies across topics and that the mechanism can be misused to create more toxic language.

  • Problem

    Controlling generated text attributes is difficult after training without modifying the model architecture or fine-tuning on attribute-specific data.

    Source: §1 · p. 1

  • Method

    PPLM combines a pretrained language model with bag-of-words or small discriminator controllers whose gradients steer hidden activations during sampling without LM retraining.

    Source: §1 · p. 2

  • Results

    PPLM generates attribute-controlled, fluent text across topics and sentiment, performing on par with or often outperforming CTRL and GPT-2 fine-tuned for positivity on attribute relevance and fluency.

    Source: §1 · p. 3

  • Takeaways & Limitations

    PPLM provides fine-grained, flexible control by combining a large pretrained LM with lightweight attribute models during generation.

    Source: §1 · p. 2, §5 · p. 10

    Controllability varies substantially across topics, with religion, science, and politics easier to control than computers and space; the mechanism can also be exploited to create more toxic language.

    Source: §4.2 · p. 8, S6 ETHICS OF CONTROLLED LANGUAGE MODELS · p. 15

Abstract

from arXiv · show

Large transformer-based language models (LMs) trained on huge text corpora have shown unparalleled generation capabilities. However, controlling attributes of the generated language (e.g. switching topic or sentiment) is difficult without modifying the model architecture or fine-tuning on attribute-specific data and entailing the significant cost of retraining. We propose a simple alternative: the Plug and Play Language Model (PPLM) for controllable language generation, which combines a pretrained LM with one or more simple attribute classifiers that guide text generation without any further training of the LM. In the canonical scenario we present, the attribute models are simple classifiers consisting of a user-specified bag of words or a single learned layer with 100,000 times fewer parameters than the LM. Sampling entails a forward and backward pass in which gradients from the attribute model push the LM's hidden activations and thus guide the generation. Model samples demonstrate control over a range of topics and sentiment styles, and extensive automated and human annotated evaluations show attribute alignment and fluency. PPLMs are flexible in that any combination of differentiable attribute models may be used to steer text generation, which will allow for diverse and creative applications beyond the examples given in this paper.

1 INTRODUCTION

PPLM enables controlled language generation by steering a pretrained language model with lightweight attribute models, avoiding model retraining or fine-tuning. Evaluations report controlled, fluent text across topics and sentiment, with applications including detoxification and structured story writing.

Method

  • Activation-space optimization guides generation ex post facto, so the language model requires no retraining or fine-tuning.Attribute influence is adjusted during sampling through controller strength, allowing the original model to be recovered at strength 0.Source: §1 · p. 2
  • PPLM introduces plug-and-play controlled generation by combining a pretrained language model with user-specified or learned attribute controllers.The approach supports bag-of-words topic controllers and a learned sentiment discriminator without changing the LM parameters.Source: §1 · p. 2, Table 1 · p. 2

Experiments

  • PPLM controls 7 topics defined by bags of words and 1 sentiment attribute defined by a discriminator, evaluated with automated and human measures.The evaluations assess perplexity, sentiment, attribute relevance, and fluency.Source: §1 · pp. 2–3

Results

  • PPLM is reported to perform on par with or often outperform CTRL and GPT-2 fine-tuned for positivity on attribute relevance and fluency.This comparison is made without LM training.Source: §1 · p. 3

Applications

  • The method is demonstrated for detoxifying likely toxic generations and for structurally constrained story writing.Detoxification follows the negative gradient of a toxicity detector.Source: §1 · p. 3

Related work

  • Unlike approaches that separately fine-tune an entire model for each attribute, PPLM combines small custom attribute models with a general pretrained language model.This plug-and-play design distinguishes it from several existing controlled-generation methods.Source: §2 · p. 3, Table 2 · p. 4

3 PLUG AND PLAY LANGUAGE MODELS

PPLM steers a pretrained transformer during generation by updating cached hidden representations with attribute and language-model gradients, then sampling from the resulting distribution. Repeated latent updates guide text toward the desired attribute while fluency-preserving mechanisms keep generation near the unconditional model.

Steering generation

  • At each time step, PPLM shifts the history H_t using gradients that increase attribute likelihood and preserve language-model likelihood.The update is restricted to H_t because future predictions depend on the past through this cached history.Source: §3.2 · p. 4
  • Generation proceeds through an attribute forward pass, latent-space gradient update, and sampling from a distribution produced by the updated latents.The three-phase cycle repeats at each time step, producing a gradual transition toward the desired attribute.Source: Figure 1 · p. 5, §3.2 · p. 5
  • 3 to 10 gradient updates are used in practice before a forward pass with updated key-value pairs produces the next-token distribution.The update uses step size α and normalization coefficient γ.Source: §3.2 · p. 5

Ensuring fluency

  • Unchecked attribute optimization can produce unrealistic low-probability text, so PPLM adds mechanisms that keep fluency near the unconditional language model.The method minimizes KL divergence between modified and unmodified output distributions and also applies post-norm fusion.Source: §3.3 · p. 5

Sampling and ranking

  • The attribute model both ranks samples by attribute likelihood and supplies the gradient direction for latent updates.PPLM can additionally discard repetitive samples using a threshold on mean Dist-1, Dist-2, and Dist-3 scores.Source: §3.4 · p. 6

4 EXPERIMENTS, RESULTS, AND EVALUATION

PPLM is evaluated for topic, sentiment, and toxicity control using automated and human measures of attribute alignment, fluency, and diversity. Across these settings, latent manipulation generally improves control while preserving fluency, though controllability varies substantially by topic.

Evaluation methodology

  • PPLM evaluations measure target-attribute satisfaction and whether stronger control degrades text quality, using automated and human assessments.Automated measures include perplexity and distinct n-gram diversity; human measures include fluency and attribute relevance.Source: §4.1 · p. 6
  • The ablation compares an unchanged GPT-2 baseline, repeated sampling with ranking, latent updating, and latent updating combined with ranking.These variants isolate the effects of sampling, latent manipulation, and their combination.Source: §4.1 · p. 7

BoW attribute models

  • 51.7% of BCR samples were judged on topic versus 15.8% for B, while BCR retained similar fluency and diversity with minimal perplexity degradation.BCR also slightly outperformed CTRL at 51.7% versus 50.0% and significantly outperformed WD at 36%.Source: §4.2 · p. 8
  • Latent updates exerted greater influence on topic relevance than output-distribution reranking, indicating that shifting latent meaning was more effective than direct reweighting.The comparison includes variants with and without ranking and latent manipulation, while WD performed relatively worse.Source: §4.2 · p. 8
  • Controllability varies widely across topics: religion, science, and politics are easier to control than computers and space.The paper also reports successful control for unusual prefix–attribute combinations.Source: §4.2 · p. 8

Discriminator attribute models

  • 73.7% sentiment accuracy for BCR substantially exceeded the baseline's 19.3% while retaining fluency similar to B and BR.BCR also performed comparably to CTRL at 73.7% versus 80.0%, and outperformed GPT2-FT-RL and WD in the reported comparisons.Source: §4.3 · p. 9

Language detoxification

  • A toxicity classifier reduced toxicity to 4.6% on average for adversarial triggers and produced 6%, 4%, and 10% toxicity rates for natural prompts.The paper notes that the same detoxification capability could potentially be misused to generate toxic language.Source: §4.4 · p. 10

5 CONCLUSION

PPLM combines a large pretrained language model with bag-of-words or small discriminator controllers and uses gradient-based sampling for fine-grained attribute control while maintaining fluency.

  • PPLM flexibly combines a large pretrained LM with a BoW or small discriminator controller.Source: §5 · p. 10
  • Gradient-based sampling provides fine-grained control over generated attributes.Source: §5 · p. 10
  • PPLM can control generation while maintaining fluency.Source: §5 · p. 10

S6 ETHICS OF CONTROLLED LANGUAGE MODELS

The section discusses ethical risks of controlled language models, including bias, disinformation, and the possibility that PPLM’s detoxification mechanism could also generate more toxic language.

S8 FURTHER DETAILS ON HUMAN AND AUTOMATED EVALUATION

The evaluation combines human and automated measures of attribute relevance and fluency across topic, sentiment, and style controls, with detailed results reported for PPLM and baselines.

S9 ODD COMBINATION OF TOPICS AND PREFIXES

PPLM is tested on deliberately mismatched topic-prefix and sentiment-prefix combinations, where control remains effective and generated examples are reported as fluent and sensible.

S10 FINE-GRAINED CONTROL WITH PPLM-BOW

Fine-grained PPLM-BoW control depends on tuning latent-update behavior and the update window. Large step sizes can cause repetition, while early stopping and recent-token updates improve control quality or fluency.

Strength factors

Experimental settings

Early stopping of latent updates

Latent-update window

  • Restricting updates to the most recent w tokens reduces computation, and w = 5 produces more fluent PPLM-BoW passages.Each hidden-state entry is modified at most w times rather than updating the entire past.Source: S11.2 FINITE HORIZON UPDATE · p. 24

Gradient normalization

S12 ADDITIONAL DETAILS AND SAMPLES OF CONTROLLED STORY TELLING

Additional examples extend controlled storytelling to multiple attribute models and language detoxification. The examples combine different controllers and compare toxicity before and after control under natural and adversarial prompts.

Multiple attribute models

Language detoxification

S14 MORE EXAMPLES

The supplementary examples cover additional controlled generations, prefix variation, skeleton storytelling, mixed attribute control, and fluency distributions for BoW and discriminator controllers.

Additional examples

Skeleton storytelling

  • Tables S20 and S21 present skeleton stories generated with fixed skeletons and one or multiple attribute models, with controlled keywords highlighted.Source: Table S21 · p. 28

Mixed attribute control

Ablation fluency

S17 WORD LISTS FOR BAG OF WORDS APPROACHES

The supplementary materials document word-list resources and topic vocabularies used for PPLM-BoW, alongside examples of controlled outputs and toxicity evaluation.

Word-list construction

Topic vocabularies

Religion vocabulary

Computers vocabulary

Multiple-knob examples

  • Table S22 provides examples of attribute-controlled text generation with multiple knobs and includes a clickbait discriminator trained on Potthast et al. (2018).Source: Table S22 · p. 32

Toxicity evaluation

  • Table S23 reports toxicity counts, sample totals, percentages, and p-values comparing Before and After detoxification.Source: Table S23 · p. 33

Related papers

Loading 1912.02164v4…