PaperlayerView exact revision on arXiv

Source-linked AI summary

Controllable Natural Language Generation with Contrastive Prefixes

Jing Qian, Li Dong, Yelong Shen, Furu Wei, Weizhu Chen

arXiv:2202.13257v1cs.CL

TL;DR

Controllable NLG seeks to guide text toward desired attributes, while prior work has used fine-tuning or discriminator guidance. This paper trains relationship-aware prefixes for frozen GPT-2 and reports successful single-aspect and promising multi-aspect control.

  • Problem

    Prior controllable-generation approaches have focused on fine-tuning language models or using attribute discriminators, and GeDi focuses on single-aspect control.

    Source: §1 · p. 1

  • Method

    The framework trains multiple small attribute-specific prefixes simultaneously, using supervised or unsupervised objectives that account for relationships among prefixes.

    Source: §1 · pp. 1–2

  • Results

    Experiments report guidance toward target attributes with high linguistic quality in single-aspect tasks and successful simultaneous sentiment and topic control.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The authors conclude that the framework can be applied to attributes beyond the control tasks they experimented with.

    Source: §5 · p. 9

    In the human-evaluated task, more severe repetition than in sentiment control produced an inconsistency that illustrates the limitation of automatic evaluations.

    Source: §4.1.3 · p. 8

Abstract

from arXiv · show

To guide the generation of large pretrained language models (LM), previous work has focused on directly fine-tuning the language model or utilizing an attribute discriminator. In this work, we propose a novel lightweight framework for controllable GPT2 generation, which utilizes a set of small attribute-specific vectors, called prefixes, to steer natural language generation. Different from prefix-tuning, where each prefix is trained independently, we take the relationship among prefixes into consideration and train multiple prefixes simultaneously. We propose a novel supervised method and also an unsupervised method to train the prefixes for single-aspect control while the combination of these two methods can achieve multi-aspect control. Experimental results on both single-aspect and multi-aspect control show that our methods can guide generation towards the desired attributes while keeping high linguistic quality.

1 Introduction

Controllable NLG steers text toward desired attributes, but prior approaches include fine-tuning or discriminator guidance. This work trains lightweight prefixes for frozen GPT-2, accounting for relationships among prefixes and reporting single- and multi-aspect control.

1 Introduction

  • Earlier controllable-generation work focused on fine-tuning language models or using discriminators, with CTRL requiring a large conditional model and GeDi adding 345M parameters.The paper also notes that GeDi focuses on single-aspect control.Source: §1 · p. 1
  • Figure 1 contrasts prefix-tuning with the proposed framework and distinguishes training arrows from inference or generation arrows.The proposed framework allows supervised, semi-supervised, or unsupervised training.Source: Figure 1 · p. 1
  • The framework uses small attribute-specific prefixes with frozen language models, introducing about 0.2%-2% of GPT-2 parameters and retaining comparable inference speed.The authors present prefixes as an alternative to attribute models or generative discriminators.Source: §1 · pp. 1–2
  • Unlike independently trained prefixes, the proposed supervised and unsupervised methods train multiple prefixes simultaneously while accounting for relationships among them.The paper notes that sentiment attributes such as positive and negative are opposites.Source: §1 · p. 2
  • Experiments report target-attribute guidance with high linguistic quality for sentiment, detoxification, and topic control, including settings with only several dozen labeled examples.Combining the supervised and unsupervised methods also achieves simultaneous sentiment and topic control.Source: §1 · p. 2

2 Related Work

Prior work controls generation through conditioned language models, hidden-state updates, and discriminator-based guidance. The paper distinguishes its approach by directly training prepended parameters rather than using target-attribute tokens.

2 Related Work

  • Ficler and Goldberg (2017) used a conditioned RNN for style control, while Holtzman et al. (2018) used discriminator committees for linguistic quality and Hu et al. (2017) combined VAEs with attribute discriminators.These approaches are described as controlling stylistic aspects, guiding linguistic quality, or controlling sentiment and tense, respectively.Source: §2 · p. 2
  • Keskar et al. (2019) trained a 1.63 billion-parameter conditional Transformer from scratch with 55 fixed control codes, while Dathathri et al. (2020) perturbed hidden activations using a discriminator.The paper describes the former as expensive and inflexible and the latter as slow because it updates gradients at token level.Source: §2 · p. 2
  • Krause et al. (2020), Yang and Klein (2021), and Lin and Riedl (2021) used generative discriminators to reweight next-token distributions during inference.The paper presents this as improving inference speed over token-level hidden-activation updates.Source: §2 · p. 2
  • Unlike Yu et al. (2021), this work does not input target-attribute tokens; unlike prefix-tuning, it controls attributes rather than a specific NLG task.The paper says avoiding attribute tokens can help when a desired attribute is difficult to describe in one word, and that attribute disentanglement is not its focus.Source: §2 · p. 2

3 Method

The method steers GPT-2 with trainable attribute-specific prefixes while keeping the pretrained model fixed. Its supervised and unsupervised approaches train multiple prefixes together, using labeled examples or latent prefix assignments, respectively.

3 Method

  • Continuous attribute-specific prefixes are prepended to GPT-2 activations and trained jointly while the pretrained GPT-2 parameters remain frozen.The prefix bank has N prefixes of length M; after training, only the trained prefixes need to be saved for generation.Source: §3 · pp. 2–3, §3.1 · p. 3

3.1 Supervised Method

  • For labeled examples, the supervised objective combines language-model loss with a discriminative loss that trains the correct prefix to generate the text and discourages other prefixes from doing so.The attribute label identifies the ground-truth prefix, and the discriminative loss improves attribute alignment by increasing p(x|y) while lowering p(x|y′) for other labels.Source: §3.1 · p. 3

3.2 Unsupervised Method

  • Without labels, an encoder assigns each text a latent prefix index, and the selected prefix conditions GPT-2 to reconstruct the text.Gumbel-Softmax relaxes discrete prefix selection; training combines language-model, KL, and contrastive losses, with the contrastive term separating the inferred and opposite-prefix posteriors by a margin.Source: §3.2 · pp. 3–5

4 Experiments

Experiments test prefix control across sentiment, detoxification, and topic tasks, then extend to human evaluation and simultaneous sentiment-topic control. Results cover supervised and unsupervised training, few-shot settings, loss ablations, generation speed, and multi-aspect control.

4.1.1 Tasks

  • Experiments use GPT2-medium on English sentiment, detoxification, and topic control, evaluating linguistic quality and attribute alignment against GPT2, PPLM, and GeDi.The study includes full-data, 1,000-example, and 24-example-per-attribute supervised settings, plus unsupervised sentiment and detoxification experiments.Source: §4 · p. 5, §4.1.1 · p. 5

4.1.2 Results

  • The unsupervised method lowers detoxification toxicity, but achieves weak negative-sentiment alignment; its contrastive loss is crucial for detoxification.The authors suggest that positive-negative sentiment differences are subtler than toxic-normal differences, challenging the encoder's separation of unlabeled sentiment data.Source: §4.1.2 · p. 6
  • With full training data, the supervised method outperforms PPLM in controllability while maintaining linguistic quality, and retains good control with 1,000 or 24 examples per attribute.GeDi has high attribute alignment but substantially higher perplexity; the authors report robustness across the three single-aspect tasks.Source: §4.1.2 · p. 6, Table 2 · p. 7, Table 1 · p. 7
  • Ablations identify discriminative loss Ld as important for detoxification and DBPedia topic alignment, while removing encoder loss Lenc trades linguistic quality for attribute relevance.The authors also report that their method's inference speed is closest to GPT2's; PPLM is several dozen times slower than GPT2.Source: §4.1.2 · p. 6, Table 5 · p. 7

4.1.3 Human Evaluation

  • Human ratings broadly track automatic evaluations, but annotators favored Ours and prompt engineering for sentiment alignment and often chose Ours for AGNews linguistic quality despite its higher perplexity.The evaluations used 100 sampled tasks per control experiment, each rated by three Mechanical Turk workers; annotators noted more severe repetition in the prompt-engineered AGNews outputs.Source: §4.1.3 · pp. 6–8, Table 4 · p. 7

4.2 Multi-Aspect Control

  • Concatenating separately trained sentiment and topic prefixes controls both aspects, and changing their order does not affect the result.Joint semi-supervised training can further improve attribute alignment without sacrificing too much linguistic quality; removing Ld degrades relevance, while removing Lenc can raise perplexity.Source: §4.2 · pp. 8–9, Table 6 · p. 8

5 Conclusion

The paper presents contrastive prefixes as a way to steer generation from frozen GPT-2 models. Its experiments report single- and multi-aspect control, while factual accuracy is not guaranteed.

5 Conclusion

  • The framework uses contrastive prefixes to guide GPT-2 generation with frozen language models.The authors say it can be applied to desired attributes beyond the control tasks they tested.Source: §5 · p. 9
  • Experiments show the framework can guide generation for both single-aspect and multi-aspect control tasks.The authors note that factual accuracy is not guaranteed and was not the focus of the work.Source: §5 · p. 9

A Hyperparameters

The appendix lists shared implementation settings and task-specific hyperparameters for the proposed methods and comparison systems.

A Hyperparameters

  • PPLM and GeDi use hyperparameters reported in their original work, with the latest available GeDi submission used for comparison.The appendix also supplies task-specific PPLM and GeDi values.Source: A Hyperparameters · p. 11
  • Across the proposed-method experiments, the optimizer is AdamW at a 2e-5 learning rate, the seed is 42, and the default prefix length is M = 10.GPT2-medium uses D = 24 × 2 × 1024; maximum evaluation generation lengths are 50 for sentiment and topic control and 20 for detoxification.Source: A Hyperparameters · p. 11
  • Sentiment control uses batch size 8; the unsupervised setup trains for 60 epochs with a 0.5 mask rate, while the supervised setup trains for 50 epochs.The unsupervised KL-loss weight anneals from 0.001 to 0.1 and temperature falls from 1.0 to 0.5; PPLM and GeDi settings are also reported.Source: A Hyperparameters · p. 11
  • Detoxification uses batch size 8; the unsupervised setup trains for 4 epochs and the supervised setup for 5.The unsupervised configuration uses a 0.5 mask rate, and PPLM settings are listed for the comparison.Source: A Hyperparameters · p. 11
  • For AGNews and DBPedia topic control, the supervised setup uses batch size 4 and weights ω1 = 0.8, ω2 = 0.2, with 8 and 2 epochs, respectively.GeDi settings are given separately for the topic-control configurations.Source: A Hyperparameters · p. 11
  • The multi-aspect concatenation setting joins sentiment and topic prefixes of length 10 into a prefix of length 20.The semi-supervised setting uses 80,000 encoder-only steps followed by 80,000 steps with updated weights and top-k/top-p filters for each aspect.Source: A Hyperparameters · p. 11

B Human Evaluation

The human-evaluation appendix identifies the worker interface, annotation payment, and examples of generated outputs.

B Human Evaluation

  • Each approved annotation paid $0.6, and the average HIT took 3 minutes 45 seconds, equivalent to a prorated hourly wage of $9.6.Source: B Human Evaluation · p. 11
  • Figure 5 shows the user interface provided to Mechanical Turk workers.The passage identifies the interface but does not describe its controls or evaluation procedure.Source: Figure 5 · p. 12
  • Table 7 gives generation examples by control code and method, including GPT2+ outputs using prompt engineering.The guiding sentences for GPT2+ are omitted for brevity.Source: Table 7 · p. 13

Related papers

Loading 2202.13257v1…