PaperlayerView exact revision on arXiv

Source-linked AI summary

GeDi: Generative Discriminator Guided Sequence Generation

Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, Nazneen Fatema Rajani

arXiv:2009.06367v2cs.CLcs.LG

TL;DR

Large language models generate realistic text but are difficult to steer away from toxicity, bias, hate, and negativity in web-trained data. GeDi uses smaller class-conditional models as generative discriminators to guide large-model decoding, achieving stronger controllability, over 30× faster generation than PPLM, zero-shot topic control, and reduced GPT-2 toxicity without sacrificing linguistic quality.

  • Problem

    Large language models imitate web-trained data well but offer limited control over which regions of that distribution they generate, including toxic, biased, hateful, or negative content.

  • Method

    GeDi uses smaller class-conditional language models as generative discriminators, applying Bayes-rule classification probabilities for candidate next tokens to guide large-language-model generation.

  • Results

    GeDi provides stronger sentiment controllability than PPLM, generates more than 30× faster, supports zero-shot new-topic control after training on 4 topics, and significantly reduces GPT-2 toxicity without sacrificing linguistic quality.

  • Takeaways & Limitations

    GeDi offers an efficient approach for controllable and safer generation from large language models, including topic control from a keyword and detoxification without linguistic-quality loss.

Abstract

from arXiv · show

While large-scale language models (LMs) are able to imitate the distribution of natural language well enough to generate realistic text, it is difficult to control which regions of the distribution they generate. This is especially problematic because datasets used for training large LMs usually contain significant toxicity, hate, bias, and negativity. We propose GeDi as an efficient method for using smaller LMs as generative discriminators to guide generation from large LMs to make them safer and more controllable. GeDi guides generation at each step by computing classification probabilities for all possible next tokens via Bayes rule by normalizing over two class-conditional distributions; one conditioned on the desired attribute, or control code, and another conditioned on the undesired attribute, or anti control code. We find that GeDi gives stronger controllability than the state of the art method while also achieving generation speeds more than 30 times faster. Additionally, training GeDi on only four topics allows us to controllably generate new topics zero-shot from just a keyword, unlocking a new capability that previous controllable generation methods do not have. Lastly, we show that GeDi can make GPT-2 (1.5B parameters) significantly less toxic without sacrificing linguistic quality, making it by far the most practical existing method for detoxifying large language models while maintaining a fast generation speed.

1 INTRODUCTION

GeDi uses smaller class-conditional language models as generative discriminators to guide large-language-model generation toward desired attributes. It improves controllability, efficiency, topic generalization, and toxicity reduction while maintaining linguistic quality.

  • Motivation: Existing control methods struggle with avoiding toxicity, preserving diversity, or computational efficiency.CTRL is limited for controlling what not to generate, while discriminator-guided methods can be substantially slower.
  • Method: GeDi uses class-conditional language models as generative discriminators to guide generation from large language models toward desired attributes.It computes classification likelihoods for candidate next tokens and applies them through weighted decoding and filtering.
  • Controllability: GeDi trained on movie-review sentiment generates book text with positive or negative tone better than state-of-the-art baselines.The sentiment controller transfers across domains from movie reviews to book text.
  • Detoxification: GeDi significantly reduces GPT-2 toxicity without sacrificing linguistic quality relative to direct GPT-2 generation.The result supports using GeDi for safer language modeling.
  • Topic control: GeDi trained on only 4 topics generalizes to new control codes zero-shot, guiding generation toward a wide variety of topics.The reported capability includes topics not used during training, such as space, history, education, cars, and climate.
  • Efficiency: GeDi-guided generation is more than 30× faster than applying PPLM with GPT2-XL using its default settings.Smaller GeDis fine-tuned for less than a day on one GPU can control larger language models.

2 BACKGROUND

Language models generate sequences autoregressively by predicting each next token from prior tokens. Class-conditional language models add control codes to represent attributes and can also serve as generative classifiers through Bayes rule.

  • Autoregressive language modeling: Autoregressive language models assign sequence probabilities by factorizing them into next-token probabilities.Generation repeatedly samples Pθ(x_t|x_<t) and feeds each sampled token back into the model.
  • Class-conditional language modeling: Class-conditional language models predict text distributions conditioned on a class variable or control code representing an attribute such as sentiment or topic.The control code is paired with each training sequence as its label or category.
  • GeDi motivation: GeDi-guided generation uses Bayes rule to compute classification probabilities for possible next tokens with element-wise operations.Contrasting opposing control codes can cancel bias toward the training domain, enabling attribute control across domains.
  • Training: CC-LMs are trained by minimizing the average negative log-likelihood of sequences conditioned on their control codes.This objective is denoted Lg.
  • Generative classification: CC-LMs can act as generative classifiers by applying Bayes rule to compute the posterior probability of a class given a sequence.This repurposes their class-conditional sequence probabilities for classification.

3 GEDI

GeDi uses class-conditional language models to contrast desired and undesired attributes, efficiently classifying candidate next tokens and steering a larger LM through decoding heuristics. It also trains these models discriminatively to improve their usefulness as generative discriminators.

  • GeDi-guided contrastive generation: GeDi contrasts Pθ(x1:t|c) with Pθ(x1:t|c̄) to estimate whether each candidate next token belongs to the desired class.The method uses a desired control code c and an anti-control code c̄ during generation.
  • GeDi-guided contrastive generation: Two parallel forward passes compute the next-token probabilities conditioned on c and c̄, while cached hidden states avoid recomputing the full prefix.At step t, only Pθ(xt|x<t,c′) must be newly computed for each class.
  • GeDi-guided contrastive generation: 10k-fold less computation is possible than with a unidirectional classifier for classifying every candidate next token, while bidirectional classifiers require t × |V| forward passes.The comparison is stated for typical vocabularies larger than 20k tokens.
  • GeDi-guided contrastive generation: GeDi normalizes log probabilities by sequence length and uses a learnable scale parameter α when computing class probabilities.Class priors can be encoded through bias parameters bc, including uniform, learned, or manually set priors.
  • Heuristics for guiding generation: Weighted decoding biases generation toward the desired class, while filtering retains high-class-probability tokens or the smallest probability-mass prefix meeting ρ.The retained set is Vk = Vp ∪ Vm, after which excluded-token probabilities are zeroed and the remainder renormalized.
  • GeDi training: GeDi training combines generative loss Lg with discriminative loss Ld so the class-conditional model improves classification accuracy and token-probability calibration.The discriminative objective is intended to make GeDis better discriminators, while the generative objective likely supports calibrated probabilities for guided generation.

4 RELATED WORK

Prior controllable-generation methods either train models with control codes or use discriminators to steer generation. GeDi differs by contrasting class-conditional distributions at the class level during both training and inference.

  • Controllable generation: Controllable-generation methods either train or fine-tune models directly, or use discriminators to guide generation.CTRL is an example of a class-conditional LM trained with predefined control codes.
  • Controllable generation: CTRL is less useful for controlling what not to generate, and fixed control codes can reduce diversity across prompts.Generated samples may resemble the data source associated with the selected control code.
  • Contrastive learning: GeDi contrasts a positive class-conditional distribution against a negative one, rather than contrasting individual positive and negative instances.The same class-level contrast is used during GeDi training and contrastive generation.

5 EXPERIMENTS

GeDi experiments show that smaller class-conditional models can guide GPT-2 toward sentiment, lower toxicity, and relevant topics while preserving linguistic quality. The method also transfers sentiment across domains and generalizes from four trained topics to many unseen control codes.

  • Training trade-offs: GeDi-trained class-conditional models achieved higher generative classification accuracy and label fidelity, but at the cost of higher perplexity.GeDi training made control codes more often correspond to the true attribute of generated samples.
  • Sentiment control across domains: GeDi-guided generation transferred positive and negative sentiment from movie reviews to book text while producing book-like generations.Direct class-conditional generation often reverted toward the movie-review training domain, whereas GeDi guidance enabled domain transfer.
  • Sentiment control across domains: GeDi provided slightly stronger sentiment control than PPLM while generating more than 30× faster.Human evaluation used 50 book prompts and rated both sentiment or tone and book resemblance.
  • Detoxifying GPT-2: GeDi-guided generation produced significantly less toxic text than GPT2-XL, while the GeDi-trained guide with λ = 0.6 achieved the highest linguistic quality.Human evaluation measured toxicity on a 1–3 scale and linguistic quality on a 1–4 scale.
  • Multi-class topic control: GeDi-guided generation achieved high relevance across four news topics while maintaining linguistic quality similar to GPT2-XL.The guide was trained on AG News with GeDi training and λ = 0.8.
  • Multi-class topic control: Training on only four topics enabled zero-shot control of many unseen topics, especially for short topic-neutral prompts.The observed topics included space, history, education, cars, and climate; performance was weaker for longer prompts and only one-token topics were tested.

6 FUTURE DIRECTIONS

The paper identifies extensions that could broaden GeDi beyond safety and controllability, including combining discriminators and guiding other desirable attributes. It also points to GeDi-inspired approaches as efficient alternatives to fine-tuning large LMs for new tasks.

  • Future directions: Multiple GeDis could be combined by multiplying attribute-classification terms to filter several undesirable attributes.This extension is described as a way to make language models safer by targeting different attributes simultaneously.
  • Future directions: GeDi could potentially guide generation toward high linguistic quality and improved commonsense reasoning, beyond safety control.These are proposed applications rather than evaluated results in the supplied passage.
  • Future directions: GeDi-inspired methods could provide more computationally efficient alternatives to fine-tuning large LMs for new generation tasks.The passage presents this as a direction for further exploration.

7 CONCLUSION

GeDi uses generative discriminators to guide large-language-model generation toward desired attributes more efficiently than prior discriminator-based methods. Its reported applications include stronger controllability, zero-shot topic control, and GPT-2 detoxification without sacrificing linguistic quality.

  • GeDi classifies candidate next tokens online using generative discriminators, making controllable generation more efficient than previous discriminator-guided methods.The approach uses CC-LMs to compare desired and undesired control conditions during inference.
  • GeDi achieves stronger sentiment controllability than PPLM while generating more than 30× faster.
  • GeDis trained on four topics can controllably generate new topics zero-shot from a keyword.
  • GeDi significantly reduces GPT-2 toxicity without sacrificing linguistic quality.

D.1 HYPER-PARAMETERS FOR GEDI GUIDED GENERATION

Generation hyperparameters were selected using generation quality and external label-fidelity evaluation. Detoxification required compensating for short, prematurely terminated samples caused by token-level discrimination.

  • Hyperparameters were chosen by combining visual generation-quality inspection with label-fidelity metrics from an external RoBERTa classifier.A 100% label-fidelity score means RoBERTa always agrees that the generated class matches the control code.
  • GeDi models generally used ω = 30, ρ = 0.2, and τ = 0.8 across tasks and training-loss settings.The GeDi-trained detoxification model was an exception.
  • The detoxification guide sometimes produced very short samples that ended mid-sentence.
  • A manually added non-toxic prior bias b_c = 2 and τ = 0.97 addressed premature endings while preserving the remaining hyperparameters.The bias assumed sequences were non-toxic, and the higher τ compensated for the resulting increase in P(c|x_1:t).

E EXPERIMENTS WITH GEDI TRAINING

The experiments fine-tune GPT2-medium CC-LMs with a hybrid GeDi objective and evaluate generative and discriminative behavior across sentiment and natural-language-inference tasks.

  • The study fine-tunes 345M-parameter GPT2-medium CC-LMs with task-specific control codes and a hybrid GeDi objective.
  • Training varies λ from 0 to 1, with λ = 1 serving as the generative-training baseline.
  • Across tasks, the evaluation measures perplexity, classifier accuracy, and label fidelity using greedy decoding with a repetition penalty.
  • IMDb tests positive and negative movie-review generation conditioned on approximately 100 characters of a review.
  • MNLI tests contradiction and entailment generation from premises, while QNLI tests passages containing answers to questions.
  • The two NLI tasks are included because their logical reasoning requirements may make them more difficult.

E.1 EVALUATION OF GEDI-TRAINED CC-LMS

GeDi-trained CC-LMs improve classification and label fidelity across three tasks, while the generative–discriminative trade-off affects perplexity. Human evaluation further finds higher label fidelity without a noticeable average loss in linguistic acceptability.

  • For λ < 1, partially discriminative GeDi training improves classification performance and label fidelity across tasks compared with generative training.
  • Higher λ improves held-out perplexity because training becomes closer to generative training.Lower perplexity is better.
  • GeDi-trained CC-LMs nearly match BERT on NLI classification, while PPLM’s attribute classifier struggles on NLI tasks.
  • Human evaluation of 300 MNLI generations asks annotators to label contradiction, neutral, or entailment and rate linguistic acceptability on a 1–4 scale.
  • GeDi-trained CC-LMs achieve higher human-judged label fidelity without noticeably reducing average linguistic acceptability.

F.1 SENTIMENT/TONE SAMPLES

GeDi-guided GPT-2 generation provides sentiment control while retaining knowledge and adapting across domains. Compared with PPLM, CTRL, and direct class-conditional generation, GeDi offers stronger or more transferable control, with a speed advantage over PPLM.

  • GeDi guides GPT-2 toward controlled sentiment while retaining pretrained knowledge about topics such as deep learning.
  • GeDi provides strong sentiment control for book prompts, with a trade-off between control strength and book-like quality for negative samples.The generatively trained GeDi with λ = 1.00 gave the strongest sentiment control but struggled to remain as book-like for negative samples.
  • GeDi transfers sentiment control to book text more effectively than direct class-conditional generation, whose outputs resemble its IMDb training domain.
  • 30× faster generation and stronger sentiment control distinguish GeDi from PPLM in human evaluation experiments.PPLM sometimes struggled to produce negative samples after long, positive prompts.
  • CTRL struggles with sentiment control on book-text prompts because its sentiment training used Amazon reviews.
  • Class-conditional language models tend to reproduce the sentiment characteristics of their training data, while CTRL often converts prompts into Amazon reviews.

F.2 DETOXIFICATION AND DEBIASING SAMPLES

GeDi samples illustrate detoxification, positive-tone control, and topic control, including generalization to topics absent from training. The comparisons also show that CTRL struggles with zero-shot topic generalization when those topics were not in its training set.

  • GeDi detoxifies GPT-2 outputs for especially difficult prompts that otherwise trigger “yourself” or a curse word.
  • Positive-sentiment GeDi guides GPT-2 toward a warm, friendly tone and sometimes produces less biased generation.The authors did not study this debiasing effect systematically in these samples.
  • GeDi effectively controls generation topics across the tested topic prompts.
  • GeDi can creatively generalize when prompts do not match the target topic, such as framing Audi’s armored A3 for the Iraqi army under “world.”
  • GeDi generates topics zero-shot for Space, Fire, and History despite those topics being absent from its training set.
  • GeDi generates Education, Cars, and Climate topics zero-shot despite their absence from its training set.
  • CTRL struggles to generalize to topics absent from its training set, and “Title:” prompts outperform “Text:” prompts for its zero-shot generations.
Loading 2009.06367v2…