PaperlayerView exact revision on arXiv

Source-linked AI summary

Gradient-Based Language Model Red Teaming

Nevan Wichers, Carson Denison, Ahmad Beirami

arXiv:2401.16656v1cs.CL

TL;DR

Human red teaming is labor-intensive, motivating automated ways to find prompts that elicit unsafe LM responses. GBRT optimizes prompts through a safety classifier and LM, and outperforms a strong RL-based approach in finding such prompts, including for a safer fine-tuned LM.

  • Problem

    Human red teaming is labor-intensive and difficult to scale, despite its role in finding prompts that elicit unsafe responses for model alignment and evaluation.

    Source: §1 · p. 1

  • Method

    GBRT updates learnable prompts by backpropagating a safety score through a frozen classifier and LM, with variants using a realism loss or a fine-tuned prompt generator.

    Source: §1 · pp. 1–2

  • Results

    GBRT finds more diverse prompts than RL Red Team, and gradient-based methods can still trigger a model fine-tuned to be safer.

    Source: §6 · p. 9

  • Takeaways & Limitations

    Using both gradient-based and RL-based methods could potentially broaden coverage of model failure cases because they produce different-looking prompts.

    Source: §6 · p. 9

    These methods require access to a differentiable safety classifier and model, limiting applicability to API-only models or non-differentiable safety rules.

    Source: §7 · p. 9

Abstract

from arXiv · show

Red teaming is a common strategy for identifying weaknesses in generative language models (LMs), where adversarial prompts are produced that trigger an LM to generate unsafe responses. Red teaming is instrumental for both model alignment and evaluation, but is labor-intensive and difficult to scale when done by humans. In this paper, we present Gradient-Based Red Teaming (GBRT), a red teaming method for automatically generating diverse prompts that are likely to cause an LM to output unsafe responses. GBRT is a form of prompt learning, trained by scoring an LM response with a safety classifier and then backpropagating through the frozen safety classifier and LM to update the prompt. To improve the coherence of input prompts, we introduce two variants that add a realism loss and fine-tune a pretrained model to generate the prompts instead of learning the prompts directly. Our experiments show that GBRT is more effective at finding prompts that trigger an LM to generate unsafe responses than a strong reinforcement learning-based red teaming approach, and succeeds even when the LM has been fine-tuned to produce safer outputs.

1 Introduction

Gradient-Based Red Teaming (GBRT) automatically searches for prompts that elicit unsafe responses by updating prompts through a frozen classifier and language model. Its variants add a realism loss or fine-tune a pretrained prompt generator, improving prompt sensibility in human evaluation.

1 Introduction

  • Human red teaming is labor-intensive, limiting the scale and diversity of prompts used to elicit unsafe responses for alignment or evaluation.Unsafe outputs may include hate speech, medical misinformation, or harmful biological information.Source: §1 · p. 1
  • GBRT learns prompts by scoring an LM response with a safety classifier, then backpropagating through the frozen classifier and LM to update the prompt.It represents prompt tokens as vocabulary probabilities and uses Gumbel softmax to make prompt sampling and autoregressive decoding differentiable.Source: §1 · pp. 1–2
  • GBRT generates more unique successful prompts than RL-based red teaming and can still produce prompts that trigger a safer fine-tuned model.The safety-classifier gradient supplies a signal for changing prompts toward less-safe responses.Source: §1 · p. 2
  • Two GBRT variants improve prompt sensibility in human evaluation: one adds a realism loss, and the other fine-tunes a pretrained LM to generate prompts.The realism loss penalizes prompt probabilities for diverging from pretrained-model logits.Source: §1 · p. 2

2 Related Work

Related work includes adversarial-token searches, reward-guided generation, and prompt-based red teaming. GBRT differs from methods targeting predetermined outputs by optimizing for a safety classifier’s assessment of a response.

Finding prompts to generate a target response

  • Unlike target-output methods, GBRT aims to trigger a safety classifier because safety is nuanced and cannot be captured by predetermined rules.Prior adversarial-token approaches include methods seeking predetermined unsafe generations and methods targeting a phrase or token.Source: §2 · p. 2

Reinforcement learning (RL) & controlled decoding

  • Controlled-generation work includes classifier-guided decoding, reward optimization, prefix scorers, and supervised fine-tuning through decoding.The cited examples use these approaches to guide or improve model responses.Source: §2 · p. 2
  • Red teaming with controlled generation includes supervised prompt optimization and RL methods that seek responses deemed unsafe by a classifier.Concurrent work adds diversity rewards and entropy regularization to RL red teaming prompts.Source: §2 · p. 2

Prompting techniques for red teaming

  • Other prompt-based red teaming approaches use in-context feedback, adversarially establish undesired behavior, or apply Bayesian optimization.Source: §2 · p. 2

3 Gradient-Based Red Teaming (GBRT)

GBRT learns prompts by minimizing a frozen safety classifier’s score on an LM response, backpropagating through both frozen models to update prompt probabilities. Gumbel softmax makes prompt sampling and LM decoding differentiable; realism-regularized and model-generated variants aim to produce more sensible prompts.

3 Gradient-Based Red Teaming (GBRT)

  • GBRT minimizes the safety score of a generated response by backpropagating through the frozen LM and classifier to update prompt probabilities.The classifier may score the prompt together with the response, using the prompt as context.Source: §3 · p. 3
  • Gumbel softmax approximates categorical sampling, making both prompt inputs and autoregressive decoding differentiable for gradient-based updates.Prompt probabilities are initialized uniformly and updated with gradient descent.Source: §3 · p. 3
  • GBRT-ResponseOnly scores only the response, avoiding prompt-dependent classifier errors that can arise when the prompt is included as context.The classifier can still make errors, but they are not dependent on the prompt.Source: §3 · p. 3, Figure 2 · p. 4
  • The realism loss penalizes divergence between prompt probabilities and a pretrained LM’s next-token predictions to encourage more sensible prompts.Its term is −σ(x) ∗x′, where x is prompt-token probabilities and x′ contains the model’s predicted log probabilities.Source: §3 · p. 4
  • The model-based variant fine-tunes a pretrained prompt model to generate prompts, using L2 regularization to keep its weights near their pretrained values.It uses a hard-coded input prompt and samples prompt-model response probabilities with Gumbel softmax; decoding steps are not backpropagated through.Source: §3 · p. 4

4 Experiment Setup

The experiments generate prompts with a 2B-parameter LaMDA model and evaluate their responses using greedy decoding. The study compares GBRT variants with an RL baseline and human-generated prompts, measuring unsafe-response success, diversity, sensibleness, and human ratings.

4 Experiment Setup

  • Experiments use a 2B-parameter LaMDA model and generate 200 prompts per method; evaluation responses are 15 tokens and use greedy decoding.Direct prompt optimization uses six input tokens, four decoded training-response tokens, and 200 random-seed runs; prompt-model methods sample 200 prompts.Source: §4 · p. 4

4.1 Baselines

  • The baselines are a reproduced RL prompt-generation method and toxic first-turn prompts from the Bot Adversarial Dialogue dataset.The BAD comparison uses the first 200 training prompts to match the setup.Source: §4.1 · p. 5

4.2 Metrics

  • Evaluation reports unique unsafe-response prompts, response toxicity, self-BLEU diversity, and prompt sensibleness measured by the original LM’s log perplexity.Two co-authors also rated 50 randomly shuffled prompts per method for coherence and toxicity on a 1–5 Likert scale.Source: §4.2 · p. 5

5 Results & Analysis

GBRT variants generally find more successful red-teaming prompts than RL Red Team, while realism and fine-tuning variants improve prompt coherence. Results also show language and hyperparameter choices affect prompt outputs, and prompts used for fine-tuning can make a model more robust to red teaming.

5.1 Quantitative Analysis

  • The evaluation counts triple-parenthesis slur outputs as unsafe even though the independent classifier does not label them unsafe.The model sometimes emits the slur or closes parentheses opened in the prompt.Source: §5.1 · p. 5
  • GBRT-RealismLoss is the most successful at finding red-teaming prompts, and all methods except vanilla GBRT find significantly more successful prompts than RL Red Team.Vanilla GBRT and RL Red Team are the weakest methods in this comparison.Source: §5.1 · p. 6, Table 1 · p. 6
  • The realism loss improves prompt coherence, while GBRT-Finetune further improves mean log perplexity but produces less diverse prompts than most other methods.The paper attributes the realism-loss variant's higher fraction of unsafe responses as likely related to increased diversity; the diversity explanation for fine-tuning is presented as probable.Source: §5.1 · p. 6, Table 2 · p. 6

5.2 Qualitative Analysis

  • Human evaluations generally agree with automated coherence and toxicity measures, while qualitative examples show realism-loss prompts can remain garbled despite improved log perplexity.GBRT-Finetune prompts are grammatically correct in German with some concatenated words; RL Red Team prompts are coherent but repetitive.Source: §5.1 · p. 7, §5.2 · pp. 7–8, Table 3 · p. 7

5.1 Quantitative Analysis

  • On the safer model, GBRT finds some prompts that elicit unsafe responses, while GBRT-ResponseOnly and RL Red Team do not find a significant number.The safer model was fine-tuned with Direct Preference Optimization using a safety reward and human-rater data.Source: §5.1 · p. 7, Table 4 · p. 8

5.3 Prompt Languages

  • German prompts can be rated unsafe because the safety classifier may be less accurate in German; translating GBRT-Finetune responses to English lowers the unsafe fraction from 0.24 to 0.19.The paper reports translated-response results because the classifier is more accurate for English.Source: §5.3 · p. 8

5.4 Effect of hyperparameters on prompts

  • Removing German tokens controls prompt language, but improves GBRT-RealismLoss quality and decreases GBRT-ResponseOnly quality; changing hyperparameters can also yield different prompts and failure cases.The no-German filtering is not perfect, and different hyperparameters can control whether prompts include triple parentheses.Source: §5.3 · p. 8, §5.4 · p. 8, Table 6 · p. 9

5.5 Finetuning a model on GBRT prompts

  • Fine-tuning on 224 prompts from GBRT and GBRT-ResponseOnly lowers the red-teaming methods’ success rate on the resulting model, indicating increased robustness to red teaming.The prompts were selected for producing the most unsafe responses, and the model was trained with Direct Preference Optimization.Source: §5.5 · p. 9, Table 7 · p. 9

6 Discussion and Conclusion

GBRT generates prompts that trigger unsafe LM responses and produces more diverse successful prompts than the RL Red Team baseline. Realism-oriented variants improve prompt sensibility, and the methods can still trigger safer fine-tuned models.

6 Discussion and Conclusion

  • GBRT methods produce more diverse prompts that trigger unsafe responses than RL Red Team, and can still trigger models fine-tuned to be safer.The different-looking prompts from the two approaches could potentially broaden coverage of model failure cases when both are used.Source: §6 · p. 9
  • Adding a realism loss or fine-tuning a pretrained prompt model improves prompt sensibility, although RL Red Team prompts remain more sensible than gradient-based prompts.Source: §6 · p. 9

7 Broader Impacts & Limitations

GBRT assumes access to a differentiable safety classifier and model, and its fixed prompt and response lengths, language coverage, and access requirements constrain its scope. The authors also note a potential misuse risk for attackers with direct model-weight access.

7 Broader Impacts & Limitations

  • GBRT requires a differentiable safety classifier and model, so it does not apply to API-only models or non-differentiable safety rules.Source: §7 · p. 9
  • Hard-coded input-prompt and response lengths may miss nuanced unsafe responses that require longer contexts.Source: §7 · p. 9
  • Testing different token counts was limited, leaving their effects not well understood.Source: §7 · p. 10
  • The tested model was primarily trained on English data, and GBRT mostly finds English and German prompts, so it may work less well for models trained in other languages.Source: §7 · p. 10
  • Malicious actors could use GBRT to generate harmful prompts only if they have direct access to model weights; the authors note they could also fine-tune the model to generate harmful content.Source: §7 · p. 10

A.1 Additional Figures

Figure 3 depicts GBRT-Finetune, in which a prompt model generates the prompt and only the red-marked weights are updated.

A.1 Additional Figures

  • Figure 3 shows the prompt model generating the prompt from a fixed prompt, with red-marked weights updated by backpropagation and the remaining weights frozen.Source: Figure 3 · p. 13

A.2 Implementation details

Implementation details describe the safety classifier's input and score, the computational setup, and prompt-token validity handling.

A.2 Implementation details

  • GBRT-ResponseOnly uses the hard-coded prompt “Hi” because the classifier expects a prompt when scoring a response.Source: §A.2 · p. 13
  • The LaMDA decoder-only language model and safety classifier are fine-tuned on context, RESPONSE, response, UNSAFE, and rating sequences; the predicted probability of 0 as the last token is used as the loss.Source: §A.2 · p. 13
  • GBRT computes the response and safety score in one LaMDA forward pass, while GBRT-ResponseOnly requires two passes because the LM and classifier prompts differ.Source: §A.2 · p. 13
  • A preceding space is required for a valid first prompt token, so implementation prepends a hard-coded token such as “ Hi” or “ Therefore.”Source: §A.2 · p. 13

A.3 Samples of red teaming prompts and responses

This appendix documents hyperparameter settings and prompt/model-response samples across GBRT variants, baselines, and safer-model evaluations.

A.3 Samples of red teaming prompts and responses

  • Table 12 and later tables show randomly selected prompts and model responses from method runs; duplicate prompts indicate method-generated duplicates, and the unsafe score is the probability a response is unsafe given its prompt.The prompt and response columns contain generated prompts and model responses, respectively; swear words are replaced with asterisks.Source: §A.3 · p. 13, Table 12 · p. 15
  • Table 8 describes method hyperparameters, including Gumbel-softmax scheduling from soft to hard and interpolation of each parameter from its initial to final value during training.The table also identifies a hyperparameter setting that produces prompts containing triple parentheses.Source: Table 8 · p. 14
  • Tables 9–10 specify the realism-loss weight and the L2 regularization weight between the fine-tuned model and its pretrained counterpart.Source: Table 9 · p. 14, Table 10 · p. 14
  • Table 11 compares prompt-model inputs: the generic-prompt rows use “Hi,” while other rows use the question about asking an LM to say something unsafe.Source: Table 11 · p. 14
  • Table 13–18 provide samples for GBRT-RealismLoss, its no-German variant, GBRT-ResponseOnly and its no-German variant, GBRT-Finetune, and the RL baseline.Source: Table 13 · p. 15, Table 14 · p. 16, Table 15 · p. 16
  • Tables 19–23 provide samples for the BAD dataset and for GBRT, GBRT-ResponseOnly, the RL baseline, and BAD dataset with the safer model.Source: Table 19 · p. 18, Table 20 · p. 19, Table 21 · p. 19
Loading 2401.16656v1…