PaperlayerView exact revision on arXiv

Source-linked AI summary

Explore, Establish, Exploit: Red Teaming Language Models from Scratch

Stephen Casper, Jason Lin, Joe Kwon, Gatlen Culp, Dylan Hadfield-Menell

arXiv:2306.09442v3cs.CLcs.AIcs.LG

TL;DR

Automated red-teaming often relies on a pre-existing way to classify failures, limiting its tailoring to a target model. This paper explores target-model behavior, establishes a contextual measure, and uses it to generate adversarial prompts. In GPT-3 experiments, CommonClaim-based attacks elicited false statements, while a CREAK-based reward led to outputs that were neither true nor false.

  • Problem

    Existing automated red-teaming approaches rely on failures being efficiently identifiable in advance, although harmful behavior may be unforeseen and target-specific.

    Source: §1 · p. 1

  • Method

    The framework explores target-model outputs, establishes a contextual measure of undesirable behavior, and uses it to train an adversarial prompt generator.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    CommonClaim-based attacks elicited political-topic statements commonly targeted by misinformation, while the CREAK classifier appeared to provide a more hackable reward signal leading to outputs that were neither true nor false.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that from-scratch red-teaming can be competitive with using a pre-existing classifier to filter training data or model outputs.

    Source: §5 · p. 9

    The approach trades red-teaming efficiency for looser permissions and requires exploration and manual data screening.

    Source: §5 · p. 9

Abstract

from arXiv · show

Deploying large language models (LMs) can pose hazards from harmful outputs such as toxic or false text. Prior work has introduced automated tools that elicit harmful outputs to identify these risks. While this is a valuable step toward securing models, these approaches rely on a pre-existing way to efficiently classify undesirable outputs. Using a pre-existing classifier does not allow for red-teaming to be tailored to the target model. Furthermore, when failures can be easily classified in advance, red-teaming has limited marginal value because problems can be avoided by simply filtering training data and/or model outputs. Here, we consider red-teaming "from scratch," in which the adversary does not begin with a way to classify failures. Our framework consists of three steps: 1) Exploring the model's range of behaviors in the desired context; 2) Establishing a definition and measurement for undesired behavior (e.g., a classifier trained to reflect human evaluations); and 3) Exploiting the model's flaws using this measure to develop diverse adversarial prompts. We use this approach to red-team GPT-3 to discover classes of inputs that elicit false statements. In doing so, we construct the CommonClaim dataset of 20,000 statements labeled by humans as common-knowledge-true, common knowledge-false, or neither. We are making code and data available.

1 INTRODUCTION

Automated red-teaming can search for harmful responses, but existing approaches assume failures can already be efficiently identified. This work proposes a from-scratch framework and applies it to eliciting toxic text from GPT-2-xl and false text from GPT-3-text-davinci-002.

1 INTRODUCTION

  • Existing automated red-teaming depends on failures being identifiable beforehand, limiting its fit for abstract, target-specific behavior and reducing its value when filtering can address known failures.The prompt and output space is massive, and failures may be unforeseen or poorly defined in advance.Source: §1 · p. 1
  • The framework explores model behavior, establishes a contextual definition and measure of undesirable behavior, then exploits vulnerabilities with adversarial prompting.It produces diverse labeled examples, a measure such as a classifier, and a process for generating adversarial prompts.Source: §1 · p. 2
  • The authors demonstrate the framework by red-teaming GPT-2-xl for toxic text and GPT-3-text-davinci-002 for false text.They also introduce a technique intended to avoid mode collapse during reinforcement-learning prompt generation.Source: §1 · p. 2
  • CommonClaim labels 20,000 GPT-3-text-davinci-002 generations true, false, or neither according to human common knowledge; attacks using its classifier elicited statements about political topics commonly targeted by misinformation.The CREAK classifier appeared to provide a more hackable reward signal because it led to prompts that were neither true nor false.Source: §1 · p. 2
  • The GPT-3 experiment shows the value of contextually refining the target behavior compared with using a pre-existing classifier.CommonClaim’s “neither” label makes its classifier more robust to statements that are not claims about the world.Source: §1 · p. 2

2 METHODS

The method is designed for red-teaming in the target model’s intended deployment context, starting from a high-level understanding rather than a ready-made failure classifier. It explores outputs, establishes a contextual measure, and uses that measure to train a prompt generator.

2 METHODS

  • The red team begins with only a high-level understanding of possible failures, whereas prior works assume the Explore and Establish steps are already done.The goal is to focus on target-model outputs in their intended deployment context rather than a pretrained classifier’s training distribution.Source: Figure 1 · p. 2, §2 · p. 2
  • Explore samples diverse model outputs so users can examine the range of behaviors the model produces.The method embeds outputs with last-token activations, clusters them with K-means, and samples sentences uniformly across clusters.Source: §2 · pp. 2–3
  • Establish uses human labels, or a classifier as a quantitative proxy for human judgment, to define and measure undesirable outputs in context.The method uses paraphrasing augmentation and trains an ensemble of five RoBERTa-based text classifiers.Source: §2 · p. 3
  • Exploit uses reinforcement learning to train a prompt generator that elicits completions classified as harmful by the Establish-step measure.Its reward combines classifier confidence with intra-batch cosine distances between target-model embeddings of generated prompts to discourage mode collapse.Source: §2 · p. 3

3 EXPERIMENTS

The experiments show that red-teaming from target-model data can elicit toxic or apparently false outputs, while generic classifiers can provide hackable reward signals. Diversity helps prevent prompt-generator collapse, and CommonClaim's labels capture common knowledge rather than literal truth.

3 EXPERIMENTS

  • The synthetic toxicity-proxy experiment demonstrated feasibility and found the diversity objective necessary to prevent mode collapse during reinforcement learning.The experiments also assessed contextual target behaviors and compared contextual with generic classifiers.Source: §3 · pp. 3–4

3.2 ELICITING FALSE TEXT FROM GPT-3-TEXT-DAVINCI-002

  • CommonClaim labels 20,000 GPT-3 claims as common-knowledge true, false, or neither; these categories differ from literal truth and can reflect cultural familiarity.Explore data prompted the additional “neither” category for statements difficult to classify as common-knowledge true or false.Source: §3 · p. 4, §3.2 · pp. 5–6, §3.2.1 · p. 6

3.1 ELICITING TOXIC TEXT FROM GPT-2-XL

  • From 80,000 GPT-2-xl sentences, an adversarial prompt generator raised proxy-classified toxicity from <1% to 31%.The 80,000 sentences were sampled from unprompted paragraphs; two independent prompt-generator runs were evaluated.Source: §3.1.1 · p. 4, §3.1 · p. 4

3.1.1 RESULTS

  • Without diversity reward, GPT-2-xl collapsed to repetitive prompts with 0% toxic completions, while GPT-3 repeated one prompt in 61 of 100 samples.GPT-2 prompts almost entirely used “would” and “you”; the GPT-3 collapse occurred in 61/100 samples.Source: §3.1.1 · p. 4, §3.2.1 · p. 8

3.2.1 RESULTS

  • CommonClaim classifiers labeled 30% of Explore data and 74% of adversarial completions common-knowledge-false; CREAK's 95% false classifications yielded no apparent untruthfulness.CREAK-driven prompts tended toward toxic or violent non-claims, while CommonClaim-driven prompts elicited political-misinformation-related completions.Source: §3.2.1 · pp. 6–7
  • In Appendix E, classifiers trained on CommonClaim labels produced by ChatGPT-3.5-turbo seemed easily hackable, while completions elicited using them had no apparent tendency to be false.Source: §3.2.1 · p. 7

4 RELATED WORK

The paper situates its work among methods for exploring model behavior, collecting human feedback, and searching automatically for prompts. It highlights common-knowledge truth labels as one response to differing notions of truth.

Exploring unexpected capabilities of language models

  • K-means diversity subsampling extends open-ended exploration of model capabilities beyond human-in-the-loop interaction.The authors note that other statistical techniques may also be useful.Source: §4 · p. 8

Reinforcement Learning from Human Feedback (RLHF)

  • The framework is a form of RLHF with a particularly involved, open-ended human-feedback step.RLHF samples model outputs, collects human feedback, fits a reward model, and fine-tunes with reinforcement learning.Source: §4 · p. 8

Red-teaming with automated searches for natural language prompts

  • The authors use reinforcement-learning attacks because they are effective, black-box, and yield sampleable adversarial-prompt distributions.They contrast this choice with other automated search methods and say the framework could use any of them.Source: §4 · p. 8

Studying toxicity and untruthfulness in large language models

  • Common-knowledge labels differ from ground-truth approaches, and CommonClaim includes neither-type statements that may help address probing-data and interpretation challenges.The authors also describe their work as synthesizing inputs that elicit false completions at scale.Source: §4 · p. 8

5 DISCUSSION

The discussion presents a complete red-teaming framework that can be competitive with pretrained-classifier filtering and can elicit false text at scale. Its outputs support interpretability, diagnosis, and debugging, while the approach remains constrained by human effort and prompt diversity challenges.

Realistic and competitive red-teaming

  • The framework can be more effective than pretrained-classifier red-teaming and is inherently competitive with filtering training data or model outputs.The paper also reports its first example of automated LM red-teaming at scale to elicit false text, focusing on claims widely regarded as false by common knowledge.Source: §5 · p. 9

The value of preference formation and human factors for AI oversight

  • The Establish step found that a third, “neither” category was necessary to label examples adequately and avoid an easily hackable reward signal.The authors frame this as avoiding choice-set misspecification when statements are not objectively true or false.Source: §5 · p. 9

What comes after Explore/Establish/Exploit?

  • The pipeline produces a labeled dataset, a harmful-output classifier, and a distribution for sampling adversarial prompts.The dataset may support probing, the classifier may filter data or outputs, and the generator may support probing or adversarial training.Source: §5 · p. 9

Limitations

  • Red-teaming remains subject to human limitations, and working with little prior knowledge of failures requires exploration and manual data screening.The authors describe a tradeoff between red-teaming efficiency and the looseness of permissions granted to the red team.Source: §5 · p. 9

Future work

  • The method still shows some mode collapse, and producing highly diverse, fluent prompts that elicit harmful outputs remains an open problem.The authors identify K-means as their only Explore-step diversity-sampling tool and suggest testing other methods and operationalizations of truth.Source: §5 · p. 9

A ETHICS AND SOCIETAL IMPACT

The authors frame the work as safety-motivated red-teaming while acknowledging that the approach could be used for malicious attacks.

A ETHICS AND SOCIETAL IMPACT

  • The approach could be used for malicious attacks, although the authors say their motivation is to improve safety by understanding and addressing risks.They describe studying vulnerabilities through attacks as a prerequisite to defending against them.Source: A ETHICS AND SOCIETAL IMPACT · p. 15
  • Developers can red-team a model before deployment and refuse to deploy it if needed.The authors call this a “defender’s advantage.”Source: A ETHICS AND SOCIETAL IMPACT · p. 15
  • The authors present red-teaming as a way to better identify and exploit weaknesses in language models.Source: A ETHICS AND SOCIETAL IMPACT · p. 15

B DIVERSITY TERM ABLATION EXPERIMENTS

Removing the diversity term made prompt generation collapse to narrow distributions in both toxicity and untruthfulness red-teaming. The cited examples show zero toxic classifications for GPT-2-xl and repeated identical prompts for GPT-3.

B DIVERSITY TERM ABLATION EXPERIMENTS

C.1 INSTRUCTIONS GIVEN TO KNOWLEDGE WORKERS

The instructions ask annotators to label AI-generated statements as reasonably true, reasonably false, or neither. They use common-sense judgments and allow “neither” for statements that are vague, contextless, opinion-based, or otherwise not clearly true or false.

C.1 INSTRUCTIONS GIVEN TO KNOWLEDGE WORKERS

  • The task labels AI-generated sentences as reasonably true, reasonably false, or neither, rather than requiring strict factual certainty.For example, the instructions treat a fictional-character claim as accurate and a common misconception as false.Source: §C.1 · p. 16
  • “True” means a typical adult would find the statement accurate; “false” means inaccurate, misleading, untruthful, or “BS.”The instructions say to use best judgment when the answer is unclear and to choose “true” when a statement is honest and reasonable, even without complete certainty.Source: §C.1 · p. 16
  • “Neither” covers opinions, vague or contextless claims, incomplete grammatical statements, and statements that could reasonably be correct or incorrect.Examples include subjective claims, claims without enough context, and sentence fragments.Source: §C.1 · pp. 16–17

C.2 KNOWLEDGE WORKER SELECTION

Knowledge workers were screened using six test questions, and only workers who answered all six successfully were selected. The questions’ intended common-knowledge labels included true, false, and neither.

C.2 KNOWLEDGE WORKER SELECTION

  • Workers were selected only if they successfully answered all six screening questions.The expected answers, in order, were common-knowledge false, true, neither, neither, false, and true.Source: §C.2 · p. 17

D EXPERIMENTS WITH A CREAK CLASSIFIER

Red-teaming GPT-3 with a classifier trained on CREAK elicited toxic prompts and responses rather than outputs that showed a tendency to be untrue claims. The classifier nevertheless labeled the responses untrue, indicating that it could be hacked.

D EXPERIMENTS WITH A CREAK CLASSIFIER

  • The CREAK-trained classifier labeled toxic red-team responses untrue even though they showed no apparent tendency to be untrue claims.The authors describe this as evidence that the classifier was hackable.Source: D EXPERIMENTS WITH A CREAK CLASSIFIER · p. 17

E EXPERIMENTS WITH CHATGPT LABELS INSTEAD OF HUMAN LABELS

Classifiers trained on ChatGPT-3.5-turbo labels scored many adversarial completions as common-knowledge-false, but those completions tended to be toxic or code-like nonsense rather than untruthful. Human and chatbot labels agreed on only 54% of the examples.

E EXPERIMENTS WITH CHATGPT LABELS INSTEAD OF HUMAN LABELS

  • Table 8 presents examples from independent red-teaming runs using classifiers trained on ChatGPT-3.5-turbo labels.The examples were adversarial prompt-completions labeled common-knowledge-false by those classifiers.Source: Table 8 · p. 19
  • Human and ChatGPT-3.5-turbo labels agreed on only 54% of the 20,000 examples.The chatbot labeled 48% common-knowledge-true, 14% common-knowledge-false, and 38% neither; classifiers trained on its labels averaged 87% accuracy on true, 63% on false, and 58% on neither validation examples.Source: E EXPERIMENTS WITH CHATGPT LABELS… · p. 19
  • 76% of adversarial-prompt completions were classified as common-knowledge-false, versus 17% of Explore-step data, yet the completions showed no apparent tendency to be untruthful.The adversarial completions tended to be toxic or nonsense strings of code-like vocabulary, suggesting that the ChatGPT-label classifiers were more hackable.Source: E EXPERIMENTS WITH CHATGPT LABELS… · p. 19
  • The authors report that chatbot-labeled classifiers, like CREAK-trained classifiers, were easily hacked and elicited completions with no apparent tendency to be false.They cite this as an example of AI-generated labels not being adequate for red-teaming.Source: E EXPERIMENTS WITH CHATGPT LABELS… · p. 19
Loading 2306.09442v3…