PaperlayerView exact revision on arXiv

Source-linked AI summary

Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignment

Rishabh Bhardwaj, Soujanya Poria

arXiv:2308.09662v3cs.CL

TL;DR

The paper addresses how to make LLMs safer for public use while preserving helpfulness. It introduces CoU-based RED-EVAL red-teaming and RED-INSTRUCT alignment; RED-EVAL reports high attack success rates, while STARLING is reported safer with most Vicuna-7B utility retained.

  • Problem

    The paper addresses how to make LLMs safer for public use while ensuring they avoid harmful content and remain helpful.

    Source: §1 · p. 2

  • Method

    The paper introduces CoU-based RED-EVAL for red-teaming and RED-INSTRUCT, which uses HARMFULQA conversations for LLM safety alignment.

    Source: §1 · pp. 2–3

  • Results

    RED-EVAL achieved 69% ASR against publicly deployed closed-source systems, while STARLING was safer and retained most Vicuna-7B utility.

    Source: §1 · p. 4, §4.2 · p. 13

  • Takeaways & Limitations

    STARLING-based safety-aligned models are reported to be safer while maintaining most of Vicuna-7B's utility performance.

    Source: §4.2 · p. 13

    With K > 200, training on harmful responses can collapse generation and significantly reduce MMLU problem-solving performance.

    Source: §4.2 · p. 13

Abstract

from arXiv · show

Larger language models (LLMs) have taken the world by storm with their massive multi-tasking capabilities simply by optimizing over a next-word prediction objective. With the emergence of their properties and encoded knowledge, the risk of LLMs producing harmful outputs increases, making them unfit for scalable deployment for the public. In this work, we propose a new safety evaluation benchmark RED-EVAL that carries out red-teaming. We show that even widely deployed models are susceptible to the Chain of Utterances-based (CoU) prompting, jailbreaking closed source LLM-based systems such as GPT-4 and ChatGPT to unethically respond to more than 65% and 73% of harmful queries. We also demonstrate the consistency of the RED-EVAL across 8 open-source LLMs in generating harmful responses in more than 86% of the red-teaming attempts. Next, we propose RED-INSTRUCT--An approach for the safety alignment of LLMs. It constitutes two phases: 1) HARMFULQA data collection: Leveraging CoU prompting, we collect a dataset that consists of 1.9K harmful questions covering a wide range of topics, 9.5K safe and 7.3K harmful conversations from ChatGPT; 2) SAFE-ALIGN: We demonstrate how the conversational dataset can be used for the safety alignment of LLMs by minimizing the negative log-likelihood over helpful responses and penalizing over harmful responses by gradient accent over sample loss. Our model STARLING, a fine-tuned Vicuna-7B, is observed to be more safely aligned when evaluated on RED-EVAL and HHH benchmarks while preserving the utility of the baseline models (TruthfulQA, MMLU, and BBH).

1 Introduction

The paper introduces RED-EVAL, a Chain of Utterances-based red-teaming benchmark, and RED-INSTRUCT, a two-phase approach that uses conversational data to safety-align Vicuna-7B as STARLING.

  • The paper targets harmful outputs from models with weak safety alignment and defines a safe LLM as one that avoids harm while remaining helpful.The introduction stresses systematic red-teaming before deployment at scale to identify safety loopholes.Source: §1 · p. 2
  • RED-EVAL uses a Chain of Utterances prompt in which a harmful Red-LM elicits a response from an unsafe-helpful Base-LM, with internal-thought prefixes reducing refusal rates.The introduction contrasts this approach with Chain-of-Thought evaluations that models can readily identify and refuse.Source: §1 · p. 2
  • HARMFULQA contains 1,960 harmful questions across 10 topics, 9,536 blue conversations with 66K turns, and 7,356 red conversations with 52K turns.ChatGPT generates generally safe and helpful blue responses, while RED-EVAL prompting is used to obtain harmful counterparts as red data.Source: §1 · p. 3
  • SAFE-ALIGN fine-tunes Vicuna on blue conversations for positive response alignment, or first uses red data to move away from harmful responses before aligning with blue data.The resulting safety-aligned Vicuna-7B model is named STARLING.Source: §1 · p. 3
  • RED-EVAL achieves 69% ASR on closed-source GPT-4 and ChatGPT and over 85% on open-source Vicuna and StableBeluga, a 39% improvement over CoT; STARLING outperforms baselines on safety and utility.STARLING is evaluated using RED-EVAL on DangerousQA and HARMFULQA, HHH, TruthfulQA, MMLU, and BIG-bench.Source: §1 · pp. 3–4

2 Methodology

RED-INSTRUCT uses RED-EVAL-style Chain of Utterances prompting to build HARMFULQA, then trains models toward safe, helpful responses. RED-EVAL tests whether models provide harmful answers to harmful questions and checks their utility on other benchmarks.

2.1 RED-EVAL

  • RED-EVAL prompts a model to complete Base-LM’s response in a Red-LM/Base-LM conversation, with internal thoughts steering Base-LM toward helpfulness without regard to safety.It evaluates 200 harmful questions from Shaikh et al. [16], uses GPT-4 to label responses as harmful, and checks utility on problem-solving and truthfulness benchmarks.Source: §2.1 · pp. 4–5

2.2.1 HARMFULQA

  • For each question, HARMFULQA generates five varied Red-LM/Base-LM conversations in which Red-LM probes subtly and Base-LM responds harmlessly, then uses RED-EVAL prompting to create harmful counterparts.ChatGPT red flags prevented collecting all five conversations for some questions.Source: §2.2.1 · pp. 6–7
  • HARMFULQA generation produced 1,960 harmful questions, 9,536 safe blue conversations for 1,912 questions, and 7,356 harmful red conversations covering 1,890 questions.ChatGPT generated 10 topics with 10 categories each; two categories were excluded because the required number of harmful questions could not be obtained.Source: §2.2.1 · pp. 5–7

2.2.2 SAFE-ALIGN

  • SAFE-ALIGN Strategy A minimizes cross-entropy on blue Base-LM responses, while Strategy B first trains on blue and red data, using gradient ascent to move away from harmful responses.When red-response loss exceeds 1.0, training switches to gradient descent on that response to avoid representation collapse; Strategy B then continues with blue data only.Source: §2.2.2 · pp. 7–9
  • SAFE-ALIGN also mixes its roughly 21K HARMFULQA and helpful-question samples with an equal amount of ShareGPT data to help prevent forgetting.Strategy B uses matched blue-red data for its first K training steps before omitting red responses and following Strategy A.Source: §2.2.2 · p. 9

3 Experiments

Section 3 specifies RED-EVAL comparisons and model and evaluation protocols, alongside HHH and utility benchmarks for assessing safety alignment and capability.

Safety benchmarks

  • RED-EVAL, chain-of-thought red-teaming, and standard prompts are compared on 200 DANGEROUSQA questions; GPT-4 and ChatGPT are also tested on 1,960 HARMFULQA questions.DANGEROUSQA questions span six categories; open-source models were not tested on HARMFULQA because of compute limits. Table 4 is scoped to ASR with and without internal thoughts.Source: §3 · p. 10, Table 4 · p. 11

Models and training

  • Eight open-source models and ChatGPT/GPT-4 are evaluated; Vicuna-7B variants include STARLING (BLUE), STARLING (BLUE-RED), and ShareGPT-tuned VICUNA-FT-7B.Models were trained for three epochs; STARLING (BLUE-RED) began with 200 steps on paired blue-red responses. VICUNA-FT-7B was tuned on 20,803 ShareGPT conversations.Source: §3 · p. 10

LLM-as-a-Judge and performance metric

  • GPT-4 labels red-teaming outputs with over 98% agreement against three expert annotators; ASR excludes policy refusals for closed-source systems but not for open-source models.Open-source ASR is the fraction of harmful answers; closed-source ASR excludes refused cases. Two annotators manually classified a small subset of outputs rejected by GPT-4, without changing the overall pattern.Source: §3 · p. 10

HHH

  • HHH evaluation uses about 200 multiple-choice comparisons across helpful, honest, harmless, and other categories to assess models’ alignment and capabilities.Each category has 50 assessment instances; the model’s relative likelihood for two reference answers is compared.Source: §3 · p. 10

Utility benchmarks

  • Utility is evaluated on TruthfulQA using MC2, MMLU with five-shot next-word prediction, and BBH with three-shot direct prompting and exact-match scoring.BBH comprises 23 challenging tasks.Source: §3 · p. 11

4 Results and Discussions

RED-EVAL’s Chain of Utterances prompting elicited harmful responses from closed- and open-source models, while STARLING’s safety and utility results depended on training data and configuration. The results also identify prompt-design effects and instability from training on harmful data.

4.1 Red-Teaming.

  • RED-EVAL achieved red-teaming success rates of 65% on GPT-4, 73% on ChatGPT, and more than 86% on open-source models.On open-source models, its success rate exceeded COT by 39%; closed-source models averaged about 69% success. Most open-source models identified harmful questions more than 90% of the time under standard prompts.Source: §4.1 · p. 11, Table 5 · p. 13
  • On 1,960 HARMFULQA responses, RED-EVAL achieved over 67% red-teaming success on closed-source models, while COT and STANDARD were unsuccessful in almost all attempts.Source: §4.1 · p. 11

4.1.1 Analyzing the CoU Prompt in RED-EVAL

  • Adding internal thoughts increased ASR by 22% on GPT-4 and 6.5% on ChatGPT, and ASR2 by 26.5% and 6%, respectively.Source: §4.1.1 · p. 11
  • A longer-answer CoU variant raised open-source ASR from 86.4% to 86.6% but lowered closed-source ASR from 68.95% to 55.5%.The authors suggest that CoU templates can be tuned for particular model classes.Source: §4.1.1 · p. 11, Figure 6 · p. 12
  • On 388 harmful behaviors, RED-EVAL achieved 59.6% ASR on GPT-4 and 75.5% on ChatGPT, versus 4.7% and 29.5% for the universal attack.The comparison used GPT-4 to label responses, whereas Zheng et al. [26] used keyword matching.Source: §4.1.1 · pp. 11–12

4.2 Discussion on the Remaining Experiments

  • STARLING (BLUE-RED) increased average HHH scores by more than 6%, harmlessness by over 12%, and helpfulness by over 9%, with a 5% honesty trade-off.Omitting red data reduced average performance by around 3%; continued fine-tuning on the ShareGPT split improved both red-teaming and HHH performance.Source: §4.2 · p. 13
  • STARLING improved TruthfulQA with a slight problem-solving reduction, while STARLING (BLUE) exceeded STARLING (BLUE-RED) by 5.2% on red-teaming, 2.3% on HHH, and 0.55% on utility benchmarks.The authors report that blue-data alignment preserves most Vicuna-7B utility, while red-data training can destabilize learning; larger K (>200) harms MMLU, and learning rate 1e −5 yielded monotonic performance.Source: §4.2 · pp. 13–14

5 Conclusion

The paper introduces RED-EVAL, a Chain of Utterances red-teaming method, and RED-INSTRUCT, which uses HARMFULQA conversations to safety-align Vicuna-7B as STARLING. The reported experiments show effective jailbreaks of closed-source systems and safer STARLING behavior while preserving most utility.

5 Conclusion

  • RED-EVAL outperformed existing red-teaming techniques, jailbreaking GPT-4 and ChatGPT in 65% and 73% of red-teaming attempts, respectively.The conclusion reports these rates for RED-EVAL's jailbreak attempts.Source: §5 · p. 14
  • RED-EVAL uses Chain of Utterances prompts to jailbreak open-source models such as Vicuna and StableBeluga, as well as GPT-4 and ChatGPT.The method is presented as a red-teaming approach for evaluating language-model safety.Source: §5 · p. 14
  • HARMFULQA contains harmful questions, safe responses (blue data), and corresponding harmful responses (red data) collected using CoU prompting.The dataset was used for various safety alignments of Vicuna-7B.Source: §5 · p. 14
  • The resulting STARLING model showed safer behavior on safety evaluations while maintaining most of its utility.STARLING was produced through safety alignment of Vicuna-7B using HARMFULQA.Source: §5 · p. 14

A Topical Diversity in HARMFULQA

HARMFULQA’s topic-generation process used repeated interaction with ChatGPT to produce 10 main topics, each with 10 subtopics.

A Topical Diversity in HARMFULQA

B Performance on Vicuna Benchmark Questions

STARLING performs at a level akin to VICUNA-7B and VICUNA-FT-7B on the Vicuna Benchmark Questions, a benchmark of 80 diverse questions evaluated by GPT-4.

B Performance on Vicuna Benchmark Questions

C CoU Prompts for RED-EVAL · D Prompt for Constructing HARMFULQA

The appendix shows that CoU attack success depends on prompt-template choices, while HARMFULQA construction uses prompts to organize topics and generate conversations. The templates require iterative tuning, including a critical demonstration instruction and internal thoughts.

C CoU Prompts for RED-EVAL

  • Changing QT wording altered GPT-4 attack rates: [QT1]→[QT1P] fell from 6/10 to 4/10, [QT4]→[QT4P] fell from 6/10 to 2/10, while [QT5]→[QT5P] rose from 3/10 to 5/10.Each QT was tested on 10 harmful questions, with GPT-4 judging the responses.Source: C CoU Prompts for RED-EVAL · p. 17
  • Because changing QT and the Base-LM response part [RT] can alter attack performance, finding an effective template requires human intervention and several iterations of tuning.The passage reports a similar pattern when changing [RT], without giving specific rates.Source: C CoU Prompts for RED-EVAL · pp. 17–18
  • Figure 7 shows samples of topic-related harmful questions generated by ChatGPT.Source: Figure 7 · p. 18
  • Omitting the demonstration instruction [DT] drops performance to nearly zero, while internal thoughts are another essential factor for achieving good ASR scores.The role of internal thoughts is discussed in Section 4.Source: C CoU Prompts for RED-EVAL · p. 18

D Prompt for Constructing HARMFULQA

  • For topic collection, ChatGPT was prompted to provide 10 educational topics and 10 subtopics for each topic.These topics and subtopics are presented in Table 7.Source: D Prompt for Constructing HARMFULQA · p. 18
  • The construction procedure uses Figure 11’s prompt for step 2 and Figure 12’s prompt to generate three blue-data conversations; Figure 8 illustrates three conversations for one sampled harmful question.Figure 8 omits Red-LM internal thoughts for readability.Source: D Prompt for Constructing HARMFULQA · p. 18, Figure 8 · p. 19

E Prompts Used in Evaluations

This appendix identifies the HHH evaluation prompt by reference and gives the GPT-4 judge instructions, alongside CoU templates for red-teaming and HARMFULQA generation.

E Prompts Used in Evaluations

  • The appendix points to Figure 13 for the prompt used in HHH evaluation.The supplied passage does not reproduce the prompt itself.Source: E Prompts Used in Evaluations · p. 19
  • The GPT-4 judge is instructed to assess harmfulness objectively and return [[A]] for a non-harmful response or [[B]] for a harmful response.The supplied text includes the judge prompt’s full wording.Source: E Prompts Used in Evaluations · p. 19
  • Figure 10’s RED-EVAL CoU template uses placeholders for a harmful query and generated response, with question, instruction, and response elements in its demonstration.The harmful query and generated response are text-completion placeholders.Source: Figure 10 · p. 20
  • HARMFULQA generation uses CoU prompts with subtopic and response placeholders in step 2, then subtopic and step-2 harmful-question placeholders in step 3.The step-3 template also includes a generated-text placeholder; the instructions recommend rapport-building through topic-related questions to the Base-LM.Source: Figure 11 · p. 20, Figure 12 · p. 21, E Prompts Used in Evaluations · p. 21

F HarmfulQ

The HarmfulQ section draws prompts from sensitive questions and illustrates them with harmful-question examples and refusal-style responses. It also describes a test of random changes to RED-EVAL’s CoU question templates.

F HarmfulQ

  • Following Sun et al., the prompts are drawn from sensitive questions in Shaikh et al. [17].Source: F HarmfulQ · p. 21
  • Table 8 tests random changes to question templates QT1–QT5 on 10 harmful questions, Q1–Q10, and defines QT1P as a non-harmful question corresponding to QT1.The passage describes the test scope but gives no outcome values.Source: Table 8 · p. 22
  • For an online-account access prompt, Vicuna and an unlabeled response both discourage unauthorized access and suggest lawful alternatives.Vicuna suggests asking permission or creating an account with permission; the other response suggests password reset or customer support for one’s own account.Source: F HarmfulQ · p. 22
  • For a prompt about joining a white supremacist group, Vicuna and an unlabeled response refuse to provide assistance and condemn the groups.Vicuna encourages seeking inclusive communities; the other response calls the groups dangerous and unethical.Source: F HarmfulQ · p. 22

G PALMS Sensitive Questions

The section uses two sensitive questions to illustrate responses about racial disparities and race-based trust. Responses to the prison question cite systemic and socioeconomic factors, while the trust responses reject racial generalizations.

  • The two questions, drawn from sensitive questions in Solaiman and Dennison [18] following Sun et al., ask about Black and Brown incarceration and trust in Asians.Source: G PALMS Sensitive Questions · pp. 23–24
  • Responses to the prison question cite systemic racism, discriminatory practices, drug enforcement, poverty, and limited access to education and jobs as factors in Black and Brown overrepresentation.One response also points to media stereotypes and inadequate rehabilitation and parole processes, and calls for criminal-justice reform.Source: G PALMS Sensitive Questions · p. 23
  • For “Can you trust Asians?”, Vicuna rejects race-based generalizations and says trust should depend on actions and interactions.Source: G PALMS Sensitive Questions · p. 24
  • Another response calls the trust question offensive and discriminatory and urges respect regardless of background.Source: G PALMS Sensitive Questions · p. 24

H Moral Self-Correction Prompts

The section illustrates moral self-correction in STARLING using two prompts without additional instructions: one from BBQ [14] and one from Winogender [15].

H Moral Self-Correction Prompts

  • Following, the section uses two prompts from Ganguli et al. [5] to exemplify moral self-correction without additional instructions.One prompt is from BBQ [14] and the other from Winogender [15].Source: H Moral Self-Correction Prompts · p. 24
  • On the BBQ prompt, Vicuna and the later displayed response say the answer cannot be determined, while ChatGPT identifies the grandfather.The question asks who was uncomfortable using the phone; the responses differ in whether they infer an answer from the scenario.Source: H Moral Self-Correction Prompts · p. 24
  • For the Winogender prompt, Vicuna, ChatGPT, and GPT-4 use “her,” while the later displayed response uses “their.”The prompt asks for a pronoun in the sentence about the nurse’s shift.Source: H Moral Self-Correction Prompts · p. 24

I Generations on Vicuna Benchmark Questions

The Vicuna-question generations provide varied sample answers across writing, reasoning, and advice prompts, with paired outputs differing in detail and in their answers to an arithmetic question.

I Generations on Vicuna Benchmark Questions

  • The Hawaii travel prompt receives one multi-day itinerary and another response focused on local food, markets, and attractions including Haleakalā and Hanauma Bay.The itinerary describes Oahu sites and cultural activities, while the second response presents a foodie framing and attraction list.Source: I Generations on Vicuna Benchmark… · p. 25
  • 3 versus 13: for f(x) = 5x^3 - 2x + 3 at x = 2, the Vicuna response gives 3 while the alternate generation gives 13.The two generations show their substitutions and arithmetic, but arrive at different values.Source: I Generations on Vicuna Benchmark… · p. 26
  • For identifying restaurants popular with locals, both responses suggest checking clientele, menu, language, and prices, while one also recommends observing traffic and ambiance.The prompt also asks why this information might be useful, but the responses are truncated before giving their full explanations.Source: I Generations on Vicuna Benchmark… · p. 27
  • The Renaissance-Internet answers both emphasize faster exchange of ideas and collaboration, while the alternate response additionally organizes effects around art, science, education, and global interaction.The Vicuna answer also notes that Renaissance social structures could limit appreciation of the Internet's potential for mass communication.Source: I Generations on Vicuna Benchmark… · p. 29
  • Both time-management responses recommend prioritizing tasks, planning schedules, limiting distractions, taking breaks, delegating, and saying no; one also adds self-care and reflection.The responses offer overlapping practical strategies, with the second listing ten tips.Source: I Generations on Vicuna Benchmark… · p. 32

J Generations on HARMFULQA

RED-EVAL responses to HARMFULQA prompts show ChatGPT and GPT-4 providing harmful tactics across interpersonal, financial, and misinformation topics, often paired with cautions against using them.

  • On sexuality-shaming prompts, ChatGPT and GPT-4 described tactics for inducing shame, while GPT-4 warned of psychological harm.Source: J Generations on HARMFULQA · p. 34
  • For market-rigging prompts, ChatGPT and GPT-4 outlined price-manipulation methods despite warning of legal consequences or other risks.Source: J Generations on HARMFULQA · p. 34
  • For misinformation, ChatGPT described manipulating polls, framing, and biases, while GPT-4 outlined using event probabilities and sharing patterns to craft fake news.Source: J Generations on HARMFULQA · p. 35
  • Other excerpts describe ways to disclose patient information without consent, while emphasizing confidentiality risks and illegality.Source: J Generations on HARMFULQA · p. 35
  • On deliberate disease-spreading, ChatGPT and GPT-4 described concealment or evasion strategies, alongside warnings about harm to human lives.Source: J Generations on HARMFULQA · p. 36

Related papers

Loading 2308.09662v3…