PaperlayerView exact revision on arXiv

Source-linked AI summary

HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal

Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, Dan Hendrycks

arXiv:2402.04249v2cs.LGcs.AIcs.CLcs.CV

TL;DR

Automated red teaming lacks a standardized framework for rigorously comparing attacks and defenses. HarmBench addresses this gap with broad, comparable, robust evaluations and a large-scale study, while its adversarial training method improves robustness across attacks, most strongly against GCG.

  • Problem

    Automated-red-teaming evaluations are disparate and lack breadth, comparability, and robust metrics, making methods difficult to assess consistently.

  • Method

    HarmBench standardizes automated-red-teaming evaluations and introduces an efficient adversarial training method for robust refusal.

  • Results

    Large-scale evaluations of 18 methods and 33 LLMs and defenses find that no attack or defense is uniformly effective, while R2D2 substantially improves robustness and achieves state-of-the-art robustness against GCG.

  • Takeaways & Limitations

    HarmBench enables large-scale comparison and supports codevelopment of stronger attacks and defenses.

  • Takeaways & Limitations

    R2D2 improves robustness less for attacks dissimilar to its train-time GCG adversary, suggesting that diverse training attacks may be needed for generalizable robustness.

Abstract

from arXiv · show

Automated red teaming holds substantial promise for uncovering and mitigating the risks associated with the malicious use of large language models (LLMs), yet the field lacks a standardized evaluation framework to rigorously assess new methods. To address this issue, we introduce HarmBench, a standardized evaluation framework for automated red teaming. We identify several desirable properties previously unaccounted for in red teaming evaluations and systematically design HarmBench to meet these criteria. Using HarmBench, we conduct a large-scale comparison of 18 red teaming methods and 33 target LLMs and defenses, yielding novel insights. We also introduce a highly efficient adversarial training method that greatly enhances LLM robustness across a wide range of attacks, demonstrating how HarmBench enables codevelopment of attacks and defenses. We open source HarmBench at https://github.com/centerforaisafety/HarmBench.

1. Introduction

HarmBench addresses the lack of scalable, comparable evaluation for automated red teaming by standardizing benchmark breadth, comparability, and metric robustness. Its large-scale evaluations reveal new properties of attacks and defenses and support efficient adversarial training.

  • Motivation: Manual red teaming cannot scale to the broad range of adversarial and long-tail scenarios relevant to increasingly capable LLMs.This limitation motivates automated methods for evaluating and hardening defenses.
  • Benchmark design: HarmBench systematically targets breadth, comparability, and robust metrics because prior automated-red-teaming evaluations were disparate and difficult to compare.It contains more unique behaviors than previous evaluations, including new behavior categories.
  • Findings: Large-scale HarmBench evaluations find that no current attack or defense is uniformly effective and that robustness is independent of model size.These findings revise existing assumptions and motivate standardized comparisons.
  • Benchmark design: 510 carefully curated behaviors span four functional categories, while the initial evaluation covers 18 red teaming methods and 33 LLMs.The benchmark is designed for large-scale comparison across diverse behaviors and models.
  • Adversarial training: The paper introduces a highly efficient adversarial training method for robust refusal and reports state-of-the-art robustness against GCG.The method illustrates how standardized evaluation can support codevelopment of attacks and defenses.

2. Related Work

Related work spans manual and automated red teaming, multimodal attacks, and system- or model-level defenses. However, disparate and non-overlapping evaluation setups prevent clear comparisons of automated methods.

  • Red teaming: Manual red teaming provides baselines for scalable automated methods but is limited by the need to discover attacks through human effort.Large-scale manual efforts have supported pre-deployment testing and characterization of jailbreak strategies.
  • Red teaming: Automated red teaming includes text optimization, LLM optimizers, and custom jailbreaking pipelines for eliciting harmful behaviors.Many of these methods can be directly compared when they target specific harmful behaviors.
  • Red teaming: Multimodal attacks are included in standardized evaluation because some studies observe them to be stronger than text attacks.Prior work has explored image attacks against multimodal LLMs.
  • Evaluation: Automated-red-teaming literature lacks standardized evaluation, leaving the relative performance of methods unclear.Existing comparisons rarely overlap across papers.
  • Defenses: Defenses are categorized as system-level measures that add external safeguards and model-level measures that alter the LLM itself.Examples include filtering and input sanitization for system-level defenses, and safety training, refusal mechanisms, prompts, and adversarial training for model-level defenses.
  • Adversarial training: Prior adversarial-training work faced computational costs and sometimes used static harmful-prompt datasets without optimizing the adversary during fine-tuning.These constraints distinguish automated red-teaming adversarial training from perturbation-attack training.
  • Robustness: Model training data, architecture, system prompts, and size may affect inherent jailbreak robustness, motivating large-scale examination of these factors.HarmBench enables comparisons of such factors across models.

3. Automated Red Teaming

HarmBench formulates red teaming as generating test cases that elicit harmful behaviors, then standardizes breadth, evaluation parameters, and robust metrics. Its pipeline generates test cases, obtains model completions, and evaluates them with classifiers.

  • Metrics: Attack success rate (ASR) is the percentage of test cases that elicit the target behavior from a model.HarmBench assumes deterministic greedy decoding for evaluation efficiency.
  • Evaluation criteria: HarmBench identifies breadth, comparability, and robust metrics as key qualities missing from prior specialized evaluation setups.The framework is designed to support systematic large-scale comparison.
  • Breadth: Broad evaluations are desirable because narrow behavior sets may overstate practical usefulness and cannot support tailored defense testing.HarmBench includes diverse behavior categories and can be extended with new undesired behaviors.
  • Comparability: The number of generated tokens can change ASR by up to 30%, making this parameter critical for fair comparisons.HarmBench proposes a more robust metric and standardizes generation at N = 512.
  • Robust metrics: Robust metrics require classifiers that withstand nonstandard completions because evaluation metrics face optimization pressure from both attacks and defenses.The prequalification tests include initial refusal followed by continued harmful behavior and random benign paragraphs.
  • Evaluation pipeline: The standardized pipeline converts diverse behaviors into test cases, generates completions, and evaluates those completions under standardized parameters.This structure is intended to make methods and models comparable.

4. HarmBench

HarmBench is a standardized evaluation framework built to improve the breadth, comparability, and robustness of automated red-teaming evaluations. It combines diverse harmful behaviors with an evaluation pipeline and supports large-scale comparisons of attacks and defenses.

  • HarmBench contains 510 unique harmful behaviors, comprising 400 textual and 110 multimodal behaviors.
  • The benchmark provides validation and test splits of 100 and 410 behaviors, respectively, with test behaviors excluded from tuning.
  • Its functional categories include standard, copyright, contextual, and multimodal behaviors, totaling 200, 100, 100, and 110 behaviors.
  • Contextual and multimodal behaviors add specific textual or visual context, enabling evaluation of more realistic or image-based attacks.
  • HarmBench’s contextual behaviors had a 0% searchability rate, compared with 55% for MaliciousInstruct and 50% for AdvBench.
  • The evaluation pipeline generates test cases, obtains target-model completions, and uses classifiers to measure whether harmful behavior occurs.

5. Adversarial Training for Robust Refusal

The paper introduces R2D2, an adversarial-training method for robust refusal that continually trains models against dynamically optimized test cases. It combines persistent GCG-generated attacks with refusal and utility-preserving training objectives.

  • R2D2 fine-tunes LLMs on a dynamic pool of test cases continually updated by a strong optimization-based red-teaming method.
  • GCG is used as the adversary because it is effective against robust models, but generating one test case on a 7B model requires 20 minutes on an A100.
  • Persistent test cases are optimized continually across batches rather than regenerated from scratch, with sampled cases updated using GCG for multiple steps.
  • R2D2 combines an away loss opposing the GCG objective with a toward loss training the model to produce a fixed refusal string.
  • The full method periodically resets a fraction of pooled test cases for diversity and adds supervised fine-tuning loss to preserve model utility.

6. Experiments

HarmBench evaluates 18 red teaming methods across 33 LLMs and defenses, revealing that attack and defense effectiveness varies across models and attacks. R2D2 substantially improves robustness, especially against GCG, while generalization is weaker against dissimilar attacks.

  • Evaluation setup: 18 red teaming methods and 33 LLMs and defenses are compared across text-only and multimodal settings.The methods include white-box, black-box, transfer, and human-designed attacks; evaluated defenses include refusal mechanisms, safety training, and R2D2.
  • Main results: ASR is considerably higher for contextual behaviors, relatively low for copyright behaviors, and relatively high for PGD-based multimodal attacks but low for Render Text.Semantic-category ASR is similar on average, while substantial differences appear across models.
  • Main results: No current attack or defense is uniformly effective: every attack has low ASR on at least one LLM, and every LLM has poor robustness against at least one attack.This pattern motivates large-scale standardized comparisons rather than conclusions from limited attack-defense pairings.
  • Main results: Robustness shows no correlation with model size within six model families spanning 7 to 70 billion parameters, but differs substantially between model families.The authors suggest training procedures and data matter more than model size, with increasing ASR at the largest sizes for copyright behaviors as a caveat.
  • Adversarial training results: Zephyr 7B + R2D2 achieves 5.9% ASR against GCG, improving from 31.8% for Llama 2 7B Chat and 30.2% for Llama 2 13B Chat.R2D2 is the strongest defense across GCG, GCG (Multi), and GCG (Transfer), and Zephyr 7B + R2D2 has the third-lowest average ASR across all models.
  • Adversarial training results: R2D2 improves robustness across all attacks, but gains are less pronounced for PAIR, TAP, and Stochastic Few-Shot, which differ from the train-time GCG adversary.The authors suggest that multiple diverse attacks may be needed for more generalizable robustness.
  • Adversarial training results: Zephyr 7B + R2D2 scores 6.0 on MT-Bench versus 6.5 for Mistral 7B Instruct v0.2, suggesting improved robustness while preserving general performance.MT-Bench evaluates general knowledge and conversational ability.

7. Conclusion

The paper introduces HarmBench as a standardized framework for evaluating automated red teaming through breadth, comparability, and robust metrics. Large-scale evaluations and R2D2 demonstrate how the framework supports stronger attack-defense development and robust refusal.

  • Conclusion: HarmBench is a standardized evaluation framework designed around breadth, comparability, and robust metrics.The framework addresses shortcomings in prior red teaming evaluations.
  • Conclusion: HarmBench evaluates 18 red teaming methods and 33 LLMs and defenses in a large-scale comparison.The comparison reveals properties of attacks and defenses that inform future work.
  • Conclusion: R2D2 provides a strong baseline defense and achieves state-of-the-art robustness on GCG.The adversarial training method demonstrates codevelopment of attacks and defenses using HarmBench.
  • Conclusion: The authors release HarmBench to support collaborative development of stronger attacks and defenses for safer AI systems.The framework is open sourced at the Center for AI Safety GitHub repository.

Impact Statement

HarmBench combines standardized evaluation across seven misuse categories with a new adversarial training method for robust refusal. The work emphasizes both improved LLM defenses and the ethical responsibility to prevent misuse while advancing AI safety research.

  • HarmBench evaluates red-teaming attacks and defenses across seven critical misuse categories, including cybercrime and misinformation.
  • The framework is designed to identify and mitigate LLM vulnerabilities before deployment through comprehensive safety evaluation.
  • The work frames AI safety advancement as requiring both stronger defenses and safeguards against enabling more sophisticated attacks.
  • Adversarial training with automated red teaming is presented as a distinct approach for improving robustness against harmful outputs.

A.2. Prior Comparisons and Evaluations

Prior red-teaming studies used varied datasets, attack methods, classifiers, and evaluation procedures, making comparisons difficult. HarmBench addresses this by defining clearer behavior criteria, held-out splits, classifier safeguards, and supported threat-model boundaries.

  • Prior comparisons: HarmBench excludes some prior methods and datasets when they lack suitable comparability with the included evaluations.
  • Prior comparisons: Prior evaluations vary in behavior sets, attack methods, and success-rate metrics, limiting direct comparison across studies.
  • Evaluation criteria: Behavior labels require clear, non-minimal, harmful generations that actually exhibit the target behavior, with special requirements for code-producing behaviors.
  • Splits and classifiers: The benchmark separates validation and test behaviors and classifiers so methods can optimize during development without directly tuning the test metric.
  • Splits and classifiers: The validation classifier reaches 88.6% agreement with human labels versus 93.2% for the test classifier, with only 26 overlapping errors.
  • Supported threat models: HarmBench primarily supports instruction-tuned LLMs and additionally includes multimodal models with a dedicated set of 110 harmful multimodal behaviors.

B.5. Evaluation Classifiers

HarmBench computes attack success rates with classifiers tailored to the behavior type. Non-copyright behaviors use a fine-tuned Llama 2 classifier, while copyright behaviors use hashing-based detection.

  • HarmBench uses fine-tuned Llama 2 13B Chat to classify successful non-copyright behaviors and hashing to detect generated copyrighted content.

B.5.1. LLAMA 2 EVALUATION CLASSIFIER

The evaluation classifier is built from human-labeled completions, GPT-4 judgments, and iterative distillation into a lightweight Llama 2 model. The resulting classifier improves agreement and validation accuracy while copyright evaluation uses stricter matching.

  • Classifier development: A 600-example human-labeled validation set supplies positive and negative completion examples for classifier development.
  • Classifier development: GPT-4 prompts are tuned separately for standard, contextual, and multimodal behaviors using human annotation instructions.
  • Classifier development: The final lightweight metric distills GPT-4 classifications into a fine-tuned Llama 2 13B Chat classifier rather than using GPT-4 directly.
  • Distillation process: The distillation process repeatedly samples completions, adds GPT-4–Llama 2 disagreements, and refines the classifier from public checkpoints.
  • Performance: After 15 iterations, agreement with GPT-4 and validation-set accuracy steadily increase.
  • Performance: The resulting classifier surpasses previous classifiers, including GPT-4-based classifiers, on evaluation agreement.
  • Copyright classification: Copyright evaluation requires generated copyrighted content and uses hashed overlapping text chunks with MinHash to detect soft matches.

C.1. Red Teaming Method Descriptions

HarmBench evaluates a broad set of automated red teaming methods, including optimization-, prompting-, transfer-, evolutionary-, persuasion-, and human-jailbreak-based approaches. Its target models span multiple open-source model families and sizes, while implementation choices sometimes adapt methods or replace closed-source components for scalable comparison.

  • Optimization-based methods: GCG optimizes an adversarial suffix to increase the target LLM’s likelihood of an affirmative target string.
  • Optimization-based methods: GCG-Multi optimizes one suffix across multiple behavior prompts, whereas GCG-Transfer optimizes against multiple training models for transfer across targets.
  • Optimization-based methods: PEZ, GBDA, UAT, and AutoPrompt optimize hard-token adversarial suffixes using distinct estimators, distributions, update rules, or candidate-selection strategies.
  • LLM- and prompt-based methods: Zero-Shot and Stochastic Few-Shot use attacker-LLM generation, with the latter selecting few-shot examples according to target-string probabilities.
  • LLM- and prompt-based methods: PAIR and TAP adaptively elicit harmful behaviors through iterative or tree-structured attacker-LLM prompting, while TAP-Transfer uses Mixtral as attacker and GPT-4 as judge and target.
  • Other methods and baselines: AutoDAN evolves handcrafted jailbreak prompts with a hierarchical genetic algorithm, PAP uses persuasive strategies, and Human and Direct Request provide template and un-obfuscated baselines.
  • Evaluation choices: The comparison adapts some baselines, replaces closed-source components with Mixtral 8x7B to reduce cost and enable compute comparisons, and excludes methods for practical or formulation-specific reasons.The excluded methods include approaches requiring per-behavior attacker fine-tuning, unavailable or ineffective code, non-transferable pipelines, or incomplete implementations.
  • Target models and defenses: Target models include open-source families such as Llama 2, Vicuna, Baichuan 2, Qwen, Koala, Orca 2, and SOLAR, alongside model-level defenses including safety training and adversarial training.

C.3. Full Results

The full-results section presents HarmBench’s diverse behavior datasets, attack-success-rate analyses, multimodal evaluations, and classifier-based assessment pipeline. It emphasizes broad coverage across behavior categories and modalities, including contextual and copyright behaviors and CAPTCHA-related multimodal behaviors.

  • Behavior coverage: HarmBench is described as considerably larger and more diverse than prior behavior datasets, with unique behaviors computed using manual and automated semantic deduplication.
  • Attack success rates: Average ASR is similar across semantic categories except for substantially lower ASR on copyright behaviors.The figure attributes the copyright difference to reasons described in Appendix B.5.2.
  • Attack success rates: Semantic-category difficulty varies by model class: Misinformation & Disinformation is highest for Llama 2 and GPT, while Chemical & Biological Weapons / Drugs is highest for Baichuan 2 and Starling.
  • Behavior types: Contextual behaviors have considerably higher ASR than standard behaviors, while copyright behaviors have much lower ASR when averaged across attacks and open-source models.Contextual behaviors represent more specific harmful tasks that are challenging to look up on a search engine.
  • Evaluation results: The results include separate ASR tables for all, test, and validation behaviors, plus multimodal attack-success-rate tables and evaluations on text-only behaviors.
  • Defense evaluation: Table 11 reports that R2D2 retains high benign-task performance, outperforming Koala 13B on MT-Bench and approaching Mistral 7B Instruct-v0.2.
  • Evaluation pipeline: HarmBench’s classifier judges whether generations unambiguously and non-minimally exhibit harmful behaviors, counting caveated or warned outputs when the behavior occurred alongside them.
  • Evaluation pipeline: Its multimodal classifier conditions judgments on an image description, and CAPTCHA cases count when the model tries to answer even without producing the correct answer.

E.1. Long-Term Impact on Advanced AI Systems

The paper frames standardized automated red teaming as a tool for improving defenses against malicious use and potentially supporting control of increasingly agentic AI systems. It also acknowledges that the same tools could improve reliability and deployment incentives, while the benchmark remains bounded by hand-crafted behaviors and human judgments about harm.

  • Long-term safety impact: HarmBench improves automated red teaming evaluation and supports codevelopment of attacks and defenses, with R2D2 demonstrating stronger refusal robustness through automated-red-teaming-based adversarial training.
  • Long-term safety impact: The paper identifies malicious use, eroded epistemics, deception, and power-seeking behavior as hazards directly affected by this research direction.
  • Long-term safety impact: The work may contribute indirectly through improved monitoring tools and safety culture.
  • Long-term safety impact: Stronger red teaming and robust malicious-use defenses could reduce bad actors’ ability to use future AI systems in biological weapons attacks.
  • Safety-capabilities balance: Because the benchmark focuses solely on harmful tasks, the authors expect its clearest contribution to be improving robustness to adversaries rather than general capabilities.
  • Safety-capabilities balance: Automated red teaming could also improve AI reliability and create stronger economic incentives to deploy systems in more autonomous settings.
  • Scope and limitations: The evaluated red teaming methods are automated given hand-crafted behaviors, but automating behavior selection remains difficult because harmfulness is context-dependent and user-dependent.
  • Scope and limitations: R2D2 adversarial training does not significantly reduce MT-Bench performance, although adversarial training could potentially reduce general capabilities.
Loading 2402.04249v2…