PaperlayerView exact revision on arXiv

Source-linked AI summary

LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yet

Nathaniel Li, Ziwen Han, Ian Steneker, Willow Primack, Riley Goodside, Hugh Zhang, Zifan Wang, Cristina Menghini, Summer Yue

arXiv:2408.15221v2cs.LGcs.CLcs.CRcs.CY

TL;DR

LLM defenses are mainly tested against single-turn automated attacks, leaving their robustness in realistic multi-turn use insufficiently evaluated. The paper uses expert human red teamers and releases the resulting MHJ dataset, finding that current defenses can be bypassed in multi-turn conversations and that human attacks outperform automated ensembles on HarmBench.

  • Problem

    LLM defenses are primarily evaluated against single-turn adversarial attacks, an insufficient threat model for malicious use in deployment involving multi-turn interactions.

  • Method

    The paper commissions expert human red teamers to conduct multi-turn jailbreaks through chat interfaces, reviews successful submissions, and compiles them into the MHJ dataset.

  • Results

    Human red teamers achieve 19%–65% higher ASR than an ensemble of automated attacks on HarmBench and bypass existing safeguards in multi-turn conversations.

  • Takeaways & Limitations

    Robustness evaluations should expand beyond single-turn attacks to longer multi-turn conversations, especially as LLMs are used in longer-context and tool-integrated applications.

  • Takeaways & Limitations

    The paper cautions that its results are not a one-to-one comparison establishing human red teaming as superior to automated attacks.

Abstract

from arXiv · show

Recent large language model (LLM) defenses have greatly improved models' ability to refuse harmful queries, even when adversarially attacked. However, LLM defenses are primarily evaluated against automated adversarial attacks in a single turn of conversation, an insufficient threat model for real-world malicious use. We demonstrate that multi-turn human jailbreaks uncover significant vulnerabilities, exceeding 70% attack success rate (ASR) on HarmBench against defenses that report single-digit ASRs with automated single-turn attacks. Human jailbreaks also reveal vulnerabilities in machine unlearning defenses, successfully recovering dual-use biosecurity knowledge from unlearned models. We compile these results into Multi-Turn Human Jailbreaks (MHJ), a dataset of 2,912 prompts across 537 multi-turn jailbreaks. We publicly release MHJ alongside a compendium of jailbreak tactics developed across dozens of commercial red teaming engagements, supporting research towards stronger LLM defenses.

1. Introduction

Existing LLM defenses are mainly tested with single-turn automated attacks, leaving their robustness in realistic multi-turn use insufficiently examined. The paper studies human multi-turn jailbreaks, finds substantial vulnerabilities, and releases the resulting MHJ dataset and tactics.

  • Single-turn automated evaluations are an insufficient threat model for malicious deployment, where users typically query LLMs across multiple turns.
  • Human red teamers use multi-turn chat interactions to elicit harmful responses, with reviewers and a language-model classifier evaluating submissions.
  • MHJ contains 2,912 prompts across 537 multi-turn conversations, with metadata, red-teamer design comments, and associated jailbreak tactics.
  • Human and automated ASRs require cautious comparison because their setups and threat models differ, although the human evaluation was controlled to provide a conservative estimate.
  • 19% to 65% higher ASR was achieved by multi-turn human red teamers than an ensemble of automated attacks on HarmBench.
  • Human jailbreaking also recovered dual-use biosecurity knowledge from a model subjected to machine unlearning.

2. Background and Related Work

Prior work develops automated attacks and defenses for LLMs, while human red teaming remains costly and uncommon at scale. This motivates comparing automated robustness benchmarks with human red-team performance.

  • Attacks Against LLMs: LLM attacks include transformations, gradient-based optimization, and LLM-assisted prompt engineering to break defenses.
  • Defenses for LLMs: LLM defenses include refusal training, adversarial training, and representation-level changes intended to improve robustness.
  • Human Red Teaming for LLMs: Human red teaming identifies vulnerabilities with limited model access, and researchers have developed taxonomies of human jailbreak tactics.
  • Human Red Teaming for LLMs: Human red teaming is rarely conducted at large scale outside industry because of its prohibitive cost.
  • The gap between automated attacks and human red teaming highlights the need for baselines supporting stronger automated attacks and robust defenses.

3. Multi-Turn Human Jailbreaking

The paper models real-world misuse through black-box, multi-turn chat interactions and uses a staged human red-teaming pipeline to develop and validate jailbreaks. Independent attempts, human review, and GPT-4o filtering are combined to reduce false positives.

  • 3.1 Threat Model: The threat model uses black-box chat interfaces where red teamers interact with models over multiple turns, without access to internals or output probabilities.Red teamers may toggle among temperatures 0, 0.5, and 1.
  • 3.2 Human Red Teaming Pipeline: Human jailbreaks are developed through diverse attempts and verified by the producing red teamer, another reviewer, and GPT-4o.Potential false positives can trigger an additional independent attack before further validation.
  • 3.2 Human Red Teaming Pipeline: The pipeline permits up to three independent human attackers to conduct multi-turn jailbreak attempts within fixed time limits.The first two attackers attempt the behavior independently; a third may reattempt rejected cases from scratch.
  • 3.2 Human Red Teaming Pipeline: The validation stage checks successful attempts with human reviewers and a language-model harm classifier to reduce false positives.Reviewer 1 verifies alignment with the HarmBench classifier prompt, while GPT-4o performs final filtering for precision.

4. Evaluation

The evaluation compares human multi-turn jailbreaks with automated attacks across LLM defenses and benchmarks, finding that human attacks consistently outperform automated attacks. The study also examines unlearning robustness, attack-time patterns, tactic vulnerabilities, and the release of MHJ for future research.

  • 4.1 Experimental Setup: The study evaluates four defenses on 240 HarmBench harmful behaviors and examines attacks against machine unlearning using 43 open-ended WMDP-Bio questions.HarmBench excludes copyright questions, while WMDP-Bio questions are converted from multiple choice to free response.
  • 4.1 Experimental Setup: Six automated attacks are evaluated, and an ensemble counts a behavior as successful when any attack succeeds, providing an upper bound on automated ASR.The attacks include four black-box and two white-box methods.
  • 4.2 Results: Human jailbreaks exceed the ensemble automated ASR by 19.6%–65.4% on three open-source defenses and reach 70.4% ASR against CYGNET.All prior white- and black-box attacks achieved 0% ASR against CYGNET in the cited original results.
  • 4.2 Results: Automated attack ASR does not reliably proxy robustness to human jailbreaks, as CYGNET is more robust to automated attacks than RR but easier for red teamers to jailbreak.Human and automated attacks also differ across HarmBench semantic categories.
  • 4.2 Results: Human attacks achieve 39.6% higher ASR than the best automated attack and 25.6% higher than the six-attack ensemble against the RMU-unlearned model.The result concerns recovery of dual-use biosecurity knowledge on WMDP-Bio.
  • 4.2 Results: Recovering unlearned biosecurity knowledge remains challenging, with human red teaming showing lower ASR than HarmBench despite an average successful submission time of 20.5 minutes.The authors hypothesize that domain-specific experience may be needed and leave domain-specific adversaries for future work.
  • 4.2 Results: 91.62% of successful HarmBench submissions and 85.7% of successful WMDP-Bio submissions require more than one conversation turn.The paper notes that some tactics, including Hidden Intention Streamline, depend entirely on multi-turn interaction.
  • 4.3 Release: The released MHJ dataset and tactic taxonomy support evaluating or training automated multi-turn red teaming and designing attacks that mimic successful human tactics.MHJ contains metadata and red teamer commentary, while sensitive and export-controlled information was removed before release.

5. Discussion

The discussion argues that current defenses and evaluations underrepresent multi-turn, longer-context threats, while cautioning that human and automated ASRs are not directly comparable. It motivates broader threat models and stronger automated attacks alongside human red teaming.

  • Threat-model gaps: Single-turn adversarial training and refusal training may leave defenses vulnerable to multi-turn human jailbreaks.The paper describes refusal training as often short and notes that even training further along completions remains insufficient.
  • Threat-model gaps: Longer-context deployments involving browsers, terminals, code, and symbols create additional out-of-distribution safety challenges.
  • Comparability limits: Human and automated ASRs warrant cautious interpretation because their pipelines differ in time limits, access, turns, temperature, personnel, and evaluation procedures.Human red teamers receive multiple turns and some temperature diversity but no model-internal access, whereas automated attacks may use model internals and single-turn settings.
  • Comparability limits: Human red teaming is orders of magnitude more costly than current automated attacks, so cost-controlled robustness analyses may produce different results.
  • Implications: The results should not be treated as a one-to-one comparison or proof that human red teaming is superior.The supported conclusion is narrower: robustness evaluations need more lifelike threat models and stronger multi-turn automated attacks.

6. Conclusion

The paper releases MHJ to expand LLM robustness evaluation beyond single-turn automated attacks. Its conclusion is that defenses robust on those evaluations remain vulnerable to multi-turn human jailbreaks.

  • Dataset: MHJ contains 2,912 prompts across 537 multi-turn jailbreak conversations for improving LLM defense robustness.
  • Threat model: The paper expands LLM red-teaming threat models to include multi-turn jailbreaks, a lifelike assumption for malicious use that existing evaluations rarely employ.
  • Conclusion: Defenses showing strong robustness against single-turn automated attacks are not robust against multi-turn human jailbreaks.
  • Resources: The release also includes jailbreak tactics collected across dozens of commercial LLM red-teaming engagements.

7. Ethics

The ethics discussion weighs the research benefits of releasing MHJ against risks of enabling malicious use. The authors reduce those risks by removing sensitive material and checking export-control compliance.

  • Risk assessment: The authors judge MHJ to pose low marginal risk because many other manual jailbreak datasets are already widely disseminated.
  • Risk mitigation: They removed model completions and jailbreaks containing sensitive information before release.
  • Compliance: Legal counsel verified MHJ’s compliance with applicable U.S. export-control requirements, including ITAR and EAR.
  • Research process: The team received permission to red-team API-access models and disclosed results to the examined defenses’ authors.

A. HarmBench Evaluation

The HarmBench evaluation combines human red-team results, automated-attack verification, semantic-category breakdowns, and qualitative reports across defenses. Its verification pipeline uses GPT-4o as a filter while human review determines accepted harmful jailbreaks.

  • Attack success rates are broken down across six HarmBench semantic categories, with category sizes ranging from 18 to 54 behaviors.The categories are misinformation/disinformation, illegal, cybercrime, chemical/biological, harassment/bullying, and harmful.
  • GPT-4o serves as a first-pass filter for automated attacks because it has strong recall but weaker precision against human labels.Human reviewers examine responses flagged positive, and harmfulness requires agreement between GPT-4o and a human verifier.
  • The human-jailbreak pipeline uses two human reviewers, while automated attacks require one human reviewer, producing a conservative lower bound for human ASR.
  • RR’s gibberish defense strongly polices responses but can also disrupt legitimate requests and terminate otherwise normal conversations after trigger words or a single refusal.
  • Qualitative red-team reports identify tactic-specific weaknesses, including trigger-word substitution, redefinition, obfuscation, acting innocent, and hidden-intention strategies.One report describes assigning new words to forbidden definitions before asking the desired question.

C.2 DERTA

DERTA’s red-team reports describe a defense that is comparatively easier to break and can remain recoverable after refusals. Reported tactics include direct requests, Opposite Day, dialogue injection, fictionalization, and persistence.

  • DERTA could often be talked back into helpfulness after refusing, unlike a gibberish defense that forced conversation restarts.
  • Other reported tactics included direct requests, fictionalization, persistence, and reframing prompts through repeated interaction.One red teamer noted that repeated requests had not been observed on other models, though the comparison was uncertain.
  • Opposite Day was reported as a particularly effective tactic, while dialogue injection was also used and later disallowed.
  • DERTA was described as more resistant to fictionalization than some prior models, while confusion, lying, and refusal remained reported behaviors.
  • DERTA was described as easier to break and slightly more helpful than the previous model, though less helpful than an unaltered model.

C.3 LAT

LAT received mixed red-team assessments: it was difficult to break and resistant to several familiar tactics, yet some stylized, fictionalized, or tone-based prompts were reported as useful. Its refusal behavior could also persist into harmless questions.

  • LAT could truncate harmful responses and sometimes refuse harmless questions after refusing a harmful request.
  • LAT was reported as resistant to Opposite Day and fictionalization, while acting innocent, authority appeals, functionalization, and educational framing sometimes worked.
  • A red-team report characterized LAT and related defenses as overly safe, limiting user-friendliness and motivating users to seek other solutions.
  • LAT was described as difficult to break, with some reports calling it stronger than most previously tested models.

C.4 CYGNET

CYGNET was characterized as resistant overall but vulnerable to selected prompt formats and persistence-based strategies. Reports also describe response truncation, short circuits, and false or scholarly-sounding framing as recurring features of red-team interaction.

  • CYGNET was generally described as difficult to break and more resistant than some alternatives, though particular stylized inputs remained effective.
  • CYGNET could punt or short-circuit, but reports also noted that its conversations sometimes remained coherent and recoverable after refusals.
  • Stylized input, Opposite Day, direct requests, false data submission, and character-based prompting were reported as useful against CYGNET.
  • False epistemology uses fake scholarly-sounding material to make harmful or false concepts appear harmless or true to the model.

C.5 RMU

The RMU defense was highly resistant to many common jailbreak tactics but remained vulnerable to obfuscated terminology and cumulative multi-turn strategies. Its strict gibberish response also impaired legitimate adjacent requests.

  • The defense was strong against keywords, crowding, and fictionalization, but weaker against obfuscated syntax, indirect reasoning, and cumulative multi-turn attacks.
  • The defense’s gibberish behavior was highly restrictive, sometimes blocking harmless questions related to the protected topic.
  • Obfuscation of Syntax, Indirect Logic, Hidden Intention Streamline, and Priming were identified as effective tactics against the defense.
  • Assigning new code words to concepts that triggered gibberish responses was fairly effective for bypassing the defense.
  • Human red teamers documented a broad tactic framework, including direct requests, obfuscation, echoing, framing, hidden intention streamline, and injection.

D.2.2 Echoing

The examples show how harmful requests can be elicited through conversational reframing, gradual escalation, and output-focused prompting. These interactions include political persuasion, phishing, romance scams, and other harmful content.

  • A multi-turn political exchange gradually reframed the Syrian Civil War as primarily an internal dispute while minimizing external intervention.
  • A romance-scam interaction elicited scripts, talking points, objection responses, and impersonation details through successive follow-up questions.
  • The conversation also included requests for definitions, protective advice, and explanations of how romance scammers operate.
  • Contextual prompting could begin with apparently harmless instructions, such as asking the model to read material without commenting or responding.
Loading 2408.15221v2…