PaperlayerView exact revision on arXiv

Source-linked AI summary

Does the Unsafe Gradient Survive a Conversation? On the Fragility of Gradient-Based Jailbreak Detection in Multi-Turn Dialogue

Omar Sheta, Rinku Deuja, Hadi Masoudi, Minghong Fang

arXiv:2609.36849v1cs.CRcs.AIcs.IRcs.LG

TL;DR

Multi-turn assistants let adversaries spread unsafe intent across several turns, but gradient-based detectors were developed for single prompts, leaving their effectiveness in dialogue unclear. The paper adapts GradSafe with fixed-size context windows and evaluates multiple benign distributions, attacks, and target models. Performance drops sharply on realistic benign conversations, and results also vary across attack families and models.

  • Problem

    Gradient-based jailbreak detection was developed for single prompts, leaving unclear whether its unsafe-gradient signal remains detectable when intent is distributed across dialogue turns.

    Source: §1 · p. 1

  • Method

    The Context Window Scanner applies GradSafe to fixed-size windows of user turns and uses the maximum window score as the conversation score.

    Source: §1 · p. 2

  • Results

    ROC-AUC drops from 0.98 on synthetic benign conversations to 0.76 on WildChat, while short windows outperform longer or accumulated contexts and results vary across attacks and models.

    Source: §1 · p. 2, §8 · p. 8

  • Takeaways & Limitations

    Reliable evaluation requires realistic benign conversations, varied context lengths, diverse attack families, and different model architectures.

    Source: §7 · p. 8, §8 · p. 8

    The study evaluates separability and threshold behavior, not a complete blocking system or robustness to attackers optimizing directly against the gradient score.

    Source: §3 · p. 3

Abstract

from arXiv · show

Safety-aligned language models are commonly deployed as multi-turn assistants, which lets adversaries spread unsafe intent across several user turns instead of a single prompt. Gradient-based jailbreak detectors such as GradSafe were developed for single prompts: they score an input by the alignment between its induced gradient and a fixed unsafe reference direction, and their effectiveness in multi-turn dialogue remains unclear. We conduct a controlled evaluation of gradient-based jailbreak detection in multi-turn settings. We extend GradSafe with a Context Window Scanner that applies the detector to fixed-size windows of user turns and uses the maximum window score as the conversation-level score. We evaluate different window sizes, attack families, benign conversation distributions, and target models. The results differ sharply between synthetic and realistic benign settings. Against synthetic benign conversations, the detector achieves an ROC-AUC of 0.98 on human-authored multi-turn jailbreaks. On WildChat benign conversations, ROC-AUC drops to 0.76, and a threshold calibrated on synthetic data flags more than 90% of benign conversations as unsafe. Under realistic benign distributions, single-turn windows give the highest separability, whereas longer windows and accumulated contexts reduce performance. The detector is also sensitive to the attack-generation method and target model: successful Crescendo attacks receive scores comparable to or lower than benign conversations, and Qwen2.5-7B-Instruct yields near-random separability with a different optimal window size. These findings show that gradient-based signals can support multi-turn jailbreak detection, but reliable deployment requires calibration on realistic benign conversations, short-window scoring, length-aware thresholds, and evaluation across attack types and model architectures.

1 Introduction

Multi-turn dialogue lets users distribute unsafe intent across seemingly benign turns, while most jailbreak defenses were developed for single prompts. This study adapts GradSafe to conversations and finds that detection varies substantially with benign-data realism, context length, attack family, and target model.

1 Introduction

  • The study asks whether GradSafe’s unsafe-gradient signature remains detectable when malicious intent is distributed across a dialogue rather than localized in one prompt.Multi-turn attacks exploit this structure through escalating or context-building turns [16] [21].Source: §1 · p. 1
  • The Context Window Scanner slides fixed-size windows over user turns, scores each with GradSafe, and uses the highest score as the conversation score.The study compares this approach with full-history accumulation and a single-turn baseline on 537 human-authored multi-turn jailbreaks and 2,000 WildChat conversations matched for turn length.Source: §1 · pp. 1–2
  • ROC-AUC falls from 0.98 on synthetic benign conversations to 0.76 on WildChat; a synthetic-calibrated threshold flags more than 90% of real benign conversations.These results show that synthetic calibration substantially overstates performance on realistic benign traffic.Source: §1 · p. 2
  • On realistic benign traffic, single-turn scoring is strongest; separability declines as windows grow or full history is accumulated.The authors attribute this to gradient dilution: benign or unrelated turns add noise and pull the gradient away from the unsafe reference direction.Source: §1 · p. 2
  • Successful Crescendo attacks approach random separability or score below benign conversations, while Qwen2.5-7B-Instruct yields near-random separability and favors accumulated scoring.The Crescendo user turns stay close to benign gradients until the model generates unsafe content; the results therefore vary with attack-generation method and target model.Source: §1 · p. 2
  • The paper recasts single-turn GradSafe as a multi-turn detector to test whether its unsafe-gradient signal survives across dialogue turns.The evaluation varies benign-data realism, context length, attack family, and target model.Source: §1 · p. 2

2 Background and Related Work

The background traces jailbreaks from single crafted prompts to coordinated dialogues, then surveys defenses by where they intervene. It places this study among internal-gradient defenses and distinguishes its pre-generation setting from response-side moderation and other protocols.

2.1 Single-Turn Jailbreak Attacks

  • GCG [28] uses white-box gradients to find transferable adversarial suffixes, while Tree of Attacks with Pruning [18] reduces query costs through tree-based prompt search.Human-written role-play, privilege-escalation, and prompt-injection patterns broaden the threat beyond optimized suffixes [22]; HarmBench [17] and JailbreakBench [1] standardize evaluation.Source: §2.1 · p. 2

2.2 Multi-Turn Jailbreak Attacks

  • Multi-turn attacks coordinate dialogue: PAIR refines prompts through feedback, while Crescendo [21] escalates innocuous requests until harmful content follows from accumulated context.The MHJ dataset [16] contains 537 human-written conversations spanning seven tactics, and indirect prompt injection can plant instructions in retrieved content.Source: §2.2 · p. 2

2.3 Jailbreak Defenses

  • Defenses intervene at input, classifier, internal-signal, or alignment stages; this study extends the GradSafe [23] internal-gradient family to distributed dialogue intent.Gradient Cuff [10] and Gradient-Controlled Decoding [4] are not baselines because their refusal-loss and first-token mitigation protocols differ from this prompt-only, pre-generation setting; forensic localization occurs after a safety failure [5] [7] [25] [26].Source: §2.3 · p. 3

3 Threat Model

The threat model gives an attacker control of a sequence of user turns and places the defender before generation. The study focuses on detection separability and threshold behavior, not a complete blocking system or robustness to direct gradient-score optimization.

3 Threat Model

  • The attacker distributes malicious intent across user turns that may individually appear benign, ambiguous, or incomplete, and knows a detector may be present but not its reference direction or threshold.Source: §3 · p. 3
  • The defender uses white-box gradients to score user turns before generation and does not see assistant responses.The chosen pre-generation point of control is intended to intercept before harmful output may be seen, logged, cached, or copied; it is also relevant where a response can launch irreversible tool actions.Source: §3 · p. 3
  • The study evaluates separability and threshold behavior, not a complete blocking system or robustness to an attacker optimizing directly against the gradient score.The Llama Guard comparison uses the same prompt-only, user-turn-only protocol, narrower than its intended prompt-and-response safeguard.Source: §3 · p. 3

4 Our Method

The method keeps GradSafe’s single-turn scoring rule fixed and adapts it to conversations by scoring windows of user turns. The Context Window Scanner uses the maximum window score, spanning isolated turns to full-history scoring as the window grows.

4.1 GradSafe Scoring

  • GradSafe scores the cosine similarity between an input-induced gradient and a fixed unsafe reference signature over safety-critical parameters.The input is paired with a compliance token; a higher score means closer alignment with the unsafe reference.Source: §4.1 · p. 3

4.2 Context Window Scanner

  • CWS addresses the choice between scoring only the latest turn, which discards accumulated context, and scoring the whole history, which mixes attack and benign turns.It varies the window size while keeping the scoring rule otherwise unchanged.Source: §4.2 · pp. 3–4
  • For each user-turn position, CWS scores a right-anchored window and assigns the conversation its highest window score.The windows are causal and can be evaluated as turns arrive; assistant turns are excluded.Source: §4.2 · p. 4
  • Max pooling makes one high-risk window sufficient to flag a conversation, matching a setting where one successful jailbreak turn counts as a failure.The window size ranges from isolated user turns to a single window covering the full history.Source: §4.2 · p. 4
  • Scoring a conversation with m user turns requires m backward passes, independent of window size; larger windows risk mixing benign turns into the scored text.CWS is a faithful extension for studying context length, not a complete defense.Source: §4.2 · p. 4

5 Experimental Setup

The evaluation compares gradient-based scoring across human-authored and automated multi-turn attacks, benign conversation distributions, target models, and scoring baselines. The realistic benign benchmark is length-matched to the human-authored attacks, and the setup includes threshold-transfer and standard discrimination metrics.

Attack data

  • The attack benchmark includes 537 human-authored multi-turn jailbreak conversations and 149 confirmed successful Crescendo conversations from 200 HarmBench objectives.The 51 Crescendo rows that failed at runtime are not counted as attack failures; excluding them restricts conclusions to genuine multi-turn context-building attacks.Source: §5 · p. 4

Benign data

  • The primary benign benchmark is 2,000 WildChat conversations, sampled after removing one-turn conversations and matching the MHJ turn-length distribution.The sample contains 740 short, 680 medium, and 580 long conversations.Source: §5 · p. 4
  • A separate synthetic benchmark contains 183 Gemini 2.5 Pro conversations; its clean, short, unmatched content is treated as an evaluation contrast, not a deployment benchmark.The authors use length-matched WildChat as the realistic reference.Source: §5 · p. 4

Models and baselines

  • The main target is Llama 3.1 8B Instruct, with Qwen2.5-7B-Instruct added as a cross-architecture target.The study compares CWS with single-turn and accumulated scoring, plus Llama Guard 3 under a prompt-only, user-turn-only protocol.Source: §5 · p. 4

Metrics

  • The evaluation reports ROC-AUC, average precision, best-F1 threshold and score, and true positive rate at a 10% false positive rate.It also transfers the synthetic best-F1 threshold to WildChat and uses nonparametric 95% bootstrap intervals.Source: §5 · p. 5, Table 1 · p. 5

Implementation

  • Gradient scoring uses bfloat16 on NVIDIA H100 GPUs, with seeded sampling and bootstrap procedures for reproducibility.The artifact script records table and figure provenance; code and redacted artifacts are planned for release upon acceptance, subject to review and responsible handling.Source: §5 · p. 5

6 Experimental Results

Detector performance changes sharply with the benign-data distribution and the attack or target model. On realistic WildChat traffic, short windows outperform accumulated context for human-authored attacks, but that pattern does not hold across attack families and architectures.

6.1 Synthetic versus Real Benign Calibration

  • Against synthetic benign conversations, three-turn and accumulated scoring exceed 0.97 ROC-AUC; on WildChat they fall to roughly 0.76 and 0.74, while a synthetic threshold flags 90.85% of real benign conversations.The same three-turn threshold flags 8.74% of synthetic benign conversations, showing that its operating point does not transfer to realistic traffic.Source: §6.1 · p. 5

6.2 Main Detector Comparison

  • On WildChat, the single-turn GradSafe setting leads at 0.8101 ROC-AUC, while two-turn, three-turn, and accumulated scoring perform progressively worse.The main-results table compares MHJ attacks with stratified WildChat benign conversations.Source: Table 2 · p. 5, §6.2 · p. 5
  • AUC falls monotonically from one to seven turns, and the one-versus-three-turn difference is ΔAUC = 0.0487 (95% CI [0.0329, 0.0643]).For conversations with seven or more user turns, two-turn and one-turn windows score 0.8625 and 0.8622 AUC, versus 0.8370 for three turns and 0.7411 for accumulated scoring.Source: §6.2 · p. 6, Figure 3 · p. 6, Table 3 · p. 6
  • The one-turn window is strongest for every nontrivial tactic group, including direct requests, injection, obfuscation, and request framing.The tactic subsets are each scored against the full WildChat benign distribution.Source: Table 4 · p. 6, §6.2 · p. 6
  • Against WildChat, successful Crescendo attacks score 0.5018 AUC at one turn and below 0.5 at three turns, seven turns, and accumulated scoring, unlike MHJ attacks.The three-turn successful-Crescendo interval lies below random ranking; the authors link the mismatch to gradual, formal prompt-side escalation and overlap with realistic benign traffic.Source: Table 5 · p. 6, §6.2 · pp. 6–7
  • On Qwen2.5-7B-Instruct, accumulated scoring is best at 0.5908 AUC while windowed settings are near random; this reverses Llama 3.1's ordering.The authors caution that these results reflect one signature realization against one benign benchmark and that thresholds need recalibration for each target.Source: Table 7 · p. 7, §6.2 · p. 7
  • None of three tested extensions outperforms the standard three-turn baseline on the real WildChat benchmark or its pilot subsets.The tested contrastive, benign-subspace, and multi-turn-reference approaches did not resolve the overlap between benign conversational structure and attack signal.Source: Table 8 · p. 8, §6.2 · p. 8

7 Discussion

Gradient-based signals remain useful for some jailbreaks on realistic traffic, but their practical utility is limited and they do not generalize reliably across attack families. The discussion attributes weaker long-context scoring to gradient dilution and recommends realistic calibration and model-specific validation.

7 Discussion

  • On realistic traffic, even the best setting remains far from a deployable operating point, despite one- and two-turn scoring outperforming random ranking.Tactic-level AUCs are strong for injection, output-format, and request-framing attacks.Source: §7 · p. 8
  • Successful Crescendo attacks approach or fall below random performance, while longer windows rank them below benign traffic; short-window results on MHJ do not transfer reliably.The authors suggest context-building attacks keep user turns close to benign gradients until the model generates unsafe content.Source: §7 · p. 8
  • Longer windows can dilute sparse unsafe signals because benign or unrelated turns move aggregate gradients away from the unsafe reference.The discussion suggests context-conditional gating or asymmetric scoring as possible ways to isolate safety-critical parameters.Source: §7 · p. 8
  • Synthetic benign data can produce optimistic thresholds, so evaluation should use realistic benign distributions, ideally length-matched to attacks.The authors warn that performance measured otherwise may not remain reliable in deployment.Source: §7 · p. 8
  • Llama Guard’s low AUC reflects a different intervention point: it classifies prompts and responses, while this protocol detects hidden intent from user turns before harmful output exists.The discussion frames this as a distinction between response-side moderation and pre-generation detection, not a defect in the classifier.Source: §7 · p. 8
  • Validate windowing and aggregation per model family, tune thresholds on deployment-like benign data, account for conversation length and attack-family drift, and assess prompt-only detection separately from response moderation.On WildChat, the synthetic three-turn threshold transferred to a 90.85% false-positive rate; accumulated scoring wins on Qwen2.5-7B-Instruct, unlike the Llama 3.1 8B Instruct default.Source: §7 · p. 8

8 Conclusion

The paper evaluates multi-turn gradient-based jailbreak detection by adapting GradSafe with fixed-size context windows. Its results vary with benign-data realism, window length, attack-generation method, and target model, motivating broader and more realistic evaluation.

8 Conclusion

  • The paper adapts GradSafe with fixed-size context windows to evaluate gradient-based jailbreak detection in multi-turn dialogue.Source: §8 · p. 8
  • Performance decreases substantially on realistic WildChat conversations, and thresholds calibrated on synthetic data do not transfer.Source: §8 · p. 8
  • Short windows separate better than longer or full-history contexts, while successful Crescendo attacks remain difficult to distinguish from benign conversations and results vary across target models.The conclusion calls for evaluation with realistic benign data, multiple context lengths, diverse attack families, and different model architectures.Source: §8 · p. 8

Related papers

Loading 2609.36849v1…