PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Unsafe Detection: Counterfactually Anchored Evidence Attribution for Multi-Turn LLM Safety Failures

Srinivasan Subramanian, Kazi Aminul Islam, Md. Abdullah Al Hafiz Khan

arXiv:2609.27773v1cs.CLcs.AI

TL;DR

Conversation-level safety evaluation does not locate evidence in multi-turn failures, where unsafe intent can emerge gradually and risky vocabulary may occur in benign contexts. The paper builds a validated, tier-supervised dataset and trains a hierarchical model to attribute user turns and tokens. It reports strong detection, lower false positives than surface-risk, and human-aligned turn attribution, while noting limits in attack coverage and counterfactual supervision.

  • Problem

    Conversation-level safety detection flags unsafe outcomes but does not identify which parts of a multi-turn conversation contributed to them, and risky vocabulary can appear in benign contexts.

    Source: §1 · p. 1

  • Method

    The paper constructs a behaviorally validated multi-turn dataset with tiered evidence supervision and trains a hierarchical model to predict safety violations and attribute user turns and token spans.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    Detection F1 = 0.988, and the top-five attributed turns contain a human-identified evidence-bearing turn in 84.5% of adversarial cases.

    Source: §6 · p. 9

  • Takeaways & Limitations

    The findings support diagnostic evidence attribution as a practical layer for analyzing detected multi-turn LLM safety failures.

    Source: §6 · p. 9

    The method is primarily a localizer for detected multi-turn failures, not a generalized universal detector, because coverage of real-world attack strategies is incomplete.

    Source: Limitations · p. 9

Abstract

from arXiv · show

As Large Language Models (LLMs) move from conversational assistants to advanced agentic systems, guardrail failures can convert adversarial intents into harmful executions. However, most guardrail evaluation frameworks focus only on the result and assess whether a user request is safe or unsafe. This approach is insufficient for multi-turn failures, where adversarial intent is distributed across multiple turns. This motivates us to go beyond detection to identify the turns and tokens that push the conversation toward unsafe trajectories. To support this, we construct a multi-turn dataset with behavioral validation and tiered evidence supervision. The dataset contains 1,762 conversations, including adversarial conversations, benign twins, and benign variants with high-risk vocabulary. We train a lightweight hierarchical attribution model that predicts safety violations and attributes them to contributing user turns and token spans. The model achieves strong detection performance (F1=0.988), and removing the top 15% of attributed tokens reduces the adversarial classification confidence by 51.1%. The model preserves low false positive rates on benign conversations with high-risk vocabulary, with false positives below 1% on both borderline benign and benign high-risk vocabulary conversations, compared to 37.3% and 94.7% for a keyword-based surface-risk baseline. Independent human annotation supports the model's attribution performance, with the top-five attributed turns containing a human-identified evidence-bearing turn in 84.5% of adversarial cases.

1 Introduction

Conversation-level safety detection flags multi-turn failures but does not locate the evidence behind them, especially when adversarial intent emerges gradually and risky vocabulary also appears in benign contexts. The paper frames attribution as localization of user tokens, spans, and turns, supported by a dataset and a lightweight hierarchical model.

1 Introduction

  • Multi-turn safety detection identifies an unsafe outcome but does not reveal which turns shifted the conversation toward it.Adversarial intent may emerge gradually, while high-risk vocabulary can also occur in benign contexts.Source: §1 · p. 1
  • The paper formulates guardrail-failure attribution as localization of user tokens, spans, and turns, moving beyond conversation-level binary labels.The dataset combines interactive adversarial generation, cross-model validation, and tiered causal and behavioral supervision.Source: §1 · p. 2
  • The proposed lightweight hierarchical model is designed to learn evidence distributed across conversation history.Unsafe intent may emerge across multiple turns rather than from a single prompt.Source: §1 · p. 2
  • Surface-risk deletion heuristics can confuse harmful evidence with benign high-risk language, motivating Attribution Utility to measure deletion impact against false attribution.Source: §1 · p. 2

2 Related Work

Prior safety benchmarks and guardrails evaluate unsafe behavior, including multi-turn interactions, while datasets also expose the challenge of distinguishing harmful intent from benign risky-looking language. Existing attribution work localizes evidence in different settings, but the cited work does not provide causal attribution across structured multi-turn histories.

Jailbreak and guardrail-failure benchmarks

  • Jailbreak benchmarks expanded from harmful-prompt evaluation to adaptive and multi-turn attacks, but remain focused on conversation-level safety classification.The cited examples include gradual intent through context, role play, task decomposition, or prior-response feedback.Source: §2 · p. 2

Safety datasets and benign conversations

  • Safety datasets pair harmful prompts with benign lookalikes or risky-looking safe prompts, underscoring that risky words alone do not establish unsafe intent.The paper's dataset pairs benign and adversarial twins with detailed attribution supervision.Source: §2 · p. 3

Guardrail models and multi-turn detection

  • Guardrail models and multi-turn detection methods improve unsafe-interaction detection but do not provide causal evidence for the unsafe outcome.Source: §2 · p. 3

Attribution and rationale evaluation

  • Prior attribution methods include feature-importance and rationale approaches, while cited safety-localization methods focus on single-turn prompts or internal representations.The paper's setting requires attribution over structured conversation history, where evidence may span user turns.Source: §2 · p. 3

3 Methodology

The methodology formulates multi-turn guardrail-failure attribution over user-side evidence and builds a behaviorally validated dataset with tiered supervision. A hierarchical model predicts conversation risk and token evidence, while evaluation combines detection, attribution, behavioral, specificity, localization, human-alignment, and robustness measures.

3.1 Problem Formulation

  • Attribution assigns scores to user tokens and turns that contribute to an unsafe trajectory, restricting evidence to user inputs as a scope choice.The stated rationale is that moderation interventions usually act on user inputs; assistant-side attribution is left for future work.Source: §3.1 · p. 3

3.2 Dataset Construction

  • The dataset construction pipeline generates adaptive adversarial conversations, benign variants, cross-model validation, and tiered evidence supervision.The source passages describe adversarial generation, benign variants, validation, causal analysis, and evidence-strength tiers.Source: §3.2 · pp. 3–4
  • Causal-region supervision combines span annotation, turn-level counterfactual replay, and span-level neutralization to identify evidence-bearing regions while excluding unrelated or benign spans.Source: §3.2 · p. 4

3.3 Attribution Model

  • The hierarchical model preserves turn structure and predicts both a conversation-level adversarial score and token-level attribution scores.Cross-turn contextualization models dependencies, gated fusion combines standard and attribution-weighted user-token representations, and an auxiliary head localizes the main causal turn.Source: §3.3 · p. 4
  • A masked forward pass suppresses selected token embeddings before contextualization and reclassifies the conversation to test deletion effects.Source: §3.3 · p. 5

3.4 Attribution Evaluation Framework

  • Evaluation combines conversation detection and span agreement with deletion-based behavioral measures and false-positive rates on benign and high-risk benign conversations.Deviation Drop measures adversarial-probability reduction, while Flip Rate measures adversarial conversations flipped to benign.Source: §3.4 · p. 5
  • Attribution Utility combines Deviation Drop after deleting top-ranked tokens with false-positive rate on boundary and high-risk benign records.The primary setting uses k = 15 and λ = 1.Source: §3.4 · p. 5
  • The framework also evaluates turn localization, human alignment, paraphrase robustness, external deletion behavior, and transfer to an independent safety evaluator.Human turn alignment uses top-k hit rate and coverage; robustness compares rankings across paraphrases.Source: §3.4 · p. 5

4 Experimental Setup

The experiments use pair-aware splits that keep linked adversarial and benign records together, with external test instances reserved for evaluation. The setup specifies generation, validation, training, and baseline comparisons for the attribution model.

4 Experimental Setup

  • Pair-aware train, development, and test splits keep adversarial conversations, benign twins, paraphrase-linked variants, and related records within the same split.External test instances are excluded from training and used only for external evaluation.Source: §4 · pp. 5–6, Table 1 · p. 6
  • Adversarial conversations are generated with Qwen2.5-14B-Instruct against a Llama-3-8B-Instruct target, while benign conversations use Qwen2.5-7B-Instruct and an independent Mistral validator.Source: §4 · p. 6
  • The main model uses a frozen DeBERTa-v3-base turn encoder and two cross-turn transformer layers, with conversations truncated to 32 turns and 192 tokens per turn.Training proceeds through classification warm-up, attribution training, and counterfactual consistency refinement.Source: §4 · p. 6
  • Attribution baselines include surface-risk, attention, Grad×Input, and Integrated Gradients, alongside flat and turn-level classifiers and ablations of gated fusion and counterfactual consistency.Source: §4 · p. 6
  • The proposed method has the best overall combination of span agreement and deletion-based attribution performance among the standard attribution baselines.Table 2 compares the proposed method with standard attribution baselines.Source: Table 2 · p. 6

5 Results

The proposed method combines strong unsafe-conversation detection with attribution that better reflects evidence across turns than keyword-based surface-risk scoring. Results also show specificity challenges on benign high-risk language, human-annotation agreement, robustness on several evaluations, and limits in external detection and exact localization.

5.1 Detection and Causal Attribution Performance

  • F1 = 0.988 and accuracy of 0.993 on the test split; the method also leads attribution baselines on span-label agreement and deletion-based scores.Attention attribution has high span-overlap F1 but little behavioral effect when its selected tokens are removed.Source: §5.1 · p. 6

5.2 Specificity under Benign High-Risk Conversations

  • DD@15 is nearly tied under noise equalization (0.504 vs. 0.502) and combined neutralization and noise (0.484 vs. 0.491), while surface-risk has higher false positives on benign high-risk conversations.The proposed method has a better tradeoff, with lower false-positive rates and higher attribution utility; its deletion behavior rises more gradually as attributed tokens are removed.Source: §5.2 · pp. 6–7, Figure 2 · p. 7

5.3 Human-Annotated Evaluation

  • The top five attributed turns overlap a human-marked causal turn in 84.5% of adversarial cases.Windowed localization accuracy rises as the evaluation window expands, indicating that evidence may lie near rather than exactly on an annotated pivot turn.Source: §5.3 · p. 8

5.4 LLM-as-judge attribution baseline.

  • Qwen2.5-7B-Instruct reaches detection F1 = 0.580 and substantially lower top-turn hit rates than the proposed model.The LLM-as-judge baseline is not sufficient for reliable multi-turn localization.Source: §5.4 · p. 8

5.5 Robustness and External Evaluation

  • Turn-level and token-level paraphrase correlations are 0.983 and 0.957, respectively, on the full adversarial test set.Stability is similar across contextual and lexical pivot subsets.Source: §5.5 · p. 8
  • On AdvBench and HarmBench converted to the multi-turn schema, precision is 1.000 and F1 = 0.607; MHJ detection generalization is weaker.For detected external cases, attributed-token removal yields DD@15 = 0.296 on average and flips 32.1% of cases; on detected MHJ conversations, DD@15 = 0.495 and Flip@15 = 0.633, outperforming surface-risk. Cross-model removal changes ShieldGemma-9B’s judgment from unsafe to safe in 90.9% of tested cases, compared with 63.6% for surface-risk.Source: §5.5 · p. 8

5.6 Ablation Analysis

  • NoFusion and NoCF retain detection F1 scores of 0.994 and 1.000, while architectural baselines have lower detection F1 of 0.758 and 0.832 and poor or negligible token-level attribution F1.Higher deletion scores do not necessarily mean better attribution: NoCF has higher DD@15 and Flip@15 than the full model, but is even more affected by surface-risk token selection.Source: §5.6 · p. 8

5.7 Error Analysis

  • Remaining errors include rare false positives on safe dual-use or safety-sensitive discussions and attribution regions that extend into nearby setup or bridge text.Detection recall remains weak on external multi-turn benchmarks such as MHJ, which the authors treat as an attribution stress test rather than evidence of broad detection generalization.Source: §5.7 · pp. 8–9

6 Conclusion

The paper frames multi-turn guardrail failures as counterfactually anchored evidence attribution over conversation history, identifying user turns and token spans that contribute to unsafe trajectories. Its results support diagnostic attribution as a practical layer for analyzing detected failures.

6 Conclusion

  • The framework localizes contributing user turns and token spans rather than only detecting unsafe outcomes.The study characterizes this as counterfactually anchored evidence attribution over conversation history.Source: §6 · p. 9
  • Detection reaches F1 = 0.988, while top-five attributed turns overlap a human-marked causal turn in 84.5% of adversarial cases.On borderline benign conversations, false positives fall from 37.3% to 0.7% relative to surface-risk.Source: §6 · p. 9
  • Raw deletion metrics can reward lexical shortcuts, while the proposed method offers a stronger tradeoff across behavioral effect, span alignment, and benign specificity.Surface-risk attribution is deletion-effective but brittle on benign false-positive conversations.Source: §6 · p. 9
  • In agentic workflows, turn-level causal attribution provides a diagnostic interface for identifying where unsafe behavior was enabled.The authors support diagnostic evidence attribution as a practical layer for analyzing detected multi-turn failures.Source: §6 · p. 9

Limitations

The authors identify limits in data coverage, counterfactual supervision, attribution scope, and model scale. They position the method primarily as a localizer for detected multi-turn failures, not a universal detector.

Limitations

  • The generated data may miss real-world multi-turn attack strategies, and detection generalization is weaker on human-crafted external attacks.The authors therefore position the method primarily as a localizer for detected failures, not a generalized universal detector.Source: Limitations · p. 9
  • Only 11 cf_strong and 30 cf_weak records, or 2.3% of conversations, provide strict counterfactual evidence.These records are high-confidence anchors; the authors do not treat all attribution labels as strict causal ground truth.Source: Limitations · p. 9
  • Attribution covers user turns alone, although assistant responses such as partial compliance can shape and redirect a multi-turn failure.The authors identify extending attribution to both user and assistant turns as a way to provide a more complete account.Source: Limitations · p. 9
  • Human annotations indicate that unsafe trajectories can involve multiple evidence-bearing turns, while the auxiliary turn-localization head uses a single-pivot target.The final token-attribution head can assign evidence across multiple turns, but future work should explore multi-label turn supervision.Source: Limitations · pp. 9–10
  • The generator, target, and validator models are smaller than frontier systems, and attribution may differ at larger scales or in tool-oriented failures.Evaluation on larger models and agentic workflows remains future work.Source: Limitations · p. 10

7 Ethical Considerations

The work is intended for defensive safety analysis, with safeguards around artifact release and annotation. Its stated scope is auditing unsafe trajectories, localizing evidence, and evaluating guardrail failures.

7 Ethical Considerations

  • The dataset and model are intended to support defensive auditing and should not be used to optimize jailbreak strategies or generate harmful instructions.The paper frames their use as auditing unsafe trajectories, localizing causal evidence, and improving guardrail-failure evaluation.Source: §7 · p. 10
  • Generated adversarial records are behaviorally validated, but the authors caution that they are not interpreted as real user intent.Reproducibility materials are separated from highly actionable content to reduce misuse risk.Source: §7 · p. 10
  • Human annotators label causal turns and spans in existing conversations without completing harmful requests or producing unsafe instructions.Ambiguous and borderline cases are retained for evaluation rather than treated as evidence of harmful intent.Source: §7 · p. 10

A Additional Results and Dataset Statistics

The appendix points readers to additional confidence intervals, ablation details, dataset statistics, and human-annotation details. Table 5 gives a detailed breakdown by conversation family and tiered evidence supervision.

A Additional Results and Dataset Statistics

  • The appendix also reports ablation details in Tables 7–9, dataset statistics in Table 5, and human-annotation details in Table 10.Table 7 is identified as reporting ablations of NoFusion and NoCF.Source: A Additional Results and Dataset… · pp. 11–12, Table 7 · p. 12
  • Table 5 details dataset breakdown by conversation family and tiered evidence supervision; main split sizes and annotation coverage are reported in Table 1.The supplied caption gives the table scope but no breakdown values.Source: Table 5 · p. 12
  • The appendix reports confidence intervals for primary attribution, specificity, and attribution utility in Table 6.The supplied passage identifies the table's scope but provides no interval values.Source: A Additional Results and Dataset… · p. 11, Table 6 · p. 12

B Dataset Generation Pipeline

The appendix details a staged dataset pipeline combining interactive adversarial and separate benign generation with behavioral validation, counterfactual evidence analysis, calibrated supervision, and pair-aware splits. It also reports model settings and summarizes selected baseline and ablation findings.

B.1 Pipeline Overview

  • Table 8 reports higher raw deletion scores for NoCF, while surface-risk deletion also rises sharply, suggesting more brittle predictions; Table 9 summarizes detection and attribution results for baselines and ablations.The supplied Table 9 caption gives its scope but no cell values.Source: Table 8 · p. 12, Table 9 · p. 12

B.2 Interactive Adversarial Generation

  • Adversarial conversations are generated turn by turn, with escalation or strategy changes after refusals to create failures arising from accumulated context.When a target refuses, the generator changes angle rather than repeating the request; implicit-trigger and misleading-pivot cases test whether attribution uses the full trajectory.Source: §B.2 · pp. 12–13, §B.3 · p. 13

B.4 Benign Generation

  • Benign conversations vary in length and include false leads; validation failures are excluded from supervised training and retained as borderline stress cases.Long benign conversations extend up to 32 turns, reducing conversation length as a shortcut feature.Source: §B.4 · p. 13

B.6 Validation model and Unsafe Score

  • The validator combines harmfulness and refusal strength into an unsafe score and identifies a candidate pivot at the first qualifying user turn.The pivot criterion requires refusal score below 0.4 and unsafe score of at least 0.3, avoiding treating low-refusal but low-harmfulness responses as failures.Source: §B.5 · p. 13, §B.6 · p. 14

B.5 Behavioral and Cross-Model Validation

  • Behavioral validation checks unsafe behavior in adversarial records and false unsafe triggers in benign records, assigning a candidate pivot where unsafe behavior first emerges or becomes strongly supported.Cross-model validation assigns transfer tiers, and cross-model failures provide stronger behavioral evidence than model-specific or ambiguous failures.Source: §B.5 · p. 13

B.7 Causal Span Analysis

  • Causal analysis annotates exact spans around candidate pivots, replaces pivot turns and selected spans with benign alternatives, and replays conversations to measure score changes.Decoy and benign-context spans serve as negative controls; an unexpected score change can relabel a span as a context bridge.Source: §B.7 · p. 14

B.8 Post-Processing and Supervision Calibration

  • Post-processing can relabel counterfactually causal decoys or benign-context spans as context bridges, while supervision tiers distinguish counterfactual, model-supported, construction, benign, incidental, and ignored evidence.The strongest span-level evidence determines the sample-level tier and its training loss weight.Source: §B.8 · p. 14, §B.9 · p. 14

B.10 Splitting and Held-Out Annotation

  • Pair-aware splitting keeps adversarial conversations, benign twins, and linked variants together, while held-out human annotation prioritizes records with stronger validation or counterfactual evidence.The model uses a frozen DeBERTa-v3-base turn encoder, represents at most 32 turns with up to 192 tokens each, and contextualizes turns in a projected cross-turn space.Source: §B.10 · pp. 14–15, §C.1 · p. 15

C.2 Training Objective

The appendix specifies a multi-task training objective and phased schedule, then documents generation, validation, and annotation prompts alongside evaluation baselines. It also defines counterfactual replay and external safety-evaluator procedures.

C.2 Training Objective

  • Training jointly optimizes conversation classification, token attribution, pivot localization, and counterfactual consistency.Counterfactual consistency is added only in the final training phase.Source: §C.2 · p. 15

C.3 Training Schedule and Optimization

  • Training proceeds in three phases: classification alone, then attribution and pivot localization, followed by counterfactual consistency.The phases run for 5, 15, and 5 epochs, respectively.Source: §C.3 · p. 15

C.4 Learned Baselines and Ablations

  • The baselines use the same data splits, while TurnLevel and flat ConvDeBERTa disable attribution, pivot, and counterfactual losses.The flat ConvDeBERTa baseline uses a maximum total input length of 2048 tokens.Source: §C.4 · p. 16

D.1 Interactive Adversarial Generation Prompt

  • Adversarial-generation prompts request natural, gradually escalating turns that adapt to the assistant’s responses, while refusal adaptation changes tactics instead of repeating a refused request.Benign and false-lead prompts separately request safe conversations, including surface-risk language in clearly benign contexts.Source: §D.1 · p. 16, §D.2 · p. 16, §D.3 · pp. 16–17

D.4 Structured Validator Rubric

  • The structured validator assigns behavior, harmfulness, refusal strength, confidence, and rationale, then uses refusal and unsafe-score thresholds to identify candidate pivots.Low-confidence COMPLY or PARTIAL judgments below 0.4 are treated as AMBIGUOUS.Source: §D.4 · p. 17

D.5 Span Annotation Instructions

  • Annotators label exact substrings in each user message, returning 0–6 spans; benign messages receive one BENIGN CONTEXT span, while adversarial messages list the most important spans first.The supplied labels include MALICIOUS TRIGGER, PAYLOAD SPAN, CONTEXT BRIDGE, and STRUCTURAL TRIGGER.Source: §D.5 · p. 17

D.6 Counterfactual Replay and Negative Controls

  • Counterfactual replay replaces candidate pivots, then selected spans when pivot effects are strong or weak, and tests decoys and benign-context spans as negative controls.Replay score-change thresholds assign strong, weak, distributed-or-unclear, or incidental labels.Source: §D.6 · p. 17

D.7 LLM-as-Judge Attribution Baseline

  • The judge baseline ranks evidence-bearing turns rather than treating its causal_user_turns field as strict causal ground truth; ShieldGemma transfer uses Yes-token probability to assess policy violations.The judge prompt distinguishes evidence-bearing turns from surface-risk decoys, while ShieldGemma cases are classified unsafe at a Yes probability of at least 0.5.Source: §D.7 · pp. 17–18, §D.8 · p. 18

E External Evaluation and Deletion Protocols

External evaluation adapts single-turn AdvBench and HarmBench prompts into multi-turn conversations while preserving the original unsafe request, and retains MHJ’s original multi-turn structure. Deletion tests remove top-ranked user tokens and measure changes in model or external safety judgments.

E.1 External Benchmark Conversion

  • AdvBench and HarmBench prompts gain benign context-building turns while retaining the original prompt as the final user turn; MHJ keeps its original structure but lacks pivot and span evidence labels for supervised localization evaluation.For converted single-turn benchmarks, added turns are not treated as counterfactually validated evidence.Source: §E.1 · p. 18

E.2 Deletion and Masking Mechanics

  • Deletion evaluation removes each attribution method’s top-ranked user tokens and measures the resulting change in adversarial probability or external safety judgment.The proposed model uses representation masking, while external evaluators use token replacement and attention masking.Source: §E.2 · pp. 18–19

Related papers

Loading 2609.27773v1…