PaperlayerView exact revision on arXiv

Source-linked AI summary

Representation Transitions Reveal Emerging Safety Risks in Multi-Turn LLM Agents

Haoyu Wang, Wei Zhao, Yedi Zhang, Christopher M. Poskitt, Jun Sun

arXiv:2610.00400v1cs.LGcs.AIcs.CR

TL;DR

Multi-turn attacks can assemble harmful outcomes from individually permissible steps, a risk that isolated action or state checks may miss. DART detects accumulated representation shifts across context updates, attributes their triggering context, and intervenes with targeted reminders. Across six models and two benchmarks, it reduces attack success, with denoising substantially improving ASEval detection.

  • Problem

    Multi-turn attacks can combine individually permissible actions into harmful outcomes, challenging defenses that assess actions or states in isolation.

    Source: §1 · p. 1

  • Method

    DART aggregates context-induced representation transitions along a denoised safety direction, attributes the contributing context, and intervenes with targeted reminders.

    Source: §1 · p. 2, §3 · p. 5, §4 · p. 6

  • Results

    Across six models, DART reduces attack success from 84% to 25% on MT-AgentRisk and from 97% to 52% on ASEval.

    Source: §1 · p. 2

  • Takeaways & Limitations

    DART provides a low-runtime-cost safety layer that can operate independently or complement heavier defenses such as speculative simulation.

    Source: §6 · p. 10

    An attributed segment may combine harmful instructions with task-relevant information, so attribution alone does not decide what content to retain.

    Source: §4 · p. 6

Abstract

from arXiv · show

Multi-turn attacks on agentic systems can compose individually permissible actions into harmful outcomes, challenging defenses that assess actions or states in isolation. We show that such attacks leave a detectable signature in the agent's internal representations: harmful behavior emerges as an accumulated representation transition across context updates, whose triggering context can be identified from the same signal. We further find that naive aggregation is confounded by benign representation drift, as a contrastive safety direction need not assign zero to benign transitions. We address this by denoising the direction, anchoring benign traffic at zero and removing its leading variation directions, with no runtime cost. These findings motivate DART, a runtime framework that detects and attributes representation shifts and intervenes with targeted reminders. Across six models and two multi-turn benchmarks, DART reduces attack success from 84% to 25% on MT-AgentRisk, catching every attack at a mean false-alarm rate of 12%, and from 97% to 52% on ASEval, at costs in benign non-refusal of 8% and 0%, respectively. On MT-AgentRisk, it outperforms ToolShield, the state-of-the-art multi-turn defense, on all six models: under the same protocol, ToolShield reaches only 55%. Denoising is critical: on ASEval, the undenoised monitor catches only 7%-40% of attacks, while the denoised monitor catches 60%-85%. The same monitor covers single-turn indirect injection without modification and adds only 0.14-0.56 s overhead per monitored step without requiring an auxiliary model, making it a lightweight complement to computation-heavy speculative defenses.

1 Introduction

Multi-turn attacks can compose individually permissible steps into harmful outcomes that are difficult to detect by assessing actions or states in isolation. DART detects and attributes representation shifts across context updates, then intervenes with targeted reminders.

1 Introduction

  • Multi-turn attacks can combine permissible steps so that risk appears only in how an agent’s behavior evolves across turns.An attack may first induce retrieval or transformation of information, then use it for a harmful action such as exfiltrating emails or redirecting a payment.Source: §1 · p. 1
  • Harmful multi-turn behavior accumulates representation transitions toward a safety-relevant direction, while benign drift can confound naive aggregation.DART anchors benign traffic at zero and removes its dominant variation directions without additional runtime computation.Source: §1 · p. 2
  • DART projects context-induced representation transitions onto a denoised safety direction, aggregates scores, and intervenes at a calibrated threshold.It attributes the shift to a triggering context and supplies a targeted reminder while preserving context and allowing execution to continue.Source: §1 · p. 2, Figure 1 · p. 2
  • DART reduces attack success from 84% to 25% on MT-AgentRisk and from 97% to 52% on ASEval across six models, with respective utility costs of 8% and 0%.On MT-AgentRisk it catches every harmful trajectory at a mean false-alarm rate of 12% and outperforms ToolShield’s 55%; denoising raises ASEval catch rates from 7%–40% to 60%–85%.Source: §1 · p. 2

2 Preliminaries

The paper models an agent’s execution as context segments that induce a sequence of internal representations and transitions. It motivates monitoring those transitions because risks can emerge across otherwise permissible steps that individual-state defenses may miss.

2 Preliminaries

  • An agent receives context segments from users, tools, memory, retrieved documents, or its environment, alongside its own reasoning and tool call since the previous segment.Bundling these into an ordered partition of the token stream lets per-segment scores telescope exactly; monitoring occurs at segment boundaries.Source: §2 · p. 3
  • Each representation is read after one segment is appended and before the next arrives, with the system-prompt-only representation as the anchor.The transition induced by a segment is the difference between consecutive representations.Source: §2 · p. 3
  • Last-token state differences include position- and length-dependent artifacts as well as the semantic contribution of each context segment.The paper identifies these artifacts as sources of drift that a naive running sum accumulates, motivating denoising.Source: §2 · p. 3
  • Existing guardrails and action monitors assess individual inputs, outputs, actions, or tokens, so they may miss risk that emerges gradually across permissible steps.Trajectory monitors either use an observer LLM or operate within one context window, whereas DART reuses the acting agent’s forward pass across context updates.Source: §2 · p. 3
  • DART uses representation transitions for detection and attribution, while mitigating risk through a targeted reminder rather than treating a predictive direction as causal control.This distinction follows the paper’s read–steer framing of representation engineering.Source: §2 · p. 4

3 Representation Transitions as a Safety Signal

Harmful behavior is reflected in accumulated representation transitions, but benign drift can overwhelm sparse attack signals. The section develops a denoised direction and evaluates its ability to separate harmful from benign trajectories.

Representation Transitions as a Safety Signal

  • Unsafe behavior appears as accumulated shifts along a consistent direction, while naive accumulation is confounded by ordinary benign drift.The contrastive direction separates harmful from benign transitions but need not assign benign transitions zero projection, allowing small benign contributions to overwhelm sparse harmful shifts.Source: §3 · p. 4

A transition-space safety direction

  • The safety direction is built from a contrast between harmful and intended-benign representation transitions extracted at a fixed model layer.A transition’s scalar score reads whether a context update moves the agent toward the target behavior.Source: §3 · p. 4

Accumulating transitions

  • DART accumulates per-transition scores rather than classifying each transition independently, measuring cumulative displacement along a safety-relevant direction.One large harmful update or several smaller transitions can cross the same threshold.Source: §3 · p. 4

Denoising for trajectory-level accumulation

  • Denoising anchors benign traffic at zero and removes leading directions of benign variation, addressing systematic and stochastic drift respectively.These are complementary fit-time operations; the selected layer and rank are set on an inner validation split, with no added runtime computation.Source: §3 · pp. 4–5

Empirical Evidence

  • On ASEval, denoising raises trajectory AUROC from 0.57–0.71 to 0.90–0.98 and catch from 7%–40% to 60%–85% without increasing false alarms.On MT-AgentRisk, both directions catch every attack, while denoising reduces false alarms on four of six models.Source: Table 1 · p. 5, §3 · p. 5
  • The evaluation measures accumulated-statistic AUROC and catch and false-alarm rates at the 90th percentile of benign calibration scores.It tests MT-AgentRisk, where every turn carries risk signal, and ASEval, where harmful instructions are injected into otherwise benign tool-using trajectories.Source: §3 · p. 5

DART runtime detection and mitigation against multi-turn attacks

  • The runtime algorithm computes each transition score, accumulates scores, and, after threshold crossing, attributes the largest-scoring segment and appends a reminder before the next action.Monitoring continues because offending content may re-enter through later retrievals; the threshold is calibrated to a false-alarm budget.Source: §3 · p. 6

Empirical Evidence

  • On ASEval, removing only the benign mean at rank p=1 yields AUROC 0.750, while removing one additional covariance direction at p=2 raises it to 0.912.Gains saturate around p ≈8–16; the removed directions reflect trajectory position, user-versus-tool modality, and tool identity rather than segment length.Source: Figure 2 · p. 5, §3 · p. 6

4 DART for Runtime Defense

DART monitors accumulated representation transitions, attributes detected shifts to contributing context segments, and intervenes with targeted reminders while allowing execution to continue.

4 DART for Runtime Defense

  • DART accumulates each segment’s projected representation transition, allowing individually permissible shifts to collectively reveal emerging harm.The accumulated statistic represents cumulative displacement along the safety direction.Source: §4 · p. 6
  • The monitor flags when the aggregate exceeds a threshold calibrated to a false-alarm budget using benign calibration scores.Monitoring continues after intervention because offending content may re-enter through later retrievals.Source: §4 · p. 6
  • DART attributes a detected shift to the segment with the largest contribution, without requiring a separate attribution model.The mechanism applies to tool outputs, retrieved documents, memory items, and user messages.Source: §4 · p. 6
  • Attribution alone cannot determine what to retain when a segment mixes harmful instructions with task-relevant information.The finer-grained retention decision is left to enforcement.Source: §4 · p. 6
  • For the attributed segment, DART adds a reminder identifying the contributing content and instructing the agent how to interpret it.For retrieved documents or tool outputs, the reminder says to treat the content as untrusted data rather than instructions; execution continues.Source: §4 · p. 7

5 Experiments

Across six models and two benchmarks, DART reduces attack success while preserving benign utility; denoising improves detection and localization, and DART outperforms ToolShield on MT-AgentRisk.

5.1 Setup

  • The evaluation uses six locally hosted models and two benchmarks: MT-AgentRisk decomposes harm across permissible turns, while ASEval injects attacks into benign trajectories.ASEval runs average 6.7 monitored segments, including 1.8 injected segments.Source: §5.1 · p. 7
  • Detection is reported as catch and realized benign FPR, while mitigation uses harmful-trajectory ASR and benign non-refusal utility.Metrics are measured with and without DART on the same trajectories to expose intervention cost.Source: §5.1 · p. 7

5.2 Main Result: DART Reduces Attack Success While Preserving Utility

  • On MT-AgentRisk, DART reduces average ASR from 84% to 25% with an 8% benign non-refusal cost; on ASEval, ASR falls from 97% to 52% at no cost.Figure 3 summarizes end-to-end mitigation across both benchmarks and all six models.Source: §5.2 · p. 7, Figure 3 · p. 8

5.3 Comparison with Existing Defenses

  • DART reduces MT-AgentRisk average ASR to 25%, versus 55% for ToolShield, outperforming it on all six models.ToolShield has slightly higher utility (91% against 85%), consistent with its always-on guidance rather than trajectory-gated intervention.Source: §5.3 · p. 8
  • DART adds 0.14–0.56 seconds per monitored step by reusing hidden states, compared with about 6.7 seconds for the reproduced speculative simulation.The SSH comparison is cost-only because its unavailable fine-tuned models prevent a faithful accuracy comparison.Source: §5.3 · p. 9

5.4 Ablation: The Role of Denoising

  • On ASEval, denoising lowers residual ASR from 0.85 to 0.55 on Qwen3-8B and from 0.97 to 0.55 on Qwen3-32B, with no utility cost.Removing the benign mean shifts the response toward injected segments; removing leading benign-variation directions limits drift and false alarms on MT-AgentRisk Qwen3-32B.Source: §5.4 · p. 9
  • With denoising, 84%–87% of flagged ASEval runs cross at or after the first injection, versus frequent pre-injection peaks without denoising.The plain aggregate peaks before attack in 45% of harmful Qwen3-8B runs and all Qwen3-32B runs.Source: §5.4 · p. 9, Figure 4 · p. 9
  • Denoising attributes the largest contribution to the injected ASEval segment 78%–83% of the time on five models, versus a 22%–30% chance rate.On Llama-3.1-8B, localization improves from 53% to 83%.Source: §5.4 · p. 9

6 Conclusion

DART detects emerging multi-turn safety risks through representation transitions, attributes them to contributing context, and intervenes with targeted reminders. Across six models and two benchmarks, it reduces attack success while preserving benign utility, with denoising enabling low-cost trajectory monitoring.

6 Conclusion

  • DART detects emerging multi-turn safety risks from representation transitions, attributes them to contributing context, and intervenes with targeted reminders.The framework is evaluated across six models and two agentic benchmarks.Source: §6 · p. 10
  • Across the evaluated benchmarks, DART substantially reduces attack success while preserving benign utility, including for attacks composed of individually permissible steps.Denoising separates sparse harmful shifts from accumulated benign drift.Source: §6 · p. 10
  • By reusing the agent’s hidden states, DART provides a lightweight safety layer that can operate independently or complement heavier speculative defenses.Source: §6 · p. 10

AI use statement

The authors report using LLMs for coding and writing assistance, while retaining responsibility for the research and final content.

AI use statement

  • LLMs assisted with implementation and debugging of evaluation and plotting code, and with clarity and concision in author-written text.Source: AI use statement · p. 10
  • The authors state that the research questions, method, experimental design, analyses, and scientific claims are their own.Source: AI use statement · p. 10
  • The authors checked AI-assisted code, reviewed AI-edited text, and take responsibility for the final content, claims, figures, and artifacts.Source: AI use statement · p. 10

Ethics statement

The experiments use public benchmarks and simulated tools, with no human subjects, private data, or live external services. The authors frame the work as evaluating defenses against attacks rather than creating them.

Ethics statement

  • Experiments use publicly available benchmarks and simulated tool environments, with no human subjects, private or personally identifiable data, or live external services.Source: Ethics statement · p. 10
  • Adversarial inputs come from or are modeled on public benchmarks and are used solely to evaluate a defense.Source: Ethics statement · p. 10
  • The authors judge the societal benefit of a cheap, attack-agnostic safety monitor to outweigh the residual risk.Source: Ethics statement · p. 10

Reproducibility statement

The paper specifies its monitor and mitigation procedures, evaluation protocol, and an anonymized reproduction repository.

Reproducibility statement

  • The monitor, attribution, and mitigation procedures are specified in §4 and Algorithm 1.The evaluation protocol covers models, benchmarks, data splits, layer and threshold selection, and metrics in §5.1 and the appendix.Source: Reproducibility statement · p. 10
  • The evaluation protocol is given in §5.1, with additional detail in the appendix.It covers models, benchmarks, data splits, layer and threshold selection, and metrics.Source: Reproducibility statement · p. 10
  • An anonymized repository provides the monitor, mitigation, evaluation harness, and benchmark records behind the results.The passage gives the repository address.Source: Reproducibility statement · p. 10

A Further analysis of the denoised direction

On ASEval, denoising addresses benign drift that can overwhelm accumulated attack signals, while further analysis identifies what is removed and when the denoising checks are informative. The section also contrasts DART with rule-based AgentSpec, which is strong for overt in-domain actions but does not transfer across domains or cover decomposed attacks.

A Further analysis of the denoised direction

  • For online ASEval Qwen3-8B, denoising shifts trajectory AUROC from 0.712 to 0.958, while attribution remains 0.675 for both directions.Segment AUROC is also unchanged within noise, at 0.697→0.685; the attribution gain seen offline is described as model- and harness-specific.Source: A Further analysis of the denoised… · p. 13
  • Denoising raises harmful-peak alignment from 0.07 to 1.00 and trajectory AUROC from 0.409/0.477 to 0.857/0.894 on ASEval diagnostics.The plain aggregate peaks on the opening segment in 40 of 40 harmful runs on both models, while segment-level AUROC remains near its prior level after denoising.Source: A Further analysis of the denoised… · p. 13, Table 2 · p. 14
  • The removed benign subspace captures trajectory position and tool-versus-user modality, while segment length explains at most 0.21 of any component.The top 16 components hold 82% and 80% of benign transition variance on Qwen3-8B and Qwen3-32B, respectively.Source: A Further analysis of the denoised… · pp. 14–15, Table 3 · p. 15
  • Benign drift alone is a deployable screen, but near-zero drift is necessary rather than sufficient; peak timing requires labelled harmful trajectories.On Llama-3.1-8B, drift of −0.72 coexists with opening-segment peaks in 60% of harmful runs.Source: A Further analysis of the denoised… · p. 15
  • Rank selection uses an inner validation split disjoint from calibration and evaluation, and fixes p=16 on ASEval and p=2 on MT-AgentRisk.The analysis warns that AUROC-only selection can choose a layer where drift returns and peaks fall back to t=0.Source: A Further analysis of the denoised… · p. 15
  • On ASEval, removing the benign mean alone leaves a 0.20 false-alarm rate; removing one covariance direction more returns false alarms to zero and raises AUROC from 0.750 to 0.912.The p=1 opening-peak share falls from 0.17 to 0.05, but its leading benign variation remains; p=2 also removes that leading variation.Source: A Further analysis of the denoised… · pp. 15–16, Table 4 · p. 16
  • The orthogonal projector achieves exact zero-mean anchoring, but its monitor results agree with sequential deflation within the reported uncertainty.The comparison covers ASEval Qwen3-8B only; the projector reduces residual benign drift, while AUROC differences at p=2 and p=16 are within ±0.007.Source: A Further analysis of the denoised… · p. 16

B Rule-based baseline: AgentSpec

  • AgentSpec blocks every overt in-domain attack on Slack, but over-blocks benign runs on some models, fails cross-domain transfer, and leaves decomposed MT attacks uncovered.The passage characterizes AgentSpec as a stronger specialist and DART as more robust to model, domain, and attack-shape transfer.Source: B Rule-based baseline: AgentSpec · p. 17, Table 6 · p. 17

C Detection across injection benchmarks

The same monitor is evaluated on single-turn indirect injection, where detection works without modification, but reminder-based mitigation can invert on some smaller models. The section also describes ActorBreaker as a secondary conversational check and frames additional probes beyond the main attack shapes.

C Detection across injection benchmarks

D Plain-direction results on the full multi-turn sets

  • ActorBreaker pairs 600 multi-turn attacks with 1,680 benign conversations sharing actor and topic but not intent; it has no tools or environment.It is presented as a secondary conversational check, and the plain direction already satisfies the denoising condition on Qwen3-8B and Qwen3-32B.Source: D Plain-direction results on the full… · p. 19

E Scope probes: non-injection detection and intrinsic misuse

  • The scope probes examine detection beyond the two main attack shapes: composition across permissible turns and indirect prompt injection.The passage states their purpose but does not report probe outcomes.Source: E Scope probes: non-injection… · p. 19

E.1 Scope: what this signal does and does not read

The transition signal extends beyond prompt injection, but its strength depends on how harm enters the agent’s context. Ablations favor transitions over states and find the direction primarily tracks injected-instruction structure, with a weaker harm-correlated component.

E.1 Scope: what this signal does and does not read

  • On R-Judge, transition drift reaches 0.77–0.80 AUROC under leave-one-domain-out cross-validation, with small-n IoT and Web domains noisier.Finance reaches approximately 0.95; Table 11 reports results across five domains.Source: §E.1 · pp. 19–20, Table 11 · p. 21
  • Cumulative drift detects multi-turn decomposition near-perfectly, although it lacks a foreign span to attribute and is not suppressed by withholding the crossing user turn.The crossing turn is one of the user’s own requests, and remaining turns still carry the decomposition.Source: §E.1 · p. 20
  • Held-out ASB AUROC tops out at 0.65–0.71; Figure 5 shows injected-vs-clean transitions separating while unsafe and safe behavior interleave.The figure’s 0.76 ASB AUROC is best-layer and in-sample; 0.65–0.71 is the held-out cross-validated range.Source: §E.1 · p. 20, Figure 5 · p. 21

F Ablations: aggregator, direction, and steering

  • The ablations favor a one-line reminder over soft-suffix steering, while direct steering works only under strong over-steering at a hand-tuned middle layer.The authors use the direction for detection and attribution, then act symbolically on the flagged context.Source: F Ablations: aggregator, direction,… · p. 21

F.1 Transition vs. state

  • Transition scoring averages 0.925 leave-one-suite-out AUROC versus 0.884 for state scoring, and recovers fragile held-out folds to 0.84–1.00.Differencing cancels the shared baseline, including depth- and length-dependent norm inflation; the deployed monitor uses cumulative transition scores.Source: §F.1 · pp. 21–22, Table 12 · p. 22

F.2 What does the direction encode?

  • The direction is primarily structural: it flags benign injected imperatives 98% of the time, yet held-out malicious and benign goals separate up to 1.00.Across layers 20–32, malicious-versus-benign separation averages 0.97; even action-matched benign goals separate at 0.89.Source: §F.2 · p. 22, Table 13 · p. 22

F.3 Mitigation lever: reminder vs. steering

A one-line reminder is the most reliable practical mitigation in the steering comparison, though quoting can backfire on some models. The section also reports fallback choices and clarifies that measured monitoring overhead is an upper bound.

F.3 Mitigation lever: reminder vs. steering

  • The reminder’s clean-flip rate is 0.20–0.52 and dominates every soft-suffix variant across models.Random-direction suffixes match or beat the injection-direction suffix on two larger models, suggesting suppression does not come from steering along that direction.Source: §F.3 · p. 23, Table 14 · p. 24
  • An always-on reminder preserves the tool call in 100% of clean contexts and the exact target function in 94%.The authors describe monitor gating as an optional refinement that saves reminders on unflagged turns.Source: §F.3 · p. 23
  • At its optimum, direct steering reaches 0.68 clean flip versus the reminder’s 0.52, but only in a hand-tuned middle-layer, strong-over-steering window.Gentle steering stays at or below 0.16, and late-layer steering nearly fails despite larger projection displacement.Source: §F.3 · p. 23, Table 15 · p. 24

F.4 What to quote, and what to do when quoting backfires

  • Quoting the attributed segment helps three larger models but raises Qwen3-8B ASR from 0.300 to 0.400; abstract warnings cost 0.06 ASR on average.Path-based top-2 attribution changes little relative to naming the single largest contributor in these short trajectories.Source: §F.4 · pp. 24–25, Table 16 · p. 25
  • Read-level withholding keeps ASR at or below 0.09, including reminder-inversion cases, but reminders preserve more task completion on five of six models.Halting is most reliable at ASR 0.00, but leaves only 0.00–0.24 completion on flagged runs.Source: §F.4 · p. 25, Table 17 · p. 26
  • The deployment rule is to use attributed reminders when calibration attacks show ASR falls, and otherwise fall back to read-level withholding.The gate is a property of the model, measured on the same red-team calibration set used to fit the direction.Source: §F.4 · p. 25

G Monitoring overhead

  • Measured on an M3 Ultra over MT-AgentRisk conversations, separate-prefill monitoring is an upper bound; integrated monitoring reuses computed states and pays only the dot product.The monitor reuses the agent’s weights and adds no resident model.Source: G Monitoring overhead · pp. 25–26, Figure 6 · p. 26
Loading 2610.00400v1…