PaperlayerView exact revision on arXiv

Source-linked AI summary

Threat-Preserving Representation Sensitivity in Agent-Security Benchmarks

Neeraj Karamchandani, Piyush Nagasubramaniam, Xinhong Xie, Sencun Zhu, Dinghao Wu

arXiv:2610.03585v1cs.CRcs.AIcs.LG

TL;DR

Agent-security benchmarks use ASR to compare models and defenses, but it is unclear how stable that score is when the benchmark changes what an agent sees about the threat. The paper measures threat-preserving representation sensitivity by changing the agent-visible representation while holding the security problem fixed. Across three benchmarks, measured effects varied, including substantial ASR shifts on ASB and MCPTox and a smaller ASR effect with a benign-utility shift on AgentDojo.

  • Problem

    Because benchmark presentation is part of the agent's input, the paper asks how stable security scores remain when representation changes but the underlying security problem and evaluation criteria stay fixed.

    Source: §I · p. 1

  • Method

    The paper introduces TPRS to measure ASR changes across threat-preserving representation changes, using three benchmarks while holding underlying security-relevant components fixed.

    Source: §I · p. 2, A. Threat-Preserving Transformations · p. 4

  • Results

    Across benchmarks, effects were nonuniform: neutralizing ASB names raised committed ASR by 11.67 percentage points on GPT-5-mini, threat-explicit MCPTox names lowered ASR, and AgentDojo's ASR effect was smaller despite a benign-utility shift.

    Source: §I · p. 2, §VII · p. 11

  • Takeaways & Limitations

    A security score measured under one representation may not generalize to other threat-preserving representations, so the paper concludes that benchmarks should characterize this sensitivity.

    Source: §V · p. 10, §VII · p. 11

    GPT-5-mini may not be exact under Haiku 4.5's tokenizer, so the sensitivity attribution is not completely resolved.

    Source: §VI · p. 11

Abstract

from arXiv · show

Security benchmarks for LLM-based agents often report the attack success rate (ASR) as a measure of model robustness and use these scores to compare different models and defense mechanisms, assuming that they describe the security of the agent. In this paper, we explore whether it also influences the benchmark's measurement. To measure the effect of the benchmark representation, we introduce threat-preserving representation sensitivity (TPRS), which measures how much the ASR changes when we change the agent-visible representation while holding the underlying task, harmful action, security policy, ground truth, environment, and the evaluation criteria fixed. On Agent Security Bench (ASB), replacing threat-related tool names with threat-neutral names raises the committed attack success rate by 11.67 percentage points on GPT-5-mini and by 13.21 points on Claude Haiku 4.5. On MCPTox, replacing the original neutral tool name with an explicit threat-related name lowers the ASR by 11.00 percentage points on GPT-5-mini and 4.11 points on Claude Haiku 4.5. On AgentDojo, adding threat-related wording to the attack-relevant tool changes ASR by only 0.50 percentage points on GPT-4o-mini, yet the benign utility falls by 5.36 points on tasks requiring that tool. We ran an experiment on MCPTox where we observed that a threat-neutral name matched on token count, length, and casing reproduces most of the shift produced by the threat-explicit name (8.54 of 11.00 points on GPT-5-mini). The results show that a security score measured under one representation may fail to generalize across threat-preserving representations of the same security problem. Robustness claims should therefore be supported by performance across a controlled set of threat-preserving representations rather than relying on a single representation-dependent score.

I. INTRODUCTION

Agent-security benchmarks use ASR to assess robustness, but their agent-visible choices may affect the behavior and scores they measure. This paper introduces TPRS to test score stability under threat-preserving representation changes and finds that sensitivity varies across benchmarks and evaluation outcomes.

I. INTRODUCTION

  • TPRS measures ASR change when the agent-visible representation changes while the underlying security problem and evaluation criteria remain fixed.The paper re-runs three benchmarks with changes to tool names or other agent-visible representations while keeping the actual threat, environment, and evaluation criteria fixed.Source: §I · pp. 1–2
  • Neutralizing threat-related names raised committed ASR by 11.67 percentage points on GPT-5-mini and 13.21 points on Claude Haiku 4.5 in ASB, while threat-explicit names lowered ASR by 11.00 and 4.11 points, respectively, on MCPTox.The study reports results across 28,904 agent runs and three attack architectures.Source: §I · p. 2
  • On AgentDojo, the ASR change was only −0.50 points on GPT-4o-mini, while benign utility fell 5.36 points on tasks requiring the renamed tool.A form-matched threat-neutral name reproduced 8.54 of the 11.00-point GPT-5-mini shift on MCPTox; ASB sensitivity also differed by success definition.Source: §I · p. 2
  • The authors argue that a score measured under one representation may not generalize to other threat-preserving representations, without identifying one universally correct representation.They distinguish attacker-controlled representation, which can become part of an attack, from designer-controlled representation, which can reveal dependence on benchmark choices.Source: §I · p. 2
  • Benchmark security scores should characterize representation sensitivity rather than treat a single representation as measurement-neutral.This matters when a score is used to compare models or defenses, because conclusions from one configuration may be narrower than the score suggests.Source: §I · pp. 1–2

II. BACKGROUND AND RELATED WORK

Agent-security benchmarks provide scores used to compare models and defenses, while prior work shows that tool metadata and evaluation context can shape agent behavior. This paper asks how much existing benchmark scores shift when only the threat's agent-visible representation changes.

A. Agent-Security Benchmarks

  • ASB, MCPTox, and AgentDojo cover varied agent-security settings, including diverse attacks and defenses, MCP tool poisoning, and prompt-injection susceptibility alongside benign utility.These benchmarks are used to evaluate models and defenses, including comparisons of attack success and task utility.Source: A. Agent-Security Benchmarks · p. 2
  • The study uses the three benchmarks to change threat representation while keeping the underlying task, harmful action, security policy, success criteria, environment, and evaluation criteria fixed.The figure describes TPRS as the resulting change in ASR and notes that the transformed representation appears in different locations across the benchmarks.Source: A. Agent-Security Benchmarks · p. 3, Figure 1 · p. 3

B. Representation and Context Sensitivity in LLM Agents

  • Prior work reports that tool metadata can alter agent actions, and that evaluation-context cues can change model behavior or benchmark performance.Examples include changes in tool use following metadata modifications and unsafe tool execution under schema-formatted specifications.Source: B. Representation and Context… · p. 3
  • This paper narrows the question to how much an existing benchmark's security score shifts under controlled representation changes when the security problem defining the outcome stays the same.The related-work discussion contrasts this benchmark-focused question with prior work on meaning-preserving rewrites and evaluation-context sensitivity.Source: B. Representation and Context… · p. 3

C. Benchmark and Measurement Validity

Benchmark scores are measurements whose conclusions can depend on evaluation design; this work extends that validity question to threat representations in agent-security benchmarks. It defines threat-preserving changes and examines whether ASR remains stable when only the agent-visible representation changes.

C. Benchmark and Measurement Validity

  • Benchmark flaws can bias performance measurement, and prior work finds conclusions depend on benchmark design; this work extends that validity question to threat representations in agent-security benchmarks.The paper asks whether security scores change when threat representation changes while the underlying security problem and evaluation criteria remain fixed.Source: C. Benchmark and Measurement Validity · pp. 3–4

III. METHODOLOGY

  • The methodology treats benchmarks as measurement tools and tests whether attack-success measurements stay stable when threat representation changes but the threat and evaluation remain fixed.It defines a threat-preserving transformation and describes its application to three benchmarks.Source: §III · p. 4

A. Threat-Preserving Transformations

  • A threat-preserving transformation changes only the agent-visible representation, keeping the environment, objective, violation policy, success criterion, and evaluation mechanism fixed.The evaluation rules are unchanged across experiments.Source: A. Threat-Preserving Transformations · p. 4
  • Orthographic, semantic-neutral, and cue-altering transformations test sensitivity to spelling, non-threat wording, and threat-specific wording, respectively.Movement under cue-altering changes but not semantic-neutral ones indicates sensitivity to threat signaling specifically; movement under both indicates broader sensitivity to descriptive content.Source: A. Threat-Preserving Transformations · p. 4

B. Per-Benchmark Construction

The study uses three independently developed benchmarks with different attack architectures and representation changes. Each benchmark is analyzed independently rather than comparing effect sizes across benchmarks.

Per-Benchmark Construction

  • The three benchmarks use distinct attack architectures, are analyzed separately, and test representation sensitivity across different security-evaluation designs.ASB presents a dedicated malicious tool, MCPTox poisons benign-tool metadata, and AgentDojo injects untrusted text through tool output.Source: B. Per-Benchmark Construction · pp. 4–5

ASB

  • ASB scores success through a fixed attack-goal sentence, while its experiments change agent-visible information such as tool names, descriptions, and injected-instruction framing.The attack-goal sentence is kept byte-identical; A2 changes task text and is used only as a diagnostic, not as a threat-preserving arm.Source: B. Per-Benchmark Construction · p. 5

MCPTox

  • MCPTox tests whether agents perform actions requested by poisoned tool descriptions, so renaming tools is threat-preserving and adds threat wording to originally neutral names.Its S3 threat-neutral control matches N1 on measured identifier properties; the experiment covers 203 cases, two models, nine arms, and three replicates.Source: B. Per-Benchmark Construction · p. 5

AgentDojo

  • AgentDojo injects untrusted text through tool output to misuse ordinary tools also needed for legitimate tasks, and tool names change without changing the environment-state success evaluator.Threat wording is added either to the attack-required tool or to an unused tool to test whether the effect depends on the needed capability.Source: B. Per-Benchmark Construction · pp. 5–6

C. Generation and Freezing

Transformations are generated deterministically from a fixed vocabulary and frozen before outcomes are observed. The experiments preserve all non-target fields and use a shared threat lexicon to quantify threat wording.

C. Generation and Freezing

  • Transformed instances use deterministic substitutions from a fixed vocabulary, are saved and hashed, and are not regenerated after model outcomes are observed.Candidate names are selected using tokenizer statistics, a fixed threat lexicon, and original-identifier properties, not model behavior.Source: C. Generation and Freezing · p. 6
  • Threat-preserving comparisons change only pre-specified representation fields, leaving tool order, schema position, non-target fields, task text, initial environment, and evaluation settings unchanged.AgentDojo randomizes execution order across arms within each replicate but not the tool order shown to the model.Source: C. Generation and Freezing · p. 6
  • Using one threat lexicon, threat words occur in 80.6% of ASB A0 tool names, 0% of ASB A1 and A3 names, 2.7% of MCPTox names, and 0% of AgentDojo names.The lexicon matches whole name components rather than substrings.Source: C. Generation and Freezing · p. 6

D. Model Configuration

The experiments use benchmark-specific model configurations and scoring procedures. They distinguish ASB's native and committed ASR, limit MCPTox scoring to an explicitly characterized subset, and account for shared transformed identifiers in the analysis.

D. Model Configuration

  • GPT-5-mini and Claude Haiku 4.5 are evaluated on ASB and MCPTox, while AgentDojo uses GPT-4o-mini with its native harness; repeated runs and replicate variability are reported.Provider errors and runs without a usable plan are treated as execution failures rather than attack outcomes.Source: D. Model Configuration · p. 6

E. What We Measure

  • ASB's native ASR counts harmful-tool invocation even when the agent refuses, whereas committed ASR counts success only when the agent never refuses.Between 5% and 28% of ASB's scored successes include a refusal; MCPTox scoring was reimplemented and validated on 203 of 1,348 cases, a subset not uniformly representative of all categories.Source: E. What We Measure · p. 6

1) What We Treat as Independent:

  • Runs sharing a transformed identifier are not independent, so the analysis clusters at the level where each benchmark's transformation is applied.Transformations are applied per tool in ASB, per case in MCPTox, and per base case in AgentDojo.Source: 1) What We Treat as Independent · p. 6

2) How We Compare Arms:

The study compares paired benchmark arms with confidence intervals, accounts for execution failures, and uses separate statistical procedures for the benchmarks. Across the three benchmarks, representation effects differ in direction and magnitude.

2) How We Compare Arms:

  • The comparisons pair the same units across arms where possible; ASB instead pairs groups of five task outcomes by tool and replicate.Some Haiku 4.5 groups are smaller because 229 of 3,240 runs had execution failures.Source: 2) How We Compare Arms · p. 6

3) Confidence Intervals:

  • Paired ASR differences use 95% confidence intervals from 10,000 cluster-bootstrap samples and a reproducible pipeline with fixed random seeds.Source: 3) Confidence Intervals · p. 7

4) Handling Failed Runs:

  • ASB Haiku 4.5 execution failures are analyzed under three rules: drop failures, count them as not committed, or count them as committed.Failure rates differ across arms, including 4.1% in A0 and 5.9% in A1.Source: 4) Handling Failed Runs · p. 7

5) Statistical Comparisons:

  • MCPTox reports Holm-adjusted p-values for multiple hypotheses, while ASB and AgentDojo report arm-versus-baseline confidence intervals without multiplicity correction.ASB cross-model differences are estimated directly.Source: 5) Statistical Comparisons · p. 7

G. Additional Design Choices

  • The MCPTox S3 arm and its N1–S3 contrast were added after initial results prompted a follow-up, with the transformation and analysis rule fixed before S3 outcomes.The authors treat S3 as an adaptively motivated follow-up control, not part of the original design.Source: G. Additional Design Choices · p. 7
  • The ASB candidate pool was expanded before model runs to improve token and length coverage, without using behavioral results or changing hypotheses or analysis rules.Source: G. Additional Design Choices · p. 7

IV. RESULTS

  • Representation effects differ by benchmark and transformation: removing threat wording raises ASB ASR, adding it lowers MCPTox ASR, and AgentDojo effects remain near zero.The benchmarks are analyzed independently; the common axis visualizes effect direction and magnitude only.Source: §IV · p. 7

A. ASB: Removing Threat-Related Tool Names

Neutralizing ASB attacker-tool names increases measured attack success, and the effect persists under alternative success definitions and failure treatments. The experiments establish representation sensitivity, but do not isolate threat wording as its mechanism.

b) The headline result holds under both native ASR and committed ASR definitions

  • Native ASR also rises under A1: 14.07 points on GPT-5-mini and 13.05 points on Haiku 4.5, relative to A0.Native ASR counts a malicious tool invocation as success; committed ASR additionally requires no refusal.Source: A. ASB: Removing Threat-Related Tool… · p. 7

c) The naming effect is robust to execution failures and individual tools

  • Across three execution-failure treatments, Haiku 4.5’s A1 effect ranges from 12.04 to 13.89 points, with every confidence interval above zero.GPT-5-mini had no execution failures.Source: A. ASB: Removing Threat-Related Tool… · p. 7
  • Leaving out any one of 36 tools keeps the A1 effect between 9.90 and 12.95 points on GPT-5-mini and 11.19 and 14.40 points on Haiku 4.5.The estimated cross-model difference is −1.54 points for A1 (95% CI [−12.45, +8.44]).Source: A. ASB: Removing Threat-Related Tool… · p. 7

d) The ASB result establishes sensitivity, but not its mechanism

  • The ASB result establishes sensitivity but does not identify which representation feature caused it, because A1 changed both wording and token use.Average token counts were 2.83 versus 3.47 for GPT-5-mini’s tokenizer and 5.39 versus 5.86 for Haiku 4.5’s.Source: A. ASB: Removing Threat-Related Tool… · p. 7
  • The planned matched neutral-name arm was not run: only 10 of 30 candidates that met structural matching requirements passed the function-preservation check.Source: A. ASB: Removing Threat-Related Tool… · pp. 7–8

a) Using neutral tool names increases the attack success rate

e) Diagnostic A2 arm

  • A2 is diagnostic rather than threat-preserving because it changes the task specification; its committed-ASR effects are +4.44 points on GPT-5-mini and −2.23 on Haiku 4.5.Haiku 4.5 had a 22.4% execution failure rate in A2.Source: A. ASB: Removing Threat-Related Tool… · p. 8

B. MCPTox: Adding Threat-Related Tool Names

Adding threat-related names lowers MCPTox ASR, especially for GPT-5-mini. A matched threat-neutral naming control reproduces much of that model’s shift, while evidence for the corresponding Haiku 4.5 contrast is inconclusive.

B. MCPTox: Adding Threat-Related Tool Names

b) Threat-related wording alone cannot explain the observed N1 effect

  • The threat-neutral S3 control lowers GPT-5-mini ASR by 8.54 points, while N1 lowers it by only 2.46 points relative to S3.N1 versus the broader S1/S2 semantic-neutral controls lowers ASR by 8.37 points.Source: B. MCPTox: Adding Threat-Related Tool… · p. 9

c) Haiku 4.5 shows a different pattern

  • For Haiku 4.5, S3 changes ASR by −0.16 points and N1 lowers ASR by 3.94 points relative to S3, but the latter contrast does not survive Holm correction.The paper describes the Haiku 4.5 evidence as inconclusive.Source: B. MCPTox: Adding Threat-Related Tool… · p. 9

d) Multiplicity

  • The S3 arm and N1–S3 contrast were added after initial results; their comparisons are treated as an adaptively motivated follow-up.The five-contrast-per-model Holm family includes N1 versus S3.Source: B. MCPTox: Adding Threat-Related Tool… · p. 9

a) Adding threat-related wording lowers the attack success rate on GPT-5-mini

d) Multiplicity

  • Only one of ten reported hypothesis tests remains significant: N1 versus semantic-neutral names on GPT-5-mini, with ∆= −8.37 points and adjusted p < 10^-4.N1 versus S3 on Haiku 4.5 has adjusted p = 0.068 and does not survive correction.Source: B. MCPTox: Adding Threat-Related Tool… · p. 9

C. AgentDojo: Small ASR Effect, Utility Shift

Adding threat-related wording to AgentDojo’s attack-relevant tool produces a near-zero ASR change, while benign utility falls on tasks requiring that tool. The utility shift is suggestive, not established.

C. AgentDojo: Small ASR Effect, Utility Shift

  • Because AgentDojo’s attacks misuse benign tools also needed for legitimate tasks, the study changes tool names only for injection tasks whose attack path requires that tool.Source: C. AgentDojo: Small ASR Effect,… · p. 9

a) The observed ASR effect is small

b) The same change affects benign utility

D. Cross-Benchmark Findings

Across the three benchmarks, TPRS reveals that ASR sensitivity can be large but is not uniform, and security conclusions can change across threat-preserving representations. The discussion connects these findings to benchmark design and identifies limits in the tested transformations, attribution, and empirical scope.

D. Cross-Benchmark Findings

  • ASB and MCPTox showed large ASR shifts in opposite directions, while AgentDojo’s ASR change was smaller despite a larger benign-utility change.Across benchmarks, sensitivity was not uniform; security scores also depended on how success was defined.Source: D. Cross-Benchmark Findings · p. 10, §V · p. 10
  • Most of GPT-5-mini’s MCPTox effect was reproduced without explicit threat vocabulary.The discussion concludes that a benchmark’s security conclusions may differ under another threat-preserving representation, motivating TPRS before interpreting a score as robustness evidence.Source: D. Cross-Benchmark Findings · p. 10, §V · p. 10

V. DISCUSSION

  • Benchmarks making generalized security claims should test threat-preserving variants, report sensitivity, and measure benign utility when representations affect it.They should also state how attack success is defined and report the benchmark configuration alongside sensitivity.Source: §V · p. 10
  • When attackers choose tool names or metadata, representation can become part of the attack; ASB’s original names understate attacker-achievable committed ASR by 11.67–13.21 points.The discussion recommends reporting TPRS and the maximum ASR across tested representations in such cases.Source: §V · pp. 10–11
  • AgentDojo’s small ASR change contrasts with ASB and MCPTox, but the paper does not establish why the sensitivity differed.One proposed possibility is greater representation importance when a benchmark explicitly states adversarial capability; this remains for future testing.Source: §V · p. 11

VI. LIMITATIONS AND FUTURE WORK

  • Transformations covered tool names and descriptions, not all representation types; ASB A1 names were not independently validated for semantic equivalence.Too few names qualified for the proposed semantically equivalent control arm, so it was not run.Source: §VI · p. 11
  • The MCPTox control reproduced most of GPT-5-mini’s shift, but did not isolate every identifier property, leaving sensitivity attribution unresolved.The token-count tokenizer may not match Haiku 4.5’s tokenizer; tighter controls could vary semantics, tokenization, and identifier structure.Source: §VI · p. 11
  • Evidence covers three benchmarks and three model configurations; the study does not establish sensitivity’s scale or generalization beyond that scope.The 28,904 runs are not necessarily independent because transformations were shared; future work includes broader benchmarks, models, attacks, and defenses, as well as ranking stability under transformations.Source: §VI · p. 11

VII. CONCLUSION

Across 28,904 runs on three benchmarks, security scores depended on how attacks were represented, with effects varying across benchmarks and sometimes including benign utility changes. The paper introduces TPRS to measure this dependence and concludes that security claims beyond one interface should be tested across controlled threat-preserving variants.

VII. CONCLUSION

  • TPRS measures representation effects while the underlying task, harmful action, security policy, ground truth, and evaluation criteria stay fixed.The paper notes that agent-security benchmarks are often interpreted as measuring model robustness to an attack.Source: §VII · p. 11
  • Across 28,904 runs, TPRS was large in some settings, increased or decreased security scores, and varied across benchmarks.ASB’s committed ASR increased when threat-related names became neutral, MCPTox ASR decreased when threat-related names were introduced, and AgentDojo’s ASR effect was smaller despite a benign-utility change.Source: §VII · p. 11
  • For claims intended to extend beyond one interface, benchmarks should evaluate controlled threat-preserving variants rather than treat one representation as measurement-neutral.Source: §VII · p. 11
Loading 2610.03585v1…