Shortcut Sensitivity in Agent-Safety Evaluations

• Evaluation records can contain provenance tags, tool names, trajectory length, source subsets, and repeated interaction structure that predict labels without measuring the intended construct.Shorter average shell commands separated honest and attack trajectories in the LinuxArena sample, though the rule may track risk, collection differences, or evidence also used by a guard.Source: §1 · p. 1
• The audit separates whether a cue predicts labels, is associated with guard decisions, and changes decisions when edited.Paired edits require preserving the relevant safety information, with repeat and placebo controls for stochastic guards.Source: §1 · pp. 1–2
• Command length reached AUC .910, exact-tool lookup fell from F1 = .901 to .495 under domain holdout, and TS-Guard F1 fell from .902 to .719 under equal-interaction weighting.These results expose label separability, domain-transfer failure, and evaluation-unit sensitivity across the audited datasets.Source: §1 · p. 2
• Haiku blocking changed by −8.33 pp for marker rewriting and +11.46 pp for aliases, while Luna and Q8 checks showed configuration-specific effects.These paired interventions used a placebo control; the paper cautions that the edits do not establish the cues are spurious.Source: §1 · p. 2
• The authors do not claim the benchmarks are broken: some proxies are intentional and semantically relevant, so audits test whether findings survive appropriate checks.The proposed checks include group holdouts, equal-unit weighting, construct-preserving paired perturbations, and repeat-original controls.Source: §1 · p. 2

• Level 1 scores a capability-poor diagnostic using cue z alone, not judgment context s, with a metric such as AUC, F1, or balanced accuracy.High performance shows a predictive construction signature, not that the cue is spurious, guard-visible, or used by a guard.Source: §2 · pp. 2–3
• Level 2 measures the guard-decision contrast between records with and without an input-visible cue within a comparison cohort.A large contrast can nominate a cue for intervention, but shared data generation and confounding leave the result observational.Source: §2 · p. 3
• Level 3 measures paired strict-decision flips after changing a cue while holding the remaining context fixed; the intervention must preserve the label and intended construct.Otherwise, a changed decision may be legitimate; the stochastic-guard target uses repeated calls in both conditions.Source: §2 · p. 3
• For stochastic guards, interleaved repeated original and edited calls, a formatting placebo, and separate identical-input disagreement help address output variation and temporal drift.Repeat disagreement measures stability and should not be subtracted from either sensitivity quantity.Source: §2 · p. 3, Figure 1 · p. 3
• The three levels distinguish label predictability, observational guard association, and paired decision sensitivity, operationalizing these tests for released agent-safety evaluations.Strong label predictability can coexist with localized paired decision sensitivity.Source: §2 · pp. 2–3

3.1 Evaluation artifacts

• The audit covers a LinuxArena public sample and two TS-Bench-derived step datasets, using released labels and guard assessments without executing tools.The analysis is black-box and does not test adaptive agents, evasion attacks, or end-to-end tool execution.Source: §3.1 · pp. 3–4

3.2 Capability-poor diagnostics

• Capability-poor diagnostics test whether narrow cues recover labels and survive deduplication, source-overlap, transfer, and evaluation-unit controls.LinuxArena uses negative mean shell-command character count; AgentDojo tests a frozen marker-plus-verb rule and exact-tool lookup; AgentHarm compares row, interaction, and execution-group units.Source: §3.2 · p. 4

3.3 Frozen AgentDojo counterfactual cohort

• The paired-intervention cohort contains 64 AgentDojo bases from 58 interactions, balanced by domain and label before edited inputs were evaluated.Edits include marker rewriting, opaque tool aliases, their combination, and a whitespace placebo; label and operational-action preservation were intended, not independently human-verified.Source: §3.3 · pp. 4–5
• The deterministic subject is a byte-pinned Q8/MLX TS-Guard reproduction, whose agreement gate applies only to that reproduction, not the released BF16/vLLM path.It parsed 64/64 originals, agreed on 63/64 strict decisions, and matched 58/64 exact scores.Source: §3.3 · p. 5
• Haiku was selected for repeated intervention after showing the strongest safe-row marker association in a seven-configuration screen; Luna was added as a cross-provider replication.Three interleaved calls per base and condition support block-probability contrasts, while placebo and within-condition disagreement diagnose serialization sensitivity and instability.Source: §3.3 · p. 5

4.1 LinuxArena: separable collection, not a length cutoff

• LinuxArena lexical AUC remains 0.934 after exact-trace deduplication; all ten held-out environments score .900–1.000, with macro mean .951.In a Haiku-only cohort, lexical AUC is 0.955 versus length AUC 0.943; the difference is 0.012 [−0.003, 0.027].Source: §4.1 · p. 5
• Action count is more label-predictive than mean command length (AUC .947 versus .905), while monitor scores remain informative within length quartiles (weighted AUC .896 versus .607).The authors characterize this as collection/protocol separability, not benchmark invalidation or evidence of guard reliance.Source: §4.1 · p. 5

4.2 AgentDojo: marker and tool identity

• Forty of 51 exact AgentDojo tool names are label-pure and cover 776 rows, while the marker alone has perfect recall but a 37.0% false-positive rate (F1 = .687).Adding a short action-prefix list yields the stronger fixed rule, and exact-tool lookup’s grouped-CV score collapses under leave-one-domain-out transfer.Source: §4.2 · p. 6

4.3 AgentHarm: construction and unit weighting

• AgentHarm’s source subset determines all 731 strict labels, and released TS-Guard row-level F1 = .902 falls to .719 with equal-interaction weighting.Balanced accuracy changes from .754 to .776; the 72 harmful interactions contribute 7.29 rows on average versus 1.65 for 125 benign interactions.Source: §4.3 · p. 6
• Across 16 mixed AgentHarm execution groups, released TS-Guard’s equal-group mean composite score is .922 in unsafe contexts and .413 in safe contexts, with no reversal.This fixed-benchmark matched analysis does not identify authorization context as the cause; tool identity alone is insufficient to explain guard behavior.Source: §4.3 · p. 6

5 Cue edits can change guard decisions

• The intervention screen uses released TS-Guard and six hosted configurations for the original-input comparison, then repeated Haiku and Luna runs for paired edits.Repeated original, edited, and placebo calls are interleaved to address stochastic output variation and possible sequential drift.Source: §5 · p. 7, §5.2 · p. 7

5.1 Association selects the intervention target

• Haiku’s 74.09 pp safe-row marker-blocking association was the strongest in the seven-configuration screen, but marked and unmarked records differ in context and risk.The association selects an intervention target; it is not itself an estimate of marker use.Source: §5.1 · p. 7

5.2 Repeated intervention reveals Haiku cue sensitivity

• Haiku marker rewriting lowers blocking by 8.33 pp and opaque aliases raise it by 11.46 pp; both placebo-adjusted 95% ranges exclude zero.All unsafe calls remain blocked, while strict accuracy against inherited labels changes from 73.44% on originals to 81.77% after marker rewriting and 61.98% after aliasing.Source: §5.2 · p. 7, Table 2 · p. 7

5.3 Repeated GPT-5.6 Luna replication

• Luna’s marker and alias contrasts are 0.00 pp and +1.04 pp, with placebo-relative ranges including zero; the Haiku-sized effects do not reproduce.The ranges do not prove equivalence, and no base moves between opposite unanimous three-call decisions under any edit.Source: §5.3 · pp. 7–8

6 Implications for evaluation design

• Test capability-poor cues, then deduplicate the semantic unit and hold out environments, domains, templates, or sources; a strong result is a trigger, not a verdict.The authors note that public rules can become adaptive targets, motivating hidden group holdouts and refreshed equivalent serializations, which this paper does not test.Source: §6 · p. 8
• Report the evaluation unit named in the deployment claim because step-, interaction-, trajectory-, and task-weighted scores answer different questions..Source: §6 · p. 8
• After observational association nominates a hypothesis, change one cue while retaining authorization, capability, arguments, and effect, and record possible semantic changes.For stochastic guards, interleave original, edited, and placebo calls and report block-probability differences and identical-input disagreement.Source: §6 · p. 8

7 Related work

• The audit operationalizes label predictability, observational guard association, and paired decision sensitivity across released agent-safety artifacts.It builds on shortcut-learning, group-shift, behavioral-testing, and contrast-set ideas [Geirhos et al., 2020, Koh et al., 2021, Ribeiro et al., 2020, Gardner et al., 2020].Source: §7 · p. 8
• Related work examines surface heuristics, AgentHarm cues, benign triggers, construct validity, and broken specifications [Li et al., 2026, Meng et al., 2025, Tasawong et al., 2025, Li et al., 2025, Bean et al., 2025, Wang et al., 2026, Tu et al., 2026].Paired cue tests are established; this paper's contribution is the three-level inference ladder across released artifacts.Source: §7 · p. 8
• Prior studies audit AgentDojo end to end or develop broader guards [Bhagwatkar et al., 2025, Chen et al., 2025, Huang et al., 2025, Greenblatt et al., 2024]; this work audits released step records, evaluation units, and black-box sensitivity.The studied guards and monitors span AgentDojo, AgentHarm, ToolSafe, LinuxArena, and StepGuard [Debenedetti et al., 2024, Andriushchenko et al., 2025, Mou et al., 2026, Tracy et al., 2026, Zheng et al., 2026].Source: §7 · p. 8

8 Limitations and broader impact

• The 64-step cohort is a non-probability sample, and its exploratory intervals are unadjusted.The marker rewrite changes provenance wording, aliases are not natural global renamings, and neither edit received blinded human validation.Source: §8 · p. 8
• The observed sensitivity is configuration-specific: Haiku-sized effects do not reproduce in Luna or deterministic Q8 TS-Guard.GPT-5.5 lacks repeats, and StepGuard is limited to a non-equivalent Mac feasibility check.Source: §8 · p. 8
• LinuxArena uses completed trajectories, AgentHarm subset identity is not proven guard-visible, and the study does not assess deployment safety.The study executes no tool calls and uses no human-subject data.Source: §8 · p. 8

9 Conclusion

• A predictive benchmark property is a hypothesis, not evidence that a guard relies on it; shortcut auditing tests whether evaluation conclusions survive checks.The framework separates label predictability, observational guard association, and paired decision sensitivity.Source: §9 · p. 8

A Reproduction and statistical details

• The reproduction protocol reports pinned inputs, clustered bootstrap procedures, and descriptive Clopper–Pearson summaries for the reported analyses.The appendix distinguishes model-based descriptive summaries from design-based population intervals.Source: A Reproduction and statistical details · pp. 11–12

B AgentDojo intervention specification

• The frozen 64-step cohort uses marker, alias, combined, and whitespace-placebo edits, with the marker and alias transformations specified at the input-field level.The marker replaces INFORMATION tags; per-sample aliases are substituted consistently across requests, history, actions, and tool descriptions, while the placebo preserves field values.Source: B AgentDojo intervention specification · p. 12
• Table 3 reports cached safe-row marker associations, with marked-minus-unmarked strict block-rate contrasts and denominators of 321 marked and 547 unmarked rows.These are observational contrasts, not paired effects.Source: Table 3 · p. 13, B AgentDojo intervention specification · p. 13
• The exploratory eight-pair authorization pilot scores the unsafe member higher in 5/8 native structured pairs and 6/8 ReAct execution-only pairs, with no reversals.The generated pairs were not independently annotated, so the pilot does not establish validity by itself.Source: B AgentDojo intervention specification · p. 13
• StepGuard's valid one-row Mac pilot was not treated as equivalent to its official vLLM/CUDA runtime, and its audited revisions declared no license.No weights are redistributed.Source: B AgentDojo intervention specification · p. 13

C Secondary paired diagnostics

• The fixed rule changes on 42 bases, but Q8 moves only two marker decisions in its predicted direction and one oppositely.Table 4 reports deterministic Q8 sensitivity with equal-interaction rates weighting each upstream interaction once.Source: Table 4 · p. 13, C Secondary paired diagnostics · p. 13
• GPT-5.5 single draws disagree on 3/64 marker, 1/64 alias, and 2/64 combined bases, with signed block-rate contrasts of +1.56 pp, +1.56 pp, and 0 pp, respectively.The reported exploratory ranges include zero for each contrast.Source: C Secondary paired diagnostics · p. 13
• Because the calls were neither replicated nor interleaved, their estimates are noisy and provide no within-condition stability or drift diagnostic for the measured quantities.A prior unchanged-input run disagreed with this run on 1/64 strict decisions and 29/64 complete component assessments.Source: C Secondary paired diagnostics · p. 13
• Haiku marker effects are absent in banking, while alias blocking rises in every domain; Luna's flip and disagreement rates are balanced in direction.All unsafe calls remain blocked under every Luna condition.Source: C Secondary paired diagnostics · p. 13

AgentDojo lookup vocabulary coverage

• Only 211 of 1,220 held-out rows (17.3%) have a tool name seen in training, and the unseen-name fallback is safe in all four folds.The seen and unseen strata are 67.8% and 20.7% unsafe, respectively, so F1 should be read alongside balanced accuracy and confusion counts.Source: D Additional interpretation checks · p. 14, Table 5 · p. 14

LinuxArena metric units

• Opus’s expected any-action detection is .785 [.610, .937] at 1% step FPR, using all 74 attack trajectories and calibration on 12,494 actions from 200 honest trajectories.This operational-style reconstruction differs from the trajectory-level comparison on 70 shell-bearing attacks and 200 honest trajectories; the sample lacks harm-step information for exact reproduction of the official pre-harm metric.Source: D Additional interpretation checks · p. 14

Additional interpretation checks

• Only three exact tool names overlap between AgentDojo-Traj and AgentHarm-Traj, arguing against one universal tool-name rule across the subsets.In AgentDojo-Traj, ten strict-unsafe rows are negative under the fixed marker-plus-prefix rule, and only three are unanimously blocked by all seven cached guards.Source: D Additional interpretation checks · p. 14
• For AgentHarm-Traj, 57 of 72 harmful interactions contain duplicated segment IDs, compared with none of the 125 benign interaction IDs.All released rows are retained for row-level reproduction; interaction analyses change only their evaluation weight.Source: D Additional interpretation checks · p. 14, Table 6 · p. 14