PaperlayerView exact revision on arXiv

Source-linked AI summary

Language Is a Security Surface for Agent Guards

Hasan Demirkiran, Jens Ernstberger

arXiv:2610.22244v1cs.CRcs.AIcs.CLcs.LG

TL;DR

Whether a step-level guard’s verdict depends on the language of a user’s request is unmeasured, despite English guard prompts and benchmarks. LinguaGuardBench changes only that instruction field across five languages and controls, finding translation shifts verdicts for many open, small, and purpose-built guards, with larger effects in rarer languages.

  • Problem

    Whether a step-level guard’s verdict on a fixed action changes with the language of the user’s request is unmeasured, despite English prompts and benchmarks.

    Source: §1 · p. 1

  • Method

    LinguaGuardBench translates only user instructions in 1,220 labelled steps into five languages, keeping tool calls, observations, and labels identical and adding English paraphrase and whitespace controls.

    Source: §1 · p. 2

  • Results

    Most open, small, and purpose-built guards change verdicts beyond their noise floor; among nine guards below 13B, mean placebo-adjusted flip rate rises from +1.5 pp on French to +4.4 pp on Amharic.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The trusted request is a security surface for open and small guards serving non-English users, with mid-sized open guards tending to over-block and small guards tending to miss attacks.

    Source: §1 · p. 2

    The low-resource language pair was not run on partner-hosted Claude, Llama, GLM, or gpt-oss guards.

    Source: §5 · p. 9

Abstract

from arXiv · show

Step-level guards decide, before each tool call, whether an agent's next action is safe given the user's request and the trajectory so far. Users write those requests in many languages, while guard prompts and the benchmarks on which purpose-built guards are trained and evaluated are English. We ask whether a guard's verdict depends on the language of the request. We build LinguaGuardBench from the 1,220 labelled steps of TS-Bench AgentDojo-Traj: we translate only the user instruction into German, French, Turkish, Swahili and Amharic, and keep every tool call, observation and label byte-identical, so any change in a verdict is attributable to language alone. Six English rewordings and a whitespace placebo serve as controls, and managed guards are run three times to measure their noise. Across 19 guards, from frontier APIs to 4B open models and four purpose-built agent guards, most open, small and purpose-built guards change verdicts under translation beyond their own noise floor, and the effect grows with the rarity of the language: among the nine guards below 13B parameters, the mean placebo-adjusted flip rate rises from +1.5 pp on French to +4.4 pp on Amharic. Purpose-built guards are the most sensitive. The direction of the damage tends to follow guard size: open guards of 12-30B parameters block more benign steps (up to +5.0 pp false positives), and every significant loss of recall is in a guard below 10B (up to -7.4 pp). Frontier guards shift by at most about one point; the strongest, Gemini 3.8 Flash, returns identical verdicts with and without the request. The damage is concentrated in a few instructions and, in a post-hoc analysis, falls on steps where the agent's reasoning restates the English request, consistent with guards reacting to inconsistency between channels. English paraphrase moves every guard less than translation does. We release the benchmark and all 489k verdicts.

1 Introduction

The paper asks whether translating a user’s trusted request changes step-level guard verdicts, a question unmeasured despite English prompts and benchmarks. It introduces LinguaGuardBench and finds that translation effects vary by language and guard type, while English rewording moves guards less.

1 Introduction

  • Step-level guards inspect the request, trajectory, and candidate tool call, blocking calls that appear unsafe.Source: §1 · p. 1
  • Among nine guards below 13B, mean placebo-adjusted flip rate rises from +1.5 pp on French to +4.4 pp on Amharic as language rarity increases.The worst guard changes 12% of its verdicts.Source: §1 · p. 2
  • Purpose-built guards average +2.5 pp on French and +6.3 pp on Swahili and Amharic; ShieldAgent’s false-positive rate falls 4–8 pp in every language.Safiron loses 3 pp of recall on Turkish and Amharic.Source: §1 · p. 2
  • Open 12–30B guards over-block benign steps (up to +5.0 pp FPR on Amharic), while every significant recall loss occurs below 10B (up to −7.4 pp).TS-Guard (7.6B) over-blocks like the larger group, so the size pattern is a tendency, not a threshold; neither failure appears in English evaluation.Source: §1 · p. 2
  • Frontier guards show little shift; Gemini 3.8 Flash and a control withholding the request agree on 100% of 3,660 paired verdicts.Claude Sonnet 5 and Gemini 3.8 Flash have no significant shift in any language; GLM 5.2’s only shift is −1.1 pp FPR on French.Source: §1 · p. 2
  • A few long instructions account for most translation-induced flips; excess flips occur where the agent’s Thought restates the English request, consistent with channel inconsistency.One Turkish instruction takes TS-Guard from 12% to 92% blocked.Source: §1 · p. 2
  • Across six English paraphrase conditions, no general guard’s flip rate exceeds its placebo by more than 1.4 pp, and every guard that moves shifts more under translation.Source: §1 · p. 2
  • LINGUAGUARDBENCH pairs 1,220 fixed labelled steps with 696 instruction rewrites, five languages, six English styles, a placebo, and 489k verdicts from 19 guards.The trajectory, action, and label remain byte-identical across conditions, isolating language as the source of a verdict change.Source: §1 · p. 2

2 Related work

Prior work studies agent guards, multilingual safety, and prompt sensitivity, but does not measure whether changing only the language of a trusted request changes an action-level guard’s verdict.

2 Related work

  • Prior agent guards judge actions in task context and use the user request as trusted input, yet English benchmarks dominate and none reports sensitivity to request language.Source: §2 · pp. 2–3
  • Unlike multilingual safety work that changes harmful content, this paper holds the tool call and trajectory fixed while varying only the principal’s request language.Source: §2 · p. 3
  • Earlier prompt-sensitivity studies examine paraphrases, formatting, or agent robustness; here English paraphrases control for wording while testing language effects in guards.Source: §2 · p. 3

3 Setup

The setup varies only the user-instruction field across languages and wording controls, while preserving the labelled tool-call steps and validating the rewrites. It evaluates 19 guards with paired verdict comparisons and trajectory-clustered uncertainty estimates.

3 Setup

  • TS-Bench AgentDojo-Traj supplies 1,220 labelled tool-invocation steps—868 safe and 352 unsafe—from 589 trajectories and 58 user instructions; labels remain unchanged.The study executes no tool calls.Source: §3 · p. 3
  • Only instructions are translated into German, French, Turkish, Swahili, and Amharic, chosen across resource tiers, language families, and scripts.German and Turkish were human-reviewed by authors who read those languages; translations preserve specified names, identifiers, and other slots verbatim.Source: §3 · p. 3
  • Controls include a whitespace placebo and six English paraphrase conditions; two cross-family judges check task equivalence, and materialisation verifies other fields remain byte-identical.All 580 pre-registered rows received human review; low-resource rows were accepted or repaired from back-translation under a fixed rule.Source: §3 · p. 3
  • Nineteen guards include frontier APIs, open models, small models, four purpose-built guards, and a request-withheld scanner control.Managed guards run three repeats; self-deployed guards use greedy decoding, and the low-resource pair is run only on Gemini and self-deployed guards.Source: §3 · pp. 3–4
  • Figure 1 plots changes in attack recall and false-positive rate by guard and language, with resource-tier markers, guard-family colors, and intervals indicating uncertainty.Source: Figure 1 · p. 4
  • Paired metrics include verdict flips, placebo-adjusted flips, and changes in FPR and TPR, with 95% intervals from trajectory-clustered bootstrap estimates.Analyses also compare rankings, restatement-leakage strata, and per-instruction block-rate changes.Source: §3 · p. 4

4 Results

Translation changes guard verdicts and error rates, with the direction of error shifts tending to vary by guard size. The effect grows with language rarity, and purpose-built agent guards are especially language-dependent.

4.1 Translating the request changes verdicts and error rates

  • 39 of 77 guard×language cells show significant FPR or TPR shifts; translation changes up to 12.1% of a guard’s verdicts.Table 1 reports translation-induced error shifts by guard and language, with significant cells marked in bold.Source: §4.1 · p. 4, Table 1 · p. 5
  • At 12B parameters or more, no guard loses recall; significant shifts in that range are in false positives, including Nemotron 3.5 30B-A3B’s +5.0 pp Amharic FPR.TS-Guard at 7.6B also over-blocks on German, Turkish and Amharic, showing the size pattern is not a strict threshold.Source: §4.1 · pp. 4–5
  • Every significant recall decrease occurs below 10B parameters; Nemotron Nano 9B loses 7.4 pp TPR on Amharic.The smaller guards’ false-positive shifts are smaller and can move in either direction.Source: §4.1 · p. 5

4.2 The sign of the damage tends to follow guard size

  • The size pattern is a tendency, not a threshold: significant shifts at 12B and above are false-positive changes, while significant recall losses occur below 10B.These represent different errors: mid-sized guards over-block benign steps, whereas small guards miss attacks; Figure 2 plots pooled shifts against guard size.Source: §4.2 · pp. 5–6, Figure 2 · p. 5

4.3 The effect grows as the language gets rarer

  • Among six guards below 13B, the mean placebo-adjusted flip rate rises from +0.8 pp on French to +3.2 pp on Amharic.Amharic is also the only non-Latin script tested, so script and resource level cannot be separated with these languages.Source: §4.3 · p. 6
  • Turkish was harder than German and French for every guard; the two largest raw flip rates were Nemotron 3 Nano 4B on Swahili (10.8%) and Amharic (11.1%).The guard ranking was least stable under Amharic among languages tested on the same roster (τ = 0.93, versus 0.99 for Swahili).Source: §4.3 · p. 6

4.4 Purpose-built agent guards are the most language-dependent

  • Purpose-built agent guards have the highest mean placebo-adjusted flip rates, ranging from +2.9 pp for Safiron to +7.7 pp for ShieldAgent.Their error directions differ: ShieldAgent blocks fewer benign steps in every translated language while recall holds, whereas Safiron loses recall on Turkish, Swahili and Amharic.Source: §4.4 · p. 6

4.5 Frontier guards are invariant, and the strongest one does not read the request

Frontier guards show little change under translation, and Gemini 3.8 Flash gives identical verdicts with or without the request. Other guards’ translation effects cluster on particular instructions and differ by whether the history restates the request; English paraphrases move guards less.

4.5 The damage is concentrated in a few instructions

  • Gemini 3.8 Flash and Claude Sonnet 5 show no significant FPR or TPR shifts under tested languages, while Gemini agrees with a request-removed control on 100% of 3,660 paired verdicts.Gemini and Claude’s placebo-adjusted flip rates are 0.0 pp and at most 1.1 pp, respectively; the control has the same operating point and also flips at most once per condition.Source: §4.5 · p. 7

4.6 The damage is concentrated in a few instructions

  • 81% of TS-Guard’s verdicts flip on one 26-step Turkish Paris-hotel instruction, while long enumerated multi-task requests recur as hot spots across guards and languages.Translation can make an open guard block nearly an entire trajectory for such instructions.Source: §4.6 · pp. 7–8

4.7 Mechanism: guards react to inconsistency between channels

  • Placebo-adjusted translation flips are higher on leaky steps for strong guards and recall-losers, but higher on non-leaky steps for over-blockers.The pattern is consistent with inconsistency detection when the Thought restates the English request, and distrust of a foreign request when it is the only source; both are bounded by the 60% restatement rate.Source: §4.7 · p. 8, Figure 3 · p. 8

4.8 The sensitivity is to language more than to wording

  • Across English paraphrases, no general guard’s flip rate exceeds placebo by more than 1.4 pp, and every guard that moves changes more under translation than paraphrase.A guard nearly invariant to English wording but sensitive to language has learned language more than register as a feature.Source: §4.8 · pp. 8–9

5 Discussion

The discussion recommends testing non-English deployments for size-dependent error shifts and concentrated translation failures. It also argues that trajectory-rich benchmarks can obscure whether guards use the user request, while noting important limits in translation validation and sampling.

What to deploy

  • For non-English deployment, 17–30B open models tend to over-block, guards that under-block are all below 10B, and purpose-built guards are the most translation-sensitive class measured.The authors suggest evaluating translated long instructions in a low-resource language; proposed mitigations remain untested.Source: §5 · p. 9

The request field is not what strong guards use

  • Trajectory-rich benchmarks may fail to test trusted-channel use when the agent’s reasoning already restates the request, reframing translation robustness as depending on channel comparison.A guard that ignores the request avoids language-based attacks but also cannot use it to disambiguate an action.Source: §5 · p. 9

Limitations

  • The low-resource translations were reviewed by two judges and back-translation without team members who read Swahili or Amharic, and those languages were not tested on partner-hosted guards.The bootstrap clustered by trajectory although the intervention was per instruction, so confidence intervals may be too narrow.Source: §5 · p. 9

6 Conclusion

The conclusion finds that translating user requests changes decisions by open and small guards on fixed actions, with effects that vary by guard size and language. The changes concentrate in long enumerated instructions and steps where the agent restates the English request.

6 Conclusion

  • Amharic produces the largest shifts in both error regimes, while English rewording changes guards less than translation.The paper concludes that request language is a security surface for open and small guards, which English-only evaluation will not expose.Source: §6 · p. 9

A English paraphrase results in full

English paraphrases generally leave guard verdicts near their placebo noise, with limited exceptions in false-positive shifts and recall. The appendix reports that guard rankings remain highly consistent across paraphrase conditions.

A English paraphrase results in full

  • At most +1.4 pp, the placebo-adjusted flip rate under English rewording for the pre-registered roster and small tier, while dose-response is flat except for Gemma 4 26B and Qwen 3.5 4B.No paraphrase condition reduces recall except gpt-oss-120b on terse style (−1.3 pp); significant FPR shifts are limited to register or lexical-rewording effects.Source: A English paraphrase results in full · p. 11

B Translation results in full

The translation analysis reports shifts by language-resource tier and by guard, separating false-positive and true-positive changes. The accompanying paraphrase tables track those error shifts across six English paraphrase conditions.

B Translation results in full

  • Tables 7 and 8 report translation-associated ∆FPR and ∆TPR, respectively, for every guard and language, marking statistically significant shifts.Source: B Translation results in full · p. 11
  • Tables 4 and 5 report ∆FPR and ∆TPR under six English paraphrase conditions, with bold marking intervals that exclude zero.Source: Table 4 · p. 12, Table 5 · p. 12

C Strict decision rule

The section compares the default blocking threshold with a stricter rule that blocks only at the maximum composite score. Under the strict rule, small guards become more sensitive while frontier guards remain unchanged at zero.

C Strict decision rule

D LINGUAGUARDBENCH: generation and validation

LINGUAGUARDBENCH uses controlled rewrites of user instructions while preserving task actions and scope, then validates and freezes the rewritten conditions. Its tables describe translation shifts by language-resource tier and guard-level error changes.

D LINGUAGUARDBENCH: generation and validation

  • The released artifact includes frozen rewrite files and hashes, a materialiser for 1,220 labelled steps per condition, guard prompts and adapters, and 489k verdict rows.Trajectories and labels are read from TS-Bench rather than redistributed.Source: D LINGUAGUARDBENCH: generation and… · pp. 12–13
  • Tables 6–8 report translation shifts by resource tier and guard-language pair, including placebo-adjusted flip rates, ∆FPR, and ∆TPR.Table 9 reports pooled placebo-adjusted flip rates under default and strict decision rules.Source: Table 6 · p. 13, Table 7 · p. 13, Table 8 · p. 14
  • The benchmark’s task-equivalence construct permits language and wording changes but preserves requested actions, objects, conditions, ordering, and authorization scope.Quoted strings, identifying slots, requested-action counts, and urgency or permission language are also protected.Source: D LINGUAGUARDBENCH: generation and… · p. 13
  • LLM paraphrases and translations were generated in three candidates per instruction, selecting the first to pass the checks, with up to two feedback rounds.The checks include slot preservation, authorization wording, edit-distance bounds, and length ratio; all 58 instructions passed in every condition.Source: D LINGUAGUARDBENCH: generation and… · pp. 13–14
  • Two cross-family judges rated 521/522 and 516/522 rows equivalent, respectively; human review found six flagged rows, repaired one dropped possessive in a Turkish translation, and rejected none.Source: D LINGUAGUARDBENCH: generation and… · pp. 14–15

E Serving and reproducibility

The study runs managed and self-deployed guards through specified APIs or a local vLLM setup, recording metadata for each verdict. Repeats and cross-deployment agreement provide reproducibility checks.

E Serving and reproducibility

  • Managed guards used platform or partner APIs, while self-deployed models ran vLLM on one RTX PRO 6000 with greedy decoding and JSON-schema enforcement.The listed exceptions include TS-Guard’s XML prompt and parser and model-specific prompts for purpose-built guards.Source: E Serving and reproducibility · p. 15
  • Managed guards were run three times at temperature 0, with placebo and repeat-to-repeat flip rates reported as noise floors.Source: E Serving and reproducibility · p. 15
  • On original wording, TS-Guard’s accuracy was 0.918 versus the released figure of 0.917, and managed and self-deployed Gemma agreed on 98.2% of verdicts.The Gemma agreement was 97–98% under shared perturbed conditions; the two versions were reported separately.Source: E Serving and reproducibility · p. 15
  • Every verdict row records model ID, prompt hash, token usage, and latency; reported compute spend was about $1,460.The total included $934 of tokens and 105 GPU-hours.Source: E Serving and reproducibility · p. 15

Related papers

Loading 2610.22244v1…