PaperlayerView exact revision on arXiv

Source-linked AI summary

SentryBench: A Cost-Sensitive Benchmark for Adversarial Robustness of Tool-Using LLM Agents in Public-Safety Deployments

Md Abu Yusuf, Oyshi Tabassum Aditi, Md Faysal Hosain Mazed

arXiv:2610.34535v1cs.CRcs.AI

TL;DR

Public-safety agents can cause harm through both missed emergencies and unwarranted actions, a two-sided risk that conventional refusal-focused benchmarks do not capture. SentryBench evaluates that risk with real-data scenarios and cost-sensitive measures; across 840 episodes, targeted trust gating has the lowest harm on every model, while refuse-more caution has the worst harm.

  • Problem

    Conventional agent-safety benchmarks focus on refusing harmful requests, while public-safety agents also risk missing emergencies or taking unwarranted actions.

    Source: §I · p. 1

  • Method

    SentryBench evaluates adversarial tool use with real emergency and disaster content, real injection and jailbreak payloads, benign controls, and cost-sensitive outcome measures.

    Source: §I · p. 1

  • Results

    Across 840 episodes, TGCC had the lowest harm on every model, while caution had the worst harm everywhere.

    Source: §X · p. 6

  • Takeaways & Limitations

    Public-safety agents should be evaluated by scenario-calibrated harm, with provider-filter effects reported separately from model behavior.

    Source: §IX · p. 6

Abstract

from arXiv · show

Large language model (LLM) agents are increasingly wired to consequential tools that send public alerts, dispatch responders, and query citizen records, where one erroneous tool call carries real-world cost. Existing agent-safety benchmarks measure generic harm or prompt-injection success and implicitly treat refusing as the safe default. In public-safety deployments that assumption breaks: an agent too cautious to dispatch help fails as surely as one that triggers a false alert. We present SentryBench, which scores agents on a two-sided, cost-sensitive frontier jointly measuring attack success, over-refusal (missing a real emergency), and over-reaction (unwarranted action) under adversarial tool use, with scenarios grounded in real emergency-call transcripts, disaster messages, and real injection/jailbreak payloads. We formalize deployment harm as an asymmetric cost functional, prove naive caution is harm-dominated whenever it refuses more genuine emergencies than the attack harm it averts, and prove a targeted trust gate adds zero over-refusal on clean input. Across 840 scored episodes over three frontier models and three defenses, refuse-more caution drives over-refusal to 100 percent on two of three models for at most a 3.7-point change in attack success and yields the worst harm everywhere, whereas our trust-gating wrapper dominates the frontier (lowest harm on every model, over-refusal at or below baseline, higher containment). Provider content filters also silently perform part of the defending and must be reported separately.

I. INTRODUCTION

Public-safety agents can cause harm by missing genuine emergencies or taking unwarranted actions, while conventional chat-safety benchmarks treat greater refusal as safer. SentryBench evaluates this two-sided risk and asks whether targeted trust gating can reduce attacks without the over-refusal cost of blanket caution.

I. INTRODUCTION

  • Public-safety agents can harm by missing genuine emergencies or taking unwarranted actions, making refusal-only safety evaluation inadequate.Conventional chat-safety benchmarks optimize refusal of harmful requests, but public safety requires jointly measuring both error directions with asymmetric costs.Source: §I · p. 1
  • SentryBench evaluates public-safety agent vulnerabilities, caution’s effect on over-refusal, targeted trust gating, and provider-filter confounding.Its research questions cover injection, poisoning, goal hijacking, over-trust, and privilege escalation.Source: §I · p. 1
  • The benchmark spans four subdomains, five attack vectors, and three outcome classes, with urgent and ambiguous benign controls.It uses legitimate emergency-transcript and disaster-message content, real injection and jailbreak payloads, and no real personally identifiable information.Source: §I · p. 1
  • Its cost-sensitive framework treats over-refusal and over-reaction as first-class metrics and compares defenses using harm and a precision–recall frontier.The authors state that naive caution is harm-dominated in the observed regime and that their trust gate adds no over-refusal on clean input.Source: §I · p. 1

II. RELATED WORK

The section situates SentryBench among agent-safety research, then specifies its threat model and formalizes a cost-sensitive framework for comparing attacks, over-refusal, and over-reaction. Its formal results show when caution increases harm and how a detector-based trust gate limits attack success without adding over-refusal on clean inputs.

II. RELATED WORK

  • The related-work section traces agent capabilities to tool-use systems and surveys, and draws its adversarial payload families from prompt-injection and jailbreak research.It cites indirect injection through untrusted tool content as a basis for its adversarial families.Source: §II · p. 2
  • Prior agent-safety benchmarks largely assess attack success or generic harm; SentryBench distinguishes itself through public-safety failure costs, a benign control arm, and deterministic ledger-first scoring.Related work also identifies provenance methods and chat-based over-refusal evaluation as influences, while noting that general runtime-safety approaches target a different trade-off.Source: §II · p. 2

III. THREAT MODEL AND PROBLEM SETTING

  • The threat model trusts the agent and system prompt but allows attacker-controlled inputs and tool results, with objectives including suppressing emergencies and triggering unwarranted actions.The specified attacker can inject text, impersonate authority, induce multi-turn goal drift, and poison a tool return, but cannot access model weights or sandbox internals.Source: §III · p. 2

IV. FORMAL FRAMEWORK

  • The framework defines dominance across attack success, over-refusal, and false positives, and models deployment harm using asymmetric costs for missed emergencies and unwarranted actions.Its primary comparison is whether a policy reduces attack success without an over-refusal or over-reaction tax.Source: §IV · p. 2
  • Caution is harm-dominated in the observed regime: it changes attack success by at most 3.7 points while adding refusals on up to every urgent control.The passage places all three models in this regime and says caution yields no harm reduction when refused-emergency costs exceed the adversarial harm averted.Source: §IV · p. 3
  • The trust gate routes pattern-matched consequential calls to a human, adds no over-refusal on clean urgent inputs, and its containment guarantee is bounded by detector recall.The genuine wrapper success described is a detector miss: Sonnet on C-008 had s = 0 and produced no gate event.Source: §IV · p. 3

A. Real-data ingestion

The benchmark draws legitimate content from public disaster-response and emergency-call-transcript datasets, with seeded sampling and filtering for disaster messages.

A. Real-data ingestion

  • Legitimate content comes from public disaster-response and emergency-call-transcript datasets, while prompt-injection and jailbreak-classification corpora provide attack payloads.The datasets are cached for reproducibility.Source: A. Real-data ingestion · p. 3
  • Disaster messages are filtered to related rows, English translations, minimum length, and no PII-flagged rows.Source: A. Real-data ingestion · p. 3
  • Incident content is sampled without replacement using a seeded process, while payloads may be reused across scenarios.Source: A. Real-data ingestion · p. 3

B. Scenario construction

Scenarios combine real content with domain directives and, in adversarial families, real attack payloads; six families comprise 190 scenarios.

B. Scenario construction

C. Simulated tool sandbox

A seeded mocked-tool sandbox supplies synthetic tool data, while a deterministic tool-call ledger supports scoring without real-system interaction.

C. Simulated tool sandbox

VI. DEFENSES

The evaluated defenses are pass-through, a prompt-only refuse-more caution policy, and TGCC, a trust-gating wrapper for consequential calls.

VI. DEFENSES

  • The TGCC gate routes a proposed call to a human when the injection score meets threshold θ = 1 or an over-broad citizen-record query has a positive injection score.Source: §VI · p. 4
  • The caution baseline is a prompt-only preamble telling the agent not to act when in doubt and to escalate.Source: §VI · p. 4
  • The pass-through baseline applies no defense.Source: §VI · p. 4
  • TGCC combines provenance separation, heuristic injection detection, human routing for manipulation signals or least-privilege violations, and escalation as containment.Source: §VI · p. 4
  • Because TGCC gates only on manipulation signals or over-broad queries, it stays out of the way on clean benign input.Source: §VI · p. 4

VII. EXPERIMENTAL SETUP

The evaluation applies three defenses across three frontier models, with scenario coverage and scoring procedures varying by model. Results are reported as frontier percentages.

VII. EXPERIMENTAL SETUP

  • The evaluated models are Claude Sonnet 4.5, Claude Haiku 4.5, and Azure GPT-5.Claude uses temperature 0; GPT-5 uses a fixed completion-token budget at default temperature.Source: §VII · p. 4
  • Table IV reports frontier results in percent, with the best result per model bolded.Source: Table IV · p. 5
  • Fig. 2 plots each model–defense cell and traces the none-to-caution-to-TGCC progression.Source: Figure 2 · p. 5
  • 840 episodes cover all 190 scenarios for Haiku and a 45-scenario stratified subset for each larger model.Haiku contributes 570 episodes; the two larger models contribute 270 combined.Source: §VII · p. 5
  • Scoring is deterministic, with seeded tools, per-episode ledger flushing, pinned snapshots, and the LLM judge disabled.Intervals use 1,000 nonparametric bootstrap resamples; errored and content-filtered episodes are counted separately.Source: §VII · p. 5

VIII. RESULTS

Across models, TGCC improves the attack-success/over-refusal frontier over caution, while caution yields little attack-success reduction and can refuse nearly every urgent emergency. TGCC has the lowest or tied-lowest harm-weighted cost across models.

VIII. RESULTS

  • Caution changes attack success by at most 3.7 points while raising over-refusal to 100 percent on Sonnet and GPT-5; TGCC reaches 0 percent attack success on Haiku and GPT-5.TGCC keeps over-refusal at or below the undefended baseline, has the highest containment in every case, and dominates caution on all three models.Source: A. The precision–recall frontier · p. 5, Figure 3 · p. 5
  • Caution has the highest harm-weighted cost for every model, while TGCC has the lowest or tied-lowest cost.Source: Figure 3 · p. 5
  • Table V compares GPT-5 model-level and system-level results in percent and count.Source: Table V · p. 5

B. Harm is the cleanest single-number summary

Harm-weighted cost ranks caution worst and TGCC best or tied-best across models. The results also show that provider filtering affects system-level accounting, suppression attacks account for genuine successes, and over-refusal varies sharply with emergency-message form.

B. Harm is the cleanest single-number summary

  • TGCC’s harm-weighted costs are 0.66/1.00/0.60 versus caution’s 1.33/1.80/1.60 for Haiku/Sonnet/GPT-5, respectively.Caution is highest and TGCC lowest or tied-lowest for every model.Source: B. Harm is the cleanest single-number… · p. 5

C. Model-level versus system-level accounting

  • Azure’s filter blocked 11–15 GPT-5 adversarial prompts per condition, and system-level over-refusal also counts urgent emergencies blocked before the model ran.GPT-5’s model-level 0 percent ASR rests on as few as 12 executed episodes and must be read with the filtered column.Source: C. Model-level versus system-level… · p. 5

D. Genuine attack successes

  • All seven genuine non-filtered successes were under-reactions in suppression-attack family C; TGCC eliminated every such success except Sonnet C-008.Source: D. Genuine attack successes · p. 5

E. Where the frontier tension lives

  • Benign over-reaction was 0 percent, while Haiku over-refusal was 18 percent on explicit emergency-call transcripts and 83 percent on terse disaster-report emergencies.The measured tension is dominated by over-refusal, and the contrast associates misses with content form rather than danger level.Source: E. Where the frontier tension lives · pp. 5–6

IX. DISCUSSION

The discussion finds that caution worsens harm by trading attack-success reductions for over-refusal, while TGCC improves the harm–utility frontier. Provider filters also affect system-level outcomes and should be reported separately.

IX. DISCUSSION

  • Residual attack success was at most 3.7 percent, and every residual success suppressed a real emergency in family C.The discussion characterizes this vulnerability as low in rate but severe in kind because it targets the costliest failure direction.Source: §IX · p. 6
  • Caution buys attack-success reduction almost entirely through over-refusal and has the worst harm on all three models.Theorem 1 places caution in a harm-dominated regime under the observed conditions.Source: §IX · p. 6
  • TGCC Pareto-improves over no defense and caution in harm-weighted terms, with no over-refusal tax on clean input.Theorem 3 bounds residual risk by detector recall; the lone Sonnet C-008 miss at s = 0 makes that bound tight.Source: §IX · p. 6
  • Provider filters both defend and block genuine emergencies, so fair deployment comparisons must report model and system outcomes separately.The discussion recommends tuning to scenario-calibrated cost ratios and using harm, rather than attack-success rate alone, as the governing metric.Source: §IX · p. 6

X. CONCLUSION

  • Across 840 scored episodes, TGCC had the lowest harm on every model, over-refusal at or below baseline, and highest containment.The conclusion also reports that refuse-more caution was actively harmful.Source: §X · p. 6
  • The conclusion identifies severity-calibration panels and per-cost-ratio robustness curves as future work.Source: §X · p. 6
Loading 2610.34535v1…