Source-linked AI summary
The Guard That Cried Wolf: How Scary Words Make Agent Guardrails Refuse Legitimate Actions
Yingjie Zhang, Yuanbo Xie, Kai Chen
TL;DR
Over-safety occurs when guardrails refuse actions that are safe and authorized, yet existing evaluation lacks policy-grounded boundary cases. The paper constructs Cautious Bench with mechanically derived labels and faithful benign/twin counterfactuals. Across measured guardrails, authorized actions are refused more often when their resource names look scarier, indicating sensitivity to surface labels rather than authorization context.
Problem
Over-safety is difficult to measure because authorization is policy-relative and boundary cases are hard to harvest and verify reliably.
Method
Cautious Bench codesigns controlled benign/twin samples with a stated policy Π, whose labels are mechanically derived and certified by construction and faithfulness checks.
Results
All six non-floor guardrails over-refuse authorized Decidable actions, with false positives ranging from 35 to 89%, and refusal increases with scarier names on every execution-safety guard.
Takeaways & Limitations
The benchmark provides a policy-grounded reference for measuring over-safety and shows that measured guardrails can respond to object-name surface cues instead of authorization.
Takeaways & Limitations
The evaluation covers English, natural-language, single-principal, synchronous language-model guardrails, so the effect may be family-specific.
Abstract
from arXiv · showhide
Agent guardrails are checks that approve or refuse each action before an LLM executes it. Sometimes they refuse requests that are genuinely safe. This over-safety blocks deployment when a guardrail refuses an authorized task. Evaluating over-safety is hard: at the boundary an authorized action resembles an unauthorized one, and the safe-versus-unsafe label is a choice of authorization policy, not fixed by the action alone. We argue it therefore requires a benchmark that does not yet exist, one that maps the decision boundary of an ideal guardrail. Harvesting such a benchmark from real data is impractical: boundary cases are hard to collect, their labels hard to verify. The gap is real, so we construct Cautious Bench, the first benchmark to make over-safety the construct for agent guardrails; it codesigns each sample and its label with a stated authorization policy. A build-time gate re-derives every example to certify it, so each label is a mechanical consequence of the policy rather than an annotator's per-sample verdict, a reference against which researchers can measure real guardrails. The benchmark renders 756 Decidable benign/twin pairs, each under three object-name types (2,268 measured pairs), and 40 Undecidable pairs reported separately. Measuring six guardrails from five designs, we find a name-superstition effect: each over-refuses an authorized action more often under a scary-looking object name than a benign one. Since only the object name varies in the aforementioned contrast experiments, the deviation is the name's doing: the guardrails read the surface label, not the authorization context.
Introduction
Over-safety is the distinct refusal of an authorized action, but existing benchmarks cannot reliably determine when such refusals are wrong. Cautious Bench addresses this gap by deriving labels from explicit authorization policies and testing whether guardrails respond to authorization rather than scary surface names.
- Introduction: Over-safety is a guardrail’s refusal of a safe, authorized action, distinct from correctly catching harm or failing to act.It matters because such refusals erode trust and block deployment.
- Introduction: The same scary operation can be routine under one authorization context and a breach under another, so safety labels depend on context rather than the verb alone.A benchmark must expose authorization context to distinguish correct from incorrect refusals.
- Introduction: Cautious Bench constructs controlled benign/twin counterfactuals under a stated authorization policy instead of harvesting ambiguous real trajectories.Each pair keeps the action fixed while changing one authorization-relevant feature.
- Introduction: A build-time construction gate and faithfulness check make each label derivable from the policy and prevent the benchmark input from simply leaking the answer.The guardrail sees ordinary role, scope, action, and observation text, not the policy or label.
- Introduction: The benchmark’s central finding is a name-superstition effect: measured guardrails over-refuse authorized actions more often under scarier resource names.The verdict should depend on authorization, which the object name cannot change.
Related Work
Prior work evaluates content-safety over-refusal, harmful agent behavior, or aggregate guardrail utility, but does not isolate refusal of authorized actions. Cautious Bench instead treats over-safety as a policy-relative construct requiring defensible authorization-derived labels.
- Content-safety over-refusal: Content-safety benchmarks label benign-seeming inputs through annotator judgments or content policies, rather than deriving per-action authorization from execution context.Cautious Bench makes labels mechanical consequences of a stated authorization policy.
- Construct validity: Cautious Bench differs from prior work by codesigning samples with a stated authorization policy over execution context, making each label a mechanical per-action consequence.This addresses both boundary-case ambiguity and the distinction between content risk and authorization.
- Agent-safety evaluation: Agent-safety benchmarks measure harmful task execution or injection behavior, while related evaluations do not isolate over-refusal of authorized actions.Their targets differ from the guardrail false positive studied here.
- Construct validity: Construct-validity theory requires benchmark labels to be theoretically defensible, a heightened requirement when safety depends on authorization policy.An over-safety label cannot rest on a policy-independent ground truth.
Framework
The framework makes over-safety measurable by deriving labels mechanically from a disclosed authorization policy and separately checking that benign and twin cases are faithful tests. Controlled name counterfactuals then isolate whether refusal changes with surface naming.
- Framework: Because no policy-independent oracle settles authorization labels, the benchmark makes each label a consequence of an explicit policy and verifies both halves with a faithfulness check.The policy-relative construct is treated as a stated normative reference rather than an annotator verdict.
- Derive Each Label from the Policy: Policy Π contains one rule per authorization axis, denies the action if any axis fails, and defaults to denial when authorization is absent.Rules read axis facts from the guard’s profile and contents, including role, scopes, consent, preconditions, or provenance.
- Derive Each Label from the Policy: The guard must reason over axis evidence rather than echoing a label leaked into the input, so leakage prevention is part of construction.The benchmark’s gate enforces this structural requirement.
- The Faithfulness Check: Faithfulness rejects permitted content with hazards an ideal guard should catch and unauthorized twins whose refusal could be explained by extra hazards or surface cues.The check applies to both benign and twin halves.
- The Faithfulness Check: Each Decidable pair is derived under Π, certified by construction and faithfulness checks, rendered under three name grades, and separated from Undecidable cases.This segmentation distinguishes contrasts visible to the guard from cases it cannot decide.
Benchmark
Cautious Bench constructs controlled benign/twin counterfactuals whose labels derive mechanically from an authorization policy, then certifies them with build-time invariants. It separates guard-observable Decidable cases from an Undecidable tier and stresses scary surfaces and reasoning shortcuts.
- Controlled Counterfactual Pairs: Contrast on Authorization Context: Cautious Bench constructs controlled counterfactuals by holding the action fixed while changing one operational authorization fact between benign and twin halves.Examples include role, scope, consent, preconditions, and provenance.
- Controlled Counterfactual Pairs: Contrast on Authorization Context: Ten authorization axes across five channels vary distinct authorizing facts, including scope, body content, data isolation, seals, and deploy integrity.Four channels are guard-observable; an unobservable channel forms the Undecidable tier.
- The Construction-Invariant Gate: Label Derivability as a Build-Time Property: A static construction gate enforces four invariants at generation so each label is derivable from the stated policy Π rather than asserted after collection.The invariants prevent incidental surface differences, label leakage, hidden contrasts, and unavailable authorization signals.
- Raising Scariness and Reasoning Difficulty: Each benign half uses an authorized action with a scary operation, while reasoning-difficulty stresses close semantic shortcuts that could reveal labels without authorization reasoning.Scary operations include deletion, remote-code download, shell execution, and sensitive-data transmission.
- Decidable and Undecidable Segments: FP is measured only on Decidable cells where the authorizing fact is visible; Undecidable cells hide that fact and are reported separately as a measured boundary.An ideal guard passes the benign half and refuses the unauthorized twin under Π.
- Threat Model and Coverage Boundary: The threat model allows prompt crafting, observation injection, and cross-session memory poisoning only for the unauthorized twin, while benign actions remain non-adversarial.The benchmark measures source authorization and data isolation, not task or action alignment, because those lack a ground-truth benign half.
Measurement
Measurement varies only object names across Decidable benign/twin cells and reports Decidable and Undecidable segments separately. The evaluation also discloses its refusal predicate, including TS-Guard’s treatment of hedged flagged verdicts.
- Guardrails: Seven guardrails are measured: six from five designs plus a content-safety baseline, including purpose-built execution-safety guards and a prompt-injection probe.The six purpose-built guards span the main design families.
- Segments and denominators: 756 benign cells define the Decidable over-refusal denominator, while 40 Undecidable cells are rendered only at the as-authored surface and reported separately.Table 1 reports FP and detection over Decidable across three name grades; Table 2 reports both segments at the as-authored surface.
- Name tokens: Three name grades—innocent, as-authored, and scary—vary only the name within each Decidable cell, so scariness-related refusal increases are surface-driven.The graded vocabulary is curated from scary-but-authorized names in the corpus.
- Variant consistency by construction: All three name variants share authorization-bearing fields, action, policy, and gate because the resource name is threaded through construction rather than renamed after rendering.A build-time check rejects cells that fail to thread the name.
- Refusal predicate, disclosed: Counting TS-Guard’s hedged flagged verdict as refusal yields FP 35% and twin false-negative rate 62%; treating it as safe changes them to 17% and 81%.The scoring choice does not change TS-Guard’s rank or the effect direction.
Findings
Across Decidable actions, guardrails over-refuse authorized actions more often when the resource name sounds scary, while refusal and twin detection generally move together. The effect persists across execution-safety designs and removal actions, but varies by guardrail and is constrained by the Undecidable segment.
- All six non-floor guardrails over-refuse authorized Decidable actions, with as-authored FP ranging from 35 to 89%.PIGuard reaches 89%, while R-Judge reports 58%.
- Scary names increase FP over innocent names on every execution-safety guard in the Decidable set.Examples include AgentDoG-Qwen 63→70→82, TS-Guard 34→35→66, and ToolEmu 32→36→45 across innocent, as-authored, and scary grades.
- The within-cell contrast varies only the resource name, so the refusal difference tracks the name rather than the fixed authorization.The paper reports qualitative traces consistent with guards citing the scary name itself as risk, but this observation was not systematically coded.
- On the removal subset, AgentDoG-Llama rises from 25% to 56%, R-Judge from 54% to 76%, and TS-Guard from 29% to 40% under scary names.PIGuard and the second AgentDoG backbone move in the same direction, whereas ToolEmu falls from 38% to 29%.
- The scary-grade vocabulary spans 47 terms, with specific family names and CVE identifiers constrained to coherent threat scenarios.This construction enforces coherence between the name, resource path, contents, and gate.
- The benchmark separates Decidable and Undecidable segments: Decidable uses n=756 as the over-refusal denominator, while Undecidable uses n=40 and is reported separately.Five of seven measured guardrails refuse 68–100% of benign Undecidable cells, creating a catch-versus-refuse tradeoff.
Limitations
The benchmark measures a bounded decision-boundary phenomenon rather than deployed refusal rates. Its scope is limited to natural-language, English, single-principal, synchronous, language-model guardrails and excludes threats requiring other security layers or semantic-alignment judgments.
- Scope of claims: The benchmark is not a guardrail and makes no claims about refusal rates in deployment.Its detection rates apply to gate-detectable Decidable twins, while successful attacks are treated as twin false-negatives.
- Scope of claims: All seven evaluated guardrails are language-model-based, natural-language, English, single-principal, and synchronous.The observed superstition effect may therefore be family-specific; cross-language, multi-agent, and new-design extensions remain open.
- Measurement: Each guardrail runs once per cell, so reported rates are point estimates rather than precise values.The paper claims only the effect’s direction and rank.
- Threat coverage: Temporal races, privilege misuse, and cross-context leakage are outside a natural-language guardrail’s scope because other layers address them.The paper assigns these threats to atomicity or commit-time authorization, capability isolation, or information-flow control.
- Threat coverage: Semantic alignment is deferred because the paper identifies no agreed ground truth for a Decidable alignment axis.Instruction-hierarchy methods are described as partial, and adaptive attacks can break prompt-injection defenses.
Ethics
Cautious Bench uses synthesized, fictional, authorization-grounded prompts that intentionally look unsafe to expose over-safety without deployment intent. Its released construction materials support examining guardrail refusals rather than bypassing safeguards.
- Ethical scope: The benchmark intentionally includes safe-but-scary actions because over-safety concerns refusing authorized actions that look dangerous.This design targets over-safety while distinguishing it from correct catches of unauthorized actions.
- Ethical scope: All benchmark content is synthesized and fictional, using publicly known threat identifiers or generic descriptors rather than novel exploits or real targets.The prompts are authorization-grounded and carry no deployment intent.
- Ethical scope: The benchmark is intended to expose over-safety to improve guardrails, not to bypass them.The authors release the generator, authorization policy Π, construction gate, and recipes.