PaperlayerView exact revision on arXiv

Source-linked AI summary

Don't Make Models Guess Security and Safety: Symbolic Guardrails for Domain-Specific AI Agents

Yining Hong, Yining She, Eunsuk Kang, Christopher S. Timperley, Christian Kästner

arXiv:2604.15579v2cs.SEcs.AIcs.CR

TL;DR

Existing agent benchmarks often leave security and safety requirements unstated or unverifiable, while commercial deployments need assurances against costly violations. This paper reviews benchmark requirements, maps them to symbolic enforcement mechanisms, and evaluates whether symbolic guardrails preserve utility; it finds them practical for many requirements without reducing agent utility.

  • Problem

    Most benchmarks do not state the security and safety requirements they test, making reliable guardrail implementation and behavior auditing difficult for commercial deployments.

  • Method

    The paper systematically reviews 80 benchmarks, classifies their policies by verifiability, maps requirements to symbolic mechanisms, and evaluates guardrails on three domain-specific agent benchmarks.

  • Results

    Symbolic guardrails enforce many analyzed security and safety requirements, usually through inexpensive API validation, schema enforcement, or hardcoded confirmation and response templates, without reducing agent utility.

  • Takeaways & Limitations

    Symbolic guardrails are an overlooked but practical path toward deploying domain-specific AI agents in risk-averse commercial settings, although they do not address every requirement.

  • Takeaways & Limitations

    The analysis covers few verifiable-rule benchmarks across only two domains, uses an LLM-generated MedAgentBench policy, and relies on automated hazard analysis without expert judgment.

Abstract

from arXiv · show

There is increasing interest in integrating AI agents that invoke tools into domain-specific commercial software, where unintended tool calls can cause serious security and safety incidents. This has drawn growing research attention, and many agent security and safety benchmarks have emerged. They implicitly shape how the community approaches security and safety. Yet existing work exhibits a blind spot: it emphasizes training-based methods and neural guardrails, which reduce the likelihood of insecure or unsafe actions but cannot guarantee their prevention. It generally overlooks opportunities for deductive, symbolic guardrails grounded in standard software engineering practices, which can provide guarantees for some security and safety requirements. Our study has three parts: (1) a systematic review of 80 agent security and safety benchmarks finding that that 85\% of benchmarks do not state verifiable requirements (61\% provide none, and 24\% give only high-level goals); (2) an applicability analysis of which security and safety requirements symbolic guardrails can and cannot enforce on $τ^2$-Bench, CAR-bench, and MedAgentBench, finding that 74\% of requirements are symbolically enforceable and 95\% of these need only simple, low-cost checks; and (3) an empirical evaluation of symbolic guardrails on the same three benchmarks, finding that symbolic guardrails improve security and safety without sacrificing utility, and often improve it. Our work draws attention to the potential for symbolic guardrails for AI agents, suggesting them as an overlooked but practical path toward deploying domain-specific AI agents in risk-averse commercial software. We release all codes and artifacts at https://github.com/hyn0027/agent-symbolic-guardrails.

I. INTRODUCTION

The paper argues that benchmarks and neural defenses overlook verifiable, symbolically enforceable requirements needed for high-assurance domain-specific agents. Its three-part study finds widespread underspecification, substantial symbolic enforceability, and improved security, safety, and utility.

  • The study combines a review of 80 benchmarks, an enforceability analysis across three benchmarks, and an empirical evaluation of security, safety, and utility.These components correspond to the study’s three-part design.
  • 85% of security and safety benchmarks do not state verifiable requirements, limiting their value for high-assurance settings.Among 80 benchmarks, 61% provide none and 24% provide only high-level goals with undefined operative terms.
  • 74% of analyzed requirements are symbolically enforceable, and 95% of those require only simple, low-cost checks.Examples include precondition checks, schema enforcement, access control, and input validation.
  • Symbolic guardrails eliminate many enforceable requirement violations while maintaining or improving task completion rates.Prompt-only requirements produce violations in 20% to 78% of tasks; symbolic blocking can provide feedback for safe retries.
  • The authors present symbolic guardrails as an overlooked but practical path for deploying domain-specific agents in risk-averse commercial applications.The contribution includes characterizing verifiable requirements, mapping them to symbolic mechanisms, and evaluating utility effects.

A. Security and Safety Requirements

The paper treats security and safety requirements as harm-preventing expectations for tool-using LLM agents, while distinguishing inductive neural defenses from deductive symbolic enforcement. It motivates symbolic guardrails because commercial deployments need guarantees and practical coverage of verifiable policies.

  • A. Security and Safety Requirements: Security addresses adversarial threats, whereas safety addresses harm under accidental or environmental faults.Security conventionally concerns confidentiality, integrity, and availability; safety keeps systems within intended operating envelopes and benign failure states.
  • A. Security and Safety Requirements: Commercial requirements can protect against both adversarial attacks and honest mistakes, making their security-versus-safety classification depend on the risk source.The paper uses approval requirements for refunds above $50 as an example.
  • A. Security and Safety Requirements: The study groups security and safety requirements as instructions intended to prevent specific undesirable outcomes, including data exfiltration, destructive actions, and policy violations.The authors deliberately avoid using strict security and safety definitions as a filter because benchmark expectations and risk sources often overlap.
  • A. Security and Safety Requirements: The paper focuses on LLM-based agents that iteratively choose tool calls or text responses, with tool outputs conditioning later actions.The agents may use APIs, command-line interfaces, or other agents and can be general-purpose or domain-specific.
  • C. AI Agent Guardrail: Neural versus Symbolic: Neural guardrails make probabilistic judgments around agent inputs, outputs, or actions, commonly using judge LLMs or machine-learning classifiers.They monitor content such as prompt injection and sensitive data, or assess the security and safety of agent actions.
  • C. AI Agent Guardrail: Neural versus Symbolic: Symbolic guardrails use deductive software-engineering mechanisms such as input sanitization, information-flow control, and access control to provide guarantees.The paper contrasts these mechanisms with inductive neural guardrails and examines whether they cover practical agent requirements without reducing utility.

III. STUDY 1: VERIFIABILITY OF SECURITY AND SAFETY REQUIREMENTS EVALUATED IN AGENT BENCHMARKS

The first study examines whether agent security and safety benchmarks specify requirements in verifiable form. Reviewing 80 benchmarks, it finds that most do not state the requirements they test.

  • III. STUDY 1: VERIFIABILITY OF SECURITY AND SAFETY REQUIREMENTS EVALUATED IN AGENT BENCHMARKS: Most benchmarks never state the security and safety requirements they test, limiting what benchmark results can establish for risk-averse deployments.Benchmarks shape research optimization, but high-assurance reasoning requires requirements to be specified in verifiable form.
  • III. STUDY 1: VERIFIABILITY OF SECURITY AND SAFETY REQUIREMENTS EVALUATED IN AGENT BENCHMARKS: The study reviews 80 agent security and safety benchmarks and examines the policies—the sets of requirements—they define.The review classifies policies along a spectrum from unstated requirements to task-specific rules.

1) Identifying Benchmarks:

The study systematically reviewed 80 AI agent security and safety benchmarks using explicit search, filtering, and policy-classification procedures. It found that benchmark policies often provide too little information for verification.

  • Search and filtering: The review targeted papers proposing benchmarks that evaluate tool-calling LLM agents on security or safety.The search covered arXiv papers from January 2022 through March 2026 and applied predefined inclusion and exclusion criteria.
  • Search and filtering: 553 candidate papers were reduced to 413 after excluding robotics and reinforcement-learning papers with different meanings of agent and policy.The exclusions removed papers cross-listed in cs.RO or containing specified robotics and reinforcement-learning terms.
  • Search and filtering: Human and GPT-5-nano annotations agreed strongly on the 100-paper sample, reaching Cohen’s kappa 0.88 with LLM precision 0.96 and recall 0.99.The LLM labels were then used to filter the larger candidate set.
  • Corpus and classification: The final corpus contained 80 benchmarks classified by agent type and policy specificity, with verifiability assessed by whether actions can be proven to satisfy stated policies.The analysis used forced-choice coding and treated omission, rather than merely ambiguity, as a central problem.
  • Corpus and classification: Most policies lack enough specification for verification, because they either leave operative terms unexplained or state no behavioral requirement.The taxonomy pragmatically distinguishes No Policy, Goal-Setting, Verifiable-Rule, and Task-Specific categories without imposing a precise verifiability threshold.

3) Threats to Validity:

The benchmark corpus provides a broad basis for analysis, but the study’s conclusions remain bounded by how benchmarks represent deployment requirements and by classification judgments. The authors also report checks intended to assess corpus completeness.

  • Threats to validity: Benchmark policies are used as proxies for the security and safety requirements that deployed agents should satisfy.The authors caution that benchmarks may reflect developer interests and researcher attention more than deployment concerns.
  • Threats to validity: The study may have missed benchmarks despite broad arXiv coverage and selection criteria.An informal snowballing search found no new Verifiable-Rule benchmarks among 61 candidates, suggesting this limitation did not alter the conclusions.
  • Threats to validity: Benchmark classifications are categorical even though domain and policy specificity lie on a spectrum and require judgment.The authors state that labels were applied consistently and uncertain cases were discussed.
  • Corpus scope: The corpus contains 80 benchmarks spanning general-purpose and domain-specific agents and differing policy specificity.The distribution is summarized in Table II.
  • Corpus scope: Only 9% of benchmarks specify plausibly verifiable requirements, while many others leave security and safety expectations implicit or context-dependent.Examples include unstated common-sense expectations and rules whose operative terms remain unclear.

C. Discussion

The discussion argues that verifiable policies are necessary for high-assurance commercial deployment and that symbolic guardrails are especially promising for domain-specific agents. The empirical program therefore focuses on benchmarks where enforceability can be meaningfully analyzed.

  • Discussion: Verifiable policies allow practitioners to implement reliable guardrails and audit agent behavior against an explicit standard.Implicit or unverifiable policies cannot provide the same assurance because common-sense expectations may be contested.
  • Discussion: Verifiable-Rule policies generalize across tasks, whereas Task-Specific policies require policies to be generated and updated for each use case.Delegating that policy generation to a model would reintroduce reliability concerns.
  • Discussion: All identified Verifiable-Rule benchmarks target domain-specific agents, although 20 of 25 domain-specific benchmarks still lack such policies.The authors characterize this as a missed opportunity for safer and more secure commercial deployment.
  • Study 2: Study 2 asks which verifiable benchmark requirements can be symbolically enforced and why others cannot.It is restricted to the seven benchmarks with verifiable requirements because those are the benchmarks capable of addressing symbolic enforceability.
  • Study 2: The analysis selects τ^2-Bench and CAR-bench, then adds MedAgentBench through a synthesized policy to broaden coverage to a high-stakes medical domain.The synthesized policy was generated from the EMR use case and tool schema, then expanded using STPA hazard analysis.
  • Study 2: The authors abandoned plans to synthesize policies for 20 additional benchmarks because conflicting task labels made consistent verifiable requirements difficult to recover.CRMArenaPro contains near-identical queries assigned conflicting privacy judgments.

2) Classifying Requirements Enforceability and Matching Symbolic Guardrails:

The enforceability analysis classifies requirements against a catalog of symbolic guardrails and validates borderline judgments through author consensus and implementation checks. It also distinguishes requirements outside the agent’s behavioral scope from those that are enforceable or likely not enforceable.

  • Classification procedure: Requirements are assessed against six symbolic guardrails: API validation, schema constraints, information flow, temporal logic, user confirmation, and response templates.The classification asks whether deductive reasoning in program code can plausibly enforce each requirement.
  • Classification procedure: Requirements are labeled out of scope when they provide information, impose system-level constraints, or are infeasible given available tools.The third category applies only to hallucinated requirements in MedAgentBench.
  • Classification procedure: Information-flow enforcement has recognized challenges involving implicit flows and environment interactions.The study acknowledges these nuances but places them beyond its scope.
  • Results: Figure 4 summarizes the distribution of enforceable, likely non-enforceable, and out-of-scope requirements across the three benchmarks.The caption presents the figure as a cross-benchmark enforceability distribution.
  • Classification procedure: Enforceable symbolically requirements are those covered by one or more considered symbolic guardrails, while likely non-enforceable requirements are covered by none.These labels are assigned after excluding requirements classified as out of scope.
  • Validation: Borderline classifications were discussed by all authors until consensus, and implemented guardrails were verified against benchmark data.The implementations were reused in the empirical evaluation.

3) Threats to Validity:

Symbolic guardrails are broadly applicable and usually inexpensive, but the evidence is limited by benchmark coverage, policy construction, and assessment choices. The enforceable-rate estimate should therefore be interpreted as a conservative view of plausible domain-specific policies rather than commercial practice as a whole.

  • The analysis covers few Verifiable-Rule benchmarks in only two domains, and MedAgentBench uses an LLM-generated rather than human-written policy.The automated hazard-analysis process also lacked expert judgment, and benchmark requirements may not match commercial concerns.
  • The enforceable-rate estimate is a conservative lower bound because the guardrail catalog and enforceability judgments may be incomplete.The judgment depends on the authors’ understanding of symbolic guardrails and ability to implement them.
  • 74% of requirements across the three benchmarks are enforceable by symbolic guardrails.API validation, schema constraints, user confirmation, and response templates cover 95% of enforceable requirements; five require costlier temporal logic or information-flow control.
  • Simple symbolic checks handle most enforceable requirements, while sophisticated mechanisms are rarely needed.Information-flow tracking and temporal logic are needed for only 5 of 93 enforceable requirements.
  • Symbolic enforcement can reduce prompt length and token costs, but deciding which requirements remain in the policy is an unresolved implementation trade-off.Keeping some requirements in prompts may help agents act correctly before receiving a guardrail error.

C. Which requirements are not enforceable with symbolic

Symbolic guardrails cannot enforce every requirement: interaction style, hallucination avoidance, procedure following, and common-sense interpretation remain difficult. Some can be reformulated into more precise or stronger rules, but that trades residual risk against implementation cost.

  • Non-enforceable requirement themes: Four recurring requirement types are not symbolically enforceable: persona and interaction style, hallucination avoidance, procedure following, and common-sense reasoning.These requirements leave substantial room for interpretation or depend on generating and evaluating content rather than checking tool-call conditions.
  • Non-enforceable requirement themes: Neural guardrails remain necessary for some requirements, such as using an LLM judge to detect hallucinations.Symbolic guardrails cannot detect unsupported or fabricated information in the same way.
  • Reformulation trade-offs: Procedure-following requirements can be enforced through stronger reformulations, such as splitting workflows into sequential sub-agent stages.This approach requires architectural change rather than only a lightweight check.
  • Reformulation trade-offs: Common-sense requirements can be weakened into precise rules, such as blocking compensation tools until the user’s response exactly matches “compensation.”The reformulation improves enforceability but may leave residual risk or alter the intended policy.
  • Reformulation trade-offs: Enforcing feasible requirements symbolically reduces attack surface and potential hazards, reserving more expensive neural guardrails for residual risks.This allocation follows from separating requirements that can be checked deductively from those that still require interpretation.

V. STUDY 3: BENCHMARKING SECURITY, SAFETY, AND UTILITY OF AGENTS WITH SYMBOLIC GUARDRAILS

Study 3 evaluates whether symbolic guardrails improve security and safety without reducing utility by comparing guarded and unguarded agents across three domain benchmarks. The evaluation reuses benchmark tools while adding enforcement mechanisms for applicable requirements.

  • Study design: The study compares agents with and without symbolic guardrails, measuring policy violations and utility across τ 2-Bench, CAR-bench, and MedAgentBench.The guardrails are the independent variable; policy violations and utility are the dependent variables.
  • Study design: User responses are simulated with an LLM because τ 2-Bench and CAR-bench require multi-turn interactions.The simulator receives relevant context but cannot access MCP tools.
  • Guardrail conditions: For τ 2-Bench and CAR-bench, released benchmark tools form the baseline, while the guardrail condition wraps them with checks for all enforceable requirements.The same benchmark tool implementations are used as the basis for both conditions.
  • Guardrail conditions: MedAgentBench uses raw, eight-tool baseline, and guarded conditions, with the guarded condition enforcing 23 requirements classified as enforceable.The raw condition exposes generic GET and POST tools; the baseline exposes one tool per endpoint.
  • Guardrail conditions: Guardrails add enforcement parameters to 6 of 16 τ 2-Bench tools and 6 of 8 MedAgentBench tools, while CAR-bench needs none.For example, cancel_ticket adds user_id to verify ownership in τ 2-Bench.

3) Datasets:

The evaluation uses domain-specific benchmark datasets spanning in-car assistance and medical-record tasks, supplemented by adversarial MedAgentBench scenarios. Utility is measured with each benchmark’s original success metric, while security and safety checks focus on symbolically enforceable requirements.

  • Datasets: CAR-bench contains 100 Base-category entries for an in-car voice assistant handling navigation and vehicle operations.Tasks include checking weather and adjusting fog lights.
  • Datasets: MedAgentBench contains 300 electronic medical-record tasks involving patient records and medication orders.The dataset is augmented with patient information so simulated users can satisfy the policy’s authorization requirement.
  • Datasets: An adversarial MedAgentBench dataset expands four task categories into 17 scenarios and samples 50 of 391 generated tasks.Each generated task seeks to violate a requirement within its scenario.
  • Evaluation measures: Utility uses first-try task success, measured as Pass^1 for τ 2-Bench and CAR-bench and Success Rate for MedAgentBench.Utility is not measured on the adversarial dataset because task completion is not expected.
  • Evaluation measures: Security and safety evaluation restricts analysis to symbolically enforceable requirements because these are the only requirements validated accurately and addressed in the study.CAR-bench additionally reports policy violations through rpolicy, which partly relies on a probabilistic LLM judge.
  • Evaluation measures: Raw and baseline conditions use guardrails only to record violation triggers, not to reject tool calls.The analysis counts tasks that trigger at least one violation in each benchmark.
  • Evaluation procedure: Replay-based evaluation handles guardrails that add tool parameters without changing the agent behavior induced by baseline schemas.The original baseline call is restored after replay-based policy assessment; inability to supply an added argument is classified as unsafe.
  • Evaluation procedure: Differences across experimental conditions are assessed with the paired McNemar test.

5) Threats to Validity:

The evaluation finds that symbolic guardrails prevent symbolically enforceable policy violations without sacrificing utility, but experimental scope and cost limit the strength of the conclusions.

  • Threats to validity: The evaluation covers only policy violations reliably detectable by symbolic guardrails and excludes requirements that are not.This bounds the empirical findings to the subset of properties amenable to symbolic detection.
  • Threats to validity: Benchmark runs cost approximately USD 80 each, restricting model choices and repetitions and limiting statistical tests to detecting large effects.The authors characterize the findings as general trends under these experimental constraints.
  • Empirical findings: 20% to 78% of task executions violated symbolically enforceable requirements without guardrails, whereas guardrails prevented such violations by construction.The difference was statistically significant with p < 0.01 in all cases.
  • Empirical findings: Utility increased under guardrails across all three datasets, although several improvements were not statistically significant.Reported significance varied by model and benchmark, including p < 0.01 on CAR-bench.
  • Scope boundary: Neural guardrails may reduce residual errors but add non-trivial runtime cost, while some requirements still depend on model-based judgment.The paper therefore presents symbolic guardrails as complementary rather than complete.
  • Interpretation: Symbolic guardrails can return explanatory error feedback that helps agents retry with safer alternatives and often complete tasks successfully.Trace inspection provides this as a possible explanation for utility improvements.
Loading 2604.15579v2…