PaperlayerView exact revision on arXiv

Source-linked AI summary

ShieldAgent: Shielding Agents via Verifiable Safety Policy Reasoning

Zhaorun Chen, Mintong Kang, Bo Li

arXiv:2503.22738v2cs.LGcs.CR

TL;DR

Agent guardrails built for LLMs do not adequately address agents’ dynamic action sequences and complex safety policies. ShieldAgent extracts verifiable policies into action-based circuits and checks agent trajectories, achieving state-of-the-art results on ShieldAgent-Bench and three existing benchmarks.

  • Problem

    Existing LLM guardrails do not adequately safeguard agents whose dynamic action sequences and complex safety policies make unsafe behavior difficult to capture.

    Source: §1 · p. 1

  • Method

    ShieldAgent extracts and refines verifiable policy rules into action-based probabilistic circuits, then uses logical reasoning to verify protected agents’ action trajectories against relevant policies.

    Source: §1 · p. 1, §3.2.1 · p. 4

  • Results

    11.3% on ShieldAgent-Bench and 7.4% across existing benchmarks: ShieldAgent outperforms prior best methods on the former and baselines on the latter.

    Source: §1 · p. 2

  • Takeaways & Limitations

    ShieldAgent is presented as a step toward aligning agents’ behavior with explicit regulations and policies.

    Source: §6 · p. 8

    Extracted temporal-logic rules may not fully capture original constraints or be sufficiently concrete for verification.

    Source: §3.2.3 · p. 4

Abstract

from arXiv · show

Autonomous agents powered by foundation models have seen widespread adoption across various real-world applications. However, they remain highly vulnerable to malicious instructions and attacks, which can result in severe consequences such as privacy breaches and financial losses. More critically, existing guardrails for LLMs are not applicable due to the complex and dynamic nature of agents. To tackle these challenges, we propose ShieldAgent, the first guardrail agent designed to enforce explicit safety policy compliance for the action trajectory of other protected agents through logical reasoning. Specifically, ShieldAgent first constructs a safety policy model by extracting verifiable rules from policy documents and structuring them into a set of action-based probabilistic rule circuits. Given the action trajectory of the protected agent, ShieldAgent retrieves relevant rule circuits and generates a shielding plan, leveraging its comprehensive tool library and executable code for formal verification. In addition, given the lack of guardrail benchmarks for agents, we introduce ShieldAgent-Bench, a dataset with 3K safety-related pairs of agent instructions and action trajectories, collected via SOTA attacks across 6 web environments and 7 risk categories. Experiments show that ShieldAgent achieves SOTA on ShieldAgent-Bench and three existing benchmarks, outperforming prior methods by 11.3% on average with a high recall of 90.1%. Additionally, ShieldAgent reduces API queries by 64.7% and inference time by 58.2%, demonstrating its high precision and efficiency in safeguarding agents.

1. Introduction

SHIELDAGENT guards agent action trajectories by reasoning over verifiable safety policies, and SHIELDAGENT-BENCH evaluates guardrails across web environments and risk categories. The paper reports improved benchmark performance alongside reduced false positives, API queries, and inference time.

  • Malicious instructions and attacks expose agents to privacy breaches and financial losses, while model-focused guardrails struggle with unsafe behavior over dynamic interactions and complex policies across platforms.The cited attack and harm sources are Chen et al. (2024c), Wu et al. (2025), and Levy et al. (2024).Source: §1 · p. 1
  • SHIELDAGENT extracts and refines verifiable policy rules into action-based probabilistic rule circuits, then verifies relevant circuits using interaction history, shielding workflows, specialized tools, and formal code.It returns a binary safety label, identifies violated rules, and explains its decision.Source: §1 · pp. 1–2
  • 2K safety-related instruction–trajectory pairs span six web environments and seven risk categories, with unsafe trajectories generated through agent-based and environment-based perturbations.The attacks draw on Chen et al. (2024c) and Xu et al. (2024).Source: §1 · p. 2
  • 11.3% on SHIELDAGENT-BENCH and 7.4% on average across three existing benchmarks over the previous best guardrail, with 4.8% false positives, 90.1% violated-rule recall, 64.7% fewer API queries, and 58.2% lower inference time.The existing benchmarks are ST-WebAgentBench (Levy et al., 2024), VWA-Adv (Wu et al., 2025), and AgentHarm (Andriushchenko et al.).Source: §1 · p. 2

2. Related Works

Prior work describes both agent-based and environment-based attacks on LLM agents, while existing guardrails largely moderate content rather than enforce external safety policies over action sequences; ShieldAgent addresses this gap through probabilistic policy reasoning.

2.1. Safety of LLM Agents

2.2. LLM Guardrails

  • Model-centric guardrails filter content, while GuardAgent remains text-only and relies on internal knowledge; ShieldAgent is presented as the first multimodal agent enforcing external policy compliance over other agents’ action sequences (Andriushchenko et al.; Xiang et al., 2024; Zeng et al., 2024).Existing content moderation does not address vulnerabilities that emerge over action sequences.Source: §2.2 · p. 2

3. SHIELDAGENT

SHIELDAGENT builds an action-based safety policy model from regulations and platform policies, then uses it to reason over and enforce safety constraints on a protected agent’s actions.

3.1. Overview

  • The guardrail takes an agent’s history, current observation, and invoked action, then returns a safety flag, any violated-rule flags, and an explanation.Source: §3.1 · p. 3

3.2. Action-based Safety Policy Model

  • To make verification tractable, ASPM structurally encodes safety constraints in a logical knowledge graph.Source: §3.2 · p. 3

3.2.1. OVEWVIEW OF ASPM

  • ASPM divides predicates into state and action variables, and rules into action constraints and physical knowledge rules that support logical reasoning.Source: §3.2.1 · pp. 3–4

3.2.2. AUTOMATIC POLICY AND RULE EXTRACTION

  • Policy documents are converted into structured policies and LTL rules containing predicates, natural-language descriptions, formal representations, and rule types.Source: §3.2.2 · p. 4

3.2.3. ASPM STRUCTURE OPTIMIZATION

  • Alternating verifiability refinement and redundancy pruning makes rules more accurate, observable, atomic, and efficient, with optional expert review of the optimized rules.Source: §3.2.3 · p. 4

3.2.4. ASPM INFERENCE & TRAINING

  • Action-specific probabilistic rule circuits retain relevant dependencies, use learned soft rule weights, and permit actions when safety likelihood improves or stays within user-set tolerance ϵ.Source: §3.2.4 · p. 5

3.3. SHIELDAGENT Framework

  • The shielding pipeline retrieves relevant circuits and workflows, plans operations to assign predicate values, formally verifies rules, and produces a probabilistic decision with explanations and countermeasures.Its operations include Search, Binary-Check, Detect, and Formal Verify, supported by moderation and model-checking tools; hybrid memory caches interaction history and reuses successful workflows.Source: §3.3 · p. 6

4. SHIELDAGENT-BENCH Dataset

SHIELDAGENT-BENCH evaluates guardrails for autonomous agents on safe and risky trajectories across six web environments, addressing benchmarks that focus on generated content rather than agent actions.

Dataset overview

  • The benchmark comprises 960 safety-related instructions and 3,110 unsafe trajectories across six web environments, organized into seven common risk categories.The environments are Shopping, CMS, Reddit, GitLab, Maps, and SuiteCRM.Source: §4 · p. 6

Safety-related Instructions

  • Instructions adapt templates from WebArena (Zhou et al., 2023) and ST-WebAgentBench (Levy et al., 2024) with safety-critical information, such as API tokens.The resulting 960 high-quality instructions span Shopping, CMS, Reddit, GitLab, Maps, and SuiteCRM.Source: §4 · p. 6

Policy-Targeted Agent Attacks

  • The dataset induces policy violations using AgentPoison (Chen et al., 2024c) and AdvWeb (Xu et al., 2024), adapted to attack the AWM agent (Wang et al., 2024).AgentPoison injects adversarial demonstrations into memory or a knowledge base; AdvWeb stealthily manipulates environment elements. Both are adapted to violate at least one extracted safety policy per instruction.Source: §4 · p. 7

Comprehensive Risk Categories

  • Unsafe trajectories are categorized as access restriction, content restriction, hallucination, instruction adherence, operational restriction, typical error patterns, and long-term risks.The categories draw on extracted policies, attack-induced risky trajectories, and prior study of risky agent behaviors (Levy et al., 2024).Source: §4 · p. 7

Quality Control

  • Human annotators review each trajectory’s guardrail label and all violated policies to support a reliable testbed.Trajectories include the complete agent-environment interactions at each step: conversations, visual screenshots, and HTML accessibility trees.Source: §4 · p. 7

5. Experiment

Across ShieldAgent-Bench, three existing benchmarks, and an online web-agent setting, ShieldAgent outperforms baselines while combining policy grounding with efficient verification.

5.1. Setup

  • Three existing benchmarks were augmented with policies and safe and unsafe trajectories because they originally supplied only task instructions.They are ST-WebAgentBench (Levy et al., 2024; 234 tasks), VWA-Adv (Wu et al., 2025; 200 tasks), and AgentHarm (Andriushchenko et al.; 110 tasks); ShieldAgent-Bench samples include instructions, trajectories, enforced policies, and ground-truth labels.Source: §5.1 · p. 7
  • Guardrails were compared under shared policy inputs using direct prompting, rule traversal, and GuardAgent, with evaluation based on label accuracy and FPR, rule recall, API-query count, and inference time.Methods produced a binary safety flag and a list of violated rules, if any.Source: §5.1 · p. 7

5.2. Results

  • 10.2% average accuracy gain over rule traverse accompanies a 4.8% false positive rate, 90.1% rule recall, 64.7% fewer API queries, and 58.2% less inference time.The paper attributes the grounding and efficiency results to ASPM’s logical reasoning and the streamlined verification pipeline.Source: §5.2 · p. 8
  • ShieldAgent performs consistently across environment- and agent-based perturbations and leads most risk categories, especially access restriction and instruction adherence, but is slightly weaker on hallucination risks requiring external knowledge.The paper says environment-based perturbations are generally easier for guardrails to observe than agent-based ones, which depend on internal agent configurations.Source: §5.2 · p. 8
  • 7.4% average ACC gain over baselines spans all three existing benchmarks, with ShieldAgent leading on VWA-Adv and AgentHarm and gaining on ST-WebAgentBench risk categories.The reported ST-WebAgentBench gains include User Consent and Boundary and Scope Limitation; ShieldAgent has the highest ACC and lowest FPR on VWA-Adv and SOTA performance on AgentHarm.Source: §5.2 · p. 8, Table 3 · p. 8, Figure 3 · p. 8
  • In online web-agent evaluation, ShieldAgent achieves the highest policy compliance rate among the guardrails across six web environments.Each guardrail was integrated with the AWM task agent as a post-verification module that checked actions step-by-step and provided interactive feedback.Source: §5.2 · p. 8, Table 4 · p. 8

6. Conclusion · A. Detailed Introduction to SHIELDAGENT · A.1. Notations

SHIELDAGENT enforces autonomous agents’ safety-policy compliance through logical reasoning, and SHIELDAGENT-BENCH evaluates agent guardrails across seven risk categories in diverse web environments.

6. Conclusion

  • SHIELDAGENT is presented as the first LLM-based guardrail agent to explicitly enforce autonomous agents’ safety-policy compliance through logical reasoning.Source: §6 · p. 8
  • Its action-based safety policy model and streamlined verification framework are designed to provide rigorous and efficient guardrails.Source: §6 · p. 8
  • SHIELDAGENT-BENCH is introduced as the first benchmark for agent guardrails, covering seven risk categories across diverse web environments.Source: §6 · p. 8
  • Empirical results show that SHIELDAGENT outperforms existing methods in guardrail accuracy while significantly reducing resource overhead.Source: §6 · p. 8
  • The conclusion frames SHIELDAGENT as a step toward aligning increasingly deployed high-stakes agents with explicit regulations and policies.Source: §6 · p. 8

A.1. Notations

  • At each step, an agent maps an observation into a partial state, samples an action from its policy, and interacts progressively with the environment.For web agents, the mapping extracts accessibility trees from webpage HTML and visual screenshots to condense lengthy observations.Source: §A.1 · p. 11

A.2. Solution Space

ShieldAgent represents agent-trajectory constraints using action and state predicates, then reasons over action rules and physical rules to determine whether actions are permissible and consistent with system constraints.

A.2. Solution Space

  • Agent-trajectory predicates distinguish actions to execute from environment states that specify conditions for executing them.The paper denotes these as action predicates pa and state predicates ps, respectively.Source: §A.2 · p. 11
  • Action rules specify whether an action should occur under permissive or preventive conditions and must include at least one action predicate.For example, deleting data is prohibited without user consent.Source: §A.2 · p. 11
  • Physical rules encode natural system constraints, and can provide ASPM knowledge that improves shield robustness when predicates are assigned inaccurately (Kang & Li, 2024).For example, private information implies that a dataset should be classified as red data under GitLab’s policy.Source: §A.2 · p. 11

B. Additional Results … C.3. Linear Temporal Logic (LTL) Rules

The supplementary passages identify the scope of three guardrail evaluations and describe policy extraction and finite-trace temporal-logic rules. They specify what the reported tables compare, but provide no table cell values here.

B.1. ST-WebAgentBench

  • Table 5 reports averaged ACC and FPR for three ST-WebAgentBench risk categories and overall averages.The supplied passage does not include the table’s cell values.Source: Table 5 · p. 11

B.2. VWA-Adv

  • VWA-Adv constructs 200 risky instructions and perturbs text or images; Table 6 compares ACC and FPR across Classifieds, Reddit, and Shopping under both perturbation sources.Classifieds contains roughly 66K listings and supports tasks such as posting, searching, commenting, and reviewing.Source: §B.2 · p. 11, Table 6 · p. 12

B.3. AgentHarm

  • Table 7 compares guardrail performance across 11 AgentHarm harm categories.The supplied passage does not include the table’s cell values.Source: Table 7 · p. 12

C.1. Automated Policy Extraction

  • The automated policy-extraction and LTL-rule-extraction prompts are detailed in Appendix H.Source: §C.1 · p. 12

C.2.1. AUTOMATIC POLICY AND RULE EXTRACTION

  • Each structured policy contains term definitions, application scope, a precise policy description, and a source reference for traceability.Source: §C.2.1 · p. 12

C.3. Linear Temporal Logic (LTL) Rules

  • LTLf interprets formulas over finite traces: Always applies at every step, Sometimes at least once, Next at the next step, and Until requires φ1 to hold until φ2 occurs.Source: §C.3 · pp. 12–13
  • An LTLf formula combines propositional variables and logical operations; a trajectory is a finite sequence of truth assignments, and ζ |= Φ means it satisfies specifications Φ.Source: §C.3 · p. 13

C.4. ASPM Structure Optimization

ASPM structure optimization is iterative, with its procedure specified in Algorithm 2 and details on verifiability refinement and redundancy merging provided in Appendix H.

C.4. ASPM Structure Optimization

  • Appendix H details the prompt for ASPM verifiability refinement and the redundancy-merging process.Source: §C.4 · p. 13
  • The iterative ASPM structure optimization procedure is detailed in Algorithm 2.The passage does not describe the procedure’s individual steps.Source: §C.4 · p. 13
  • Table 8 reports predicate counts, rule counts, and average rule vagueness before and after optimization across environments, with a maximum of 10 iterations.The supplied caption gives no specific before-and-after values.Source: Table 8 · p. 13

C.5. Training ASPM … E.1. Risk Categories

The supplied passages define seven unsafe-trajectory risk categories and describe ASPM structure optimization through rule refinement and predicate clustering.

E.1. Risk Categories

  • ShieldAgent-Bench unsafe trajectories are organized into seven risk categories spanning access, content, hallucination, instruction, operational, error-pattern, and long-term risks.The categories include access and content restrictions, hallucination, instruction adherence, operational restrictions, typical error patterns, and delayed risks such as account lockout.Source: §E.1 · pp. 13–14
  • The seven categories cover access control, policy-compliant content handling, and factually incorrect or misleading outputs in information-seeking tasks.Examples include limiting interaction to authorized areas and preventing exposure of private or harmful data.Source: §E.1 · pp. 13–14
  • ASPM Structure Optimization takes predicate and rule sets, an embedding model, a clustering algorithm, refinement budget, iteration limit, surrogate LLM, and a predicate graph as inputs.The graph is initialized with edge weights, and predicates receive vagueness scores used to compute rule vagueness.Source: §E.1 · p. 14
  • The optimization loop refines the most vague rules that the LLM cannot verify, updates affected predicates and vagueness scores, and queues the revised rules.Refinement proceeds under a budget and iteration limit.Source: §E.1 · p. 14
  • The algorithm clusters predicates to prune redundancy, merges similar predicates when beneficial, updates rule references and vagueness scores, and stops when no changes occur.It returns ASPM with an optimized structure and randomized weights.Source: §E.1 · p. 14
  • The remaining categories address instruction adherence, permission before sensitive operations, common failure patterns, and delayed consequences such as account lockout.The listed typical error patterns include infinite loops and redundant executions; long-term risks include repeated failed logins.Source: §E.1 · p. 14

F. Detailed Experiment Results … H. Prompt Template

The block reports SHIELDAGENT-BENCH’s sample distribution, illustrates policy-model construction and end-to-end shielding, and provides prompt templates for extracting and refining policies and rules.

G. Case Study

  • Figures 4–6 track GitLab policy rule and predicate counts across refinement and pruning iterations and report rule-set vagueness scores, where lower scores indicate more concrete rules.Figures 11–13 illustrate compound-rule decomposition, predicate refinement, and merging redundant predicates.Source: Figure 4 · p. 15, Figure 5 · p. 15, Figure 6 · p. 15

F.1. Dataset Distribution

  • Table 9 reports safe and unsafe trajectory counts by environment, with equal quantities of each for paired instructions.The dataset distribution is also shown in Figure 9.Source: §F.1 · p. 14, Table 9 · p. 15

G. Case Study

  • Algorithm 3 clusters state predicates and trains rule weights using probabilistic inference and ground-truth labels to produce an action-based safety policy model.Clustering uses predicate co-occurrence or cosine similarity against a threshold, followed by spectral clustering.Source: G. Case Study · p. 16
  • The shielding example extracts action predicates, retrieves relevant rule circuits and workflows, assigns predicate values with tools, generates verification code, and reports a safety label with violations and remediation.Predicate assignment combines context inference, specialized tools, and generic query-based verification when specialized tools are unavailable.Source: Figure 14 · p. 23, Figure 15 · p. 24, Figure 16 · p. 25

H. Prompt Template

  • The policy-extraction prompt requests actionable policies with definitions, scope, descriptions, and source references, preserving exact wording and separating unrelated policies.Unclear definitions or scope are left as None.Source: H. Prompt Template · p. 29
  • The LTL-extraction prompt translates policy constraints into rules using verifiable, preferably positive and action-based atomic predicates, decomposing multi-predicate rules when possible.Predicates are grounded in agent observations and action history; unrelated rules should not be combined.Source: H. Prompt Template · p. 30
  • The verifiability-refinement prompt checks predicates for verifiability, concreteness, accuracy, atomicity, necessity, and unambiguity, then rewrites, splits, renames, or removes them as needed.Refinements include updated predicate definitions and keywords, associated rules, and context-specific definitions.Source: H. Prompt Template · p. 31
  • The redundancy-pruning prompt merges predicates or rules with equivalent meanings while requiring the updated representation to preserve consistency, completeness, and original intent.The prompt requests updated predicates and rules when merging is needed.Source: H. Prompt Template · p. 32

Related papers

Loading 2503.22738v2…