PaperlayerView exact revision on arXiv

Source-linked AI summary

ShieldAgent: Shielding Agents via Verifiable Safety Policy Reasoning

Zhaorun Chen, Mintong Kang, Bo Li

arXiv:2503.22738v2cs.LGcs.CR

TL;DR

Autonomous agents are vulnerable to malicious instructions and attacks, while conventional LLM guardrails do not adequately address their dynamic action trajectories or complex safety policies. SHIELDAGENT constructs policy-derived action-based probabilistic rule circuits and verifies relevant circuits during agent execution, with SHIELDAGENT-BENCH supporting evaluation. It achieves state-of-the-art results across the reported benchmarks while reducing API queries and inference time, although extracted rules may not fully capture original policy constraints.

  • Problem

    Existing guardrails do not adequately safeguard agents’ sequential action trajectories against malicious instructions and attacks under complex safety policies.

  • Method

    SHIELDAGENT extracts and refines verifiable policy rules into action-based probabilistic rule circuits, retrieves relevant circuits, and formally verifies agent actions.

  • Results

    SHIELDAGENT achieves state-of-the-art performance across SHIELDAGENT-BENCH and three existing benchmarks, with 11.3% improvement on SHIELDAGENT-BENCH and 90.1% violated-rule recall.

  • Takeaways & Limitations

    SHIELDAGENT improves guardrail accuracy while reducing API queries by 64.7% and inference time by 58.2%.

  • Takeaways & Limitations

    Structured LTL rules extracted from policy documents may not fully capture the original constraints or be sufficiently concrete for verification.

Abstract

from arXiv · show

Autonomous agents powered by foundation models have seen widespread adoption across various real-world applications. However, they remain highly vulnerable to malicious instructions and attacks, which can result in severe consequences such as privacy breaches and financial losses. More critically, existing guardrails for LLMs are not applicable due to the complex and dynamic nature of agents. To tackle these challenges, we propose ShieldAgent, the first guardrail agent designed to enforce explicit safety policy compliance for the action trajectory of other protected agents through logical reasoning. Specifically, ShieldAgent first constructs a safety policy model by extracting verifiable rules from policy documents and structuring them into a set of action-based probabilistic rule circuits. Given the action trajectory of the protected agent, ShieldAgent retrieves relevant rule circuits and generates a shielding plan, leveraging its comprehensive tool library and executable code for formal verification. In addition, given the lack of guardrail benchmarks for agents, we introduce ShieldAgent-Bench, a dataset with 3K safety-related pairs of agent instructions and action trajectories, collected via SOTA attacks across 6 web environments and 7 risk categories. Experiments show that ShieldAgent achieves SOTA on ShieldAgent-Bench and three existing benchmarks, outperforming prior methods by 11.3% on average with a high recall of 90.1%. Additionally, ShieldAgent reduces API queries by 64.7% and inference time by 58.2%, demonstrating its high precision and efficiency in safeguarding agents.

1. Introduction

SHIELDAGENT addresses the difficulty of safeguarding agents whose unsafe behavior unfolds across dynamic action trajectories and whose safety policies are complex. It combines policy-derived probabilistic rule circuits with targeted verification and introduces a broader benchmark for evaluating agent guardrails.

  • Guardrail approach: SHIELDAGENT shields autonomous agents’ action trajectories by enforcing explicit safety-policy compliance through probabilistic logic reasoning and verification.It is designed for agent actions rather than only text-based inputs and outputs.
  • Safety policy model: It constructs action-based probabilistic rule circuits by extracting, refining, and grouping verifiable rules from policy documents.During inference, it verifies only circuits relevant to the invoked action.
  • Benchmark: SHIELDAGENT-BENCH addresses limitations of existing guardrail benchmarks by covering 2K safety-related instruction–trajectory pairs across six web environments and seven risk categories.Its unsafe trajectories use agent-based and environment-based attacks.
  • Results: 11.3% higher performance than the previous best guardrail method on SHIELDAGENT-BENCH accompanied the reported 4.8% false positive rate and 90.1% violated-rule recall.The method also reduced closed-source API queries by 64.7% and inference time by 58.2%.

2. Related Works

The related work frames agent safety around attacks that manipulate either agent internals or external environments, while existing guardrails inadequately protect sequential decision-making. SHIELDAGENT responds by accounting for both attack types and constructing a benchmark from them.

  • Agent-based attacks: Agent-based attacks manipulate instructions, memory, knowledge bases, or tool libraries and can force arbitrary malicious requests, but typically require internal access.These attacks target components inside the agent system.
  • Environment-based attacks: Environment-based attacks manipulate agent behavior through vulnerabilities such as malicious HTML elements or deceptive web pop-ups.They are easier to execute in real-world settings but may have a lower success rate.
  • Safety consequences: Both attack types can produce life-threatening failures, privacy breaches, and financial losses.The paper therefore treats both categories as relevant safety risks.
  • Paper’s response: SHIELDAGENT accounts for agent-based and environment-based perturbations when designing its shield and benchmark.SHIELDAGENT-BENCH uses state-of-the-art attacks from both categories to generate risky web-agent trajectories.
  • Existing guardrails: Existing guardrails focus on models and content moderation, leaving sequential agent decision-making and externally defined policy compliance insufficiently addressed.GuardAgent explores agent guardrailing but remains confined to textual space and internal model knowledge.

3. SHIELDAGENT

SHIELDAGENT constructs an action-based safety policy model from policy documents and uses probabilistic logic reasoning to verify protected agents’ actions. It combines iterative rule optimization, relevant rule circuits, and executable formal verification to support efficient, explainable shielding.

  • Overview: SHIELDAGENT has two stages: constructing an action-based safety policy model and verifying protected agents’ actions against it.The model encodes constraints from government regulations or platform policies, while verification applies probabilistic safety-policy reasoning.
  • Inference: During inference, SHIELDAGENT retrieves action-relevant rule circuits, uses history and tools to generate a shielding plan, and executes formal verification.It then performs probabilistic inference to produce a binary safety label, identify violated rules, and generate an explanation.
  • ASPM Construction: The model represents safety constraints as LTL rules over state and action predicates, organized into action rules and physical rules.Action rules constrain target actions, while physical rules capture internal system-variable constraints that support logical reasoning.
  • ASPM Construction: SHIELDAGENT extracts policies into structured rules, translates them into LTL representations, and clusters optimized rules by agent actions.Policy extraction records definitions, scope, descriptions, and references so each policy can be interpreted independently and traced during shielding.
  • ASPM Structure Optimization: Iterative Verifiability Refinement and Redundancy Pruning improve rule accuracy, concreteness, verifiability, and verification efficiency.Refinement decomposes vague rules and makes predicates observable, while pruning merges semantically similar redundant or contradictory rules; optimization continues until convergence or a budget limit.
  • Inference: SHIELDAGENT permits an action when its safety likelihood increases or remains within a tolerance bounded by |ϵ|, with ϵ adjustable for different safety requirements.This relative condition avoids directly thresholding unstable absolute world probabilities and supports dynamic safety assessment.

4. SHIELDAGENT-BENCH Dataset

SHIELDAGENT-BENCH evaluates agent guardrails on safety-related instructions and trajectories across six web environments, using adversarial perturbations and seven risk categories. Human review supports the reliability of its labels and violated-policy annotations, while Table 2 defines the reported performance and cost measures.

  • Dataset construction: 3110 unsafe trajectories are collected from 960 safety-related web instructions across six web environments.The environments are Shopping, CMS, Reddit, GitLab, Maps, and SuiteCRM.
  • Trajectory representation: Each trajectory includes complete agent-environment interactions at every step, including conversations, visual screenshots, and HTML accessibility trees.
  • Dataset construction: Two practical attacks generate unsafe trajectories by perturbing the agent or its environment.The benchmark adapts AgentPoison for agent-based attacks and AdvWeb for environment-based attacks.
  • Risk coverage: Seven risk categories organize the unsafe trajectories, including access restriction, content restriction, hallucination, instruction adherence, operational restriction, typical error patterns, and long-term risks.
  • Evaluation protocol: Human annotators manually review each trajectory’s guardrail label and all violated policies.Table 2 reports accuracy by risk category and perturbation source, along with ACC@G, FPR@G, ARR@R, query count, and inference time.

5. Experiment

The experiments compare SHIELDAGENT with prompt-based, rule-traverse, and GuardAgent baselines across SHIELDAGENT-BENCH and three existing benchmarks. Results show strong accuracy, policy grounding, robustness across attack types and categories, and reduced inference cost.

  • Setup: The evaluation covers SHIELDAGENT-BENCH, ST-WebAgentBench, VWA-Adv, and AgentHarm using accuracy, false positive rate, rule recall, and inference cost.The benchmark setup includes agent instructions, trajectories, enforced policies, and ground-truth labels.
  • Robustness and Categories: SHIELDAGENT performs consistently across agent-based and environment-based attacks, while hallucination-related risks remain comparatively weaker.The paper links the latter category to external knowledge needs beyond the policy.
  • Existing Datasets: SHIELDAGENT outperforms baselines across the three existing benchmarks by an average of 7.4% in accuracy.It achieves the highest accuracy and lowest false positive rate on VWA-Adv and SOTA performance on AgentHarm.
  • Online Guardrail: Online evaluation integrates guardrails as post-verification modules that check actions step by step and provide interactive feedback to the AWM agent.The setting measures accuracy, policy grounding, explanation clarity, and policy compliance conditioned on task success across six web environments.

6. Conclusion

The paper presents SHIELDAGENT as a logical-reasoning guardrail for enforcing explicit safety policies on autonomous-agent actions, alongside a benchmark for evaluating agent guardrails. Empirical results report improved guardrail accuracy and lower resource overhead.

  • Conclusion: SHIELDAGENT explicitly enforces safety policy compliance for autonomous agents through logical reasoning.Its streamlined verification framework uses an action-based safety policy model.
  • Conclusion: SHIELDAGENT-BENCH evaluates agent guardrails across seven risk categories and diverse web environments.The benchmark addresses evaluation of safety-related agent behavior.
  • Conclusion: Empirical results show higher guardrail accuracy and significantly lower resource overhead for SHIELDAGENT than existing methods.The conclusion summarizes the reported performance and efficiency gains without specifying additional metrics.

A. Detailed Introduction to SHIELDAGENT

The notation models an agent as interacting progressively with an environment: observations are mapped to partial states, which determine sampled actions under the agent policy.

  • A.1. Notations: X denotes the environment and πagent denotes the action policy of the agent being shielded.At each step, the agent receives an observation oi and maps it to a partial state si via f.
  • A.1. Notations: For web agents, f extracts accessibility trees from webpage HTML and visual screenshots to condense lengthy observations.The resulting partial state supplies the input for action generation.
  • A.1. Notations: The agent samples action ai from πagent(si) and progressively interacts with environment X.This defines the sequential trajectory that the shielding system evaluates.

A.2. Solution Space

SHIELDAGENT represents agent behavior with action and physical rules. These rules connect executable actions and environmental constraints, enabling reasoning over whether actions satisfy safety conditions.

  • A.2. Solution Space: Predicates are categorized as action predicates describing actions and state predicates describing relevant environment conditions.Examples include deleting data and whether information is private.
  • A.2. Solution Space: Action rules specify whether an action should be executed under permissive or preventive conditions.For example, deletion is prohibited when user authorization is absent.
  • A.2. Solution Space: Physical rules specify natural system constraints whose conditions can logically depend on other conditions.An example classifies private information as red data under a platform policy.
  • A.2. Solution Space: Physical rules provide knowledge that can improve robustness when predicates are inaccurately assigned.Together, the rules support reasoning over the agent’s solution space for shielding actions.

B. Additional Results

The additional results define evaluation settings across ST-WebAgentBench, VWA-Adv, and AgentHarm, and formalize the LTL-based policy representation used for verification.

  • B.1. ST-WebAgentBench: ST-WebAgentBench reports averaged accuracy (ACC) and false positive rate (FPR) across three risk categories, including overall averages.
  • B.2. VWA-Adv: Classifieds models realistic marketplace tasks such as posting, searching, commenting, and reviewing across roughly 66K listings.
  • B.2. VWA-Adv: VWA-Adv evaluates guardrails across Classifieds, Reddit, and Shopping under text-based and image-based perturbations.
  • C.3. Linear Temporal Logic (LTL) Rules: LTLf represents temporal logic over finite traces using propositional variables, logical connectives, and temporal operators such as Always, Sometimes, Next, and Until.
  • C.3. Linear Temporal Logic (LTL) Rules: A trajectory is a finite sequence of truth assignments, and ζ |= Φ denotes that trajectory ζ satisfies the LTLf specifications Φ.

C.4. ASPM Structure Optimization

ASPM structure optimization refines vague rules and merges redundant predicates iteratively, with Table 8 comparing the resulting policy-model statistics across environments.

  • C.4. ASPM Structure Optimization: The optimization process includes verifiability refinement and redundancy merging, as detailed in the associated prompts and algorithm description.
  • C.4. ASPM Structure Optimization: Table 8 compares predicate counts, rule counts, and average rule vagueness before and after optimization across environments.The maximum number of iterations is set to 10 for all environments.

C.5. Training ASPM

The training section organizes safety policies around risk categories and uses an iterative algorithm to refine, cluster, and merge predicates before returning an optimized ASPM.

  • Risk Categories: Unsafe trajectories are categorized into seven risks, including access restriction, content restriction, hallucination, instruction adherence, operational restriction, typical error patterns, and long-term risks.
  • ASPM Structure Optimization: Algorithm 2 initializes predicate and rule sets, computes vagueness scores, and iteratively refines the most vague rules within a refinement budget.
  • ASPM Structure Optimization: The algorithm clusters predicates, merges similar predicates or rules when beneficial, updates affected structures, and stops when no changes occur.
  • ASPM Structure Optimization: The optimized ASPM is returned with an optimized structure and randomized weights.

F.1. Dataset Distribution

The dataset and case-study materials describe balanced safe and unsafe trajectories, policy-model construction, and examples of rule refinement, grounding, and redundancy pruning.

  • F.1. Dataset Distribution: SHIELDAGENT-BENCH reports sample distributions by environment, pairing each instruction with equal numbers of safe and unsafe trajectories.
  • F.1. Dataset Distribution: The optimization figures track rules, predicates, and GPT-4o-judged vagueness across refinement and pruning iterations.
  • ASPM Training: The training pipeline clusters predicates, associates action circuits with relevant rule clusters, performs probabilistic inference, and learns rule weights from safety labels.
  • Case Study: Policy examples cover GitLab access and content restrictions, with structured descriptions, scopes, definitions, and sources.
  • Case Study: A content restriction rule prohibits publishing, downloading, or creating content classified as illegal or harmful.
  • Case Study: The case studies refine compound or vague predicates into concrete atomic actions and merge redundant predicates with equivalent contextual implications.
Loading 2503.22738v2…