PaperlayerView exact revision on arXiv

Source-linked AI summary

AGrail: A Lifelong Agent Guardrail with Effective and Adaptive Safety Detection

Weidi Luo, Shenghong Dai, Xiaogeng Liu, Suman Banerjee, Huan Sun, Muhao Chen, Chaowei Xiao

arXiv:2502.11448v2cs.AI

TL;DR

LLM agents pose task-specific and systemic risks, while existing defenses inadequately adapt to varied tasks and agent actions. AGrail uses collaborative LLMs to generate and optimize safety checks, with optional tool support. Experiments report strong risk detection across tasks and attacks, with generalizability across agents and tasks.

  • Problem

    Existing defenses inadequately adapt to task-specific and systemic risks involving agent objectives, interactions, and information confidentiality, integrity, or availability.

  • Method

    AGrail is a lifelong guardrail that generates adaptive safety checks, optimizes them through test-time adaptation, and selectively invokes auxiliary tools.

  • Results

    AGrail demonstrates strong performance across task-specific and systemic-risk evaluations, including 0% prompt-injection ASR and 96% preserved benign actions with Claude-3.5-Sonnet.

  • Takeaways & Limitations

    AGrail reduces attack success while maintaining agent effectiveness and generalizes across diverse agents and tasks.

  • Takeaways & Limitations

    The framework relies on off-the-shelf LLM components and reasoning-based defenses because advanced agent-security tools remain scarce.

Abstract

from arXiv · show

The rapid advancements in Large Language Models (LLMs) have enabled their deployment as autonomous agents for handling complex tasks in dynamic environments. These LLMs demonstrate strong problem-solving capabilities and adaptability to multifaceted scenarios. However, their use as agents also introduces significant risks, including task-specific risks, which are identified by the agent administrator based on the specific task requirements and constraints, and systemic risks, which stem from vulnerabilities in their design or interactions, potentially compromising confidentiality, integrity, or availability (CIA) of information and triggering security risks. Existing defense agencies fail to adaptively and effectively mitigate these risks. In this paper, we propose AGrail, a lifelong agent guardrail to enhance LLM agent safety, which features adaptive safety check generation, effective safety check optimization, and tool compatibility and flexibility. Extensive experiments demonstrate that AGrail not only achieves strong performance against task-specific and system risks but also exhibits transferability across different LLM agents' tasks.

1 Introduction

LLM agents face task-specific and systemic risks that existing defenses do not adequately detect adaptively. AGrail addresses these gaps with adaptive safety-check generation and optimization, flexible tool use, and strong results across agent tasks.

  • Motivation: LLM agents can fail to consider vulnerabilities that threaten confidentiality, integrity, or availability in real-world scenarios.Task-specific risks follow administrator-defined objectives and constraints, while systemic risks arise from vulnerabilities in agent interactions.
  • Challenges: Existing approaches struggle with adaptive risk detection for dynamic tasks and with identifying effective safety policies for individual agent actions.Manually specified trusted contexts can limit generalization, motivating a more adaptive framework.
  • AGrail: AGrail dynamically generates safety checks from universal criteria or task-specific trusted contexts, then iteratively optimizes them during test-time adaptation.Two cooperative LLMs refine the checks, while customized auxiliary tools can support verification.
  • Results: 96% of benign actions were preserved while prompt-injection ASR was 0% with Claude-3.5-Sonnet, alongside low ASR for other Safe-OS and web attacks.Reported ASR was 3.8% for environmental sabotage, 5% for system sabotage, 0% on AdvWeb, and 17% on EIA on average.

2 Related Work

Related work covers LLM-agent memory and guardrails, while highlighting that agent risks extend beyond language outputs to code, commands, and environment interactions. AGrail instead collaborates on safety-check refinement to balance robustness and utility.

  • LLM-based Agents: LLM agents use available tools to follow language instructions and perform complex tasks, with memory supporting later actions.Memory can store environmental information and inform future behavior.
  • AGrail: AGrail differs from GuardAgent by collaboratively optimizing stored safety checks through test-time adaptation rather than using memory only for knowledge-enabled reasoning.The framework revises, merges, adds, or removes checks as actions are processed.
  • Guardrails: Existing LLM guardrails often focus on harmful language outputs and overlook risks in modalities such as Python code and Linux commands.Agent-specific defenses assess whether behavior poses risks, but may not cover the broader interaction surface.

3 Safe-OS

Safe-OS is designed to evaluate online OS-agent robustness in realistic execution settings where risks depend on environment state and diverse system interactions. It includes attack scenarios spanning prompt injection, system sabotage, and environment attacks.

  • Motivation: Safe-OS addresses the need to evaluate security during online execution rather than relying primarily on LLM-generated test data.Real-world actions cannot always be predefined or fully simulated without the environment.
  • Overview: OS agents face broad attack surfaces across process management, permissions, and file-system access control.These interaction routes expose agents to risks that may emerge only from the surrounding system state.
  • Attack Scenarios: Safe-OS defines prompt-injection, system-sabotage, and environment attacks to test distinct systemic-risk behaviors.Examples include altered agent actions, corrupted files or halted processes, and apparently harmless actions causing environmental damage.

4 Methodology

AGrail formulates safety-check selection as an optimization problem over universal criteria and optional guard requests, using memory to approximate the best subset. Its Analyzer and Executor iteratively refine checks and selectively invoke tools before deciding whether actions may execute.

  • 4.1 Preliminary: The framework seeks an optimized subset of safety checks that aligns with safety goals while incorporating optional manually specified guard requests.The complete search space combines safety criteria and guard requests, while memory stores an approximate optimized subset.
  • 4.1 Preliminary: AGrail processes safety criteria, guard requests, agent specifications and actions, environment observations, user requests, and auxiliary tools.The objective is formulated as a goal-based optimization problem.
  • 4.2 Safety Criteria: Universal safety criteria organize checks around information integrity, confidentiality, and availability while allowing manually designed criteria for specific tasks.These criteria define the objective safety standards an agent must follow during user tasks.
  • Workflow: The Analyzer revises retrieved checks, and the Executor removes unsuitable checks, validates essential ones with reasoning or tools, and updates memory iteratively.The guardrail blocks an action when safety status is False and otherwise permits execution.

5 Experimental Setup

AGrail is evaluated across task-specific and systemic risks using real-world agent outputs, multiple datasets, defense-agency baselines, and predictive and agreement metrics.

  • Datasets: The evaluation covers task-specific risks in Mind2Web-SC and EICU-AC, and systemic risks in AdvWeb, EIA, and the Safe-OS benchmark.Safe-OS contains three attack scenarios designed to assess online OS-agent robustness.
  • Datasets: Mind2Web-SC tests web-agent enforcement of safety rules, while EICU-AC tests ICU-data access restrictions based on user roles.The benchmarks evaluate SeeAct and EHRAgent, respectively.
  • Datasets: AdvWeb uses imperceptible adversarial strings, whereas EIA injects hidden malicious HTML elements to mislead web agents.The evaluation uses 80 AdvWeb instances and 100 successfully attacked EIA instances, plus 30 clean benign EIA samples.
  • Baselines: Baselines include model-based defenses using advanced LLMs and guardrail-based defenses including LLaMA-Guard3, GuardAgent, ToolEmu, and AgentMonitor.ToolEmu is evaluated only on Safe-OS because it simulates agent environments and requires an online setting.
  • Evaluation Configuration: AGrail uses task-specific safety criteria for Mind2Web-SC and EICU-AC, and universal safety criteria for Safe-OS, EIA, and AdvWeb during test-time adaptation.The criteria align with dataset guard requests or universal guard requests, depending on the evaluation setting.
  • Metrics: Defense agencies are assessed with accuracy, precision, recall, F1-score with ASR, and task-specific agreement metrics against ground-truth risks.Predictive metrics evaluate trade-offs on normal and harmful actions, while agreement metrics measure alignment with actual dataset risks.

6 Result and Analysis

AGrail shows strong performance across task-specific and systemic risk detection, with low attack success rates, accurate risk identification, memory benefits, and transferability across tasks and models.

  • Main Results: AGrail consistently ranks second on task-specific risk detection across Mind2Web-SC and EICU-AC with GPT-4o and Claude-3.5-Sonnet.
  • Main Results: ASR remains 5% for environment attacks and 3.8% for system-sabotage attacks, while EIA produces 6% action-grounding ASR and 28% action-generation ASR with 86.7% normal-task accuracy.
  • OOD and ID Analysis: Training and freezing memory improves performance over no memory in both ID and OOD settings for Claude-3.5-Sonnet and GPT-4o-mini.
  • Sequence Analysis: Claude-3.5-Sonnet shows minimal accuracy variation across input sequences, whereas GPT-4o-mini exhibits wider metric fluctuations.
  • Domain Transferability: AGrail transfers across domains without performance degradation with Claude-3.5-Sonnet and with only a 5.6% drop using GPT-4o.
  • Error Analysis: 46.2% higher average agreement and 100% agreement on OS prompt injection show that AGrail identifies risks corresponding to ground truth more effectively than baselines.
  • Learning Analysis: Memory similarity toward ground truth stabilizes after the fourth iteration, with simple actions converging faster than complex actions.

7 Conclusion

AGrail introduces Safe-OS and a lifelong framework for adaptive risk detection and effective safety-policy identification, outperforming existing defense agencies while generalizing across agents and tasks.

  • AGrail introduces Safe-OS, a high-quality and comprehensive dataset for evaluating online OS-agent robustness.
  • AGrail detects risks adaptively and identifies effective safety policies for those risks.
  • AGrail reduces attack success rates while maintaining LLM-agent effectiveness and demonstrates strong generalizability across diverse agents and tasks.

Limitation

The framework has two stated limitations: it relies on off-the-shelf LLMs rather than a trained guardrail, and its tool support is constrained by the scarcity of agent-security tools.

  • The current framework explores guardrailing with existing LLMs, using off-the-shelf LLM components and memory for lifelong learning.The authors identify training the guardrail as future work.
  • The framework primarily relies on reasoning-based defenses because existing tools for LLM-agent security are scarce.External tools are invoked only when necessary to minimize unnecessary tool usage.
  • Future work should develop more advanced tools that can be directly integrated into the framework to strengthen LLM-agent security.

B Methodology

AGrail retrieves and adapts safety checklists for agent actions, processes them through cooperative validation, and supports customizable criteria and auxiliary detection tools across agent settings.

  • Retrieve: AGrail stores safety checks by agent action and uses step-back prompting to generalize actions beyond particular elements.Stored actions can be overwritten when the analyzer marks them as in-memory or their similarity exceeds 0.8.
  • Workflow: The workflow generates a checklist, augments or refines retrieved checks when needed, and passes processed checks to an Executor for validation.The Analyzer identifies potential risks, while the Executor removes unnecessary checks and records validation results.
  • Application Details: AGrail supports customizable guard requests, safety criteria, and tools, including permission, OS-environment, and web-HTML detection tools.Developers can design and integrate additional detection tools through the framework’s interface.
  • Guardrail Workflow: The guardrail workflow takes memory, usage principles, specifications, requests, actions, environment, safety criteria, and tools as inputs, producing updated memory and a Boolean safety status.
  • Application Details: The OS Environment Detection Tool is used for Safe-OS, while the Web HTML Detection Tool is used for EIA attack and normal scenarios.The Web HTML Detection Tool is not feasible for Mind2Web-SC because its outputs lack complete browser-operation content and user inputs.
  • Retrieve: For missing or mismatched retrieved checklists, AGrail generates new checks; for incomplete or redundant lists, it augments, merges, or refines them.

C.2.2 Agreement Metrics

The Agreement metric evaluates whether an agent-safety detector identifies the specific reasoning behind unsafe actions, complementing safe/unsafe classification metrics. Experiments report agreement alongside attack success rates across datasets.

  • Agreement addresses the limitation of accuracy, precision, recall, and F1-score, which evaluate safe-versus-unsafe classification without assessing underlying reasoning.
  • A prediction is consistent when its identified violation criteria align with the ground-truth violation information.
  • Agreement measures the proportion of predictions whose algorithm-identified reasoning matches the ground-truth reasoning.For prediction set P, r(x) is extracted reasoning and g(x) is ground-truth reasoning.
  • Agreement rates were computed with Claude-3.5-Sonnet for Safe-OS, AdvWeb, and EIA, and with GPT-4o-mini for EICU and MindWeb.
  • In AdvWeb, AGrail’s ASR was 8.8% versus the baseline’s 5.0%, but the higher agreement rate indicated more effective identification of danger types.

D.1 OOD and ID Analysis Details

AGrail’s test-time adaptation and memory mechanisms support performance across ID and OOD settings, with evidence of domain transfer and a trade-off between universal and task-specific safety criteria.

  • OOD and ID Analysis: TTA consistently achieved the best performance, while Freeze Memory outperformed No Memory during TTA across ID and OOD evaluations.The reported results associate memory integration with stronger OOD generalization.
  • Data Sequence Robustness: Stronger models were more robust to random data-sequence effects than weaker models across experiments.This pattern was observed in evaluations using Claude-3.5-Sonnet and GPT-4o-mini across three random seeds.
  • Domain Transferability: GPT-4o experienced a 5.7% decrease in average performance when adapted memory and inference transferred from Mind2Web-SC to EICU-AC, while Claude-3.5-Sonnet showed minimal impact.The transfer evaluation used universal safety criteria and the same LLM across domains.
  • Universal versus Task-Specific Criteria: Universal safety criteria caused only a 2.7% accuracy decrease on Mind2Web-SC compared with task-specific criteria.The paper identifies this result as evidence of a trade-off between generalizability and performance.

E.1 Error Analyze

The error and case-study analyses examine learning behavior, computational cost, observation-aware evaluation, memory convergence, and tool-assisted or reasoning-based defenses across agent tasks and attacks.

  • Error Analyze: After bypasses during the first 30 steps on AdvWeb, AGrail’s attack success rate dropped to 0%, indicating a learning phase affected overall ASR.The analysis calculated ASR every 10 steps for defense agencies using GPT-4o.
  • Computing Cost: AGrail’s token consumption fell between the two-shot GuardAgent and one-shot GPT-4o model-based baseline, representing a trade-off between efficiency and overall performance.The comparison was conducted on the ID test set of Mind2Web-SC.
  • Experiment with Observation: AGrail achieved 89% accuracy in normal scenarios, whereas the baseline blocked 54.2% of normal data while both defenses blocked prompt-injection attacks.This comparison evaluates defenses with environment observations included as agent output.
  • Learning Analysis: By the fifth step, memory trajectories from three seed initializations converged, with average cosine similarity exceeding 95%.The analysis uses memory similarity rather than a single safety check because EICU-AC ground truth is complex.
  • Safe-OS Case Studies: The framework defended against OS prompt-injection and system-sabotage attacks through logical reasoning without invoking external tools.For environment attacks, it invoked an OS environment detector instead.
  • Web and EHR Case Studies: AGrail detected anomalous AdvWeb options and mitigated EIA attacks using personal-data-protection checks and a custom web HTML detector.The framework can also reason about unsafe EICU-AC actions without tools and invoke a permission detector for verification.

H Contribution

The framework combines universal and task-specific guard requests, adaptive checking, memory-based workflow components, and optional auxiliary tools across web, EHR, and OS-agent examples.

  • Contribution: The Safe-OS benchmark includes system sabotage, prompt injection, normal data, and environment-attack cases for evaluating online OS-agent defenses.The listed contributions include manual construction of the Safe-OS dataset and experiments spanning these scenarios.
  • Contribution: AGrail uses universal safety criteria or task-specific agent usage principles to ground safety checks for web, EHR, and OS agents.Examples include access restrictions based on user identity, location, licenses, permissions, and attack type.
  • Contribution: The Web HTML Detector extracts HTML input choices and selects shorter, more secure options to protect user input data against EIA attacks.Its prompts separate next-action and HTML-choice extraction from secure-choice selection.
  • Contribution: The paper demonstrates AGrail on Mind2Web-SC, EICU-AC, Safe-OS, AdvWeb, and EIA scenarios through safe and unsafe agent-interaction examples.The examples cover web access control, EHR database permissions, OS attacks, anomalous web options, and action-targeting attacks.
Loading 2502.11448v2…