PaperlayerView exact revision on arXiv

Source-linked AI summary

GuardAgent: Safeguard LLM Agents by a Guard Agent via Knowledge-Enabled Reasoning

Zhen Xiang, Linzhi Zheng, Yanjie Li, Junyuan Hong, Qinbin Li, Han Xie, Jiawei Zhang, Zidi Xiong, Chulin Xie, Carl Yang, Dawn Song, Bo Li

arXiv:2406.09187v3cs.LG

TL;DR

LLM agents can be misused, while existing safeguards are either hardcoded for particular agents or focused on textual outputs. GuardAgent turns safety requests and target-agent logs into executable guardrail code using memory-retrieved demonstrations. It achieves over 98% and 83% guardrail accuracies on its healthcare and web-agent benchmarks, respectively, without affecting target-agent task performance.

  • Problem

    LLM-agent research often overlooks misuse, hardcoded safeguards do not generalize across agents, and text-focused guardrails cannot address agents’ diverse actions and specific safety requests.

    Source: §1 · pp. 1–2

  • Method

    GuardAgent uses an LLM and memory-retrieved demonstrations to turn safety requests and target-agent inputs, outputs, and logs into an action plan, then generates and executes guardrail code.

    Source: §1 · p. 2

  • Results

    Over 98% and 83% guardrail accuracies were achieved on EICU-AC and Mind2Web-SC, respectively, using four core LLMs without affecting target-agent task performance.

    Source: §1 · p. 3

  • Takeaways & Limitations

    GuardAgent offers flexibility for diverse guardrail requests, code-based guardrails, and low computational overhead.

    Source: §6 · p. 11

    More complicated guardrail requests, such as access rules involving multiple user attributes, are identified as a setting for future multi-agent designs.

    Source: More discussion on future research · p. 21

Abstract

from arXiv · show

The rapid advancement of large language model (LLM) agents has raised new concerns regarding their safety and security. In this paper, we propose GuardAgent, the first guardrail agent to protect target agents by dynamically checking whether their actions satisfy given safety guard requests. Specifically, GuardAgent first analyzes the safety guard requests to generate a task plan, and then maps this plan into guardrail code for execution. By performing the code execution, GuardAgent can deterministically follow the safety guard request and safeguard target agents. In both steps, an LLM is utilized as the reasoning component, supplemented by in-context demonstrations retrieved from a memory module storing experiences from previous tasks. In addition, we propose two novel benchmarks: EICU-AC benchmark to assess the access control for healthcare agents and Mind2Web-SC benchmark to evaluate the safety policies for web agents. We show that GuardAgent effectively moderates the violation actions for different types of agents on these two benchmarks with over 98% and 83% guardrail accuracies, respectively. Project page: https://guardagent.github.io/

1 Introduction

GuardAgent addresses the difficulty of applying diverse safety requests to LLM agents by turning those requests and target-agent logs into executable guardrails. The paper introduces healthcare and web-agent benchmarks and reports high guardrail accuracy without reducing target-agent task performance.

1 Introduction

  • Existing LLM-agent research largely overlooks misuse, while hardcoded safeguards do not generalize across agents and textual guardrails cannot handle agents’ varied actions and specific requests.Examples include privacy-related access restrictions and prohibiting certain users from purchasing specified items.Source: §1 · pp. 1–2
  • GuardAgent uses an LLM to turn safety requests and target-agent inputs, outputs, and logs into an action plan, then generates and executes guardrail code with memory-retrieved demonstrations.The code uses callable functions; the framework can adapt to new agents by adding relevant functions, uses in-context learning, and requires no additional training.Source: Figure 1 · p. 1, §1 · p. 2
  • The paper introduces EICU-AC for healthcare-agent access control and Mind2Web-SC for web-agent safety control, evaluating privacy and safety aspects of AI trustworthiness.The benchmarks are used to evaluate GuardAgent.Source: §1 · pp. 2–3
  • Over 98% and 83% guardrail accuracies were achieved on EICU-AC and Mind2Web-SC, respectively, using four core LLMs without affecting target-agent task performance.EICU-AC evaluates healthcare access control; Mind2Web-SC evaluates web-agent safety control.Source: §1 · pp. 2–3

2 Related Work

Prior guardrails focus on harmful textual outputs and do not address LLM agents’ diverse actions and safety requirements. GuardAgent is presented as an agent-guarding framework that surpasses the model-guarding approach in the authors’ experiments.

2 Related Work

  • LLM agents use LLMs for task understanding and planning, interact with environments, and commonly use retrievable memory for knowledge-enabled reasoning.GuardAgent is described as an LLM agent with the distinct objective of safeguarding other LLM agents.Source: §2 · p. 3
  • Existing LLM guardrails target harmful content in model outputs and cannot address LLM agents’ diverse output modalities and safety requirements.Earlier approaches include classifiers and “model guarding models” or “agent guarding models” approaches.Source: §2 · p. 3
  • GuardAgent, an “agent guarding agents” framework, surpasses the “model guarding agents” approach in the authors’ experiments.The paper considers both approaches as initial steps toward guardrails for LLM agents.Source: §2 · p. 3

3 Safety Requests for Diverse LLM Agents

The section introduces benchmarks for evaluating access control in healthcare agents and safety control in web agents. EICU-AC and Mind2Web-SC target gaps in agent-specific privacy and safety evaluation.

3 Safety Requests for Diverse LLM Agents

  • The benchmarks address the lack of existing privacy or safety evaluations for healthcare and web agents, using EICU and Mind2Web as their source datasets.EICU-AC assesses healthcare-agent access control, while Mind2Web-SC evaluates web-agent safety control.Source: §3 · p. 3

3.1 Access Control for Organizational Agents

  • EICU-AC evaluates access control for EHRAgent using physician, nursing, and general-administration roles, with labels indicating whether required databases and columns are accessible.Each example also identifies inaccessible databases and columns when access should be denied.Source: §3.1 · p. 4
  • EICU-AC questions are sampled or adapted from EICU and checked for correct answers with GPT-4 before label balancing focuses evaluation on access control.Examples are labeled for each of three roles, then adjusted because the initial labels were highly imbalanced.Source: §3.1 · p. 4

3.2 Safety Policies for Web Agents

  • Mind2Web-SC evaluates web-agent safety against six rules covering shopping membership, flights, car rentals, hotels, media searches, and job applications.Examples use SeeAct-predicted tasks and user profiles to label actions as permitted or denied.Source: §3.2 · pp. 4–5

4 GuardAgent Framework

GuardAgent checks target-agent actions against textual safety requests without being hardcoded into the target agent. It uses retrieved demonstrations to turn requests into plans and executable guardrail code.

4 GuardAgent Framework

  • GuardAgent is designed to address varied safety requests without interfering with target-agent decision-making, requires successful code execution for outputs, and needs no additional LLM training or fine-tuning.Its flexibility relies on non-invasive deployment, extendable memory, and a toolbox that can accommodate new agents and requests.Source: §4 · p. 5

4.1 Overview of GuardAgent

  • GuardAgent takes safety requests, a target-agent specification, user inputs, and target-agent logs, then labels actions as compliant or violating and can give reasons for violations.The label determines whether the target agent's proposed output or action is admitted or denied.Source: §4.1 · p. 5

4.2 Task Planning

  • For task planning, GuardAgent retrieves demonstrations by Levenshtein distance over target-agent inputs and outputs, then prompts an LLM to produce an action plan.Users can also inject demonstrations whose four steps summarize requests, filter relevant input and output information, and guide code generation.Source: §4.2 · p. 6

4.3 Guardrail Code Generation and Execution

  • For code generation, GuardAgent supplies callable-function specifications and retrieved examples, executes the generated code in the specified engine, and uses LLM debugging for execution errors.The LLM is instructed to use only the provided functions to reduce invented, nonexistent functions; debugging was seldom activated in the experiments.Source: §4.3 · p. 6

5 Experiments

The experiments evaluate GuardAgent on healthcare access control and web safety, comparing it with prompted, model-based, and hardcoded baselines. GuardAgent outperforms the prompted baseline on both benchmarks, while role, memory, and toolbox studies probe its breadth and dependencies.

5.1 Setup

  • The evaluation tests EHRAgent on EICU-AC and SeeAct on Mind2Web-SC, using inputs, output logs, and five metrics covering label prediction, explanations, and final responses.The main experiments use one demonstration for EICU-AC and three for Mind2Web-SC, and evaluate four core LLMs.Source: §5.1 · p. 7

5.2 Baselines

  • The baselines include a prompted Model-Guarding-Agent, LlamaGuard, and safety rules hardcoded into the target agents.The hardcoded rules are incorporated into EHRAgent and SeeAct system prompts.Source: §5.2 · pp. 7–8

5.3 Guardrail Performance

  • Over 98% LPA on EICU-AC and over 83% on Mind2Web-SC across core LLM choices; GuardAgent also has higher EAs and 100% FRAs across settings.The table compares GuardAgent with Model-Guarding-Agent, Hardcoded Safety Rules, and LlamaGuard.Source: Table 1 · p. 8, §5.3 · pp. 8–9
  • For a nurse lacking diagnosis-database access, the prompted baseline grants access when a query requires both diagnosis and medication data; GuardAgent avoids this mistake by executing guardrail code.The baseline also struggles to distinguish identically named columns across databases, whereas GuardAgent represents databases and columns in a dictionary.Source: §5.3 · pp. 8–9
  • Hardcoded Safety Rules degrade target-task performance, with EHRAgent’s FRA at 3.2%, while GuardAgent maintains 100% FRA across settings.The paper describes GuardAgent as non-invasive to target agents and the hardcoded rules as non-transferable to differently designed agents.Source: §5.3 · p. 9

5.4 Ablation Studies

  • GuardAgent performs well across three EICU-AC roles and six Mind2Web-SC rules, except for GPT-4 on rule 5, whose broad coverage and query details impede matching.The rule concerns movies, music, and videos.Source: §5.4 · p. 10, Figure 4 · p. 10
  • Varying the number of retrieved demonstrations shows that memory matters, while GuardAgent retains good guardrail performance with very few demonstrations.The reported comparison uses GPT-4 and tracks LPA and EA on both benchmarks.Source: §5.4 · p. 10, Figure 5 · p. 10
  • Removing required toolbox functions lowers EICU-AC performance to 90.8% LPA and 96.1% EA from 98.7% LPA and 97.5% EA, though GuardAgent can define functions or generate procedural code.Without the required functions, the paper also reports that 68.0% of examples prompted code-based guardrails, mostly inexecutable.Source: §5.4 · pp. 10–11

6 Conclusion and Future Research

GuardAgent is presented as a framework for safeguarding LLM agents against diverse safety and privacy requests. It uses knowledge-enabled reasoning to plan and generate guardrail code, with future work aimed at toolbox design, planning, and multi-agent frameworks.

6 Conclusion and Future Research

  • GuardAgent uses LLM reasoning to turn guard requests into a task plan and then into guardrail code for execution.The paper characterizes the framework as flexible for diverse requests, reliable through code-based guardrails, and low in computational overhead.Source: §6 · p. 11
  • Future research directions include automated toolbox design, stronger task-planning strategies, and multi-agent frameworks for safeguarding varied agents.Source: §6 · p. 11
  • The paper frames GuardAgent as its first framework for protecting other LLM agents against diverse user safety or privacy requests.Source: §6 · p. 11

Details About the EICU-AC Benchmark

The appendix details how EICU-AC and Mind2Web-SC examples, roles, and safety conditions are assembled and represented. It also documents the prompts, inputs, outputs, and callable functions used in GuardAgent experiments.

Role-Based Access Permission

  • EICU-AC uses physician, nursing, and general-administration roles, selected for their prevalence in ICU settings.Access permissions for databases and columns are based on ChatGPT recommendations verified by human experts, with rules for handling limited-access suggestions.Source: Role-Based Access Permission · p. 13

Sampling from EICU

  • EICU-AC examples are sampled from EICU questions, assigned roles and access labels, then filtered and supplemented to balance labels and reduce duplicate questions.The final benchmark contains 316 examples.Source: Sampling from EICU · p. 14, Table 3 · p. 15

Healthcare Questions Involved in EICU-AC

Details About the Mind2Web-SC Benchmark

  • Mind2Web-SC defines six rules over user-profile attributes, including membership, vaccination, driver’s-license status, age, and domestic status.The benchmark balances its two classes by removing some benign examples and adding examples with rule violations or non-violating profiles.Source: Details About the Mind2Web-SC Benchmark · p. 15, Table 4 · p. 15

Illustration of Examples in EICU-AC and Mind2Web-SC

Detailed System Prompts for Naive Access Control and Safety Control Based on Instructions

Complete Inputs and Outputs of GuardAgent

Callable Functions

  • The experiments use two callable functions, while users can extend GuardAgent’s toolbox by uploading additional functions.The paper notes that practical toolboxes may contain many more functions than the experimental one.Source: Callable Functions · p. 17

A Case Study on the Reliability of Code-Based Guardrails

A shared column name can have different access permissions depending on its database. GuardAgent’s code-based representation distinguishes these database-column pairs in a case where a model-based guardrail fails.

A Case Study on the Reliability of Code-Based Guardrails

Self-Defined Function by GuardAgent

When no toolbox or related functions are installed, GuardAgent defines necessary functions itself, including an access-control function for EICU-AC.

Prompts for Baselines

The baselines create guardrails by prompting an LLM with target-agent specifications, guard requests, inputs and output logs, and examples. Their demonstrations rely on textual reasoning rather than guardrail code generation.

  • Baseline prompts provide the LLM with the target-agent specification, guard requests, user inputs, associated output logs, and examples.Source: Prompts for Baselines · p. 17
  • The baseline system instruction adapts GuardAgent's task-planning instruction and adds directions for guardrail-output format.Source: Prompts for Baselines · p. 17
  • Unlike GuardAgent's code-generation approach, the baselines generate guardrails through reasoning over textual inputs.Source: Prompts for Baselines · p. 17

Manually Created Demonstrations

The authors manually created demonstrations for each benchmark, including examples for EHRAgent on EICU-AC and SeeAct on Mind2Web-SC.

Further Analysis of the Debugging Mechanism

With both toolbox and memory removed, 29/316 generated codes were initially not executable; debugging fixed 9/29, but none of the syntax errors.

Further Analysis of the “the trend of code-based guardrails”

Further analyses test GuardAgent’s risk-based code guardrails, memory retrieval, runtime, and model assumptions. They also examine how input formatting affects code generation and outline extensions for broader tool and request complexity.

Further Analysis of the “the trend of code-based guardrails”

  • Removing code-based-guardrail instructions still yields 68% code-based guardrails on EICU-AC, while natural-language formatting of access permissions reduces that share to 62%.The authors suggest structured access-permission formatting may facilitate code generation.Source: Further Analysis of the “the trend of… · pp. 17–18

More Details about Memory Usage of GuardAgent

Cost of GuardAgent

  • GuardAgent runs slower than the baseline but at a runtime comparable to its target agent; the authors expect human inspectors would take longer to moderate inputs and outputs.The paper also reports debugging iterations of 15.2s on EICU-AC and 17.8s on Mind2Web-SC, though most generated code runs without debugging.Source: Cost of GuardAgent · p. 18

Choice of the Core Model for GuardAgent

  • EHRAgent achieves 53.1% task accuracy on EICU and SeeAct achieves 40.8% on Mind2Web with GPT-4, leading the authors to expect GuardAgent to share their powerful core model.They do not consider GuardAgent with a weak core model an interesting case for these target agents.Source: Choice of the Core Model for GuardAgent · pp. 18–19

Investigating the Code Generation Design for GuardAgent

  • On CSQA, the authors test code-based guardrails on 80 questions using four pseudo-rules and five risk levels, although the Q&A task itself requires no code generation.The sample includes 39 questions without rule violations and 41 with at least one violation; GPT-4 answers all test questions correctly.Source: .16.1 Setup · pp. 19–20

Results and Case Studies

  • GuardAgent outperforms the GPT-4 model-guarding baseline on CSQA risk-level recall; the baseline’s recall is 10% lower, possibly because it entangles rule identification, validation, and risk estimation.GuardAgent identifies relevant rules, generates code to validate them, and counts violations to estimate risk.Source: Results and Case Studies · p. 20, Table 8 · p. 20

More discussion on future research

  • The authors identify manual toolbox specification and unvalidated chain-of-thought reasoning as improvement areas, and propose multi-agent designs for complex requests and more complex tools.They suggest assigning planning, code generation, and memory management to separate agents.Source: More discussion on future research · pp. 20–21
Loading 2406.09187v3…