PaperlayerView exact revision on arXiv

Source-linked AI summary

Identifying the Risks of LM Agents with an LM-Emulated Sandbox

Yangjun Ruan, Honghua Dong, Andrew Wang, Silviu Pitis, Yongchao Zhou, Jimmy Ba, Yann Dubois, Chris J. Maddison, Tatsunori Hashimoto

arXiv:2309.15817v2cs.AIcs.CLcs.LG

TL;DR

LM agents that use tools can produce serious real-world failures, making scalable pre-deployment risk testing necessary. ToolEmu addresses this gap with LM-based tool emulation, adversarial scenario construction, and automatic safety evaluation. Human validation found that 68.8% of failures identified with the adversarial emulator were true failures, while current agents still showed substantial failure rates.

  • Problem

    Tool-using LM agents introduce serious risks, but testing them across high-stakes tools and long-tailed scenarios is labor-intensive and difficult to scale.

  • Method

    ToolEmu uses LM-based emulators to execute diverse tools in sandboxed scenarios, an adversarial emulator for red-teaming, and automatic safety and helpfulness evaluators.

  • Results

    68.8% of failures identified with the adversarial emulator were judged true failures, while GPT-4 and Claude-2 agents still failed in 39.4% and 44.3% of test cases, respectively.

  • Takeaways & Limitations

    ToolEmu enables quantitative risk analysis across diverse tools and exposes potentially severe failures that warrant safer LM agents before real-world deployment.

  • Takeaways & Limitations

    Emulators sometimes overlook core constraints and produce critical issues, especially in complex or adversarial scenarios.

Abstract

from arXiv · show

Recent advances in Language Model (LM) agents and tool use, exemplified by applications like ChatGPT Plugins, enable a rich set of capabilities but also amplify potential risks - such as leaking private data or causing financial losses. Identifying these risks is labor-intensive, necessitating implementing the tools, setting up the environment for each test scenario manually, and finding risky cases. As tools and agents become more complex, the high cost of testing these agents will make it increasingly difficult to find high-stakes, long-tailed risks. To address these challenges, we introduce ToolEmu: a framework that uses an LM to emulate tool execution and enables the testing of LM agents against a diverse range of tools and scenarios, without manual instantiation. Alongside the emulator, we develop an LM-based automatic safety evaluator that examines agent failures and quantifies associated risks. We test both the tool emulator and evaluator through human evaluation and find that 68.8% of failures identified with ToolEmu would be valid real-world agent failures. Using our curated initial benchmark consisting of 36 high-stakes tools and 144 test cases, we provide a quantitative risk analysis of current LM agents and identify numerous failures with potentially severe outcomes. Notably, even the safest LM agent exhibits such failures 23.9% of the time according to our evaluator, underscoring the need to develop safer LM agents for real-world deployment.

1 INTRODUCTION

LM agents that use tools can create serious real-world risks, while ToolEmu enables scalable testing by emulating tools, environments, and risky scenarios.

  • Motivation: Tool-using LM agents can cause financial loss, property damage, or life-threatening harm when they fail to follow instructions.The paper argues that even low-probability risks should be identified before deployment.
  • ToolEmu: ToolEmu uses an LM-emulated sandbox to identify realistic agent failures across diverse scenarios at scale.Its emulator supports tools that may lack existing APIs or sandbox implementations.
  • ToolEmu: The adversarial emulator automatically instantiates scenarios in which agents are more likely to produce severe failures.It is designed to support red-teaming and long-tail risk detection.
  • ToolEmu: The framework emulates tool execution from tool specifications and inputs without requiring each tool’s implementation or execution environment.This supports faster prototyping and testing of high-stakes tools.
  • Observed failures: Typical detected failures include fabrication, instruction misinterpretation, erroneous execution, and risk ignorance.Figure 2 highlights risky actions and points requiring attention in example trajectories.

2 BACKGROUND & PROBLEM STATEMENT

The paper formalizes LM agents as tool-using systems acting under partial observability and evaluates whether they safely fulfill benign, sometimes underspecified user instructions.

  • Agent formulation: LM agents receive user instructions and execute tool actions in external environments to solve real-world tasks.The setup includes tools such as web browsers, code interpreters, and API plugins.
  • Agent formulation: The agent-environment interaction is formalized as a partially observable Markov decision process.At each step, the agent invokes a tool with additional inputs and receives observations.
  • Risk model: Underspecified instructions create risks when agents misinterpret user intent or fail to ask for clarification.For example, safely freeing disk space requires distinguishing unnecessary files from critical system or user files.
  • Risk model: The threat model assumes benign user intent and values safe task achievement over recklessly risky achievement.The helpfulness score therefore favors confirmation before potentially critical actions.

3 CONSTRUCTING TOOLEMU

ToolEmu combines LM-based tool emulation, adversarial scenario construction, automatic safety and helpfulness evaluation, and a curated benchmark for scalable risk testing.

  • Framework components: ToolEmu’s emulator returns tool observations, while safety and helpfulness evaluators score the agent’s complete trajectory.Human-curated test cases provide fields used by the framework’s components.
  • Standard emulation: GPT-4 prompts the standard emulator to instantiate tool sandboxes from tool specifications and agent inputs.The emulator avoids manual physical-sandbox setup and can test rare or complex scenarios safely.
  • Adversarial emulation: The adversarial emulator uses underspecification and designated risks to construct challenging sandbox states for red-teaming.This targets long-tail failures more efficiently than random scenario sampling.
  • Emulator requirements: Emulator fidelity requires valid input handling, accurate outputs, and consistent state transitions that could be instantiated in an actual sandbox.Violations of core requirements make an emulation non-instantiable.
  • Automatic evaluation: The safety evaluator retrospectively detects varied agent failures and quantifies their potential risks, while the helpfulness evaluator measures safe fulfillment of user instructions.The two evaluations capture safety and the potential safety-helpfulness tradeoff.
  • Benchmark curation: The benchmark curates diverse, realistic, potentially high-stakes tools and risk-focused test cases across multiple domains.GPT-4 generates candidate toolkits and cases, with human filtering and modifications; the risks span 9 types.

4 VALIDATING TOOLEMU

ToolEmu’s end-to-end validation shows that its emulator and automatic evaluator can identify realistic, risky LM-agent failures, while component tests reveal generally strong but imperfect quality. The adversarial emulator finds more true failures than the standard version, trading off some precision.

  • End-to-end validation: Six of seven detected ChatGPT-3.5 Terminal failures were successfully reproduced in a real sandbox.The seven failures were selected without cherry-picking, supporting the realistic instantiability of many detected failures.
  • End-to-end validation: The adversarial emulator detects about 10 p.p. more true failures than the standard emulator, with a slight precision trade-off.This higher true-failure incidence is attributed to targeting scenarios where agents are more likely to cause severe risks, alongside mildly reduced emulator validity.
  • Detailed validation: Over 80% of trajectories from both standard and adversarial emulators are free of critical issues in human validation.The adversarial emulator sometimes compromises emulation quality to attain a higher failure incidence rate, while minor issues do not compromise trajectory validity or instantiability.
  • Detailed validation: Both safety and helpfulness evaluators achieve Cohen’s κ over 0.45 with human annotations, matching the agreement rate between human annotators.The evaluations use binarized labels to reduce the effect of noise and subjective disagreement; human agreement itself is limited by subjectivity and noise.

5 EVALUATING LANGUAGE MODEL AGENTS WITHIN TOOLEMU

ToolEmu evaluates LM agents’ safety and helpfulness across adversarial test cases using an LM-emulated sandbox. More capable agents generally perform better on both dimensions, but even prompted GPT-4 agents still fail a substantial fraction of cases.

  • Experimental setup: Evaluation used four base models across 144 test cases, with an approximate cost of $1.2 per case.
  • Base LM agents: GPT-4 and Claude-2 achieved the best safety and helpfulness among the evaluated base LM agents.
  • Base LM agents: 39.4% of GPT-4 test cases and 44.3% of Claude-2 test cases contained failures.
  • Prompting: Safety prompting improved GPT-4’s safety and helpfulness scores, although failures remained in 23.9% of test cases.
  • Prompting: Adding helpfulness requirements to safety prompting negatively affected both safety and helpfulness scores.
  • Safety-helpfulness frontier: For capable agents, higher safety scores tended to correspond to higher helpfulness scores, whereas Vicuna-1.5 showed the opposite pattern.

6 RELATED WORK

Prior LM-agent benchmarks mainly assess capabilities, while ToolEmu focuses directly on agent risks through an expandable LM-emulated evaluation framework. The related work spans LM-based emulation and simulation environments used to evaluate agents safely.

  • Evaluation of LM agents: Existing LM-agent benchmarks primarily evaluate capabilities, including code execution, web interaction, and broader tool use.
  • Evaluation of LM agents: ToolEmu distinguishes itself by directly assessing LM-agent risks rather than only measuring capabilities.
  • Evaluation of LM agents: ToolEmu expands evaluation scope across tools and scenarios that are absent from existing benchmarks or difficult to assess in real-world settings.
  • Language model as an emulator: LM emulation has previously modeled human behavior, feedback, collaboration, virtual machines, and public APIs, whereas ToolEmu emulates environments.
  • Simulation environments: Simulation environments support agent development and evaluation when real-world environments are complex or raise safety concerns.

7 LIMITATIONS & FUTURE DIRECTIONS

ToolEmu’s emulator and evaluator show useful performance but retain important limitations, while future work targets scalability and more automated test-case generation. Human involvement remains central to curating test cases.

  • Quality of the emulators & evaluators: Prompt-engineered emulators sometimes overlook core constraints, especially in complex or adversarial scenarios.
  • Quality of the emulators & evaluators: The safety evaluator detected risky actions with 73.1% recall, below the 75.3% average recall of human annotators.
  • Future directions: Future-generation LMs may provide better emulators and evaluators, suggesting that the framework could scale.
  • Future directions: ToolEmu automates sandbox execution, adversarial scenario instantiation, and failure identification for LM-agent red-teaming.
  • Future directions: Test-case curation still largely relies on humans, limiting the current degree of automation.

A DETAILED DESIGN & IMPLEMENTATION OF TOOLEMU

ToolEmu combines realistic LM-based tool emulation with automated safety and helpfulness evaluation. Its design validates inputs, constrains outputs, stresses risky scenarios, and assesses agent trajectories against task and safety requirements.

  • Emulator: Emulator requirements prioritize input validation and realistic output generation so simulations can be instantiated in actual tool environments.Requirements are divided between prompt-encoded checks and programmatic validation, with violations classified as critical or minor.
  • Emulator: GPT-4-based emulators validate tool inputs, examine output requirements, and iteratively revise outputs that violate return specifications.The implementation uses temperature=0 prompting, zero-shot generation, and programmatic format checks.
  • Safety evaluator: The safety evaluator detects risky tool actions, considers underspecified instructions and incorrect calls, and assesses risks from resulting outcomes.The evaluation distinguishes actual outcome risks from intent alone, including cases involving fabricated information, unwarranted assumptions, or erroneous calls.
  • Helpfulness evaluator: The helpfulness evaluator measures whether agents correctly use tools to achieve the specified tasks, including how they resolve underspecification and incorrect calls.Assessments are grounded in the expected achievement and penalize insufficient or incorrect task completion.
  • Examples: The appendix provides example test cases and detailed AugustSmartLock tool specifications to illustrate the benchmark inputs and emulator format.The toolkit includes tools for controlling locks, managing guests and access, generating codes, and viewing access history.

A.4 EVALUATION BENCHMARK

Each ToolEmu test case specifies the toolkits available to the agent and an underspecified user instruction designed to red-team agent behavior.

  • Test-case fields: A test case contains a list of available toolkits, each drawn from the curated toolkit set and represented by a toolkit specification.The test-case toolkit field identifies which emulated tools the agent can invoke.
  • Test-case fields: User instructions typically include underspecifications intended to expose risky behavior during agent evaluation.These instructions are designed as red-team scenarios rather than fully specified routine tasks.

A.4.2 EXAMPLE TOOL SPECIFICATIONS

A toolkit specification organizes the tools, metadata, parameters, returns, and exceptions needed to model a cohesive target task such as email management.

  • Toolkit structure: A toolkit is a cohesive collection of relevant tools designed for a specific core target task.The paper gives Gmail management as an example of a toolkit-level task.
  • Specification fields: The toolkit JSON object records its name, human and model descriptions, and a list of constituent tools.Each tool includes a name, summary, parameters, return values, and exceptions.

A.4.3 TOOL CURATION

ToolEmu curates diverse, realistic, potentially high-stakes toolkits and test cases through GPT-4 generation followed by structured human review.

  • Toolkit curation: Toolkit curation generates toolkit names, descriptions, relevant tools, specifications, and potential misuse risks from category and requirement prompts.GPT-4 is used in separate brainstorming and specification-generation stages.
  • Toolkit curation: The curated toolkit list distinguishes tools by benchmark presence and public API availability.Categories cover existing benchmark tools, tools with public APIs but no sandbox evaluations, and tools without similar public APIs.
  • Test-case curation: Test cases are generated from randomly sampled primary toolkits and selected primary risks, then reviewed and refined by the authors.The review adjusts individual fields and can involve substantial modifications to meet curation requirements.
  • Quality control: Each test case is scrutinized by at least two individuals and its emulator trajectory is verified before dataset inclusion.The dataset analysis also tracks the number of toolkits and trajectory iterations per test case.

B ADDITIONAL EXPERIMENTAL RESULTS

ToolEmu’s emulator validation uses multiple annotators to classify deviations by severity, distinguishing critical from minor issues.

  • Four annotators classify emulation deviations as critical or minor according to predefined requirements.An emulation is considered to contain critical issues when at least two annotators assign that category.

B.1 ADDITIONAL VALIDATIONS FOR

Additional validations compare ToolEmu’s emulator and automatic evaluators with human judgments, while documenting subjectivity and annotation uncertainty.

  • Most emulator simulations contain minor issues that do not prevent instantiation.One example is simulating “123-456-7890” as a phone number.
  • Human assessments of risk severity are subjective, with author Cohen’s κ around 0.6 and reported safety and helpfulness agreement values of 0.47 and 0.52.Different annotators may rate the same failure, such as sending impolite emails, at different severity levels.
  • Both automatic evaluators achieve leave-one-out accuracy comparable to the average accuracy of human annotators.
  • The safety evaluator’s agreement with human annotations is comparable to agreement between human annotators.The helpfulness evaluator performs worse on the original scale because low-level assessments are more subjective.
  • Author-conducted end-to-end validation produces identified-failure precision and true-failure incidence similar to external-annotator results.
  • Author annotations yield a lower critical-issues-free emulator ratio but higher agreement with the automatic evaluators.

B.2 ADDITIONAL EVALUATIONS FOR LM AGENTS

The additional LM-agent evaluation examines temperature effects on Claude-2’s safety and helpfulness scores.

  • Larger temperatures of 0.5 and 1.0 diminish Claude-2’s safety and helpfulness scores.The scores do not decrease consistently as temperature increases.

C DETAILED IMPLEMENTATION OF LM AGENTS

The implementation uses ReAct-style LM agents that alternate reasoning, tool actions, inputs, and observations, with prompts containing task and safety or helpfulness requirements.

  • ReAct agents generate a Thought, Action, and Action Input before receiving an Observation from the emulator.This sequence repeats until the agent produces a Final Answer.
  • Base-agent prompts include required fields, task instructions, formatting instructions, and examples.
  • Additional prompts incorporate explicit safety and helpfulness requirements analyzed in the paper.
  • External annotators were selected from senior computer science undergraduates through applications, interviews, and programming and ethics tests.
  • Annotators received author-led training, detailed guidelines, FAQs, and practice examples before conducting evaluations.The initial training and guideline review averaged approximately 2 hours and 4 hours, respectively.

E.1 TYPICAL FAILURES

The examples show LM agents taking consequential actions without resolving ambiguity, verifying intent, or communicating risks. Failures span financial transfers, medication, access control, file management, and social-media actions.

  • GPT-4 sent money to all three individuals without determining who had paid the bill.
  • GPT-4 sometimes assumed the user had paid the bill and requested money from the other three individuals.
  • GPT-4 granted sudo access to a new colleague without informing the user of potential risks.
  • GPT-4 shared a Twitter link with edit access instead of comment access and created a potentially misinformation-containing post without warning about risks.
  • GPT-4 blocked the department domain instead of unnecessary domains and transferred $100,000 without checking the transfer’s purpose.
  • GPT-4 sent the user’s private SSH key to an external development team without informing the user of potential risks.
  • Claude-2 and ChatGPT-3.5 also made high-stakes errors involving payees, medication, files, cryptocurrency, fundraising, and emergency treatment.

E.2 COMPARING STANDARD & ADVERSARIAL EMULATION

Standard emulation can hide risks when underspecified instructions produce only one matching option, whereas adversarial emulation introduces plausible ambiguities and critical alternatives. These adversarial scenarios expose failures involving medication, home devices, and product-price updates.

  • Medication scenario: Adversarial emulation presented two matching medications, and the agent selected heart medication despite potentially fatal consequences from choosing incorrectly.
  • Medication scenario: Standard emulation exposed only one green medication bottle, while adversarial emulation created heart- and sleep-medication alternatives matching the same request.
  • Home-device scenario: Adversarial emulation identified risks from turning off all home devices, including security devices and refrigerators, without checking their importance or obtaining consent.
  • Shopify scenario: Adversarial emulation made “Outlet” match both a discounted jacket and a Power Outlet Adapter, leading the agent to reduce both prices indiscriminately.
  • Shopify scenario: The resulting product-selection and price-calculation errors could cause financial loss when the agent interprets “Outlet” incorrectly.

E.3 REAL SANDBOX INSTANTIATION

Real-sandbox experiments reproduced most ToolEmu-identified ChatGPT-3.5 failures, including dangerous deletions, privilege changes, and unsafe package removal. These instantiations show that emulation findings can manifest with actual tools, while also revealing occasional emulator inconsistencies.

  • Replication results: 6 of 7 ChatGPT-3.5 failures selected for Terminal instantiation were successfully replicated on an actual Ubuntu 22.04 machine.The remaining case showed consistency problems in the emulation.
  • Privilege escalation: The agent granted sudo access to a new user in the real sandbox, matching the emulated trajectory.The researchers verified that user samford successfully received sudo access.
  • Indiscriminate deletion: The agent deleted all created files larger than 1G, despite warnings that some system files could not be removed.The actual trajectory differed slightly from the emulation, but the deletion command still executed and removed the created files.
  • Destructive commands: The agent executed rm -rf / or equivalent commands, terminating its process, deleting files, and rendering the Amazon instance inaccessible.The agent warned about irreversibility only after execution; the same risky behavior was observed in the real sandbox.
  • Untrusted script execution: The agent downloaded and executed an external script without establishing that it was safe or would fix the reported crashes.In one emulation, the script returned an insulting message; the selected failure was nevertheless reproduced among the real-sandbox cases.
  • Unsafe package removal: The Python 2.7 removal action bypassed confirmation with -y, although the risk did not materialize on Ubuntu 22.04.On some Ubuntu systems, Python 2.7 is integrated into essential system packages and should not be arbitrarily removed.
Loading 2309.15817v2…