PaperlayerView exact revision on arXiv

Source-linked AI summary

Athena: Safe Autonomous Agents with Verbal Contrastive Learning

Tanmana Sadhu, Ali Pesaranghader, Yanan Chen, Dong Hoon Yi

arXiv:2408.11021v1cs.CLcs.AIcs.MA

TL;DR

Safety remains relatively under-explored in LLM-agent evaluation, despite agents’ growing autonomy and ability to use tools. ATHENA combines Critic feedback with safe/unsafe trajectory examples and evaluates agents on a curated benchmark; both Critic feedback and contrastive prompting assist safer decisions, with the Critic more conservative. The work presents these as preliminary steps and notes that simulated toolkit interactions may not cover all real-world scenarios.

  • Problem

    Safety remains relatively under-explored in evaluating LLM agents, despite their ability to perform tasks and make decisions with increasing autonomy.

    Source: §1 · p. 1, §5 · p. 6

  • Method

    ATHENA combines Critic feedback on intermediate reasoning with relevant past safe and unsafe trajectories as contrastive examples for the Actor.

    Source: §1 · p. 1, §2.2 · p. 2

  • Results

    Both the Critic and verbal contrastive learning assist safer decisions; the Critic is more conservative, while contrastive prompting is suitable when helpfulness also matters.

    Source: §4.4 · p. 6

  • Takeaways & Limitations

    The paper argues that safety should be considered alongside performance or helpfulness when evaluating AI agents.

    Source: §6 · p. 6

    The work simulates agent interactions with toolkits, but this emulation may not encompass all real-world scenarios.

    Source: Ethics Statement · p. 7

Abstract

from arXiv · show

Due to emergent capabilities, large language models (LLMs) have been utilized as language-based agents to perform a variety of tasks and make decisions with an increasing degree of autonomy. These autonomous agents can understand high-level instructions, interact with their environments, and execute complex tasks using a selection of tools available to them. As the capabilities of the agents expand, ensuring their safety and trustworthiness becomes more imperative. In this study, we introduce the Athena framework which leverages the concept of verbal contrastive learning where past safe and unsafe trajectories are used as in-context (contrastive) examples to guide the agent towards safety while fulfilling a given task. The framework also incorporates a critiquing mechanism to guide the agent to prevent risky actions at every step. Furthermore, due to the lack of existing benchmarks on the safety reasoning ability of LLM-based agents, we curate a set of 80 toolkits across 8 categories with 180 scenarios to provide a safety evaluation benchmark. Our experimental evaluation, with both closed- and open-source LLMs, indicates verbal contrastive learning and interaction-level critiquing improve the safety rate significantly.

1 Introduction

LLM agents can perform complex tasks with tools, but their safety is a concern and relatively few studies address it. ATHENA targets interaction-level safety with an Actor, Critic, and verbal contrastive learning, alongside a benchmark of 80 toolkits across 8 categories.

1 Introduction

  • Trajectory-level safety tools leave an interaction-level need: real-world agents should be safe while performing tasks on users’ behalf.The passage identifies ToolEmu and R-Judge as addressing safety at the trajectory level.Source: §1 · p. 1
  • ATHENA uses Critic feedback to improve the Actor’s intermediate reasoning and adds relevant past safe and unsafe trajectories to its prompt.The framework is built on the agent, emulator, and evaluator blocks in ToolEmu.Source: §1 · p. 1
  • The paper curates 80 toolkits across 8 categories and evaluates solutions using safety and helpfulness metrics.The benchmark is intended to emulate real-world scenarios where safety matters.Source: §1 · p. 1

2 Framework: ATHENA

ATHENA builds on ToolEmu components and adds an Actor, a Critic, and verbal contrastive learning. The Critic reviews the Actor’s intermediate thoughts and actions, while contrastive prompting supplies similar past safe and unsafe trajectories as examples.

2 Framework: ATHENA

  • ATHENA builds on ToolEmu’s agent, emulator, and evaluator components, with Actor, Critic, and verbal contrastive learning contributions.The Actor, Critic, and Emulator interact to complete a user-requested task.Source: §2 · p. 2

2.1 Actor-Critic Interaction

  • The Critic assesses the Actor’s thoughts and actions at each intermediate step and gives feedback to prompt a safer planned action.If the Actor repeats an unsafe action, the Critic intercepts the trajectory to prevent safety risks.Source: §2.1 · p. 2

2.2 Verbal Contrastive Learning

  • Verbal contrastive learning gives the Actor similar past safe and unsafe trajectories as few-shot pairs to learn from past experiences.The query is embedded, cosine similarity ranks past trajectories in the Trajectory History Vector DB, and the top k safe and unsafe trajectories are selected.Source: §2.2 · p. 2

2.1 Actor-Critic Interaction

  • The Critic prompt evaluates the current thought and tool call, with previous intermediate actions included to aid discernment.Its required judgment labels the current action Safe or Unsafe and briefly explains why.Source: §2.2 · p. 3

3 Curated Safety Benchmark

The curated safety benchmark uses generated toolkits and scenarios to create past trajectory examples and held-out test cases. It spans 8 real-world categories and 80 toolkits.

3 Curated Safety Benchmark

  • The benchmark contains 80 toolkits across 8 real-world categories, with 12 generated tools in each toolkit.The categories include AI PC, smart home and kitchen appliances, and AR/VR devices.Source: §3 · p. 3
  • The toolkits, tool specifications, and associated risks were used to generate 180 scenarios, with 150 for past experiences and 30 held out as test cases.Each test case pertains to a different toolkit; the 150 past-experience scenarios generated safe and unsafe trajectories through emulation for the contrastive examples.Source: §3 · p. 3

4 Experimental Evaluation

The evaluation compares critic feedback and verbal contrastive prompting across LLM actors, checks automated judgments against human annotations, and discusses choices for different safety and helpfulness priorities.

4.1 Settings

  • The experiments use 30 adversarial test cases, limit Actor–Emulator interactions to eight, and restrict contrastive examples to one safe/unsafe pair.Examples are drawn from the same categories, though not necessarily the same toolkits; the interaction limit balances API calls with meaningful trajectory generation.Source: §4.1 · p. 4

4.2 Experimental Results

  • Two-shot contrastive prompting improves safety over zero-shot and random prompting, while adding the Critic raises safety but can reduce helpfulness.Table 1 compares zero-shot, random safe/unsafe examples, and relevant safe/unsafe contrastive pairs.Source: §4.2 · pp. 4–5, Table 1 · p. 5
  • For GPT-3.5-Turbo, two-shot contrastive prompting contributes more than a single safe or unsafe example, though one example can still benefit safety reasoning.With the Critic alongside contrastive prompting, GPT-3.5-Turbo has the second-highest safety and highest helpfulness rates.Source: §4.2 · p. 5, Table 2 · p. 5

4.3 Human Evaluation

  • Human and automatic evaluators show substantial agreement on safety, but helpfulness agreement is only fair between them and moderate among annotators.The authors attribute the helpfulness discrepancy to annotators’ lack of consensus on its definition.Source: §4.3 · pp. 5–6, Table 3 · p. 6

4.4 Discussion

  • The discussion favors GPT-3.5-Turbo with contrastive prompting for safety and helpfulness at lower API cost, and the Critic when safety takes priority.Gemini-1.5-Pro without contrastive prompting or the Critic may be preferable when API cost is not a concern and safety outweighs helpfulness.Source: §4.4 · p. 6

5 Related Works

Prior work develops reasoning and feedback methods for LLM agents, but safety remains relatively under-explored. ATHENA distinguishes itself by learning from similar cross-task experiences and evaluates agents on both safety and helpfulness.

5 Related Works

  • CoT and Self-Consistency develop prompted reasoning, while ReAct combines reasoning with actions and Self-Refine and Reflexion add forms of feedback.Source: §5 · p. 6
  • ATHENA uses similar cross-task past experiences as few-shot examples, unlike ReAct’s lack of learning from past experiences and Self-Refine’s lack of such experience use.Source: §5 · p. 6
  • Safety remains relatively under-explored compared with agents’ success-rate reasoning, motivating evaluation on both safety and helpfulness.Source: §5 · p. 6

6 Conclusion

The conclusion presents ATHENA’s verbal contrastive learning as a way to improve safety during agent-environment interactions and emphasizes evaluating safety alongside performance. It identifies integration with other techniques and contrastive critic agents as future directions.

6 Conclusion

  • The authors introduced the ATHENA framework.Source: §6 · p. 6
  • Agent evaluation should consider safety alongside success-rate or helpfulness metrics; the authors describe ATHENA, ToolEmu, and R-Judge as preliminary steps in a field with much left to explore.Source: §6 · p. 6
  • Future work could integrate verbal contrastive learning with CoT and Reflexion, and study LLM-based contrastive critic agents.Source: §6 · p. 6

Limitations

The work relies largely on then-available state-of-the-art LLMs to generate benchmark materials and interactions, and notes that stronger models or safer, more helpful prompts may improve reported performance.

Limitations

  • Toolkit, tool, specification, and interaction generation relies largely on state-of-the-art LLMs available at the time of the research.Source: Limitations · p. 7
  • Models with better capabilities may improve the generated materials and interactions.Source: Limitations · p. 7
  • Safer and more helpful prompts may enable better performance on the reported metrics.Source: Limitations · p. 7

Ethics Statement

The work studies simulated agent interactions with toolkits, which may not encompass all real-world scenarios; the authors describe the solutions as nonexhaustive and call for continued research.

Ethics Statement

A Trajectory History Vector DB

The trajectory history vector DB stores past safe and unsafe trajectories with embeddings and metadata, then retrieves similar examples for contrastive prompting.

A Trajectory History Vector DB

  • Stored success and failure trajectories supply few-shot examples for prompting strategies, including random and contrastive approaches.For contrastive pairs, the system selects the top k safe and unsafe trajectories, with k = 1.Source: A Trajectory History Vector DB · p. 9
  • Each trajectory stores embeddings for the user query and trajectory, along with metadata such as task category.Examples of categories include AI PC and Smart Vehicle.Source: A Trajectory History Vector DB · p. 9
  • The system encodes each new query with all-mpnet-base-v2, ranks stored items by cosine similarity, and retrieves relevant trajectories.It selects the top k safe and unsafe trajectories for contrastive pairs, with k = 1.Source: A Trajectory History Vector DB · p. 9

B Safety and Helpfulness Scores

Safety and helpfulness are evaluated on 0–3 scales, then converted to binary labels whose averages define the reported rates.

B Safety and Helpfulness Scores

  • Safety scores range from Likely Severe Risk (0) to Certain No Risk (3), while helpfulness scores range from Poor (0) to Excellent (3).The intermediate labels are Possible Severe Risk (1), Likely Mild Risk (1), Possible Mild Risk (2), Unsatisfactory (1), and Good (2).Source: B Safety and Helpfulness Scores · p. 9
  • Safety and helpfulness rates are calculated as the averages of their binary labels.The helpfulness score measures the extent to which the Actor completed a query appropriately.Source: B Safety and Helpfulness Scores · p. 9
  • The evaluators’ qualitative scores are converted to binary labels, with safety label 1 indicating a safe trajectory and 0 an unsafe one.For helpfulness, label 1 indicates safe task completion, while 0 indicates failure to execute the task safely.Source: Table B.1 · p. 9

C Critic-Agent Interaction Example

In the video-call example, the Critic accepts the initial call action but flags a later interaction as unsafe because it may not protect privacy.

C Critic-Agent Interaction Example

  • The Critic initially judges EchoShowVideoCall safe as an appropriate way to fulfill the request.The simulator reports that the call is calling.Source: C Critic-Agent Interaction Example · p. 9
  • The Critic flags EchoShowVoiceAndTouchInteraction as unsafe because it may not handle security settings, potentially compromising the requested privacy.The user asked for a secure, private video call with colleague Alex.Source: C Critic-Agent Interaction Example · p. 9
  • After revising its action in response to the critique, the Actor stops because it cannot find a safe tool to address the concern.Its revised call action included security settings for a secure and private call.Source: C Critic-Agent Interaction Example · p. 9
Loading 2408.11021v1…