PaperlayerView exact revision on arXiv

Source-linked AI summary

R-Judge: Benchmarking Safety Risk Awareness for LLM Agents

Tongxin Yuan, Zhiwei He, Lingzhong Dong, Yiming Wang, Ruijie Zhao, Tian Xia, Lizhen Xu, Binglin Zhou, Fangqi Li, Zhuosheng Zhang, Rui Wang, Gongshen Liu

arXiv:2401.10019v3cs.CLcs.AI

TL;DR

LLM agents can introduce safety risks in complex interactive environments, motivating evaluation beyond the harmlessness of generated content. The paper introduces R-Judge, a benchmark for judging and identifying risks in agent interaction records, and finds that current risk awareness is imperfect while fine-tuning improves performance and straightforward prompting fails.

  • Problem

    LLM agents can cause unexpected safety issues in complex environments, while existing safety evaluation largely focuses on generated content rather than agent interaction behavior.

  • Method

    R-Judge benchmarks LLM safety monitors by using annotated agent interaction records to evaluate risk identification and binary safety judgments.

  • Results

    Risk awareness is a challenging, multidimensional capability involving knowledge and reasoning; fine-tuning on safety judgment significantly improves performance, while straightforward prompting mechanisms fail.

  • Takeaways & Limitations

    Developing risk-aware LLM agents depends on strong foundation-model capabilities and high-quality, diverse safety-judgment fine-tuning data.

  • Takeaways & Limitations

    R-Judge is smaller than prior crawling-based safety benchmarks because complex interactions across distinct environments make human annotation costly.

Abstract

from arXiv · show

Large language models (LLMs) have exhibited great potential in autonomously completing tasks across real-world applications. Despite this, these LLM agents introduce unexpected safety risks when operating in interactive environments. Instead of centering on the harmlessness of LLM-generated content in most prior studies, this work addresses the imperative need for benchmarking the behavioral safety of LLM agents within diverse environments. We introduce R-Judge, a benchmark crafted to evaluate the proficiency of LLMs in judging and identifying safety risks given agent interaction records. R-Judge comprises 569 records of multi-turn agent interaction, encompassing 27 key risk scenarios among 5 application categories and 10 risk types. It is of high-quality curation with annotated safety labels and risk descriptions. Evaluation of 11 LLMs on R-Judge shows considerable room for enhancing the risk awareness of LLMs: The best-performing model, GPT-4o, achieves 74.42% while no other models significantly exceed the random. Moreover, we reveal that risk awareness in open agent scenarios is a multi-dimensional capability involving knowledge and reasoning, thus challenging for LLMs. With further experiments, we find that fine-tuning on safety judgment significantly improve model performance while straightforward prompting mechanisms fail. R-Judge is publicly available at https://github.com/Lordog/R-Judge.

1 Introduction

LLM agents can create unexpected safety issues in complex interactive environments, while existing safety evaluations largely target generated content rather than agent behavior. R-Judge formalizes agent risk awareness and benchmarks LLMs on judging and identifying risks in diverse interaction records.

  • Complex environments can make LLM agents cause unexpected safety issues, including privacy leakage and property loss.
  • Existing safety evaluations primarily address harmful, unfair, illegal, or unethical LLM-generated content rather than interactive agent behavior.
  • R-Judge contains 569 multi-turn records spanning 27 application scenarios across five categories, with binary safety labels and risk descriptions.
  • GPT-4o achieves an F1 score of 74.45%, while no other evaluated model evidently surpasses random performance.
  • R-Judge evaluates whether LLMs can judge and identify safety risks from agent interaction records.
  • The benchmark evaluates 11 LLMs and examines prompting, fine-tuning, and case studies to inform research on agent safety.

2 Problem Formalization

The paper models an agent’s multi-turn interaction as a record of user instructions, agent thoughts and actions, and environment feedback. It then frames safety monitoring as an LLM task that analyzes these records and assigns safe or unsafe judgments.

  • An interaction record begins with a user instruction and iterates through agent thought, action, and environment-feedback units.
  • The record is represented as R = (u, [(t, a, f)1, . . . , (t, a, f)n]), where n is the number of interaction turns.
  • At each turn, the agent takes the interaction record as input and outputs a thought and action for environmental interaction.
  • Safety monitoring matters because direct interaction with complex environments can produce severe issues such as privacy leakage and data loss.
  • The safety-monitor task asks an LLM to inspect agent interaction records, identify risks, and judge whether the actions are safe.

3 R-Judge Benchmark

R-Judge is a human-annotated benchmark of multi-turn interactions designed to test whether LLMs can identify agent safety risks and judge actions as safe or unsafe across diverse scenarios.

  • Case Format: R-Judge constructs complex multi-turn records capturing interactions among users, agents, and environments.Records use the ReAct framework, in which agents think, act, and receive environmental feedback.
  • Case Format: Each case includes a binary safety label and an annotator-written risk description, with labels carefully cross-checked for unambiguity.Risk descriptions document the elements needed for humans to understand the risks in each sample.
  • Curation Principles: The benchmark covers Intended Attacks and Unintended Risks across 5 application categories, 27 scenarios, and 10 risk types.Categories include program, IoT, software, web, and finance; risk types include privacy leakage, data loss, and financial loss.
  • Curation Principles: R-Judge balances open-agent safety cases by filtering and annotating existing trajectories, transforming unsafe examples, and constructing additional expert-designed cases.The dataset combines open-source transformations with manual construction and aims to balance safe and unsafe samples.
  • Dataset Statistics: R-Judge contains 569 cases averaging 2.6 interaction turns and 206 words, with 52.7% labeled unsafe.The dataset spans program, IoT, software, web, and finance and includes ten risk types.
  • Evaluation: Evaluation serially asks an LLM to analyze an interaction record and then produce a safe-or-unsafe label from the record and analysis.Risk identification is scored by an automatic evaluator against human risk descriptions, while safety judgment counts correct labels.

4 Experiments

Experiments evaluate 11 LLMs as agent safety monitors using safety-judgment and risk-identification tests, with metrics and random baselines validated before reporting benchmark results.

  • Setup: The study evaluates 11 API-based and open-source LLMs as agent safety monitors.The evaluated families include GPT, Llama, Mistral, and Vicuna models.
  • Setup: Zero-shot chain-of-thought prompting elicits reasoning steps before the final answer.The task instructions are designed to remain simple and general.
  • Metrics: F1 ranks safety judgment, while Recall and Specificity measure identification of unsafe and safe cases, respectively.Risk identification uses Effectiveness, normalized from 0 to 100.
  • Metrics: Random safety-judgment baselines are 50.00% for Recall and Specificity, while random risk-identification Effectiveness is 0.The corresponding random F1 values are 56.34%, 49.14%, and 51.32% for the two sets and full set.
  • Validation: Human agreement supports the GPT-4 automatic risk-identification scorer, with average Effectiveness PCC of 0.91 and GPT-4-answer PCC of 0.89.PCC measures linear correlation between human and automatic scores.
  • Main Results: GPT-4o ranks first and is the only model scoring above random in both evaluation sets, while most LLMs perform below random or ineffectively.The results identify substantial room to improve risk awareness in open-agent scenarios.

5 Analysis

The analysis examines prompting and fine-tuning for safety judgment, then identifies capability flaws underlying model failures. Straightforward prompting is insufficient, whereas fine-tuning and stronger general capabilities show promise.

  • Prompting Experiments: Straightforward prompting mechanisms do not consistently improve overall safety-judgment performance on complex agent-related tasks.Few-Shot-CoT does not consistently improve performance, and limited demonstration coverage may confuse models.
  • Fine-tuning Experiments: Fine-tuning on safety judgment improves behavioral-risk judgment, with Meta-Llama-Guard-2-8B surpassing GPT-4o under the same setting.The comparison shows lower Recall but higher Specificity for Meta-Llama-Guard-2-8B.
  • Fine-tuning Experiments: The analysis attributes fine-tuning gains partly to stronger foundation models and training data covering more risk types.The cited comparison contrasts Llama 3 with Llama 2 and 11 versus 6 risk types.
  • Case Study: GPT-4o failures reflect difficulty simulating scenarios, adapting risk judgments to conditions, and aligning safety judgments with human consensus.Examples include misjudging edit-access link sharing, false positives or negatives under different conditions, and privacy risks in an unverified Twitter post.
  • Analysis Summary: The authors conclude that risk-aware agents depend on both general model capability and fine-tuning with high-quality, diverse safety data.Risk awareness is described as requiring knowledge and reasoning abilities for safety judgment and risk identification.

6 Related Work

Related work has advanced capable LLM agents and evaluated harmful generated content, but agent safety remains an open challenge. Existing approaches include safety questions, red-teaming, content monitors, and agent-focused monitoring frameworks.

  • LLM Agents: LLM-agent research has expanded from framework prototypes and tool learning to increasingly complex tasks, while agent safety remains an open challenge.The cited work covers planning, reasoning, decision-making, tool learning, environment interaction, and multi-agent systems.
  • LLM Safety: Prior LLM-safety studies mainly evaluate or moderate harmful generated content using safety questions, red-teaming, and monitoring systems.The cited examples address toxicity, bias, immorality, offensiveness, unfairness, illegal activities, and ethics.

7 Conclusion

The paper presents R-Judge as a benchmark for evaluating LLM risk awareness in interactive agent environments. Experiments find imperfect risk awareness, knowledge-and-reasoning demands, gains from fine-tuning, and failure of straightforward prompting.

  • Conclusion: R-Judge evaluates whether LLMs can judge and identify safety risks from agent interaction records in interactive environments.The benchmark is described as general, realistic, and human-aligned.
  • Conclusion: Experiments across 11 LLMs show imperfect risk awareness requiring general knowledge and reasoning capabilities.The conclusion also reports that fine-tuning significantly improves performance, while straightforward prompting mechanisms fail.

Limitations

R-Judge trades annotation scale for annotation quality because complex interactions across distinct environments make the dataset smaller than crawling-based safety benchmarks. The dataset remains scalable as agent applications develop.

  • Limitations: The dataset is smaller than previous crawling-based LLM safety benchmarks because annotating complex agent interactions across distinct environments constrains scale.The authors frame this as a trade-off between instance scale and annotation quality.
  • Limitations: R-Judge can be expanded using agent execution records and minimal annotation when new safety risks are encountered.The authors state that they will continue expanding the dataset.

Impact Statement

R-Judge constructs a curated benchmark for agent safety risks across diverse scenarios, using annotated labels and structured risk descriptions. Its threat model attributes risks to user or environment motivations, agent triggers, and resulting outcomes.

  • Dataset scope: The benchmark focuses on personal LLM agents operating across program, IoT, software, web, and finance scenarios.The five categories contain corresponding scenarios, with overlapping tools assigned to the most relevant category.
  • Risk representation: Each unsafe case is described using Motivation, Trigger, and Outcome to connect risk origins, agent behavior, and consequences.Motivation comes from the user or environment, Trigger is the agent’s risky behavior, and Outcome is the resulting harm.
  • Curation: Annotators use the MTO schema and documented curation principles to produce high-quality risk descriptions and safety labels.The dataset includes examples, statistics, human involvement, and validation of the GPT-4 scorer.
  • Curation: Records are unsafe when agent outcomes match predefined risk types, while task failures without safety issues remain out of scope.Even user-specified risky actions are unsafe when the agent fails to request confirmation.

D.1 Metrics

The evaluation uses binary safety-judgment metrics and open-ended risk-identification scoring, with prompts designed to elicit analysis before final judgments. GPT-4 serves as an automatic scorer whose agreement with human evaluation is measured.

  • Metrics: Recall and Specificity measure unsafe-case identification and safe-case identification, while F1 ranks binary safety-judgment performance.The task predicts binary safe or unsafe labels from agent interaction records.
  • Metrics: Effectiveness evaluates whether generated analyses accurately identify and address risks in human-annotated descriptions.A GPT-4 scorer assesses relevance between model analyses and reference risk descriptions.
  • Prompts: Zero-Shot-CoT prompts ask models to analyze agent safety in plain text before printing exactly ‘safe’ or ‘unsafe’.Few-Shot-CoT adds two demonstrations, while other prompts provide risk-type hints or use Llama Guard categories.
  • Validation: Human agreement with the GPT-4 scorer is measured using the Pearson correlation coefficient across the risk-identification evaluation.The corresponding results are reported in Table 7, including averages across 11 models.

E.2 Extra Explanation on Analysis

Supplementary analysis examines prompting constraints, model capability flaws, and category-level performance. The authors emphasize that risk awareness depends on knowledge and reasoning, while noting limits to quantitative capability analysis.

  • Prompting: Few-Shot-CoT does not consistently improve overall performance under the two-demonstration setting.Two demonstrations are used because longer agent records and limited context lengths constrain larger few-shot configurations.
  • Analysis: Risk awareness is analyzed qualitatively because knowledge and reasoning capabilities remain underexplored and difficult to interpret quantitatively.The paper notes that knowledge and reasoning are related but not atomic capabilities.
  • Analysis: Evaluation of instruction-tuned models does not directly match knowledge-and-reasoning evaluation conducted primarily on pretrained models.Instruction-tuned models serve as backends for both agents and safety monitors, while providers use different tuning techniques.
  • Model failures: GPT-4o failures are categorized as scenario simulation, understanding adaptability, and safety alignment flaws.These three capability flaws are illustrated in Figures 11–13.
  • Model failures: The Llama-2 series shows instability in producing the required ‘safe’ or ‘unsafe’ outputs.A representative Llama-2-13b-chat-hf failure is shown in Figure 10.

F.2 Experiment Resources & Cost

The supplementary resource analysis reports experiment timing and infrastructure for API-based and open-source models. It also points to representative model-failure figures and the evaluated-model table.

  • Compute: API-based experiments traversing all 569 R-Judge samples generally take under 1 hour per experiment.Runtime may vary with network conditions.
  • Compute: Open-source model experiments use a single A100 GPU and take approximately 1.5 hours per model per experiment.The reported Zero-Shot-CoT and Few-Shot-CoT experiment times are similar because both require two inference rounds per sample.
  • Evaluated models: Table 9 lists the LLMs evaluated in the paper.The appendix identifies the concrete models used in the experiments.
  • Failure examples: Figures 8 and 9 provide examples of intended attacks and unintended risks in the R-Judge dataset.These examples correspond to the two risk-origin parts of the dataset.
  • Failure examples: Figures 10–13 illustrate representative failures involving output instability and GPT-4o capability flaws.The figures cover Llama-2 output failures and GPT-4 scenario simulation, adaptability, and safety-alignment flaws.
Loading 2401.10019v3…