PaperlayerView exact revision on arXiv

Source-linked AI summary

R-Judge: Benchmarking Safety Risk Awareness for LLM Agents

Tongxin Yuan, Zhiwei He, Lingzhong Dong, Yiming Wang, Ruijie Zhao, Tian Xia, Lizhen Xu, Binglin Zhou, Fangqi Li, Zhuosheng Zhang, Rui Wang, Gongshen Liu

arXiv:2401.10019v3cs.CLcs.AI

TL;DR

Agent safety evaluation needs to assess whether LLMs recognize risks in interactive environments. R-Judge benchmarks this ability using annotated multi-turn agent interaction records. Across 11 LLMs, risk awareness remains limited: GPT-4o leads with a 74.45% F1 score, while no other model evidently surpasses random performance.

  • Problem

    Agent safety evaluation needs to assess whether LLMs can judge and identify risks arising in interactive environments.

    Source: §1 · p. 2, §7 · p. 9

  • Method

    R-Judge evaluates LLM safety judgments and risk identification using curated multi-turn agent interaction records annotated with safety labels and risk descriptions.

    Source: §1 · p. 2

  • Results

    Across 11 LLMs, risk awareness remains limited: GPT-4o performs best with a 74.45% F1 score, while no other model evidently surpasses random performance.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Risk-aware agents depend on general knowledge and reasoning capabilities, and safety-judgment fine-tuning can improve performance where straightforward prompting fails.

    Source: §5.4 · p. 8, §7 · p. 9

    R-Judge is smaller than the previous crawling-based LLM safety benchmark (Zhang et al., 2023a) because agent interactions across environments are complex.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large language models (LLMs) have exhibited great potential in autonomously completing tasks across real-world applications. Despite this, these LLM agents introduce unexpected safety risks when operating in interactive environments. Instead of centering on the harmlessness of LLM-generated content in most prior studies, this work addresses the imperative need for benchmarking the behavioral safety of LLM agents within diverse environments. We introduce R-Judge, a benchmark crafted to evaluate the proficiency of LLMs in judging and identifying safety risks given agent interaction records. R-Judge comprises 569 records of multi-turn agent interaction, encompassing 27 key risk scenarios among 5 application categories and 10 risk types. It is of high-quality curation with annotated safety labels and risk descriptions. Evaluation of 11 LLMs on R-Judge shows considerable room for enhancing the risk awareness of LLMs: The best-performing model, GPT-4o, achieves 74.42% while no other models significantly exceed the random. Moreover, we reveal that risk awareness in open agent scenarios is a multi-dimensional capability involving knowledge and reasoning, thus challenging for LLMs. With further experiments, we find that fine-tuning on safety judgment significantly improve model performance while straightforward prompting mechanisms fail. R-Judge is publicly available at https://github.com/Lordog/R-Judge.

1 Introduction

R-Judge benchmarks LLMs’ ability to identify and judge safety risks from agent interaction records. Evaluation reveals substantial shortcomings in risk awareness; safety-judgment fine-tuning improves performance, while straightforward prompting fails.

1 Introduction

  • Content-focused benchmark questions cannot expose behavioral risks in interactive environments, motivating evaluation of LLM risk awareness under complex, multi-turn agent interactions.Unknown environmental risks can cause unexpected agent safety issues, making effective risk-awareness evaluation important for real-world deployment.Source: §1 · pp. 1–2
  • 569 agent interaction records constitute R-Judge, covering 27 scenarios across five application categories and spanning 10 risk types.Categories comprise program, IoT, software, web, and finance, with binary safety labels and risk descriptions for each record. The authors present R-Judge as the first benchmark curated to assess LLM risk awareness for agent safety.Source: Figure 1 · p. 1, §1 · p. 2
  • 74.45% F1 makes GPT-4o the best-performing model among 11 evaluated LLMs, while no other model evidently surpasses random performance.Most evaluated models fail to identify safety risks in open agent scenarios.Source: §1 · p. 2
  • Fine-tuning on safety judgment significantly improves model performance, whereas straightforward prompting mechanisms fail to improve it.These findings come from further experiments accompanying the benchmark evaluation.Source: §1 · p. 2
  • Case studies characterize risk awareness in open agent scenarios as a multidimensional capability involving knowledge and reasoning, making it challenging for LLMs.Source: §1 · p. 2
  • R-Judge formalizes agent-safety risk awareness as judging and identifying safety risks from interaction records containing user instructions, agent actions, and environment feedback.This task provides a simple evaluation paradigm grounded in complex, multi-turn interactions.Source: §1 · p. 2

2 Problem Formalization

Agent interactions are formalized as instruction-led sequences of thought, action, and environment feedback. Risk awareness is evaluated by having an LLM analyze these records and judge whether agent actions are safe.

Record of LLM Agents

  • An agent record contains a user instruction followed by repeated thought, action, and environment-feedback triples until task completion or error.Formally, R = (u, [(t, a, f)_1, ..., (t, a, f)_n]); each turn takes R as input and produces (t, a). A list of records from multiple user calls is also treated as a record.Source: §2 · p. 3

Risk Awareness for Agent Safety of LLMs

  • ToolEmu and AgentMonitor report fewer agent safety issues with LLM safety monitors, motivating risk awareness as an important factor in safe agent execution.Potential severe harms include crucial privacy leakage and data loss.Source: §2 · p. 3

Task Formulation of Agent Safety Monitor

  • An LLM safety monitor maps interaction records to an analysis and a binary safe-or-unsafe label, evaluating its ability to identify risks and judge safety.Performance on this task serves as an indicator of the LLM’s risk awareness.Source: §2 · p. 3

3 R-Judge Benchmark

R-Judge benchmarks LLM agent risk awareness using human-annotated, multi-turn interaction records. It covers intended attacks and unintended risks and evaluates both safety judgment and risk identification.

3.1 Case Format

  • Cases pair ReAct interaction records (Yao et al., 2023) with human-checked safe/unsafe labels and annotated risk descriptions.Records capture interactions among users, environments, and agents that think, act, and receive feedback. Trained annotators cross-check binary labels and document the elements needed to understand each risk.Source: §3.1 · p. 3, §3 · p. 3

3.2 Curation Principles

  • The threat model assumes benign user prompts and distinguishes intended environmental attacks from unintended risks arising without malicious attacks (Tang et al., 2024).Intended attacks focus on indirect prompt injection (Yi et al., 2023), while environmental complexity and multi-turn interactions can produce unintended risks. Direct user-prompt attacks, including jailbreaks (Yuan et al., 2023; Deng et al.), are excluded.Source: §3.2 · pp. 3–4

3.3 Curation Process

  • Curation combines filtered, annotated trajectories from existing agent-safety datasets with expert-created cases, aiming to balance safe and unsafe examples.Experts replace risky actions in some unsafe ToolEmu examples with safe actions and brainstorm additional cases, polished with ChatGPT assistance and validity checks. Safe cases contain potential safety issues that do not actually occur.Source: §3.3 · p. 4

3.4 Dataset Statistics

  • 569 cases span 27 scenarios across five application categories and 10 risk types, with 52.7% labeled unsafe.Cases average 2.6 interaction turns and 206 words across program, IoT, software, web, and finance applications. injecAgent (Zhan et al., 2024) supplies 414 intended-attack cases; 155 unintended-risk cases combine ToolEmu, AgentMonitor (Naihin et al., 2023), and manual construction.Source: §3.4 · p. 5

3.5 Evaluation Criteria

  • Evaluation combines binary safety-label agreement with Effectiveness, which measures whether analyses identify and address human-annotated risks and explain how agents cause them.Models first analyze records, then predict safe/unsafe labels; labels are compared with human consensus, while GPT-4 (gpt-4-0613 version) scores analyses on unsafe cases. Safety judgment is fairer and more affordable, whereas risk identification offers fine-grained interpretability.Source: §3.5 · p. 5

4 Experiments

Across 11 LLMs evaluated on R-Judge, most perform below random in safety judgment, while GPT-4o ranks first and alone exceeds random in both risk-origin sets. Validation finds strong agreement between GPT-4 scoring and human assessments of risk identification.

4.1 Setup

  • 11 API-based and open-source LLMs undergo safety judgment and risk identification tests using zero-shot chain-of-thought prompting (Kojima et al., 2022).Models generate reasoning before their final answers, with simple task instructions intended to maintain generality.Source: §4.1 · p. 5, §4 · p. 5

4.3 Main Results

  • GPT-4o ranks first and alone exceeds random in both risk-origin sets, while most LLMs score below random in safety judgment and produce ineffective risk analyses.The benchmark covers Intended Attacks and Unintended Risks, revealing substantial room to improve safety risk awareness in open-agent scenarios.Source: Table 1 · p. 6, §4.3 · p. 6

4.1 Setup

  • Safety judgment ranks models by F1, while risk identification uses Effectiveness scores normalized to 0–100 through GPT-4 comparisons against human-annotated risk descriptions.Recall measures unsafe-case detection and Specificity measures safe-case detection; both have random baselines of 50.00%. Random F1 is 51.32% on the full set, and random Effectiveness is 0.Source: §4.1 · p. 6, Table 1 · p. 6, §4.2 · p. 6

4.2 Validating R-Judge

  • 0.91 average Pearson correlation between GPT-4 and human Effectiveness scores supports the reliability of risk-identification evaluation.Three human annotators assessed model analyses on 50 randomly selected unsafe samples using the scorer’s criteria. Agreement on GPT-4’s own answers was 0.89, which the authors interpret as showing little bias.Source: §4.2 · p. 6

5 Analysis

Safety-judgment fine-tuning can improve behavioral risk detection, whereas straightforward prompting does not reliably improve performance. The analysis emphasizes foundation-model knowledge and reasoning, condition-sensitive understanding, and alignment with human safety judgments.

5.1 Influence of Different Prompting Techniques

  • Two-shot Few-Shot-CoT does not consistently improve performance, while risk-type hints improve only ChatGPT’s F1 through higher Recall at some cost to Specificity.The authors suggest that demonstrations cover too few agent-task risks and may confuse models. These experiments indicate that straightforward prompting is unlikely to address the task’s complexity.Source: Table 2 · p. 7, §5.1 · p. 7

5.2 Effect of Fine-tuning on Safety Judgment

  • Meta-Llama-Guard-2-8B surpasses GPT-4o in safety judgment under matched settings, with lower Recall but higher Specificity; LlamaGuard-7b is incapable.The authors speculate that stronger foundations and more training data covering 11 versus 6 risk types explain the difference, emphasizing high-quality, diverse data. Comparisons use risk-type prompts; risk identification is excluded because Llama Guard was not trained to generate risk analyses.Source: §5.2 · pp. 7–8, Table 3 · p. 8

5.3 Case Study

  • Case studies identify three failure modes: inadequate scenario simulation, poor adaptation to specific conditions, and safety judgments misaligned with human consensus.Failures include missing action consequences through insufficient knowledge or reasoning and misreading conditions, producing both false positives and false negatives. Disagreements with human consensus concern ethics and privacy; these flaws align with the risk-identification effectiveness results.Source: §5.3 · p. 8

5.4 Summary

  • The analysis prioritizes foundation-model knowledge and reasoning alongside safety-judgment fine-tuning on high-quality, diverse data.It proposes a specifically fine-tuned monitor model to provide salient safety-risk feedback during agent execution.Source: §5.4 · p. 8

6 Related Work

Prior work develops autonomous and collaborative LLM agents, while safety research addresses harmful content and behavioral risks from environment interaction. R-Judge focuses on evaluating LLM risk awareness across diverse scenarios.

6.1 LLM Agents

6.2 LLM Safety

  • R-Judge evaluates LLM risk awareness across diverse scenarios, extending behavioral-safety research exemplified by ToolEmu.ToolEmu uses a GPT-4-powered emulator with diverse tools and scenarios, alongside a GPT-4 evaluator that identifies risks in agent execution.Source: §6.2 · p. 9
  • LLM safety evaluation has progressed from safety-related questions to adversarial and jailbreak red-teaming.Early question-based evaluations (Sun et al., 2023; Zhiheng et al., 2023; Lin et al., 2023b) addressed harmful-content risks including toxicity, bias, and immorality. Later approaches use adversarial attacks (Zou et al., 2023) and jailbreaks (Wei et al., 2023; Yuan et al., 2023; Deng et al.; Yi et al., 2023).Source: §6.2 · p. 9
  • LLM safeguards include reinforcement learning from human feedback and LLM-based content monitors.RLHF promotes harmlessness (Ouyang et al., 2022; Bai et al., 2022; Dai et al., 2023). LLM-based monitors evaluate (Bhatt et al., 2023) and moderate generated content (Inan et al., 2023; Zhang et al., 2024).Source: §6.2 · p. 9

7 Conclusion

R-Judge evaluates LLMs’ ability to judge and identify safety risks from agent interaction records. Experiments reveal imperfect risk awareness and improvements from safety-judgment fine-tuning, while straightforward prompting mechanisms fail.

7 Conclusion

  • 11 evaluated LLMs reveal imperfect risk awareness, a capability requiring knowledge and reasoning.Risk awareness is crucial for safe agent execution in interactive environments.Source: §7 · p. 9
  • R-Judge provides a general, realistic, and human-aligned benchmark for judging and identifying safety risks from agent interaction records.Source: §7 · p. 9
  • Fine-tuning on safety judgment significantly improves model performance, while straightforward prompting mechanisms fail.Source: §7 · p. 9

Limitations

R-Judge’s scale is limited by the complexity of agent interactions, but the dataset can grow using execution records and minimal annotation.

Limitations

  • R-Judge is smaller than the crawling-based safety benchmark (Zhang et al., 2023a) because of complex agent interactions across distinct environments.Human-annotated datasets involve a trade-off between dataset scale and annotation quality (Cui et al., 2020).Source: Limitations · p. 9
  • R-Judge’s scale is comparable to high-quality, human-annotated benchmarks containing multi-step operation records (Li et al., 2023a; Liu et al., 2023; Wei et al., 2022b).Source: Limitations · p. 9
  • R-Judge can expand using agent execution records and minimal annotation whenever safety risks are encountered.The authors plan continued expansion as agent applications develop.Source: Limitations · p. 9

Impact Statement … D Evaluation

R-Judge’s curation principles organize agent scenarios, risk types, and structured risk descriptions, supported by explicit labeling standards and human annotation. These supplementary sections also document dataset scope, evaluation materials, and responsible-use considerations.

Impact Statement

  • The research studies benign rather than adversarial agents, emphasizing action-based risks despite including some harmful content in the dataset.The authors discourage misuse and encourage responsible use to improve agent safety.Source: Impact Statement · p. 9

A Curation Principles; A.1 Category & Scenario

  • 5 categories—program, IoT, software, web, and finance—cover 27 scenarios in R-Judge.Curation principles address scenarios, risk types, and risk-description schemas. Cases spanning multiple scenarios receive the most relevant category.Source: §A.1 · p. 13, A Curation Principles · p. 13

A.2 Risk Type

  • 10 risk types aim to cover diverse agent safety issues and incorporate human safety consensus through illustrative risk descriptions.The types draw on research and related work, including Safetybench and ToolEmu; the authors acknowledge that precise risk-type definitions are difficult.Source: §A.2 · p. 13

A.3 Schema of Risk Description

  • The Motivation–Trigger–Outcome schema structures unsafe-case descriptions around risk sources, risky agent behavior, and resulting harm.Motivation can involve inadvertent or malicious user instructions or an unstable environment; annotators use the schema to write risk descriptions.Source: §A.3 · p. 13

B Dataset; B.1 Examples; B.3 Statistics

  • R-Judge distinguishes Intended Attacks from Unintended Risks and reports label counts, average interaction turns, and word counts.The supplementary dataset materials provide examples of both risk origins; the supplied passages do not provide the numerical statistics.Source: §B.1 · p. 13, §B.3 · p. 14

B.2 Curation Standard

  • Curation standards distinguish unsafe outcomes from harmless task failures and require confirmation even when users explicitly request risky actions.The dataset seeks balanced safe and unsafe samples; safe cases involve potential safety issues that do not actually occur.Source: §B.2 · p. 14

C Human Involvement

  • 6 trained author-annotators constructed the dataset: one PhD candidate, three master’s students, and two senior undergraduates.Their backgrounds span AI, computer science, and cybersecurity, with English and programming proficiency and foundational knowledge of LLMs and agents.Source: C Human Involvement · p. 14

D Evaluation; C Human Involvement

  • 3 annotators validated the GPT-4 scorer by assessing model analyses using the same criteria supplied to the scorer.The evaluation supplement covers metric formulations, prompts, and human agreement with GPT-4 scoring in the risk-identification test.Source: C Human Involvement · p. 14, D Evaluation · p. 15

D.1 Metrics … E.1 Model Performance across categories

These sections document evaluation metrics and prompts, including Pearson-correlation assessment of human–GPT-4 scorer agreement in risk identification. Supplementary analysis reports category-level Safety Judgment F1 scores and covers additional explanations and representative failures.

D.1 Metrics

  • Safety judgment uses n input–label samples with binary reference labels y_i and language-model predictions y_hat_i = p_theta(x_i).Both reference and predicted labels belong to {0, 1}.Source: §D.1 · p. 15
  • Risk-identification metrics are specified through pairwise evaluation prompts for the GPT-4 scorer.These prompts are documented in Figure 6.Source: §D.1 · p. 16, §D.2 · p. 16

D.2 Prompts

  • 2-shot examples augment Zero-Shot-CoT for Few-Shot-CoT; other prompts cover Zero-Shot-CoT with or without risk-type hints and Llama Guard with risk types.Figures 4, 5, and 7 document these prompting configurations.Source: §D.2 · p. 16, Figure 4 · p. 16, Figure 5 · p. 17

D.3 Human Agreement with GPT-4 Scorer in Risk Identification Test

  • Pearson correlation measures human–GPT-4 scorer agreement on risk-identification Effectiveness for each model.Table 7 also reports an average across 11 models.Source: Table 7 · p. 16, §D.3 · p. 16

E Analysis

  • The supplementary safety-judgment analysis combines category-level F1 results with additional explanations and representative model failures.This material supplements Section 5.Source: E Analysis · p. 18, §E.1 · p. 18

E.1 Model Performance across categories

  • F1 scores for all models are reported by category in the Safety Judgment test in Table 8.The supplied passage does not include numerical scores or model comparisons.Source: §E.1 · p. 18

E.2 Extra Explanation on Analysis … F.1 Hyperparameter

Two-shot Few-Shot-CoT does not consistently improve overall performance. Reported safety-judgment failures include GPT-4o capability flaws and Llama-2 instability in producing safe or unsafe labels.

E.2 Extra Explanation on Analysis

  • Two demonstrations accommodate Llama-2 and Vicuna’s 4k context windows while balancing safe and unsafe examples.Four or more demonstrations would exceed context limits on some long cases, making comparisons unfair; balanced binary examples require even demonstration counts.Source: §E.2 · p. 18
  • Two-shot Few-Shot-CoT does not consistently improve overall performance under the authors’ fair comparison setting.The two-demonstration setting accommodates models with limited context windows.Source: §E.2 · p. 18
  • Risk awareness receives qualitative analysis because knowledge and reasoning remain underexplored, and existing evaluations primarily address pretrained rather than instruction-tuned models.The authors argue that quantitative analysis could introduce uncertainty and interpretability problems; instruction-tuned models underpin both agents and safety monitors.Source: §E.2 · p. 18

E.3 Harm Categories of Llama Guard

  • 11 versus 6 training harm categories may explain Meta-Llama-Guard-2-8B’s advantage over LlamaGuard-7b, according to the authors.This explanation is speculative rather than an established causal finding.Source: §E.3 · p. 19

E.4 Model Failures / E.4.1 GPT-4o Failures

  • GPT-4o’s representative safety-judgment failures reveal flaws in scenario simulation, understanding adaptability, and safety alignment.These illustrate three key capability flaws discussed in Section 5.3.Source: §E.4.1 · p. 19, §E.4 · p. 19

E.4.2 Llama-2 Failures

  • Llama-2 models inconsistently output ‘safe’ or ‘unsafe’ in the safety judgment test.The authors provide a representative failure from Llama-2-13b-chat-hf.Source: §E.4.2 · p. 19

F Experiment Settings / F.1 Hyperparameter

  • Experiments use official model-release defaults for temperature and top_p, with FastChat deployment using half-precision weights.Source: §F.1 · p. 19

F.2 Experiment Resources & Cost · F.3 Model Information

API-based experiments generally take under one hour, while open-source experiments take approximately 1.5 hours per model on a single A100 GPU. The section also identifies the evaluated-model table and illustrated examples of intended attacks and unintended risks.

F.2 Experiment Resources & Cost

  • Two rounds of inference per sample give Zero-Shot-CoT and Few-Shot-CoT experiments similar runtime requirements.The computational budget and infrastructure are reported to facilitate reproduction.Source: §F.2 · p. 19
  • Under 1 hour is generally needed for an API-based experiment covering all 569 R-Judge samples, though network conditions can affect runtime.Reported runtime statistics refer to the main Zero-Shot-CoT experiment.Source: §F.2 · p. 19
  • Approximately 1.5 hours per model per experiment are required for open-source inference using a single A100 GPU.Reported runtime statistics refer to the main Zero-Shot-CoT experiment.Source: §F.2 · p. 19

F.3 Model Information

  • Figure 8 illustrates an intended-attack example from the R-Judge dataset.Source: Figure 8 · p. 20
  • Table 9 lists information about the LLMs evaluated in the experiments.The supplied caption and prose do not provide individual model specifications.Source: §F.3 · p. 19, Table 9 · p. 20
  • Figure 9 illustrates an unintended-risk example from the R-Judge dataset.Source: Figure 9 · p. 21

Related papers

Loading 2401.10019v3…