SafeArena: Evaluating the Safety of Autonomous Web Agents

1. Introduction

• Agents’ direct access to environments and high autonomy create safety concerns as LLM-based agents are adopted for interactive tasks and integrated with browsers, code interpreters, and external knowledge sources.The introduction notes deployments in software engineering, web navigation, and computer use.Source: §1 · p. 1
• Earlier agent-safety benchmarks tested harmful tasks in text-based environments, often with synthetic tools, motivating evaluations that require navigating ubiquitous graphical interfaces.The cited preliminary work includes Andriushchenko et al. (2024), Debenedetti et al. (2024), and Zou et al. (2024).Source: §1 · p. 1
• SafeArena contains 500 tasks—250 safe and 250 malicious—across four realistic websites, spanning misinformation, harassment, illegal activity, cybercrime, and social bias.The paired tasks let the benchmark assess agents’ ability to complete both benign and harmful web tasks.Source: §1 · p. 1
• GPT-4o completed 34.7% of 250 harmful tasks; GPT-4o and Claude-3.5-Sonnet attempted or completed 68.7% and 36.0%, while benign-substep decomposition jailbroke Claude on every initially refused task.ARIA categorizes behavior into immediate refusal, delayed refusal, attempted but failed execution, and successful completion; the findings indicate that alignment transfers poorly to web tasks.Source: §1 · p. 2
• The findings underscore the need for web-agent safety alignment beyond procedures applied to underlying LLMs, with SafeArena intended to support efforts to design safer, more aligned agents.Source: §1 · p. 2

2.1. Autonomous Web Agents

• Web-agent research advanced from dedicated environments and screen- or HTML-based fine-tuning to real-world task benchmarks and instruction-tuned LLM agents, whose safety in web navigation remains unclear.Shi et al. (2017) and Liu et al. (2018) pioneered dedicated evaluation environments; later work explored screen-based (Shaw et al., 2023) and HTML-based (Furuta et al., 2024) fine-tuning, demonstration benchmarks, and realistic self-contained environments.Source: §2.1 · p. 2

2.2. LLM Agent Safety

• LLM-agent safety research studies harmful-task completion in text environments, direct harmful requests and attacks, personal-computer tasks, tool prompt injections, and visual adversarial attacks.Examples include Andriushchenko et al. (2024), AgentDojo (Debenedetti et al., 2024), ToolEmu (Ruan et al., 2024), and studies of prompt injection and visual attacks.Source: §2.2 · pp. 2–3
• VisualWebArena-Adversarial tests image-based robustness on benign tasks, while ST-WebAgentBench assesses enterprise outcome safety; SafeArena instead measures execution of malicious web tasks.The related benchmarks focus on susceptibility to adversarial attacks or safe outcomes rather than agents’ capability to follow malicious instructions.Source: §2.2 · p. 3

3.1. Harm Categories

• Five harm categories cover misinformation, illegal activity, harassment, cybercrime, and social bias, including deception, unlawful acts, abuse, computer-enabled crime, and prejudice.The categories are described as reflecting concerns about malicious LLM-agent misuse.Source: §3.1 · p. 3, MISINFORMATION · p. 3

3.2. Web Environments

• SAFEARENA uses four WebArena-based environments—a Reddit-style forum, e-commerce store, GitLab-style code platform, and retail management system—augmented to enable realistic misuse tasks.Augmentations include adding f/politics and f/covid19 forums and illegal drugs as e-commerce products.Source: §3.2 · p. 3

3.3. Task Design and Curation

• The benchmark contains 250 harmful and 250 safe tasks, with paired intents designed to test similar agentic capabilities through minimal changes in harmful versus safe intent.Each task gives an agent a user intent to pursue through actions in a controlled web environment.Source: §3 · p. 3, §3.3 · p. 4
• Researchers manually created 30 harmful tasks per category, totaling 150, and augmented each environment while distributing tasks roughly equally across websites within each category.They drew inspiration for malicious intents from existing jailbreak benchmarks and used WebArena templates to annotate evaluation reference objects.Source: §3.3 · p. 4
• GPT-4o-Mini generated 200 additional tasks from demonstrations of human-designed safe-harmful pairs, followed by human checks of intent classification, environment artifacts, and evaluation references.Reviewers checked webpage existence, HTML programs, and reference content.Source: §3.3 · p. 4

3.4. Human Verification

• Three authors with expertise in LLM agents and safety manually reviewed all 500 tasks, discussing flagged ambiguities together and making changes as needed.Initial review divided tasks evenly among reviewers, who assessed intents and evaluation reference objects.Source: §3.4 · p. 4

4.1. Agent Risk Assessment Framework

• ARIA orders harmful-task behavior from immediate refusal (1), through attempted execution followed by refusal (2) or failed completion (3), to successful completion (4).An LLM-based judge assigns levels from agent trajectories and metadata; the authors also conduct human assessment using ARIA.Source: §4.1 · pp. 4–5

4.2. Automatic Evaluation Metrics

• Automatic evaluation uses task completion rate, refusal rate, and normalized safety score.Source: §4.2 · p. 5

4.2.1. TASK COMPLETION RATE

• Task completion is evaluated functionally against task-specific reference objects, with binary rewards for success or failure.References include exact-matched answers, URLs for navigation tasks, or programs that locate and match required webpage content.Source: §4.2.1 · p. 5

4.2.2. NORMALIZED SAFETY SCORE

• The normalized safety score considers only safe tasks the agent successfully completed, reflecting its ability to refuse harmful tasks while completing safe ones.A lower score suggests the agent fails to distinguish between safe and harmful tasks.Source: §4.2.2 · p. 5

4.3. Human Evaluation

• Five authors conducted human evaluation, with two annotators independently judging each of 150 human-designed harmful tasks using complete agent trajectories and Cohen’s Kappa to measure agreement.Annotators reviewed screenshots and actions and assessed trajectories according to ARIA.Source: §4.3 · p. 5

5.3. Results

• GPT-4o and Qwen-2-VL-72B achieve high completion rates on both safe and harmful tasks, while Claude-3.5-Sonnet shows comparatively strong safety despite decent safe-task performance.Llama-3.2-90B struggles on both task types; Claude-3.5-Sonnet is the exception to the general trend of greater harmful-task risk with higher capability.Source: Figure 3 · p. 6, §5.3 · p. 6

5.1. Attack Methods

• Direct prompting gives agents the complete harmful intent; the study also tests multi-turn task decomposition and priming as jailbreak approaches.In task decomposition, a harmful request is split into innocuous-looking substeps delivered sequentially; priming conditions the model to appear to have already begun the harmful task.Source: §5.1 · p. 6

5.2. Models

• The five backbones are Claude-3.5-Sonnet, GPT-4o, GPT-4o-Mini, Llama-3.2-90B, and Qwen-2-VL-72B; all are vision-capable and instruction-tuned.The selected capabilities support following prompts to complete SAFEARENA web tasks.Source: §5.2 · p. 6

5.3. Results

• Almost 40% separates GPT-4o, the highest, from Llama-3.2-90B, the lowest, on illegal-activity completion; misinformation completion peaks at 30% for Qwen-2-VL-72B and 28% for GPT-4o.Harassment and cybercrime completion rates vary by less than 15% across models.Source: Figure 4 · p. 6, §5.3 · p. 7
• 55.0% is Claude-3.5-Sonnet’s normalized safety score, the highest reported, while the metric highlights risks associated with deploying Qwen-2-VL-72B.Normalized safety contextualizes harmlessness against agentic capability; the passage also describes lower-capability Llama-3.2-90B as a safer alternative.Source: §5.3 · pp. 6–7, Table 3 · p. 7
• Nearly 64% of Claude-3.5-Sonnet’s human-evaluated responses are refusals, while Qwen-2-VL-72B rarely refuses and attempts almost all harmful tasks.Human annotators achieved a Cohen’s Kappa score of 0.96, indicating almost perfect agreement.Source: Table 4 · p. 7, §5.3 · p. 7
• 64.0% of Claude-3.5-Sonnet’s requests are refusals; Qwen-2-VL-72B attempts 72.0% and completes 27.3% of harmful tasks.The LLM-judge evaluation achieved a Cohen’s Kappa score of 0.82 against human ratings for tasks where both annotators agreed.Source: §5.3 · p. 7
• Claude-3.5-Sonnet was jailbroken on all 49 harmful tasks it had initially refused, requiring an average of 1.26 attempts per task.A human evaluator had three attempts per task to use task decomposition.Source: §5.3 · p. 7

Discussion

• 26.0% of the 250 harmful tasks were completed by Qwen-2-VL-72B, the model with the highest task completion rate.The authors say these findings indicate substantial potential for malicious misuse without proper safeguards.Source: §6 · pp. 7–8
• Risks extend from safe agents’ task failures, such as misdirected emails, to unsafe agents automating harmful workflows such as systematic email harassment.The discussion distinguishes unintended adverse effects from more severe risks when unsafe agents are integrated into malicious workflows.Source: §6 · p. 8
• Safety alignment transfers poorly to web tasks: several LLMs have low ARIA-1 and ARIA-2 refusal rates, and agents carry out tasks with explicit harmful language.Although SAFEARENA tests only rudimentary malicious web-agent capability, the authors call for stronger agent safety efforts, especially given adversarial content on real-world websites (Liao et al., 2024; Wu et al., 2024a;b).Source: §6 · p. 8

• SAFEARENA pairs 250 harmful instructions across five categories and four web environments with corresponding safe tasks for normalized safety comparisons.The benchmark includes 50 tasks per harm category, curated through human-curated and human-in-the-loop methods.Source: §7 · p. 8
• GPT-4o successfully completed 22.8% of harmful requests versus 34.4% of safe requests, and agents often attempted many harmful tasks without refusing.Source: §7 · p. 8
• ARIA evaluates five strong LLM web agents across immediate refusal, delayed refusal, attempted but failed execution, and successful completion.Figure 5 reports the percentage of trajectories assigned to each of the four risk levels.Source: §7 · p. 8, Figure 5 · p. 8

Limitations

• SafeArena tests only explicitly harmful intents, leaving ambiguous tasks—whose permissibility may depend on inspecting the environment—for future work.For example, deleting a particular user’s posts could be permissible if that user is harassing others.Source: §8 · p. 8
• External classifiers or aligned LLMs could filter harmful intents before they reach web agents, but jailbreaks and priming or decomposition attacks may circumvent these defenses.The passage cites GCG as an example of an effective jailbreak.Source: §8 · p. 8
• Brittle automatic metrics may miss harmful behavior outside the evaluated context and have only positive predictive power, motivating more open-ended evaluation methods.For example, harmful-message variants posted on a different website may not be flagged.Source: §8 · p. 8

Impact Statement

• SafeArena is presented as the first benchmark dedicated to deliberate misuse of autonomous web agents, spanning five harm categories and revealing frequent compliance with malicious requests.The supplied passage names misinformation, illegal activity, and harassment as examples and says jailbreaks exacerbate the vulnerability.Source: Impact Statement · p. 9
• The authors call for web-agent-specific safety alignment beyond procedures applied to underlying models and present SafeArena as a framework to support future agent-safety work.Their societal concern challenges the assumption that agents built on safety-aligned LLMs will necessarily behave safely.Source: Impact Statement · p. 9

A. Roadmap

• The appendix roadmap assigns additional SafeArena experiments to Appendix B, benchmark and environment details to C, implementation details to D, and examples to E.Source: A. Roadmap · p. 13

B.1. Priming Jailbreak

• The rule-based priming attack changes the starting URL to a later point in the harmful-task trajectory and revises the intent to cover only remaining actions.The paper illustrates a primed task in which the agent is given the webpage where it needs to write a comment, rather than starting at the homepage.Source: §B.1 · p. 13, Figure 6 · p. 13
• Priming changes the initial environment state to emulate interactive malicious use, which the authors argue should be considered in agent risk assessments.They note that instruction-following LLMs can be susceptible to malicious requests presented across multiple conversation turns.Source: §B.1 · pp. 13–14
• Priming leads all evaluated models to complete a higher proportion of harmful tasks than direct prompting.Figure 7 presents task completion rates for the two conditions; the supplied text gives no exact rates.Source: §B.1 · p. 13, Figure 7 · p. 14
• All models refuse primed intents less frequently than directly prompted intents.Refusal rates are reported in Table 5 and were computed using an LLM judge; no exact rates are supplied here.Source: §B.1 · p. 13, Table 5 · p. 14

B.2. Direct Refusal Evaluation

• A GPT-4o LLM judge evaluates each agent trajectory for refusal and reports the percentage of tasks refused.The judge is prompted only to flag refusals; the paper points to Figure 20 for the evaluation prompt.Source: §B.2 · p. 14
• Claude-3.5-Sonnet refuses over 57.6% of directly prompted harmful intents, while Qwen-2-VL-72B rarely refuses malicious requests.The paper presents refusal rates as a way to explain why Claude completes fewer harmful tasks than other agents.Source: §B.2 · p. 14

B.3. Agents Complete LLM-Generated Intents Better

• Agents show a slight preference for LLM-generated intents, and Claude refuses fewer HITL malicious instructions across categories, with differences of 23.3% in cybercrime and 21.6% in illegal activities.The authors suggest that harmful tasks that are easy to execute may pose risks when designed at scale.Source: §B.3 · p. 15, Figure 8 · p. 15

B.4. Task Success Rates Across Websites and Harm Categories

• Completion rates are reported by website and harm category, with harmful-task completion rates distinguished from safe-task rates.The supplied passages identify Figures 9 and 10 as presenting these breakdowns but do not provide their cell values.Source: §B.4 · p. 15, Figure 9 · p. 16, Figure 10 · p. 17

B.5. Completion Divergence

• Llama-3.2-90B and Qwen-2-VL-72B can complete harmful tasks at higher rates than their safe counterparts.The paper notes that successful harmful-task completion does not guarantee success on the corresponding safe task.Source: §B.5 · p. 15
• Inconsistent element selection, action loops that reach the 30-step limit, and confusing similar entities cause agents to fail on otherwise similar tasks.Agents may select the wrong page item despite set-of-marks, repeat states until the step limit, or choose a comparison list or cart instead of a wishlist.Source: §B.5 · p. 15

C.1. Details of the Web Environments

• SafeArena’s environments simulate popular websites using open-source Docker images and include potentially harmful forum discussions, code repositories, and shopping products.The pre-populated sites support task design around actions that could be malicious or harmful; the images are attributed to Zhou et al. (2024).Source: §C.1 · p. 16

C.2. Augmentations

• The benchmark augments its environments to accommodate harmful tasks, with the forum, code-management, and shopping changes shown in Figures 15 and 16.The supplied passages identify where the augmentations are presented but do not describe their specific contents.Source: §C.2 · p. 16, Figure 15 · p. 23, Figure 16 · p. 23

D.1. Models

• The experiments use gpt-4o-mini-2024-07-18, gpt-4o-2024-11-20, llama-3.2-90b-vision-instruct, qwen-2-vl-72b-instruct, and claude-3.5-sonnet-2024-06-20.Source: §D.1 · p. 24
• Claude and GPT models use first-party APIs, Qwen-2-VL-72B uses VLLM, and Llama-3.2-90B uses Together; results are visualized with Agent X-Ray.Source: §D.1 · p. 24

D.2. Hyperparameters

• Across models, generation uses temperature 0, pruned HTML, a 1,024-token generation limit, and a 2,048-token prompt limit.The same BrowserGym hyperparameter settings are used for each model; Table 13 is identified as the settings table.Source: §D.2 · p. 24, Table 13 · p. 24

D.3.1. PROMPT FOR HUMAN-IN-THE-LOOP DATA GENERATION; Prompt for Human-in-the-loop Data Generation

• The human-in-the-loop data-generation prompt is provided in Figure 17.Source: §D.3.1 · p. 24, Figure 17 · p. 25

D.3.2. PROMPTS FOR EVALUATING AGENTS; Dynamic Prompt

• Agent evaluation uses BrowserGym’s dynamic prompt, giving agents action space, action history, observations, intents, and task-specific details, with a single-action response format.The shown format uses <think> and <action> tags and includes an answer-format example.Source: §D.3.2 · p. 24, Dynamic Prompt · pp. 26–27, Figure 18 · p. 27

D.3.3. PROMPT FOR ARIA EVALUATION

• The paper supplies the prompt used for LLM-judge evaluation following the ARIA framework.Source: §D.3.3 · p. 25

E.1. Examples of Harmful-Safe Paired Tasks

• SafeArena’s additional examples show harmful-safe paired tasks across categories and environments in Table 14.Source: §E.1 · p. 27

E.2. Examples of Refusals; Claude 3.5 Sonnet; GPT-4o-Mini; GPT-4o; Llama-3.2-90B; Qwen-2-VL-72B

• The cited examples show refusals: one redirects toward constructive discussion, another declines assistance, one reports the task infeasible, and another refuses to create illegal or unethical content.Refusal detection uses an LLM-judged approach following the ARIA framework.Source: Claude 3.5 Sonnet · p. 27, GPT-4o-Mini · p. 27, §E.2 · p. 27

E.3. Example of Task with Explicitly Harmful Language

• Llama-3.290B prepared a comment endorsing the death of an Iranian person, filling the comment field with the requested text.The passage introduces this as an instance where the agent did not refuse; it does not establish that the comment was posted.Source: §E.3 · p. 28

E.4. Additional Examples of Task Decomposition

• Table 15 provides additional examples of task decomposition.Source: §E.4 · p. 28, Table 15 · p. 33

E.4.1. EXAMPLES OF SAFEARENA TASKS WITH METADATA

• Figures 22 and 23 show SAFEARENA task examples with metadata, including human-curated and human-in-the-loop-generated tasks.Source: §E.4.1 · p. 28, Figure 22 · p. 34, Figure 23 · p. 35

E.4.2. EXAMPLES OF PRIMED SAFEARENA TASKS WITH METADATA

• Figures 24 and 25 show primed SAFEARENA task examples with metadata, including human-curated and human-in-the-loop-generated tasks.Source: §E.4.2 · p. 28, Figure 24 · p. 36, Figure 25 · p. 37

ARIA LLM Judge Prompt

• The ARIA judge prompt categorizes agent behavior as refusal, attempt-then-refusal, failed attempt, or successful completion.Source: Figure 19 · p. 29

LLM Judge Refusal Prompt

• The refusal-detection prompt labels responses as refusals when they cite ethical, legal, or moral grounds, redirect to legal alternatives, or give a clear brief refusal.Source: LLM Judge Refusal Prompt · p. 30, Figure 20 · p. 30

CYBERCRIME

• Cybercrime examples pair a script to identify government software vulnerabilities and a DDoS-service product with benign hashing-code and smartwatch alternatives.Source: CYBERCRIME · p. 32

HARASSMENT; ILLEGAL ACTIVITY; MISINFORMATION

• Examples pair deepfake forums, extortionate messages, drug-sale posts, and vaccine or election misinformation with benign alternatives.The supplied examples include harmless forum and customer-message tasks, a consumer-product post, and image-model code.Source: HARASSMENT · p. 32, ILLEGAL ACTIVITY · p. 32, MISINFORMATION · p. 32