PaperlayerView exact revision on arXiv

Source-linked AI summary

VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks

Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, Daniel Fried

arXiv:2401.13649v2cs.LGcs.CLcs.CV

TL;DR

Routine computer tasks require visual understanding, but agents must also process language instructions and act on the web. VisualWebArena introduces visually grounded tasks and evaluates LLM and VLM agents; VLMs outperform text-based LLMs, yet remain far below human performance.

  • Problem

    Routine computer-task agents must process visual and textual inputs because digital interfaces are primarily built for human eyes.

    Source: §1 · p. 1

  • Method

    VisualWebArena introduces 910 visually grounded tasks across three web environments and evaluates LLM and VLM agents, alongside a Set-of-Marks-inspired agent.

    Source: §1 · pp. 1–2

  • Results

    VLMs outperform text-based LLMs, but the best VLM agents achieve 16.4% success on VisualWebArena versus human performance of 88.7%.

    Source: §1 · p. 1

  • Takeaways & Limitations

    The analysis identifies reasoning, visual understanding, and planning as directions for future agent improvement.

    Source: §7 · p. 9

    The evaluation does not consider non-binary rewards, leaving continuous performance scales unexplored.

    Source: §3.3 · p. 4

Abstract

from arXiv · show

Autonomous agents capable of planning, reasoning, and executing actions on the web offer a promising avenue for automating computer tasks. However, the majority of existing benchmarks primarily focus on text-based agents, neglecting many natural tasks that require visual information to effectively solve. Given that most computer interfaces cater to human perception, visual information often augments textual data in ways that text-only models struggle to harness effectively. To bridge this gap, we introduce VisualWebArena, a benchmark designed to assess the performance of multimodal web agents on realistic \textit{visually grounded tasks}. VisualWebArena comprises of a set of diverse and complex web-based tasks that evaluate various capabilities of autonomous multimodal agents. To perform on this benchmark, agents need to accurately process image-text inputs, interpret natural language instructions, and execute actions on websites to accomplish user-defined objectives. We conduct an extensive evaluation of state-of-the-art LLM-based autonomous agents, including several multimodal models. Through extensive quantitative and qualitative analysis, we identify several limitations of text-only LLM agents, and reveal gaps in the capabilities of state-of-the-art multimodal language agents. VisualWebArena provides a framework for evaluating multimodal autonomous language agents, and offers insights towards building stronger autonomous agents for the web. Our code, baseline models, and data is publicly available at https://jykoh.com/vwa.

1 Introduction

VisualWebArena addresses the limited evaluation of agents on visually grounded web tasks with a 910-task benchmark and a Set-of-Marks-inspired agent. Evaluation finds that VLMs outperform text-based LLMs, while substantial gaps from human performance remain.

1 Introduction

  • The benchmark targets a gap in text-focused agent benchmarks by assessing visual and textual capabilities on tasks that integrate visual understanding with language processing.It builds on WebArena's reproducible self-hosted environments and execution-based evaluations.Source: §1 · p. 1
  • VisualWebArena comprises 910 tasks across Classifieds, Shopping, and Reddit; Classifieds is new with real-world data, while Shopping and Reddit are inherited from WebArena.All introduced tasks require visual understanding; 25.2% also include images as input and require interleaved image-text understanding.Source: §1 · p. 1, Figure 1 · p. 2
  • VLM agents outperform text-based LLMs, but the best VLM agents achieve 16.4% success on VisualWebArena versus human performance of 88.7%.The proposed Set-of-Marks-inspired agent also outperforms baseline LLM agents, especially on more visually complex sites.Source: §1 · p. 1
  • The proposed VLM agent draws on Set-of-Marks prompting to simplify the action space and performs especially well against baseline LLM agents on visually complex sites.The approach is inspired by Set-of-Marks prompting (Yang et al., 2023a).Source: §1 · p. 2

2 Related Work

Related work spans web-agent benchmarks, LLM-based web agents, and advances in vision-language models. The paper distinguishes its Set-of-Marks agent from prior action grounding by using marked elements directly as the VLM's observation and action space.

Language-Guided Web Agent Benchmarks

LLM Agents

Vision-Language Models

  • VLM research spans image captioning, visual question answering, and multimodal architectures; contemporaneous action grounding identifies HTML elements, whereas this SoM agent uses marked elements as the VLM's observation and action space.The contrast with action grounding is described in Zheng et al. (2024); the SoM uses Yang et al. (2023a).Source: §2 · pp. 2–3

3 VisualWebArena Environment

VisualWebArena models web interaction as a deterministic, partially observable environment and provides visual and structural observations for agents to act on. Its evaluation uses task-specific binary rewards, including checks for navigation, text answers, and visual questions.

3 VisualWebArena Environment

  • The environment is a deterministic POMDP: agents act from partial observations, transitions update the state, and a binary final reward indicates whether the task goal was achieved.The websites are standalone self-hosted applications intended to support reproducibility, realism, and determinism.Source: §3 · p. 3

3.1 Observation Space

  • Observations can include webpage URLs, tabs, and page content represented as HTML, an accessibility tree, screenshots, or Set-of-Marks screenshots.In 25.2% of tasks, the intent also involves one or more input images.Source: §3.1 · p. 3

3.2 Action Space

  • For Set-of-Marks agents, actions reference unique IDs assigned to interactable elements, avoiding coordinate prediction and focusing action selection on higher-level reasoning.Accessibility-tree agents instead use the element ID in the tree.Source: §3.2 · pp. 3–4, Figure 2 · p. 4

3.3 Evaluation

  • Visual QA awards success when a BLIP-2-T5XL answer contains the ground-truth response; the evaluation uses binary rewards rather than graded scores.The authors identify non-binary rewards as a direction for future work.Source: §3.3 · p. 4
  • Navigation tasks are checked against the final webpage state, while text outputs can be evaluated by exact, inclusion, exclusion, or semantic matching.The paper also describes visual question answering rewards and notes that non-binary rewards are not considered in this work.Source: §3.3 · pp. 4–5

4 Curating Visually Grounded Tasks

VisualWebArena spans three realistic web environments and provides 910 visually grounded tasks, with difficulty labels and human performance measurements. The benchmark includes tasks that require visual interpretation rather than relying only on page text or HTML.

4.1 Web Environments

  • VisualWebArena includes Classifieds, Shopping, and Reddit, with tasks that can involve its Wikipedia knowledge base or interactions across multiple websites.Classifieds has 65,955 listings, and Reddit contains 31,464 posts with varied images.Source: §4.1 · p. 5

4.2 Tasks

  • The benchmark adds 910 tasks from 314 templates, including 46 unachievable tasks that test whether agents terminate early and explain why.Six graduate-student co-authors wrote intent templates, which annotators expanded into tasks and paired with reward functions.Source: §4.2 · p. 5
  • Every task requires visual understanding, and difficulty labels combine visual and reasoning complexity with action and visual difficulty annotations.Action difficulty is based on estimated human actions; visual levels range from basic identification to multiple images, small or lengthy OCR, and fine details.Source: §4.2 · pp. 5–6

4.3 Human Performance

  • 88.7% was the overall human success rate on a representative 230-task sample; remaining errors often involved missed objectives, exhaustive searches, or comparing products.Seven college students were measured, with task creators excluded from the tasks they had initially created.Source: §4.3 · p. 6

5 Baselines

The baselines evaluate prompt-based LLM and VLM agents on VisualWebArena, including text-only, caption-augmented, and multimodal approaches. The caption-augmented experiments compare two image-captioning models, while multimodal agents are tested with webpage screenshots and Set-of-Marks inputs.

5.1 Text-based LLM Agents

  • Text-based LLM agents use Chain-of-Thought prompting over accessibility-tree inputs, with API-based and open-source models tested.The shared baseline setup uses three non-overlapping in-context examples, one from each environment.Source: §5.1 · p. 7, §5 · p. 7

5.2 Image Caption Augmented LLM Agents

  • Caption-augmented agents add captions for webpage images as accessibility-tree alt-text and include captions of task input images in the prompt.With GPT-3.5, LLaVA and BLIP-2 have comparable performance; BLIP-2 has a slightly higher success rate, is smaller, and requires less GPU VRAM, so it is used for remaining experiments.Source: §5.2 · p. 7

5.3 Multimodal Agents

  • Multimodal agents are evaluated with screenshots plus captions and an accessibility tree, or with screenshots annotated by bounding boxes and unique IDs alongside captions.The second setting applies Set-of-Marks preprocessing to interactable webpage elements.Source: §5.3 · p. 7

6 Results and Analysis

VisualWebArena results show that multimodal agents outperform text-only baselines, while all evaluated models remain far below human performance. SoM further improves GPT-4V, and task-subset results show differing performance across OCR, image-matching, and image-input tasks.

6 Results and Analysis

  • All existing models substantially underperform humans; text-only GPT-4 reaches 7.25%, rising to 12.75% with captions.The results are summarized in Table 3.Source: §6 · p. 7, Table 3 · p. 8
  • GPT-4V reaches 15.05% overall, while Gemini-Pro rises from 3.85% with captions to 6.04% multimodally.The authors suggest text-based agents may struggle with complex images requiring OCR or recognition of non-salient objects.Source: §6 · pp. 7–8
  • 15.05% → 16.37%: SoM raises GPT-4V’s overall success rate, with especially large gains on Classifieds and Reddit.The authors attribute the gains to denser visual content and difficulty distinguishing spatially close elements in the accessibility tree.Source: §6 · p. 8
  • In one Reddit task, GPT-4V + SoM navigates to /f/memes, identifies the target image among many, and blocks its author.The described trajectory reaches the goal without unnecessary actions.Source: §6 · pp. 8–9, Figure 3 · p. 8

6.1 Performance by Task Type

  • 13.4% versus 16.9%: GPT-4V + SoM succeeds less often on OCR tasks than on tasks without OCR.OCR is required by 17.1% of VisualWebArena tasks; the authors suggest it may bottleneck current agents.Source: §6.1 · p. 9
  • 18.9% versus 16.2%: GPT-4V + SoM succeeds slightly more often on exact-image-match tasks than on other tasks.Exact image matching is required by 8.7% of tasks; the authors suggest it is not a primary bottleneck.Source: §6.1 · p. 9
  • 19.0% versus 14.9%: GPT-4V + SoM succeeds more often on tasks that include image inputs than on tasks without them.Image inputs appear in 25.2% of VisualWebArena tasks.Source: §6.1 · p. 9

7 Conclusion

The paper introduces VisualWebArena to evaluate autonomous multimodal agents on realistic, visually grounded web tasks. Its evaluation finds that VLMs show promise but remain far behind humans, leaving substantial room for improving agent capabilities.

7 Conclusion

  • VisualWebArena is a benchmark of realistic tasks designed to evaluate and advance autonomous multimodal web agents.The paper presents it as addressing the gap in evaluation of visually grounded tasks.Source: §7 · p. 9
  • VLMs show promise, but their performance remains considerably below the very high success rates achieved by humans.The evaluation also identifies common failure modes among existing LLM and VLM agents.Source: §7 · p. 9
  • The authors identify reasoning, visual understanding, and planning as directions for future agent research.They also report the potential of Set-of-Marks-inspired prompting to simplify action spaces and improve performance on visually complex websites.Source: §7 · p. 9

8 Ethical and Broader Impacts

The paper discusses possible accessibility and work-automation benefits alongside employment, bias, and safety concerns. It frames VisualWebArena as a research sandbox and says the presented models are not intended for practical deployment in their current state.

8 Ethical and Broader Impacts

  • Autonomous agents could improve accessibility and automate routine computer work, while deployment could raise employment concerns.The paper says broader economic and social implications should be considered if agents are deployed in real-world applications.Source: §8 · p. 9
  • The authors call for further analysis of bias and safeguards, noting agents could cause more harm than regular LLMs without careful safeguards.They say further research is needed to understand and mitigate possible harmful behaviors.Source: §8 · p. 9
  • VisualWebArena is intended as a self-contained sandbox for research, and the paper’s models are not intended for practical deployment in their current state.The deployment warning especially applies to high-risk domains.Source: §8 · p. 9

Appendix

  • The appendix directs readers to further material on collected tasks, model failure modes, the Classifieds environment, and task collection.Source: Appendix · p. 12

A Tasks Breakdown

The appendix describes a 910-task benchmark spanning three web environments, with multi-site tasks and varied visual and action difficulty. It also reports that agent success tends to fall as task difficulty increases.

A Tasks Breakdown

  • VisualWebArena contains 910 tasks across Classifieds, Reddit, and Shopping, including some multi-site tasks and tasks referencing Wikipedia.The site breakdown is shown in Figure 4.Source: A Tasks Breakdown · p. 12, Figure 4 · p. 12
  • Tasks span a variety of visual and action difficulty levels, as summarized in Figure 5.The difficulty classification follows the specifications in Section 4.2.Source: A Tasks Breakdown · p. 12, Figure 5 · p. 12
  • Agent success rates are roughly correlated with task difficulty and decrease as difficulty increases.This relationship is discussed in the appendix’s analysis across difficulty levels.Source: A Tasks Breakdown · p. 12

B Additional Results

Additional evaluations compare newer models and show that more in-context examples and harder visual tasks relate to agent performance. The analyses examine few-shot prompting and success across task difficulty levels.

B Additional Results

  • GPT-4o achieves 19.78% success versus GPT-4V’s 16.37%, while Llama-3-70B-Instruct reaches 9.78%, slightly below caption-augmented GPT-4’s 12.75%.Source: B Additional Results · p. 13

C.1 Few-shot Prompting

  • Overall success rate tends to rise with more in-context examples, with a significant jump from 1 to 3 examples.Source: §C.1 · p. 13, Table 6 · p. 13

C.2 Performance by Task Difficulty

  • On hard visual tasks, GPT-4V + SoM averages 12.4% success versus caption-augmented 8.0% and text-only 4.8%; trajectories lengthen as action difficulty rises.Source: §C.2 · p. 13

C.3 Task Subset Analysis

Task-subset analyses reveal different challenges for OCR, exact image matching, and image-input tasks, while qualitative comparisons describe agent failures and differences in navigation. The Classifieds environment provides a large listing collection with search and filtering tools.

C.3 Task Subset Analysis

  • On OCR tasks, multimodal models improve over captions (6.4% to 12.2%), although GPT-4 models do worse on OCR tasks than non-OCR tasks.OCR tasks comprise 17.1% of the benchmark; Gemini-Pro multimodal agents show a different pattern, achieving higher success on OCR tasks than non-OCR tasks.Source: §C.3 · p. 14, Figure 7 · p. 15

C.4 Failure Modes

  • Other failures included undoing correct selections, missing an easy red-item task, giving up before scrolling to a button, and looping between pages.Source: §C.4 · pp. 15–16

C.3 Task Subset Analysis

  • GPT-4V + SoM performs better on exact-image-match tasks, while other GPT-4 agents and Gemini models have lower success on exact-match than non-exact-match tasks.The authors suggest SoM’s visual-centric observation and action space may suit exact matching.Source: §C.3 · p. 14, Figure 8 · p. 16

C.4 Failure Modes

  • Agents wasted steps on unused tabs and often appended rather than replaced field text, producing repeated queries or addresses and sometimes action loops.The authors believe the looping in the phone-number example stems from appending the corrected number rather than replacing the incorrect one.Source: §C.4 · pp. 16–17

C.3 Task Subset Analysis

  • GPT-4 agents generally do better on image-input tasks, unlike text-only GPT-4; Gemini-Pro agents generally do worse, while GPT-4V + SoM terminates most tasks under 10 steps.GPT-4V + SoM’s error rate remains relatively uniform across longer trajectories.Source: §C.3 · pp. 14–15, Figure 9 · p. 17, Figure 10 · p. 18

C.5 Comparison Between Agents

  • The text-only GPT-4 agent failed to identify a post from its title alone, and captioning could omit fine visual details needed to identify a location.For the Pittsburgh skyline example, GPT-4V + SoM identified the city, while the captioning model described only a skyline with tall buildings.Source: §C.5 · pp. 17–18
  • SoM’s simplified observation and action space helped GPT-4V succeed at a multi-step Classifieds task that the multimodal agent could not navigate efficiently.The task involved finding a white Google Pixel listing and posting a comment offering $10 less than its asking price.Source: §C.5 · pp. 18–19

D The Classifieds Environment

  • Classifieds contains 65,955 listings scraped from Northeastern Craigslist over three weeks, with PII redacted; users can search and filter by category or location.Source: D The Classifieds Environment · p. 19

E Task Collection Process

The task collection process used familiarization with the benchmark sites and encouraged realistic, visually grounded tasks. Baseline settings varied by model, while multimodal and SoM prompts included screenshots in their examples.

E Task Collection Process

  • Six computer-science graduate students familiar with the sites collected 910 tasks after exploring the environments; task creation encouraged visual layouts, input images, and cross-site functions.Templates were instantiated with different arguments to avoid repeated tasks and an overabundance of one task type.Source: E Task Collection Process · p. 19

F Baseline Agent Settings

  • Baseline viewport sizes and text-observation limits varied by model context length, and sampling temperatures and top-p settings also differed across model groups.Source: F Baseline Agent Settings · pp. 19–20
  • Baseline prompts used three in-context examples; multimodal and SoM agents received screenshots of examples and the current page, unlike text-only and caption-augmented agents.Source: F Baseline Agent Settings · p. 20
Loading 2401.13649v2…