PaperlayerView exact revision on arXiv

Source-linked AI summary

VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?

Junpeng Liu, Yifan Song, Bill Yuchen Lin, Wai Lam, Graham Neubig, Yuanzhi Li, Xiang Yue

arXiv:2404.05955v1cs.CLcs.AI

TL;DR

Web-page evaluation remains challenging because general multimodal benchmarks miss web-specific characteristics and web-agent benchmarks do not measure fine-grained skills such as OCR, understanding, and grounding. VisualWebBench assesses MLLMs with seven web tasks, and evaluation finds substantial challenges and performance gaps. The analysis also identifies inadequate grounding in text-rich settings and limitations with low-resolution image inputs.

  • Problem

    General multimodal benchmarks miss web-page characteristics, while end-to-end web-agent benchmarks do not assess fine-grained skills such as OCR, understanding, and grounding.

    Source: §1 · p. 1

  • Method

    VisualWebBench is a QA-style benchmark of seven web tasks, comprising 1.5K instances for evaluating MLLMs’ web capabilities.

    Source: §1 · p. 2

  • Results

    GPT-4V and Claude Sonnet average 64.6 and 65.8, respectively, while leading open-source LLaVA-1.6-34B averages 50.5 on VisualWebBench.

    Source: §1 · p. 2

  • Takeaways & Limitations

    VisualWebBench offers a standardized evaluation resource for web understanding that the paper says can support development of web-capable models and applications.

    Source: §1 · p. 3

    Current MLLMs show inadequate grounding in text-rich environments and subpar performance with low-resolution image inputs.

    Source: §1 · p. 3

Abstract

from arXiv · show

Multimodal Large Language models (MLLMs) have shown promise in web-related tasks, but evaluating their performance in the web domain remains a challenge due to the lack of comprehensive benchmarks. Existing benchmarks are either designed for general multimodal tasks, failing to capture the unique characteristics of web pages, or focus on end-to-end web agent tasks, unable to measure fine-grained abilities such as OCR, understanding, and grounding. In this paper, we introduce \bench{}, a multimodal benchmark designed to assess the capabilities of MLLMs across a variety of web tasks. \bench{} consists of seven tasks, and comprises 1.5K human-curated instances from 139 real websites, covering 87 sub-domains. We evaluate 14 open-source MLLMs, Gemini Pro, Claude-3 series, and GPT-4V(ision) on \bench{}, revealing significant challenges and performance gaps. Further analysis highlights the limitations of current MLLMs, including inadequate grounding in text-rich environments and subpar performance with low-resolution image inputs. We believe \bench{} will serve as a valuable resource for the research community and contribute to the creation of more powerful and versatile MLLMs for web-related applications.

1 Introduction

Web-page evaluation is difficult because pages combine structured layouts, rich text, and interactive elements, while existing benchmarks do not assess web skills comprehensively. VisualWebBench evaluates MLLMs across seven web tasks, and results show substantial challenges and gaps between model groups.

1 Introduction

  • Web pages combine structured layouts, rich textual information, and diverse interactive elements, making accurate website comprehension a distinct challenge for MLLMs.The paper frames accurate website comprehension as a capability that could unlock web-domain applications.Source: §1 · p. 1
  • General multimodal benchmarks miss web characteristics, while web-agent benchmarks focus on end-to-end abilities rather than fine-grained OCR, semantic understanding, and grounding.Assessing these component skills supports targeted improvement and a clearer picture of model performance.Source: §1 · p. 2
  • VisualWebBench comprises seven QA-style tasks assessing website-, element-, and action-level understanding, reasoning, and grounding.The tasks include captioning, webpage QA, heading OCR, element OCR, element grounding, action prediction, and action grounding.Source: §1 · p. 2, Figure 1 · p. 2
  • GPT-4V and Claude Sonnet average 64.6 and 65.8, respectively, while leading open-source LLaVA-1.6-34B averages 50.5 on VisualWebBench.The results also identify weak grounding and limited image-resolution handling among current MLLMs.Source: §1 · pp. 2–3

2 Related Work

Related work spans general multimodal benchmarks and web-agent environments, but the paper positions VisualWebBench as a comprehensive evaluation of MLLM capabilities in web contexts. It addresses the need to assess basic perception, understanding, grounding, and reasoning alongside agent tasks.

2.1 MLLM Benchmarks

  • VisualWebBench differs from prior MLLM benchmarks by spanning perception, comprehension, grounding, and reasoning in web-based scenarios.The paper situates web pages as structured, text-rich, and interactive, unlike the focus of many general benchmarks.Source: §2.1 · p. 3

2.2 Web Agent Benchmarks

  • Figure 3 presents the benchmark’s domain distribution and tasks, with U, R, and G denoting understanding, reasoning, and grounding.Its domain counts represent the number of sub-domains in each domain.Source: Figure 3 · p. 4
  • Web-agent research moved from simplified simulated navigation environments to realistic, reproducible environments, yet comprehensive assessment of basic web-scene capabilities remains lacking.The paper names MiniWob++ and WebShop among earlier efforts, and Mind2Web, WebArena, and VisualWebArena among recent environments.Source: §2.2 · pp. 3–4

3 The VisualWebBench Benchmark

VisualWebBench evaluates MLLM understanding and grounding across web pages using seven tasks and a curated, diverse set of websites. Its construction spans website selection, task-specific annotation, and task-appropriate evaluation metrics.

3.1 Overview of VisualWebBench

  • VisualWebBench covers 139 websites, 1.5K samples, 12 domains, and 87 sub-domains, assessing website-, element-, and action-level abilities across seven tasks.The tasks evaluate understanding, OCR, grounding, and reasoning; samples are human-verified and curated.Source: §3.1 · p. 4

3.2 Website Selection

  • Websites were selected from top-ranking sites across 87 sub-domains, prioritizing rich interactive elements and excluding sites used in Mind2Web and WebArena.The authors rendered and saved selected sites with Playwright.Source: §3.2 · p. 4

3.3 Task Construction

  • Captioning uses GPT-4V to produce unified-style webpage descriptions from screenshots and extracted meta descriptions, while WebQA uses up to five human-written reasoning questions per screenshot.Captions are verified and curated by the authors; WebQA answers are designed to be precise and objective.Source: §3.3 · p. 5
  • Heading OCR extracts the first HTML <h1> from a raw screenshot, while Element OCR targets text descriptions exceeding 20 words within a marked screenshot region.Element OCR target boxes and text are extracted from the HTML DOM tree.Source: §3.3 · p. 5
  • Element Grounding asks models to match an HTML-element description to one of eight candidate boxes, using automatically extracted target and negative examples.The simplified candidate-box setting was adopted after preliminary studies found that models struggled to directly provide target coordinates.Source: §3.3 · p. 5
  • Action Prediction asks models to select a redirected-page title after a click, while Action Grounding asks them to choose an element matching a human instruction.Action Prediction uses a highlighted target and eight choices; Action Grounding presents eight candidate boxes and was annotated by seven experienced annotators.Source: §3.3 · p. 5
  • All tasks use a VQA-style formulation, and screenshots are standardized to 1280 pixels wide; samples undergo collaborative verification and curation.Two authors divide the verification and curation work.Source: §3.3 · p. 6

3.4 Evaluation Metrics

  • Open-ended generation is scored with ROUGE-L, WebQA with SQuAD-style F1, and multiple-choice tasks with accuracy.The cited metric sources are Lin (2004) for ROUGE-L and Rajpurkar et al. (2016) for SQuAD-style F1.Source: §3.4 · p. 6

4 Experiments

The experiments show that web-page tasks remain difficult for MLLMs, with proprietary models generally ahead of open-source systems. Performance also varies across web subtasks and does not consistently track results on general or agent benchmarks.

4.1 Evaluated MLLMs

  • The evaluation covers 14 open-source MLLMs, plus Gemini Pro, Claude models, GPT-4V, and two GUI-specialized models.The evaluated GUI-specialized models are SeeClick and CogAgent.Source: §4.1 · p. 6

4.2 Main Results

  • GPT-4V scores only 64.6 on average, while many models struggle to exceed 12.5 random chance on Action Prediction and Action Grounding.The authors describe substantial room for improvement on web tasks.Source: §4.2 · p. 7
  • Proprietary models outperform open-source models; LLaVA-1.6-34B scores 50.5, scaling boosts almost all tasks, and GUI-specialized models show no significant gains.SeeClick does not outperform its Qwen-VL base across tasks; the authors also report catastrophic forgetting in general instruction following after GUI-grounding training.Source: §4.2 · p. 7

4.3 Correlations with General Scenario and Agent Benchmarks

  • VisualWebBench and Mind2Web scores are generally low-correlated, while MMMU’s overall relationship is nonsignificant despite strong links with WebQA and Action Prediction.VisualWebBench scores are generally higher than Mind2Web scores; Captioning and Action Grounding show negative correlations with Mind2Web.Source: Figure 5 · p. 7, Figure 6 · p. 8, §4.3 · p. 8

4.4 Correlation Between VisualWebBench Tasks

  • Captioning, WebQA, and Element OCR correlate strongly; Action Prediction and Action Grounding correlate minimally, and Action Grounding is distinct from non-grounding tasks.The authors interpret the weak Action Prediction–Action Grounding correlation as indicating distinct required skill sets.Source: Figure 7 · p. 8, §4.4 · p. 8

4.5 Analysis of Image Resolution

  • Higher input resolutions generally accompany higher scores, and increasing LLaVA-1.6 resolution improves performance across all three sizes.For LLaVA-1.6, gains are greater from 336 to 448 than from 448 to 672; the authors identify 448×448 as the minimum for adequate web-task performance.Source: §4.5 · pp. 8–9, Figure 8 · p. 9

4.6 Analysis of Grounding Capability

  • GUI-agent models outperform general MLLMs at predicting target coordinates, while models trained on general grounding data still fail to locate targets accurately.The study evaluates unannotated screenshots using bounding-box or point prediction; the table distinguishes Multi-Choice, Bbox, and Point settings.Source: Table 2 · p. 9, §4.6 · p. 9

4.7 Case Studies

5 Conclusion

The conclusion presents VisualWebBench as a seven-task benchmark spanning website, element, and action levels. Its evaluation finds substantial challenges for current MLLMs, including weak grounding in text-rich settings and poor performance with low-resolution inputs.

5 Conclusion

  • VisualWebBench evaluates MLLM web-page understanding and grounding across seven tasks and three levels: website, element, and user action.The conclusion identifies inadequate grounding in text-rich environments and subpar performance with low-resolution image inputs as current limitations.Source: §5 · p. 10

A Annotation Tool of Action Grounding

The action-grounding annotation tool guides annotators from understanding a website and writing an instruction to selecting and confirming its interactive target.

A Annotation Tool of Action Grounding

  • Annotators first learn the website’s purpose from its description and may search for the site name to understand it better.They use GPT-4V-generated action-description examples as a reference when writing diverse instructions.Source: A Annotation Tool of Action Grounding · p. 15
  • Annotators hover over the interactive element that completes the instruction and press “s” to select it.A green rectangle indicates the selected element.Source: A Annotation Tool of Action Grounding · p. 15
  • Annotators verify the selected element matches the instruction, submit it, and allow screen capture.The tool marks the selected rectangle with a blinking green indicator during confirmation.Source: A Annotation Tool of Action Grounding · p. 15

B Data Verification and Curation

The authors describe a collaborative verification and curation process for benchmark samples, with checks tailored to each task’s annotations and content.

B Data Verification and Curation

C Details of Evaluated MLLMs

The evaluated MLLMs include a range of open-source model families and several commercial models. Experiments use deterministic generation at temperature 0.0 on NVIDIA A100 80G GPUs.

C Details of Evaluated MLLMs

  • The open-source lineup spans varied architectures, including Q-Former-based BLIP-2, high-resolution DeepSeek-VL, and decoder-only Fuyu.Other listed design approaches include modular networks, vision encoders, and visual expert modules.Source: C Details of Evaluated MLLMs · p. 16
  • The evaluation considers various general large multimodal models, including BLIP-2, InstructBLIP, mPLUG-Owl2, Otter, VILA, Fuyu, SPHINX, LLaVA-1.5/1.6, Qwen-VL, DeepSeek-VL, Yi-VL, and CogVLM, alongside Gemini Pro, Claude Sonnet, Claude Opus, and GPT-4V(ision) for comparison.By default, for each model family, the latest, largest, and best-performing available checkpoint to date is used.Source: C Details of Evaluated MLLMs · p. 16
  • All models use temperature 0.0 for deterministic generations, and experiments run on NVIDIA A100 80G GPUs.Source: C Details of Evaluated MLLMs · p. 16

D VisualWebBench vs. Mind2Web

Table 3 presents MLLM scores on VisualWebBench and Mind2Web.

D VisualWebBench vs. Mind2Web

E Additional Case Studies

The case studies illustrate model responses on WebQA and four additional VisualWebBench tasks. In the WebQA example, GPT-4V answers correctly while CogAgent and LLaVA-34b do not.

E Additional Case Studies

  • In the WebQA case, CogAgent gives unrelated content, LLaVA-34b gives a wrong answer, and GPT-4V's reasoning and answer are correct.The passage characterizes GPT-4V's performance in this case as demonstrating strong understanding and reasoning.Source: E Additional Case Studies · p. 16
  • Separate case studies cover Captioning, Heading OCR, Element Grounding, and Element OCR.Source: E Additional Case Studies · p. 16, Figure 14 · p. 17, Figure 15 · p. 18
  • Table 3 compares scores on VisualWebBench and Mind2Web, with some results taken from Cheng et al. (2024) and Zheng et al. (2024).The table caption marks the respective sourced results with * and †.Source: Table 3 · p. 17
Loading 2404.05955v1…