PaperlayerView exact revision on arXiv

Source-linked AI summary

Harnessing Webpage UIs for Text-Rich Visual Understanding

Junpeng Liu, Tianyue Ou, Yifan Song, Yuxiao Qu, Wai Lam, Chenyan Xiong, Wenhu Chen, Graham Neubig, Xiang Yue

arXiv:2410.13824v3cs.CVcs.CL

TL;DR

Text-rich visual understanding requires MLLMs to combine dense text with visual context, but existing web-data methods provide limited contextual and interactive coverage. The paper synthesizes multimodal instructions from webpage accessibility trees, pairs them with screenshots, and releases MultiUI, whose training improves UI tasks and generalizes to document, OCR, and chart understanding.

  • Problem

    MLLMs need to integrate dense textual content with visuals, while existing web-data approaches suffer from noise, rigid representations, or limited text-visual interaction coverage.

    Source: §1 · p. 2

  • Method

    MultiUI uses text-based LLMs to generate multimodal instructions from structured webpage accessibility trees and pairs them with screenshots across diverse tasks and layouts.

    Source: §1 · p. 2, §2.1 · p. 3, §2.3 · p. 4

  • Results

    MultiUI training improves UI benchmarks and generalizes to document understanding, OCR, and chart interpretation, including up to 48% improvement on VisualWebBench and 19.1% higher Mind2Web element accuracy.

    Source: §1 · p. 2, §6 · p. 12

  • Takeaways & Limitations

    Structured web UI data is a broadly useful training resource for text-rich visual understanding across UI and non-UI scenarios.

    Source: §1 · p. 2, §6 · p. 12

Abstract

from arXiv · show

Text-rich visual understanding-the ability to process environments where dense textual content is integrated with visuals-is crucial for multimodal large language models (MLLMs) to interact effectively with structured environments. To enhance this capability, we propose synthesizing general multimodal instructions from webpage UIs using text-based large language models (LLMs). Despite lacking direct visual input, text-based LLMs are able to process structured text representations from webpage accessibility trees. These instructions are then paired with UI screenshots to train multimodal models. We introduce MultiUI, a dataset containing 7.3 million samples from 1 million websites, covering diverse multimodal tasks and UI layouts. Models trained on MultiUI not only excel in web UI tasks-achieving up to a 48% improvement on VisualWebBench and a 19.1% boost in element accuracy on a web agent dataset Mind2Web-but also generalize surprisingly well to non-web UI tasks and even to non-UI domains, such as document understanding, OCR, and chart interpretation. These results highlight the broad applicability of web UI data for advancing text-rich visual understanding across various scenarios.

1 INTRODUCTION

The paper addresses text-rich visual understanding by using structured webpage UIs to synthesize multimodal instructions with text-based LLMs. MultiUI scales this approach into a diverse dataset that improves UI performance and generalizes to non-UI visual tasks.

  • Text-rich visual understanding requires MLLMs to integrate dense textual information with surrounding visuals across documents, webpages, charts, and related tasks.Source: §1 · p. 2
  • Webpage UIs provide a naturally structured, diverse, and text-dense source for learning complex text-visual interactions.Source: §1 · p. 2
  • Existing web-data methods introduce noise, impose rigid simplified structures, or overlook interactions between textual and visual content.Source: §1 · p. 2
  • MultiUI synthesizes instructions from cleaned accessibility trees with text-based LLMs, then pairs them with UI screenshots for multimodal training.Source: §1 · p. 2
  • MultiUI contains 7.3 million samples from 1 million websites, spanning three task categories, varied layouts, devices, aspect ratios, and question formats.Source: §1 · p. 2
  • Training on MultiUI improves VisualWebBench by up to 48% and Mind2Web element accuracy by 19.1%, while also improving document understanding, OCR, and chart interpretation.Source: §1 · p. 2

2 DATASET CONSTRUCTION

MultiUI is constructed by scraping and curating webpage representations, extracting diverse multimodal tasks, and expanding them with varied instruction templates. The resulting dataset contains 7.3 million VQA samples across nine tasks.

2 DATASET CONSTRUCTION

  • The construction pipeline has four stages: raw website scraping, website curation, task extraction, and instruction construction.Source: §2 · p. 3

2.1 WEBSITE RAW DATA SCRAPING

  • The raw dataset combines HTML/CSS, high-resolution screenshots, and accessibility trees from 1.1 million websites rendered on Windows and iPhone devices.Source: §2.1 · p. 3

2.2 WEBSITE CURATION

  • Llama-3-70B-Instruct filters harmful or invalid websites, removing approximately 10% of crawled sites and leaving 1 million website instances.Source: §2.2 · pp. 3–4

2.3 TASK EXTRACTION FROM SCRAPED WEBSITES

  • Task extraction covers visual understanding and reasoning, text recognition, and grounding, with prompts manually refined from generated samples.Source: §2.3 · p. 4

2.3.1 VISUAL UNDERSTANDING AND REASONING TASKS

  • Visual understanding tasks include webpage and embedded-image captioning, webpage and image QA, and action prediction using accessibility trees, contextual captions, and interaction outcomes.Source: §2.3.1 · pp. 4–5

2.3.2 OCR TASKS.

  • OCR tasks target long textual elements and webpage headings, using screenshots and extracted headline text to support recognition.Source: §2.3.2 · p. 5

2.3.3 GROUNDING TASKS

  • Grounding tasks generate or select element locations from descriptions or instructions, using DOM-derived boxes in multi-choice and coordinate-prediction settings.Source: §2.3.3 · pp. 5–6

2.4 INSTRUCTION TEMPLATE CONSTRUCTION

  • GPT-4o generates 200 varied prompt templates per task, producing MultiUI’s 7.3 million VQA samples across nine perception, comprehension, grounding, and reasoning tasks.Source: §2.4 · p. 6

3 EXPERIMENTAL SETUP

The experiments use UIX models with multiple LLM backbones, dynamic high-resolution image processing, and a two-stage training strategy. Evaluation spans GUI, OCR, general multimodal, grounding, and agent benchmarks against broad baseline families.

3.1 IMPLEMENTATION DETAILS

  • UIX uses Qwen2-7B-Instruct as its primary backbone and additionally tests Vicuna-7B-v1.5 and Llama-3.1-8B-Instruct.Source: §3.1 · p. 6
  • Dynamic high-resolution processing independently encodes resolution-matched image patches and combines them with a downsampled global image representation.Source: §3.1 · p. 6
  • Training first uses 95% of MultiUI for GUI knowledge, then combines general visual instruction data with the remaining 5% for broader multimodal capabilities.Source: §3.1 · pp. 6–7

3.2 BENCHMARKS

  • Evaluation covers GUI understanding and grounding, OCR-related tasks, general multimodal benchmarks, natural-image grounding, and the Mind2Web agent task.Source: §3.2 · p. 7

3.3 BASELINES

  • Comparisons include LLaVA variants, matched reimplemented baselines, Pix2Struct, S4, SeeClick, CogAgent, ScreenAI, GPT-4V, and GPT-4o.Source: §3.3 · p. 7

4 EXPERIMENTAL RESULTS AND ANALYSIS

MultiUI improves GUI understanding, web navigation, grounding, and general multimodal performance, with gains increasing with data scale and balanced task mixtures.

4.1 RESULTS ON GUI-RELATED TASKS

  • UIX-Qwen2 reaches 75.9 on VisualWebBench, exceeding LLaVA-1.6-34B (50.5) and GPT-4V (64.6).Across three backbones, MultiUI improves all nine GUI benchmarks, with gains above 20% for LLaVA-1.5 training and up to 64.9% for LLaVA-1.6 training.Source: §4.1 · p. 7, Table 2 · p. 8

4.2 RESULTS ON GUI AGENT TASKS

  • MultiUI improves Mind2Web element accuracy by up to 19.1%, and UIX-Qwen2-M2W surpasses SeeClick and CogAgent across all three test subsets.The 7B UIX model is smaller than SeeClick (9.6B) and CogAgent (18B).Source: §4.2 · p. 8, Table 4 · p. 9

4.3 RESULTS ON GENERAL DOCUMENT UNDERSTANDING AND GROUNDING TASKS

  • UIX-Qwen2 outperforms LLaVA-1.6-34B and DocOwl across evaluated OCR-related benchmarks, while UIX-Vicuna improves RefCOCO+ by 4.0% over models trained exclusively on general instruction data.Source: §4.3 · pp. 9–10

4.4 RESULTS ON GENERAL MULTIMODAL TASKS

  • MultiUI preserves general multimodal capability: mixing MultiUI with LLaVA data performs on par with training on general visual instruction data alone.The two-stage strategy also outperforms single-stage training in both GUI and general scenarios.Source: §4.4 · p. 10, §4.5 · p. 10, Table 5 · p. 10

4.6 EFFECT OF SCALING UP DATA

  • Increasing MultiUI data generally improves all four task categories, with GUI grounding rising from approximately 12% to 60%.OCR-related, document, chart, and general grounding tasks show more moderate gains of about 5% to 10%.Source: §4.6 · p. 10, Figure 6 · p. 10

4.7 ABLATION ON TASK TYPE

  • QA and Caption samples improve GUI understanding and OCR benchmarks, OCR samples mainly aid text-recognition GUI tasks, and all sample types provide the most balanced performance.Grounding-only web samples can hurt RefCOCO+, which uses natural-scene images rather than webpage-like images.Source: §4.7 · p. 11, Figure 7 · p. 11

5 RELATED WORK

Prior web-based multimodal datasets underuse webpage structure or constrain generalization; MultiUI instead exploits accessibility trees to synthesize broad multimodal instructions.

Web-Based Multimodal Pre-training Dataset Collection

  • LAION, MMC4, and OBELICS capture image-text associations but do not fully represent webpage layouts and diverse UI elements.Rule-based extraction and interleaved image-text formats miss rich UI structure.Source: §5 · p. 11
  • Pix2Struct and S4 use webpage structure but remain limited by task coverage, downstream fine-tuning requirements, or weak transfer beyond web pages and HTML.These limitations motivate a dataset spanning broader multimodal capabilities and domains.Source: §5 · p. 11

MLLMs for GUI-related Multimodal Tasks

  • Unlike screenshot-only GUI models, MultiUI uses structured textual information from accessibility trees to synthesize instructions for perception, understanding, reasoning, and grounding.The approach targets web UI tasks while improving text-rich visual understanding beyond UI settings.Source: §5 · pp. 11–12

6 CONCLUSION

The paper presents MultiUI as a large-scale accessibility-tree-based dataset that improves multimodal generalization, while documenting screenshot cropping and coordinate-augmented tree representations.

6 CONCLUSION

  • MultiUI contains 7.3 million samples spanning UI types and tasks, and improves document understanding, GUI comprehension, grounding, and web navigation.The conclusion emphasizes structured web data as a resource for text-rich visual environments.Source: §6 · p. 12

A SCREENSHOT CROPPING & ACCESSIBILITY TREE REFINEMENT

  • Full-page screenshots are horizontally cropped at sampled height-width ratios to fit image-model size constraints while preserving vertical webpage flow.The ratio ranges are [0.5, 1.5] for Windows 10 and [1.5, 2.5] for iPhone 12 Pro.Source: A SCREENSHOT CROPPING & ACCESSIBILITY… · p. 18

B EXAMPLE OF OUR PROCESSED ACCESSIBILITY TREE

C PROMPTS FOR TASK EXTRACTION

The task-extraction prompts use webpage descriptions, screenshots, and accessibility trees to generate grounded captions, questions, and executable actions. They impose constraints on answer format, difficulty, objectivity, spatial reasoning, and interaction validity.

Embedded Image QA

  • Embedded-image QA prompts use truncated accessibility trees and image captions to generate difficult, objective, diverse questions with short and detailed answers.Questions must be answerable from the caption and current webpage without relying on background knowledge.Source: C PROMPTS FOR TASK EXTRACTION · pp. 20, 23

Action Grounding

  • Action extraction additionally asks whether crawling succeeded and whether the webpage contains harmful content, returning one YES or NO per question.The prompt lists blocked, forbidden, blank, and missing pages as examples of unsuccessful crawls.Source: C PROMPTS FOR TASK EXTRACTION · p. 20
  • The prompts provide webpage descriptions and accessibility trees, with screenshots used as the visual context for task generation.Accessibility trees encode element IDs, types, text or descriptions, coordinates, and hierarchy.Source: C PROMPTS FOR TASK EXTRACTION · pp. 21, 24

Webpage Captioning

  • Captioning prompts require detailed descriptions of webpage visual and textual components, including element counts and relative positions.Bounding-box coordinates should be converted into natural-language spatial descriptions rather than reported directly.Source: C PROMPTS FOR TASK EXTRACTION · p. 21
  • Spatial descriptions must be derived from bounding-box coordinates rather than accessibility-tree order, using normalized reference points for absolute positions.The prompts explicitly distinguish relative ordering from coordinate-based vertical and absolute placement.Source: C PROMPTS FOR TASK EXTRACTION · pp. 21–22

Webpage QA

  • Webpage QA prompts generate diverse, objective questions with short answers under 10 words and detailed answers explaining the reasoning.Questions must be answerable from the current webpage, avoid advertisements, cookie sections, and main-heading queries, and omit references to the accessibility tree.Source: C PROMPTS FOR TASK EXTRACTION · p. 22
  • Generated outputs are constrained to JSON lines containing question-answer pairs or instruction-bounding-box pairs, with no additional text.The required formats are specified separately for webpage QA, embedded-image QA, and action grounding.Source: C PROMPTS FOR TASK EXTRACTION · pp. 22–24

Action Grounding

  • Action-grounding prompts generate one-step executable instructions paired with exact target bounding boxes for interactive webpage elements.Instructions must identify a single interactive element, exclude cookie controls and root elements, and use simple natural descriptions without element IDs.Source: C PROMPTS FOR TASK EXTRACTION · p. 24

D EXAMPLE TRAINING DATA BY TASK

The examples cover training data for webpage and embedded-image captioning and QA, action prediction, OCR, and grounding. Together, they illustrate the breadth of task formats included in the dataset.

E TRAINING SETUP

The section provides a table describing three models developed in the paper.

  • Table 6 summarizes details of the three models developed in this paper.Source: Table 6 · p. 28

F DETAILS OF EVALUATED BENCHMARKS

The evaluated benchmarks span GUI understanding, GUI grounding, web-agent interaction, document and chart question answering, OCR, and general multimodal reasoning. They measure both web-specific capabilities and broader visual understanding skills.

GUI Understanding Benchmarks

  • VisualWebBench, WebSRC, ScreenQA, and WidgetCap evaluate web or mobile UI understanding through web tasks, structural reading comprehension, screen QA, and UI description.VisualWebBench emphasizes fine-grained OCR, understanding, and grounding; ScreenQA contains 86K QA pairs over RICO.Source: GUI Understanding Benchmarks · p. 29

GUI Grounding

  • ScreenSpot and RefExp evaluate GUI grounding by testing cross-platform component localization and prediction of coordinates from natural-language expressions.Source: GUI Grounding · p. 29

GUI Agent Benchmark

  • Mind2Web evaluates generalist web agents on complex tasks across more than 2,000 tasks, 137 websites, and 31 domains.Source: GUI Agent Benchmark · p. 29

General OCR / DocQA / ChartQA Benchmarks

  • DocVQA, ChartVQA, TextVQA, InfoVQA, VisualMRC, and OCRBench assess document QA, chart reasoning, image-text understanding, infographic reasoning, document layout comprehension, and broad OCR-related skills.OCRBench spans 29 datasets covering text recognition, scene-text VQA, document VQA, key information extraction, and handwritten mathematical expression recognition.Source: General OCR / DocQA / ChartQA Benchmarks · pp. 29–30

General-domain Tasks

  • MMMU, MMBench, VQAv2, and RefCOCO+ evaluate broader multimodal reasoning, bilingual vision-language ability, visual-information use, and referring-expression understanding.Source: General-domain Tasks · p. 30

G EVALUATION DETAILS AND FULL RESULTS

This section describes the evaluation setup and reports full baseline and UIX results across the paper’s benchmarks. Results are organized through benchmark metrics and comparisons across three model backbones.

G EVALUATION DETAILS AND FULL RESULTS

H MIND2WEB EXPERIMENT DETAILS

The Mind2Web experiments test next-action prediction from screenshots, task descriptions, and previous actions, using cropped observations and a CogAgent-Chat baseline. The prompt example illustrates the grounded web-agent setting.

H MIND2WEB EXPERIMENT DETAILS

  • Mind2Web models receive a raw screenshot, task description, and previous actions, then directly predict the next action.This setup evaluates action prediction in an interactive webpage context.Source: H MIND2WEB EXPERIMENT DETAILS · p. 33
  • Mind2Web screenshots are randomly cropped around the ground-truth element to avoid excessively tall full-page observations.Original captures can exceed an 8:1 height-to-width ratio, so cropping follows a procedure similar to SeeClick.Source: H MIND2WEB EXPERIMENT DETAILS · p. 33

Prompt example for CogAgent

  • CogAgent-Chat is evaluated as a baseline using the recommended agent-operation version and a bounding-box-to-click-point conversion.The model’s predicted bounding-box center is used as the webpage operation point, as illustrated by a prompt containing task instructions and previous actions.Source: H MIND2WEB EXPERIMENT DETAILS · p. 33, Prompt example for CogAgent · p. 33
Loading 2410.13824v3…