Source-linked AI summary
Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding
Kenton Lee, Mandar Joshi, Iulia Turc, Hexiang Hu, Fangyu Liu, Julian Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wei Chang, Kristina Toutanova
TL;DR
Visually situated language blends text and visual context in documents, tables, infographics, and interfaces, while prior research often uses domain-specific recipes. Pix2Struct pretrains a pixel-only image-to-text model by parsing masked web-page screenshots into HTML, with variable-resolution inputs and rendered task prompts. Across four domains and nine tasks, it strongly outperforms Donut and improves substantially over domain-specific pipelines in low-resource domains.
Problem
Visually situated language requires jointly recognizing text, understanding language, and incorporating visual context, but prior work often relies on domain-specific recipes rather than shared general-purpose methods.
Method
Pix2Struct pretrains a pixel-only image-to-text model to predict HTML-based parses from masked web-page screenshots, using variable-resolution inputs and rendering task inputs onto images.
Results
Across four domains and nine tasks, Pix2Struct strongly outperforms Donut by 9 to 53 points and improves over domain-specific pipelines by 1 to 44 points in low-resource domains.
Takeaways & Limitations
A single pretrained pixel-only model can transfer across diverse visual language tasks and domains, while OCR pipelines or metadata may remain appropriate in some domains.
Takeaways & Limitations
Pixel-only models remain highly sensitive to input resolution because high-resolution images produce longer sequences, leaving OCR-based pipelines advantageous in some settings.
Abstract
from arXiv · showhide
Visually-situated language is ubiquitous -- sources range from textbooks with diagrams to web pages with images and tables, to mobile apps with buttons and forms. Perhaps due to this diversity, previous work has typically relied on domain-specific recipes with limited sharing of the underlying data, model architectures, and objectives. We present Pix2Struct, a pretrained image-to-text model for purely visual language understanding, which can be finetuned on tasks containing visually-situated language. Pix2Struct is pretrained by learning to parse masked screenshots of web pages into simplified HTML. The web, with its richness of visual elements cleanly reflected in the HTML structure, provides a large source of pretraining data well suited to the diversity of downstream tasks. Intuitively, this objective subsumes common pretraining signals such as OCR, language modeling, image captioning. In addition to the novel pretraining strategy, we introduce a variable-resolution input representation and a more flexible integration of language and vision inputs, where language prompts such as questions are rendered directly on top of the input image. For the first time, we show that a single pretrained model can achieve state-of-the-art results in six out of nine tasks across four domains: documents, illustrations, user interfaces, and natural images.
1. Introduction
Pix2Struct targets general-purpose visually-situated language understanding with a unified pixel-to-text model. It uses masked screenshot parsing, variable-resolution inputs, and rendered language prompts to transfer across diverse tasks and domains.
- Visually-situated language combines text and visual context in documents, tables, infographics, and user interfaces, requiring recognition, language understanding, and contextual reasoning.
- Previous approaches are fragmented across domains, relying on task-specific tools such as OCR, platform metadata, or diagram parses.These pipelines limit sharing of data, architectures, and objectives while increasing engineering complexity and computational cost.
- Pix2Struct pretrains by predicting an HTML-based parse from a masked webpage screenshot, using web structure to provide signals about text, images, and layouts.Masked inputs encourage joint reasoning about the co-occurrence of textual and visual elements.
- Variable-resolution inputs preserve image aspect ratios, while rendering questions or other prompts directly onto images unifies language and vision inputs.This design consumes downstream inputs through a single visual modality.
- 282M- and 1.3B-parameter models trained on 80M screenshots outperform Donut by 9–53 points across evaluated tasks.Against domain-specific pipelines, Pix2Struct improves low-resource illustrations and UI domains by 1–44 points but trails state of the art in high-resource documents and natural images.
2. Method
Pix2Struct adapts vision-transformer inputs to variable image layouts and pretrains on masked screenshot-to-HTML parsing. Downstream tasks are expressed through image preprocessing and text decoding.
- Pix2Struct uses raw pixels as input and token sequences as output, aiming to share one pretrained model across visually-situated tasks and domains.Downstream adaptation largely resides in preprocessing, analogous to text-to-text task formulation.
- 2.2. Variable-resolution inputs: Variable-resolution preprocessing extracts the maximal number of fixed-size patches fitting a sequence length while preserving the input aspect ratio.Two-dimensional absolute positional embeddings disambiguate variable resolutions and support changes in sequence length.
- 2.3. Pretraining: Screenshot parsing pairs webpage screenshots with simplified HTML derived from visible DOM structure and selected textual or image metadata.The target is shortened by selecting the largest linearized subtree that fits the decoder length, with its region marked on the screenshot.
- 2.3. Pretraining: Masking 50% of text spans and decoding the entire selected subtree supplies a BART-like context-modeling objective.The objective combines signals resembling OCR, masked language modeling, and image captioning.
- 2.4. Curriculum learning: A short text-reading warmup stabilizes screenshot-parsing pretraining, speeds convergence, and improves finetuning performance.The warmup trains on text snippets rendered with random colors and fonts.
- 2.5. Finetuning: For downstream tasks, captions use images directly, bounding-box tasks draw target boxes on images, and VQA questions are rendered as image headers.This lets Pix2Struct process questions and images jointly through the visual modality.
3. Experimental Setup
The evaluation spans four visual-language domains and nine tasks using standard splits without additional labeled data. Pix2Struct is compared with domain-specific state-of-the-art methods and a consistent Donut baseline.
- Evaluation covers illustrations, user interfaces, natural images, and documents across multiple visually-situated language benchmarks.The benchmark collection was designed for diversity in domains and task formats.
- Metrics follow the original benchmarks: ANLS, exact match, relaxed accuracy, and CIDEr for the corresponding task groups.
- Comparisons emphasize single-model, single-task baselines trained on standard splits for fair comparison and easier experimentation.Some state-of-the-art methods instead use ensembles, extra labeled data, validation data, or domain-specific inputs.
4. Results
Pix2Struct outperforms prior visual methods on 8 of 9 benchmarks and achieves state-of-the-art results on 6 across four domains. Gains are strongest in lower-resource illustrations and UI tasks, while specialized pipelines remain stronger in some high-resource settings.
- Illustrations: 58.6 relaxed accuracy improves ChartQA over VisionTaPas's 45.5 by using charts directly instead of extracted tables.Pix2Struct-Large does not rely on table extractors.
- Illustrations: Pix2Struct-Large exceeds DQA-NET by 3.6 points and Donut by 11.27 points on AI2D without domain-specific modifications.AI2D evaluates questions about illustrative science diagrams.
- UIs: Pix2Struct-Large improves UI captioning from CIDEr 127.4 to 136.7 on Widget Captioning and from 64.3 to 109.4 on Screen2Words.Both tasks use app screenshots, with widget- or page-function captions.
- Documents: Pix2Struct-Large achieves ANLS 76.6 on DocVQA and ANLS 40 on InfographicVQA despite using purely visual representations and no in-domain pretraining data.It outperforms Donut by 9 points on DocVQA; extreme aspect ratios challenge fixed-ratio image scaling on InfographicVQA.
- Common trends: Pix2Struct outperforms Donut on all tasks, and scaling from Base to Large improves every task despite Base receiving 3× more training iterations.The authors identify further scaling as a promising direction.
5. Analysis
The ablations show that screenshot parsing is the most important pretraining component, while variable-resolution inputs preserve image structure and use the patch budget efficiently.
- Removing screenshot parsing causes the largest performance drop, reducing pretraining effectively to reading linear text.
- Ablating warmup and masking has nearly equivalent effects on DocVQA and Widget Captioning, while warmup matters slightly more for TextCaps.
- The pretraining results indicate that visually-situated language understanding involves recognizing text, understanding language, and incorporating visual context.
- Variable-resolution inputs maintain the original aspect ratio while maximizing use of the sequence-length budget.Padded inputs waste effective resolution, whereas stretched inputs distort the image.
6. Discussion
The discussion frames Pix2Struct as a simple general-purpose pixel-only approach while identifying resolution, web-data scope, harmful content, and domain-specific tooling as important boundaries.
- High-resolution inputs remain a bottleneck because they often require longer sequence lengths, despite progress toward closing the gap with OCR pipelines.
- The pretraining corpus uses C4, which is significantly smaller and narrower than corpora used to train today’s largest language models.
- Web-based multimodal pretraining requires care because models may also be sensitive to harmful web content.
- OCR pipelines or metadata can be appropriate or necessary in certain domains, even though Pix2Struct focuses on general pixel-only models.
7. Related Work
Related work is organized by domain-specific visually-situated language systems, markup-structure models, and datasets, contrasting specialized intermediate structures with Pix2Struct’s broader scope.
- Document understanding: Document-understanding systems commonly combine external OCR with models that integrate images and OCR annotations.
- UI understanding: UI models often benefit from view hierarchies and element annotations, although some approaches use image-only inputs.
- Natural image understanding: Natural-image work includes image-to-text architectures for core vision tasks and objectives for images containing short text segments.
- Illustrations: Illustration models have lacked full large-scale pretraining, while Pix2Struct outperforms current state-of-the-art models without intermediate structures.
- Models learning from markup structure: MarkupLM and Webformer encode HTML, while HTLM and CM3 generate simplified HTML for zero-shot prompting with text and natural images.
- Datasets: The paper’s dataset selection represents diverse challenges but is not intended to be exhaustive.
A. Resolution in visually-situated language understanding tasks
Resolution strongly affects visually-situated language understanding: variable-resolution processing avoids distortion, while high-resolution inputs remain important for DocVQA and incur measurable inference costs.
- Variable-resolution inputs prevent aspect-ratio distortion by extracting the maximum number of patches that fit within the sequence length.
- Performance on DocVQA improves with higher resolution for both Pix2Struct and Donut, with diminishing returns after about 1M pixels.Pix2Struct uses 4096 patches of 16×16 pixels at this scale, while fixed-resolution models use 1024×1024 images.
- At full resolution, Pix2Struct-Base processes 62 documents per second and Pix2Struct-Large processes 20 documents per second.
- Pix2Struct achieves state-of-the-art results on 6 out of 9 benchmarks spanning 4 domains among single-task, single-model methods.
B. Full Results
Table 3 compares single-model, single-task pipeline and pixel-only baselines on standard splits, while noting that some task-specific state-of-the-art methods use domain-specific inputs.
- Table 3 reports full results for pipeline and pixel-only methods using single-model, single-task baselines trained on standard splits.
- Some per-task state-of-the-art methods use domain-specific inputs such as UI view hierarchies or gold chart data tables.These inputs make comparison and transfer across domains difficult.
C. Finetuning Dataset Details
The benchmark section summarizes the datasets and hyperparameters used for finetuning Pix2Struct across visually-situated language tasks.
- The benchmark includes datasets for visually-situated language understanding, summarized in Table 4.
- Base and large models use input sequence lengths of 4096 and 3072, except the base model on InfographicVQA, which uses 6144.The large variant cannot use a longer sequence length because of TPU/GPU memory constraints.
- Finetuning runs for 5000 or 10000 steps with batch sizes of 32, 128, or 256, using validation-based hyperparameter tuning and early stopping.Table 5 contains task-specific hyperparameter values.
E. Warmup Stage Data
Pix2Struct begins with a warmup stage that renders synthetic text snippets as images and trains the model to parse them, improving pretraining and transfer.
- Warmup images render BooksCorpus text snippets with random colors, Google Fonts 12 fonts, and font sizes from 12pt to 36pt on white backgrounds.Snippets are up to 128 bytes, rendered at 640-pixel width, wrapped when necessary, with height fitted to content.
- The warmup stage teaches the model to read by pairing rendered text images with their parses.Figure 6 shows an example of these rendered text-and-parse pairs.
- Warmup makes pretraining more stable and faster to converge while improving finetuning performance.
F. Pretraining Data
Pix2Struct pretraining uses web screenshots paired with HTML-derived parses, with examples spanning text, images, navigation, articles, and contact information.
- The pretraining corpus contains 80M screenshot–HTML pairs collected from URLs in C4.Screenshots are 1024 pixels wide, with height fitted to page content.
- Ground-truth and predicted parses illustrate how screenshot content is converted into simplified textual representations.
- Additional examples contain eyewear advertisements, fashion posts, article titles, and menu elements represented in the parses.
- The corpus also includes pages describing games, social sharing, comments, coronavirus updates, legal services, and fitness offerings.
- Further examples show business contact information, appointment notices, legal-service navigation, and fitness membership details.
- Predicted parses reproduce most page text and structure while sometimes changing wording, such as replacing “prevent the spread of the virus” with “prevent the spread of infection.”