Understanding HTML with Large Language Models

1 INTRODUCTION

• The study asks whether LLMs can improve HTML-understanding performance and sample efficiency without custom neural-network architecture design.The motivation is that specialized datasets and architectures are expensive, time-consuming, and require domain-specific expertise.Source: §1 · pp. 1–2
• The three benchmarks cover element-category classification, natural-language description generation, and actions across sequential pages to fulfill a command.The tasks are designed to require understanding both HTML structure and content.Source: §1 · p. 2
• LLMs show HTML understanding across tasks, with up to 192× more sample-efficiency than models trained from scratch and a new supervised-learning SoTA on MiniWoB.Encoder-decoder architectures with bidirectional attention perform best across the board, even without HTML in pretraining.Source: §1 · p. 2
• The paper presents the first work to embed and train an LLM as an autonomous web-navigation agent.The implementation adapts behavior cloning and integrates text generation into a perception-compute-action cycle in a stateful web environment.Source: §1 · pp. 2–3
• The authors open-source model weights for WoB agents and a dataset for description generation to encourage further research.The paper frames this work as connecting language models with autonomous web agents.Source: §1 · p. 3

2 RELATED WORK

• The paper uses MiniWoB's simulated websites for navigation, while its CommonCrawl-derived description-generation dataset is motivated by the need to evaluate across varied real-world websites.The authors identify the lack of a mechanism for broad real-world-site evaluation as a motivation for the dataset.Source: §2 · p. 3
• Earlier web-navigation and related models often use architectures tailored to HTML structure or preprocess HTML to simplify model inputs.The cited work spans web navigation, information retrieval, and question answering.Source: §2 · p. 3
• Prior LLM approaches either preprocess pages into natural language for navigation or pretrain on HTML but evaluate standard NLP tasks rather than HTML-understanding tasks.The paper describes its approach as reversing the latter direction: keeping pretraining unchanged and focusing on transfer to HTML-relevant tasks.Source: §2 · p. 3

3 BRIEF BACKGROUND ON HTML AS SEMI-STRUCTURED TEXT DATA

• HTML organizes web-page structure and content through elements, each of which can carry key-value attributes describing content such as style and human-readable text.When rendered, attributes affect how an element appears and where it is positioned.Source: §3 · p. 3
• In the example, the email input's tag, type, and id attributes identify it as an email field and provide a programmatic identifier.The listed attributes are tag="input", type="email", and id="uName".Source: §3 · p. 3
• Table 1 summarizes tasks, datasets, and models; all models receive raw HTML, with navigation using the entire page and other tasks using salient-element snippets.The snippet inputs are extracted given the salient element.Source: Table 1 · p. 4

4 CANONICAL TASKS FOR HTML UNDERSTANDING

• Autonomous Web Navigation evaluates sequential decision-making across HTML pages to satisfy a natural-language instruction.At each step, the agent clicks or types using an integer element selector, and an episode ends at a terminal state or step limit.Source: §4 · p. 4
• Semantic Classification assigns an HTML element to a role category, such as identifying a login-page email input as a username.The model may use labels, input attributes, and the ordering of other page elements.Source: §4 · p. 4
• Description Generation locates an element's textual description in HTML and outputs it, testing whether the model can infer rendered relationships from raw markup.The description may be far from the element in the HTML plaintext even when it appears alongside that element when rendered.Source: §4 · p. 4

Dataset overview

• The three canonical tasks each use a separate dataset, with description generation introducing an auto-labeled dataset based on CommonCrawl.Source: §5 · pp. 4–5

Autonomous Web Navigation

• MiniWoB supplies 12K demonstrations across 62 website applications, each a sequence of instruction, HTML, and action tuples.Page-local reference numbers serve as element selectors; the model receives the instruction and HTML as text and predicts the action as text.Source: §5 · pp. 4–5

Semantic Classification

• Semantic classification uses 28K labeled examples spanning 66 categories from real-world shopping websites, focused on payment and checkout form-filling.Source: §5 · p. 5

Description Generation

• CommonCrawl description generation uses 85K balanced examples derived from 400K label-linked HTML examples collected from 100 April 2019 WARC files.Each example pairs page HTML with the element whose id matches a label’s for attribute and the label text.Source: §5 · p. 5

HTML representation

• HTML is supplied as raw text, with snippet extraction applied only when pages exceed typical LLM context windows.The approach minimizes HTML tree preprocessing before model input.Source: §6 · p. 5

Snippet Extraction

• For semantic classification and description generation, snippets center on a salient element and include its ancestor subtree, bounded by tree-height and descendant limits.The salient element receives a target attribute; MiniWoB retains full HTML because its pages are typically smaller.Source: §6 · p. 5

HTML un-Parsing

• MiniWoB inputs concatenate action history, instruction, and HTML into a single sequence.Source: §6 · p. 5

Model overview

• The study compares transformer LLMs of different sizes and architectures across the HTML understanding tasks.Source: §7 · p. 6

Encoder-Decoder and Decoder-only Models

• T5 encoder-decoder and LaMDA and PaLM decoder-only models use text inputs and outputs for HTML pages or snippets and task predictions.Categories, natural-language descriptions, and actions are represented as text; actions are encoded as concatenated key-value pairs.Source: §7 · p. 6

Autonomous Web Navigation

• For multi-interaction MiniWoB tasks, training tuples include past actions alongside the instruction, HTML, and target action.The action history is added to help models learn from prior experience when website state changes between interactions.Source: §7 · p. 6

Encoder-only Models

• BERT encoder-only models predict discrete semantic categories through a classification layer, with Mobile-BERT and Albert-XL also evaluated.Mobile-BERT uses inverted bottlenecks; Albert-XL uses parameter sharing and embedding split.Source: §7 · p. 6

8.1 AUTONOMOUS WEB NAVIGATION RESULTS

• WebN-T5-3B improves task success 16% over CC-Net (SL) while using 192 times less data, and LLMs outperform prior supervised models but are surpassed by RL systems.The comparison covers 48 MiniWoB websites shared across models; RL systems require orders of magnitude more online website interaction.Source: §8.1 · pp. 6–7, Figure 2 · p. 7

8.2 SEMANTIC CLASSIFICATION TASK RESULTS

• All WebC-LLMs outperform the same architectures without pretraining; WebC-T5-3B reaches 100% accuracy on password new with 56 training examples, while ambiguous email fields are often mistaken for usernames.The category accuracy distribution is reported for the development dataset.Source: §8.2 · p. 8, Figure 3 · p. 8

8.3 DESCRIPTION GENERATION TASK RESULTS

• Description-generation models score approximately 84% on exact-match accuracy and approximately 91% on BLEU and ROUGE-1, indicating they locate descriptions but do not always reproduce them exactly.The evaluation tests generalization to HTML from unseen URL top-level domains.Source: §8.3 · p. 9, Table 3 · p. 9

8.4.1 PRETRAINING EFFECT: PRETRAINING ON LARGE TEXT CORPORA MATTERS

• Fine-tuned pretrained LLMs outperform models trained on HTML-only data by more than 34.1% on autonomous navigation and 10% to 12.7% on semantic classification.Without pretraining, models are comparable to pretrained models mainly on simple text-matching websites; pretraining is key for harder interactions such as clicking checkboxes.Source: §8.4.1 · p. 9

8.4.2 ARCHITECTURE EFFECT: T5-BASED MODELS PERFORM BEST ACROSS ALL TASKS

• Encoder-decoder T5 models perform better across all three tasks; the authors suggest bidirectional attention may help process HTML from both ends of its linearized sequence.T5 models also perform strongly without HTML pretraining, while PaLM and LaMDA include HTML and other code in their pretraining corpora.Source: §8.4.2 · p. 9, §8.5 · p. 10

8.4.3 MODEL SIZE EFFECT: SIZE (SUB-LINEARLY) MATTERS

• Model size and performance are positively correlated, but gains diminish: performance grows slowly on navigation and plateaus on description generation as models get larger.On semantic classification, the paper describes the relationship to model size as roughly O(log log n).Source: §8.4.3 · p. 10

8.5 DISCUSSION

• The discussion reports up to 4.5x performance improvements from pretraining and says model-size gains must be weighed against training and inference costs.Pretrained LLMs require 200x less expert data for navigation and 30x less for classification; on classification, WebC-PaLM-62B takes days to fine-tune and evaluates at 30 Hz, versus hours and 700 Hz for WebC-T5-large.Source: §8.5 · p. 10
• Context-window length is a major bottleneck: real websites are orders of magnitude larger than MiniWoB pages, and increasing snippet size degrades performance.The authors identify compressing web-content state representations as a possible direction for future methods.Source: §8.5 · p. 10

• Pretraining is critical and can reduce labeled-data requirements, improving sample efficiency by up to 200x.Source: §9 · p. 10
• T5 models with bidirectional attention and an encoder-decoder architecture perform best across the evaluated tasks.Source: §9 · p. 10
• Model size correlates sub-linearly with performance, so size should be considered alongside training and inference performance.Source: §9 · p. 10
• Short context windows limit current LLMs on HTML-understanding tasks and motivate research that incorporates or removes this constraint.Source: §9 · p. 10

• Only 20 descriptions covered 50% of the original 400K dataset, so the authors capped each unique description at 10 examples.Source: §A.1 · p. 13
• To discourage reliance on label elements, the authors replaced their tags with randomly sampled div, span, a, or label tags.The passage notes that for attributes are almost always defined for HTML labels, which could let a model find the label and ignore other content.Source: §A.1 · p. 13
• After removing examples with only one text, which could make description generation trivial, the dataset contained 85K labeled examples.Source: §A.1 · p. 13

A.2.1 SEMANTIC CLASSIFICATION

• 32% of examined errors came from missing snippet information, while 30% involved annotation errors or email/username ambiguity.Other errors were incorrect model predictions; long snippets could also mislead the model with nearby text.Source: §A.2.1 · p. 13, Table 4 · p. 14
• Few-shot PaLM prompting scored 45.6 accuracy, rising to 52.1 with dictionary mapping, 57.9 with token reordering, and 64.2 after input cleanup.Source: §A.2.1 · p. 14

A.6 STRUCTURE DEPENDENCE ABLATION STUDY

• Removing closing tags while preserving element order yielded 45.4% MiniWoB success, 6% lower than before, suggesting partial dependence on DOM topology.Source: §A.6 · p. 15

A.7 TASK-SPECIFIC MODELS

• MarkupLM-base reached 65% development and 66% test accuracy, more than 16% below similar-size WebC-BERT-base results.The authors suggest domain-specific models may suit HTML NLP tasks, while LLMs offer advantages for autonomous navigation.Source: §A.7 · p. 16