Source-linked AI summary
Mapping Natural Language Commands to Web Elements
Panupong Pasupat, Tian-Shun Jiang, Evan Zheran Liu, Kelvin Guu, Percy Liang
TL;DR
The paper studies how to ground natural-language commands in web pages whose textual, structural, and spatial properties vary. It introduces a command dataset and three baseline models; experiments find that both neural models outperform retrieval and that functional, relational, and visual reasoning matter.
Problem
Web pages combine varied textual, structural, and spatial properties, making it useful to study how natural-language commands can identify page elements.
Source: §1 · p. 1Method
The authors collected over 50,000 commands, implemented retrieval, element-embedding, and text-alignment models, and analyzed command phenomena and model behavior.
Source: §1 · pp. 1–2Results
Takeaways & Limitations
The task offers a challenging setting for language understanding with potential applications in assistive voice interfaces and browser automation.
Source: §6 · p. 5The models generally struggle with relational, ordinal, and spatial reasoning, particularly when a text box must be identified from a nearby label.
Source: §4.2 · p. 5
Abstract
from arXiv · showhide
The web provides a rich, open-domain environment with textual, structural, and spatial properties. We propose a new task for grounding language in this environment: given a natural language command (e.g., "click on the second article"), choose the correct element on the web page (e.g., a hyperlink or text box). We collected a dataset of over 50,000 commands that capture various phenomena such as functional references (e.g. "find who made this site"), relational reasoning (e.g. "article by john"), and visual reasoning (e.g. "top-most article"). We also implemented and analyzed three baseline models that capture different phenomena present in the dataset.
1 Introduction
The paper frames mapping natural-language commands to web elements as a grounding task in pages with varied textual, structural, and spatial properties. It introduces a dataset and baseline models to examine the reasoning this task requires.
1 Introduction
- Web-page grounding maps natural-language commands to elements such as links, buttons, and form inputs, using text, attributes, DOM structure, and spatial data.Web pages’ diverse content and design make them a rich environment for grounding.Source: §1 · p. 1
- Natural-language element identification is proposed for voice interaction, especially assistive technology for visually impaired users, and browser automation less brittle than CSS or XPath selectors.The paper also describes web pages as open-ended compared with environments that have a fixed, known schema.Source: §1 · p. 1
- The authors collected over 50,000 commands and implemented retrieval, element-embedding, and text-alignment models.The dataset captures multiple command phenomena for analysis.Source: §1 · pp. 1–2
- Commands include functional, relational, and visual references, and experiments identify all three as important for correct element selection.Examples include asking who made a site, identifying an article by its author, or selecting the top-most article.Source: §1 · pp. 1–2
2 Task
The task is to select the web-page element described by a command. The authors build a 51,663-command dataset and analyze the varied ways commands refer to elements.
2 Task
- Given a web page, its elements, and a command, the task is to select the element described by that command.Training and test examples are (web page, command, element) triples.Source: §2 · p. 2
2.1 Dataset
- The dataset contains 51,663 commands from 1,835 web pages, split into training, development, and test sets with no page overlap.Crowdworkers created commands for interactive elements after pages were archived and filtered; they were encouraged to avoid the selected element’s exact wording.Source: §2.1 · p. 2
2.2 Phenomena present in the commands
- Analysis of 200 training examples found varied references beyond exact element text; commands overlap with an average of 5.9 leaf elements.Commands average 4.1 tokens, while collected pages average 1,051 elements.Source: §2.2 · p. 2, §2.1 · p. 2
3 Models
The section presents three ways to match commands to web elements: retrieval over element text and attributes, separate command and element embeddings, and token-level alignment. The embedding model can also incorporate spatial neighbors, while the alignment model extends that approach with direct text interaction.
3.1 Retrieval-based model
- The retrieval model ranks elements using TF-IDF scores between the command and bags of tokens from element text and attributes.Attribute tokens are downweighted by α=3, and ties are resolved by choosing the element earliest in DOM pre-order.Source: §3.1 · pp. 2–3
3.2 Embedding-based model
- The embedding model separately represents commands and element properties, scores normalized embeddings, and trains by maximizing the correct element’s log-likelihood.Element properties are concatenated and projected to match the command embedding’s length.Source: §3.2 · p. 3
- Spatial context adds embeddings of adjacent elements in four directions; the authors also report that an LSTM command encoder brought no improvement.Neighboring labels and author names motivate context because they can help identify a text box or an article.Source: §3.2 · p. 3
3.3 Alignment-based model
- The alignment model builds a token-similarity matrix for command and element text, applies convolution and pooling, and scores the resulting representation.For context, it concatenates representations of the four neighboring elements to the final scoring layer.Source: §3.3 · pp. 3–4
4 Experiments
The neural models outperform retrieval, while ablations and error analysis show that text is the strongest input signal and that reasoning about context remains difficult. The models have complementary strengths in matching strings and interpreting indirect descriptions.
4.1 Ablation analysis
- Text contents are the most important input signal; attributes also help embedding and alignment models, while spatial context yields only a very small alignment gain.The authors suggest context is difficult to incorporate because its types vary and its signals are sparse.Source: §4.1 · p. 4
4 Experiments
- Both neural models significantly outperform the retrieval model.The reported results are shown in Table 2.Source: Table 2 · p. 4, §4 · p. 4
4.2 Error analysis
- The authors analyze predictions from embedding-based and alignment-based models on 100 development examples where at least one model made an error; errors are summarized in Table 3.Source: §4.2 · p. 4
- The alignment model handles partial text matches better because it explicitly models text matching.The embedding model can miss targets whose text only partially matches the command.Source: §4.2 · p. 4
- The alignment model can misidentify targets when many elements share command substrings or a matching substring belongs to the wrong element.For example, “get the program” can refer to a “Download” link rather than a “Microsoft developer program” link.Source: §4.2 · p. 4
- The embedding model handles paraphrases, goal descriptions, and element properties better than alignment, but still misses harder indirect descriptions.One example is interpreting “close this notice” as the button with hidden text “Hide.”Source: §4.2 · pp. 4–5
- Both models largely fail at relational, ordinal, or spatial reasoning, especially when a text box must be identified through a nearby label.Many pages lack semantic annotations for text boxes, so this requires logical or visual context.Source: §4.2 · p. 5
- When several elements satisfy a command, annotations usually select the most prominent one, a target convention the models should arguably learn.“Log in” may match multiple “Sign In” buttons on the same page.Source: §4.2 · p. 5
5 Related work and discussion
Related work spans language-to-action semantic parsing, interface interaction, and reference games. This task extends those settings to web pages with many diverse elements and contextual properties.
Mapping natural language to actions
- Semantic parsing maps language to actions such as API calls, database queries, navigation, and object manipulation.The cited examples include API calls (Bordes and Weston, 2017), database queries, navigation (Janner et al., 2018), and object manipulation (Andreas and Klein, 2015; Guu et al., 2017; Fried et al., 2018).Source: §5 · p. 5
- Prior GUI-language work includes web-table computation and form submission, while instruction-interpretation work uses distant supervision for shallower interactions in a narrower domain.The paper contrasts that work with its own shallower interactions across a much broader domain.Source: §5 · p. 5
- The dataset may also support the reverse task of describing actions on web pages.Prior work explores generating natural-language descriptions of objects (Vinyals et al., 2014).Source: §5 · p. 5
Reference games
- Reference games select an object from an utterance, but prior work focuses on few similar objects; web pages add many objects, diverse properties, and contextual interpretation.The cited reference-game work includes Golland et al. (2010), Smith et al. (2013), Çelikyilmaz et al. (2014), Andreas and Klein (2016), and Yu et al. (2017).Source: §5 · p. 5
Interacting with web pages
- Web interaction alternatives include image-based and natural-language selectors, alongside keystrokes, speech, haptics, and eye gaze.The cited examples include images, natural language, keystrokes, speech, haptics, and eye gaze.Source: §5 · p. 5
6 Conclusion
The paper presents natural-language grounding on open-ended, semi-structured web pages as a new task. Its mix of reference types, element attributes, and contextual demands makes the task challenging and potentially useful in real-world applications.
6 Conclusion
- The paper presents a new task of grounding natural-language commands on open-ended, semi-structured web pages.Source: §6 · p. 5
- The task combines different ways of referring to elements, textual and nontextual attributes, and the need to incorporate context.Source: §6 · p. 5
- This combination makes the task challenging, with potential for real-world applications.Source: §6 · p. 5