Grounding Open-Domain Instructions to Automate Web Support Tasks

Task

• RUSS targets previously unseen web tasks from open-domain instructions, including customer-service operations such as password resets and gift-card redemption.The task is motivated by conversational access to web knowledge and services without domain-specific APIs.Source: §1 · pp. 1–2

Approach

• The system uses a BERT-LSTM pointer semantic parser to map each instruction to ThingTalk, then grounds requested elements to active-webpage IDs for execution.Its runtime supports both user interaction and webpage operations.Source: Figure 1 · p. 1, §1 · p. 2

ThingTalk

• ThingTalk is a typed, expressive DSL designed as a target for natural-language semantic parsing and for synthesizing training data.The DSL provides the intermediate representation for web operations.Source: §1 · p. 2

Dataset

• The RUSS Dataset contains 741 real-world step-by-step instructions with webpage DOMs, ThingTalk, and actions, plus 1.5M synthetic instructions mapped to ThingTalk.The dataset supports evaluation and training-data augmentation.Source: §1 · p. 2

Evaluation

• 76.7% end-to-end accuracy was achieved on the RUSS evaluation dataset, while the semantic parser reached 85% and the grounding model 75%.A user study also found preference for RUSS’s natural-language interface over existing web UIs.Source: §1 · p. 2

Grounding in the Visual and Physical Worlds

• Prior grounding research spans physical environments, images and videos, mobile interfaces, desktop GUIs, and web elements.Examples include robot-human dialogue, visual question answering, Android, Adobe interfaces, and TV web elements.Source: Grounding in the Visual and Physical… · pp. 2–3

Natural Language Digital Interfaces

• Unlike prior web-element grounding, RUSS parses natural-language instructions into formal executable actions that support both web navigation and user interactivity.Pasupat et al. (2018) mapped commands to web elements without actions.Source: Grounding in the Visual and Physical… · p. 3

Dialogue Agents for The Web

• Earlier web dialogue agents commonly use single-use heuristics or programming-by-demonstration, whereas RUSS operates from existing real-world instructions without user demonstration.The authors note that their intermediate representation and grounding model could also improve PBD robustness.Source: Grounding in the Visual and Physical… · p. 3

3 Task and Model

• RUSS models tasks as sequences of six actions, including navigation, text entry, clicking, user interaction, reading results, and requesting information.The task starts from natural-language instructions and a webpage, producing an action sequence executed against uniquely identified webpage elements.Source: §3 · p. 3

3.1 ThingTalk

• ThingTalk is a compositional DSL in which @retrieve queries webpage elements and passes their IDs to actions such as @click, @read, or @enter.Queries can combine multiple retrieval operations and use descriptions, element types, absolute locations, and relative positions.Source: §3.1 · pp. 3–4

3.2 Semantic Parser Model

• The semantic parser uses a BERT encoder, LSTM decoder, and pointer-generator to produce ThingTalk token by token from natural-language instructions.Entity placeholders for URLs, locations, and element types reduce tokenization and copying errors before values are restored in the parsed program.Source: §3.2 · p. 4

3.3 Grounding Model

• The grounding model matches ThingTalk retrieval descriptions to DOM elements by filtering type and location, checking relations, and ranking Sentence-BERT text similarities.The highest-scoring candidate supplies the element ID required by the grounded action.Source: §3.3 · pp. 4–5

3.4 The Run-Time

• The runtime executes grounded programs in automated Chrome sessions, uses Puppeteer for web actions, and supports voice-based asking, saying, and reading.For requested information, RUSS stores user responses under validated dictionary keys and later retrieves values for entry actions.Source: §3.4 · p. 5

4 Datasets

• The paper contributes a real-world RUSS Evaluation Dataset and a synthetic dataset for training semantic parsers.The evaluation dataset contains annotated instructions, webpage DOMs, ThingTalk programs, and grounded actions; the synthetic dataset maps generated instructions to ThingTalk.Source: §4 · p. 5, §4.1 · p. 5

4.1 RUSS Evaluation Dataset

• The evaluation data were collected from line-by-line help instructions, then executed, scraped, and annotated with ThingTalk and grounded actions; missing steps were inserted and multi-action instructions split.Human workers wrote instructions as if teaching another person and did not participate in ThingTalk’s design.Source: §4.1 · p. 5
• The evaluation dataset contains 80 tasks and 741 instructions from 22 online help centers spanning domains including music, email, retail, and software.It is split into 304 development instructions from 30 tasks and 437 test instructions from 50 tasks, and is not used for training.Source: §4.1 · p. 5
• 61.4% of natural-language instructions require retrieving a webpage element, while click, ask, and enter are the three most common actions.Click, enter, and read require webpage elements; the dataset also includes dialogue-oriented actions such as ask.Source: §4.1 · p. 5, Figure 5 · p. 5
• RUSS instructions average 9.6 tokens, compared with 4.1 tokens for crowdsourced PhraseNode web instructions.The evaluation vocabulary contains 684 words, including domain-specific terms, and frequent vocabulary coverage is concentrated in the most common 300 words.Source: §4.1 · p. 5, Evaluation Dataset · p. 6

Evaluation Dataset.

• The dataset supports complex web grounding: 76 of 455 element queries use two @retrieve functions, while 53.7%, 42.7%, and 3.6% of @retrieve functions use one, two, and three descriptors.Webpages contain 689 DOM elements on average, and possible action-element combinations number in the thousands.Source: Evaluation Dataset · pp. 5–6

4.2 Synthetic Dataset

• The synthetic dataset uses typed templates for ThingTalk primitives and combinations, sampled DOM text, URLs, and variable-like phrases to generate 1.5M training instructions.The compositional synthesizer creates roughly 840 templates and a vocabulary of 9305 words.Source: §4.2 · p. 6

5 Evaluation

• 76.7% overall accuracy was achieved on the RUSS Evaluation Dataset despite training entirely with synthetic data.The evaluation comprised three experiments covering parsing, grounding, and end-to-end user performance.Source: §5 · p. 6

5.1 Semantic Parsing Accuracy

• 87.0% semantic-parsing accuracy was obtained, including 81.4% for instructions involving web elements and 94.6% for other instructions.The metric uses exact token-by-token matching against gold annotations.Source: §5.1 · pp. 6–7

5.2 Grounding Evaluation

• RUSS achieved 63.6% grounding accuracy, while an end-to-end baseline performed 12.6% worse and PhraseNode reached 46.5%.The modular approach combines correct parsing with web-element grounding rather than mapping instructions directly to elements.Source: §5.2 · p. 7

5.3 Analysis

• RUSS outperformed PhraseNode across reasoning types, although spatial instructions involving image features were not supported.Additional constraints in instructions can reduce the candidate element set, while ThingTalk remains expressive enough for many open-domain web instructions.Source: §5.3 · pp. 7–8, Table 5 · p. 7

5.4 User Study

• 98% of tasks were completed with RUSS versus 85% on the web, and 69% of users preferred RUSS over online help pages.Completion times were generally similar, but RUSS was significantly faster on one unfamiliar complex task; users also cited ease of use, efficiency, and speed.Source: §5.4 · p. 8, Figure 7 · p. 8, Figure 8 · p. 9
• The study was small-scale, while runtime failures could arise from limited conversational retries or webpages taking more than 60 seconds to load.Other limitations include six instructions with attributes poorly expressed in ThingTalk and webpage attributes hidden behind site-specific classes.Source: §5.4 · p. 8, §5.3 · p. 8

6 Conclusion

• RUSS achieved 76.7% accuracy on the evaluation dataset and outperformed end-to-end neural models for grounding natural-language web instructions.PhraseNode achieved 46.5% grounding accuracy, indicating substantial room for further improvement toward 100%.Source: §6 · p. 9
• The RUSS Evaluation Dataset provides real instructions paired with executable web actions for studying semantic parsing and web grounding.The dataset supports future work on new formal-language targets and grounding formal representations on webpages.Source: §6 · p. 9

• The study received IRB Exempt status, used informed consent without deception, lasted about 20 minutes, and compensated participants $10.Source: §7 · p. 9
• The dataset used public webpages and non-identifying fake accounts to limit privacy risks when websites required access.Source: §7 · p. 9
• The authors envision web agents improving access for visually impaired, less technologically advanced, or otherwise preoccupied individuals.Source: §7 · p. 9
• Before scaling RUSS, the authors call for research on user behavior, system-error consequences, and information delivery in high-stakes health or finance transactions.Source: §7 · p. 9