PaperlayerView exact revision on arXiv

Source-linked AI summary

A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis

Izzeddin Gur, Hiroki Furuta, Austin Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, Aleksandra Faust

arXiv:2307.12856v4cs.LGcs.AIcs.CL

TL;DR

Real-world web automation remains difficult because websites have open-ended action spaces, long HTML documents, and insufficient HTML-specific modeling. WebAgent combines self-supervised planning and HTML summarization with grounded Python program generation, and reports over 50% higher success on real websites, alongside strong HTML-understanding benchmark results.

  • Problem

    Real-world web automation is limited by open-ended actions, long HTML documents, and insufficient domain-specific knowledge for understanding HTML.

  • Method

    WebAgent combines HTML-T5 for planning and task-relevant HTML summarization with Flan-U-PaLM for grounded executable Python program generation.

  • Results

    WebAgent improves success on real websites by over 50%, while HTML-T5 achieves 18.7% higher success than a prior language model agent on MiniWoB++ and SoTA performance on Mind2Web.

  • Takeaways & Limitations

    The results support modular specialist language models for planning, HTML summarization, and code generation in real-world web automation.

  • Takeaways & Limitations

    The modular approach may introduce additional computational costs and latency, and broader internet generalization requires enough self-experience data.

Abstract

from arXiv · show

Pre-trained large language models (LLMs) have recently achieved better generalization and sample efficiency in autonomous web automation. However, the performance on real-world websites has still suffered from (1) open domainness, (2) limited context length, and (3) lack of inductive bias on HTML. We introduce WebAgent, an LLM-driven agent that learns from self-experience to complete tasks on real websites following natural language instructions. WebAgent plans ahead by decomposing instructions into canonical sub-instructions, summarizes long HTML documents into task-relevant snippets, and acts on websites via Python programs generated from those. We design WebAgent with Flan-U-PaLM, for grounded code generation, and HTML-T5, new pre-trained LLMs for long HTML documents using local and global attention mechanisms and a mixture of long-span denoising objectives, for planning and summarization. We empirically demonstrate that our modular recipe improves the success on real websites by over 50%, and that HTML-T5 is the best model to solve various HTML understanding tasks; achieving 18.7% higher success rate than the prior method on MiniWoB web automation benchmark, and SoTA performance on Mind2Web, an offline task planning evaluation.

1 INTRODUCTION

WebAgent addresses real-world web automation challenges by combining planning, task-relevant HTML summarization, and grounded program synthesis with specialized language models. It improves performance on real websites and HTML-based benchmarks.

  • Challenges: Real-world web automation is harder than simulation because actions are open-ended and HTML documents are much longer and less simplified.Real websites contain dynamic environments, open-ended actions, and substantial task-irrelevant HTML information.
  • WebAgent: WebAgent decomposes natural-language instructions into sub-instructions, summarizes long HTML into task-relevant snippets, and generates executable Python programs.The system combines canonical web actions in a program space and learns from self-experience.
  • WebAgent: HTML-T5 specializes in planning and HTML summarization using local-global attention and long-span denoising pre-training on a large-scale HTML corpus.The corpus is extracted from CommonCrawl, and the model is designed to capture HTML structure.
  • Results: WebAgent improves real-world website success rates by over 50%.The reported improvement is attributed to the modular system combining specialist and generalist language models.
  • Results: HTML-T5 achieves 18.7% higher success than a previous language model agent on MiniWoB++ and achieves SoTA performance on Mind2Web.HTML-T5 also outperforms competitive attention-model and instruction-finetuned baselines on MiniWoB++.

2 RELATED WORKS

Prior web-automation research largely relies on simulated environments with simplified HTML and predefined actions. WebAgent instead combines HTML-specialized understanding with program-based interaction for real-world web tasks.

  • Web Automation: Web automation is a sequential decision-making task in which agents manipulate browsers through computer-action sequences.Examples include form filling and information retrieval.
  • WebAgent: WebAgent uses HTML-T5 for planning and HTML summarization, while prompted Flan-U-PaLM generates executable Python programs.This modular two-stage design targets long HTML documents and real-world web interaction.
  • Web Automation: Simulators differ from real web environments because real HTML pages average about 15 times more tokens and predefined action spaces may harm generalization.Predefined actions can limit generalization to out-of-distribution pages or instructions.
  • Web Automation: MindAct summarizes raw HTML into task-relevant snippets and predicts web actions with another model in a multiple-choice question-answering format.It uses DeBERTa and Flan-T5 and is evaluated on an offline dataset.

3 WEBAGENT

WebAgent combines HTML-specialized planning and summarization with generalist program synthesis to automate tasks on real-world websites. Its design addresses HTML structure, long documents, open-domain actions, and limited supervision through modular processing and self-experience.

  • Architecture: WebAgent uses HTML-T5 for planning and HTML summarization, while Flan-U-PaLM generates executable Python programs from the resulting sub-instructions and snippets.The two-stage architecture iteratively connects domain-specific HTML understanding with grounded browser actions.
  • Workflow: WebAgent repeatedly predicts the next sub-instruction, extracts task-relevant HTML snippets, and synthesizes a Python program until termination or an iteration limit.The workflow conditions each step on the user instruction, prior planning steps, and newly extracted page HTML.
  • HTML-T5: HTML-T5 models hierarchical HTML with local and global attention mechanisms and trains with mixed long-span denoising objectives.Longer spans are intended to capture more meaningful HTML syntax and semantics than short spans that often mask fragments such as </ or id=.
  • Evaluation: 65%, 70%, and 80% success rates are achieved by WebAgent on real-estate, social-media, and map websites, respectively.The table defines success as the percentage of instruction-specified attributes covered and reports WebAgent as outperforming the listed baselines.
  • Self-experience supervision: Self-experience supervision uses sampled instructions, scripted planning, and prompted program synthesis to create demonstrations for fine-tuning planning and summarization.The approach reduces the need to manually annotate every planning, summarization, and program-synthesis step.
  • Open-domain actions: WebAgent addresses open-domain web automation by acting through programs rather than relying on a predefined categorical action space.Programming allows generated actions to specify interactions with real websites whose available elements cannot be exhaustively predefined.

4 EXPERIMENTAL RESULTS

Experiments evaluate WebAgent and HTML-T5 across real-world navigation, offline action prediction, and MiniWoB++ ablations. Results show strong gains from modular planning and summarization, HTML-specific architecture and denoising, while long-horizon planning remains a key failure source.

  • Real-World Web Automation: WebAgent evaluates real-world instruction-following on real-estate, social-media, and map websites using planning, HTML-T5 summarization, and Flan-U-PaLM programming.The evaluation covers domains with different planning horizons and numbers of page transitions.
  • Real-World Web Automation: 65% success and 87.6% score on real-estate, 70% success and 85.8% score on social-media, and 80% success and 93.8% score on map were achieved by WebAgent.These results outperformed single Flan-U-PaLM and systems using only partial language-model modules.
  • Error Analysis: Planning remains the dominant WebAgent failure source in long-horizon settings, with incorrect plans occurring in 70% of real-estate failure episodes.Self-experience reduced programming and summarization errors, but consistent long-horizon prediction without error accumulation remains difficult.
  • Offline Action Prediction: HTML-T5 increases Mind2Web element accuracy by 5-8%, operation F1 by 6-8%, and step success rate by 4-8% across task, website, and domain generalization.The comparison includes baselines using Flan-T5-XL or GPT-4.
  • MiniWoB++: 18.7% higher success than WebN-T5 was achieved by HTML-T5-XL on MiniWoB++, while HTML-T5-XL reached 85.6% success with 347K demonstrations.The 85.6% result exceeded Flan-T5-XXL despite using 3B rather than 11B parameters.
  • Architecture and Objective: Over 18% higher success was obtained from local-global attention than instruction-finetuned dense attention on 56 MiniWoB++ tasks.The architecture is motivated by matching HTML's hierarchical tree structure.
  • Architecture and Objective: HTML-denoising generally improves offline task-planning performance, with longer span lengths outperforming alternative mixtures.The reported comparisons cover real-estate offline planning and MiniWoB.

5 DISCUSSION AND LIMITATION

WebAgent’s modular specialist design improves real-world web automation but introduces computational, data, program-feedback, and evaluation constraints.

  • Modular planning, HTML summarization, and code generation improve web automation but may add computational costs and latency.
  • Open-loop planning with prompted Flan-U-PaLM achieves at most 10 - 30% success, motivating self-experience supervision for planning.
  • Broad internet generalization depends on sufficient data, with larger domain-expert models and demonstrations identified as future directions.
  • Using Flan-U-PaLM with 540B parameters makes incorporating feedback about generated-code errors challenging.
  • Real-world evaluation is costly, creating a need for automated evaluation with minimal human intervention.

6 CONCLUSION

WebAgent combines HTML-T5 for planning and HTML summarization with Flan-U-PaLM for grounded program synthesis, achieving strong real-world and HTML-understanding performance.

  • WebAgent combines HTML-T5 for planning and HTML summarization with Flan-U-PaLM for grounded program synthesis.
  • Around 70-80% success on real websites surpasses a single LLM approach by over 50% through self-experience supervision.
  • HTML-T5 uses local and global attention with long-span denoising objectives to capture hierarchical HTML structures.
  • HTML-T5 achieves the best results on multiple HTML-based benchmarks, including Mind2Web and MiniWoB++.

ETHICS STATEMENT

The paper presents autonomous web agents as potentially useful beyond simulations while emphasizing security, safety, privacy, and evaluation safeguards.

  • Autonomous agents could support sophisticated computer and smartphone assistants that enhance productivity and accessibility.
  • LLM-based web automation risks prompt injection, cybersecurity threats, scams, misuse, and unintended consequences.
  • Real-website experiments were conducted under human supervision, with access rates limited to avoid stressing servers and sites anonymized for safety and privacy.

C IMPLEMENTATION DETAILS OF HTML-T5

HTML-T5 is pretrained on a filtered CommonCrawl HTML corpus using local-global attention and long-span denoising, while its workflow iteratively links planning, summarization, and code generation.

  • HTML-T5 uses local radius r = 127 and transient global-attention block size k = 16.
  • Pretraining masks 0.15 of input tokens with long mean span lengths µ ∈{8, 64}, using 4096-token inputs.
  • The HTML corpus draws from 100 CommonCrawl WARC files and filters documents before extracting subtrees around labeled elements with a for attribute.
  • The preprocessing produces about 3.41M examples for HTML-T5 denoising pretraining.
  • Figure 6 shows HTML-T5 iteratively predicting decomposed sub-instructions and task-relevant snippets before Flan-U-PaLM generates Python programs.

E WEBSRC: STATIC HTML COMPREHENSION

WebAgent is evaluated on WebSRC, a static HTML comprehension benchmark spanning value extraction, comparison, and table-structure understanding. Its modular approach improves over single LLMs overall, but remains weaker on some structural tasks.

  • Benchmark: WebSRC tests HTML comprehension across KV value extraction, entity comparison, and table structure understanding.KV extracts values from attribute keys; Comparison handles multiple entities with shared attributes; Table requires matching headers to row values.
  • Method: WebAgent combines HTML-T5 snippet extraction with Flan-U-PaLM answer prediction using five-shot examples.Predicted snippets are expanded through the HTML hierarchy and shortened when necessary to fit the generalist model’s context budget.
  • Results: WebAgent outperforms its single generalist and specialist LLM components on WebSRC, showing complementary benefits from modular collaboration.The evaluation compares WebAgent with Flan-U-PaLM, HTML-T5, and an oracle-snippet variant, alongside established baselines.
  • Results: WebAgent is better on Comparison tasks but inferior to other baselines on KV and Table tasks.The authors attribute these weaker results to the continuing difficulty generalist LLMs have with structural data such as tables.

H.2 OFFLINE EVALUATION ON TASK PLANNING WITH MODEL SCALING

Offline task-planning experiments compare HTML-T5 with LongT5 across model sizes and related HTML-comprehension evaluations. HTML-denoising improves sub-instruction prediction and HTML understanding while retaining instruction comprehension.

  • Offline task planning: HTML-T5 outperforms LongT5 on offline sub-instruction prediction accuracy across Base, Large, and XL model sizes.The evaluation predicts the next sub-instruction from current HTML, user instructions, and prior sub-instruction history.
  • Offline task planning: HTML-denoising captures HTML’s structural bias without sacrificing the ability to understand natural-language instructions.The result also suggests that the pre-training approach scales consistently to larger model sizes.
  • Description Generation: The Description Generation benchmark evaluates whether models can describe HTML elements as they would appear to users without rendered-website access.The benchmark targets structural HTML comprehension through textual descriptions of elements marked with an accessibility-related attribute.
  • Description Generation: Local-global attention improves the previous best Description Generation performance by over 10%, while HTML-T5 marginally outperforms LongT5.The comparison reports accuracy, BLEU, and ROUGE-1 across LaMDA, T5, LongT5, and HTML-T5.

I FLAN-LONGT5

Flan-LongT5 extends LongT5 with instruction tuning for comparison against HTML-T5’s HTML-focused pre-training. It performs competitively with Flan-T5, though results vary by model size and task type.

  • Instruction tuning: Flan-LongT5 is created by instruction-finetuning pre-trained LongT5 with the Flan dataset.The authors use 100K training iterations and select the best checkpoints.
  • Reasoning tasks: Flan-LongT5 is evaluated on reasoning benchmarks, including GSM8K, StrategyQA, SVAMP, Asdiv, CommonsenseQA, BBH, and MMLU.The evaluation includes few-shot and zero-shot settings and rechecks official Flan-T5 checkpoints.
  • Reasoning tasks: Flan-LongT5-Large scores 36.64 versus Flan-T5-Large at 35.25 on average, while Flan-LongT5-XL scores 39.05 versus Flan-T5-XL at 43.03.The authors report that XL-size training instability might contribute to the lower XL result.
  • Downstream tasks: Flan-LongT5 is also compared with LongT5 on downstream summarization using ROUGE-1, ROUGE-2, and ROUGE-L.The supplied table caption identifies the metrics and comparison, without reporting their numerical values.

J PER-TASK PERFORMANCE ON MINIWOB++

The paper reports per-task MiniWoB++ performance and compares generalist language-model modules for map automation. Larger or publicly accessible models achieve stronger map-task results in the reported comparison.

  • MiniWoB++: MiniWoB++ results are summarized as per-task average success rates across 56 tasks.The table uses baseline performances reported by Furuta et al. and Gur et al.
  • Map automation: Flan-U-PaLM-540B and gpt-3.5-turbo each achieve 80% success and a 93.8% score on the map website.The comparison follows the same 20 instructions for all tested generalist LLMs.
  • Map automation: Flan-PaLM-62B achieves 60% success and an 86.3% score, below Flan-U-PaLM-540B in the map comparison.The evaluated model variants are Flan-PaLM-8B, Flan-PaLM-62B, and Flan-U-PaLM-540B, plus gpt-3.5-turbo.
Loading 2307.12856v4…