Source-linked AI summary
AutoScraper: A Progressive Understanding Web Agent for Web Scraper Generation
Wenhao Huang, Zhouhong Gu, Chenghao Peng, Zhixu Li, Jiaqing Liang, Yanghua Xiao, Liqian Wen, Zulong Chen
TL;DR
Wrappers have limited adaptability to new websites, while language agents have poor reusability in diverse web environments. AutoScraper uses progressive generation and synthesis to generate scrapers, and experiments report that it outperforms baselines across three datasets. The framework is restricted to vertical web information extraction and depends on backbone LLM performance.
Problem
Wrappers struggle to adapt to new website structures, while language agents have limited reusability across diverse web environments.
Source: §1 · p. 1Method
AutoScraper progressively generates scrapers using HTML hierarchy and synthesizes results from multiple pages into a website-specific scraper.
Source: §1 · p. 2, §4.2 · p. 5Results
Across three datasets, AutoScraper consistently outperforms all baselines and achieves state-of-the-art results in zero-shot settings.
Source: §1 · p. 2Takeaways & Limitations
The experiments demonstrate that AutoScraper can outperform state-of-the-art baselines in scraper generation.
Source: §7 · p. 9The framework is restricted to vertical web information extraction and relies on the performance of its backbone LLMs.
Source: Limitation · p. 9
Abstract
from arXiv · showhide
Web scraping is a powerful technique that extracts data from websites, enabling automated data collection, enhancing data analysis capabilities, and minimizing manual data entry efforts. Existing methods, wrappers-based methods suffer from limited adaptability and scalability when faced with a new website, while language agents, empowered by large language models (LLMs), exhibit poor reusability in diverse web environments. In this work, we introduce the paradigm of generating web scrapers with LLMs and propose AutoScraper, a two-stage framework that can handle diverse and changing web environments more efficiently. AutoScraper leverages the hierarchical structure of HTML and similarity across different web pages for generating web scrapers. Besides, we propose a new executability metric for better measuring the performance of web scraper generation tasks. We conduct comprehensive experiments with multiple LLMs and demonstrate the effectiveness of our framework. Resources of this paper can be found at \url{https://github.com/EZ-hwh/AutoScraper}
1 Introduction
Web scraping methods face challenges adapting across websites: wrappers require custom work for new structures, while language agents are costly and less reusable. AutoScraper generates scrapers in two stages and evaluates their executability across multiple pages.
1 Introduction
- Wrappers struggle with new website structures, while language agents incur time and financial costs and have limited reusability across repeated tasks.These shortcomings constrain scalability in dynamic and diverse web environments.Source: §1 · pp. 1–2
- LLM-based scraper generation must address long, hierarchical HTML, scraper reuse across pages with differing content and structure, and evaluation that reflects usability across a website.Traditional extraction metrics focus on individual pages and may misrepresent scraper usability.Source: §1 · p. 2
- AutoScraper combines progressive generation, which uses HTML hierarchy to handle long documents, with synthesis that integrates scrapers from multiple pages into a website-specific scraper.The paper also introduces executability, an evaluation metric measured across multiple pages within a website.Source: §1 · p. 2
- Across three datasets and eight LLMs, AutoScraper consistently outperforms all baselines in zero-shot settings, achieves state-of-the-art results, and can surpass supervised methods.It also demonstrates superior efficiency on large-scale web information extraction tasks.Source: §1 · p. 2
2 Related Work
Prior web-scraping approaches either rely on human-intensive wrappers or use language agents that do not effectively exploit structural similarities across pages. The paper distinguishes these approaches and their scaling challenges.
2 Related Work
- Wrapper-based methods include rules, learned wrappers, heuristics, and neural networks, but often require human annotations, feature design, scoring rules, or verification.This human involvement makes scaling across many different websites difficult.Source: §2 · pp. 2–3
- General language agents use LLM reasoning and reflection, but do not effectively use web structural features or simplify the environment after unsuccessful planning.These limitations constrain subsequent planning optimization.Source: §2 · p. 3
- Current language-agent frameworks focus largely on open-world interactive tasks, while scraping requires accuracy and efficiency suited to information extraction.They also do not effectively exploit HTML similarities across pages to reduce LLM dependence on repetitive operations.Source: §2 · p. 3
3 Preliminaries
The preliminaries define scraper generation as producing an executable extraction sequence for pages on one website, then describe the benchmarks and an evaluation scheme that measures execution outcomes across pages.
3.1 Task Formulation
- Given webpages from one website and a target attribute, scraper generation produces an executable rule or action sequence that extracts the target information from all pages.Source: §3.1 · p. 3
3.2 Datasets
- The benchmark statistics table reports cases, distinct extraction tasks, and total webpages for the web-scraping task benchmarks.Source: Table 1 · p. 3
- The evaluation uses semi-structured information extraction as its testbed and includes SWDE, Extended SWDE, and DS1 benchmarks.SWDE covers 80 websites in eight domains; Extended SWDE provides finer-grained annotations, and DS1 contains annotated pages from 30 real-life websites.Source: §3.2 · p. 3
3.3 Evaluation Metrics
- Executability classifies extraction outcomes across a collection of pages into six mutually exclusive situations, then reports each situation's proportion.The scheme distinguishes Correct, Precision, Recall, Un-executable, Over-estimate, and Else; higher Correct and lower Un-executable ratios are preferred.Source: §3.3 · p. 4
4 AUTOSCRAPER
AutoScraper generates scrapers in two phases: progressive generation uses HTML’s hierarchy to handle long pages, and synthesis combines page-specific scrapers into a website-specific scraper. It models scraping as an action sequence of XPath expressions and uses top-down and step-back traversal to find target information.
4 AUTOSCRAPER
- AutoScraper first progressively generates an action sequence using webpage hierarchy, then synthesizes sequences from multiple pages into a website-specific scraper.Synthesis addresses differences in target locations and page structure that can make a sequence derived from one page less reusable.Source: §4 · p. 5, §4.3 · p. 5
4.1 Modeling
- Scraper generation is modeled as a sequence of XPath expressions from seed webpages; all but the last XPath prune the page, and the last extracts the target value.The sequence is executed in order using a parser.Source: §4.1 · p. 5
4.2 Progressive Generation
- Top-down traversal refines from the DOM root toward the target node, while step-back revises the selection by moving upward to a broader, more reliable node.If execution fails, the process retreats from the failed node to ensure the page includes the target information.Source: §4.2 · p. 5
4.3 Synthesis
- Synthesis generates sequences for randomly selected seed pages, executes them, and selects one that extracts all target information across those pages.It is introduced to improve action-sequence reusability across pages with different target locations and structures.Source: §4.3 · p. 5
5 Experiment
Experiments assess AutoScraper across multiple LLMs and datasets, with the main results showing improved executable-action outcomes over prompting baselines. Ablations examine the synthesis module, while a seed-page study reports gains that diminish as more pages are added.
5.4 Seed Websites
- Figure 3 tracks AutoScraper performance as seed-page count changes; more seed pages increase the correct ratio and decrease the unexecutable ratio, with diminishing gains.The results suggest that adding seed pages can improve performance, but the improvement has an upper limit.Source: Figure 3 · p. 7, §5.4 · p. 7
5.3 Ablation Study
- Without synthesis, AutoScraper still beats the two baseline methods across LLMs, while synthesis improves AutoScraper and other methods.Using more webpages for inference can make generated scrapers more stable and improve generalization.Source: §5.3 · p. 7, Table 3 · p. 7
6 Discussion
The discussion compares AUTOSCRAPER with direct extraction and supervised baselines, analyzes when it becomes more efficient, and identifies remaining extraction failures.
6.1 Comparison with LLM Direct Extraction
- All LLMs except GPT-4-Turbo outperform AUTOSCRAPER in direct extraction, but the gap narrows as LLM capability improves.The authors attribute the contrast to LLMs understanding webpage content better than webpage structure, while expecting stronger LLMs to improve AUTOSCRAPER's extraction performance and stability.Source: §6.1 · p. 7
6.2 Comparison with supervised baselines
- AUTOSCRAPER beats all five supervised baselines on F1, despite being evaluated zero-shot while they were trained on seed websites.Each supervised result in Table 5 uses one seed site.Source: §6.2 · p. 8, Table 5 · p. 8
6.3 Efficiency Analysis
- On SWDE, AUTOSCRAPER's average page-count threshold was 19.5, below the dataset's average number of pages per site.The analysis estimates the threshold using timing measurements from 10 randomly selected websites, repeated three times for AUTOSCRAPER.Source: §6.3 · p. 8
6.4 Error Analysis
- Page-to-page variation in target information and DOM structure can prevent AUTOSCRAPER from reusing the same rules across a website.One job-careerbuilder page placed “Not Available” for the company name on a different DOM node.Source: §6.4 · p. 9
- When target information appears in multiple locations, AUTOSCRAPER may extract only portions rather than capture all information in one action sequence.The examples include restaurant addresses and university contact numbers appearing in locations such as an information bar or title.Source: §6.4 · p. 9
7 Conclusion
The paper introduces scraper generation as a way to improve reusability in language-agent-based frameworks, and presents AUTOSCRAPER as a two-phase approach that outperforms the state-of-the-art baseline in scraper generation.
7 Conclusion
- AUTOSCRAPER combines progressive generation and synthesis to produce more stable, executable action sequences, and its experiments show it outperforming the state-of-the-art scraper-generation baseline.Source: §7 · p. 9