AutoData: A Multi-Agent System for Open Web Data Collection

1 Introduction

• Web-sourced datasets have supported research in natural language processing, computer vision, and recommendation systems.The paper motivates data collection by describing the web as a broad source of real-world signals.Source: §1 · p. 1
• Existing approaches face trade-offs: wrapper-based crawlers require manual rule engineering, LLM-based scrapers incur time and financial overhead, and REST-API harvesting needs expert interpretation.The introduction frames fully automatic collection that balances coverage, accuracy, and efficiency as the central challenge.Source: §1 · p. 2
• AutoData’s research squad turns a natural-language instruction into a development blueprint, which the development squad converts into executable code to collect and validate data.The system uses eight specialized agents and an oriented hypergraph cache for collaboration.Source: §1 · pp. 2–3
• Instruct2DS is presented as the first open-web data-collection benchmark, covering Academic, Finance, and Sport.Existing relevant benchmarks mainly use static, archived web pages.Source: §1 · p. 2
• Experiments across Instruct2DS and three existing benchmarks report AutoData’s effectiveness, efficiency, and applicability; case studies further highlight applicability and plug-and-play adaptability.The experiments compare AutoData with baseline methods.Source: §1 · p. 3

2 Preliminary

• A hyperedge connects multiple nodes and represents interactions beyond pairwise graph edges.The definition contrasts hyperedges with pairwise edges in graphs.Source: §2 · p. 3
• An oriented hyperedge partitions nodes into source and target subsets to represent direction among connected nodes.Source: §2 · p. 3
• The collection problem maps a natural-language instruction and open web sources to a dataset that meets the instruction’s requirements.The stated objective is to do this effectively and efficiently with a fully automated system.Source: §2 · p. 3

3.1 Specialized Agents

• AutoData uses eight specialized agents in research and development squads, coordinated by a central manager for open-web collection.The design follows a workflow of learning collection logic from web sources, then coding and executing a program.Source: §3 · p. 3, §3.1 · p. 3
• Research agents plan tasks, gather web knowledge and tools, and produce a blueprint; development agents implement, debug, execute, and validate the collection pipeline.The manager allocates tasks, tracks progress, and adapts coordination across agent roles.Source: §3.1 · pp. 3–4

3.2 Multi-Agent Collaboration

• Existing multi-agent communication can overload agents with broadcast histories, unstructured messages, and bulky artifacts.These problems can impede information extraction and add inefficiency to complex, multi-stage collection tasks.Source: §3.2 · p. 4
• OHCache combines an oriented message hypergraph, message formatter, and local cache to structure information flow and manage artifacts.Hyperedges direct a message from its sender to a selected recipient group rather than broadcasting it to every agent.Source: §3.2 · p. 4
• The manager orchestrates sequential information flow, while role-specific formatting makes messages machine-interpretable and the local cache shares artifacts by identifier.For example, a web agent can store an HTML file locally and send a concise message so another agent can retrieve it when needed.Source: §3.2 · p. 5

4 Instruct2DS: Benchmark Dataset for Open Web Data Collection

• Instruct2DS benchmarks instruction-driven collection from live web sources across academic, finance, and sports domains.It was introduced to address the lack of a publicly available benchmark tailored to open-web data collection.Source: §4 · p. 5

4.1 Task Definition

• Instruct2DS varies task instructions through symbolic templates and complexity levels to assess collection performance and robustness.Its ground truth is generated from a curated database according to the task instruction; models collect data from open web sources without access to that database.Source: §4.1 · pp. 5–6

4.2 Data Collection

• The academic database was collected from official conference sites and cross-validated for completeness, quality, and integrity.Researchers also separately coded and cross-validated the algorithm that constructs ground-truth datasets for valid instructions.Source: §4.2 · p. 6

4.3 Comparison with Existing Works

• Unlike prior static-page benchmarks, Instruct2DS tests live sources, REST API collection, and symbolic information extraction rather than only fixed fields.Its curated database is also presented as a resource for applications such as academic recommendations and finance or sports prediction.Source: §4.3 · pp. 6–7
• Tables 1 and 2 report baseline performance comparisons on Instruct2DS and on SWDE and EXTENDED SWDE, respectively.The supplied table captions identify their scopes but do not provide comparison values.Source: Table 1 · p. 7, Table 2 · p. 7

5.1 Experiments over Dataset Instruct2DS

• AutoData surpasses all baselines in every Instruct2DS domain while requiring less implementation time and lower expenses.The comparison includes general multi-agent systems and human-effort methods.Source: §5.1 · p. 7

5.2 Experiments over Coding Benchmark Dataset

• On HUMANEVAL, AutoData with GPT-4 is comparable to MetaGPT, outperforms GPT-4, and with GPT-4o incrementally improves over GPT-4o.The authors note AutoData was not specifically designed for code generation.Source: Table 3 · p. 7, §5.2 · p. 7

5.3 Experiments over IE Benchmark Datasets

• AutoData outperforms baseline methods across most evaluation criteria on SWDE and EXTENDED SWDE.Manus is competitive on many metrics but leaves room for improvement on these datasets.Source: §5.3 · p. 8

5.4 Ablation Study

• Removing either agent squad lowers performance, although each ablation costs less than the original system.The ablations retain the manager, web agent, and engineering agent.Source: Figure 3 · p. 8, §5.4 · p. 8
• Removing OHCache reduces performance and increases cost; removing its formatter lowers performance, while removing local cache yields marginal performance changes and substantial cost reduction.The formatter ablation uses plain natural-language communication; without local cache, artifacts are embedded in communication.Source: Figure 3 · p. 8, §5.4 · p. 8

5.5 Case Study

• For picture-book collection, AutoData outperforms Manus across all human-evaluation aspects and costs $0.91, half as much.The task uses a children’s picture-book database searchable by topics, concepts, and skills.Source: Table 4 · p. 8, §5.5 · pp. 8–9
• For survey-paper reference collection, AutoData surpasses Manus across all evaluation metrics and incurs lower overall cost.The case study crawls BibTeX references from five MAS-field surveys.Source: Table 5 · p. 9, §5.5 · p. 9

• General-purpose web agents do not address the specific demands of open-web data collection, including token-intensive operations, and can show performance gaps and redundant LLM calls.The related-work discussion describes web sources as current, domain-specific, and diverse.Source: §6 · p. 9
• Traditional web extraction methods depend on annotated or weakly supervised labels, limiting automatic understanding of diverse website layouts.These methods commonly formulate extraction as span-based tasks and fine-tune pretrained language models.Source: §6 · p. 9
• Generative LLM extraction reads every webpage, which the authors describe as expensive and inapplicable to large-scale collection; AutoData instead extracts knowledge to build collection programs.Development agents then build scripts for scalable open-web collection.Source: §6 · p. 9

• AutoData organizes eight specialized agents into research and development squads for fully automatic open-web data collection.The paper reports experiments on Instruct2DS and three existing benchmark datasets.Source: §7 · p. 9
• OHCache combines an oriented hypergraph for message sharing, a profile-based message formatter, and local on-demand artifact storage.Source: §7 · p. 9
• Experiments on Instruct2DS and three existing benchmarks demonstrate effectiveness, efficiency, and applicability compared with state-of-the-art baselines.Source: §7 · p. 9

A.1 Overview

• Instruct2DS uses manually collected, researcher-cross-validated data to construct ground truth from instructions, and selects static sources to support reproducibility.The dataset is designed to assess complex, multi-step collection tasks; the models receive instructions but not the underlying benchmark data.Source: §A.1 · p. 15

A.2 Academic

• The academic domain covers papers from nine conferences across machine learning, NLP, and computer vision, with three instruction templates for conference, year, track, and year-range requirements.The ground-truth construction algorithm selects records that meet the instruction’s specified requirements.Source: §A.2 · p. 16

A.3 Finance

• The finance domain contains daily S&P 500 constituent data from 2015–2024, including OHLCV, corporate-action-adjusted fields, and cash dividends.Instructions can specify stocks, years, date ranges, or dates.Source: §A.3 · p. 16

A.4 Sports

• The sports domain includes regular-season NBA team and player statistics from 1946–2024 and MLB team hitting and pitching statistics from 1876–2024.The NBA data comes from its official website; the MLB data comes from its official website as well.Source: §A.4 · pp. 16, 18

B Benchmark Datasets

• SWDE covers 80 websites in 8 domains and 124,291 webpages; Extended SWDE adds relation annotations for 21 websites across 3 domains.Extended SWDE labels binary relations involving the main entity, including object values and predicate strings.Source: B Benchmark Datasets · p. 18
• MLB examples provide separate hitting and pitching data structures and instruction templates; pitching fields include team, season, outcomes, and pitching statistics.The pitching schema defines fields including ERA, innings pitched, hits allowed, and strikeouts.Source: B Benchmark Datasets · p. 19, Figure 9 · p. 19, Figure 10 · p. 19
• HumanEval comprises 164 hand-crafted problems for evaluating code generation capabilities of large language models.It assesses whether models can understand, reason about, and generate correct code solutions.Source: B Benchmark Datasets · p. 20

C Baseline Methods

• Baselines include human-written collection scripts, programming assistants, web crawlers, and four multi-agent systems.Programming assistants receive dataset instructions plus source or collection-method guidelines; Manus is evaluated on a representative task subset because of its high cost.Source: C Baseline Methods · p. 20

D.1 Environments

• Experiments use Linux servers with four Nvidia A40 GPUs and GPT-4o as the default LLM backbone.The implementation uses PyTorch 2.4.0, CUDA 12.1, and Python 3.11.5.Source: §D.1 · p. 21

D.2 Evaluation Metrics

• Collection tasks are evaluated with F1, precision, and recall, while reported time measures human effort from instruction release to data-collection start, excluding program execution.SWDE and EXTENDED SWDE also use correct, execution precision, and execution recall metrics.Source: §D.2 · p. 21

D.3 Human Evaluation Setup for Case Study

• Picture-book samples are assessed for completeness, accuracy, and uniqueness as binary tasks, with three researchers rating independently and majority vote determining final labels.The human evaluation randomly draws 500 collected samples.Source: §D.3 · p. 21

D.4 Experiment Setup for Dataset Instruct2DS

• Instruct2DS evaluation samples 234 valid tasks: 81 academic, 81 stock-market, and 72 sports collection tasks.A valid task yields non-empty results containing the data samples specified in its instruction.Source: §D.4 · p. 21

D.5 Experiment Setup for Benchmark Datasets SWDE and EXTENDED SWDE

• SWDE and EXTENDED SWDE experiments follow AutoScraper’s setup, using three seed webpages and the remaining pages for testing, with GPT4-Turbo used for fair comparison except for Manus.The experiments reproduce AutoScraper results from its source code.Source: §D.5 · p. 22

D.6 Experiment Setup for Benchmark Dataset HUMANEVAL

• HUMANEVAL experiments follow MetaGPT settings, with AutoData’s manager coordinating research and development squads for code generation and prompts adjusted to Python format requirements.Source: §D.6 · p. 22

D.7 Experiment Setup for Case Study Paper Collection from Survey

• For survey-paper collection, two researchers independently extract and cross-validate cited BibTeX entries to form ground truth for evaluating AutoData and Manus.Evaluation reuses precision, recall, and F1 metrics.Source: §D.7 · p. 22

E.1 Performance of AutoData with different LLMs

• GPT-4o offers the best overall performance among tested LLMs at relatively high expense; GPT-4o mini is faster and lower-cost on simple STOCK and SPORT tasks.DeepSeek R1 70B has the lowest performance among the LLMs but remains comparable to Manus.Source: §E.1 · p. 22, Table 6 · p. 22

E.2 Error Analysis

• Conference pages may encode track information inside a volume field, causing AutoData to miss or mis-parse attributes; the authors avoid hard-coded heuristics for a general-purpose system.They note that more task-specific prompts could help.Source: §E.2 · pp. 22–23
• For MLB statistics, AutoData sometimes chooses error-prone HTML crawling over available third-party APIs; better decision policies or user hints may help.Source: §E.2 · p. 23

F Additional Discussion for Children Picture Book Collection

• A picture-book dataset can support educators’ theme-based book selection and help content creators identify gaps in children’s literature.Source: F Additional Discussion for Children… · p. 23

G Limitations

• The system depends on proprietary LLMs and focuses on text collection from publicly accessible HTML pages or REST APIs; client-side-only, login-walled, CAPTCHA, and anti-bot sites remain largely out of reach.Multimodal assets are unsupported, and legal retrievability is assumed; headless browsing and credential management are left for future work.Source: G Limitations · p. 23

H Broader Impacts

• Automation may threaten web-scraping and data-annotation jobs, while uncontrolled harvesting risks legal liability and misuse for disinformation or deepfake-related data.The authors say re-skilling programs will be important if adoption scales.Source: H Broader Impacts · p. 23
• Planned safeguards include license checks, PII redaction, rate limits, and provenance logs, but safe deployment at scale requires continuous refinement.The authors frame AutoData as a research platform and default to sharing task descriptions and checksums rather than raw dumps.Source: H Broader Impacts · p. 24