PaperlayerView exact revision on arXiv

Source-linked AI summary

WebAnything: From the Wild Web to Scalable Agent Environments

Tianci Xie, Zikun Cai, Chaoyun Zhang, Pinxue Guo, Bingxuan Li, Rui Zheng, Hang Yan

arXiv:2610.37786v1cs.AI

TL;DR

Web agents need stable, controllable environments, and the paper investigates how to transform specified real websites into source-grounded, executable, reusable environments without site-by-site reconstruction. WebAnything reconstructs and verifies local environments from browser evidence; evaluations report fidelity gains over WebHarbor and improved downstream benchmark performance after fine-tuning on interaction trajectories.

  • Problem

    The paper investigates how to transform specified real websites at scale into source-grounded, executable, reusable Web agent environments without manual site-by-site reconstruction.

    Source: §1 · p. 1

  • Method

    WebAnything reconstructs executable local environments from browser-observable evidence and independently verifies frozen candidates to guide local repair or targeted source recapture.

    Source: §1 · p. 2, §3.3 · p. 4

  • Results

    12 wins, two ties, and no losses in a blind overall-fidelity audit, with higher automated fidelity metrics than WebHarbor; fine-tuning on interaction trajectories improves performance on REAL, Odysseys-Easy, and WebArena-Verified Overall.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The environments support repeatable interaction and task evaluation, and the paper reports their use for collecting trajectories that improve downstream Web agent performance.

    Source: §1 · p. 2, §5 · p. 8

    The system reconstructs and verifies source states at particular times, without systematic evaluation of mirror timeliness or long-term maintenance under continuing website changes.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Web agents require diverse, executable, and resettable website environments for reproducible training and evaluation. Human-assisted mirrors demand per-site engineering, while generative construction can diverge from specified real websites. Capture-and-replay preserves observations, with interaction coverage constrained by captured content and replay capabilities. We present WebAnything, a fully automated framework that reconstructs source-grounded, reusable Web environments from real URLs without per-site manual implementation or repair after initial setup and authorization. An Explorer-Builder collects browser-observable structures, resources, and interaction outcomes; an independent verifier checks frozen candidates and guides local repair or targeted source recapture. We construct 244 website artifacts across 14 application domains and 53 categories, supporting 12,984 executed and accepted tasks. Compared with human-assisted WebHarbor, WebAnything achieves higher aggregate scores on three fidelity metrics, with 12 wins, two ties, and no losses in a blind audit. Fine-tuning Qwen3.5-9B on 5,856 interaction trajectories improves REAL success by 9.9 percentage points, Odysseys-Easy Rubric Avg and Task Avg by 6.25 and 6.32 percentage points, and WebArena-Verified Overall success by 13.33 percentage points under the fixed harness. Code: https://github.com/china-qijizhifeng/Web-Anything

1 Introduction

Web agents need stable, controllable environments, while reconstructing specified real websites at scale raises source-grounding and verification challenges. WebAnything automates reconstruction and verification, producing a large environment and task pool and outperforming WebHarbor in a blind fidelity audit.

1 Introduction

  • The central challenge is transforming real websites into source-grounded, executable, reusable environments without manual site-by-site reconstruction.Existing approaches trade off automation, source fidelity, and interaction coverage; verification must distinguish implementation defects from insufficient source observation.Source: §1 · p. 1
  • WebAnything reconstructs executable local environments from browser-observable structures, resources, and behaviors, then independently verifies frozen candidates and directs local repair or targeted recapture.Fixed initial states, reset mechanisms, and task evaluators support repeated interaction and trajectory collection.Source: §1 · p. 2
  • 12 wins, two ties, and no losses in a blind overall-fidelity audit, alongside higher automated fidelity metrics than WebHarbor.The evaluation compares five matched WebHarbor sites and 14 historical states.Source: §1 · p. 2
  • 244 website artifacts across 14 domains support 12,984 executed and accepted tasks.The resource pool spans 53 categories.Source: §1 · p. 2

2 Related Work

Related work spans real-site task and execution benchmarks, generated or replayed website environments, and feedback or interaction-data systems. WebAnything connects real-source diversity with repeatable execution and uses a unified task and evaluator pipeline to assess utility on external benchmarks.

2 Related Work

  • Real-site benchmarks study tasks and interactions, while controlled-execution platforms emphasize state initialization and enterprise or computer-environment evaluation.These lines of work include Mind2Web, WebVoyager, WebArena, WorkArena, BrowserGym, and OSWorld.Source: §2 · pp. 2–3
  • Website-generation methods scale executable sites from specifications or references, while replay preserves captured content and Weblica combines replay with source-grounded synthesis.WebHarbor instead constructs mirrors through human–Coding Agent collaboration; WebAnything grounds refinement in URL-specific evidence.Source: §2 · p. 3
  • Feedback systems revise outputs or agents, while interaction-data efforts expand GUI demonstrations; WebAnything supplies executable environments and collects data through a fixed harness and unified task/evaluator pipeline.Its utility is measured on external benchmarks.Source: §2 · p. 3

3 Methodology

WebAnything reconstructs local executable environments from browser evidence and uses independent verification to direct evidence-aware refinement. Released environments include repeatable states and interfaces for task execution and trajectory collection.

3 Methodology

  • The methodology couples an Explorer–Builder that uses source evidence for reconstruction with independent verification that directs implementation repair or targeted recapture.Candidates are frozen and reverified before release within their declared scope.Source: §3 · p. 3, Figure 2 · p. 4, §3.3 · p. 5

3.1 Problem Formulation

  • The problem is to preserve observed presentation, interactions, and state changes within a site contract, using browser observations rather than private backend code.The evidence records page structure and appearance, resources and dependencies, and before-and-after action observations.Source: §3.1 · p. 3

3.2 Source-Grounded Reconstruction

  • The Builder associates evidence with each visited state and action, using structural, resource, and behavioral observations to constrain local components, routes, and state relationships.When observations do not establish an outcome, the system seeks further evidence rather than treating inferred behavior as observed; construction can use Full Coverage or task-prioritized Just-in-Time Coverage.Source: §3.2 · pp. 3–4

3.3 Independent Verification and Evidence-Aware Refinement

  • A separate verifier tests a frozen candidate against source evidence and the site contract, then routes supported implementation errors to local repair and unresolved evidence gaps to targeted recapture.Updated candidates are frozen again and affected checks and applicable regression tests are repeated before release.Source: §3.3 · pp. 4–5

3.4 Environment and Verifiable Task Supply

  • Released environments package local implementations, initial data, runtime configuration, and reset interfaces, with fixed starting conditions supporting repeatable tasks and accepted-trajectory collection.Task generation, acceptance, trajectory collection, and training export remain separate stages and counting units.Source: §3.4 · p. 5

4 Experiments and Results

The experiments assess construction scale, environment quality, and downstream utility. WEBANYTHING constructs a large pool of executable environments, compares favorably on source-state fidelity, and supports downstream gains, while interaction coverage varies across tested subsets.

4.1 Experimental Setup

  • The evaluation uses frozen artifacts, matched historical references, blind human judgments, separate offline interaction tests, and external benchmark evaluations.SFT uses 5,856 earlier interaction trajectories; the newer task collection is not used in that experiment.Source: §4.1 · p. 5

4.2 RQ1: Construction Scale and Efficiency

  • The pipeline constructs 244 artifacts across 14 domains and 53 categories; among 85 historical sites, no per-site manual implementation, debugging, or repair was recorded after setup and authorization.Recorded model-call cost for 84 sites totals $220.62, a lower bound excluding unpriced calls and system overhead.Source: §4.2 · p. 5, Figure 3 · p. 6

4.3 RQ2: Environment Quality and Independent Verification

  • Independent verification found evidence-supported discrepancies in 39 historical report pairs across 22 sites and confirmed 18 resolution records across six sites.The 18 records comprise 14 local implementation repairs, three runtime repairs, and one request-lifecycle issue.Source: §4.3 · p. 6

4.4 RQ3: Task Support and Downstream Learning Utility

  • The two disjoint task collections provide 12,984 executed and accepted tasks, with diverse retrieval, search, comparison, and state-operation workflows.In the earlier collection, observed tool-interaction depth averages 10.4 steps, and 34.7% of executions exceed 10 steps.Source: §4.4 · pp. 6–7

4.3 RQ2: Environment Quality and Independent Verification

  • WEBANYTHING exceeds WebHarbor on all three aggregate fidelity metrics and receives 12 blind-audit wins, two ties, and no losses.Pixel Similarity is 0.9240 vs. 0.8083, SSIM is 0.7904 vs. 0.6260, and Text F1 is 0.8541 vs. 0.2617.Source: §4.3 · p. 6, Table 1a · p. 7
  • Offline interaction tests show lower capability coverage and all-test pass for WEBANYTHING than WebHarbor on their matched subset, while selected pool artifacts pass more VeriEnv-subset tests.WEBANYTHING scores 75% capability coverage and 63.3% all-test pass versus WebHarbor's 80% and 76.7%; on the separate VeriEnv subset, selected artifacts pass 29/33 tests versus 21/33.Source: §4.3 · p. 6, Table 1b · p. 7

4.4 RQ3: Task Support and Downstream Learning Utility

  • SFT improves results on REAL, Odysseys-Easy, and WebArena-Verified Overall, including a 13.33pp Overall gain on WebArena-Verified under the fixed harness.On Odysseys-Easy, Rubric Avg rises by 6.25pp and Task Avg by 6.32pp; on REAL, success rises by 9.9 percentage points.Source: §4.4 · p. 7
  • On 189 jointly valid Odysseys-All tasks, WEBANYTHING leads WebHarbor and VeriEnv on both Rubric Avg and Task Avg.The comparison uses the same backbone, training configuration, and evaluation harness with similar trajectory counts.Source: §4.4 · p. 8, Table 3 · p. 8

5 Conclusion

WEBANYTHING automates the reconstruction of real websites into source-grounded, executable Web agent environments. Its construction and verification pipeline supports a broad artifact and task supply, while downstream SFT improves performance on external benchmarks.

5 Conclusion

  • WEBANYTHING automates source exploration, evidence collection, reconstruction, independent verification, and iterative refinement without per-site manual implementation or debugging in audited runs.Structural, resource, and behavioral evidence guides reconstruction; verification directs discrepancies to local repair or targeted recapture.Source: §5 · p. 8
  • The framework constructs 244 website artifacts across 14 application domains and 53 categories, supporting 12,984 executed and accepted tasks.The paper reports source fidelity, operational validity, verification-driven refinement, and the importance of evaluating functional coverage separately from visual similarity.Source: §5 · p. 8
  • SFT on 5,856 interaction trajectories improves Web agent performance on REAL, Odysseys-Easy, and WebArena-Verified.The paper presents the framework as a practical path toward scaling real-Web agent environments with minimal site-specific human engineering.Source: §5 · p. 8

Limitations

The paper identifies limits in model-scale validation, ongoing maintenance evaluation, and authorization for reproducing and distributing real-site content.

Training validation with large-scale models

  • Current fine-tuning experiments primarily cover Qwen3.5-9B, without systematic tests of larger models, other post-training methods, or larger training budgets.The authors propose studying environment scale, data diversity, and model capabilities in larger-scale agent training.Source: Limitations · p. 9

Continuous real-Web updates and environment maintenance

  • Long-term mirror timeliness and maintenance efficiency under continuing source changes have not been systematically evaluated.The authors identify source-change detection, incremental evidence capture, and local updates as future research directions.Source: Limitations · p. 9

Copyright, content authorization, and environment distribution

  • Real-site content, images, fonts, and branding may be protected, so construction, research use, and public release must comply with applicable authorization requirements.Future releases may prioritize the general framework, evaluation protocols, and de-identified data while determining mirror and resource distribution scope by license.Source: Limitations · p. 9

Ethical Considerations

The paper sets authorization, privacy, security, and distribution boundaries for real-site reconstruction, and documents qualifications on historical construction records and evaluation scope.

Ethical Considerations

  • Reconstruction should be limited to authorized, publicly licensed, or research-agreement-covered sites, while avoiding collection of credentials, cookies, personal information, and payment content.Released environments should run in isolated networks and sandboxes; scope, public content, and model capabilities should be restricted for potentially abusive automation.Source: Ethical Considerations · p. 9

A.1 Historical Construction and Cost Records

  • Historical cost records cover 84 of 85 sites and provide a lower bound for recovered, priceable model calls, excluding unpriced calls, browser execution, and other overhead.The recorded total is $220.62; it is not extrapolated to 244 sites or treated as the Verifier's marginal cost under a controlled ablation.Source: §A.1 · p. 12
  • No unified construction success rate is computed because neither the 244-site pool nor the historical 85-site subset has a complete attempt cohort under uniform acceptance criteria.The historical subset's conclusions about intervention and pricing are restricted to that subset and recovered calls.Source: §A.1 · p. 12

A.2 Evidence Scope of Independent Evaluation

  • Independent evaluation keeps historical fidelity eligibility (A) separate from offline interaction validity (B), and reports different artifact subsets separately rather than combining them into a ranking.The four added sites include imported artifacts, whose execution results do not count retroactively toward independent SDK construction success.Source: Table 4 · p. 12, §A.2 · p. 12

A.3 Historical Source-State Fidelity: References and Sensitivity Analysis

Historical and current-source checks reveal limits in page-state comparability and metric interpretation; interaction and presentation diagnostics are reported separately from fidelity results.

A.3 Historical Source-State Fidelity: References and Sensitivity Analysis

  • Excluding Apple and ESPN from the sensitivity analysis yields SSIM of 0.8051 for WEBANYTHING and 0.6885 for WebHarbor; this does not replace the five-site main results.Historical references for Apple and ESPN have rendering defects, and JavaScript rendering conditions differ between source and mirror pages.Source: §A.3 · pp. 12–13
  • Historical Text F1 uses a replay-derived source reference with JavaScript disabled, unlike mirror text from earlier JavaScript-enabled captures, so the rendering asymmetry can affect scores.The paper retains the historical scores and distinguishes them from matched live rendering.Source: §A.3 · p. 13

A.5 Website Stratification and Coverage Statistics

  • The 244-site inventory spans 14 application domains and 53 fine-grained categories, but is not a preregistered random sample with fixed quotas.Usage categories, tested capabilities, and trajectory visitation scopes are counted separately and are not interchangeable.Source: §A.5 · p. 14

A.6 Presentation Diagnostics and Quality Depth of the Resource Pool

  • All 244 startup records have HTTP 200 responses and homepage screenshots, but these do not establish complete functional acceptance.Only 113 sites received additional review; 131 received none.Source: §A.6 · p. 14

A.4 Current-Source Spot Checks

  • Current-source paired similarity uses only pages rendering normally on both sides, while source-unavailable states and mirror runtime failures are reported separately.The selected Coursera and Weather artifacts differ in product scope or location from current sources, and live weather, news, and advertising vary over time.Source: §A.4 · p. 14, Table 5 · p. 15

A.7 Interaction Tests and State Recovery

  • In the five-site WebHarbor subset, WebHarbor has higher coverage and interaction-test completion rates, while WEBANYTHING's Cambridge lacks explicit no-results feedback for nonexistent words.Image-resource errors are reported separately from interaction-test success rates.Source: §A.7 · p. 15

B.7 Evidence Definitions for Verification Decoupling and Feedback Analysis

The analysis distinguishes evidence-supported verification discrepancies from broader status differences, then documents selected repair cases and the acceptance process for executable tasks.

B.7 Evidence Definitions for Verification Decoupling and Feedback Analysis

  • Of 229 construction-verification pairs retaining actual report associations, Builder-pass/Verifier-fail status discrepancies occur in 139; 39 have supporting evidence of concrete check targets and defects.Source: §B.7 · p. 17
  • 18 issue records across six sites jointly establish feedback execution, candidate modification, and semantic reverification; a later overall pass alone does not establish issue closure.The 18 records are not independent bugs or statistical samples, and existing records do not establish closure of original evidence gaps targeted by recapture.Source: §B.7 · p. 17

B.8 Evidence for Verification and Feedback Cases

  • Ctrip records five repair issues followed by corresponding semantic-reverification passes; three initial failures involved read-only filesystem/SQLite runtime conditions.The first two issues are classified as local-implementation repairs, while the three runtime-condition issues are classified as runtime-environment repairs.Source: §B.8 · pp. 17–18

C.1 Interfaces for Downstream Task Generation, Evaluation, and Collection

  • Tasks become end-to-end accepted only after actual agent interaction and outcome checks, not merely because specifications or evaluation materials exist.An independent evaluator checks final answers and goal completion against hidden success conditions using terminal DOM, screenshots, and applicable state evidence.Source: §C.1 · p. 19

C.2 Diversity and Interaction Complexity of Accepted Tasks

The accepted source-task collection covers 6,172 tasks across 74 sites and 14 domains, with automatically assigned overlapping capability labels and varied observed interaction depth. A separate new batch reports 6,812 accepted tasks across 229 sites, with task counts and acceptance denominators distinct from the earlier collection.

C.2 Diversity and Interaction Complexity of Accepted Tasks

  • The post-training source collection reports 6,172 accepted tasks covering 74 site labels, while 5,856 training trajectories, 5,167 unique training task IDs, and 60,517 step-level samples are distinct counting units.Source: §C.2 · p. 19
  • Capability labels are automatically derived, may overlap, and are not human gold-standard annotations, so attribute counts cannot be summed as unique-task totals.The multi-page label is not directly equivalent to the number of unique paths after query parameters are removed.Source: §C.2 · p. 19, Table 8 · p. 19
  • Approximately 34.7% of accepted tasks exceed 10 tool-interaction steps, while 10.4% exceed 20 steps.These counts use all 6,172 tasks, including zero-step tasks; observed tool operations, URL states, and page paths are distinct statistics.Source: §C.2 · pp. 19–20, Figure 6 · p. 20

C.3 New Task Batch and Acceptance Definitions

  • 6,812 tasks across 229 sites completed visual-agent execution and evaluator acceptance; 6,172 earlier source tasks and the new batch are independent collections.The newer batch was not used in the reported 5,856-trajectory SFT experiment.Source: §C.3 · p. 20
  • The new batch reached 6,812/7,320 target-quota completion (93.1%), but these accepted outputs do not measure the success rate of all proposals or execution attempts.Across 244 sites, the median was 30 tasks per site, the range 0–30, and the mean 27.9.Source: §C.3 · p. 20

D Human Blind Audit: Protocol and Results

In a blind audit of 14 historical source states, annotators compared anonymous mirrors against designated source pages. WebAnything won the overall-fidelity comparison on 12 pages, tied on two, and had no losses; discrepancy-severity ratings also favored WebAnything.

D Human Blind Audit: Protocol and Results

  • 12 of 14 pages were overall-fidelity wins for WebAnything, with two ties and no losses against WebHarbor.Three annotators discussed labels after viewing the designated historical reference and two anonymous mirrors in randomized order; no samples were selected by automatic scores.Source: D Human Blind Audit: Protocol and… · pp. 20–21
  • Information-fidelity judgments gave WebAnything 10 wins, three ties, and one loss across the 14 pages.The sole WebHarbor win was for ESPN; ties were included in the denominator.Source: D Human Blind Audit: Protocol and… · p. 21
  • WebAnything had 12/14 equivalent, 2/14 minor, and 0/14 severe discrepancies, versus WebHarbor’s 0/14, 8/14, and 6/14.Mean severity was 0.143 for WebAnything and 1.429 for WebHarbor; the reported discrepancy shares were 0% and 42.9%, respectively.Source: D Human Blind Audit: Protocol and… · p. 21
  • The audit assessed visible differences in 14 designated historical-reference states, not functional correctness for unexecuted operations.Source versions, Apple/ESPN reference-rendering defects, and JavaScript rendering conditions remain applicable limitations.Source: D Human Blind Audit: Protocol and… · p. 21

E.1 RQ3: Protocol for Post-Training Utility

The paper evaluates post-training utility with a fixed browser-agent harness and separately compares data sources on Odysseys-All. Qwen3.5-9B improves on the reported REAL and Odysseys-Easy comparisons, while the data-source comparison favors WebAnything-trained models on two aggregate metrics.

E.1 RQ3: Protocol for Post-Training Utility

  • The fixed hybrid harness combines visual and DOM-assisted observations with semantic and coordinate-based actions, and is shared across data collection and downstream evaluations.The Odysseys-Easy comparison uses the hybrid setting; the WebArena-Verified comparison uses the paper’s fixed harness implementation.Source: §E.1 · pp. 21–22
  • SFT used 5,856 trajectories from historical interaction-collection batches, deduplicating to 5,167 task IDs; only each sample’s final agent turn was supervised.Filtering required task-evaluation success and a mean score of at least 4 across process, completion, and efficiency.Source: §E.1 · p. 22
  • REAL success rose from 22/91 (24.2%) to 31/91 (34.1%), a gain of 9.9 percentage points; Odysseys-Easy Rubric Avg rose from 19.71% to 25.96% (+6.25pp) and Task Avg from 18.86% to 25.18% (+6.32pp).REAL results use 91 tasks with valid results for both models; Odysseys-Easy uses 44 mutually valid tasks.Source: §E.1 · p. 22
  • WebArena-Verified supplementary rescoring raised Retrieve passes from 39/325 (12.00%) to 105/325 (32.31%), a difference of 66 passes (+20.31pp), when JSON-format judgment was ignored.This rescored the same SFT outputs without new agent execution and does not replace the strict comparison.Source: §E.1 · p. 23
  • Task generation has limited coverage of long-horizon, multistage tasks and tasks with strong intermediate-state dependencies.The stated limitation concerns data-construction capabilities; source fidelity is measured separately.Source: §E.1 · p. 23

E.2 Data-Source Comparison on Odysseys-All

  • On the common Odysseys-All subset, WebAnything’s Rubric Avg exceeded WebHarbor and VeriEnv by 3.51 and 4.02 percentage points, and Task Avg by 3.91 and 4.17 percentage points.The common comparison uses 189 tasks under shared training and evaluation settings.Source: §E.2 · pp. 23–24

F Reproducibility Materials and Data Governance

The reproducibility materials preserve inventories, artifacts, screenshots, and evaluation and training provenance with result-specific evidence scopes. Distribution is constrained by licensing, privacy, and authorization, with restricted resources handled separately from shareable anonymized statistics.

F Reproducibility Materials and Data Governance

  • The supporting materials include the site inventory and domain/category mapping, additional-review markers, original screenshots and digests, and historical qualitative comparisons.The atlas retains frozen homepages without filtering presentation or resource anomalies.Source: F Reproducibility Materials and Data… · p. 24
  • Reproduction materials separately associate the resource pool, historical subset, comparison artifacts, and training sources, while preserving models, harnesses, tasks, evaluators, budgets, initial states, adapters, and nonexecution reasons.Each result’s evidence scope is specified separately.Source: F Reproducibility Materials and Data… · p. 24
  • Environment distribution is constrained by content licenses, privacy, and access authorization; released environments contain neither real credentials nor private user content.Payments, posting, deletion, and real-account modification are prohibited by default.Source: F Reproducibility Materials and Data… · p. 24

F.1 Complete Site Inventory

  • The resource inventory contains 244 site artifacts across 14 application domains and 53 fine-grained categories, and inclusion does not imply uniform functional acceptance.The inventory retains artifacts with presentation anomalies.Source: §F.1 · p. 24

Related papers

Loading 2610.37786v1…