PaperlayerView exact revision on arXiv

Source-linked AI summary

Safe and Scalable Web Agent Learning via Recreated Websites

Hyungjoo Chae, Jungsoo Park, Alan Ritter

arXiv:2603.10505v1cs.CL

TL;DR

Training web agents on real websites is unsafe, difficult to reset, and often lacks reliable task verification. VERIENV recreates websites as executable synthetic environments with verifiable tasks, improving WebArena success rates by up to +9.09 points over base models.

  • Problem

    Effective self-evolving web agents require safe environments and verifiable task construction, which real-world websites and LLM-based evaluation do not reliably provide.

  • Method

    VERIENV uses coding agents to clone real websites into executable synthetic environments and generate tasks with deterministic, programmatically verifiable validators.

  • Results

    On WebArena, VERIENV improved success rates by +6.06 points for Qwen3-4B and +9.09 points for LLaMA-3.2-3B-Instruct over corresponding base models.

  • Takeaways & Limitations

    The results support environment-centric scaling as a practical route to robust web agents.

  • Takeaways & Limitations

    The framework failed to successfully implement 39 of 136 analyzed websites, indicating that end-to-end environment construction remains challenging.

Abstract

from arXiv · show

Training autonomous web agents is fundamentally limited by the environments they learn from: real-world websites are unsafe to explore, hard to reset, and rarely provide verifiable feedback. We propose VeriEnv, a framework that treats language models as environment creators, automatically cloning real-world websites into fully executable, verifiable synthetic environments. By exposing controlled internal access via a Python SDK, VeriEnv enables agents to self-generate tasks with deterministic, programmatically verifiable rewards, eliminating reliance on heuristic or LLM-based judges. This design decouples agent learning from unsafe real-world interaction while enabling scalable self-evolution through environment expansion. Through experiments on web agent benchmarks, we show that agents trained with VeriEnv generalize to unseen websites, achieve site-specific mastery through self-evolving training, and benefit from scaling the number of training environments. Code and resources will be released at https://github.com/kyle8581/VeriEnv upon acceptance.

1. Introduction

VERIENV addresses unsafe exploration and unreliable self-generated rewards by cloning real-world websites into executable synthetic environments with internal access and verifiable task validation. Experiments show that agents trained in this framework generalize to unseen websites and support analyses of verifiability, environment scaling, and coding agents.

  • Introduction: Real-world web exploration can be unsafe or restricted, while poorly specified tasks and unverifiable outcomes make self-evolving learning unstable and inefficient.Agents may interfere with users, violate platform policies, or encounter Cloudflare and CAPTCHAs.
  • Introduction: VERIENV generates tasks with executable validation, enabling safe and reliable self-evolving learning without direct training on real-world websites.The framework is designed to provide safe environments and verifiable task construction together.
  • Introduction: VERIENV automatically clones target websites, including frontend, backend logic, and databases, into fully executable synthetic environments for safe agent training.A coding agent performs the reconstruction, while internal access supports task generation and executable validation programs.
  • Introduction: +6.06 points for Qwen3-4B and +9.09 points for LLaMA-3.2-3B-Instruct on WebArena demonstrate VERIENV's gains over corresponding base models.Experiments also evaluate generalization on Mind2Web-Online and realistic web tasks.
  • Introduction: Systematic analyses examine the importance of verifiability in task construction and reward assignment, along with environment scaling and coding agents' effects on learning.These analyses accompany experiments on WebArena and Mind2Web-Online that assess generalization to unseen websites.

2. Related Work

Prior work emphasizes verifiable rewards, controlled environments, and self-evolving curricula for web and reasoning agents, while coding agents increasingly build web applications end to end. VERIENV combines these directions by cloning real websites into executable, database-verified environments that support deterministic rewards and self-generated training.

  • Agent learning with verifiable reward: Verifiable rewards and controlled settings address sparse, brittle learning signals in long-horizon web interaction and tool-use tasks.This research direction has also improved reasoning and tool use in mathematics and coding.
  • Self-evolving agents: Self-evolving agent research uses exploration, curricula, and automated task construction to reduce reliance on static human supervision, including realistic web-agent benchmarks and online curricula.Mind2Web (Deng et al., 2023), WebVoyager (He et al., 2024), and WebArena (Zhou et al., 2024) support systematic study of end-to-end agents and iterative improvement.
  • Self-evolving agents: VERIENV differs from prior web-agent methods by cloning real sites into executable environments with database-backed verification, enabling valid self-generated tasks and deterministic rewards.Its pipeline uses a coding agent, task and judge generators, and a Python SDK to verify tasks and judges through website and database interaction.
  • Coding agents for web development: Coding agents can develop web applications end to end, but they frequently introduce implementation errors and require iterative debugging with compiler, runtime, language-server, and vision-language feedback.Their capabilities span frontend design, backend implementation, and deployment, supported by file-system access, terminal execution, and external search.

3. Method

VeriEnv uses coding agents to recreate real websites as executable synthetic environments, generate deterministically verifiable tasks, and train agents through reproducible reward feedback. Human evaluation indicates high-quality websites and largely executable, rule-based task judges.

  • Environment Construction: A coding agent reconstructs websites from screenshots into executable synthetic environments while iteratively writing, executing, and refining code that captures application logic and data semantics.The reconstruction process uses local filesystem and terminal access, and the resulting environment includes application code, database state, and a Python SDK for controlled querying and verification.
  • Environment Construction: Each synthetic environment is represented as (C, D, P), combining executable code, database state, and a Python SDK that supports controlled state queries, verification, startup, and resets.The coding agent also creates auxiliary environment-control scripts to facilitate repeated experimentation and agent training.
  • Task Generation: LLMs generate natural-language tasks with Python-SDK validation programs that check task executability and define executable predicates for deterministic judges.This enables scalable task generation without manual annotation while grounding correctness checks in the synthetic environment.
  • Agent Training: Agents solve sampled tasks in synthetic environments, after which SDK validators query database state to produce reproducible rewards for stable learning without heuristic or LLM-based judgments.Verified rewards update the agent through a self-evolving learning loop, supporting scalable training without manual annotations or human supervision.
  • Evaluation: 90.3% functionality, 4.7/5 visual quality, 90% task executability, and 76% judge correctness were achieved in human evaluation of generated websites and tasks.Judge errors most commonly arose when database resets failed to preserve random seeds; interannotator agreement was substantial (mean Cohen’s κ = 0.61).

4. Experiments

VERIENV is evaluated for cross-domain generalization and site-specific mastery using recreated websites, verifiable tasks, and self-evolving agent training. It consistently improves out-of-domain WebArena performance and progressively strengthens in-domain capabilities within cloned environments.

  • Experimental scope: The experiments cover cross-domain generalization on unseen sites and tasks, and site-specific mastery through repeated training within a single recreated website.WebArena-Lite and Mind2Web-Online provide the cross-domain evaluation benchmarks, while cloned WebArena websites support site-specific training.
  • Training setup: Training uses rejection-filtered trajectories that satisfy executable validation criteria, producing supervised data from verifiable task-completion signals.The implementation uses GPT-5.2 for environment construction and trains agents based on Qwen3-4B and LLaMA-3.2-3B-Instruct.
  • Cross-domain generalization: VERIENV improves fully out-of-domain WebArena performance by +6.06 and +9.09 points for Qwen3-4B and LLaMA-3.2-3B-Instruct, respectively.The gains are attributed to self-generated trajectories with verified task completion, which stabilize reasoning and action-token learning across base models.
  • Site-specific mastery: Across three representative website categories, VERIENV agents consistently improve from the base model through later self-evolution phases and outperform PAE with larger, more stable gains.Repeated interaction within a fixed cloned environment strengthens in-domain capabilities without requiring direct interaction with real-world websites.

5. Analyses

VERIENV performance improves as the number of training environments increases, while automated construction failures reveal orchestration challenges and comparisons with PAE highlight the value of unambiguous task verification.

  • Scaling with Training Environments: Performance generally improves as the number of training environments increases across WebArena and Mind2Web-Online, unlike fixed-dataset or non-verifiable baselines.The upward trend indicates that additional environments provide useful learning signals, whereas baseline performance remains relatively stable.
  • Failure Modes: 39 of 136 websites failed implementation, with incomplete system setup and failed task generation among the most common failure modes.These failures indicate that end-to-end orchestration remains a major challenge for coding agents.
  • Task Verification: Compared with PAE, VERIENV avoids ambiguous tasks and false-positive judgments by constructing tasks with a single, well-defined target.PAE tasks can admit multiple plausible answers, allowing a vision-language judge to mark seemingly relevant outcomes successful even when the intended page is not reached.

6. Discussion and Future Directions

VeriEnv can reconstruct many websites, but multimedia-heavy services remain difficult to reproduce faithfully because they require additional delivery infrastructure. Future work will use these executable environments for reinforcement learning with deterministic, reproducible rewards that support more principled study of self-evolving web agents.

  • Discussion: Multimedia-heavy websites such as arXiv and YouTube are challenging to reproduce faithfully because serving PDFs or video streams requires additional infrastructure.
  • Discussion: Lightweight placeholders such as dummy PDFs or sample videos can preserve functional behavior while avoiding the complexity of full media pipelines.
  • Future Directions: VeriEnv’s executable, verifiable judges provide deterministic and reproducible rewards, reducing instability associated with LLM-based or heuristic evaluation.
  • Future Directions: These environments enable reinforcement-learning research on self-evolving web agents that continuously generate tasks and interact with environments.

7. Conclusion

VERIENV trains web agents in recreated websites using executable, verifiable task validators, improving safety and reproducibility without real-service interaction or subjective LLM judges. Experiments show consistent gains over open-source baselines and steady improvement through self-evolving training.

  • Conclusion: VERIENV trains web agents in recreated websites by generating tasks with executable, verifiable validators.The framework avoids interaction with real services and reduces reliance on subjective LLM judges, improving safety and reproducibility.
  • Conclusion: Experiments on WebArena and Mind2Web-Online show consistent gains over open-source baselines.
  • Conclusion: A site-specific setting demonstrates steady improvement through self-evolving training.

8. Impact Statements

VeriEnv aims to make web-agent learning safer, more scalable, and reproducible through recreated websites, executable validators, and sandboxed training. The paper also identifies misuse, intellectual-property, bias, and deployment risks and proposes safeguards.

  • Positive impacts: Executable validators reduce reliance on subjective LLM judging, supporting more reliable benchmarking, ablation studies, and auditable agent-training pipelines.Recreated websites also enable rapid research iteration without repeated interaction with real services, reducing risks such as spamming, policy violations, and accidental data modification.
  • Risks and negative impacts: High-fidelity recreated environments could be misused for large-scale scraping, account abuse, or online-service manipulation, while raising intellectual-property, terms-of-service, bias, and safety concerns.These environments may inadvertently encode biased or unsafe content from source websites.
  • Mitigations: Sandboxed execution with disabled external network access, safety-constrained validators, and exclusion of payments, authentication, and personally identifiable information limit harmful training actions.The framework emphasizes recreated environments rather than direct interaction with real services.
  • Mitigations: The authors recommend permission-based cloning, reduced fidelity for sensitive workflows, safeguarded releases, and explicit safety evaluations before real-world deployment.Suggested safeguards include usage policies, rate limits, safety-focused evaluation, and continued study of transfer from recreated environments to real deployment.

A. Implementation Details of VERIENV · A.1. Agent Architecture and Training Hyperparameters · A.1.1. CODING AGENT AND LLMS FOR IMPLEMENTING VERIENV

VERIENV implementation was evaluated with multiple coding-agent systems and backbone language models, but reliable environment construction was limited by premature agent termination and insufficient multimodal capabilities in open-source models.

  • A.1.1. CODING AGENT AND LLMS FOR IMPLEMENTING VERIENV: Cursor CLI, Claude CLI, and OpenHands were evaluated as coding-agent systems for implementing VERIENV.
  • A.1.1. CODING AGENT AND LLMS FOR IMPLEMENTING VERIENV: Claude CLI and OpenHands frequently terminated implementation prematurely before target websites were fully functional.
  • A.1.1. CODING AGENT AND LLMS FOR IMPLEMENTING VERIENV: These early exits sometimes left critical components, including the Python SDK, missing.
  • A.1.1. CODING AGENT AND LLMS FOR IMPLEMENTING VERIENV: Premature termination made it difficult to construct complete, production-ready synthetic environments reliably.
  • A.1.1. CODING AGENT AND LLMS FOR IMPLEMENTING VERIENV: GPT-5.2 was used throughout the experiments, alongside evaluations of Qwen3-Coder-30B-A3B-Instruct and GLM-4.7-Flash.
  • A.1.1. CODING AGENT AND LLMS FOR IMPLEMENTING VERIENV: The open-source models’ lack of strong multimodal capabilities limited environment construction, including diagnosing and fixing frontend layout inconsistencies.

A.1.2. TRAINING DETAILS AND HYPERPARAMETERS … B.3.1. INPUTS

The appendix specifies reproducible training settings, an automated synthetic-environment construction workflow, cloned-site examples, and a structured annotation protocol for evaluating website quality, task executability, and judge validity.

  • A.1.2. TRAINING DETAILS AND HYPERPARAMETERS; A.2. Synthetic Environment Construction Pipeline; A.3. Task Generation and Validation Implementation: Training uses LLaMA-Factory with a 1 × 10^-5 learning rate, two epochs, 10% linear warmup, 8,000-token sequences, ZeRO-3, gradient accumulation of 2, and two NVIDIA A40 GPUs.The reconstruction, implementation/debugging, and task-generation prompts are provided in Figures 10–12.
  • A.1.2. TRAINING DETAILS AND HYPERPARAMETERS: Website reconstruction times are grouped as fast (<45 minutes), medium (45–90 minutes), and slow (>90 minutes), with most sites in the moderate-time groups and a long tail of complex implementations.Figure 8 reports the distribution of implementation durations across reconstructed websites.
  • A.2. Synthetic Environment Construction Pipeline; A.3. Task Generation and Validation Implementation: VERIENV’s construction pipeline combines screenshot-guided implementation, executable backend and SDK development, Playwright-based bug discovery, structured reports, and iterative debugging to produce auditable synthetic environments.The workflow includes server-reset and authentication fixes plus visual and content refinement, supporting reproducible task generation and self-evolving agent learning.
  • A.4. Cloned Synthetic Website Examples: Representative cloned websites are illustrated with screenshots from randomly sampled sites, including CarMax, CVS, Eventbrite, and Google Finance, mapped to their corresponding pages in Table 6.Table 6 links cloned synthetic website examples to figures covering Parts 1–3.
  • B. Synthetic Website Evaluation Interface; B.1. Annotation Task: The Label Studio interface presents annotators with a synthetic website URL, natural-language task instruction, and machine-checkable judge code, then records website-quality and task-and-judge-validity assessments.Website quality covers functional checks and visual scoring, while task-and-judge validity uses binary judgments of executability and completion evaluation.
  • B.2. (A) Website Quality: Website quality separates functional correctness from visual realism to avoid conflating broken behavior with poor styling.This separation structures the evaluation into functional checks and independent visual-appearance scoring.
  • B.2.1. 1) CORE FUNCTIONAL CHECKS (CHECKLIST): Core functional checks test available signup, login, search, navigation, forms, filters, sorting, and pagination, classifying each feature as working, broken, or not applicable.Annotators test each feature, select one status, and may describe distinct issues.
  • B.2.2. 2) VISUAL / APPEARANCE (LIKERT SCALE); B.3. (B) Task and Judge Validation; B.3.1. INPUTS: Visual appearance receives a 1–5 Likert rating based on distinct issue counts, while task-and-judge validation checks whether instructions are executable and judges correctly reflect completion using task instructions and validation specifications.Ratings range from excellent for 0–2 minor issues to very poor for more than 12 issues or multiple severe failures; judge code may require target substrings and an evaluation type.

B.3.2. BINARY JUDGMENTS · Steps to reproduce · Steps to reproduce

VeriEnv evaluates task executability and judge correctness with binary annotations, while reproduction checks expose visual, content, and responsive-layout mismatches in cloned websites.

  • B.3.2. BINARY JUDGMENTS: Task executability is judged “Yes” when a task is doable with the website’s functionality and matches the instruction, and “No” when it is ambiguous, impossible, or unsupported.
  • B.3.2. BINARY JUDGMENTS: Judge correctness is “Yes” when the validator accepts correct and rejects incorrect completions consistently with the instruction, and “No” for false positives, false negatives, or semantic mismatches.
  • B.3.2. BINARY JUDGMENTS: Example benchmark tasks span information retrieval, shopping, finance, travel, sports, healthcare, and account workflows, with exact, inclusion, fuzzy-match, and numeric judge criteria.
  • B.3.2. BINARY JUDGMENTS: Reproduction checks found cloned homepages visually incomplete, with missing major sections, lower content density, and footer or navigation layouts that differed from their references.
  • Steps to reproduce: The reproduction procedure starts servers with ./start_servers.sh and, for mobile checks, opens http://localhost:12040/ at a viewport such as 390×844.
  • Steps to reproduce: Informational-page checks found shortened clones missing the reference’s long FAQ and educational sections, resulting in substantially shorter pages.
  • Steps to reproduce: Mobile reproduction checks found a simple stacked desktop layout instead of the reference’s mobile header, navigation patterns, icon rail, content density, and section ordering.
Loading 2603.10505v1…