PaperlayerView exact revision on arXiv

Source-linked AI summary

Training Needs Trustworthy Worlds: Verified Synthetic Web Environments for Agent Learning

Chenghao Zhang, Canran Xiao, SaiSai Hu, Dan Roth

arXiv:2608.21898v1cs.AI

TL;DR

Synthetic web environments can appear plausible while containing unreachable links, inconsistent backend state, missing controls, or infeasible workflows that mislead agent training. This paper constructs verified environment scaffolds, repairs defects, and combines state-safe interaction with backend-grounded rewards; the resulting environments improve executability and PPO performance.

  • Problem

    Generated web environments may contain globally invalid workflows despite locally plausible pages, creating a mismatch between apparent interaction and executable, state-grounded training.

  • Method

    The framework represents websites as structured scaffolds, verifies and repairs defects, and trains agents with rewards derived from verified backend-state progress.

  • Results

    Verified environments, marker-level validation, and dense backend-grounded rewards jointly improve environment executability and PPO performance.

  • Takeaways & Limitations

    Verified synthetic environments provide a scalable training substrate for compact web agents by grounding supervision in executable, state-consistent, auditable interactions.

  • Takeaways & Limitations

    The event-driven simulator may be less efficient for applications where nearly every interaction changes backend state.

Abstract

from arXiv · show

Web agents promise to automate complex digital workflows, but their training remains limited by synthetic environments that look plausible while hiding broken links, inconsistent states, or infeasible tasks. We address the gap between scalable environment generation and trustworthy agent learning by constructing synthetic web environments that are executable, auditable, and grounded in backend state. Our framework represents each generated website as a structured scaffold of pages, navigation links, database records, state-change markers, and task constraints, then verifies and repairs structural, semantic, consistency, and feasibility defects before policy training. During interaction, ordinary UI transitions are executed deterministically, while persistent backend updates are invoked only through validated state-change markers, enabling dense rewards compiled from verified task-progress predicates. Across 500 synthetic environments spanning six domains, our method reduces task-blocking defects and improves feasible-task rate from 48.6% to 94.8%, while producing stronger PPO policies and improving transfer to WebArena, WebShop, and MiniWoB++ without LLM calls at evaluation time. These results show that verified synthetic environments can serve as a scalable and reliable training substrate for compact web agents, shifting synthetic webagent learning from surface-level plausibility toward executable, state-grounded supervision.

1 Introduction

The paper identifies a mismatch between plausible page-level generation and executable, state-grounded web interaction. It proposes verified environment scaffolds that are checked and repaired before policy learning.

  • Motivation: Synthetic web environments can contain globally invalid workflows despite locally plausible pages.Examples include unreachable links, contradictory database values, missing controls, and success conditions without executable state transitions.
  • Approach: The framework represents websites with pages, navigation, data records, state-changing events, and task constraints before verification and repair.
  • Motivation: Web-agent training depends on executable tasks under consistent backend dynamics, not merely plausible interfaces.
  • Approach: Dense rewards are derived from verified backend-state progress, while ordinary interactions execute without LLM calls at evaluation time.

2 Related Work

Related work established reproducible browser-control tasks, realistic web benchmarks, and LLM-based experience generation. The paper positions verified environment construction alongside these efforts by focusing on executable and reliable synthetic interaction.

  • Web interaction benchmarks and agents: World of Bits and MiniWoB++ established reproducible browser-control tasks for reinforcement learning.
  • Web interaction benchmarks and agents: WebShop, Mind2Web, WebArena, VisualWebArena, WorkArena, and BrowserGym expanded web-task realism, diversity, demonstrations, and evaluation protocols.
  • Synthetic environments and experience generation: LLM-based methods generate environments, tasks, trajectories, or experiences to reduce the cost of interactive agent learning.
  • Reward, verification, and policy learning: Prior reward and verification research addresses sparse terminal feedback through demonstrations, workflow constraints, programmatic matching, and automatic evaluators.

3 Preliminaries

The paper models web interaction as goal-conditioned control over rendered interfaces and persistent backend state. Its scaffold separates static structure from dynamic updates, motivating verification and sparse event-driven state writing.

  • Web interaction model: Web interaction is modeled as a goal-conditioned sequential decision process with DOM-grounded actions and programmatic completion constraints.The agent observes rendered interfaces while completion is evaluated on environment state.
  • Environment scaffold: An environment scaffold E = (P, L, D, T) contains pages, a navigation graph, database records, and tasks with verifiable constraints.
  • Environment scaffold: Separating static scaffolds from dynamic state updates addresses interfaces that are locally plausible but globally invalid.
  • Motivating diagnostics: The diagnostic study covers 500 raw environments from six domains, 6,842 tasks, and 1,800 successful trajectories.
  • Motivating diagnostics: Raw environments average 12.4 defects, while feasibility defects are especially likely to block executable workflows.
  • Event-driven updates: Most interaction steps are deterministic UI operations, with persistent backend writes restricted to sparse marker-triggered events.

4 Method

The method constructs synthetic web environments as verified, structured scaffolds and trains policies through deterministic interaction, validated state updates, and state-grounded rewards.

  • Environment representation: Environments are represented as structured objects containing pages, navigation links, database records, state-change markers, and task constraints.This representation connects backend records, rendered DOM nodes, UI events, state updates, and task progress.
  • Scaffold verification and repair: Generated scaffolds are canonicalized, checked, and repaired for reachability, link integrity, schema validity, bindings, marker consistency, and workflow feasibility before training.Repair proceeds in dependency order and applies structural, semantic, consistency, and feasibility-specific operators.
  • Scaffold verification and repair: Defect reports combine verifier outputs with locations, affected objects, severity, evidence, and confidence, while coordinated rechecks target interacting defect categories.Reports sharing a canonical key are merged before repair prioritization.
  • Event-driven simulation: Ordinary navigation and interface actions execute deterministically, whereas persistent backend writes occur only through validated marker-triggered operations.Accepted updates must satisfy marker preconditions, permitted read-write fields, database schema, and environment invariants; rejected updates preserve the previous state.
  • State-grounded reward and policy learning: Verified task constraints are compiled into intermediate progress predicates that provide dense state-grounded rewards and penalize rejected transitions during PPO training.Predicates check conditions such as required pages, form constraints, target entities, updated attributes, and agreement between rendered views and backend state.

5 Experiments

The experiments evaluate whether verification produces executable environments, whether verified training improves PPO policies, and which pipeline components drive the gains. Across these evaluations, verification improves task feasibility and environment fidelity, while state-grounded dense rewards strengthen policy learning.

  • Experimental design: The study evaluates environment executability, downstream PPO learning, and the contributions of verification, repair, and reward components.Experiments use 500 synthetic environments across six domains, a compact policy with fewer than 10M parameters, and multiple construction and training baselines.
  • Environment quality: 48.6% of raw tasks admit bounded executable traces, whereas the verified method achieves the highest feasible-task rate and the lowest state-violation rate.The verified method also produces the fewest total and blocking defects and lower curation time than self-consistency and AutoGen.
  • Environment quality: Within three repair iterations, defects fall from 12.4 to 4.2 and feasible tasks rise from 48.6% to 90.7%, after which gains saturate.This convergence supports targeted verification and repair instead of repeated full regeneration.
  • Policy learning: Verified environments with state-grounded dense rewards produce the strongest PPO learning curve and best final success rate.Raw environments provide noisy supervision; verification improves terminal-reward learning, while dense rewards remain limited on inconsistent scaffolds.
  • Component ablations: Feasibility verification, marker validation, and dense rewards respectively support executable supervision, backend fidelity, and policy learning.The ablation matrix reports metrics with 95% confidence intervals and shows these components are complementary.

6 Conclusion

The paper frames reliable synthetic web training around executable, state-consistent, and auditable environments. It combines offline scaffold verification with event-driven state updates, while diagnostics characterize defects across 500 raw environments and 6,842 tasks.

  • Conclusion: The conclusion argues that synthetic websites must be executable, state-consistent, and auditable to provide useful supervision for web-agent reinforcement learning.The paper positions environment reliability as the central perspective for training web agents.
  • Conclusion: The framework combines offline scaffold verification with event-driven state updates so compact policies can learn from state-grounded synthetic interactions.The supporting diagnostics motivate offline verification and sparse event-driven runtime updates rather than generative calls at every step.
  • Conclusion: The paper concludes that feasibility verification, marker-level validation, and dense backend-grounded rewards jointly improve environment executability and PPO performance.Future work extends the paradigm to richer multi-site and multimodal web workflows.
  • Diagnostic scope: The diagnostic suite covers 500 raw environments, six domains, and 6,842 generated tasks.Each environment records pages, navigation, database schema, initialized records, interactive elements, and task sets.
  • Defect analysis: The defect framework categorizes detected defects by type, location, affected object, severity, and supporting evidence.The reported categories include symbolic, structural, semantic, cross-page consistency, and task-feasibility defects.

A.1.3 Bounded Trace Analysis

Bounded trace analysis tests whether a task has an executable action sequence from its initial states to a completion constraint, then attributes blocked tasks to the earliest defect category preventing progress.

  • Feasibility analysis: A task is executable when bounded search finds actions reaching its completion constraint within horizon Hmax; otherwise it is marked blocked.The analysis searches transitions from initial states and is used for diagnostics, not policy training.
  • Feasibility analysis: The search expands rendered observations into candidate actions, records transitions, tracks visited states, and returns either a successful trace or blocking evidence.Algorithm 1 performs breadth-wise expansion through the bounded horizon.
  • Blocking attribution: Blocking attribution assigns failure to the earliest defect category that prevents progress along the attempted workflow.Feasibility defects take priority, followed by structural, consistency, and semantic defects according to execution proximity.
  • Blocking attribution: Feasibility defects indicate missing workflow steps or inaccessible controls, structural defects block necessary pages, and consistency or semantic defects invalidate reachable state or content.This priority order distinguishes failures of reachability from failures after a task path exists.
  • Verification and repair: The verification pipeline combines deterministic scaffold checks with specialized structural, semantic, consistency, and feasibility verifiers whose reports are merged, routed, and scheduled for repair.Accepted defects carry evidence, severity, and confidence, while dependencies determine repair order.

A.6 Reward Compilation and PPO Training

The framework compiles verified task-completion predicates into dense rewards and trains a compact PPO policy over DOM-grounded candidate actions, while keeping backend state and verification signals hidden from the policy.

  • Reward compilation: Completion constraints are decomposed into database, session-state, and DOM predicates that provide verified task-progress signals.Examples include visited pages, valid fields, created entities, updated attributes, and rendered backend values.
  • Policy learning: The policy scores and normalizes only DOM-grounded candidate actions extracted from the current observation and task instruction.Unavailable DOM actions are excluded from the policy distribution.
  • Policy learning: PPO collects rollouts in the event-driven simulator and computes rewards without exposing backend state, completion constraints, or progress predicates to the policy.The policy is optimized with a clipped objective and generalized advantage estimation.
  • Verification coordination: Four specialized verification agents detect structural, semantic, cross-page consistency, and task-feasibility defects, coordinating through defect-triggered messages.Their roles cover navigation and layout, content validity, entity consistency, and reachable task goals.
  • Repair and training protocol: Priority-weighted repair scheduling orders accepted defects using dependency-aware scores while preserving shared scaffolds, training settings, and held-out evaluation splits across baselines.The compact policy has fewer than 10M parameters, and hyperparameters are selected only on validation environments.

B.2 Compute Resources

Experiments use shared CPU/GPU infrastructure, offline LLM-based environment verification, and unified DOM-grounded transfer protocols across external web-agent benchmarks.

  • Compute resources: A PPO run for one training condition takes approximately 3.2 GPU-hours on one NVIDIA A10G GPU.The main learning-curve experiments require approximately 48 GPU-hours, while component ablations require approximately 67 GPU-hours.
  • Environment construction: Full verification curates 500 raw synthetic environments at an average of 18 minutes per environment, or approximately 150 CPU-hours excluding parallelization overhead.LLM-based verifiers are used during offline scaffold verification and repair.
  • Transfer protocol: Transfer evaluation covers WebArena-compatible tasks, WebShop, and MiniWoB++ under a shared textual instruction, serialized DOM observation, and DOM-grounded action interface.WebArena uses a DOM-compatible subset that excludes tasks requiring unavailable private credentials, uploads, or visual-only information.
  • Transfer results: Compared with PPO trained on raw synthetic environments, verified training improves success by 6.2 points on WebArena-compatible tasks, 14.2 points on WebShop, and 16.5 points on MiniWoB++.The compact policy uses zero LLM calls at evaluation time and receives no external benchmark fine-tuning.

C.2 Does event-driven simulation reduce cost while preserving state fidelity?

Event-driven simulation reduces per-episode generation cost by invoking backend updates only at marker-triggered events, while validation preserves state fidelity and prevents unsafe state drift.

  • Cost–fidelity comparison: Step-wise LLM simulation reaches 97.1% state fidelity but requires 17.8 LLM calls and 22.4K tokens per episode.It has the highest latency and lowest rollout throughput among the compared simulators.
  • Cost–fidelity comparison: The deterministic-only simulator reaches 71.5% state fidelity, showing that eliminating backend writes entirely is inadequate for stateful web interaction.Its low cost and speed come at the expense of realistic persistent state.
  • Cost–fidelity comparison: 82.6% lower token usage reduces cost from 22.4K to 3.9K while event-driven simulation preserves 96.3% versus 97.1% state fidelity.The event-driven simulator also improves rollout throughput by about 8.7×, from 210 to 1820 episodes per hour.
  • Cost–fidelity comparison: The unconstrained marker simulator uses 2.3 LLM calls and 3.6K tokens per episode but produces 10.7% state violations and 88.2% fidelity.Compared with it, the validated event-driven method improves fidelity by 8.1 points and reduces state violations by more than 3.8×.
  • Design implication: Sparse marker triggering is effective only when combined with state-safe validation, which removes unnecessary per-step generation while constraining persistent updates.The simulator therefore targets scalable PPO training without allowing low-cost rollouts to drift from faithful backend dynamics.

C.3 Which defects most harm policy learning?

Defect frequency does not predict learning harm: feasibility and structural defects can block executable workflows, while state-grounded rewards are designed to track real task completion rather than local activity. Verification should therefore prioritize defects by their effect on executability and reward validity.

  • Feasibility defects produce the largest PPO success-rate drops despite being less frequent than semantic defects.Missing submit actions, inaccessible forms, and unsatisfiable completion constraints can make tasks impossible under any policy.
  • Structural defects are highly damaging when they disconnect required pages or remove navigation paths.
  • Verification should prioritize task-blocking defects rather than minimizing defect counts alone.Rare feasibility defects can corrupt reward signals by making tasks infeasible, while common semantic defects may have modest learning impact.
  • State-grounded dense reward is better aligned with terminal success than surface-level or LLM-judged alternatives.Its calibration curve stays close to the diagonal, whereas competing rewards assign high progress to unsuccessful rollouts.
  • State-grounded reward yields an almost monotonic increase in terminal success across progress deciles and a lower high-progress failure rate.These results support compiling task constraints into verified backend-state predicates to reduce reward hacking.

C.5 What are the failure modes of learned policies?

Failure attribution separates invalid environments and backend violations from errors attributable to the learned policy. Verification and independent audits reduce environment-induced failures while preserving the method’s executability advantage and improving verifier reliability.

  • Failure attribution uses a deterministic diagnostic order covering environment invalidity, state-update violation, reward mismatch, grounding error, planning error, and timeout or exploration.The protocol first checks bounded executability, then backend invariants, reward progress, DOM targeting, unmet subgoals, and residual cases.
  • Raw synthetic environments produce many non-policy failures from environment invalidity and state-update violations.PPO may be penalized for tasks that are not executable or for backend transitions that are not faithfully simulated.
  • After verification and marker validation, most remaining failures are attributable to policy navigation, grounding, and planning errors.These failures are learnable through training, unlike broken scaffolds and invalid backend updates.
  • 94.8% internal feasible-task rate is supported by 92.9% held-out analyzer feasibility and 91.7% author-audited executability.The false-feasible rate drops from 12.4% for raw environments to 2.3% for the proposed method.
  • The coordinated verifier improves recall while keeping precision high and false repairs low on audited defects.Feasibility defects receive higher recall because missed feasibility errors are especially harmful for policy learning.
  • 87.4% feasible-task rate is achieved by the token-matched variant versus 70.4% for Single-LLM verification.This indicates that structured verifier coordination and repair scheduling, rather than a larger LLM budget, account for the improvement.
  • In the e-commerce case study, verification changes failures from infeasible-task failures to policy errors such as wrong product selection, omitted checkout, or timeout.

D Limitations

The event-driven simulator is most efficient when persistent backend updates are sparse, and the work also highlights misuse risks for more capable web agents. Sandboxing, synthetic data, marker schemas, database invariants, and programmatic constraints are described as safeguards.

  • The event-driven simulator may be less efficient when nearly every interaction changes backend state.Its cost reduction depends on invoking constrained state writing only at marker-triggered operations.
  • More capable web agents could be misused for spam, unauthorized automation, credential abuse, or synthetic phishing workflows.
  • The proposed environments use sandboxing and synthetic data rather than real user records.Persistent state changes are constrained by marker schemas, database invariants, and programmatic task constraints.
  • LLMs support raw scaffold generation, semantic verification, and constrained marker-triggered state-delta proposals during synthetic training.
Loading 2608.21898v1…