PaperlayerView exact revision on arXiv

Source-linked AI summary

WebWorld: The Browser as a World Model for Self-Improving Web Code

Jiajun Wu, Jian Yang, Yaxin Du, Wei Zhang, Haowen Wang, Junhang Cheng, Yuxuan Zhang, Tuney Zheng, Xianglong Liu, Ming Zhou

arXiv:2608.30530v1cs.CLcs.SE

TL;DR

VLM self-improvement of web code is limited when the same model proposes and judges repairs using visual plausibility rather than executable behavior. WebWorld lets the VLM interact with the browser as a deterministic world model, admitting only contract-verified transitions into supervision. WebWorld-27B improves over Raw-27B by 5.3 points on HTMLBench-400 and 14.9 points on MiniAppBench-Val, while removing the certificate reduces the matched 9B lift to 0.4 points over Raw.

  • Problem

    VLM-driven web-code improvement conflates proposing and judging, making screenshot-based visual plausibility an inadequate proxy for whether interactive pages work.

  • Method

    WebWorld uses the browser as a deterministic executable world model: a planner compiles VLM critiques into typed interaction contracts, and browser certificates admit only progress-preserving transitions to memory and SFT.

  • Results

    5.3 points on HTMLBench-400 and 14.9 points on MiniAppBench-Val improve WebWorld-27B over Raw-27B, while the matched 9B no-certificate ablation reaches only 0.4 points over Raw-9B.

  • Takeaways & Limitations

    Browser-backed admission makes supervision reliable by separating VLM hypotheses from browser-verified proof and admitting only certified transitions.

  • Takeaways & Limitations

    The current system isolates single-file sandboxed HTML and depends on faithful executable verification; perceptual defects and some dynamic applications still require VLM judgment or deterministic replay.

Abstract

from arXiv · show

VLM-driven self-improvement of web code has a structural flaw: the model that proposes the repair is the model that judges it, and visual plausibility under that judge is a poor proxy for whether the page actually works. What the loop is missing is a counterparty the VLM cannot fool, and the browser already is that counterparty: a deterministic, executable simulator of how an HTML artifact behaves under user actions, and in everything but name a world model for web code. We present WebWorld, the interface that lets a VLM prior interact with this browser-as-world-model autonomously and decides which interactions become supervision. Each round, the VLM emits a critique that the planner compiles into a typed interaction contract; the browser re-executes the candidate and issues an acceptance certificate only when both target progress and preservation of every previously verified capability hold; certified transitions accumulate as a quality ratchet that is the only thing the SFT export ever sees. Under matched training, WebWorld-27B improves Raw-27B by 5.3 points on HTMLBench-400 and 14.9 points on MiniAppBench-Val, and reaches the level of strong frontier systems such as Kimi-K2.6 and GPT-5.4 on interactive HTML generation. Equal-size ablations show that browser-backed admission carries the gain: without the certificate, the matched 9B lift nearly disappears.

1 Introduction

WebWorld addresses the structural weakness of VLM self-improvement loops by making the browser a deterministic counterparty that verifies interactive behavior. Typed contracts, browser-issued certificates, and certified-transition ratcheting turn verified browser interactions into supervision.

  • Motivation: Screenshot-based VLM judging can accept visually improved edits that fail under browser re-execution or remove previously working controls.The underlying failure is behavioral rather than merely visual: submit buttons may be inert, games may ignore keyboard input, and repairs may regress existing functionality.
  • Results: Without the browser-issued certificate, the matched 9B ablation lifted HTMLBench by only 0.4 points over Raw, more than ten times below the full gate.This ablation supports browser-backed admission as the source of the improvement.
  • Motivation: The browser serves as a deterministic, executable world model for how HTML artifacts behave under user actions.WebWorld uses the existing browser rather than training a separate world model, allowing the VLM prior to interact with it autonomously.
  • Method: A typed interaction contract converts VLM critique into a checkable repair target with replay actions, preservation requirements, and an evidence path.This interface separates the VLM’s proposal from the browser’s later verification.
  • Method: Acceptance certificates are issued only when browser re-execution establishes target progress and preserves every previously verified capability.Only certified transitions become the next baseline and enter the quality ratchet and SFT export.
  • Results: 52.7 on HTMLBench-400 and 85.5 on MiniAppBench-Val represent WebWorld-27B’s reported results, improving over Raw-27B by 5.3 and 14.9 points, respectively.The model matched frontier systems such as Kimi-K2.6 and GPT-5.4 on interactive HTML generation.

2 WEBWORLD

WEBWORLD replaces screenshot-only self-improvement with browser-verified transitions: the VLM proposes typed repairs, while deterministic re-execution decides what becomes certified supervision. Its quality ratchet admits only changes that achieve the target and preserve previously verified capabilities.

  • Motivation: The naive critique-and-rewrite loop can improve screenshots while breaking interaction or regressing previously working controls.Its closest re-implementation lifts HTMLBench by only 0.4 points over Raw when the browser certificate is removed.
  • Overview: WEBWORLD uses the browser as a deterministic world model that re-executes HTML artifacts under user actions.The browser provides the execution-based counterparty, while the VLM supplies a learned prior over plausible repairs.
  • Interaction Contract: The planner compiles each evidence-backed VLM critique into an interaction contract specifying the target, replay check, preservation set, and optional impact scope.Contracts reject uncertifiable or perception-only claims before repair and constrain edits to the affected region when defects are local.
  • Acceptance Certificate: The browser issues an acceptance certificate only after re-execution proves target progress and preservation of every previously verified capability.Proof checks prioritize same-trace replay and capability gain, followed by target-issue, localized visual, and static structural evidence.
  • Quality Ratchet: Only certificate-backed QUALITYSTEP transitions enter the SFT export, while refused candidates remain available for routing and debugging.The runtime distinguishes safe exploratory ITERSTEPs from certified training-quality transitions, making certified progress the quality ratchet.

3 Experiments

The experiments evaluate whether browser-verified transitions improve interactive HTML generation under matched training and benchmark-independent construction. WEBWORLD improves over Raw across HTMLBench and MiniAppBench, with the largest controlled gains arising from the full certificate gate.

  • Setup: WEBWORLD trains on 32,800 certificate-accepted transitions and evaluates on HTMLBench-400 plus MiniAppBench-Val without querying benchmark items during construction.The pipeline uses browser behavior for evaluation, while construction probes and benchmark assertions remain disjoint.
  • Main SFT Results: 5.3 points on HTMLBench and 14.9 points on MiniAppBench-Val are gained by WEBWORLD-27B over Raw-27B.At 27B, WEBWORLD reaches 52.7 on HTMLBench and 85.5 on MiniAppBench-Val.
  • Main SFT Results: The performance gap widens with capacity, reaching 3.4 points at 4B, 4.8 at 9B, and 5.3 at 27B on HTMLBench.The strongest improvements occur on TC pass, Functionality, and MiniApp transfer, while Rendering, Visual, and Code differ by less than a point without a consistent direction.
  • Certificate Mechanism Ablation: 0.4 points is the HTMLBench lift from removing the certificate at 9B, while the full gate gains 4.8 points over Raw-9B.The NoCertificate variant also drops 3.1 TC-pass points below Raw, whereas partial substitutes remain 6.6 to 8.7 TC-pass points behind the full gate.

4 Analysis

Analysis tests whether certificate admission supports deeper, denser supervision and preserves verified capabilities across multi-round repair. The results connect deeper certified trajectories, full gating, and reduced behavioral regressions to the certificate mechanism.

  • Depth and Trajectory Diagnostics: HTMLBench rises monotonically through certified depth ≥5 and plateaus at ≥8 under a fixed 5,000-example budget.The 5,000 depth-≥5 subset comes within 0.5 point of full training on 32,800 examples.
  • Depth and Trajectory Diagnostics: Certified depth counts rounds rather than runtime activity, and the certificate gate reduces per-page certified depth by more than half with zero hard-verified accepted regressions.The runtime begins multi-step trajectories on roughly a third of pages in the held-out snapshot.
  • Certificate Mechanism: The rule and depth analyses both implicate the certificate: one varies which gate admits, while the other varies which certified trajectories are retained.Together, they examine the same property from complementary axes.
  • Depth and Trajectory Diagnostics: Accepted repairs span rounds 4–6 for harder GUI failures, while the late-accept tail fades rather than growing.This bounded multi-round window exposes harder failures without diluting the export pool.
  • Certificate Mechanism: Only the full gate reaches the strongest 9B acceptance-rule results: 49.3 HTMLBench, 40.6 TC, 26.1 Functionality, and 66.3 MiniApp.Removing the gate leaves a 0.4-point lift, removing preservation leaves 1.5 points, and hypothesis-only rules reach about 3.5 points.
  • Capability Preservation: Verification-backed repair decreases console, gameplay, control, keyboard, and layout failures while keeping preservation regressions near zero.NoPreserve loses 3.3 HTMLBench points to the full gate, concentrated on TC pass and Functionality.

5 Related Work

Related work studies web agents, browser benchmarks, feedback-driven repair, and visual judging, but WEBWORLD assigns the browser a distinct certifying role for web-code supervision. Its contribution is to separate learned repair hypotheses from browser-verified proof.

  • Web Agents and Front-End Evaluation: Web-agent research interleaves reasoning, actions, and feedback in external environments, while browser benchmarks extend this paradigm to realistic websites and multimodal browsing.Examples include ReAct, WebShop, Reflexion, Mind2Web, WebArena, WebVoyager, and VisualWebBench.
  • Web Agents and Front-End Evaluation: Front-end benchmarks use browsers to navigate environments or score rendered artifacts, whereas WEBWORLD treats the browser as a world model for code.The distinction is the browser’s role in certifying repair transitions rather than merely evaluating navigation or rendering.
  • Repair, Judging, and Self-Improvement: Existing repair loops turn execution or critique into revisions, but web-code defects often lack a test-suite oracle because they are visual or interactional.SWE-bench provides an oracle through repository tests, unlike many web-code settings.
  • Repair, Judging, and Self-Improvement: WEBWORLD confines the VLM to hypothesis generation and gives the oracle role to browser re-execution under a typed interaction contract.The contract must certify both target progress and preservation, and accepted transitions become SFT data.
  • Concurrent Related Work: HTMLCure supplies the HTMLBench-400 testbed, while WEBWORLD uses it only for held-out evaluation and contributes browser-certified supervision instead.The two efforts therefore differ in the role assigned to browser execution.

6 Conclusion

WEBWORLD separates VLM hypothesis generation from browser-based proof, admitting only transitions that achieve target progress while preserving verified capabilities. Certified trajectories improve both interactive HTML benchmarks, while ablations show browser-backed admission carries the gain.

  • WEBWORLD routes every accepted transition through browser re-execution and an acceptance certificate before it becomes the next baseline or enters SFT.The VLM proposes via a typed interaction contract; only certified transitions accumulate in verified-capability memory and the SFT export.
  • 5.3 points on HTMLBench-400 and 14.9 points on MiniAppBench-Val are the improvements of WEBWORLD-27B over Raw-27B under matched training.The model reaches Kimi-K2.6/GPT-5.4-level performance in interactive HTML generation.
  • 0.4 points over Raw-9B is the lift after dropping the certificate in equal-size acceptance-rule ablations.This is more than ten times below the full gate, indicating that world-model-backed admission carries the gain.
  • Browser admission makes supervision reliable by checking intended interactions, target predicates, and preservation of earlier capabilities.The central design choice is the interface rather than a stronger prompt or critic.

Limitations

WEBWORLD is currently evaluated on single-file interactive HTML artifacts, and its certificate depends on the browser being a faithful oracle for the property at stake. Perceptual defects, incorrect critiques, and nondeterministic replay remain outside or beyond the verifier’s full protection.

  • WEBWORLD currently isolates single-file interactive HTML artifacts, leaving multi-file and framework-heavy settings with additional deployment variables.These variables include dependency replay, project organization, and agent scaffolding.
  • Perceptual defects such as aesthetic taste, brand identity, or accessibility nuance lack executable certificates and fall back to the VLM proxy.Those cases inherit the proxy’s noise rather than receiving browser-based certification.
  • A wrong critique can still waste repair budget, and dynamic applications still require deterministic replay.The verifier reduces false positives but does not make VLM critique infallible.

Ethics Statement

Because WEBWORLD edits executable web code, it inherits risks including insecure scripts, misleading interfaces, and broken accessibility behavior. The current setting uses sandboxing, verification gates, artifact scanning, and removal of flagged examples, but deployment still requires independent review.

  • Executable web-code editing inherits risks of accidental insecure scripts, misleading interfaces, and broken accessibility behavior.
  • Sandboxed HTML artifacts and verification gates are used to reject regressions before data export.
  • Generated artifacts and trajectories are scanned for private credentials, personal identifiers, external network endpoints, malicious scripts, and offensive content.Flagged examples are removed before release.
  • Deployment of repaired artifacts should still include security, privacy, and accessibility review.

A HTMLBench Benchmark Details

HTMLBench-400 is a frozen single-file benchmark with deterministic browser tests, while MiniAppBench-Val provides an out-of-distribution transfer check. HTMLBench scores rendering, visual design, functionality, interactivity, and code quality on a 100-point scale, with functionality measured from weighted test pass rates.

  • HTMLBench-400 is a frozen single-file benchmark with deterministic browser test cases, and MiniAppBench-Val is a held-out mini-application set for out-of-distribution transfer.
  • HTMLBench-400 scores each prompt along rendering, visual design, functionality, interactivity, and code quality, with scores summing to 100.
  • Functionality is measured as the weighted pass rate over a deterministic test pool, while coverage is recorded only as execution metadata.
  • Test programs use actions including click, type, hover, key press, resize, visibility check, JavaScript assertion, and screenshot-change check.They avoid framework-specific selectors, hidden source assumptions, real credentials, payments, and private services.
  • Five explicit non-overlap conditions separate HTMLBench-400 from the WEBWORLD data-construction pipeline, including item, prompt-family, test-case, and probe-set disjointness.The evaluation runner is used only on held-out items and is never queried during data construction or training.
  • WEBWORLD artifacts include the browser-verification runtime, certified transition corpus, SFT checkpoints, and external evaluation artifacts.HTMLBench-400 and MiniAppBench-Val are evaluation benchmarks, while web-agent and web-code benchmarks are comparison-only.
  • The artifacts are intended for research on browser-verified web-code generation rather than deployment without independent security, privacy, and accessibility review.
  • Table 5 organizes HTMLBench-400 task families as natural-language prompts paired with deterministic browser test cases used to score functionality.

C SFT Training Details

The training setup uses matched routes and a fixed software and evaluation environment across model scales. It also specifies the optimizer, sequence, hardware, and inference settings used for reproducibility.

  • Controlled training routes: Raw and WEBWORLD variants are trained at 4B, 9B, and 27B with the same conversation template, sequence length, optimizer recipe, and evaluation runner within each scale.Mechanism runs use equal-size route construction, while depth runs keep example counts fixed to separate trajectory quality from sample quantity.
  • Optimization and scale: The backbone is Qwen3.5 at 4B, 9B, and 27B parameters, trained with AdamW, learning rate 2×10−5, weight decay 0.01, 3% warmup, effective batch 128, and three epochs.Sequence length is 16,384 tokens.
  • Evaluation: Evaluation uses the pinned public HTMLCure analysis runner and the released MiniAppBench-Val validation split with default runner settings.The evaluation configuration is fixed to released runner specifications.
  • Runtime and inference: Browser execution uses deterministic headless Chromium/Playwright settings, while critique decoding is greedy and patch generation uses temperature 0.7 with one candidate per contract.The software environment is shared across routes, and package versions, runner commit, and checkpoint identifiers are released with the code artifact.

D Data Construction Pipeline

The data pipeline iteratively filters, repairs, re-executes, and certifies candidate transitions before exporting training examples. Its diagnostics emphasize behavioral evidence, preservation, rejection, and the cost-quality tradeoff of verified admission.

  • Pipeline: Each artifact undergoes up to Tmax = 10 critique-repair-verify rounds, ending with either a certificate that establishes the next baseline or a typed rejection.Admitted critiques become contracts, routed skills generate one patch, and rejected attempts feed the next critique.
  • Pipeline: 42,860 certified transitions result from roughly 60K admitted candidates reaching browser re-execution after contract filtering.Re-execution refuses 24,350 candidates for reasons including partial progress, visual-only evidence, preserve risk, and stale replay.
  • Pipeline: The final 32,800-example SFT corpus is a uniform random subsample of the certified pool, matching the Raw training budget.The sample uses seed 42.
  • Evidence paths: Different page types require different executed evidence paths, including replay, goal progress, state matching, workflow evidence, localized visual evidence, and static structural checks.This reflects failures involving clicks, typed input, keyboard control, state replay, and console inspection rather than appearance alone.
  • Rejection accounting: The full WEBWORLD gate rejects visually improved candidates that fail replay, drift after state changes, or lack preservation evidence, keeping them out of SFT targets.Rejected attempts remain diagnostic rather than becoming training supervision.
  • Cost and quality: WEBWORLD occupies a higher-cost, cleaner-accept frontier because interaction, critique, patching, and verification improve accepted-transition precision.The paper frames this as paying more to avoid noisy positives in SFT data construction.
  • Case trajectories: Case timelines show accepted paths reaching browser execution, replay, and final acceptance, while rejected paths fail when replay exposes regression or evidence remains visual-only.The trajectory view makes replayable progress and preserved prior capability the admission criteria.

F.1 Case A: Zoo Tycoon GUI Repair

The case studies show browser-verified repair trajectories across Zoo Tycoon, Smart Finance, and Knife Master. Accepted rounds add interactive, visual, or structural improvements while preserving prior capabilities; plausible but unsupported edits are rejected.

  • Zoo Tycoon: Zoo Tycoon begins as an unverified baseline because its guide state has not yet been replayed as a concrete browser target.The initial page already contains the zoo scene, paths, economy cards, and guide modal.
  • Zoo Tycoon: Round 1 accepts a same-trace repair that makes the guide action a visible clickable target while keeping the zoo scene intact.The product screenshot and replay record resolve the same target.
  • Zoo Tycoon: Rounds 2 and 3 accept native habitat refinements and docked status/help copy while preserving working controls, animals, and board visibility.These rounds use capability-gain and localized-visual proofs.
  • Zoo Tycoon: Round 4 rejects a plausible page because the candidate targets the wrong semantic control and replay never reaches the start gate.The rejection is shown without covering the zoo scene.
  • Zoo Tycoon: Rounds 5 and 6 accept a stable native status region and an export-ready zoo state after preserving the replay target, grid, status text, toolbar, economy card, and visible content.The final state enters the SFT export pool.
  • Smart Finance: Smart Finance starts with a visually complete dashboard whose filter is only visual and has not yet produced a replayed state transition.The baseline includes filter chips, balance cards, and a transaction table.
  • Smart Finance: Rounds 1 and 2 verify the Today filter and synchronize cards, rows, and chart with the filtered state, preventing a cosmetic-only tab repair.The first round uses same-trace replay; the second establishes capability gain.
  • Smart Finance: Rounds 3 and 4 preserve readable transactions and add an auditable filter card naming the active filter, row count, and synchronized cards.The trajectory becomes export-ready through replay-auditable visual and structural evidence.
Loading 2608.30530v1…