WebPageBench: Event-Level Verification and Controlled UI-Variant Generation for Web Agents

1 Introduction

• Existing benchmarks trade off reproducible path matching, live-web scale with judge and content challenges, and self-hosted final-state checks that cannot distinguish steps taken from the state left behind.They also do not isolate sensitivity to individual control implementations while keeping the prompt and success condition fixed.Source: §1 · p. 1
• WebPageBench instruments mock-site interfaces so task success is checked against typed events, while the same instrumentation makes interface form a controlled variable.The framework is designed to help distinguish model failures from harness and interface failures and reproduce them locally.Source: §1 · p. 1
• The event-level verification contract defines success through typed event parameters and matching conditions, without a judge model, and records agent completion signals separately.The contract includes 19 event types and exact, wildcard, comparison, and last-state matching.Source: §1 · pp. 1–2
• Controlled variants change a UI control while keeping the prompt and success conditions fixed; the release includes 87 variants from 25 donor tasks, 82 changing exactly one control.Figure 1 illustrates the design with the same departure-date task rendered as a popup calendar or a native date field.Source: §1 · p. 2, Figure 1 · p. 2
• The runner evaluates browser/DOM harnesses and screenshot-only GUI agents under one verifier, and the release includes a public leaderboard of 24 model–harness pairs on 152 tasks.The runner covers six browser/DOM harness configurations and five screenshot-only GUI-agent families.Source: §1 · p. 2

2 Related Work

• Benchmark approaches include reference matching, judge models, hybrid checks, and deterministic programmatic verification, each with different verification scope or reliability trade-offs.WebPageBench belongs to the deterministic group, which includes reward checks, backend API checks, replay, and final-state comparison.Source: §2 · p. 2
• Unlike final-state comparison, WebPageBench matches conditions against typed events emitted during a run, including intermediate steps and parameters.Its task conditions are authored by hand, while variant generation is automated.Source: §2 · p. 2
• Among the surveyed systems, none renders one task through alternative implementations of a control while holding its prompt and verifier fixed.The paper distinguishes this from WorkArena++ brand sampling, which repaints the same interface without changing controls.Source: §2 · p. 2
• WebPageBench differs from a WebArena-style environment with logging by using event-based task specifications, per-condition verdicts, fixed task specifications across UI realizations, and one runner for both action spaces.The related-work discussion describes BrowserGym and AgentLab as complementary, with WebPageBench potentially exposed through that ecosystem.Source: §2 · pp. 2–3

3.1 Mock sites and anonymization

• Six mock sites cover everyday online activities; site contributors hand-wrote canonical scenarios, and variants copy rather than create scenarios.The sites use neutral route and state names to avoid branding and navigation cues from the real platforms they are based on.Source: §3.1 · p. 3

3.2 Events and UI configuration keys

• Pages emit typed semantic events with parameters into isolated task-track logs, rather than reconstructing them from the DOM.The 152 tasks declare 521 conditions over 19 semantic event types.Source: §3.2 · p. 3
• A UI configuration key selects a control implementation; controls emit the same event across implementations, keeping their conditions valid.The nine keys cover eight controls and the colour theme, with 33 implementations in total.Source: §3.2 · p. 3, Figure 2 · p. 4

3.3 The verification contract

• Conditions match named events and required parameters independently; strings support exact or wildcard matching, numeric comparisons, and grouped alternatives.Quantity conditions use the last event for an item unless a condition explicitly uses match: any.Source: §3.3 · p. 4
• The methodology separately records harness completion and event-based success, distinguishing over-claims from under-claims and tracking their gap.An over-claim is completed with EMS 0; an under-claim is EMS 1 without a completion claim.Source: §3.3 · p. 4
• Table 1 describes the evaluated harnesses and their input modes, including DOM trees, extracted page text, and screenshots where listed.DOM is an indexed element tree that agents can act on by element handle.Source: Table 1 · p. 5

3.4 Generating controlled UI variants

• The release contains 87 variants from 25 donor tasks; 82 change one control, while five document-cabinet variants change three keys.Variants copy the donor prompt and conditions, preserving the task specification while changing control implementation and its associated interface properties.Source: §3.4 · p. 5

4 System Demonstration

• A single command runs the Docker stack; environment variables select the harness and model, and the runner creates a track for each task before checking its log.For gemini-3.8-flash with openmanus, a 152-task run averaged 7.8 steps and 221 seconds of harness time per task.Source: §4 · p. 5
• The track page lists events beside task conditions and marks matches, allowing visitors to compare an identical task run with another control implementation.Visitors can also open leaderboard slices by harness, interaction class, or UI key.Source: §4 · p. 6
• Finished runs are exported to a public leaderboard with success, speed, and cost views broken down by domain and interaction class.New tasks are added as JSON files, while new control implementations are registered as values for a UI key.Source: §4 · p. 6

5.1 Protocol

• The protocol fixes a harness, model, and task set per run, reporting EMS, Completion, steps, duration, and cost; Done&pass is counted from run logs.Each submission's EMS covers all 152 tasks, variants included, without separately scoring donors and variants.Source: §5.1 · p. 6

5.2 Results

• Completion minus EMS reaches 41 points for gpt-5.6-luna with openmanus (1.00 vs. 0.59); Gemini on OpenHands instead reports 0.461 completion against 0.572 EMS.For Gemini on OpenHands, 18 of 87 verified successes ended without a completion signal.Source: §5.2 · p. 6
• Harness rankings depend on the model: openmanus leads Gemini-3.8-flash at 0.822, while GPT-5.6-luna peaks on ouroboros-full-isolated at 0.796.GPT-5.6-luna scores 0.592 on openmanus, its lowest configuration.Source: §5.2 · p. 6
• On the same 152-task EMS, screenshot agents score below DOM harnesses: Qwen3.8-27B scores 0.737–0.763 through DOM harnesses and 0.480 through screenshots.Screenshot-agent EMS scores include Fara-1.5-9B at 0.638, OpenCUA-72B at 0.421, and UI-TARS-1.5-7B at 0.336.Source: §5.2 · p. 6
• Screenshot agents also under-report verified successes: Fara-1.5-9B solves 0.638 but reports and solves only 0.493, while Qwen3-VL scores 0.480 against 0.375.Between one in nine and one in four verified successes of these agents ends without a completion signal.Source: §5.2 · p. 6
• Cost and speed do not track EMS: glm-5.2 with openhands costs $76.08 and scores 0.500, while gpt-5.6-luna with browser-use scores 0.704 for $3.23.OpenHands is fastest per task (132–158 s) and most expensive in each of the three model groups where it appears.Source: §5.2 · p. 6
• For gemini-3.8-flash × openmanus, primary-class scores range from FILES 18/18 and FAV 14/14 to PAY 2/4; classes with four or fewer tasks are diagnostics, not a ranking.The submission does not report a donor–variant split, so ∆h,w is not reported.Source: §5.2 · pp. 6–7

6 Conclusion

• WebPageBench verifies web-agent tasks from the interface’s own event log, using typed events rather than model judgments or interface text.The event contract does not depend on the language.Source: §6 · p. 7
• Rendering one task through different implementations of one control makes sensitivity to interface form measurable under unchanged prompts and success conditions.Source: §6 · p. 7
• On the public 152-task leaderboard, the completion–verification gap reaches 41 points and changes sign with the harness.Source: §6 · p. 7

Limitations

• WebPageBench is not a measurement of production websites; anti-bot defenses, real payment rails, and content drift are outside its modeled setting.Browser-stored application state also prevents scenarios spanning two browser sessions, and the suite is Russian-only without broader codeswitching or transliteration testing.Source: Limitations · p. 7
• Published submissions pool donor tasks with variants, so this snapshot does not report ∆h,w.Variants change DOM structure, action count, and screenshot coordinates together, limiting interpretation of interface-form effects.Source: Limitations · p. 7
• Event matching is not proof of understanding: conditions are checked independently and without order, so tasks requiring distinctions between event orderings cannot be expressed.Task-specific parameter values make emitting the required events without understanding unlikely, but no adversarial audit confirmed this.Source: Limitations · p. 8

Ethics Statement

• The environment uses synthetic accounts, payment cards, passenger records, and book catalog entries; its mock payment flow reaches no external system.Evaluation runs use headless browsers against a local deployment and issue no traffic to third-party websites.Source: Ethics Statement · p. 8
• Catalog photographs have redistribution-compatible licenses, the software is released under Apache License 2.0, and third-party images retain their source licenses.The attribution workbook lists CC BY, CC BY-SA, CC0, or Public Domain for the photographs.Source: Ethics Statement · p. 8
• The authors used Cursor, Claude, and GPT for drafting LaTeX and code review, and checked task files, event contracts, evaluation numbers, and claim wording.The authors state that remaining errors are theirs.Source: Ethics Statement · p. 8

A Example Tasks

• The examples are translated for readability; original prompts ship with the suite, and translation affects the paper rather than the evaluation.Source: A Example Tasks · p. 9
• The marketplace example asks the agent to add any catalog item to the cart and requires one basket_add event with no parameters.Source: A Example Tasks · p. 10
• The hotel example specifies a city, check-in and check-out dates, and two guests, with conditions for selections and the resulting search state.Source: A Example Tasks · p. 10
• The rail example requires seven conditions covering origin, destination, date, train, tariff, cart addition, and successful payment.Source: A Example Tasks · p. 10

B Variant Catalog

• The 87 variants instantiate 20 profiles over 25 donor tasks; 12 are dark-theme clones, while five document-cabinet variants change three controls together.Those three controls share one screen.Source: B Variant Catalog · p. 10
• The release contains 65 canonical tasks and 87 variants, for 152 tasks total, with 19 event types and 521 conditions.It registers 33 implementations across nine UI configuration keys.Source: B Variant Catalog · p. 10

C Interaction Classes

• The interaction taxonomy defines 19 classes, and Table 5 lists nine primary classes present in every public submission.The supplied passage does not provide the table’s scores.Source: C Interaction Classes · p. 10
• NAV, RADIO, SEARCH, and SEAT occur in task files but not in ui_classes; six other classes have no released task.The six are button, free text, checkbox, filter, login, and low-level DOM activity.Source: C Interaction Classes · p. 10

• WebPageBench matches conditions against typed events emitted during a run, including intermediate steps and parameters, rather than only the final application state.Source: D Related Benchmarks · p. 10
• Table 4 presents one donor per domain and its variants, which retain the donor’s prompt and conditions while changing only the named control.Source: Table 4 · p. 11
• Table 5 covers nine primary classes in every public submission and reports passed/total for gemini-3.8-flash × openmanus; classes with four or fewer tasks are not rankings.Source: Table 5 · p. 11
• Among the surveyed systems, none re-renders a fixed task with an alternative control implementation; WorkArena++ changes colours and logos only.Source: D Related Benchmarks · p. 10, Table 6 · p. 12
• Table 6 groups benchmarks by success-deciding approach and describes environment, domain/site, and agent-connection information.Source: Table 6 · p. 12