Where Did It Go Wrong? Process-Level Evaluation of Web Agents with Semantic State Tracking

Jiwan Chung; JiHyuk Byun; Vibhav Vineet; Seon Joo Kim

Source-linked AI summary

Where Did It Go Wrong? Process-Level Evaluation of Web Agents with Semantic State Tracking

Jiwan Chung, JiHyuk Byun, Vibhav Vineet, Seon Joo Kim

arXiv:2606.15673v2Apr 8, 2026cs.AIcs.LG

TL;DR

Existing web-agent benchmarks largely reduce long interaction trajectories to terminal success, limiting diagnosis of where agents fail. WEBSTEP adds semantic state tracking for process-level evaluation and shows that agents with similar success can have distinct, actionable failure patterns.

Problem
Most web-agent benchmarks rely on terminal success, which compresses trajectories into binary outcomes and provides little insight into failure location or capability limits.
Method
WEBSTEP pairs self-hosted websites with semantic MDPs that record structured states and transitions while agents interact through rendered GUIs.
Results
Process-level evaluation separates agents with similar terminal success into qualitatively distinct exploration, execution, skill, trajectory, and complexity-dependent failure signatures.
Takeaways & Limitations
WEBSTEP makes web-agent differences directly diagnosable, identifying specific skills, temporal failure patterns, and complexity regimes as improvement targets.
Takeaways & Limitations
WEBSTEP evaluates agents in controlled, self-hosted replicas with synthetic data rather than live online services, real user accounts, or production backends.

Abstract

from arXiv · show

Web agents act through long interaction sequences, yet existing benchmarks evaluate only terminal success, discarding all process information and offering little guidance on improvement. In this work, we conduct a process-level analysis of web agents. We introduce WebStep, a benchmark of 1,800 task instances with controlled difficulty and automatic semantic state tracking. Each website exposes a deterministic semantic MDP alongside the GUI: the agent operates on the interface, while the environment records high-level states and transitions in the background, enabling fine-grained analysis without manual annotation. Based on the semantic trajectory, we first show that process metrics reveal differences invisible to outcome evaluation: three agents whose success rates cluster within 34-37% diverge in exploration reach versus execution accuracy. Then, decomposing by skill characterizes the nature of these differences, exposing opposite per-skill rankings hidden within the same website: e.g., on Q&A, Claude CUA outperforms OpenAI CUA by 30% on navigation actions yet underperforms it by 6.7% on inspection, pinpointing a concrete skill to improve even within a domain. Bifurcation analysis further localizes the decisive error that loses the task and shows that this error is agent-specific rather than shared. Finally, these differences widen as tasks grow harder: success rate is similar on easy tasks but separates sharply as exploration becomes more demanding. Our process-level analysis opens a new avenue in web agent evaluation, providing fine-grained and actionable insight into where and how each agent should be improved. Project page: https://jiwanchung.github.io/webstep

1 Introduction

WebStep evaluates web agents through their semantic interaction processes rather than terminal success alone, automatically tracking trajectories to diagnose distinct failure modes. It shows that agents with similar outcomes can differ in exploration, execution, skills, and failure locations.

Motivation: Terminal-success evaluation assigns the same score to qualitatively different failures, obscuring whether an agent failed during navigation or execution.An agent may reach the correct message thread but apply the wrong reaction, while another never finds the relevant page.
Method: Because the GUI is rendered directly from the semantic model, WEBSTEP recovers full semantic trajectories exactly and evaluates processes without manual annotation.This enables automatic state tracking and process-level evaluation.
Process-level analysis: The framework decomposes behavior into exploration success, execution outcomes, and skill invocation patterns, while comparing trajectories at shared semantic states to localize failures.It extends evaluation from ranking to diagnosis by identifying where unsuccessful trajectories first diverge.
Findings: Agents with similar terminal success rates exhibit qualitatively distinct failure signatures across exploration, execution, skills, and task-complexity regimes.Process analysis separates agents that appear similar under terminal success and identifies when their failures become most pronounced.

2 Process-level evaluation via semantic MDP duals

WebStep evaluates web-agent processes by pairing each visual website with a deterministic semantic MDP that converts GUI interactions into interpretable state transitions and traces. Its task generator guarantees unique solvability while controlling difficulty through hard negatives that require detail-page inspection.

Semantic MDP duals: Each website pairs a visual interface with a semantic MDP dual that maps GUI actions into interpretable semantic transitions.The trace records visited items, newly available information, and high-level actions such as VIEWREPO, STARREPO, and CREATEISSUE.
Semantic MDP duals: The deterministic MDP represents interface position and revealed item attributes, uses typed semantic actions, and records each run as a state-action trace.Transitions follow st+1 = T(st, at), and the resulting trace supports process-level evaluation.
Semantic MDP duals: The GUI–MDP coupling makes the semantic model the source of truth: GUI actions update the model, which determines visible information before the page is re-rendered.This alignment ensures that observed interface behavior corresponds to semantic state transitions.
Task construction: Every generated task is solvable with exactly one target satisfying its natural-language constraints.Instructions combine templates with randomly sampled item-attribute constraints, and the world is populated to satisfy the specification.
Task construction: Hard negatives control task complexity by matching visible list-page attributes while differing on detail-page attributes, requiring agents to navigate to individual details.The number of planted hard negatives is itself a design variable, and each task also has a statically derivable oracle trajectory.

3 WEBSTEP

WEBSTEP is a process-level benchmark for web agents built from deterministic semantic models underlying visually realistic, self-hosted websites. It combines explicit semantic-MDP evaluation and complexity control with a validated construction pipeline and process metrics beyond terminal success.

Benchmark design: WEBSTEP spans 10 self-hosted websites modeled after real sites across productivity, commerce, professional, collaborative, and technical-knowledge domains.The websites retain realistic visual environments while replacing their backends with deterministic semantic models.
Benchmark design: WEBSTEP evaluates process through explicit semantic MDPs, deterministic self-hosted environments, and task-conditioned world generation for complexity control.This distinguishes it from live-website settings and terminal-level benchmarks.
Construction pipeline: The benchmark is constructed by converting live website workflow traces into formal MDP specifications and self-hosted sites through coding-agent generation with iterative human review.Each site is subsequently validated through automated testing, including MDP unit tests, GUI interaction tests, and oracle trajectory replay, plus manual verification.
Evaluation metrics: Semantic MDP traces report exploration, execution, and information coverage alongside terminal success, exposing behavioral differences that terminal outcomes alone miss.These aggregate process-level metrics are designed to reveal differences beyond terminal success.

4 Experiments and Results

Process-level metrics expose behavioral differences hidden by similar terminal success rates, then localize those differences to specific skills, trajectory errors, and exploration demands. Across agents, harder exploration requirements widen performance gaps and reveal distinct failure patterns.

4.1 Outcome versus process: 34–37% terminal success rates among small agents conceal divergent behavior: UI-TARS is exploration-strong but execution-weak, whereas Fara is under-exploratory.UI-TARS identifies correct targets more often than GUI-Owl (∆+5.9%) but performs worse during execution (∆-4.7%); Fara takes the fewest steps and has the lowest information coverage.
4.2 Skill-level analysis: ∆+30.0% on Navigate and ∆−6.7% on Inspect separate Claude CUA from OpenAI CUA on Q&A, pinpointing item inspection as Claude CUA’s weakness.Skill-level analysis reveals opposing proficiencies that website-level scores would average away; Claude CUA is generally stronger at Commit and Filter, but relative skills vary by website.
4.3 Bifurcation analysis: 48% of GUI-Owl and FARA’s delayed-failure actions are Navigate, compared with OpenAI CUA’s continued Inspect (64%) and Search (32%).The agents therefore exhibit different delayed-failure forms: continued navigation for GUI-Owl and FARA versus continued evidence gathering and query refinement for OpenAI CUA.
4.3 Bifurcation analysis: 33–79% of agents’ first wrong branches arise through Inspect, while GUI-Owl diverges through Filter (31%) and Fara and UI-TARS fall back to Search (30%).Bifurcation points are defined as the last shared semantic MDP state before trajectory divergence, enabling agent-specific localization of decisive errors.
4.4 Difficulty scaling: Exploration performance separates increasingly as hard-negative counts, successful trajectory lengths, and required information access increase.Performance is relatively similar on easier card tasks, while filter and detail tasks increasingly distinguish larger models and the strongest agents, Claude CUA and OpenAI CUA.

5 Conclusion

WEBSTEP enables automatic, fine-grained process-level evaluation of web agents by pairing self-hosted websites with semantic state tracking. It reveals behavioral differences that terminal success alone hides, making exploration, action, and failure directly diagnosable without manual trajectory annotation.

Contribution: WEBSTEP pairs self-hosted websites with semantic state tracking for automatic process-level evaluation of web agents.This design enables fine-grained analysis without manual trajectory annotation.
Findings: Process-level evaluation exposes major differences in how agents explore, act, and fail that terminal success alone hides.These differences become directly diagnosable through process-level analysis.
Impact: WEBSTEP is intended to support deeper diagnosis of web agents.

Ethics Statement

WEBSTEP evaluates web agents in controlled, self-hosted environments without human-subject experiments, real user accounts, live services, or real user data. Its websites reproduce publicly observable interactions and use synthetic world data rather than production backends.

Evaluation Setting: WEBSTEP avoids human-subject experiments, real user accounts, live online services, real user data, and production backends during evaluation.The benchmark instead uses controlled, self-hosted environments, independently implemented website reproductions, and synthetic world data.

Reproducibility Statement · Appendix · G Artifact Viewer

The paper states that it will release the complete WebStep benchmark and evaluation suite with deterministic, seed-controlled environments for reproducible benchmark conditions. The appendices include an artifact viewer, data examples, trajectory comparisons, transition graphs, and case studies.

Reproducibility Statement: 10 websites and 1,800 task definitions with world seeds and oracle trajectories are included in the planned public benchmark release.The release also includes the complete evaluation suite.
Reproducibility Statement: Deterministic MDPs with fixed world seeds make environment dynamics, verification outcomes, and semantic traces reproducible given identical agent actions.This guarantees reproducibility of benchmark conditions, while the supplied passage does not complete the statement about full end-to-end runs.
G Artifact Viewer: The Artifact Viewer and appendix contents provide supporting views of benchmark artifacts, example data, trajectories, transition graphs, and agent behavior case studies.This sentence synthesizes the supplied appendix listings without adding details beyond their named sections.
Appendix: The data examples appendix catalogs task templates, compares agent and oracle trajectories, and presents MDP surface transition graphs.These materials appear under sections H.1, H.2, and H.3.
Appendix: The appendix contains a case study section with analyses of model scale and browsing quality, premature commitment to hard negatives, and thorough exploration before commitment.The listed case-study subsections are I.1, I.2, and I.3.

A Limitations … C.4 Information Coverage

The paper prioritizes reproducible, trajectory-level evaluation through controlled environments, semantic verification, and information-coverage measurement, while acknowledging limits in realism, domain, data, task, skill, and action-space coverage. It situates WebStep among benchmark and process-evaluation approaches and specifies its agents, runtime, episode protocol, and evidence-based commit metric.

A Limitations: WebStep improves measurement precision through controllability, reproducibility, and trajectory-level diagnosis, but sacrifices realism, coverage, and action scope.Its limitations include semantic abstraction, synthetic data, template-based tasks, restricted domains, invocation-based skill measurement, and coordinate-based interaction.
A Limitations: WebStep covers ten domains but excludes applications involving sensitive data, complex authorization flows, or rich media interaction.The authors describe this as incomplete coverage, while noting broader scope and substantially more task instances than existing alternatives.
B Related Work: Existing web benchmarks span controlled synthetic, self-hosted realistic, and online live-web environments, whereas WebStep emphasizes built-in process instrumentation.Each WebStep environment has an explicit semantic MDP dual grounding process evaluation in its own state and transition structure.
C.1 Agent Configurations: WebStep evaluates UI-TARS, Fara, GUI-Owl, Qwen3.5, OpenAI CUA, and Claude CUA under a shared browser-interaction setup.Claude CUA receives screenshots plus interactive DOM elements and uses reference targeting with coordinate fallback; the other agents operate purely from pixels.
C.2 Compute and Runtime Statistics: GPU-based agents run on one node with 4× NVIDIA H200 GPUs, completing the full evaluation in approximately 3–4 days.OpenAI CUA and Claude CUA use vendor APIs and each complete within a day with 8 parallel workers.
C.3 Evaluation Protocol: Each episode launches headless Chromium, initializes task state, and runs a screenshot-and-action loop until completion, failure, or the 50-turn limit.The runner supplies screenshots, URL, and conversation history, then executes returned clicks, text inputs, or scrolls.
C.3 Evaluation Protocol: Terminal success requires all verifier conditions on the stored semantic MDP trajectory to be satisfied, with a safe pass for agents reaching the correct decision point without committing.The safe pass addresses models such as Fara that identify the target entity but fail to execute the final commit action.
C.4 Information Coverage: COVtcommit measures the proportion of benchmark-defined task-relevant evidence gathered when the agent first executes a terminal action.Information constraints are derived from each task’s information gap, template type, and reasoning type; site visibility rules determine which observed surfaces satisfy them.

C.5 Reproducibility · C.6 LLM Usage · D Benchmark Specification

The work supports reproducibility through public release of benchmark artifacts, deterministic evaluation conditions, standardized agent settings, and explicit limits on closed-source repeatability. It also documents Claude Opus 4.6 as a human-supervised accelerator for benchmark construction.

C.5 Reproducibility: The release will include the complete benchmark codebase, 10 site implementations, task-generation and evaluation scripts, metric computation, and all 1,800 task definitions.Task definitions include world seeds, instructions, oracle trajectories, and verifier conditions.
C.5 Reproducibility: Fixed random seeds fully determine each task’s world data, ensuring identical evaluation conditions across runs.World data includes user profiles, item catalogs, and message contents.
C.5 Reproducibility: All site implementations are pure deterministic MDPs, so each state–action pair always produces the same next state and observation.This eliminates environmental stochasticity.
C.5 Reproducibility: Open-weight agents Fara, UI-TARS, and GUI-Owl use temperature 0, while Qwen3.5 uses temperature 0.7, top-p 0.8, and top-k 20.These settings ensure deterministic open-weight outputs except for Qwen3.5’s official implementation configuration.
C.5 Reproducibility: All agents receive a maximum of 50 GUI turns per task and share a 1440 × 900 browser viewport.These shared constraints standardize episode budgets and visual input dimensions.
C.5 Reproducibility: Exact reproducibility is not guaranteed for OpenAI CUA and Claude CUA because of possible server-side API non-determinism, although repeated runs show low variance.The limitation applies specifically to the closed-source agents.
C.6 LLM Usage: Claude Opus 4.6 assists benchmark construction by converting recorded interaction traces into semantic MDP specifications and executable websites within a human-driven development loop.The process includes substantial manual validation, editing, and re-iteration.

D.1 Task Distribution · D.2 Site Screenshots · D.3 Per-Site MDP Specifications

WEBSTEP comprises 1,800 tasks distributed evenly across ten instrumented web applications, while its per-site deterministic MDPs formalize typed state, visibility, and skill structure for process-level analysis.

D.1 Task Distribution: The hierarchical task distribution uses multiple templates within each site to capture varied interaction patterns.Figure 7’s inner ring represents the ten sites, while its outer ring represents template instance counts.
D.1 Task Distribution: 1,800 tasks are distributed across 10 websites, with 180 tasks per site and templates containing 4–30 instances.The distribution spans diverse interaction patterns and difficulty levels.
D.2 Site Screenshots: WEBSTEP includes ten standalone HTML/CSS/JS applications spanning mail, calendars, shopping, accommodation, food delivery, housing, coding Q&A, repositories, job networking, and team chat.Each application has full MDP instrumentation.
D.2 Site Screenshots: The ten sites model distinct visual layouts, navigation patterns, and interaction paradigms based on representative real-world applications.The domains range from e-mail and calendaring to e-commerce, social networking, and developer tools.
D.3 Per-Site MDP Specifications: Each site is modeled as a deterministic MDP with typed variables covering navigation surfaces, UI component states, and application-level data.Examples include the current inbox or search results, open dropdowns or scroll position, and selected items or form contents.
D.3 Per-Site MDP Specifications: Visibility partitioning determines which state variables are observable on each surface, making cross-surface reasoning require multi-step navigation.Tables distinguish Card, Detail, and Commit surfaces and identify attributes visible only on detail pages.
D.3 Per-Site MDP Specifications: Tasks are labeled with five skill categories—Navigate, Search, Filter, Inspect, and Commit—for fine-grained capability analysis across sites.The four exploration categories are global action-type sets, whereas Commit is task-relative and denotes the final task-completing step.

E Data Generation Pipeline … E.3 Oracle Trajectory Generation

WEBSTEP uses an iterative, validated pipeline that converts grounded user workflows into deterministic semantic-MDP websites and reproducible task worlds. Deterministically constructed oracle trajectories then provide optimal reference behavior for evaluation, efficiency, and solvability checks.

E.1 Site Construction Pipeline: WEBSTEP sites are built iteratively with an LLM coding agent, while intermediate outputs receive manual review and downstream failures trigger revisions to earlier stages.The process is explicitly iterative rather than strictly linear.
E.1 Site Construction Pipeline: Grounded exploration records user-visible actions and screenshots from representative workflows, supplying evidence for the subsequent semantic-MDP specification.The observations rely solely on screenshots and user actions, not backend code or API traffic.
Algorithm 1 Specification construction from grounded traces: The specification defines typed entities, relations, surfaces, navigation, state schemas, observations, and action preconditions before implementation as a self-contained web application.The GUI renders the underlying semantic MDP state, and task-relevant interactions are mapped to typed semantic actions.
Algorithm 1 Specification construction from grounded traces: Validation combines semantic unit tests, coordinate-level GUI tests, end-to-end oracle replay, and manual QA to check transitions, visibility, interaction fidelity, trajectory execution, and verifier correctness.Issues involving verifier logic, world-generation ambiguities, and specification gaps are fixed and re-verified.
E.2 World Generation: Each task world contains a constraint-satisfying target, controlled hard negatives, and distinguishable fillers, with hard-negative counts calibrating the discrimination effort required.Hard negatives share surface-level attributes with the target but differ on task-relevant attributes.
E.2 World Generation: A single integer seed deterministically generates each world, so repeating a task seed produces an identical environment.This provides full reproducibility for world generation.
E.3 Oracle Trajectory Generation: Every task includes a minimal-length semantic-action oracle trajectory generated deterministically from a template-defined surface-visit plan.The plan emits typed semantic actions rather than pixel coordinates.
E.3 Oracle Trajectory Generation: Oracle generation navigates to the target region and, when comparison is required, inspects up to two top-ranked hard negatives before opening the target; replay verifies optimality and task completion.Oracle trajectories support coverage and skill metrics, define an upper bound on task efficiency, and verify that tasks are solvable within the MDP.

F Extended Quantitative Results … F.4 Exploration SR by problem complexities

The extended quantitative results provide per-site performance decompositions, skill-invocation analyses, aggregate skill rates, and exploration-success breakdowns by problem complexity. An interactive artifact viewer supports turn-level diagnosis, trajectory comparison, verification inspection, and batch agent analysis.

F.1 Per-site performance decomposition: Tables 9–12 break down terminal success, exploration success, execution success, and informational coverage at commit by site.Table 2 reports aggregate metrics, while Tables 9–12 provide the full per-site breakdown.
F.2 Per-site skill invocation: Tables 13 and 14 report site-level skill-invocation rates, with each cell conditioned on episodes requiring that skill.Commit is task-relative and counts as invoked only when the agent trajectory ends with the oracle trajectory’s final task-completing action.
F.3 Aggregate skill invocation: Table 15 reports aggregate skill-invocation rates across all sites, including the task-relative commit-invocation criterion.Each cell is the fraction of episodes requiring that skill where the agent invoked it, with best values marked per column.
F.4 Exploration SR by problem complexities: Table 16 reports exploration success by hard negative count, treating hard negatives as a designed lower bound on identification ambiguity.HN=0 tasks lack identification ambiguity and vary along other difficulty axes, so monotone trends should be interpreted over HN≥1.
F.4 Exploration SR by problem complexities: Table 17 reports exploration success by oracle trajectory length, comparing agents across increasing trajectory requirements.The table reports percentages and marks the best value in each column.
F.4 Exploration SR by problem complexities: Table 18 reports exploration success by information-access level across Detail, Filter, and Card tasks.Detail tasks require opening detail pages, Filter tasks use search, filter, or sort actions without a detail gap, and Card tasks are solvable from list-level cards.
F.4 Exploration SR by problem complexities: The artifact viewer supports turn-by-turn replay, side-by-side agent–oracle trajectory visualization, verifier-condition inspection, and batch comparison with aggregate statistics.These capabilities expose screenshots, actions, state changes, matches, divergences, verifier status, and failure-mode distributions for diagnosis.
F.4 Exploration SR by problem complexities: The authors will release the artifact viewer together with the evaluation codebase.The viewer is designed for inspecting and analyzing WebStep agent evaluation results.

H Data Examples … I.7 Case 7: Failure by Safeguard, Not by Competence

The paper grounds its process-level evaluation in parameterized task templates, semantic MDP trajectories, and qualitative cases showing how agents differ in exploration, execution, efficiency, and safeguards. Across examples, oracle comparisons and case studies localize both competent behavior and failure mechanisms that terminal success alone obscures.

H.1 Task Template Catalog: WEBSTEP templates parameterize natural-language instructions, instantiated constraints, and verifier conditions across navigation, constraint complexity, and reasoning demands.Examples span information retrieval, comparison, and multi-step form completion.
H.2 Example Trajectories: Agent vs. Oracle Trajectory: Oracle trajectories provide minimal ground-truth action sequences, while agent trajectories expose unnecessary exploration, premature commitment, and error-recovery attempts.On Shopping, the oracle uses 3 actions; Fara inspects wrong products, while UI-TARS repeatedly reformulates queries without reaching the target.
H.3 MDP Surface Transition Graphs: Semantic MDP graphs represent surfaces as nodes and typed semantic actions as edges, separating inter-surface transitions from intra-surface state changes.The Mail site contains 3 surfaces and 36 typed actions, including 8 inter-surface and 28 intra-surface actions; other sites vary in graph structure.
I.1 Case 1: Model Scale and Browsing Quality: Case 1 shows that equal terminal success can conceal sharply different browsing quality: OpenAI-CUA searches, expands messages across three threads, and succeeds in 14 MDP actions.The comparison is between OpenAI-CUA and UI-TARS on the same Mail task; both succeed, but their trajectories diverge.
I.2 Case 2: Premature Commitment to Hard Negative; I.3 Case 3: Thorough Exploration Before Commitment: UI-TARS prematurely commits to a Shopping hard negative by adding an unverified dress to the cart, whereas systematic inspection of all three results supports correct commitment.The positive example checks Weight = 0.3 lbs in Technical Details before continuing through the remaining results.
I.4 Case 4: Exploration Success, Execution Failure; I.5 Case 5: Exploration Failure, Execution Success: Fara’s cases separate exploration from execution: it can reach the correct StackOverflow answer or GitHub repository yet miss the intended upvote or Star button.A contrasting Code Repo case shows exploration failure when Fara mistakes a repository language label for the TypeScript filter, while still completing the later execution path.
I.6 Case 6: Redundant Process Despite Success: Fara completes a Shopping task despite executing 13 MDP actions versus the oracle’s 9, with 6 extra steps caused partly by garbled concatenated queries.The case illustrates redundant process hidden behind a passing outcome.
I.7 Case 7: Failure by Safeguard, Not by Competence: Fara identifies the correct accommodation listing and reaches “Confirm and pay” with Exploration: Pass, Coverage: 100%, and Hard Neg: 0, but stops because completion requires personal information.The failure is therefore attributed to a safeguard rather than a demonstrated competence deficit.