Source-linked AI summary
Auditing Web Agent Evaluation on WebArena-Lite: Human Review of Outcomes and Trajectories
Chengguang Gan, Zimeng He, Yoshihiro Tsujii, Ken-ichiro Kobayashi, Hiroki Itoh, Kotaro Funakoshi
TL;DR
Final-outcome evaluators provide limited evidence about whether web agents completed tasks and how failed trajectories unfolded. The paper audits all 165 WebArena Lite tasks with human outcome review and trajectory analysis; across four GPT 5.5 settings, review recovers 5.45 to 8.49 percentage points of success missed by the evaluator.
Problem
Automatic evaluation often reduces web-agent performance to a final score, while human verification of completion and detailed analysis of failed trajectories remain limited.
Source: §1 · pp. 1–2Method
The authors audit all 165 WebArena Lite tasks, retaining original scores while correcting evaluator false negatives, identifying first consequential errors, and assessing verified progress across trajectories.
Source: §1 · p. 2Results
Human review recovers 47 missed successes, or 5.45 to 8.49 percentage points per setting, across four GPT 5.5 settings.
Source: §7 · p. 9Takeaways & Limitations
Web-agent evaluation should report original and corrected outcomes together and preserve trajectory evidence to explain failures, progress, and stopping behavior.
Source: §7 · p. 9Each condition has one rollout per task, so paired task flips describe observed runs rather than a distribution over repeated trials.
Source: §6 · p. 9
Abstract
from arXiv · showhide
Web agents are an important application of large language models, yet their evaluation often depends on rule based or language model evaluators that inspect only the final outcome. Human verification of task completion and detailed analysis of failed trajectories remain limited. We audit all 165 WebArena Lite tasks under six evaluation conditions built from GPT 5.5 and an untrained Qwen3.5 9B model. The audit retains the original score, corrects false negatives from the automatic evaluator, identifies the first consequential error, and examines progress across the trajectory. We also study a Memory and Analysis Support Mechanism (MASM), which maintains explicit execution state, and Guide Text, which provides task relevant procedural guidance. Across four GPT 5.5 settings, human review recovers 5.45 to 8.49 percentage points of success missed by the evaluator. With a 25 step budget, Guide Text raises corrected success with MASM from 34.55% to 38.18%. On the untrained Qwen3.5 9B model, MASM raises the evaluator score from 13.90% to 18.80%. Review of 102 failed GPT 5.5 trajectories reveals frequent scrolling loops, unfinished exploration, premature answers, invalid actions, and incomplete form workflows. Step level evidence further shows that substantial early progress can coexist with a final failure. These results show why final scores alone provide an incomplete account of web agent behavior and motivate human grounded, trajectory aware verification.
1 Introduction
Final scores conflate whether an agent completes a task with whether an evaluator recognizes completion, leaving both false negatives and failure trajectories difficult to assess. The paper audits all 165 WebArena Lite tasks to correct outcomes and trace errors and progress, finding substantial evaluator misses and distinct behavior hidden by binary scores.
1 Introduction
- Automatic scores conflate actual task completion with evaluator recognition, which can diverge because of formatting, strict matching, partial state checks, or stopping behavior.A binary score also cannot distinguish little progress from near-completion, post-completion actions, or a correct answer in an incompatible format.Source: §1 · pp. 1–2
- The audit covers all 165 WebArena Lite tasks, retaining original scores while reviewing outcomes, first consequential errors, and verified progress after each action.The retained conditions compare GPT 5.5 with matched MASM and Guide Text settings at 15 and 25 steps, plus an untrained Qwen3.5 9B MASM ablation.Source: §1 · p. 2
- Human review recovers 5.45 to 8.49 percentage points of success across four GPT 5.5 settings, while 102 failed runs reveal recurring trajectory errors.With 25 steps, Guide Text raises corrected success with MASM from 57 of 165 tasks (34.55%) to 63 of 165 (38.18%).Source: §1 · p. 2
- The findings target evaluation and debugging of web agents, not a leaderboard claim.The analysis also uses paired task evidence to study MASM, Guide Text, and action budgets, with implications for evaluator design, stopping criteria, and deployment monitoring.Source: §1 · p. 2
- The paper identifies first consequential errors and separates navigation, information gathering, action execution, stopping, and answer-generation failures.Step-level progress can differ markedly even when two trajectories receive the same final binary score.Source: §1 · p. 2
2 Related Work
Web-agent research has developed interactive benchmarks and increasingly capable policies, but evaluation commonly compresses trajectories into final success signals. Prior work on evaluator reliability and trajectory analysis motivates this paper’s human-grounded audit.
2 Related Work
- Most web-agent results still reduce each trajectory to a final success value, despite benchmarks supporting executable or live-site evaluation.WebArena Lite retains 165 representative tasks for a more practical evaluation cycle, and BrowserGym provides shared interfaces for controlled experiments.Source: §2 · p. 2
- Browser policies have advanced from prompted reasoning to browser-experience learning, yet their gains are commonly measured with the same automatic signals used during development.Examples include ReAct [Yao et al., 2023], NNetNav, WebRL [Qi et al., 2025], and WebAgent R1 [Wei et al., 2025].Source: §2 · p. 2
- Prior studies report inconsistencies in LLM judges, and AgentRewardBench finds that rule-based evaluators can underreport successful web-agent completion.AgentBoard measures partial progress, while WebGraphEval analyzes recurring paths and critical decisions across trajectories [Ma et al., 2024, Qian et al., 2025].Source: §2 · p. 3
- This work audits the complete WebArena Lite task set, preserving original and corrected outcomes while analyzing first errors, failure patterns, and stepwise progress.Matched conditions examine MASM, Guide Text, and action budget, separating changes in agent behavior from evaluator artifacts.Source: §2 · p. 3
3 Study Scope and Experimental Design
The study evaluates web-agent outcomes and trajectories on a fixed set of 165 WebArena Lite tasks without updating model weights. It compares matched prompt-support and action-budget conditions while retaining detailed execution evidence.
3 Study Scope and Experimental Design
- The study aligns all conditions by task identifier across 165 WebArena Lite tasks, enabling paired analysis of gains, regressions, action counts, and stopping behavior.Runs use the same task definitions, reset website containers and authentication states, and receive a maximum of 15 or 25 actions.Source: §3 · p. 3, §3.1 · p. 3
3.1 WebArena-Lite Evaluation Setting
- Six conditions compare GPT 5.5 with MASM alone or Guide Text plus MASM at 15 and 25 steps, and an untrained Qwen3.5 9B MASM ablation at 25 steps.The Qwen comparison isolates structured memory support without behavior cloning or parameter updates; recorded task, action, page, screenshot, and response evidence is retained.Source: §3.1 · p. 3
3.2 MASM and Guide Text
- MASM analyzes each browser transition against the agent’s intention, then updates explicit memory from recent action records and observed results.The memory records verified information, last-step feedback, estimated progress, next steps, and obstacles for the next action prompt.Source: §3.2 · p. 3
- Guide Text supplies static procedural evidence for planning and memory updates, while action-result analysis remains grounded in the executed transition and environment evidence.It describes navigation routes, required fields, and verification checkpoints without supplying the target answer.Source: §3.2 · pp. 3–4, Figure 1 · p. 4
4 Human-Grounded Verification Protocol
The verification protocol retains automatic scores while checking failed outcomes and reviewing trajectories against task-specific evidence. It then labels consequential errors and estimates verified progress at each step.
4 Human-Grounded Verification Protocol
- The protocol links evaluator false-negative correction, first-error annotation, and step-level progress estimation under a shared evidence hierarchy.Original scores are retained alongside human-corrected outcomes rather than replaced.Source: §4 · p. 4, §4.1 · p. 4
4.1 Automated Evaluation and Human Correction
- A failed run is corrected to success only when task requirements are demonstrably satisfied and the evaluator rejects it for formatting, strict matching, or another documented limitation.Semantically incomplete answers, unexecuted intentions, and claims supported only by the agent’s memory are not corrected.Source: §4.1 · p. 4
4.2 Trajectory Review and Failure Annotation
- Reviewers label the earliest action, omission, or unsupported belief that materially reduces completion chances within the remaining budget, then assign one primary failure mode.The 16-category taxonomy includes navigation and exploration loops, premature answers, invalid actions, incomplete forms, and answer-format mismatches; ambiguous cases receive second review.Source: §4.2 · p. 5
4.3 Step-Level Progress Assessment
- Progress is measured from verified task subgoals after each action, with missing or ambiguous evidence treated as incomplete.Positive changes indicate progress and negative changes record regression; completion is the first step when all required subgoals are verified, and later failure is checked for evaluator error versus completion before stopping.Source: §4.3 · p. 5
5 Results and Analysis
Human review changes measured outcomes, while matched comparisons show that guidance can improve corrected success but also reshuffle which tasks are solved. Failure and step-level analyses reveal why final scores alone miss important differences in agent behavior.
5.1 Overall Performance and Human-Corrected Success
- 63 of 165 tasks (38.18%) were human-verified with 25-step Guide Text and MASM, versus 51 (30.91%) accepted by the original evaluator.For MASM alone, review raises the result from 43 (26.06%) to 57 (34.55%). Across four GPT 5.5 settings, reviewers recover 47 outcomes, or 7.12 percentage points per setting on average.Source: §5.1 · p. 5, Table 1 · p. 6
5.2 Effects of MASM
- MASM memory is a model-generated state summary, not a verifier, so completion claims need checking against environment evidence.Failed runs sometimes report high completion despite a missing click, unsaved form, or absent final answer.Source: §5.2 · p. 6
5.3 Effects of Guide Text
- At 25 steps, Guide Text raises corrected success with MASM from 57 of 165 tasks (34.55%) to 63 (38.18%), with gains and regressions.At 15 steps, corrected success rises from 58 (35.15%) to 61 (36.97%); among tasks with available guides, it rises from 53 (37.06%) to 56 (39.16%).Source: §5.3 · p. 6
- At 25 steps, Guide Text gains action and mixed tasks but loses retrieval and map tasks; benefits cluster where procedural routes are stable.Action tasks gain five successes and mixed tasks three, while retrieval tasks lose two; shopping administration gains four tasks, Reddit two, and GitLab and shopping one each, while map tasks lose two.Source: §5.3 · p. 7
5.4 Task-Level Gains, Regressions, and Efficiency
- 34 of 165 tasks (20.61%) flip outcomes in the 25-step comparison despite a net gain of six; on Guide Text-only successes, mean actions fall from 15.0 to 8.9.The median action count falls from 14 to 7, and runs reaching the 25-action limit fall from seven to one.Source: §5.4 · p. 7
5.5 Failure Modes and Completion Before Stopping
- 32 of 102 failed trajectories (31.4%) involve scroll oscillation; unfinished exploration and premature answering each account for 13 (12.7%).The review also distinguishes reaching the requested state before stopping from ultimately completing the task, since invalid final actions or continued interaction can still lead to failure.Source: §5.5 · pp. 7–8, Table 6 · p. 8
5.6 Step-Level Progress and Trajectory Analysis
- 56.2% was the mean maximum verified progress among 40 analyzable failures that exhausted the action budget, and none reached 100%.The first two actions have the largest mean reported increments, 23.6 and 19.0 percentage points; the paper recommends treating a later plateau as a cue to verify completion or change strategy.Source: §5.6 · p. 8, Table 7 · p. 8, Table 8 · p. 8
5.7 Implications for Reliable Web Agent Evaluation
- The audit recommends reporting original and corrected outcomes together, comparing interventions by paired task flips and action counts, and grounding monitoring in environment evidence.It also calls for repeated-action detection and an explicit completion gate.Source: §5.7 · p. 9
6 Limitations and Threats to Validity
The study’s validity is bounded by single rollouts per task, judgment-based human labels, partial progress records, and nonrandomized Guide Text conditions.
6 Limitations and Threats to Validity
- Each task-condition has one rollout, so paired flips describe observed runs rather than variation across repeated trials.The 15- and 25-action budgets are not directly comparable; strongest intervention claims use matched Guide Text comparisons at the same budget.Source: §6 · p. 9
- Human labels remain judgment based, and structured progress covers only 55.2% of tasks and is used to locate evidence, not as ground truth.Ambiguous cases receive second review under fixed evidence rules.Source: §6 · p. 9
- Guide Text is observational rather than randomized, varies in specificity, and is absent for 22 tasks.The audit flags possible state carryover and guide-related answer leakage, and API behavior, website state, and authentication may vary across reruns.Source: §6 · p. 9
7 Conclusion
The audit finds that evaluator scores miss some completed tasks and that final scores alone do not capture agent behavior. It recommends reporting original and corrected outcomes alongside trajectory evidence.
7 Conclusion
- Human review recovers 47 missed successes across four GPT 5.5 settings, or 5.45 to 8.49 percentage points per setting.The audit assigns 102 failed runs to reproducible causes and finds that high self-reported progress often lacks sufficient environment evidence.Source: §7 · p. 9
- MASM improves the original evaluator score for an untrained Qwen3.5 9B model, while Guide Text changes both GPT 5.5 success rates and the identity of solved tasks.The conclusion reports these comparisons without supplying their numeric values in this passage.Source: §7 · p. 9
- The authors recommend reporting original and corrected outcomes together and preserving trajectory evidence about failures, progress, and stopping behavior.A final benchmark score is useful but does not fully describe task completion or agent behavior.Source: §7 · p. 9
8 Ethical Consideration
The reported study is an academic evaluation and does not use GPT-generated outputs or trajectories to improve another model.
8 Ethical Consideration
- The retained study is evaluation only, using GPT 5.5 outputs to assess the originating API model and analyze its recorded behavior.No GPT-generated output or trajectory is used to train, fine-tune, distill, or otherwise improve another model in the reported experiments.Source: §8 · p. 9
- The records are used solely for this academic evaluation.Source: §8 · p. 9
- The records are not used to develop, train, evaluate, or deploy a commercial product or service.Source: §8 · p. 9
A Complete 165-Task Outcome Matrix
The appendix describes a task-indexed outcome matrix, evidence-based human correction rules, and operational definitions for fine-grained failure modes.
A Complete 165-Task Outcome Matrix
- The matrix organizes six retained conditions by WebArena Lite task and records evaluator outcomes plus human corrections for four GPT 5.5 conditions.Failed audited runs also record the first consequential error, primary failure mode, maximum verified progress, and completion step when present.Source: A Complete 165-Task Outcome Matrix · p. 12
B Human Review Guidelines and Correction Rules
- Human reviewers correct failures only when the requested result is supported by environment evidence or the final answer, checking task state and records before model Memory or Analysis.Ambiguous cases receive a second reviewer rather than relying on model self-report.Source: B Human Review Guidelines and… · p. 12
C Definitions of the Fine-Grained Failure Modes
- The failure taxonomy defines operational categories including scroll oscillation, unfinished exploration, premature answers, incomplete workflows, invalid action formats, and completion before stopping.The primary label is the first cause that materially reduces completion chances; later symptoms remain in the reviewer note.Source: C Definitions of the Fine-Grained… · p. 12
D Per-Task Guide Text Gains and Regressions
This material reports paired Guide Text efficiency, the step-level progress protocol, contrasting audited trajectories, and the review interface.
D Per-Task Guide Text Gains and Regressions
- Among twenty Guide Text-only successes, mean actions fell from 15.0 to 8.9, median actions from 14 to 7, and action-limit cases from seven to one.These efficiency values are computed on the paired flip set, not across all tasks.Source: D Per-Task Guide Text Gains and… · p. 13
E Step-Level Progress Annotation Protocol
- Annotators decompose instructions into checkable atomic conditions, track current and maximum verified progress, and count overrun after first verified completion.Maximum progress preserves evidence of earlier completion even if later actions overwrite a correct state.Source: E Step-Level Progress Annotation… · p. 13
- None of the 40 analyzable failures exhausting the action budget reached 100% verified progress; memory progress selects review steps but does not assign final labels.The seven failed runs with a 100% memory report were reviewed against screenshots, action results, and the task rubric.Source: E Step-Level Progress Annotation… · p. 13
F Representative Audited Trajectories
- Task 35 recovers the answer but has invalid final action formats, whereas Task 149 enters dates but stops before running and checking the report.Task 149 is genuinely incomplete despite substantial reported progress.Source: F Representative Audited Trajectories · p. 13
G Review Interface and Analysis Tool
- The review interface places outcomes, screenshots, action analysis, memory, and final responses together, while filters expose disagreements and unreviewed cases.Reviewers record verdicts, failure modes, first error steps, corrected answers, and notes.Source: G Review Interface and Analysis Tool · p. 13
H Evaluation and Environment Details
The evaluation details specify the task reset procedure, model conditions, and retained study records.
H Evaluation and Environment Details
- All 165 WebArena Lite task identifiers were evaluated after website reset and authentication state recreation.Source: H Evaluation and Environment Details · p. 13
- GPT 5.5 was evaluated in four prompt and budget conditions, while untrained Qwen3.5 9B was tested with and without MASM at 25 steps.The GPT 5.5 conditions were MASM and Guide Text with MASM, each at 15 and 25 steps.Source: H Evaluation and Environment Details · p. 13
- Prompts, action records, screenshots, evaluator outputs, and human corrections were retained; no recorded GPT output was used as training data.Source: H Evaluation and Environment Details · p. 13