Source-linked AI summary
WebFovea: When the Model Is Right but the Click Is Wrong -- Reliable Round Trips for Vision-Based Web Agents on Live Websites
Jiangang Han
TL;DR
Live-site retrieval requires agents to navigate a website’s own interface and return verifiable answers, yet a capable model alone is not sufficient. WebFovea treats each step as a four-stage round trip with guardrails. Across four submissions, its score rose from 31.0 to 57.0, placing it second in the challenge.
Problem
WebRetriever tasks require agents to navigate live websites and return exactly checkable answers, and navigation success alone does not predict task completion.
Source: §1 · p. 1Method
WebFovea hardens parsing, execution, feedback, and observation in each agent step and surrounds the loop with guardrails.
Source: §1 · p. 1, §4.1 · p. 3Results
Takeaways & Limitations
The four-stage view provides a model-agnostic way to analyze failures in how an agent’s decisions are carried out and reported back.
Source: §1 · p. 1, §7 · p. 8Component evidence comes from targeted sets of 1–24 tasks and indicates direction rather than statistical significance.
Source: §6.2 · p. 8
Abstract
from arXiv · showhide
We present WebFovea, a vision-based web agent that placed 2nd in the WebRetriever Challenge 2026 with a final score of 57.0 out of 100. The challenge evaluates agents end to end on Protocol III of the WebRetriever benchmark (arXiv:2607.06118): starting from an entry URL on a live website, the agent must operate the site's own interface and return a verifiable answer. A capable multimodal large language model (LLM) is necessary for this, but not sufficient. The model's decisions reach the browser through the harness, the code between the model and the page. At every step, four things must go right: the model's reply must be parsed into the intended action, the action must take effect on the page, the result must be reported back accurately, and the model must be shown the information it needs. On real websites, many of the failures we observed occurred at one of these four stages rather than in the model's reasoning. A coordinate-space mismatch placed every click at 3/4 of its intended coordinates; actions on native dropdowns, inside iframes, and in text boxes failed silently; and self-generated chat-template tokens contaminated 4.9% of task episodes. WebFovea hardens each stage and surrounds the loop with guardrails that keep the agent within the rules and its budget. The four-stage view does not depend on the model, although some individual fixes do. Because we used the same model in all four submissions, the rise of our official hidden-set score from 31.0 to 57.0 reflects changes to the harness, up to run-to-run variance on live sites. We describe the design, the evidence for each component (including negative results), a failure analysis, the limitations, and a roadmap that includes routing different steps to different models. Code is available at https://github.com/jianganghan/WebFovea.
1 Introduction
WebRetriever tasks require agents to reach live-site evidence through the interface and return an exactly checkable answer; navigation success alone does not ensure task completion. WebFovea analyzes each agent step as a four-stage round trip, with guardrails around the loop.
1 Introduction
- Reliable retrieval requires both an answer grounded in authoritative on-screen evidence and actions that reach the intended page elements.The benchmark reports that navigation success alone is an insufficient predictor of task completion [1].Source: §1 · p. 1
- WebFovea treats each step as a round trip through parsing, execution, feedback, and observation, with guardrails around the loop.The four stages help diagnose failures that can resemble reasoning errors even when the model's decision is right.Source: §1 · p. 1
- WebFovea1 placed second in the 2026 WebRetriever Challenge with 57.0/100; across four submissions, harness changes—not model changes—raised its score from 31.0 to 57.0, with no manual-review points lost.Source: §1 · pp. 1–2
- The paper presents paired evaluations, case studies, negative results, limitations, and a roadmap that includes routing steps to different models and classical vision tools.Source: §1 · p. 2
2 Related Work
Prior web-agent work spans benchmarks on self-hosted, offline, and live websites, as well as vision- and DOM-based agent designs. WebFovea focuses on the interface that carries a given model’s decisions to the page and back.
2 Related Work
- WebRetriever scales live-site evaluation to 1,550 tasks on 800 websites and finds that navigation success alone is insufficient to predict task completion [1].Its Protocol III evaluates end-to-end retrieval; NavEval agrees with human experts 91.2% of the time.Source: §2 · p. 2
- Vision-based agents act through screenshot coordinates, while DOM-based agents serialize page elements; Set-of-Mark prompting overlays labels on screenshots.Examples include Claude computer use, Browser Use, and Set-of-Mark prompting [14].Source: §2 · p. 2
- WebFovea studies how the model–environment interface affects web-agent performance, an analogous question to SWE-agent’s study of software-engineering agents [15].Source: §2 · p. 2
- Table 1 lists human-verified Protocol III leaderboard baselines, which the paper says are not directly comparable to challenge scores.Source: Table 1 · p. 2, §2 · p. 2
3 Task Setting
Protocol III tasks start from an entry URL and require operating a live website to return an answer. The agent acts through a constrained browser setup, while each step is framed as a round trip bounded by guardrails.
3 Task Setting
- Protocol III requires step-by-step interaction with a live site and a final answer, with no partial credit and validity rules against bypassing the interface or using cached answers.The 100 public tasks were split into a 70-task development set and a 30-task holdout set.Source: §3 · p. 2
- The agent drives Chrome over CDP at a 1920×1080 viewport, uses an approved model, and operates under a fixed evaluation-run time budget.Tasks cannot be retried or reset once started, although individual steps may be retried after a failed model call.Source: §3 · p. 2
- The action set is closed, hard constraints apply, and the executor enforces budgets; task retries are disallowed while model-call retries are allowed.Source: §3 · pp. 2–3
- Each step passes through parsing, execution, feedback, and observation; the step succeeds only if all four stages work.Guardrails bound the loop.Source: Figure 1 · p. 3, Table 2 · p. 3
4 The WebFovea Agent
WebFovea treats each model–browser step as a four-stage round trip—parsing, execution, feedback, and observation—surrounded by guardrails for rules, budgets, and infrastructure faults. Its implementation hardens each stage against failures that can otherwise look like reasoning errors.
4.1 Four stages and guardrails
- A step succeeds only when parsing, execution, feedback, and observation all work; guardrails separately limit permitted round trips and protect against infrastructure faults.The agent receives screenshots and acts through pixel coordinates; a single executor function is the only place it can affect the browser.Source: §4.1 · p. 3
4.2 Parsing: reading the model’s reply as intended
- Self-generated chat-template tokens contaminated 15 of 305 episodes; stripping them changed all 551 contaminated replay steps and none of 7,154 clean steps.Twelve of 13 scorable contaminated episodes failed, versus 7.7% success for contaminated episodes and 41.7% for clean ones; that comparison is observational.Source: §4.2 · p. 4
4.3 Execution: making actions land
- On 24 paired dev tasks, coordinate alignment raised success from 8/24 to 20/24, with 12 fixed and none broken.The fix downsizes screenshots to at most 1440×810 and maps coordinates using the sent-image-to-viewport ratio; it also adds direct selection, reliable text filling, Enter handling, and an iframe DOM fallback.Source: §4.3 · pp. 4–5
4.4 Feedback: reporting what actually happened
- DOM fingerprints include child frames and text hashes for hovers and SVG/canvas targets, while action feedback reports entered values or available select options.Accurate feedback alone did not change the model’s behavior, so the authors pair it with executor-enforced budgets.Source: §4.4 · p. 5
4.5 Observation: showing the model what it needs
- Observation tools expose information unavailable in the screenshot: hover reveals tooltip values, find_text scrolls matches into view, and read_text verifies nearby DOM text.Prompt rules specify when to use tools; on six targeted tasks, find_text use rose from two tasks to three, while read_text fixed none.Source: §4.5 · p. 5
4.6 Guardrails: rules, budgets, and robustness
- The executor rejects prohibited navigation and enforces 80 steps and 1,800 seconds per task, while retrying transient model-call failures up to six times within 240 seconds.Hard constraints require use of the site’s interface and answers visible on the current screenshot; tasks are not retried after starting.Source: §4.6 · p. 5
5 Experiments
Experiments track the official score trajectory, test components on small targeted sets, analyze errors on live sites, and report runtime cost.
5.1 Setup
- 52.0% locally versus 46.0% officially on 25 scorable holdout tasks; the 6-point gap was within one binomial standard deviation.Sixteen of the 100 sites were unreachable from the authors’ network.Source: §5.1 · p. 6
5.2 Official score trajectory
- +10 points at the second official submission came mostly from finishing more tasks: roughly 7 points from completion and 3 from accuracy.The first run finished 79 of 100 tasks; the second finished 97. Changes were bundled, so gains cannot be attributed to individual components.Source: §5.2 · p. 6
5.3 Evidence for each stage
- Hover solved 4 of 5 tooltip tasks, raising the step limit from 40 to 80 increased solved tasks from 0–1 to 4, and reloads recovered two stuck tasks without harm.The targeted comparisons were small and intended to indicate direction, not statistical significance.Source: §5.3 · p. 6
5.4 Negative results
- Feedback notes left 10 idle-looping tasks at 0/10 with 400 total steps; an idle breaker recovered none, and read_text fixed none of five tasks.The idle breaker also terminated a task that would otherwise have succeeded.Source: §5.4 · p. 6
- For fill, repeated inputs stayed at 6, while the observed improvement came through 12% fewer steps; the earlier 40→65 test was outdated.The repeated 40→80 comparison on current code showed a gain; the version-level comparison had alignment as its main change and retained the goto shortcut in the baseline.Source: §5.4 · p. 7
5.5 Failure analysis
- Thirteen public-task references no longer matched live data or metric definitions, 16 sites were unreachable, and remaining agent errors involved capability or decision gaps.The count of correct answers across tasks’ best outcomes is an upper bound for any single version; the authors suggest timestamped evidence screenshots for benchmark references.Source: §5.5 · p. 7, Table 6 · p. 7
5.6 Cost
- The average step used about 10.6k input and 127 output tokens, with input exceeding output 83-fold; a 100-task run cost about $170–230.The repeated transmission of the five most recent screenshots accounted for almost all of the input-token load.Source: §5.6 · p. 7
6 Discussion
The discussion offers design principles for diagnosing and hardening web-agent loops, qualifies the evidence, and outlines future directions.
6.1 Design principles
- Diagnose the failing round-trip stage, keep action definitions consistent across prompt, parser, executor, and feedback, enforce budgets in the executor, use rule-based triggers, and add compound actions to remove silent failures.The authors distinguish an action’s mechanism from its measured outcome; fill was added because screenshot-observed keyboard focus could not be verified.Source: §6.1 · p. 7
6.2 Limitations
- Evidence uses one model, lacks a hidden-set reference baseline, relies on 1–24-task component samples, varies across live sites, and uses public development tasks.The authors say component results show direction rather than significance and local numbers are not comparable to leaderboard scores.Source: §6.2 · pp. 7–8
6.3 Future work
- Future work includes routing steps among models and tools, computer-vision feedback, an independent verifier, native API use with caching, hybrid observations, file parsing, task skills, and synthetic-task fine-tuning.Native API use with prompt caching is estimated to cut cost by 55–70%; the hybrid observation proposal is to be A/B-tested.Source: §6.3 · p. 8
7 Conclusion
The authors conclude that live-site agent failures often arise in how decisions reach the page and how page responses return to the model. They recommend checking those links before blaming the model.
7 Conclusion
- Across four submissions, WebFovea’s score rose from 31.0 to 57.0 and placed 2nd; the authors say the four-stage view does not depend on the model.Their practical advice is to check that decisions reach the page and the page’s response reaches the model.Source: §7 · p. 8
Disclosure of AI Use
The author reports using Claude Code to implement and test components, analyze logs, and help draft the report, while retaining responsibility for the research decisions and final content.
Disclosure of AI Use
- Claude Code assisted implementation, testing, log analysis, and drafting; the author made the research decisions, reviewed the work, and takes responsibility for the content.The author says they conceived the approach, designed experiments, judged results, and checked the agent’s behavior against trajectories.Source: Disclosure of AI Use · p. 8
A Component Inventory
Table 7 inventories WebFovea’s components by pipeline stage and origin, including briefly mentioned components and inherited components left unchanged.
A Component Inventory
- Table 7 maps each WebFovea component to its stage and identifies whether it was inherited unchanged, modified, or newly added.The table also covers components mentioned only briefly in the main text.Source: A Component Inventory · p. 10
- Back, reload, click_many, and the task dispatcher appear in the inventory despite only brief mention in the main text.Source: A Component Inventory · p. 10
- The inventory leaves unchanged the CDP connection and authentication, click/scroll/drag primitives, randomized keystroke delay, and no-cache request headers.Source: Table 7 · p. 10