PaperlayerView exact revision on arXiv

Source-linked AI summary

WebChallenger: A Reliable and Efficient Generalist Web Agent

Jayoo Hwang, Xiaowen Zhang, Vedant Padwal

arXiv:2606.10423v1cs.CL

TL;DR

Autonomous web agents still struggle with realistic long-horizon tasks, while leading generalist systems are costly because they rely on proprietary reasoning models. WebChallenger addresses this through PageMem-based attention, memory, and compound workflows, achieving state-of-the-art open-model results across four benchmarks without fine-tuning.

  • Problem

    Autonomous web agents remain below human performance on realistic long-horizon tasks, while leading generalist systems rely on prohibitively costly proprietary reasoning models.

  • Method

    WebChallenger uses DOM-derived PageMem with selective observation, reusable website memory, and compound action workflows that generalize without site-specific adapters.

  • Results

    WebChallenger achieves state-of-the-art open-model results on all four benchmarks without fine-tuning, including 56.3% on WebArena.

  • Takeaways & Limitations

    The results support architectural scaffolding as a way to close much of the gap between small open-weight and frontier proprietary agents on long-horizon web navigation.

  • Takeaways & Limitations

    The framework depends on hand-designed structural priors, may degrade on atypical sites, and uses more sequential LLM calls that increase task latency.

Abstract

from arXiv · show

Autonomous web navigation remains challenging for LLM agents, and the strongest generalist systems rely on proprietary reasoning models whose inference cost is prohibitive for the repetitive tasks where such agents would be most useful. We argue this gap stems not from insufficient model capability but from agent architectures that fail to replicate three human cognitive advantages: selective attention to relevant page regions, persistent memory of website structure, and procedural fluency with common interaction patterns. We introduce WebChallenger, a web agent framework that addresses each gap through architecture design rather than model scale, built around PageMem: a structured page representation deterministically constructed from the DOM that exposes each page as a hierarchy of semantic sections with short summaries. On this shared substrate we build three mechanisms that mirror the three cognitive advantages: a divide-and-conquer observation pipeline that lets the agent skim section summaries and extract details only from task-relevant regions; a lightweight exploration and memory system that traverses each website once to build a reusable map of pages and element behaviors; and compound action workflows that collapse common multi-step interactions into single agent actions, handling partial state changes automatically. Because all three operate over PageMem, the framework generalizes across websites without site-specific adapters. Using off-the-shelf open-weight models without fine-tuning, our system achieves 56.3% on WebArena, 48.7% on VisualWebArena, 51.0% on Online-Mind2Web, and 70.9% on WorkArena, approaching frontier proprietary systems at a fraction of the cost. Our code is released at https://github.com/jayoohwang1/webchallenger

1 Introduction

WebChallenger argues that web-agent difficulty reflects architectural mismatches rather than insufficient model capability, and addresses this with PageMem plus attention, memory, and procedural-action mechanisms. Using off-the-shelf open-weight models without fine-tuning, it achieves strong benchmark performance at a fraction of frontier-system inference cost.

  • Core architecture: PageMem deterministically converts the DOM into hierarchical semantic sections with short summaries, providing a shared representation that generalizes across websites without site-specific adapters.The representation supports all three architectural mechanisms uniformly.
  • Selective attention: Divide-and-conquer observation lets agents skim section summaries, select task-relevant regions, and extract details without processing entire pages.This produces information-dense observations while reducing the need to process irrelevant page content.
  • Persistent memory: Lightweight exploration builds a persistent website map by recording PageMems, pages, navigation paths, and interactive-element behaviors before task execution.The system traverses new websites before executing tasks, creating reusable structural memory.
  • Procedural fluency: Compound workflows collapse common multi-step interactions into single agent actions and automatically surface partial state changes such as expanded dropdowns.They implement site-agnostic routines for searching, menu selection, and form submission, dispatched by section type.
  • Results: 56.3% on WebArena, 48.7% on VisualWebArena, 51.0% on Online-Mind2Web, and 70.9% on WorkArena demonstrate state-of-the-art open-weight performance at a fraction of inference cost.Results use an off-the-shelf 32B LLM and 7B VLM without fine-tuning, approaching frontier proprietary systems.

2 Method

WebChallenger organizes web navigation around PageMem, a DOM-derived semantic page representation that preserves browser-control selectors and serves exploration, observation, and action. It combines persistent offline website memory, focused multi-stage observation, and compound workflows to support efficient, reusable interaction without site-specific adapters.

  • Observation: The observation pipeline selects relevant section summaries, extracts details from selected sections, and synthesizes them into a task-focused page summary.This decomposition reduces dilution of task-relevant information from large pages and flattened accessibility trees.
  • Action: Compound actions execute multi-step workflows for interactions such as dropdown selection, form submission, and search, handling intermediate partial state changes automatically.Direct actions advance after a page transition, while workflows invoke additional model calls and browser operations within one high-level agent action.
  • PageMem: PageMem is a hierarchical DOM representation of websites, pages, semantic sections, and interactive elements that preserves selectors for direct browser control.It provides the common interface for exploration, observation, and action, allowing higher-level components to operate on abstract page objects.
  • PageMem: PageSections are constructed by recursively splitting the DOM, grouping repeated sibling structures, assigning clickable elements to ancestor sections, and generating short summaries.The construction uses size thresholds, grouping tags, and clickable-element heuristics adapted from BrowserUse.
  • Persistent website memory: Offline exploration deterministically traverses each website before task execution to build reusable WebsiteMem containing PageMems, page templates, and element behaviors.The memory requires no LLM guidance, demonstrations, or external resources, amortizing environmental knowledge upfront at a fixed one-time cost.

3 Experiments

WebChallenger achieves strong zero-shot results across four web-navigation benchmarks, setting new open-model state-of-the-art results while approaching proprietary systems. Ablations show that observation, compound actions, and memory each contribute to accuracy and efficiency, with observation having the largest accuracy impact.

  • Benchmark results: 56.3% on WebArena exceeds Mobile-Agent-v3.5’s 48.4% by 7.9 points and surpasses ScribeAgent’s 53.0%.WebChallenger sets a new open-model state-of-the-art result without fine-tuning.
  • Benchmark results: 48.7% on VisualWebArena outperforms all open-model baselines and trails only WALT’s 52.9%, while 70.9% on WorkArena exceeds Claude 3.5 Sonnet’s 56.4% and GPT-4o’s 45.5%.Online-Mind2Web reaches 51.0%, showing generalization through structural patterns shared across the web.
  • Backbone model comparison: 58.8% on WebArena-lite is close to the full WebArena score of 56.3%, while GLM-4-32B in GenericAgent scores 19.4% versus 58.8% in WebChallenger.GPT-5 reaches 68.7% and GPT-4o-mini reaches 46.7% within the framework, showing backbone sensitivity and retained performance with a weaker backbone.
  • Component ablations: Removing the observation pipeline lowers accuracy by 17.6 points, compared with 9.7 points for compound actions and 7.6 points for memory.Compound-action removal has its largest effect on CMS, causing a 20.0-point drop; removing memory has no effect on Reddit, where both conditions score 71.4.
  • Token and step efficiency: Removing observation reduces total tokens from 47.0M to 36.0M but increases average prompt size from 1850 to 8793 tokens and steps from 7.2 to 11.26.Removing compound actions raises total tokens to 64.9M and steps to 9.85, demonstrating their efficiency benefit.

4 Related Work

WebChallenger’s approach differs from prior web-agent work through deterministic site-map construction, cross-site compound workflows, and structured observation refinement over PageMem. These mechanisms avoid task-experience requirements and per-site adaptation while addressing limitations of existing memory, action-space, and observation strategies.

  • Agent Memory: WebChallenger builds a structured site map through deterministic exploration without task experience, demonstrations, or documentation.This complements prior methods that accumulate insights from task trajectories.
  • Web Action Space: Its compound workflows operate over PageMem’s abstract elements and sections, generalizing across sites without per-site adaptation.Prior higher-level action approaches typically learn site-specific code.
  • Observation Refinement: Existing web-agent observations use text, screenshots, or both, but these modalities are token-heavy and information-sparse, motivating refinement strategies.Text-based agents commonly prune irrelevant HTML elements as one refinement approach.

5 Conclusion

WebChallenger argues that small open-weight models can approach frontier proprietary systems on long-horizon web navigation when supported by architecture that supplies selective attention, persistent memory, and procedural fluency. Its framework provides these capabilities through divide-and-conquer observation, offline exploration and memory, and compound actions.

  • WebChallenger closes much of the gap between small open-weight models and frontier proprietary systems on long-horizon web navigation.
  • The framework argues that current LLMs already possess sufficient intelligence for many common web tasks.
  • WebChallenger scaffolds this intelligence with selective attention, persistent memory, and procedural fluency through three architectural mechanisms.These mechanisms are a divide-and-conquer observation pipeline, an offline exploration and memory system, and compound actions.

A Implementation Details · A.1 PageMem

PageMem is constructed in two stages that first recover the page’s structural skeleton and then populate it with interactable elements and summaries. This process yields both section-level and page-level representations.

  • A.1 PageMem: PageMem construction uses DIVIDEPAGE to recursively partition the live DOM into ordered empty PageSections, followed by UPDATEPAGEMEM to add interactable elements and LLM-generated summaries.UPDATEPAGEMEM also generates the page-level summary after populating the sections.

A.1.1 Memory Structure · A.1.2 Page division.

WebChallenger organizes website memory hierarchically from websites to pages, sections, and elements, combining immutable DOM attributes with summaries and mutable agent state. Its page-division procedure recursively splits the DOM into meaningful semantic, visual, or repetitive groups, including list sections formed from repeated siblings.

  • A.1.1 Memory Structure: WebsiteMem stores encountered pages by URL, list-page templates for structural matching, and all encountered elements for deduplication.It is defined as Mw = (Pw, Tw, Ew), where Pw maps URLs to PageMem objects.
  • A.1.1 Memory Structure: A PageMem records the URL, title, VLM-generated page summary, ordered sections, and page-level agent state.Mutable state includes extracted information and the agent’s past interaction history on the page.
  • A.1.1 Memory Structure: PageSections contain summaries, ordered elements, optional list subsections, DOM-derived selector attributes, and mutable task state.Mutable section state can include task-relevant extractions, image descriptions, and a staleness flag for changed DOM subtrees.
  • A.1.1 Memory Structure: Elements retain DOM-derived identifiers and interaction attributes, dropdown-item structure, and mutable state such as current input values and click flags.Element attributes include id, tag, class, role, label, and type.
  • A.1.2 Page division.: DIVIDEPAGE recursively traverses the DOM and appends terminal nodes as ordered sections, producing a PageMem with the current URL, extracted title, and section list.The procedure returns the page’s ordered section list as its structural skeleton.
  • A.1.2 Page division.: Groups of ≥4 consecutive siblings sharing tag and class are replaced by a single list-section node before recursive splitting.GROUPSIBLINGS scans sibling sequences and returns the shortened sequence after these replacements.
  • A.1.2 Page division.: A node is terminal when it is a list section, has a grouping tag, or is not oversized under the specified rendered-size thresholds.ISTERMINAL returns v.isListSection ∨ v.tag ∈ Tgroup ∨ ¬OVERSIZED(v).
  • A.1.2 Page division.: The grouping tag set includes details, p, img, embed, code, group, nav, header, and footer, while dimensions use rendered CSS-pixel bounding boxes.The browser’s layout engine supplies each node’s height and width.

A.1.3 PageMem update. · A.2 Exploration

PageMem is refreshed at every observation step by updating section elements, detecting state diffs, and selectively regenerating summaries. Exploration then deterministically traverses website pages and clickable elements, reuses structural templates, records interaction outcomes, and restores state between clicks.

  • A.1.3 PageMem update.: PageMem is refreshed at every observation step, including initial element and summary population for newly divided pages.UPDATEPAGEMEM updates each section and supplies the live representation used by observations and workflows.
  • A.1.3 PageMem update.: UPDATESECTION computes added, removed, and modified-element diffs, re-summarizing sections when summaries are absent or structural changes are sufficiently large.The routine returns the diff so observations and workflows can respond to partial state changes.
  • A.1.3 PageMem update.: GETELEMENTS resolves section locators, filters descendants through an accessibility-and-visibility clickable predicate, and constructs elements from DOM attributes.Interactable nodes must pass the gate and satisfy a positive signal such as an interactable tag, event attribute, ARIA role, or pointer cursor.
  • A.2 Exploration: Exploration builds WebsiteMem through deterministic depth-first traversal of pages and clickable elements, deduplicating globally seen elements and restoring pre-click state by reloading URLs.Per-page, total-page, and per-website timeout budgets provide additional early-return checks.
  • A.2 Exploration: EXPLOREPAGE navigates to unexplored URLs, constructs and updates full PageMem representations, registers pages, and skips iteration when their section structures match known templates.Template matching uses equal section counts and structural equivalence of corresponding DOM-derived section attributes.
  • A.2 Exploration: ITERATEPAGE explores non-list elements once globally and handles list sections through a representative list-item traversal that promotes resulting pages to templates.List-item exploration avoids redundantly revisiting structurally identical neighbors.
  • A.2 Exploration: EXPLOREELEMENT skips unsafe or unhelpful targets, records same-site navigations as page stubs, recursively explores newly revealed dropdown elements, and restores the pre-click URL.The skip filter excludes off-site, authentication, tel/mailto/print, and persistent-state-mutating links or buttons.

A.3 Observation Pipeline

The observation pipeline selects relevant PageMem sections, extracts task-specific details, and synthesizes a page-level summary. It handles long lists through chunked selection with early termination and reuses unchanged section extractions while regenerating the page summary each call.

  • Detail extraction and summarization: ANALYZEPAGE extracts details from selected sections and passes the resulting strings to a page summarization step.It invokes SELECTLISTITEMS for list sections, formats each section, calls LLMEXTRACTDETAILS, and then calls LLMSUMMARIZEPAGE.
  • Per-section detail extraction: 50 x 50 pixels is the minimum image size for including URLs and VLM-generated image descriptions in normal-section extraction details.Normal sections include the accessibility subtree plus qualifying image information; list sections instead format selected items.
  • List item selection: List sections are processed in fixed-size sequential chunks, with LLM selection and an explicit early-termination check.The selector tracks searched indices and selected items while deciding whether remaining entries need processing.
  • Summary caching: Section summaries persist across tasks, per-section task extractions persist while unchanged during a task, and page-level task summaries regenerate on every ANALYZEPAGE call.Per-section extractions are cached with the details string that produced them, whereas page summaries can change as task history progresses.

A.4 Agent Loop

AGENTLOOP runs one timestep at a time, combining PageMem-based observation, candidate assembly, action selection, execution, and end-task verification until completion or the step budget is exhausted. Same-URL partial actions trigger a continuation phase that refreshes page state and selects follow-up actions without restarting the full observation pipeline.

  • Observation phase: At each timestep, AGENTLOOP retrieves or constructs PageMem, detects modals, selects relevant sections, and analyzes them into a task-focused observation.A detected modal bypasses section selection and becomes the sole relevant section.
  • End-task verification: End-task selections receive one verification check per task; verified completion produces LLMFINALANSWER, while failed verification removes end-task for that timestep and re-prompts.The end-task action remains available on subsequent timesteps after failed verification.
  • Action phase: GATHERCANDIDATES combines elements selected from relevant sections with coverage of the first five non-selected sections, while filtering navigation actions and retaining end-task.The first-five heuristic preserves access to upper-page navigation bars, search boxes, and primary buttons.
  • Action phase: The action selector returns an action and reason, and execution dispatches navigation, form, element, or end-task choices to their corresponding workflows.Navigation uses NAVIGATE; form elements use SUBMITFORM; other elements use ELEMENTACTION.
  • Intra-step continuation: Same-URL partial actions enter intra-step continuation, refreshing PageMem, incorporating a VLM screen-difference description, and selecting follow-up actions until navigation, a modal, end-task, or five actions ends continuation.The continuation phase avoids restarting the observation pipeline for follow-up actions on the same page.

A.5 Action Workflows · A.5.1 Action Logging and History Format

WebChallenger dispatches element actions through structured workflows that distinguish forms, element types, dropdown behavior, and common input operations. Successful basic actions are logged after execution in timestep histories, with compound actions represented as action lists.

  • A.5 Action Workflows: Element actions route form-contained elements to SUBMITFORM and all others to ELEMENTACTION, while navigation and end-task actions remain in the agent loop.ELEMENTACTION dispatches by tag, role, DOM attributes, and behaviors recorded during exploration.
  • A.5 Action Workflows: ELEMENTACTION prioritizes explored dropdown behaviors, input-specific handlers, popup or unexplored signals, and finally a plain click.Handlers include file upload, select or combobox selection, search, radio or checkbox clicks, and copy-to-clipboard.
  • A.5 Action Workflows: SUBMITFORM selects fields with an LLM, fills them through type-specific actions, repairs empty-required or aria-invalid fields, and then reviews the form.The review loop permits up to Kmax = 15 iterations and can submit, continue with an element action, exit, or stop when the URL changes.
  • A.5 Action Workflows: DROPDOWNACTION clicks a trigger, compares PageMem sections, returns after navigation or no disclosure, and otherwise either submits a revealed form or clicks an LLM-selected element.A revealed cluster is treated as a form when it contains at least two input-like elements and a submit-like element.
  • A.5 Action Workflows: SEARCH enters text, optionally selects a revealed suggestion, and presses Enter, while ENTERINPUT and UPLOADFILE respectively fill fields and present file choices.These workflows are part of the per-element-type action routing.
  • A.5.1 Action Logging and History Format: Every successful basic action contributes a post-execution string to history, whereas failed actions are omitted and only the eventually successful action is logged.Post-execution construction supports formats requiring realized values, such as entered text or the selected option.
  • A.5.1 Action Logging and History Format: Compound actions and continuation chains emit multiple basic-action strings under Actions:, while single-action timesteps use Action: within blocks containing observation, summary, reason, and action fields.The page name and URL come from PageMem, the task summary from ANALYZEPAGE, and the reason is generated during action selection.

B Additional Experiment Details

This section specifies exploration limits, benchmark-specific interaction handling, and largely heuristic hyperparameter choices. It also describes URL substitution, multi-website selection, and image grounding procedures used for benchmark evaluation.

  • Exploration parameters: Exploration is capped at 75 clickable elements per page, 500 pages per website, search depth 2, and a 12-hour wall-clock timeout.After timeout, the partial WebsiteMem is used as-is; Online-Mind2Web uses depth 1 across 136 distinct websites.
  • URL replacement on WebArena and VisualWebArena: Bidirectional URL substitution maps simulated benchmark URLs to real site names for the LLM and maps the LLM’s real URLs back to simulated sites.This addresses confusion caused by locally hosted WebArena and VisualWebArena environments whose instructions use real website names.
  • Multi-website selection (WebArena): At task start, WebArena agents select relevant additional websites, whose homepages are added to bookmark set Bτ as one-click navigation actions.The agent receives the full benchmark website list and may select sites beyond the starting URL.
  • Input image grounding (VisualWebArena): For VisualWebArena image tasks, the VLM describes the input images in relation to the task, appending that description to the task instruction throughout execution.The prompt includes the task instruction, input images, and current page screenshot at task start.
  • Hyperparameters: Hyperparameters were chosen largely as heuristic defaults and were not extensively swept because pilot runs showed no strong sensitivity.Table 7 consolidates configuration values across system components, with benchmark-specific exploration overrides noted parenthetically.

B.1 Compute Cost Estimates · C Broader Impacts

Experiments ran locally for roughly 23 days across four benchmarks, with estimated electricity costs of about $23. The work highlights affordability, privacy, and accessibility benefits of locally runnable open-weight web agents while acknowledging lowered misuse barriers.

  • B.1 Compute Cost Estimates: Experiments used a Ryzen 5 3600 CPU, NVIDIA RTX 3090 GPU, and 64GB RAM with local vLLM inference.The setup ran inference locally on a desktop machine.
  • B.1 Compute Cost Estimates: ~7 days, ~8 days, ~3 days, and ~2 days were required for WebArena, VisualWebArena, Online-Mind2Web, and WorkArena, respectively.These durations sum to roughly 20 days of benchmark execution.
  • B.1 Compute Cost Estimates: $1.15 per day was the estimated electricity cost based on system power draw and regional electricity prices.The estimate was derived from the desktop system’s power consumption and local electricity pricing.
  • B.1 Compute Cost Estimates: $23 was the reported total electricity estimate for the experiments.The passage reports this figure after estimating approximately $1.15 in electricity per day.
  • C Broader Impacts: Locally runnable open-weight models can make tedious web-task automation economical at scales where frontier-model APIs would not.The broader-impact argument links local execution and smaller models to lower automation costs.
  • C Broader Impacts: Sensitive browsing sessions need not leave the user’s device when agents run locally.The passage identifies privacy as a positive implication of the approach.
  • C Broader Impacts: Locally runnable open-weight agents make reproducible research more tractable for groups without large compute budgets.The passage frames improved research accessibility as another positive implication.
  • C Broader Impacts: Lowering the barrier to capable web agents also lowers the barrier for misuse such as spam.The passage explicitly presents misuse as a potential negative consequence.

D Limitations · E Prompts

The paper notes that WebChallenger depends on hand-designed structural and interaction priors, which may reduce performance on atypical websites. Its prompts operationalize the agent loop through structured selection, observation, navigation, completion, and final-response instructions.

  • D Limitations: The framework uses hand-designed DOM decomposition, clickable-element heuristics, deterministic exploration, and fixed compound-action workflows.These components encode structural priors about typical web-page organization.
  • D Limitations: Performance may degrade on websites that diverge significantly from common web patterns.The implementation is described as generally robust across a wide range of websites, but not universally robust.
  • E.1 Observation Prompts: The observation prompts first select potentially relevant page sections from task, history, page, and section summaries.The selected sections are returned as comma-separated integer indices for further analysis.
  • E.1 Observation Prompts: Section-analysis prompts summarize potentially relevant information while avoiding premature next-step instructions and acknowledging uncertainty.They provide task, history, page, section, and content fields, then record noteworthy details in a labeled format.
  • E.1.1 List item selection prompts: List-item prompts select all potentially relevant items, including partial or closest matches when exact matches are unavailable.They also support selecting uncertain candidates for additional inspection and continuing searches when more matches may remain.
  • E.2.1 Agent Loop Prompts: Agent-loop prompts identify relevant pages, clickable elements, and navigation actions using task instructions, history, and current-page observations.The action-selection prompts require choosing one or more candidate pages or elements, while the final navigation prompt selects a next action.
  • E.2.1 Agent Loop Prompts: Completion prompts keep task status false whenever required steps remain, including full checkout for purchase tasks.After completion, a separate prompt generates a user-facing message and final answer from the task history and current page information.
Loading 2606.10423v1…