Surfer 2: The Next Generation of Cross-Platform Computer Use Agents

1 Introduction

• Prior agents often require environment-specific interfaces, while Surfer 2 uses purely visual interaction across desktop, web, and mobile.Examples of prior adaptations include DOM parsers, accessibility trees, and specialized desktop APIs.Source: §1 · p. 3
• Surfer 2 separates strategic planning from tactical execution and uses adaptive validation to continue, replan, or terminate tasks.Its Orchestrator can be bypassed for simple tasks; the Navigator acts through screenshot-based reasoning and pixel-level controls.Source: §1 · p. 3
• Without task-specific fine-tuning, Surfer 2 achieves state-of-the-art results on four benchmarks and surpasses the human baseline on OSWorld and AndroidWorld.The benchmarks span web, desktop, and mobile environments.Source: §1 · p. 3

2.1 Agents for GUI Control

• GUI control has progressed from scripts and rules through reinforcement learning and computer vision to LLM- and VLM-based agents.GUIs are presented as a general control surface when tools or APIs are unavailable.Source: §2.1 · p. 3, §2 · p. 3

2.1.1 Browser Use Agents

• Browser agents often parse DOMs [18] [19] [12], but text-based methods struggle with visually rich, dynamic pages; Surfer 2 uses raw screenshots and pixel coordinates [9].Set-of-Marks instead overlays labeled bounding boxes [11].Source: §2.1.1 · pp. 3–4

2.1.2 Desktop Computer Use Agents

• Desktop work includes Agent S3’s behavior best-of-N selection over summarized trajectories [34], and it can also invoke coding agents for programmatic edits.The passage situates Agent S3 among later systems expanding on early open-source efforts.Source: §2.1.2 · p. 4

2.1.3 Mobile Use Agents

• Mobile-agent research has developed specialized models for interpreting mobile UIs and executing gestures, while Surfer 2 uses orchestration of existing models across platforms.The passage also identifies AndroidWorld and AITW as mobile-agent research resources [14] [35] [36] [37].Source: §2.1.3 · p. 4

2.2 Frontier Models for Agents

• Rather than scaling or fine-tuning models, Surfer 2 uses frontier models and a carefully designed system to target state-of-the-art performance.The related work names GPT-4.1 [5], o3, Claude 4.5 Sonnet, and Gemini 2.5 [6] as examples of frontier models.Source: §2.2 · p. 4

2.3 Localization Models

• Visual grounding maps a high-level action, such as clicking Submit, to precise screen coordinates; Surfer 2 uses Holo1.5 for this localization.The passage identifies UI-TARS [27] and CogAgent [19] as other localization models.Source: §2.3 · pp. 4–5

2.4 Agent Architectures and Learning Paradigms

• Hierarchical planning and execution have precedents [8] [43] [44], including K²-Agent’s training-free planner and learning-based executor.Surfer 2 adds persistent environment state and validation mechanisms to support consistency and recoverability across extended workflows.Source: §2.4 · p. 5
• Surfer 2 performs no parameter updates, using system-level coordination and self-correction instead of weight updates in several related approaches [49] [50] [51] [52] [53] [54] [41].The passage relates its feedback loop to autonomous evaluation [46] [10] and linguistic feedback or reflective episodic memory [47].Source: §2.4 · p. 5

3.1 Design Philosophy

• Five design principles combine separated planning and execution, hierarchical context, shared environment state, multistage validation, and chain-of-thought reasoning outside the Localizer.The context mechanism provides relevant global information while preserving each component’s scope-specific focus.Source: §3.1 · p. 5

3.2 Orchestrator

• The Orchestrator decomposes goals into verifiable subtasks, tracks progress and interaction history, evaluates outcomes, and replans when needed.It is triggered for complex multistage tasks, while simple tasks go directly to the Navigator.Source: §3.2 · pp. 5–6

3.3 Navigator

• The Navigator uses screenshots and its trajectory to produce a note, thought, and action, then executes grounded controls and awaits a new screenshot.A Validator checks answer actions before termination and can return the Navigator to execution with feedback.Source: §3.3 · p. 7

3.4 Localizer

• The Localizer converts textual UI references into precise screen coordinates, such as turning “blue submit button” into clickable coordinates.Surfer 2 uses Holo1.5 models as its Localizer.Source: §3.4 · p. 7

3.5 Validator

• The Validator checks the task specification, execution trace, proposed answer, and recent screenshots against observable evidence before accepting termination.At the Orchestrator level, its assessment is combined with the Navigator’s report to accept, refine, or replan the outcome.Source: §3.5 · p. 7

4 Evaluation Methodology

• Experiments use WebVoyager, WebArena, OSWorld, and AndroidWorld without task-specific fine-tuning to assess the hierarchical architecture separately from model improvements.The suite covers web, desktop, and mobile environments.Source: §4 · p. 7

4.1 Benchmarks

• The benchmarks test multistep reasoning, realistic environments, precise visual grounding, and distinct interaction modes across mouse, keyboard, and touch.They use actual websites, production desktop applications, and authentic mobile apps rather than simulators.Source: §4.1 · p. 8
• WebVoyager evaluation uses the curated 590-task subset established by Magnitude, since changes to live websites can make tasks obsolete.The original formulation contains 643 tasks across 15 websites.Source: §4.1 · p. 8

WebArena

• WebArena comprises 812 navigation tasks across web environments including e-commerce, a social forum, GitLab, a content management system, and a map interface.The evaluation also includes refinements to the original implementation.Source: WebArena · p. 8

OSWorld

• OSWorld evaluation uses 369 real Ubuntu tasks in the Foundation E2E GUI category, restricting actions to human-performable GUI operations without APIs or code execution.This excludes the broader All category, which permits code-level operations.Source: OSWorld · p. 8

AndroidWorld

• AndroidWorld covers 116 tasks across Android and 20 real-world apps, scored by ADB-based state inspection; Surfer 2 relies on screenshots rather than the available accessibility tree.Scores are 0, 0.5, or 1 for failure, partial success, and success, respectively.Source: AndroidWorld · p. 8

4.2 Configuration

• Configuration varies by benchmark: web tasks use the Orchestrator, while OSWorld and AndroidWorld use only the Navigator; OSWorld and AndroidWorld also have step limits.The limits are 15–100 steps for OSWorld and 150 for AndroidWorld.Source: §4.2 · pp. 8–9

5.1 WebVoyager

• 97.1% success on WebVoyager surpasses the previous best of 93.9%, with 100% pass@10.Performance was consistent across nearly all tested websites; Cambridge Dictionary was an exception because anti-bot measures hindered execution.Source: §5.1 · p. 9, Figure 3 · p. 9

5.2 WebArena

• 69.6% pass@1 on WebArena establishes a new state of the art, with stronger results on Reddit (77%) than shopping sites (58%).Many tasks in the domain remain challenging.Source: §5.2 · p. 9, Figure 4 · p. 10
• 84.9% pass@10 on WebArena improves on 69.6% pass@1, primarily by expanding task coverage rather than single-run reliability.The diversity of successful paths indicates that many residual failures arise from local exploration traps rather than systematic reasoning errors.Source: §5.2 · p. 10

5.3 OSWorld

• 60.1% success on OSWorld's Foundation E2E GUI category is state of the art; pass@10 reaches 77.0%, above the 72.4% human baseline.Success exceeds 50% across task categories, with especially strong performance on programming-related tasks; the Holo1.5 72B localizer reached 60.1%, versus 58.4% for Holo1.5 7B and 56.9% for UI-TARS 7B.Source: §5.3 · p. 10, Figure 5 · p. 11
• 60.1% with Holo1.5 72B exceeds 58.4% with Holo1.5 7B and 56.9% with UI-TARS 7B on OSWorld.The authors attribute the difference to the importance of accurate spatial grounding for GUI reasoning.Source: §5.3 · p. 10

5.4 AndroidWorld

• 87.1% accuracy across AndroidWorld's 116 tasks surpasses prior visual-only approaches; accuracy is 98.4% on Easy, 86.1% on Medium, and 52.6% on Hard tasks.Performance decreases on tasks requiring long-horizon reasoning or intricate multi-step coordination.Source: §5.4 · pp. 10–11, Figure 6 · p. 11
• 50% accuracy on memorization and transcription and 37.5% on multi-app tasks identify AndroidWorld's weakest categories.The authors identify limited memory and text handling as difficulties for the first two categories, and cross-application reasoning as the primary multi-app challenge.Source: §5.4 · p. 11, Figure 7 · p. 12
• 93.1% pass@3 on AndroidWorld improves on 87.1% pass@1 by 6.0 percentage points; replacing Holo1.5 with UI-TARS reduces pass@1 to 81.9%.The authors connect localization errors on small targets and iconographic widgets to incorrect actions; they identify real-time perception, long-sequence memory, and GUI semantics as remaining challenges.Source: §5.4 · pp. 11–12

6 Key Insights and Path Forward

• Prompt wording shifts accuracy 5–10%; persistent context cuts navigation steps 30–40%; multi-stage validation intercepts 15–20% of errors.Model variance remains substantial at temperature 0, while hierarchical task decomposition provides retry boundaries and interpretability.Source: §6 · p. 12
• Complex tasks cost $1–5 with frontier reasoning models, and localizers fail on 5–8% of UI elements.Tasks exceeding 50 steps face context-window limits and compounding errors; LLM-based evaluation carries 5–10% error rates.Source: §6 · p. 12
• Surfer 2 achieves state-of-the-art accuracy with fixed, general-purpose models, and modular designs generalize across web, desktop, and mobile.Source: §6 · p. 12
• The proposed path forward is smaller, specialized models targeting comparable or better performance at a fraction of current cost.The stated goal is Pareto optimality.Source: §6 · p. 12

• The WebVoyager test set contains 590 tasks across 15 websites.The listed sites include Amazon, Google Flights, GitHub, Google Maps, and Wolfram Alpha.Source: §A.1 · p. 17
• Tasks cover information retrieval, comparison, and navigation.Source: §A.1 · p. 17
• Evaluation replaces a single GPT-4V judge with majority voting across three GPT-4o calls at temperature 0 using the last five screenshots.The protocol retains the original WebVoyager prompt and is intended to reduce variance, single-judge bias, and spurious results.Source: §A.1 · p. 17
• Tasks run on live websites in Selenium-controlled Chrome, with proxy rotation and Bing redirection used to mitigate bot restrictions.The cited restrictions include CAPTCHAs and IP-based access blockers.Source: §A.1 · p. 17

• WebArena comprises 812 tasks across six self-hosted sites, including GitLab, Reddit, OpenStreetMap, and English Wikipedia.Source: §A.2 · p. 17
• Tasks span retrieval, comparison, navigation, and multi-step or multi-site actions, with category-specific initialization prompts for grounding.Examples include form filling and searches across one or two websites.Source: §A.2 · p. 17
• The modular evaluation combines URL matching, required HTML artifacts, and model-based assessment, with five GPT-4.1 judges using majority voting.The criteria can be combined or used independently.Source: §A.2 · p. 17
• The benchmark runs on controlled web servers with fixed initial states accessed through Selenium-controlled Chrome.Source: §A.2 · p. 17

• WebArena’s self-hosted sites require manual intervention, tasks can inherit residual state, and statefulness limits reproducibility and large-scale parallel execution.These constraints increase the computational cost and time needed for robust metrics.Source: §A.2.1 · p. 17
• Five independent GPT-4.1 judges replace heuristic string matching and a single LLM assessment, with majority voting to reduce variance and single-judge bias.The method allows minor semantically irrelevant variations in exact-match tasks and penalizes false positives in must-include tasks.Source: §A.2.1 · p. 18
• Manual review corrected 71 benchmark tasks; without those fixes, the approach achieved 67.4% success using an LLM judge for string comparison.That ablation still surpassed prior state-of-the-art performance.Source: §A.2.1 · p. 18

A.3 OSWorld Evaluation

• OSWorld covers 369 Ubuntu desktop tasks across applications such as LibreOffice, Thunderbird, Gimp, Chrome, and VS Code.Tasks can involve several applications at once.Source: §A.3 · p. 18
• Tasks range from document editing and image manipulation to coding, with complexity varying from 10–100 steps.Source: §A.3 · p. 18
• Evaluation uses programmatic checks of the final file or application state on AWS-hosted Ubuntu VMs at 1920×1080 with VNC control.Source: §A.3 · p. 18

A.4 AndroidWorld Evaluation

• 116 parameterized AndroidWorld tasks run on a Pixel 6 emulator and require multi-app, multi-step workflows across a broad set of apps.SMS-related tasks ran outside Docker containers because of SIM card constraints.Source: §A.4 · pp. 18–19
• AndroidWorld spans tasks from data entry and navigation to verification and game playing, scored by programmatic checks as 1, 0.5, or 0.Information-retrieval tasks use exact or fuzzy matching.Source: §A.4 · p. 19

B.1 OSWorld Example

• Surfer 2 changed Chrome’s interface language to Korean by opening a terminal, running system commands through visual interaction, and relaunching Chrome.Human evaluators had labeled the task infeasible because no visible language selector was available.Source: §B.1 · p. 19, Figure 8 · p. 20

B.2 AndroidWorld Examples

• The AndroidWorld examples pair a complex success with a representative failure; they show robust cross-app reasoning but difficulty with long-horizon temporal perception.Source: §B.2 · p. 19, Figure 9 · p. 21