Coding Agents with Multimodal Browsing are Generalist Problem Solvers

1 Introduction

• Specialist agents are typically optimized for narrow task sets and may lack capabilities needed beyond their intended domains.The paper contrasts coding agents that cannot typically gather web information with browsing agents that cannot write, debug, or execute code.Source: §1 · pp. 1–2
• 9.1, 1.3, and 9.1 points are the absolute success-rate improvements over the best published results on SWE-Bench Multimodal, GAIA, and The Agent Company, respectively.OpenHands-Versa also improves OpenHands’ browsing and information access while maintaining its software-engineering capabilities.Source: Figure 1 · p. 2, §1 · p. 2
• The paper argues that many tasks can be addressed with three capabilities: coding, multimodal web browsing, and information access.These include interactive browsing and processing multimodal content from files such as PDFs and spreadsheets.Source: §1 · p. 2
• OpenHands-Versa combines coding, multimodal web browsing, and information access in one agent built on the OpenHands framework.Information access includes web search and processing multimodal content from files.Source: §1 · p. 2
• The authors report that current state-of-the-art multi-agent systems fail to generalize beyond their intended scope.They also analyze tool-use patterns and find that OpenHands-Versa’s tool choices align better with task requirements than OpenHands’ choices.Source: §1 · p. 2

2.1 Preliminaries

• OpenHands provides a flexible event-stream architecture, sandboxed runtime, evaluation harness, shell, interactive Python, text-based browsing, and plain-text file tools.OpenHands-Versa builds on this framework to implement a generalist agent.Source: §2.1 · p. 3

2.2 Ingredients of Our Agent

• OpenHands-Versa uses Set-of-Marks prompting to pair webpage screenshots with labeled interactable elements and AXTree context.The AXTree is truncated to the current viewport when it is too large for the LLM’s context window.Source: §2.2 · p. 3
• A browsing condenser retains the k most recent observations and replaces older ones with placeholders to reduce large inputs and inference costs.The paper motivates this design by the screenshots and potentially large AXTrees included in browser observations.Source: §2.2 · p. 3
• OpenHands-Versa uses search APIs for web information and Markdown converters to represent multimodal files in a unified format.The paper says search APIs can mitigate CAPTCHA blocking and reduce costs for factual searches; supported files include PDFs, audio, and presentation slides.Source: §2.2 · pp. 3–4
• After every τ steps, a fixed planning prompt asks the agent to summarize progress and plan the next part of execution.This replaces approaches using an orchestrator, a planning agent, a Think tool, or Chain-of-Thought prompting.Source: §2.2 · p. 4

2.3 Comparison with Existing Agents

• The comparison evaluates agents’ coding, browsing, and information-access capabilities, as well as whether they use single- or multi-agent frameworks.The comparison is limited to agents with open-source implementations so their internal designs can be examined.Source: §2.3 · p. 4
• Browser-only agents lack code-related abilities, while several multi-agent systems lack file editing or shell access needed for software-engineering tasks.SWE-Agent, OpenHands, and OpenHands-Versa support code editing and execution capabilities; Agentless supports selected repository tests in a human-defined workflow.Source: §2.3 · p. 4
• Existing agents vary in browsing and information access: some lack interactive visual browsing, search APIs, or multimodal file viewing.OpenHands supports interactive but text-only browsing, while SWE-Agent and Agentless do not support browsing; plain-text file viewing is the limit for several coding agents.Source: §2.3 · p. 5

3 Experimental Setup

• The setup evaluates OpenHands-Versa on three benchmarks chosen to cover diverse agent capabilities and use cases.The paper describes the benchmarks, metrics, and baselines in this section.Source: §3 · p. 5, §3.1 · p. 5

3.1 Evaluation Benchmarks

• SWE-Bench M tests fixing frontend-library issues across 17 JavaScript repositories, with more than 28% of instances requiring edits across at least two programming languages.Some tasks also provide visual assets or links to online IDEs that require multimodal processing.Source: §3.1 · p. 5
• GAIA tests general-assistant tasks involving web browsing and search, coding, reasoning, and multimodal content such as audio, spreadsheets, and PDFs.Its coding tasks generally require standalone programs rather than edits to existing code files.Source: §3.1 · p. 5
• The Agent Company evaluates digital co-worker tasks in a self-hosted simulated software company, including development, project management, data science, and financial analysis.Tasks involve websites for code, files, project management, and communication, as well as writing and editing files.Source: §3.1 · p. 5

3.2 Baseline Agents

• Baselines are the best-performing open-source agents with reproducibility guidelines, and OpenHands v0.28.1 is included to assess the effect of OpenHands-Versa’s modifications.The study uses claude-3-7-sonnet-20250219 for direct architecture comparisons and also evaluates OpenHands-Versa with claude-sonnet-4-20250514.Source: §3.2 · p. 6

3.3 Evaluation metrics

• GAIA and SWE-Bench M use resolve rate, while The Agent Company uses full and partial completion scores; the study also reports pass @ 1 for all benchmarks.The reported evaluation uses test splits for all three benchmarks, except that GAIA uses validation because test ground truth is unavailable.Source: §3.3 · p. 6

4 Main Results

• 1.33 points in GAIA resolve rate, 6.9 points in The Agent Company full completion score, and 6.8 points in partial completion score exceed the strongest baselines.On SWE-Bench M, the resolve rate improves by 9.09 points over Agentless-Lite and by more than 22 points over SWE-Agent and its variants, without specific SWE-Bench M optimizations.Source: §4 · p. 6
• The comparison is restricted to agents with open-source implementations and method descriptions, excluding systems without reproducibility guidance or configuration details.The paper notes that other GAIA leaderboard methods report scores up to 77% but lack published code or technical details.Source: §4 · p. 6, Table 2 · p. 7
• 51.16% resolve rate on GAIA, 34.43% resolve rate on SWE-Bench M, and 33.14% full completion score on The Agent Company put OpenHands-Versa at or near state of the art.The reported results use claude-sonnet-4 as the backbone LLM.Source: §4 · p. 6, Table 2 · p. 7
• 13.9 points in GAIA resolve rate and 4.6 points in The Agent Company full completion score and 3.8 points in partial completion score exceed OpenHands, while SWE-Bench M resolve rates differ by only 0.39 points.The GAIA comparison uses the same claude-3.7 sonnet backbone.Source: §4 · pp. 6–7
• OWL-roleplaying's 58.18% GAIA validation resolve rate does not generalize to The Agent Company, where it scores 4% full and 11.0% partial completion.The Agent Company scores are 29.1 points lower in full completion and 32.2 points lower in partial completion than OpenHands-Versa.Source: §4 · p. 7

5.1 Tool Use Patterns across Benchmarks

• OpenHands-Versa's tools align with benchmark tasks: it uses browser and search for GAIA, browser navigation for The Agent Company, and coding tools plus browser-based visual verification for SWE-Bench M.For GAIA, it uses IPython and bash flexibly; for SWE-Bench M, it often uses bash, edit_file, and view_file.Source: §5.1 · p. 8, Figure 2 · p. 8
• OpenHands-Versa uses search snippets to select targeted URLs for GAIA and more often uses the browser to visually verify SWE-Bench M changes, while The Agent Company's tool-use behavior is similar between agents.OpenHands lacks a search engine in the comparison and uses text-only browsing, unlike OpenHands-Versa's visual browsing.Source: §5.1 · p. 8, Figure 2 · p. 8

5.2 Error Analysis

• GAIA errors include reliance on inaccurate search snippets and inaccessible CAPTCHA-protected sites; SWE-Bench M errors include incomplete testing and premature exits, while The Agent Company errors include OwnCloud loops and unfinished complex tasks.The GAIA error analysis uses the validation split because test-set ground truth is unavailable.Source: §5.2 · pp. 8–9

5.3 Effect of Search API on GAIA

• 64.24% resolve rate with Tavily exceeds 56.96% with Brave and 58.18% with Exa on the GAIA validation split; switching from Brave to Tavily improves resolve rate by 7.28 points.Tavily provides an LLM-generated answer synthesized from retrieved results, which the analysis identifies as one primary reason for its higher resolve rate; inaccurate summaries can still cause hallucinations.Source: §5.3 · p. 9

6 Conclusion

• The results suggest generalist agents can perform across domains by providing necessary tools and letting the backbone LLM decide how to use them.The authors report strong performance across GAIA, SWE-Bench M, and The Agent Company.Source: §6 · p. 9
• OpenHands-Versa is proposed as a strong baseline for future research on generalist agents.The authors also discuss limitations and insights for future improvement.Source: §6 · pp. 9–10

A Webpage Screenshot with Set-of-Marks Annotation

• Interactable webpage elements are annotated with bounding boxes and corresponding browsergym-ids.The figure is an example rather than a reported performance result.Source: A Webpage Screenshot with Set-of-Marks… · p. 12, Figure 3 · p. 12
• The screenshot illustrates the annotation of interactable elements on a webpage.The caption identifies it as an example screenshot.Source: A Webpage Screenshot with Set-of-Marks… · p. 12
• Figure 3 provides an example of a webpage screenshot with set-of-marks annotation.The figure depicts interactable elements marked by bounding boxes and corresponding browsergym-ids.Source: Figure 3 · p. 12, A Webpage Screenshot with Set-of-Marks… · p. 12

B Experimental Setup

• The identical OpenHands-Versa implementation is used across all three benchmarks rather than selecting tools or prompts for each benchmark.Browsing condensation retains the most recent observation (k=1), and the planning prompt is appended every 10 steps (τ=10).Source: B Experimental Setup · p. 12
• The agent runs with a 100-step limit on SWE-Bench M and The Agent Company and a 60-step limit on GAIA.GAIA answers are extracted using an LLM to match the benchmark’s exact-answer format and help address formatting errors.Source: B Experimental Setup · p. 12
• Evaluations use CPU-only cloud machines, and the authors report runtimes of approximately 24 hours for GAIA, 54 for The Agent Company, and 12 for SWE-Bench M.The setup uses AWS EC2 t3.2xlarge instances with 32GB RAM, 8 vCPUs, and 512GB disk space.Source: B Experimental Setup · p. 12

B.1 Baseline Agents

• Magentic-One and OpenDeepResearch use multiple agents, but their described coding and browsing tools have constraints, including limited support for editing files or interactive browsing.Magentic-One lacks native code editing and support for other programming languages; OpenDeepResearch’s browser cannot click or type and its coding libraries are fixed.Source: §B.1 · p. 13
• OWL-roleplaying combines assistant and browsing agents with file tools, Python execution, web search, and visual browsing, but lacks native file editing and unrestricted programming support.Its Python programs are limited to pre-installed packages.Source: §B.1 · p. 13
• SWE-Agent and its SWE-Bench M variants focus on code tasks, while Agentless-Lite retrieves relevant files and asks an LLM to generate a patch.SWE-Agent Multimodal can visually reproduce image-based issues and verify fixes, but these coding agents lack public-web browsing or search; Agentless-Lite also lacks code execution and multimodal file processing.Source: §B.1 · p. 13
• Table 4 presents example tasks from the three benchmarks alongside the tools or capabilities required to solve them.The examples qualitatively illustrate that strong performance across the benchmarks requires several capabilities.Source: Table 4 · p. 14

C Performance on GAIA Validation Split

• The evaluation considers open-source agents with reproducibility guidelines and documented configurations.This criterion is stated for the GAIA validation comparison and the other experiments.Source: C Performance on GAIA Validation Split · p. 14
• 64.24% resolve rate on the GAIA validation split, exceeding Magentic-One (46.06%), OpenDeepResearch (55.15%), and OWL-roleplaying (58.18%).The evaluation uses the described agent configuration with claude-3.7-sonnet as the backbone LLM.Source: C Performance on GAIA Validation Split · p. 14

• Examples across GAIA, SWE-Bench M, and The Agent Company illustrate that performing well across the benchmarks requires several capabilities.The tasks and their required tools or capabilities are presented in Table 4.Source: D Example Tasks · p. 14
• Without browsing, Agentless-Lite and SWE-Agent cannot solve the given GAIA and The Agent Company examples.Source: D Example Tasks · p. 14
• Without JavaScript code execution, none of the multi-agent systems can solve the example SWE-Bench M tasks.Source: D Example Tasks · p. 14

E Limitations of Our Approach

• OpenHands-Versa does not cover GUI-based desktop tasks and cannot view local video files with its file viewer.The authors note that a file or video summarizer could potentially mitigate the video limitation.Source: E Limitations of Our Approach · p. 14
• The work primarily relies on closed-source LLMs, and proprietary-model costs prevented evaluating every baseline on all three benchmarks.The authors limit evaluation to strong baseline agents.Source: E Limitations of Our Approach · p. 15

F Broader Impact of Our Work

• The broader-impact discussion identifies potential misuse, labor-market disruption, and governance challenges as agents become more sophisticated.Source: F Broader Impact of Our Work · p. 15
• The authors call for safeguards, ethical frameworks, and policies to guide real-world implementation.Source: F Broader Impact of Our Work · p. 15