Source-linked AI summary
MolmoWeb: Open Visual Web Agent and Open Data for the Open Web
Tanmay Gupta, Piper Wolters, Zixian Ma, Peter Sushko, Rock Yuren Pang, Diego Llanes, Yue Yang, Taira Anderson, Boyuan Zheng, Zhongzheng Ren, Harsh Trivedi, Taylor Blanton, Caleb Ouellette, Winson Han, Ali Farhadi, Ranjay Krishna
TL;DR
Existing web agents are often proprietary, limiting reproducibility and scientific understanding. This paper releases MolmoWebMix and screenshot-based MolmoWeb agents, which outperform comparable open-weight and larger closed-model baselines while benefiting from parallel-rollout test-time scaling.
Problem
Proprietary web agents often disclose limited training data and complete recipes, constraining reproducibility and scientific understanding.
Method
The paper releases MolmoWebMix and fully open vision-language action policies trained on task, skill, and GUI-perception data that operate from screenshots without HTML or accessibility trees.
Results
MolmoWeb agents outperform comparable open-weight agents and set-of-marks agents built on much larger closed frontier models, with further gains from parallel-rollout best-of-N selection.
Takeaways & Limitations
The released checkpoints, data, code, and evaluation tools support reproducible research on open web agents.
Takeaways & Limitations
Performance is best for specific instructions and may degrade with ambiguity, extensive constraints, or required exploration.
Abstract
from arXiv · showhide
Web agents--autonomous systems that navigate and execute tasks on the web on behalf of users--have the potential to transform how people interact with the digital world. However, the most capable web agents today rely on proprietary models with undisclosed training data and recipes, limiting scientific understanding, reproducibility, and community-driven progress. We believe agents for the open web should be built in the open. To this end, we introduce (1) MolmoWebMix, a large and diverse mixture of browser task demonstrations and web-GUI perception data and (2) MolmoWeb, a family of fully open multimodal web agents. Specifically, MolmoWebMix combines over 100K synthetic task trajectories from multiple complementary generation pipelines with 30K+ human demonstrations, atomic web-skill trajectories, and GUI perception data, including referring expression grounding and screenshot question answering. MolmoWeb agents operate as instruction-conditioned visual-language action policies: given a task instruction and a webpage screenshot, they predict the next browser action, requiring no access to HTML, accessibility trees, or specialized APIs. Available in 4B and 8B size, on browser-use benchmarks like WebVoyager, Online-Mind2Web, and DeepShop, MolmoWeb agents achieve state-of-the-art results outperforming similar scale open-weight-only models such as Fara-7B, UI-Tars-1.5-7B, and Holo1-7B. MolmoWeb-8B also surpasses set-of-marks (SoM) agents built on much larger closed frontier models like GPT-4o. We further demonstrate consistent gains through test-time scaling via parallel rollouts with best-of-N selection, achieving 94.7% and 60.5% pass@4 (compared to 78.2% and 35.3% pass@1) on WebVoyager and Online-Mind2Web respectively. We will release model checkpoints, training data, code, and a unified evaluation harness to enable reproducibility and accelerate open research on web agents.
1 Introduction
MolmoWeb addresses the reproducibility gap in proprietary web agents by releasing an open visual-agent suite and diverse training data. Its screenshot-based agents achieve strong performance while avoiding HTML, accessibility-tree, and specialized-API access.
- Proprietary web agents limit reproducibility and scientific understanding because their training data and complete recipes are often undisclosed.
- MolmoWeb provides MolmoWebMix and fully open multimodal agents trained as instruction-conditioned policies that repeatedly map task context and screenshots to browser actions.
- MolmoWeb agents operate from the visual interface alone, without specialized APIs or access to HTML or accessibility trees.
- MolmoWebMix combines synthetic and human task trajectories, atomic web-skill supervision, and GUI perception data for screenshot understanding and task execution.
- MolmoWeb agents outperform comparable open-weight agents and set-of-marks agents built on much larger closed frontier models despite using only visual input.
2 MolmoWebMix
MolmoWebMix is assembled from multiple trajectory-generation sources and targeted visual-perception supervision. The pipelines combine human and synthetic demonstrations, deterministic navigation data, atomic skills, and screenshot-grounded interaction data.
- MolmoWebMix training data includes task trajectories, atomic skill trajectories, and GUI perception data, with failed trajectories filtered out.
- AxTree-agent trajectories provide synthetic browser demonstrations from instructions, serialized accessibility-tree observations, and browser-ID actions that are converted into pixel-space coordinates.
- A Planner–Operator–Verifier harness decomposes tasks, executes browser actions, and checks screenshot-based subgoal completion before proceeding.
- 78.5 versus 74.4 on WebVoyager shows higher task-completion success for the multi-agent setup than the Gemini AxTree agent alone.
- Human demonstrations capture browser events and screenshots, while ordered subtask annotations enable extraction of short, focused atomic-skill trajectories.
- Node-traversal data uses breadth-first website graphs, deterministic scrolling and clicking, URL-based verification, and goals generated from validated paths.
- GUI perception supervision teaches screenshot-based element grounding, OCR-like reading, and comprehension needed to identify controls and answer webpage questions.
3 MolmoWeb
MolmoWeb is a vision-language action policy that uses screenshots, instructions, and recent action history to produce browser operations. It is trained end-to-end with supervised fine-tuning on the MolmoWebMix mixture.
- MolmoWeb builds on a multimodal language model and predicts a natural-language thought followed by the next browser action from webpage screenshots, instructions, and action history.
- At each step, the observation includes the current viewport screenshot, task instruction, ten prior actions, and the current page URL and title.
- The model emits a JSON object containing a rationale and action, with mouse operations represented by normalized spatial coordinates later mapped to viewport pixels.
- All task, skill, and GUI-perception data are mixed in one supervised-fine-tuning stage, with source ratios selected as hyperparameters for benchmark balance.
4 Experiments
Experiments evaluate MolmoWeb across browser-use benchmarks, compare it with open- and closed-model agents, and study test-time compute, training data, and decoding choices. MolmoWeb achieves strong benchmark performance, while parallel rollouts, larger datasets, and randomized sampling provide additional gains.
- Comparison to prior work: MolmoWeb establishes a new state-of-the-art among open-weight models across browser-use benchmarks.MolmoWeb-8B improves over leading open-weight models across all benchmarks, while MolmoWeb-4B leads on WebVoyager and DeepShop and remains competitive elsewhere.
- Comparison to prior work: MolmoWeb-8B outperforms GPT-4o-based set-of-marks agents, matches o3 on WebVoyager, and trails GPT-5 and o3 by only 6 points on DeepShop.It also outperforms OpenAI computer-use-preview on WebVoyager and DeepShop according to reported performance.
- Test-time scaling: 86.2% via three parallel 30-step runs exceeds 78.2% from one 100-step run for the 8B model.The comparison indicates that parallel rollout scaling can outperform simply increasing the maximum inference steps.
- Training data ablations: About 85 to 90% of performance is achieved with 10% of the training dataset, while performance continues improving with data scale.The ablation used an earlier MolmoWebMix version with fewer human and synthetic trajectories but a similar composition.
- Training data ablations: Human-data gains are limited and inconsistent across benchmarks, with differences in volume, task distribution, actions, annotation style, and noise identified as likely factors.The authors hypothesize that synthetic and human trajectories encode distinct completion policies, complicating generalization across both sources.
- Sampling Strategies: Top-k and top-p sampling improve performance by over 5% versus greedy decoding, with nucleus sampling at p=0.8 and temperature=0.7 performing best on WebVoyager.Randomized strategies can avoid repetitive stuck states observed with greedy decoding; nucleus sampling is used by default.
- Comparison to prior work: MolmoWeb-4B trails a specialist grounding model by only a few points while also supporting web task completion.This comparison highlights competence across both grounding and browser-task completion rather than grounding alone.
5 Related work
Related work spans language-based agents operating on DOM-derived representations, screenshot-based multimodal agents, isolated GUI understanding, and live-website evaluation.
- LLM-driven web agents: LLM-driven web agents commonly predict browser interactions from language representations derived from the DOM, such as accessibility trees.
- Multimodal web agents: Multimodal web agents instead process screenshots to produce actions, ranging from modular planning systems to unified vision-language action policies.
- GUI understanding: GUI understanding research studies referring-expression grounding, screenshot question answering, and structured screenshot parsing separately from web-task execution.
- Evaluation of web agents: Web-agent evaluation has progressed from sandboxed and desktop environments toward live websites with automatic verifiers, static answers, or VLM-based success judgments.
6 Capabilities and Limitations
MolmoWeb performs best with specific instructions but has limitations involving ambiguity, complex constraints, OCR-heavy reading, and inference latency.
- Instruction following: MolmoWeb performs best on specific instructions, while ambiguous requests, many constraints, or extensive search filters can degrade performance.
- OCR and reading comprehension: MolmoWeb can struggle with OCR on small text and with complex questions requiring comprehension of large passages.
- Latency: MolmoWeb trajectories target action efficiency, but the model is not optimized for per-step feedforward inference latency.
7 Conclusion
The paper introduces an open data-and-model suite for screenshot-based web agents that outperforms several baselines and supports reproducible research.
- MolmoWebMix and MolmoWeb form a fully open data and model suite for multimodal web agents.
- Operating purely from screenshots, MolmoWeb agents outperform comparable open-weight models and SoM agents built on much larger proprietary models.
- The authors will release checkpoints, data, code, and evaluation tools to support reproducible research and progress on open web agents.
B.2 Task Sampling
Task sampling combines manually authored templates with LLM-generated tasks, multiple instruction granularities, persona-based diversity, and constrained browser-step specifications.
- Manually written task templates: Manually authored templates cover domains including shopping, news, real estate, travel, maps, food, jobs, health, and cars through sequences of atomic skills.
- Manually written task templates: Workers use an annotation tool to mark completed steps or record why a step could not be completed.
- Instruction specificity: An LLM rewrites step-by-step instructions into low-, mid-, and high-level versions with different specificity.
- Instruction specificity: Training randomly samples among four instruction forms, with slightly higher probability assigned to high-level instructions.
- LLM-sampled tasks: Persona-conditioned LLM prompts generate tasks as allowed browser-step sequences followed by natural-language instructions, typically targeting 3–10 steps.
B.2.3 LLM sampled tasks with navigation and QA.
This task-generation approach combines structured navigation-and-question tasks with more flexible requests to reach a webpage and optionally answer a question there.
- Rigid generated tasks can be infeasible when strict search constraints produce no website matches.
- Generated tasks target a webpage through navigation and may append a question answered from that page.The flexible format is exemplified by finding Lavazza coffee for pickup on Walgreens and asking its price.
- The generation prompt asks for both allowed action steps and a natural-language instruction for the web agent.
- The task set contains only navigation-only tasks or two-step tasks consisting of navigation followed by a question.
B.2.4 LLM sampled benchmark-like tasks.
Benchmark-like synthetic tasks are designed to reduce distribution mismatch by incorporating benchmark task patterns while varying the language-model sources used for generation.
- Benchmark-like task generation addresses biases from author-written templates and prompts that may differ from downstream benchmark task distributions.
- Synthetic task sampling primarily uses GPT-4o and cycles among GPT-4o, GPT-4.1, GPT-5-mini, and GPT-5 for some tasks.The stated purpose is to avoid reliance on a single language model and increase task diversity.
C.1 Task Sampling
Task sampling combines manually authored templates, benchmark-like generation, and a four-axis taxonomy to produce diverse, metadata-tagged browsing tasks across intents, domains, difficulty, and ambiguity.
- The sampling strategies include manually authored templates and benchmark-distribution-matched generation.
- Task taxonomy: The taxonomy spans intent, domain, difficulty, and ambiguity, covering 13 website domains and four difficulty levels from D0 to D3.
- Task taxonomy: Difficulty increases with simultaneous constraint count, from D0 with 0–1 constraints to D3 with 6+ constraints.
- Task taxonomy: Ambiguity ranges from A0 single-answer tasks to A3 vague or underspecified goals requiring clarification.
- Generation pipeline: The pipeline iterates over the taxonomy and prompts GPT-4o for diverse tasks in each Cartesian-product cell.
- Generation pipeline: Each task includes a target website, task type, concrete browser action steps, three instruction verbosity levels, and taxonomy metadata.
D Additional Dataset Statistics
The dataset spans diverse web domains and websites, with visual summaries showing its hierarchical website coverage and action distribution.
- The training data covers a wide variety of web browsing domains and websites, approximating human web-browsing action and input distributions.
- Figure 9 uses a sunburst chart to show the hierarchical breakdown of broad domains and websites, listing the top nine websites per category.
- Figure 8 presents the action distribution breakdown across the dataset.