Source-linked AI summary
Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agents
Saaket Agashe, Kyle Wong, Vincent Tu, Jiachen Yang, Ang Li, Xin Eric Wang
TL;DR
Computer-use agents struggle with precise GUI grounding, long-horizon planning, and bottlenecks from relying on single generalist models. Agent S2 addresses these issues with compositional generalist-specialist modules, Mixture of Grounding, and Proactive Hierarchical Planning, achieving SOTA results across three benchmarks.
Problem
Current computer-use agents have inaccurate GUI grounding, difficulty with long-horizon tasks under evolving observations, and performance bottlenecks from using generalist models for diverse subtasks.
Method
Agent S2 delegates high-level reasoning, low-level execution, and fine-grained grounding across generalist and specialist modules using Mixture of Grounding and Proactive Hierarchical Planning.
Results
Agent S2 achieves SOTA performance across OSWorld, WindowsAgentArena, and AndroidWorld, including 27.0% (↑18.9%) on OSWorld’s 15-step evaluation.
Takeaways & Limitations
Composing generalist and specialist models can outperform monolithic models across computer-use and smartphone-use benchmarks.
Takeaways & Limitations
Agent S2 still exhibits planning, grounding, interaction, navigation, and infeasible-task failure modes.
Abstract
from arXiv · showhide
Computer use agents automate digital tasks by directly interacting with graphical user interfaces (GUIs) on computers and mobile devices, offering significant potential to enhance human productivity by completing an open-ended space of user queries. However, current agents face significant challenges: imprecise grounding of GUI elements, difficulties with long-horizon task planning, and performance bottlenecks from relying on single generalist models for diverse cognitive tasks. To this end, we introduce Agent S2, a novel compositional framework that delegates cognitive responsibilities across various generalist and specialist models. We propose a novel Mixture-of-Grounding technique to achieve precise GUI localization and introduce Proactive Hierarchical Planning, dynamically refining action plans at multiple temporal scales in response to evolving observations. Evaluations demonstrate that Agent S2 establishes new state-of-the-art (SOTA) performance on three prominent computer use benchmarks. Specifically, Agent S2 achieves 18.9% and 32.7% relative improvements over leading baseline agents such as Claude Computer Use and UI-TARS on the OSWorld 15-step and 50-step evaluation. Moreover, Agent S2 generalizes effectively to other operating systems and applications, surpassing previous best methods by 52.8% on WindowsAgentArena and by 16.52% on AndroidWorld relatively. Code available at https://github.com/simular-ai/Agent-S.
1 Introduction
Agent S2 addresses inaccurate GUI grounding, weak long-horizon planning, and bottlenecks from using generalist models alone by composing specialist and generalist modules. It reports SOTA results across multiple computer-use benchmarks.
- Motivation: Current agents struggle with pixel-level GUI grounding, long-horizon tasks under changing observations, and diverse cognitive responsibilities handled by generalist models.Generalist models can underperform specialists on domain-specific subtasks.
- Approach: Agent S2 delegates reasoning, execution, and grounding across generalist and specialist modules in a compositional hierarchical framework.Its contributions include Mixture of Grounding and Proactive Hierarchical Planning.
- Approach: Mixture of Grounding routes actions to specialized grounding experts for precise GUI localization across diverse applications.The mechanism targets the grounding bottleneck directly.
- Approach: Proactive Hierarchical Planning dynamically refines action plans at multiple temporal scales as new observations and state changes arrive.This contrasts with methods that follow fixed plans or revise only after failures.
- Results: 29.8% (↑52.8%) accuracy on WindowsAgentArena and 54.3% (↑16.5%) accuracy on AndroidWorld establish further SOTA results.The evaluations cover additional operating systems and smartphone-use tasks.
2 Background
Computer-use agents interact with partially observed digital environments through multimodal observations and action sequences. Research has developed monolithic and hierarchical approaches, while reactive planning remains vulnerable to environmental changes.
- Computer-use setting: Computer-use tasks are multimodal interactions with digital environments, formally modeled as partially observable Markov decision processes.The formulation includes states, observations, actions, transitions, and rewards.
- Benchmarks: OSWorld evaluates desktop control in Ubuntu, WindowsAgentArena extends evaluation to Windows, and AndroidWorld targets smartphone-use tasks.These benchmarks use online functional evaluation scripts.
- Approaches: Existing methods either train monolithic generalist models or compose multiple models into planning, execution, and grounding pipelines.These represent the two primary research directions described in the background.
- Approaches: Monolithic agents handle planning, execution, and grounding with one model, requiring both long-term reasoning and precise UI grounding.Fine-tuning for specialization can diminish broader capabilities.
- Approaches: Hierarchical methods decouple planning, execution, and grounding by assigning high-level subgoal generation to a Manager and atomic actions to a Worker.Each subgoal specifies an intermediate task objective.
- Planning limitation: Reactive planning methods struggle with unexpected environmental changes, limiting overall robustness.This limitation motivates more adaptive planning strategies.
3 Agent S2: Compositional Grounding and Planning for Computer Use
Agent S2 composes hierarchical planning modules with specialist grounding experts: the Manager generates subgoals, the Worker executes actions, and grounding experts localize targets. Proactive replanning updates objectives as observations evolve.
- Framework: The Manager decomposes tasks into high-level subgoals, while the Worker generates actions for the current topmost subgoal.The framework combines these generalist planning modules with specialist grounding experts.
- Framework: At each execution step, the Worker routes an action’s grounding task to a corresponding specialist expert.This separates action generation from precise target localization.
- Mixture of Grounding: The Visual Grounding Expert maps a screenshot and language description to precise point coordinates, enabling screenshot-only interaction across screen regions.The Worker can progressively refine descriptions to self-correct actions.
- Mixture of Grounding: The Textual Grounding Expert uses OCR with start and end phrases to output coordinates for a text span.It targets fine-grained alignment with words or sentences.
- Mixture of Grounding: The Structural Grounding Expert addresses precise localization in spreadsheets and tabular interfaces.These interfaces are difficult because cells can vary in size and translated tables can shift row and column positions.
- Proactive Hierarchical Planning: Proactive Hierarchical Planning replans at Manager and Worker levels across different temporal scales in response to noisy, evolving observations.The Manager updates remaining subgoals after each completed subgoal, while the Worker routes each action to an expert.
4 Experiments
Agent S2 is evaluated across desktop and mobile-oriented computer-use benchmarks, with comparisons at multiple horizons and ablations of grounding, planning, scaling, and failure modes. It achieves strong cross-benchmark results while showing that specialist grounding and proactive replanning contribute especially at longer horizons.
- Experimental Setup: Agent S2 is evaluated on OSWorld, WindowsAgentArena, and AndroidWorld using screenshot-input baselines and both 15-step and 50-step OSWorld settings.OSWorld includes 369 tasks, WindowsAgentArena includes 154 tasks, and Table 2 reports categorized OSWorld 50-step success rates across manager and worker MLLMs.
- Main Results: Agent S2 achieves new SOTA on OSWorld, outperforming all compared results at both 15-step and 50-step evaluations.With Claude-3.5-Sonnet (new), Agent S2 relatively outperforms Claude Computer Use with Claude-3.7-Sonnet by 58.1% at 15 steps and 29.6% at 50 steps.
- Main Results: 52.8% relative improvement over NAVI on WindowsAgentArena demonstrates generalization across operating systems, with gains in 4/6 categories.Agent S2 uses screenshots only, whereas Agent S and NAVI use screenshots and accessibility trees.
- Ablation Study: Mixture of Grounding raises success from 27.69% to 30.77% at 15 steps and from 33.85% to 38.46% at 50 steps.Removing textual grounding lowers subtask success from 70.6% to 65.2%, while removing structural grounding lowers it from 73.7% to 69.4%.
- Ablation Study: Smaller specialist visual-grounding models can outperform large generalist models when used within Agent S2’s modular framework.The visual grounding expert is used in every example task, and Figure 6 compares 15-step performance across grounding models.
- Ablation Study: Proactive Hierarchical Planning improves success by +4.62% at 15 steps and +6.15% at 50 steps compared with reactive planning.The study attributes additional-step gains to adaptive navigation, self-correction, alternative approaches, backward correction, and task complexity.
- Error Analysis: Agent S2 reduces grounding errors relative to other reported bottlenecks, while planning failures become its most frequent failure category.Interaction and navigation failures are less common, consistent with the reported scaling analysis.
- Main Results: Agent S2 outperforms the previous best AndroidWorld method by 16.5% relatively, supporting generalization beyond desktop operating systems.The AndroidWorld study is presented as evidence of the framework’s generalizability and modularity.
5 Conclusion
Agent S2 combines generalist and specialist models for high-level reasoning, low-level execution, and detailed grounding, and achieves state-of-the-art results across computer- and smartphone-use benchmarks.
- Agent S2 integrates generalist and specialist models for high-level reasoning, low-level execution, and detailed grounding.
- Its Mixture of Grounding approach uses multiple experts for precise grounding across diverse applications.
- Proactive Hierarchical Planning refines plans and contextualizes observations based on user instructions.
- Agent S2 achieves state-of-the-art performance on two computer-use benchmarks and one smartphone-use benchmark.
A Domain Specific Results on OSWorld
The OSWorld results are organized by domain for both 50-step and 15-step evaluations, with additional comparisons across Manager and Worker backbone models.
- The 50-step OSWorld evaluation reports success rates by domain across ten desktop and application categories.
- Both evaluations compare Agent S2 configurations using different models as the Manager and Worker backbones.
- The 15-step OSWorld evaluation reports success rates divided by domains.
B App Specific Results on WindowsAgentArena
The WindowsAgentArena results report Agent S2 success rates by application while using Claude-3.7-Sonnet.
- WindowsAgentArena success rates are reported separately for twelve applications, including browsers, development tools, office software, and system utilities.
- The evaluation uses Claude-3.7-Sonnet with results divided by apps such as Chrome, Microsoft Edge, VS Code, and File Explorer.
C Agent S2 Action Space
Agent S2 uses a structured, function-calling-inspired action space to streamline action selection and parameter specification, with action types and arguments documented in Table 8.
- The structured action space streamlines action selection and parameter specification through a function-calling-inspired interface.
- Table 8 documents each Agent S2 action type, its arguments, and a brief usage description.
D Case Studies on OSWorld
OSWorld case studies illustrate Agent S2’s Mixture of Grounding and Proactive Planning through precise spreadsheet and text operations, plus adaptive replanning in settings. The examples show specialist selection for cell- and word-level interactions and revised plans when labels or panels differ from expectations.
- Mixture of Grounding: Agent S2 uses a textual grounding expert for precise word-level selection when applying strikethrough to a document paragraph.Figure 9 presents this behavior as an example of grounding experts in action.
- Mixture of Grounding: Agent S2 uses structural grounding for cell-level spreadsheet operations, creating a Profit column and calculating weekly profit as Sales minus COGS.The example specifies the header in D1, the formula =B2-C2 in D2, and filling it through D11.
- Proactive Planning: The settings example progressively refines actions from opening Settings and Power to searching for equivalent controls and toggling the relevant option off.The initial plan targets “Dim screen when inactive,” while later plans account for version-specific differences.
- Proactive Planning: When the requested setting is absent under its expected name or panel, Agent S2 revises its plan by checking Displays, returning to Power, and considering related dimming controls.The replanning sequence ultimately considers the Screen Blank setting and available alternatives.
E Case Studies on WindowsAgentArena
The WindowsAgentArena case study demonstrates consistent replanning while creating a Desktop shortcut for the Projects folder in Documents. Agent S2 revises the shortcut-creation subtask into increasingly detailed navigation and naming steps.
- Initial plan: The initial plan opens the Create Shortcut wizard, browses to This PC and Documents, selects Projects, and enters the shortcut name.The subsequent plans retain the same target while specifying the wizard interactions more concretely.
- Replanning: Later plans refine the subtask by selecting the item-location field, entering the name, and clicking Finish.These steps provide a more detailed execution sequence for the same shortcut task.
- WindowsAgentArena case study: Agent S2 creates a Desktop shortcut for the Projects folder in Documents and names it “Projects - Shortcut”.Figure 12 uses this task to demonstrate dynamic plan updates and subtask revision.
F Case Studies on AndroidWorld
The AndroidWorld case study shows Agent S2 completing a new-contact task through open, touch, and type interactions. The example enters the specified contact identity, phone number, and Work label.
- AndroidWorld case study: Agent S2 uses open, touch, and type interactions to enter Grace Taylor’s contact details with the specified phone number and Work label.Figure 13 presents this as a qualitative AndroidWorld example.