Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agents

• 27.0% (↑18.9%) on OSWorld 15-step and 34.5% (↑32.7%) on 50-step; WindowsAgentArena 29.8% (↑52.8%) and AndroidWorld 54.3% (↑16.5%) set new SOTA.Source: Figure 1 · p. 1, §1 · p. 2
• Existing agents struggle with pixel-level GUI grounding, long-horizon tasks amid distractions and changing contexts, and bottlenecks from using generalist models for specialized subtasks.Source: §1 · p. 2
• Agent S2 combines Mixture of Grounding, which routes actions to specialist experts for GUI localization, with Proactive Hierarchical Planning that refines plans at multiple temporal scales as observations change.This compositional system delegates high-level reasoning, low-level execution, and fine-grained grounding rather than relying on a single generalist model.Source: §1 · p. 2
• Ablations examine Mixture of Grounding and Proactive Planning against reactive planning, while further analyses cover compute and time-step scaling, errors, and emergent behaviors.Source: §1 · p. 2
• Composing generalist and specialist models can outperform the best monolithic models even when each component is slightly suboptimal on its own.Source: §1 · p. 2

Computer Use Tasks and Benchmarks

• Computer-use tasks require agents to interpret instructions and screen observations, then execute action sequences in a partially observable environment.The POMDP formulation defines states, observations, actions, transitions, and rewards.Source: §2 · pp. 2–3
• Recent online benchmarks evaluate computer-use agents across desktop, Windows, and mobile tasks, while other benchmarks focus on visual grounding.OSWorld covers Ubuntu desktop tasks, WindowsAgentArena provides a parallelizable Windows environment, and AndroidWorld tests mobile UI navigation.Source: §2 · p. 3

Monolithic Methods for Computer Use Agents

• Monolithic agents combine planning, execution, and UI grounding, but specialization can erode broader capabilities and a single model may not suit each responsibility.These agents need long- and short-term reasoning plus precise localization of interactable elements.Source: §2 · p. 3

Hierarchical Methods for Computer Use Agents

• Hierarchical systems separate planning into subgoals from workers’ atomic actions, but worker grounding bottlenecks and reactive planning can reduce robustness to unexpected changes.Prior work also explores knowledge augmentation, skill libraries, verification, multi-agent responsibility sharing, and separate grounding models (Agashe et al., 2024; Wu et al., 2024a; Wang & Liu, 2024; Tan et al., 2024; Wu et al., 2024b; Gou et al., 2024; Yang et al., 2024).Source: §2 · pp. 3–4

3 Agent S2: Compositional Grounding and Planning for Computer Use

• Agent S2 pairs a high-level Manager that decomposes tasks into subgoals with a low-level Worker that executes the topmost subgoal and routes actions to grounding experts.The framework also uses Agent S’s knowledge base of high-level and low-level interaction experience and contextual web knowledge.Source: §3 · p. 4

3.1 Mixture of Grounding

• The Worker acts as a gate, routing each action to the appropriate grounding expert, which supplies pixel-level coordinates so the Worker can focus on reasoning.The Worker receives the subgoal and latest observation, generates an atomic action with a language description of its target, then delegates grounding.Source: §3.1 · p. 4
• Visual grounding maps screenshot descriptions to arbitrary pixel coordinates without accessibility trees; OCR grounds text spans from exact start/end phrases; structural grounding updates mapped cells, rows, columns, or tables.The Worker can refine visual descriptions to self-correct actions, while structural grounding can update multiple cells or entire tables at once.Source: §3.1 · p. 5

3.2 Proactive Hierarchical Planning

• After each completed subgoal, the Manager updates the remaining plan using new observations, while the Worker selects the next subgoal and routes each action to an expert.This proactive replanning adapts to evolving observations and preserves context from previous subgoals to reduce susceptibility to UI noise.Source: §3 · p. 4, §3.2 · p. 5

4.1 Experimental Setup

• Experiments cover OSWorld, WindowsAgentArena, and AndroidWorld, with a 65-example OSWorld subset used for ablations.The benchmarks comprise 369 OSWorld tasks, 154 WindowsAgentArena tasks, and 116 AndroidWorld tasks across 20 applications. Comparisons include screenshot-input baselines; OSWorld is evaluated at 15 and 50 steps, while the WindowsAgentArena comparison uses a baseline with both accessibility-tree and screenshot inputs.Source: §4.1 · p. 6

4.2 Main Results

• Agent S2 outperforms the previous best WindowsAgentArena agent, NAVI, by 52.8%, with improvements across 4/6 categories.The benchmark comparison notes that the prior agent uses accessibility-tree and screenshot inputs, whereas Agent S2 requires only screenshots; it also performs well on Windows-specific tasks.Source: Table 3 · p. 7, §4.2 · p. 7
• Agent S2 with Claude-3.5-Sonnet (new) relatively outperforms Claude Computer Use with Claude-3.7-Sonnet by 58.1% at 15 steps and 29.6% at 50 steps, while both Agent S2 variants achieve new SOTA on OSWorld.On the 50-step category breakdown, Agent S2 performs strongly on OS, Daily, and Professional tasks across backbones and competitively on Office; Claude-3.5-Sonnet (new) surpasses the Claude-3.7-Sonnet variant on Office and relies on textual and structural grounding experts almost twice as often.Source: §4.2 · p. 7, Table 2 · p. 7

4.3 Ablation Study

• Mixture of Grounding raises success rate from 27.69% to 30.77% at 15 steps and from 33.85% to 38.46% at 50 steps.Removing the textual grounding expert lowers subtask success rate from 70.6% to 65.2%, while removing the structural grounding expert lowers it from 73.7% to 69.4%.Source: §4.3 · pp. 7–8, Figure 5 · p. 8
• Proactive Hierarchical Planning improves success rate over reactive planning by +4.62% at 15 steps and +6.15% at 50 steps.Its advantage grows with more steps because it adapts plans to evolving observations.Source: Figure 5 · p. 8, §4.3 · p. 8
• UI-TARS-7B-DPO and UGround-V1-7B can outperform Claude-3.7-Sonnet as visual grounding experts within Agent S2’s modular framework.Among 29 OSWorld tasks that fail at 15 steps but succeed at 50, the analysis identifies adaptive navigation, adaptive interaction, backward correction, and tasks requiring more than 15 steps as contributing behaviors.Source: Figure 6 · p. 8, §4.3 · pp. 8–9, Figure 7 · p. 9

4.4 Error Analysis

• Planning failures are Agent S2’s most frequent failure type, while grounding errors are relatively less frequent and interaction and navigation failures are less common.The analysis groups failures into planning, grounding, interaction, navigation, and infeasible tasks; planning problems typically involve noisy or misaligned subtask information.Source: Figure 8 · p. 9, §4.4 · p. 9

4.5 Generalization to Smartphone Use

• Agent S2 outperforms the previous state-of-the-art method on AndroidWorld by 16.5% relatively.The evaluation covers smartphone use across Android applications.Source: §4.5 · p. 9

5 Conclusion

• Agent S2 achieves state-of-the-art performance on two computer-use benchmarks and one smartphone-use benchmark.The conclusion also reports ablation studies and error analysis examining the roles of system components.Source: Table 4 · p. 9, §5 · p. 10
• Agent S2 integrates generalist and specialist models for high-level reasoning, low-level execution, and detailed grounding.Mixture of Grounding uses a team of experts for precise grounding across diverse applications; Proactive Hierarchical Planning refines plans and contextualizes observations based on user instructions.Source: §5 · pp. 9–10
• The authors conduct ablation studies and error analysis to highlight the roles of each system component.Source: §5 · p. 10

A Domain Specific Results on OSWorld

• Table 5 divides 50-step OSWorld success rates across ten domains, from OS and GIMP to VLC and Multiapps.It also reports performance using different Manager and Worker backbones.Source: Table 5 · p. 13
• For the 50-step evaluation, Table 5 includes comparisons using different Manager and Worker backbone models.Source: Table 5 · p. 13
• Table 6 reports Agent S2 success rates by domain in the 15-step OSWorld evaluation.It also reports performance using different Manager and Worker backbones.Source: Table 6 · p. 13

B App Specific Results on WindowsAgentArena

• Table 7 presents Agent S2’s WindowsAgentArena success rates for 12 apps, including browsers, productivity tools, Settings, and File Explorer.The caption identifies Chrome, Microsoft Edge, VS Code, Notepad, LibreOffice Calc, Settings, Windows Calc, Clock, Microsoft Paint, File Explorer, LibreOffice Writer, and VLC Player; it does not provide the success-rate values.Source: Table 7 · p. 13

C Agent S2 Action Space

• Agent S2’s structured action space streamlines action selection and parameter specification.Table 8 summarizes each action type, its parameters, and a brief description of its usage.Source: C Agent S2 Action Space · p. 13, Table 8 · p. 13

Textual grounding

• Agent S2 uses textual grounding for precise word-level selection when applying strikethrough to a paragraph.The example selects words in the last paragraph identified by the task instruction as redundant.Source: Figure 9 · p. 14

Structural grounding

• For spreadsheet cell operations, Agent S2 adaptively uses structural grounding to fill a new Profit column with weekly Sales minus COGS.The example shows the agent choosing the structural grounding expert for the column operation.Source: Figure 10 · p. 15

Proactive Planning

• When the requested setting is not found verbatim, Agent S2 proactively replans to identify the correct option.The example asks it to turn off “Dim screen when inactive” in settings.Source: Figure 11 · p. 16

E Case Studies on WindowsAgentArena

• In a WindowsAgentArena task to create a named Desktop shortcut to Documents/Projects, Agent S2 consistently replans, updating its plan and refining its current subtask.The requested shortcut name is “Projects - Shortcut”.Source: Figure 12 · p. 17, E Case Studies on WindowsAgentArena · p. 17

F Case Studies on AndroidWorld

• Figure 13 shows Agent S2 using open, touch, and type interactions to enter the specified details on a new AndroidWorld contact screen.The requested contact is Grace Taylor, with phone 799-802-XXXX and the Work label.Source: Figure 13 · p. 18, F Case Studies on AndroidWorld · p. 18