Agent S: An Open Agentic Framework that Uses Computers Like a Human

Saaket Agashe; Jiuzhou Han; Shuyu Gan; Jiachen Yang; Ang Li; Xin Eric Wang

Source-linked AI summary

Agent S: An Open Agentic Framework that Uses Computers Like a Human

Saaket Agashe, Jiuzhou Han, Shuyu Gan, Jiachen Yang, Ang Li, Xin Eric Wang

arXiv:2410.08164v1Oct 10, 2024cs.AIcs.CLcs.CV

TL;DR

Autonomous GUI agents must acquire changing domain knowledge, plan over long task horizons, and operate dynamic interfaces. Agent S addresses these challenges with experience-augmented hierarchical planning and a language-centric Agent-Computer Interface, achieving state-of-the-art OSWorld performance and generalization across operating systems. The paper identifies task-completion time and agent steps as important directions for future evaluation.

Problem
GUI task automation requires specialized knowledge, long-horizon planning, and reliable interaction with dynamic, non-uniform interfaces.
Method
Agent S combines external web knowledge, hierarchical retrieval of narrative and episodic experience, continual memory updates, and a bounded language-centric ACI for GUI control.
Results
Agent S improves OSWorld success rate from 11.21% to 20.58%, an 83.6% relative improvement, and generalizes to WindowsAgentArena without explicit adaptation.
Takeaways & Limitations
Experience-augmented planning and an ACI support state-of-the-art computer-task automation on OSWorld and generalization across different operating systems.
Takeaways & Limitations
The evaluation does not address agent steps or wall-clock time, and the experiments rely on GPT-4o and Claude-3.5-sonnet rather than smaller open-source MLLMs.

Abstract

from arXiv · show

We present Agent S, an open agentic framework that enables autonomous interaction with computers through a Graphical User Interface (GUI), aimed at transforming human-computer interaction by automating complex, multi-step tasks. Agent S aims to address three key challenges in automating computer tasks: acquiring domain-specific knowledge, planning over long task horizons, and handling dynamic, non-uniform interfaces. To this end, Agent S introduces experience-augmented hierarchical planning, which learns from external knowledge search and internal experience retrieval at multiple levels, facilitating efficient task planning and subtask execution. In addition, it employs an Agent-Computer Interface (ACI) to better elicit the reasoning and control capabilities of GUI agents based on Multimodal Large Language Models (MLLMs). Evaluation on the OSWorld benchmark shows that Agent S outperforms the baseline by 9.37% on success rate (an 83.6% relative improvement) and achieves a new state-of-the-art. Comprehensive analysis highlights the effectiveness of individual components and provides insights for future improvements. Furthermore, Agent S demonstrates broad generalizability to different operating systems on a newly-released WindowsAgentArena benchmark. Code available at https://github.com/simular-ai/Agent-S.

1 INTRODUCTION

Agent S targets the domain knowledge, long-horizon planning, and dynamic-interface challenges of autonomous GUI task automation through experience-augmented planning and an Agent-Computer Interface. It improves benchmark performance and generalizes across operating systems.

Motivation: Autonomous GUI agents aim to automate varied tasks such as data entry, scheduling, and document creation through direct keyboard and mouse interaction.The paper also connects this automation to efficiency and accessibility gains.
Challenges: GUI task automation requires specialized knowledge, long-horizon plans with interdependent actions, and interpretation of dynamic, non-uniform interfaces.Agents must distinguish relevant elements, interpret graphical cues, and respond to visual feedback during execution.
Approach: Agent S uses external web knowledge and stored task experiences to decompose complex tasks into manageable subtasks and refine execution over time.Narrative Memory supports high-level planning, while Episodic Memory provides detailed subtask experience; successful experiences are stored for continual improvement.
Approach: Agent S’s language-centric ACI combines visual input, an image-augmented accessibility tree, and bounded language-based primitives for precise GUI interaction.The interface is designed to improve grounding, safety, efficiency, and task-relevant feedback for MLLM-based agents.
Results: Agent S improves OSWorld overall performance from 11.21% to 20.58%, an 83.6% relative improvement, establishing new state-of-the-art results.The introduction reports consistent improvements across five broad computer task categories.
Contributions: The framework integrates hierarchical planning, continual memory updates, and an ACI, with experiments on OSWorld and WindowsAgentArena demonstrating component effectiveness and cross-operating-system generalizability.These contributions are presented as a unified framework for complex computer tasks.

2 RELATED WORK

Prior work develops MLLM agents with memory, planning, tools, and environmental action, while GUI-agent research has expanded from web navigation to operating-system environments. Retrieval augmentation supplies external knowledge, exemplars, guidelines, and past experiences, with Agent S distinguishing its hierarchical use of experience.

MLLM Agents: MLLM agents augment language models with memory, structured planning, tool use, and the ability to act in external environments.These systems have been applied across domains including embodied simulators and video games.
GUI Agents: GUI-agent research has progressed from web navigation tasks toward OS-level environments and benchmarks such as OSWorld and WindowsAgentArena.The cited work reflects a shift toward direct interaction with operating-system interfaces.
Retrieval-Augmented Generation: Retrieval augmentation supports MLLM agents with up-to-date external knowledge, task exemplars, state-aware guidelines, and past experiences.Agent S’s use of experience differs in its hierarchical planning across full-task and subtask experience.

3 AGENT S

Agent S combines hierarchical planning, continual memory, and a specialized Agent-Computer Interface in a closed loop for complex GUI tasks. Managers retrieve external and internal experience to plan subtasks, workers execute them with grounded actions, and self-evaluation updates memory for future tasks.

Framework overview: Agent S integrates experience-augmented hierarchical planning, continual narrative and episodic memory updates, and an Agent-Computer Interface for GUI control.These strategies operate together in a closed loop for complex operating-system tasks.
Hierarchical planning: The Manager retrieves web knowledge and similar narrative-memory experiences, fuses them, and decomposes user tasks into a topologically sorted queue of subtasks.The query incorporates the user task and initial ACI observation before retrieval and planning.
Subtask execution: Workers retrieve subtask experiences and use reflective advice to generate structured, grounded actions that the ACI executes sequentially.Each action includes status checking, observation analysis, semantic planning, and grounding.
Memory and feedback: Self-evaluation summarizes successful subtask and full-task trajectories, storing them as experience for Worker episodic memory and Manager narrative memory.The evaluator provides textual rewards without human feedback or ground-truth information.
Memory construction and update: Initial memories are bootstrapped through self-supervised exploration and then continually updated as Agent S encounters new inference tasks.Exploration uses environment-independent and environment-aware synthetic tasks, while later updates occur during interaction with novel tasks.
Agent-Computer Interface: The ACI combines screenshots with tagged accessibility trees for perception and grounding, while restricting agents to primitive one-action-at-a-time commands with immediate feedback.The bounded action space includes operations such as click, type, and hotkey, improving control over individual transitions.

4 EXPERIMENTS

Agent S is evaluated on OSWorld and WindowsAgentArena, with experiments covering main results, qualitative execution, component ablations, and error analysis. Results show stronger performance than baselines and identify contributions from experience, ACI, hierarchical planning, and memory construction.

Main Results: 20.58% overall success on OSWorld nearly doubles the best corresponding GPT-4o baseline of 11.21%.Agent S also reaches 27.06% on Daily and 36.73% on Professional tasks, versus baseline results of 12.33% and 14.29%.
Qualitative Examples: Agent S completes a Thunderbird account-removal task through sequential desktop interactions involving screenshots, subtasks, and grounding actions.The example follows the task across steps 1, 4, and 6.
Ablation Study: Removing experience components shows that web knowledge, task experience, and subtask experience contribute to GUI-agent performance.The ablation evaluates how different experience sources affect task categories in OSWorld testsub.
Ablation Study: ACI improves reasoning and agentic learning, while removing hierarchical planning lowers performance from 26.15% to 20.00%.The hierarchical-planning drop is measured for the ACI-only plus Experiential Learning setup compared with full Agent S.
Error Analysis: Execution and grounding errors are the most common error types among failed Agent S tasks.The error analysis distinguishes planning, grounding, and execution errors and reports subtask failure and error rates.

5 CONCLUSION

The paper presents Agent S as a GUI-agent framework combining experience-augmented hierarchical planning, continual learning, and an Agent-Computer Interface. It reports state-of-the-art OSWorld performance, cross-operating-system generalization, and a need to evaluate efficiency and smaller models.

Conclusion: Agent S combines experience-augmented hierarchical planning, Online Web Knowledge, continual memory, and an Agent-Computer Interface for autonomous GUI control.The framework is designed for agents that directly control keyboards and mice while learning from external sources and environment interaction.
Conclusion: The framework achieves state-of-the-art performance on OSWorld and generalizes across operating systems.The conclusion identifies generalizability across different operating systems as a supported outcome of the framework.
Future Work: The paper leaves agent steps and wall-clock completion time unaddressed as evaluation metrics.Future work proposes shortest-path navigation and Pareto analysis across time and accuracy.
Future Work: Future work could extend experiential learning and ACI to smaller open-source MLLMs.The authors use GPT-4o and Claude-3.5-Sonnet and suggest fine-tuning smaller models to narrow the gap.

A AGENT-COMPUTER INTERFACE

Agent S uses a constrained action space to make GUI action selection more structured and precise. The action space defines action types together with their parameters and descriptions.

A.1 Constrained Action Space: The constrained action space simplifies action selection by restricting the agent to a structured set of operations.This design is intended to facilitate accurate and effective task execution.
A.1 Constrained Action Space: Table 5 organizes each action type with its required parameters and detailed description.The table specifies the available agent actions and their arguments.

A.2 ABLATIONS ON AGENT COMPUTER INTERFACE

The ablation study examines how Retrieval-as-Learning affects the baseline and Agent S models. Its impact is greater for Agent S.

Retrieval-as-Learning improves performance for both the Baseline and Agent S models.
The performance improvement from Retrieval-as-Learning is notably greater for Agent S.
The ablation is reported in Table 6.

A.3 ABLATIONS ON LEARNING

The learning ablation evaluates components of Agent S’s experience-augmented hierarchical planning. Continual Learning and the Self-Evaluator are identified as critical contributors to performance.

Continual Learning enhances Agent S performance.
The Self-Evaluator enhances Agent S performance.
The experience-augmented hierarchical planning ablation is reported in Table 7.

B DETAILED RESULTS ON OSWORLD AND WINDOWSARENA

The detailed-results material describes Agent S’s retrieval, planning, execution, reflection, and memory-update pipeline, alongside app-domain breakdowns for OSWorld and WindowsAgentArena.

Domain breakdowns: The detailed results are organized by application domains for OSWorld and WindowsAgentArena.
Manager planning: Agent S retrieves external knowledge and internal narrative experience using an observation-aware query.
Manager planning: The Manager uses fused knowledge to generate ordered subtasks with associated contexts for Worker execution.
Worker execution: Workers retrieve subtask experience, reflect on trajectories, generate grounded actions, and signal DONE or FAIL.
Experience updates: Successful subtask execution produces episodic experience updates, while completed tasks produce narrative-memory rewards.

D SUPPLEMENTARY EXAMPLES FOR QUALITATIVE ANALYSIS

The supplement provides additional successful and failed task examples and expands the paper’s error analysis to complement the qualitative analysis.

The supplement includes additional examples of successful tasks.
The supplement includes additional examples of failed tasks.
The supplement provides more detailed error analysis alongside the qualitative analysis.

D.1 SUCCESS EXAMPLES

Agent S completes diverse desktop tasks across applications and operating-system workflows, while its trajectories still contain recoverable mistakes and failures to recognize completion.

Successful task examples: Agent S demonstrates successful task completion across multi-app cloning, LibreOffice Impress, VSCode, LibreOffice Writer, and Ubuntu file recovery.The examples span repository management, application settings, text replacement, paragraph formatting, and Trash recovery.
Execution issues: The successful trajectories still include incorrect field selection, inappropriate actions, and redundant recovery attempts after the task is complete.These issues occurred in the VSCode, LibreOffice Writer, and Ubuntu recovery examples, respectively.

D.2 DETAILED ERROR ANALYSIS AND FAILURE EXAMPLES

Agent S’s failures arise from planning, grounding, and execution errors that can interact and produce repetitive or unsuccessful behavior. Across 39 execution errors, planning or grounding errors accounted for 46%.

Error sources: 46% of Agent S’s 39 execution errors were caused by planning or grounding errors.The analysis identifies grounding errors as frequent causes of repetitive actions and suggests reducing them could improve performance.
Failure examples: In the Chrome failure, an irrelevant “Updating of Chrome” subtask produced an incorrect sequence that prevented the extension from being turned off.The planning deficiency propagated into an execution error.
Failure examples: In the GIMP failure, repeatedly grounding to “Activity” instead of the “Fuzzy Select Tool” led to an Overview state and subsequent execution errors.The agent did not promptly correct the repeated misselection.
Failure examples: The analyzed failure cases include Calc sequence filling, Chrome privacy cleanup, and GIMP background transparency tasks.These examples illustrate distinct interactions between planning, grounding, and execution errors.