PaperlayerView exact revision on arXiv

Source-linked AI summary

OS-Copilot: Towards Generalist Computer Agents with Self-Improvement

Zhiyong Wu, Chengcheng Han, Zichen Ding, Zhenmin Weng, Zhoumianze Liu, Shunyu Yao, Tao Yu, Lingpeng Kong

arXiv:2402.07456v2cs.AI

TL;DR

General computer agents remain limited by their focus on individual applications rather than entire operating systems. OS-Copilot provides a unified OS-level interface and supports FRIDAY, a self-improving agent that learns unfamiliar applications; on GAIA, FRIDAY achieves a 35% relative improvement over the previous best system.

  • Problem

    Existing language agents mainly target standalone applications, leaving effective interaction with the entire heterogeneous operating system underexplored.

  • Method

    OS-Copilot unifies code, terminal, mouse/keyboard, and API control, while FRIDAY uses self-directed learning to accumulate tools for unfamiliar applications.

  • Results

    FRIDAY demonstrates generalization across unseen applications and achieves a 35% relative improvement over the previous best system on GAIA level-1 tasks.

  • Takeaways & Limitations

    OS-Copilot provides infrastructure for generalist computer agents across heterogeneous operating-system applications and future situated-agent research.

  • Takeaways & Limitations

    OS-Copilot and FRIDAY rely on prompt engineering and cannot operate closed-source applications.

Abstract

from arXiv · show

Autonomous interaction with the computer has been a longstanding challenge with great potential, and the recent proliferation of large language models (LLMs) has markedly accelerated progress in building digital agents. However, most of these agents are designed to interact with a narrow domain, such as a specific software or website. This narrow focus constrains their applicability for general computer tasks. To this end, we introduce OS-Copilot, a framework to build generalist agents capable of interfacing with comprehensive elements in an operating system (OS), including the web, code terminals, files, multimedia, and various third-party applications. We use OS-Copilot to create FRIDAY, a self-improving embodied agent for automating general computer tasks. On GAIA, a general AI assistants benchmark, FRIDAY outperforms previous methods by 35%, showcasing strong generalization to unseen applications via accumulated skills from previous tasks. We also present numerical and quantitative evidence that FRIDAY learns to control and self-improve on Excel and Powerpoint with minimal supervision. Our OS-Copilot framework and empirical findings provide infrastructure and insights for future research toward more capable and general-purpose computer agents.

1 INTRODUCTION

Existing language agents largely target standalone applications, leaving effective interaction with an entire operating system underexplored. OS-Copilot addresses this gap with a unified OS interface and FRIDAY, a self-improving agent that automates general computer tasks and performs strongly on GAIA.

  • Motivation: Language-agent efforts have primarily focused on standalone applications, with limited exploration of agents that interact effectively with an entire operating system.Examples include web browsers, command-line terminals, Minecraft, and databases.
  • OS-Copilot: OS-Copilot provides a universal interface for building computer agents on Linux and MacOS across heterogeneous OS applications.The interface consolidates Python code interpretation, bash terminals, mouse/keyboard control, and API calls.
  • FRIDAY: FRIDAY is a self-improving embodied agent built on OS-Copilot that automates computer tasks and learns to control unfamiliar applications through self-directed learning.Its self-evolving configurator enables this learning capability, distinguishing FRIDAY from agents such as AutoGPT.
  • Evaluation: 40.86% success rate on GAIA level-1 tasks marked a 35% relative improvement over the previous best system (30.3%) and exceeded AutoGPT-4 (14.4%).The evaluation positions FRIDAY as a strong general computer-task solver on a benchmark for general AI assistants.
  • Contributions: OS-Copilot supports future research on personalized digital assistants, multimodal agents, and situated agent learning, while FRIDAY shows generalization across unseen applications.The evaluation results and case studies also underscore FRIDAY’s potential as a helpful OS assistant.

2 THE OS-COPILOT FRAMEWORK

OS-Copilot decomposes user requests into subtasks, configures each with retrieved tools and knowledge, and has an actor execute and critique operations iteratively. Its architecture combines directed acyclic graph planning, memory modules, universal OS runtimes, and self-correction.

  • Planner: The planner decomposes complex user requests into subtasks while retrieving capability information to produce plans at the appropriate granularity.OS-Copilot supports planning methods including Plan-and-Solve and directed acyclic graph-based planning.
  • Planner: The directed acyclic graph planner parallelizes independent tasks, reducing the need for sequential execution.The paper illustrates this with simultaneously monitoring deep-learning training and generating inference code.
  • Configurator and memory: The configurator uses working, declarative, and procedural memory to retrieve relevant context, tools, user preferences, environment knowledge, and learned skills for each subtask.Declarative memory includes a User Profile and Semantic Knowledge, while procedural memory is primarily a tool repository.
  • Working memory: Working memory connects the planner, configurator, and actor by retrieving and updating long-term information, supplying subtask context, and incorporating execution feedback.It can update tool code following self-correction and revise information based on actor feedback.
  • Actor: The actor executes commands or function calls through Python, bash, APIs, and mouse/keyboard controls, then uses system-state assessment and LLM-generated criticism to correct errors and update memory.The critic evaluates completion, analyzes failures, suggests tool or action corrections, and may recommend restructuring subtasks.

3 THE FRIDAY AGENT

FRIDAY is designed as a generalist computer agent that improves through self-refinement and self-directed learning. It retrieves context, creates or executes tools, evaluates outcomes, and accumulates reusable capabilities while pursuing task curricula.

  • FRIDAY maximizes generality by combining self-refinement with self-directed learning for unfamiliar computer applications.The framework emphasizes both iterative correction during task execution and autonomous acquisition of new skills.
  • Dense retrieval constructs a configuration prompt containing relevant tools, user profiles, OS versions, and the working directory.This process is illustrated for the subtask “Change the system into the Dark mode.”
  • When no suitable tool exceeds the similarity threshold, the Tool Generator creates an application-tailored Python class using AppleScript.In the example, the generated tool changes the system to dark mode.
  • The Executor converts the finalized prompt into an executable action, stores generated tool code in a Python file, and runs it through the command-line terminal.
  • After execution, the critic scores successful tools from 0 to 10, preserving tools scoring above 8 in procedural memory; failures trigger refinement for up to three attempts.The refiner uses critic feedback to correct the responsible action, tool, or subtask.
  • Given a learning objective, FRIDAY generates an easy-to-challenging task curriculum, solves tasks through trial and error, and accumulates tools and semantic knowledge.

4 EXPERIMENTS

Experiments evaluate FRIDAY on the general-purpose GAIA benchmark and examine its self-directed learning on spreadsheet and PowerPoint tasks. FRIDAY achieves strong performance on GAIA and demonstrates that self-directed learning enables control of unfamiliar applications.

  • GAIA Evaluation: FRIDAY is evaluated on GAIA’s 466 challenging question-answering tasks after expanding its initial four tools with nine additional tools on the development set.The test-set results use GPT4-turbo-1106 because of limited budget.
  • Root of Improvement: FRIDAY’s advantage over AutoGPT-4 highlights self-criticism and refinement, while its broader tool access is contrasted with GPT4 Plugins.The paper presents OS-Copilot as a cohesive combination of these components and a design guideline for future general computer agents.
  • Ablation Study: Disabling self-directed learning still leaves FRIDAY significantly ahead of all baselines, but the performance gain from learning exposes the limitations of predefined tool sets in open environments.The ablation compares FRIDAY with FRIDAY (w/o learning).
  • Self-Directed Learning: On 20 spreadsheet control tasks, FRIDAY initially completes none without self-directed learning but surpasses SheetCopilot after learning, despite SheetCopilot’s manually crafted and verified tools.The tasks cover formatting, management, charts, pivot tables, and formulas.
  • Self-Directed Learning: In a PowerPoint slide construction task, self-directed learning enables FRIDAY to master unfamiliar application control after initially struggling with fonts, image positioning, and image sizing.The evaluated slide specifies content, font, font size, and other details in the task instruction.

5 RELATED WORK

Related work situates LLM-powered language agents across robotics, web manipulation, and games, while emphasizing that most digital agents remain tailored to specific scenarios. It also characterizes language agents as complex pipelines with diverse components, including task planning.

  • Autonomous agents powered by LLMs, also called language agents, are a rapidly growing research field attracting substantial attention.
  • Most digital language agents are tailored to specific scenarios, including web manipulation, command-line coding, spreadsheet control, and Minecraft.Language agents have also been designed for robotics, web manipulation, and games.
  • Language agents form complex systems with lengthy pipelines whose components each represent substantial bodies of research.Recent surveys are recommended for comprehensive methodological coverage.
  • During planning, researchers decompose complex tasks into sequential series of simpler subtasks.

6 CONCLUSION · A GRAPH-BASED PLANNER

OS-Copilot provides an OS-level framework that enables FRIDAY to solve open-environment computer tasks and learn to control unseen applications through self-directed learning. Its graph-based planner represents task dependencies in a directed acyclic graph, enabling parallel execution of independent subtasks and sequential execution where dependencies require it.

  • 6 CONCLUSION: OS-Copilot is a framework for OS-level language agents that supports FRIDAY, an embodied computer agent.The framework is designed for interaction with operating-system-level tasks.
  • 6 CONCLUSION: FRIDAY demonstrates strong performance on open-environment computer tasks and learns to control previously unseen applications through self-directed learning.The passage identifies self-directed learning as the mechanism enabling adaptation to unseen applications.
  • A GRAPH-BASED PLANNER: Current planners generally require tasks to execute sequentially, despite computers’ capacity to process multiple tasks concurrently.The passage contrasts linear and non-linear planners that still execute tasks one after another with computers’ ability to handle concurrent work.
  • A GRAPH-BASED PLANNER: Parallelizing independent tasks can reduce overall execution time, such as training a machine-learning model while generating inference code.This example illustrates how independent coding-agent subtasks can proceed simultaneously.
  • A GRAPH-BASED PLANNER: The planner formalizes tasks as a directed acyclic graph in which arrows represent interdependencies between subtasks.The graph structure captures which tasks depend on the completion of others.
  • A GRAPH-BASED PLANNER: Tasks 1, 2, and 3 can initially run simultaneously, while tasks 4 and 6 follow completion of tasks 2 and 3, respectively.The running example combines parallel execution at the start with dependency-triggered sequential execution afterward.
  • A GRAPH-BASED PLANNER: Task 5 must wait until both tasks 1 and 4 have been completed, demonstrating coordination across multiple prerequisites.This dependency illustrates how the graph enforces execution constraints among subtasks.

B EXTENDED EVALUATION · C POSSIBLE USE CASES OF OS-COPILOT

The extended evaluation shows that FRIDAY is substantially faster than AutoGPT-4 and humans on Level 1 tasks, while excelling in coding and file I/O but remaining limited in web interaction and multimodal handling. OS-Copilot’s supported OS interaction methods are illustrated through exemplary use cases.

  • B EXTENDED EVALUATION: 105 seconds versus more than 500 seconds for AutoGPT-4, FRIDAY executes Level 1 tasks with a fourfold speed increase.FRIDAY’s measurement includes tool retrieval and generation time.
  • B EXTENDED EVALUATION: FRIDAY is three times faster than human performance on Level 1 tasks.
  • B EXTENDED EVALUATION: FRIDAY’s timing is slower than GPT-4 Plugins, whose measurement excludes manual plugin-selection time by humans.
  • B EXTENDED EVALUATION: The evaluation manually verified the correctness of all 93 Level 1 tasks because the test data were not released.
  • B EXTENDED EVALUATION: FRIDAY excels at processing coding and file I/O, but shows weaknesses in web browsing and multimodal handling.
  • B EXTENDED EVALUATION: FRIDAY currently retrieves information from websites but cannot perform web actions such as clicks.The paper anticipates enhancement through recent advances in web agents.
  • C POSSIBLE USE CASES OF OS-COPILOT: OS-Copilot supports four OS interaction methods, illustrated with exemplary use cases in Table 4.

D DISCUSSION · E PROMPTS

The discussion identifies limitations in prompting, closed-source software, evaluation, safety, and interpretability, while outlining multimodal, fine-tuning, and reinforcement-learning directions. The following prompts section points readers to FRIDAY’s core-component prompts in Tables 5–9.

  • D DISCUSSION: OS-Copilot and FRIDAY rely on prompt engineering and cannot operate closed-source applications.Prompt-based construction may change substantially when the underlying LLM changes.
  • D DISCUSSION: OS-Copilot follows the OpenAI-Gymnasium interface to facilitate future reinforcement learning and fine-tuning research.Effective language-agent training remains difficult because it requires substantial data.
  • D DISCUSSION: Multimodal control is needed because code-and-language interaction is infeasible for closed-source commercial software.The proposed direction includes visual input and screenshot-to-action generation, building on visual-language agents for web and mobile environments.
  • D DISCUSSION: Evaluating general computer agents is difficult because subtasks lack reliable ground truth in OS environments.FRIDAY decomposes tasks into subtasks, but assessing whether each subtask succeeds remains challenging.
  • D DISCUSSION: OS-interacting systems must be transparent, interpretable, and safe, with outputs that are not harmful.The discussion calls for comprehensible natural-language explanations and safety alignment during learning.
  • D DISCUSSION: The authors note that their annotation results slightly differ from those returned by the evaluation server.This discrepancy is recorded as an annotation-related qualification in the discussion.
  • E PROMPTS: The core-component prompts for FRIDAY are provided in Tables 5 to 9.This section serves as an implementation reference for the prompting setup.

F TOOLS … F.4 EXAMPLES OF TOOLS GENERATED BY FRIDAY

FRIDAY is equipped with foundational tools for internet access and multimodal understanding, then self-directed learning expands its toolset for spreadsheet and slide tasks. Generated tools are designed to be atomic, broadly reusable, and compatible with the Executor’s required code format.

  • F.1 TOOLS FOR GAIA: FRIDAY receives basic tools for internet access and image and audio understanding to support GAIA task completion.These capabilities are described as crucial for completing tasks in the GAIA dataset.
  • F.1 TOOLS FOR GAIA: During GAIA development tasks, FRIDAY generates tools with atomicity and universality that improve subsequent testing performance.The generated tools are documented in Table 10.
  • F.2 TOOLS FOR SHEETCOPILOT: FRIDAY uses self-directed learning with openpyxl to accumulate spreadsheet-control tools for the SheetCopilot dataset.The accumulated tools significantly improve FRIDAY’s performance on SheetCopilot.
  • F.3 TOOLS FOR PPT: Through self-directed learning, FRIDAY learns to use python-pptx for slide generation and manipulation.The accumulated tools and their descriptions are provided in Table 12.
  • F.4 EXAMPLES OF TOOLS GENERATED BY FRIDAY: The paper provides a representative example of a tool generated by FRIDAY to make its learned tools more intuitive.The example appears in Table 13.
  • F.4 EXAMPLES OF TOOLS GENERATED BY FRIDAY: Generated tool code follows a strictly defined format that matches the Executor.The specific code-format requirements are detailed in the Tool Generator prompts shown in Table 5.

G TASK INSTRUCTION · Graph-based Planner

The appendix specifies prompts and task instructions for FRIDAY’s graph-based planner and related components. The planner decomposes tasks into atomic subtasks represented as a directed acyclic graph, while other prompts guide tool generation and execution.

  • G TASK INSTRUCTION: The PowerPoint task instruction asks FRIDAY to generate a slide introducing OS-Copilot.Following Guo et al. (2023), it specifies content, font, font size, and the positioning of text and images.
  • Graph-based Planner: The graph-based planner decomposes a task into atomic subtasks that form a directed acyclic graph.The complete task is executed by topologically sorting the subtasks.
  • Graph-based Planner: Each subtask records a name, description, dependencies, and task type in the planner’s JSON output.The type distinguishes Code, API, and QA tasks, while dependencies identify tasks that must execute first.
  • Graph-based Planner: The planner prompt illustrates file retrieval and organization as separate subtasks connected by dependencies.The example retrieves text files containing “agent” and organizes the retrieved files into an agent folder.
  • Graph-based Planner: The appendix identifies dedicated prompts for the planner, tool generator, refiner, executor, and critic.Their tables distinguish information contributed by declarative or procedural memory and, where applicable, by other components.
  • Graph-based Planner: The tool generator prompt instructs FRIDAY to produce Python classes for user-specified operating-system tasks.The generated response includes Python code and an invocation statement, using the specified BaseAction structure.
  • Graph-based Planner: The appendix catalogs basic and FRIDAY-generated tools, including tools acquired for spreadsheet control and PowerPoint creation.It also includes an example of tool code generated by FRIDAY.
Loading 2402.07456v2…