Source-linked AI summary
OS-Symphony: A Holistic Framework for Robust and Generalist Computer-Using Agent
Bowen Yang, Kaiming Jin, Zhenyu Wu, Zhaoyang Liu, Qiushi Sun, Zehao Li, JingJing Xie, Zhoumianze Liu, Fangzhi Xu, Kanzhi Cheng, Qingyun Li, Yian Wang, Yu Qiao, Zun Wang, Zichen Ding
TL;DR
Current CUA frameworks have limited control over historical visual context and lack visual-aware tutorial retrieval, hindering long-horizon robustness and novel-domain generalization. OS-SYMPHONY addresses these gaps with an Orchestrator coordinating milestone-driven reflection memory and a multimodal Searcher. It reports gains across three benchmarks, including 65.8% on OSWorld, while retaining desktop-only scope and substantial multi-agent overhead.
Problem
Current CUA frameworks lack granular historical visual-context curation and visual-aware tutorial retrieval for robust long-horizon and out-of-distribution task execution.
Method
OS-SYMPHONY uses an Orchestrator coordinating a milestone-driven Reflection-Memory Agent and Versatile Tool Agents with a SeeAct-based Multimodal Searcher.
Results
65.8% on OSWorld, 63.5% on WindowsAgentArena, and 46.0% on MacOSArena were achieved with substantial gains over current state-of-the-art methods.
Takeaways & Limitations
The framework supports robust long-horizon execution and generalization to unseen scenarios across diverse operating systems and model scales.
Takeaways & Limitations
Evaluation is confined to desktop ecosystems, while multi-agent interactions impose high token consumption and significant latency.
Abstract
from arXiv · showhide
While Vision-Language Models (VLMs) have significantly advanced Computer-Using Agents (CUAs), current frameworks struggle with robustness in long-horizon workflows and generalization in novel domains. These limitations stem from a lack of granular control over historical visual context curation and the absence of visual-aware tutorial retrieval. To bridge these gaps, we introduce OS-Symphony, a holistic framework that comprises an Orchestrator coordinating two key innovations for robust automation: (1) a Reflection-Memory Agent that utilizes milestone-driven long-term memory to enable trajectory-level self-correction, effectively mitigating visual context loss in long-horizon tasks; (2) Versatile Tool Agents featuring a Multimodal Searcher that adopts a SeeAct paradigm to navigate a browser-based sandbox to synthesize live, visually aligned tutorials, thereby resolving fidelity issues in unseen scenarios. Experimental results demonstrate that OS-Symphony delivers substantial performance gains across varying model scales, establishing new state-of-the-art results on three online benchmarks, notably achieving 65.84% on OSWorld.
1 Introduction
OS-SYMPHONY addresses weak visual-context management in long-horizon CUA tasks and inadequate visual-aware retrieval for unseen scenarios. Its Reflection-Memory Agent and Versatile Tool Agents improve robustness and generalization across benchmarks.
- Current CUA frameworks lack granular control over historical visual information, limiting retrospective error detection and meaningful reflection during complex long-horizon tasks.
- OS-SYMPHONY coordinates an Orchestrator with a Reflection-Memory Agent and Versatile Tool Agents to address these two challenges.
- 65.8% on OSWorld, 63.5% on WindowsAgentArena, and 46.0% on MacOSArena demonstrate substantial gains over current state-of-the-art methods.
- The Reflection-Memory Agent retains milestone screenshots alongside abstract trajectories and audits historical states to guide robust long-horizon planning.
- The Multimodal Searcher navigates browser-based pages to acquire visually aligned tutorials that provide external multimodal knowledge for out-of-distribution scenarios.
- Evaluations across operating systems and model scales show that open-source VLMs can execute long-horizon or unseen tasks that previously challenged them.
2 Related Work
Related work establishes CUAs, retrieval augmentation, and memory management as complementary approaches to digital task automation. OS-SYMPHONY’s framework is presented against these existing paradigms.
- Computer-Using Agents: Computer-Using Agents use Vision-Language Models to pursue digital autonomy and interact with graphical user interfaces.
- OS-SYMPHONY: OS-SYMPHONY’s pipeline combines an Orchestrator, Tool Agents, and a Reflection-Memory Agent that compresses trajectories for long-term memory and reflection.
- RAG for CUAs: Retrieval-augmented strategies access external knowledge to strengthen CUA generalization, including through general-purpose AI search engines.
- Memory for CUAs: Memory methods for long-horizon tasks compress interactions through history summarization, context folding, or multi-scale short- and long-term memory.
3 OS-SYMPHONY
OS-SYMPHONY uses an Orchestrator to coordinate reflection-driven memory and specialized tools. The RMA supports trajectory auditing, while the Searcher retrieves visually grounded tutorials in an isolated browser sandbox.
- 3.1 Orchestrator: The Orchestrator interprets tasks, coordinates Tool Agents, incorporates reflection and retrieved tutorials, and selects actions from current observations and short-term history.
- 3.1 Orchestrator: The Orchestrator restricts short-term memory to a sliding window of the last K turns for next-action prediction.
- 3.2 Reflection-Memory Agent: The Reflection-Memory Agent compresses interaction history and selectively retains milestone screenshots to reduce contextual overhead and address intent drift.
- 3.2 Reflection-Memory Agent: At each step, the RMA identifies milestones, generates trajectory-level reflections, and extracts relevant information from visual inputs.
- 3.2 Reflection-Memory Agent: The RMA classifies off-track states into GUI Error, Lack of Tutorial, Code Error, and Other Error categories.
- 3.3 Versatile Tool Agents: The Orchestrator synthesizes a Searcher for OOD knowledge, Grounders for UI localization, and a Coder for direct system interaction.
- 3.3 Versatile Tool Agents: The Multimodal Searcher uses a VLM-driven SeeAct strategy to navigate rendered pages and synthesize tutorials while preserving visual cues beyond text parsing.
- 3.3 Versatile Tool Agents: The Searcher pairs a how-to query with the current observation, navigates a sandbox using click, type, and scroll, and returns a structured tutorial when relevant.
4 Experiment
OS-SYMPHONY is evaluated on desktop benchmarks across operating systems, model scales, and component ablations. It achieves strong benchmark performance, with analyses linking gains to multimodal search, reflection memory, and framework adaptability while identifying visual-perception and planner-worker limitations.
- 4.1 Experiment Setup: OSWorld comprises 369 real-world tasks across five domains, reduced to 361 after excluding eight Google Drive tasks; WindowsAgentArena and MacOSArena test cross-platform generalization.OSWorld-Verified is the primary benchmark, while the additional arenas assess software consistency and system-specific configurations.
- 4.2 Main Results: 65.84% on OSWorld with GPT-5 at 100 steps surpasses Agent S3 with GPT-5 by approximately 3%, while Workflow improves over the runner-up by 7%.The 50-step OSWorld score is 63.61%; the authors attribute long-horizon strength largely to the Reflection-Memory Agent.
- 4.2 Main Results: 63.5% on WindowsAgentArena with GPT-5 at 50 steps surpasses Agent S3 by 9.4% at 50 steps and 6.9% at 100 steps.The GPT-5-Mini variant also exceeds the 100-step Agent S3 GPT-5 baseline by 5.6%.
- 4.2 Main Results: 45.3% on WindowsAgentArena with Qwen3-VL-32B-Instruct improves 13.6% over its vanilla baseline, despite trailing the specialist UI-TARS-2.These results indicate adaptability across model scales and distinct operating-system characteristics.
- 4.3 Ablation Study: Multimodal Search gains 22.1% and 10.3% over w/o Search and Unimodal Search in the Daily domain, while RMA improves Workflow by approximately 20.0% and 20.7% over STM reflection and no reflection.The Workflow domain requires cross-application, long-horizon memory; removing RMA adds 3.3 steps per task, while naive Last-K reflection reduces token consumption by approximately 36.5% yet can be ineffective or detrimental.
- 4.4 Discussion: GPT-5-Mini trails GPT-5 by only 5% on average while reducing costs by approximately 80%, whereas grounding-model performance remains stable.The framework’s Coder delegation is cited as reducing dependence on the Grounder’s specific proficiency; visually subtle cues and planner-worker textual abstraction remain limitations.
5 Conclusion
OS-SYMPHONY uses an Orchestrator to coordinate specialized modules addressing long-horizon robustness and domain generalization. Experiments report state-of-the-art performance across diverse operating systems and show that open-source VLMs can solve complex tasks effectively.
- OS-SYMPHONY coordinates specialized modules through an Orchestrator to address long-horizon robustness and domain generalization.
- 65.84% on OSWorld establishes a state-of-the-art result for OS-SYMPHONY.
- The framework achieves state-of-the-art performance across diverse operating systems.
- OS-SYMPHONY demonstrates that complex problems can be effectively solved using open-source VLMs.
Limitations
The evaluation is confined to desktop ecosystems, leaving the framework’s adaptability to mobile platforms unverified. Full cross-platform universality therefore remains unexplored.
- The evaluation is strictly confined to desktop ecosystems.
- Adaptability to Android and iOS remains unverified because mobile interfaces require distinct action-space adaptations.
- Full cross-platform universality remains a subject for future exploration.
Ethical Considerations
The evaluations use isolated sandboxed environments to protect host systems and external networks. Real-world deployment raises security, privacy, and dual-use concerns requiring explicit safeguards and human control.
- Evaluations were conducted in isolated Docker containers and virtual machines to prevent damage to host systems or access to unauthorized networks.
- Real-world deployment requires renewed evaluation of system security and user privacy.
- Continuous screenshot processing can expose sensitive personally identifiable information displayed on screens.
- Strict permission controls and data sanitization are required for deployment, with users retaining authority over operational boundaries.
- The framework’s automation capabilities carry dual-use risks because they could enable malicious workflow auto-execution without safeguards.
A.1 Task Definition
CUA interaction is modeled as sequential decision-making under partial observability, where historical context supports belief-state construction and action selection. OS-SYMPHONY compresses this history through recent interactions, reflections, milestone memories, and auxiliary error detection, while retaining open design questions around action spaces.
- Task Definition: CUA interaction is modeled as a POMDP with environmental states, executable actions, observations, and probabilistic transitions.
- Task Definition: A CUA receives a task instruction, observes the environment, generates thoughts, and executes actions until completion or failure.
- Task Definition: Historical observations are aggregated into an implicit belief state because a single observation cannot fully depict the environment.
- Task Definition: Full multimodal histories create computational overhead, possible hallucinations from information overload, and redundancy in long-horizon GUI tasks.
- OS-SYMPHONY Details: The Orchestrator compresses context by combining a sliding window of recent interactions with reflections and retrieved procedural knowledge from the Reflection-Memory Agent.
- OS-SYMPHONY Details: The Reflection-Memory Agent uses milestone screenshots, step-transition histories, retrospective step summaries, and loop detection to support error identification.
- OS-SYMPHONY Details: UI-TARS-1.5-7B performs best among evaluated open-source grounding models in desktop environments.
- OS-SYMPHONY Details: Optimal action-space and action-parameter design remains an open question despite strong performance without specialized actions for every software feature.
B.1 More Results on OSWorld
On OSWorld, OS-SYMPHONY shows strong test-time scaling, sensitivity to visual-context length, and benefits from reasoning, instruction handling, coding, and hybrid GUI-API interaction.
- Impact of Thinking: Thinking variants generally perform better with OS-SYMPHONY, with the 8B Thinking model exceeding its Instruct counterpart by approximately 5%.The comparison spans Qwen3-VL 8B and 32B models with Instruct and Thinking variants.
- Instruction Rewriting: Instruction rewriting produced negligible performance differences, so the study retained the first image strategy for evaluation integrity.The authors note that changing test instructions could risk data leakage or task simplification, while retaining rewriting for real-world deployment.
- Impact of Coder: Removing the Coder degraded performance by approximately 6.2%, while GUI interactions and API-style coding together support the hybrid interaction paradigm.The Coder ablation disabled call_code_agent and constrained the model to GUI interactions; the hybrid-paradigm conclusion is supported by the reported action analysis.
- Visual Context Length: Performance depends on visual-context length: too few retained images omit interaction history, while too many saturate the context window.The study evaluates varying maximum image counts with Qwen3-VL-32B-Instruct and UI-TARS-1.5-7B.
B.2 More Results on WindowsAgentArena & MacOSArena
OS-SYMPHONY improves performance across WindowsAgentArena and MacOSArena, including challenging domain-specific and cross-platform tasks.
- WindowsAgentArena: 54.76% is achieved in the challenging Office domain of WindowsAgentArena.The benchmark contains Office, Web, Windows System, Code, Media, Utilities, and infeasible-task categories.
- WindowsAgentArena: 24.6% average relative improvement is reported for Qwen3-VL-32B-Instruct compared with its vanilla baseline across nearly every WindowsAgentArena domain.The result is presented as evidence that the framework addresses adaptation to Windows-specific characteristics.
B.3 Other Statistics
Additional OSWorld statistics examine task-step distributions, action usage, resource costs, reflection accuracy, and qualitative success and failure cases.
- Step Distribution: Successful and failed tasks are predominantly concentrated within the first 15 steps, despite standard limits of 50 or 100 steps.The authors identify premature termination and infeasible-task scoring as evaluation concerns.
- Action Utilization: Click accounts for 56.4% of actions, while click, hotkey, type, and scroll together comprise 83.9%.The call_search_agent action occurred in 0.6% of actions, and 85% of those searches yielded valuable tutorials.
- Token Usage: The Reflection-Memory Agent and Orchestrator average approximately 270k tokens per task, with about 96% attributed to context prompts.The General Grounder averages 33.2k tokens per task, while specialized modules are invoked less often.
- Message Protocol: Approximately 90% of GUI errors identified by the step-summary module were recognized by the Reflection-Memory Agent, and roughly 90 of 100 reviewed cases were actual GUI errors.The protocol marked about 25% of steps as various error types, while approximately 75% were judged normal operations.
- Qualitative Analysis: The qualitative analysis examines specific success and failure cases to demonstrate framework strengths and limitations.The referenced analysis includes OSWorld statistics and case studies of the framework’s behavior.
C.1 Correct Case
A successful OSWorld case illustrates complementary benefits from visual tutorial search and trajectory reflection during GUI execution.
- Correct Case: The Multimodal Searcher synthesized a relevant Thunderbird tutorial after a baseline lacked the domain knowledge to identify the correct button.Guided by the tutorial, OS-SYMPHONY clicked the target button at Step 4 and completed the task.
- Correct Case: The Reflection-Memory Agent verifies actions using history summaries and the current-step screenshot.The case study contrasts OS-SYMPHONY with Agent S3 on a slide-orientation task.
C.2 Error Case
OS-SYMPHONY’s failures arise from reflection errors and evaluation or instruction ambiguity, despite mechanisms intended to support robust execution. The examples also specify recovery, verification, and milestone procedures for handling tool and planning errors.
- Reflection errors: False alarms can arise when visual obstruction causes the Reflection-Memory Agent to issue erroneous negative feedback that misleads orchestration.A mouse cursor obscured a correct update, causing the RMA to report a false alarm.
- Reflection errors: Missing alarms occur when the VLM overlooks subtle execution errors, although ablations report that the RMA mitigates adverse collaborative side effects.The cited example involved a partially applied right-alignment task that the VLM failed to detect.
- Ambiguous instructions: Failures can reflect overly restrictive evaluation functions or ambiguous instructions rather than agent deficiencies.Examples include accepting only “STO” instead of the selected ARN airport and requiring “pure green” instead of the selected “bright green.”
- Tool recovery: The Search Agent retrieves complete tutorials when available and otherwise directs the system to use its own GUI and Code agents.Retrieved tutorials may include steps already completed, so the agent should not blindly restart from step one.
- Tool recovery: Reflection feedback guides recovery by prompting retries for GUI errors, Search Agent calls when stuck, and GUI continuation after Code Agent failures.The workflow requires reading reflection before every step and verifying Code Agent results through GUI actions.
- Milestone evaluation: Milestones are defined as successful completion of significant, self-contained sub-goals rather than ordinary incremental actions.This criterion supports selective historical context retention for planning.