PaperlayerView exact revision on arXiv

Source-linked AI summary

OSCAR: Operating System Control via State-Aware Reasoning and Re-Planning

Xiaoqiang Wang, Bang Liu

arXiv:2410.18963v1cs.AIcs.CL

TL;DR

OSCAR addresses the limited ability of language-model agents to generalize across applications and GUI environments. It combines code-centric OS control, GUI-grounded observation, state-aware recovery, and task-driven re-planning, and reports strong success across desktop and smartphone benchmarks. The paper positions these results as evidence of adaptable general-purpose OS task automation.

  • Problem

    Existing LLM and LMM agents have limited generalization across applications with different observation and action spaces, motivating standardized OS control for diverse workflows.

    Source: §1 · pp. 1–2

  • Method

    OSCAR combines executable Python OS control with GUI-grounded observations and a state-machine loop that uses execution and verification feedback for adaptive re-planning.

    Source: §1 · p. 2, §2.2 · p. 5, §5 · p. 10

  • Results

    OSCAR outperforms prior methods across GAIA, OSWorld, and AndroidWorld, including 28.7% average success on GAIA, 24.5% on OSWorld, and 61.6% on AndroidWorld.

    Source: §1 · p. 3, §3 · p. 7, §5 · p. 10

  • Takeaways & Limitations

    OSCAR offers a versatile and efficient approach for automating workflows across dynamic desktop and smartphone operating systems.

    Source: §5 · pp. 10–11

Abstract

from arXiv · show

Large language models (LLMs) and large multimodal models (LMMs) have shown great potential in automating complex tasks like web browsing and gaming. However, their ability to generalize across diverse applications remains limited, hindering broader utility. To address this challenge, we present OSCAR: Operating System Control via state-Aware reasoning and Re-planning. OSCAR is a generalist agent designed to autonomously navigate and interact with various desktop and mobile applications through standardized controls, such as mouse and keyboard inputs, while processing screen images to fulfill user commands. OSCAR translates human instructions into executable Python code, enabling precise control over graphical user interfaces (GUIs). To enhance stability and adaptability, OSCAR operates as a state machine, equipped with error-handling mechanisms and dynamic task re-planning, allowing it to efficiently adjust to real-time feedback and exceptions. We demonstrate OSCAR's effectiveness through extensive experiments on diverse benchmarks across desktop and mobile platforms, where it transforms complex workflows into simple natural language commands, significantly boosting user productivity. Our code will be open-source upon publication.

1 INTRODUCTION

OSCAR addresses limited cross-application generalization by combining standardized OS controls with screen-based interaction, executable code, state-aware recovery, and task-driven re-planning. It is evaluated across desktop and smartphone benchmarks, where it reports strong success rates and broad adaptability.

1 INTRODUCTION

  • OSCAR targets agents that struggle to generalize across applications with varied observation and action spaces, using standardized mouse and keyboard controls over screen outputs.The motivation includes workflows spanning multiple applications and diverse graphical or command-line interfaces.Source: §1 · p. 1
  • Dynamic environments let agents adapt to real-time feedback and complete tasks through multiple valid action trajectories, unlike fixed static workflows.The Report.docx example contrasts a predetermined sequence with alternative methods such as direct directory navigation and Ctrl+P.Source: §1 · p. 2, Figure 1 · p. 2
  • OSCAR combines unified OS controls, GUI grounding, and exploration-based re-planning to support precise interaction, screen interpretation, retries, exception handling, and adaptive strategy changes.These challenges define the approach’s intended robustness across computer applications.Source: §1 · p. 2
  • OSCAR generates executable Python code for semantically clear OS actions and uses visual and semantic grounding to improve GUI interaction across diverse tasks.Its code-centric control is paired with a state-machine loop that handles execution failures and system exceptions.Source: §1 · pp. 2–3
  • 28.7% average success on GAIA, 13.5% on Level 3 tasks, 24.5% on OSWorld, and 61.6% on AndroidWorld demonstrate reported effectiveness across desktop and smartphone environments.The paper states that OSCAR outperformed previous methods, with Level 3 GAIA performance nearly doubling the prior state of the art.Source: §1 · p. 3

2 METHODOLOGY

OSCAR uses a state machine that repeatedly observes, plans, executes, verifies, and recovers from failures while grounding GUI elements and generating executable controls. Task-driven re-planning refines affected subtasks rather than restarting entire workflows, supporting efficient interaction across desktop and mobile operating systems.

2.1 FORMULATION OF STATE TRANSITIONS

  • OSCAR’s state machine processes requests through observation, planning, execution, verification, success or failure handling, reset, and error recovery.Transitions are triggered by user requests, planning completion, verification outcomes, execution failures, and OS exceptions.Source: §2 · p. 3, Figure 2 · p. 4, §2.1 · p. 4
  • Successful execution returns OSCAR to observation and planning with updated screenshots, while invalid code sends interpreter feedback back into planning.Verification can produce success, another planning cycle, or failure after the maximum allowed attempts.Source: §2.1 · pp. 3–4
  • Continuous feedback loops and modular transitions support error recovery, fine-grained task-driven re-planning, and generalization across desktop and smartphone operating systems.The architecture is contrasted with linear action sequences and re-planning from scratch.Source: §2.1 · pp. 3–4

2.2 GUI-GROUNDED OBSERVATION

  • OSCAR’s dual-grounding observation combines screenshot context, SoM visual prompts derived from the Accessibility tree, and semantic labels for precise element references.The Accessibility tree supplies UI-element locations, properties, and states, while labels can identify elements by ID and coordinates.Source: §2.2 · p. 5, Figure 3 · p. 5
  • The Accessibility-tree approach is more precise and robust than detection plus OCR on complex screens, where OCR often fails.The method is designed for interfaces densely populated with small text, icons, widgets, and other interaction elements.Source: §2.2 · p. 5

2.3 TASK-DRIVEN RE-PLANNING

  • Task-driven re-planning uses context memory and OS feedback in two planning levels: decomposing instructions into SOP-based tasks, then generating actions step by step.Planning and execution are interleaved within the state machine to avoid the inefficiency of fully exploring before deployment.Source: §2.3 · p. 6, Figure 4 · p. 6
  • When verification fails, OSCAR can re-plan only affected tasks rather than the entire workflow or current action, reducing error propagation across multi-application workflows.The paper illustrates this with workflows moving among Word, Photos, and PowerPoint.Source: §2.3 · p. 6

2.4 CODE-BASED ACTION

  • OSCAR operationalizes actions as executable Python code, using grounded element IDs or coordinates with PyAutoGUI for mouse and keyboard interactions.Supported controls include movement, clicks, scrolling, key presses, and shortcuts.Source: §2.4 · pp. 6–7

3 EXPERIMENTS

OSCAR is evaluated against generalist agents on desktop and mobile workflow benchmarks, with additional ablations and analyses of planning efficiency. Results show broad gains, especially on complex tasks, while task-driven re-planning and GUI grounding account for important performance and efficiency differences.

3 EXPERIMENTS

  • OSCAR achieves the best GAIA performance across all workflow levels, reaching 13.5% on Level 3 versus MMAC’s 6.1%.The comparison uses exact match on GAIA; Table 1 reports the benchmark results.Source: §3 · p. 7, Table 1 · p. 7
  • OSCAR surpasses competing agents across dynamic desktop and mobile environments, including 12.9% on multi-application OSWorld tasks and stronger average AndroidWorld performance than AppAgent.The AndroidWorld advantage is especially pronounced on medium and hard subsets.Source: §3 · p. 7, Table 2 · p. 7, Table 3 · p. 7

3.1 ABLATION ANALYSIS

  • GUI grounding and task-driven re-planning are necessary for full OSWorld performance: raw screenshots reach 70%, while direct prompts without fine-grained planning reach 80% of OSCAR’s performance.Detection+OCR is weaker than A11Y-tree grounding on professional-tool tasks and also adds processing time.Source: §3.1 · p. 8, Figure 5 · p. 8
  • ReAct and Chain-of-Action are competitive in daily applications and office software but struggle on professional and multi-application scenarios without real-time feedback and efficient re-planning.These comparisons motivate OSCAR’s task-driven strategy for dynamic OS environments.Source: §3.1 · p. 8

3.2 IN-DEPTH ANALYSIS

  • OSCAR avoids false-completion and invalid-action errors through execution and verification feedback, while its remaining failures involve less redundant re-planning than FRIDAY’s.FRIDAY re-plans the entire workflow in 52.8% of failures, often repeating previously unsuccessful trajectories.Source: §3.2 · pp. 8–9, Table 4 · p. 9
  • Over 80% of OSCAR’s successful cases require fewer than three re-planning attempts, whereas more than 50% of FRIDAY’s successful cases require three to four.The experiments allow at most four re-planning attempts.Source: §3.2 · p. 9, Figure 6 · p. 9
  • OSCAR’s re-planning uses smaller, more efficient steps, with stored task lists and action histories guiding later trajectories after verification failures.The proxy action matching score worsens for FRIDAY as re-planning attempts increase.Source: §3.2 · pp. 8–9
  • A qualitative example shows OSCAR handling a request that moves across operating-system, office, and daily applications.The example illustrates multi-application planning in a dynamic environment.Source: §3.2 · p. 9

4 RELATED WORKS

Prior GUI agents span mobile, desktop, and broader OS-control settings, but many remain domain-specific, static, or inefficient when replanning. OSCAR addresses these gaps with state-aware, task-driven replanning for real-time dynamic OS feedback.

4 RELATED WORKS

  • Most existing agents are limited to specific domains, static environments, or application-specific APIs, restricting broader GUI automation.These limitations include sandboxed games and office workflows, predefined OS environments, and insufficient grounding in real-time executable environments.Source: §4 · p. 9
  • Some GUI agents re-plan from scratch after verification failures, lacking fine-grained strategies that would improve efficiency in real-world scenarios.Source: §4 · pp. 9–10
  • Unlike OS-integration work focused on kernels, resource management, or traditional OS functions, OSCAR acts as a generalist GUI co-pilot for user workflows.Source: §4 · p. 10

5 CONCLUSION

OSCAR is presented as a code-centric, state-aware agent for dynamic desktop and smartphone OS tasks. Across GAIA, OSWorld, and AndroidWorld, it improves task success, especially on complex multi-step workflows, with ablations supporting its core components.

5 CONCLUSION

  • OSCAR uses code-centric control, task-driven replanning, and GUI-grounded observations to navigate dynamic desktop and smartphone OS environments.Source: §5 · p. 10
  • Ablations identify GUI-grounded observation and task-driven replanning as important for improving performance and reducing redundant replanning.Source: §5 · pp. 10–11
  • OSCAR achieves reported gains across GAIA, OSWorld, and AndroidWorld, particularly for complex, multi-step tasks, supporting its use for workflow automation.Source: §5 · pp. 10–11

A OVERVIEW

The appendices provide supporting implementation, baseline, benchmark, and qualitative materials. They cover GUI understanding, static GUI navigation, and additional examples beyond the main text.

A OVERVIEW

  • Appendix B contains implementation details, while Appendix C documents the baselines used for comparison.Source: A OVERVIEW · p. 17
  • Appendix D reports experiments on GUI understanding benchmarks.Source: A OVERVIEW · p. 17
  • Appendix E reports experiments on static GUI navigation benchmarks.Source: A OVERVIEW · p. 17
  • Appendix F presents additional qualitative results.Source: A OVERVIEW · p. 17

B IMPLEMENTATIONS

OSCAR's implementation combines observation extraction, standardized action spaces, controlled model settings, and repeated benchmark runs. The setup adapts these components to dynamic OS environments and other benchmark formats.

B IMPLEMENTATIONS

  • In dynamic OS environments, OSCAR extracts Set-of-Mark prompts from Accessibility trees; screenshot-only benchmarks use YOLO-v8 and Google OCR instead.Source: B IMPLEMENTATIONS · p. 17
  • The action space uses desktop and smartphone controls for dynamic OS benchmarks, while other benchmarks require free-form answers or structured actions.Source: B IMPLEMENTATIONS · p. 17
  • OSCAR and baselines use GPT-4o with eight in-context demonstrations, except GAIA results use GPT-4-turbo to match the baseline setting.Source: B IMPLEMENTATIONS · p. 17
  • Experiments average results across four runs, allow up to four attempts per run, and use a Detection+OCR pipeline that can run on ordinary Windows 11 machines with WSL.Source: B IMPLEMENTATIONS · p. 17

C BASELINE DETAILS

OSCAR is compared against four baseline categories defined by model specialization and target GUI environment. The comparison highlights OSCAR’s ability to navigate both desktop and smartphone OS environments with real-time feedback.

C BASELINE DETAILS

  • Four baseline categories span general-purpose versus GUI-specialized LMMs and desktop versus smartphone OS targets.This two-dimensional organization supports a comprehensive comparison across model type and GUI scenario.Source: C BASELINE DETAILS · p. 17
  • OSCAR uniquely navigates both desktop and smartphone OS environments while responding to real-time OS feedback.Source: C BASELINE DETAILS · p. 17, Table 6 · p. 18

D GUI UNDERSTANDING BENCHMARK

The GUI understanding evaluation tests OSCAR across six desktop and smartphone GUI domains using screenshots and language or multiple-choice outputs. OSCAR achieves the best performance across five domains and reaches state-of-the-art website performance when provided HTML input.

D GUI UNDERSTANDING BENCHMARK

  • The benchmark covers six desktop and smartphone GUI scenarios evaluated through captions, layout descriptions, GUI elements, screenshot relations, and multiple-choice accuracy.Automatic evaluation uses BERTScore, LLM-as-a-Judge, or accuracy depending on the task.Source: D GUI UNDERSTANDING BENCHMARK · p. 18
  • OSCAR achieves the best GUI understanding performance across five GUI domains, while website leadership requires HTML input for parity with the state of the art.The reported result is attributed to OSCAR’s GUI-grounded observation.Source: D GUI UNDERSTANDING BENCHMARK · p. 18, Table 7 · p. 19
  • Fine-tuning on domain-specific data can reduce general-domain performance, with the Web Agent scoring 83 on iOS GUI versus 93 on website GUI.Average differences among agents remain marginal, indicating strong single-step GUI understanding from the GPT-4o base model used in the experiments.Source: D GUI UNDERSTANDING BENCHMARK · pp. 18–19

E STATIC GUI NAVIGATION BENCHMARK

The static GUI navigation evaluation measures next-action prediction on desktop and smartphone benchmarks using historical actions and screen states. OSCAR without re-planning performs best, especially on cross-website and cross-domain tasks, while single-step GUI fine-tuning transfers poorly to multi-step navigation.

E STATIC GUI NAVIGATION BENCHMARK

  • Mind2Web and AITW evaluate next-action prediction with element accuracy, operation F1, step success rate, and screen-wise action matching.Correctness depends on matching the selected element and operation, with defined tolerances for click and scroll actions.Source: E STATIC GUI NAVIGATION BENCHMARK · p. 19
  • OSCAR without re-planning consistently achieves the best multi-step navigation performance, particularly on cross-website and cross-domain data.The comparison includes competitive baselines such as UFO and AUTO-GUI.Source: E STATIC GUI NAVIGATION BENCHMARK · p. 19
  • Single-step GUI fine-tuning contributes little to multi-step decision-making, as CogAgent performs competitively in understanding but poorly in navigation.The paper suggests hallucinated actions may increase when intermediate feedback is unavailable in static environments.Source: E STATIC GUI NAVIGATION BENCHMARK · p. 19

F QUALITATIVE RESULTS

The qualitative materials show OSCAR handling daily applications and professional tools, alongside system-prompt details and quantitative navigation-result tables. The examples include creating and saving a text file and installing a VS Code extension.

F QUALITATIVE RESULTS

Loading 2410.18963v1…