Source-linked AI summary
JarvisGUI: Towards Cross-Device GUI Agents with Dynamic Task Composition
Zixiang Chen, Yuheng Lu, Zihao Cheng, Zeming Liu, Jizeng Bai, Ziye Huang, Zhiyin Lin, Zihan Li, Yuhang Guo, Yunhong Wang, Haifeng Wang
TL;DR
Existing GUI benchmarks largely omit cross-device workflows, leaving coordination across heterogeneous platforms insufficiently evaluated. JarvisGUI introduces a typed, dynamically composable benchmark spanning Android, Windows, and Ubuntu, and evaluations show persistent weaknesses in state transfer, cross-platform reasoning, and long-horizon dependency management. The benchmark also depends on specific virtualization resources and currently lacks accessibility-focused scenarios.
Problem
Existing GUI benchmarks predominantly evaluate single-device tasks, leaving cross-device capabilities and realistic multi-platform workflow performance insufficiently assessed.
Method
JarvisGUI uses slot-typed task modeling to compose and evaluate cross-device workflows across Android, Windows, and Ubuntu.
Results
JarvisGUI reveals that current agents remain weak in state transfer, cross-platform reasoning, and long-horizon dependency management.
Takeaways & Limitations
Cross-device coordination remains a major bottleneck for reliable real-world GUI agents and is obscured by existing benchmarks.
Takeaways & Limitations
Running JarvisGUI depends on Docker and KVM, and its scenarios currently lack accessibility-focused cases for elderly and disabled users.
Abstract
from arXiv · showhide
Real-world GUI usage frequently involves workflows that span multiple devices and platforms, requiring the transfer of intermediate results, maintenance of shared state, and coordination across heterogeneous environments. However, existing GUI benchmarks overwhelmingly evaluate agents on single-device, statically defined tasks, thus leaving such cross-device capabilities largely unexamined, resulting in an overly optimistic assessment of agents' readiness for real-world usage. We introduce JarvisGUI, a dynamic benchmark that evaluates GUI agents on cross-device workflows requiring coordinated interaction across heterogeneous platforms, including Android, Windows, and Ubuntu. Specifically, JarvisGUI formulates GUI tasks as input-output transformations under a lightweight type system, which allows us to automatically compose multi-step, cross-device workflows and dynamically evaluate agent performance within a unified framework. By evaluating agents in virtual environments spanning multiple operating systems, JarvisGUI reveals that state-of-the-art open-source GUI agents struggle with the state-transfer awareness, cross-platform contextual reasoning, and long-horizon dependency management required for real-world workflows, exposing a critical capability gap invisible to existing benchmarks.
1 Introduction
JarvisGUI addresses the gap between single-device GUI benchmarks and real-world workflows spanning heterogeneous platforms. It introduces typed task composition and a benchmark for evaluating cross-device coordination, where open-source agents show persistent weaknesses in context preservation and inter-device reasoning.
- Existing GUI benchmarks predominantly confine tasks to one operating system, neglecting the cross-device interactions common in real-world digital workflows.
- JarvisGUI evaluates realistic multi-device scenarios across Android, Windows, and Ubuntu, requiring information transfer, file sharing, and task-context maintenance.
- Evaluations reveal substantial disparities between agents’ single-device proficiency and cross-device competence, especially in context preservation and inter-device reasoning.
- JarvisGUI is presented as the first systematic study of GUI-agent tasks requiring information, file, and execution-context transfer across heterogeneous devices.
- Its slot-typing-based task framework models transferable entities and inter-task dependencies for reliable, extensible, and verifiable composition of complex cross-device tasks.
2 Related Work
GUI-agent research has progressed toward end-to-end visual interaction and stronger grounding, but most agents and benchmarks still assume single-device observations and execution. Existing cross-device coverage remains too limited to assess realistic end-to-end workflows.
- GUI agents evolved from structured visual representations toward end-to-end action prediction from raw screenshots.
- Recent agents improve language-to-interface grounding through larger training data and reinforcement learning, including static and online methods.
- Most current GUI-agent approaches assume one device’s screenshot per observation, leaving cross-device interaction largely unaddressed.
- Benchmarking has advanced from static grounding tasks to multi-step interactive settings and virtual environments with diverse execution paths.
- CRAB contains only 18 cross-device tasks and does not model realistic file and data transfer, limiting end-to-end workflow evaluation.
3 Environment
JarvisGUI provides a reproducible, controllable environment for dynamically composing, executing, and evaluating typed GUI tasks across heterogeneous devices. Its layered architecture standardizes platform control, coordinates model interaction, and automates final-state scoring.
- Task Definition and Evaluation: JarvisGUI models each task as a typed, parameterized system executed in heterogeneous environments and evaluated by inspecting the final environment state.
- Task Definition and Evaluation: A task contains input and output slots, a natural-language template, evaluation functions, and a target platform, with dynamic components instantiated from concrete inputs.
- Task Definition and Evaluation: Agent execution is represented as iterative state transitions in which actions transform environment observations, including observations from multiple platforms, until final-state evaluation.
- Task Definition and Evaluation: A deterministic evaluator verifies completion by programmatically inspecting observable final-state artifacts such as file contents or DOM elements.
- System Architecture: The four-layer architecture combines Docker-based infrastructure, platform-agnostic control, Planner-Grounder model interaction, and automated evaluation execution.
- Environment Configuration: The evaluation workflow provisions environments, initializes task prerequisites, runs an observation-and-action loop, retrieves state data, and automatically scores completion.
4 JarvisGUI
JarvisGUI builds a dynamic benchmark by composing typed GUI task graphs across heterogeneous platforms and filtering them for realism, coherence, and evaluability. Its resulting benchmark contains diverse cross-platform workflows and quality-controlled task statistics.
- 4 JarvisGUI: JarvisGUI combines a type system, dynamic task generation, quality control, and benchmark statistics into a pipeline for cross-device GUI evaluation.The pipeline covers compositional task construction, data collection, quality control, and dataset characterization.
- Task Composition: Tasks are automatically composed with subtype checking, cross-device file-transfer assistants, and rewritten merged instructions for coherent multi-step workflows.The composition procedure uses typed task instances and auxiliary transfer tasks when files move between devices.
- Task Graph: Each workflow is modeled as a directed graph whose task outputs feed compatible downstream inputs and dynamically instantiate descriptions and evaluators.Runtime value propagation conditions downstream tasks on concrete upstream results rather than static placeholders.
- Quality Control: The quality-control process samples 2,000 tasks for LLM judgments of realism, coherence, and evaluability, retaining only tasks scoring 5 on every dimension.Atomic tasks also undergo interaction-step constraints, evaluator checks before and after execution, and manual inspection.
- Quality Control: Human inspection found 92% of 50 sampled composed tasks fully correct, while LLM and human filtering decisions achieved approximately 0.92 Cohen’s Kappa.Four sampled tasks had imprecise descriptions that might introduce ambiguity.
- Data Statistics: JarvisGUI contains 150 compositional tasks across three categories and 442 platform-specific subtasks, including 187 Ubuntu, 138 Windows, and 52 Android subtasks.The remaining 65 subtasks are file-transfer assistant tasks supporting cross-platform coordination.
5 Experiments
Experiments evaluate GUI agents across atomic and multi-task workflows using TSR and SSR. Results show substantial weaknesses in cross-device coordination, dependency management, platform robustness, and long-horizon execution.
- Evaluation Setting: The evaluation covers atomic tasks on Android, Windows, and Ubuntu, plus single-device dependent, multi-device independent, and multi-device dependent workflows.TSR measures terminal-state success across task types, while SSR captures intermediate-step completion for compositional tasks.
- Main Results: Cross-device dependent tasks produce the largest performance drop because agents struggle with target-device state inference and fragile sequential critical paths.A single failure in file location or cloud transfer can invalidate all subsequent steps.
- Main Results: Multi-device independent tasks also underperform because simultaneous screens increase visual grounding demands and implicit platform references are often routed incorrectly.Examples include mapping “C drive” to Windows and “back to the computer” to the previously used system.
- Main Results: Single-device dependent tasks perform below independent and atomic tasks, showing that preserving intermediate results and sequencing actions remains difficult.The reported challenges include maintaining clipboard content or temporary file paths across logically dependent steps.
- Main Results: Atomic performance is consistently better on Ubuntu and Android than Windows, which the authors attribute to differences in pre-training data distributions.The paper links stronger performance on Ubuntu and Android to command-line or mobile interaction data, while diverse Windows GUI data is relatively scarce.
- Error Analysis: Task success declines sharply as subtask count increases, reaching near zero for workflows requiring four or more subtasks across evaluated models.Unexecuted dependency steps are treated as completed, causing cascading errors in later actions.
6 Conclusion
JarvisGUI is a dynamic benchmark for cross-device GUI workflows across Android, Windows, and Ubuntu. Its evaluations reveal persistent weaknesses in state transfer, cross-platform reasoning, and long-horizon dependency management.
- Conclusion: JarvisGUI evaluates diverse cross-device workflows by composing atomic tasks and inspecting final environment states.The benchmark spans Android, Windows, and Ubuntu.
- Conclusion: Current agents remain weak in state transfer, cross-platform reasoning, and long-horizon dependency management.These weaknesses are identified as key challenges for reliable real-world GUI agents.
Limitations
JarvisGUI’s reproducibility and coverage are bounded by infrastructure requirements and practical constraints on scenario design.
- Infrastructure: Running parallel virtual machines depends on Docker and the KVM module, which may hinder reproducibility in restricted or cloud-based environments.The limitation specifically concerns environments lacking KVM support.
- Coverage: Scenario coverage excludes accessibility-focused workflows and is constrained by legal compliance, evaluation costs, and hardware simulation limits.The authors identify these constraints as practical trade-offs affecting benchmark expansion.
Ethical Statement
The system uses strict cross-device planner prompts, standardized action interfaces, and supervised data filtering to support reproducible GUI-agent evaluation while acknowledging privacy and dual-use concerns.
- Ethics and safety: The study anonymizes tasks, supervises manual annotations, attributes open-source components, and plans dataset release under data-protection agreements.It also acknowledges potential misuse involving file theft or unauthorized data leakage.
- Planner protocol: The planner receives screenshots from Android, Windows, and Ubuntu and outputs one next step by selecting a platform, action, and tool invocation.The prompt prohibits coordinate inference and restricts the planner to structured JSON output.
- Planner–grounder separation: Coordinate-controlled actions are delegated to the grounding agent, while the planner supplies only the action type and target description.This separation applies across the supported desktop and Android action spaces.
- Data filtering: LLM-based judging filters randomly generated tasks for semantic coherence and evaluability beyond structural type compatibility.The filtering process retains tasks receiving full scores across the judge’s evaluation aspects.
- Environment control: The unified action space maps abstract actions to platform-specific primitives and includes controls such as WAIT, FAIL, and DONE.Desktop actions use PyAutoGUI, while Android actions use ADB commands.
A.7 Experiment Details
Additional planner experiments test whether the cross-device bottleneck depends on the backbone model. Alternative models provide mixed results, but cross-device degradation persists beyond the original planner.
- Alternative planners: Llama 4 Maverick frequently violated the required action format and achieved near-zero scores across all evaluated tasks.Its failures prevented consistent generation of executable actions.
- Alternative planners: Kimi K2.6 improved over Qwen3-VL-Plus on atomic and compositional tasks, but cross-device performance remained limited.The Multi-Overall score increased from 8.0 to 11.3 without eliminating cross-device compositional degradation.
- Generalization: The results indicate that cross-device bottlenecks generalize across planner backbones rather than being specific to Qwen3-VL-Plus.The conclusion follows from persistent degradation under the alternative-model evaluation.
- Statistical analysis: The evaluation reports successful-task counts, percentages, and 95% bootstrap confidence intervals for model performance.These statistics are provided in Table 9.
C Future Extension
JarvisGUI provides a substantial benchmark base while identifying limits in scenario coverage, trajectory length, transfer simulation, and user diversity that future versions aim to address.
- Current scale: The benchmark contains 150 composite workflows, 442 platform-specific subtasks, and 65 file-transfer auxiliary tasks.Workflows average approximately three subtasks and span many applications and domains.
- Current scale: Future extensions can automatically recompose atomic input–output tasks to expand workflow diversity.The structured task representation supports continued sampling of cross-device workflows.
- System coverage: The current setup excludes macOS and iOS, caps trajectories at 50 steps, and uses a private AList-based network drive instead of physical USB or Bluetooth transfer.These choices reflect reproducibility, licensing, stability, and inference-cost trade-offs.
- User coverage: The present benchmark primarily covers generalized mainstream workflows, leaving accessibility-oriented scenarios for future expansion.Proposed examples include cross-device text-to-speech synchronization and remote health-monitoring setups.
- Failure analysis: Failure analysis identifies single-device errors, cross-device transfer failures, and context loss as recurring failure modes across evaluated models.The analysis focuses on observable trajectories because identical errors can have multiple underlying causes.
D.2.1 Atomic Tasks
Atomic-task cases show that agents can complete some localized GUI operations but may misjudge completion or fail to execute required follow-up actions.
- Successful case: A successful Windows case opens a text file, replaces the specified content, saves it, closes Notepad, and verifies the result.The agent correctly recognizes the file text and performs the requested replacement.
- Completion judgment: A LibreOffice failure applies bullets only to selected lines, yet the planner incorrectly judges the task complete.The required formatting should have covered all lines.
- Action selection: Another failure captures a photo but does not rename it because the planner cannot determine the appropriate next action.The missing rename step prevents completion of the full task.
- Action selection: A separate case fails because the planner cannot predict the atomic action needed to complete file renaming.The failure prevents correct execution of the renaming operation.
D.2.2 Compositional Tasks
The compositional-task examples show successful coordination within one device and across multiple devices, alongside failures involving file transfer and action planning.
- Successful Cases: The system successfully increased an image’s brightness with ImageMagick and set the processed image as the Ubuntu wallpaper via gsettings.This dependency-free workflow was executed entirely on an Ubuntu desktop.
- Successful Cases: The system coordinated a cross-device workflow by changing Ubuntu to Dark mode while modifying target.txt on Windows.The example demonstrates simultaneous task execution across different device contexts.
- Failure Cases: One failure occurred when the agent opened Chrome but did nothing further despite reasoning that it had transferred the file to Android and attempted to open it.The failure is associated with a compositional task requiring file transfer across devices.
- Failure Cases: The figures also document atomic-task failures caused by identifying the wrong next action or failing to predict the atomic action.These cases distinguish planning failures from the successful compositional examples.