PaperlayerView exact revision on arXiv

Source-linked AI summary

UI-Copilot: Advancing Long-Horizon GUI Automation via Tool-Integrated Policy Optimization

Zhengxi Lu, Fei Tang, Guangyi Liu, Kaitao Song, Xu Tan, Jin Ma, Wenqi Zhang, Weiming Lu, Jun Xiao, Yueting Zhuang, Yongliang Shen

arXiv:2604.13822v1cs.LG

TL;DR

Long-horizon GUI agents face memory degradation, progress confusion, and math hallucination when tasks exceed their intrinsic capabilities. UI-Copilot decouples memory from execution and selectively invokes a Retriever or Calculator, with TIPO separately optimizing tool selection and task execution. UI-Copilot-7B achieves state-of-the-art MemGUI-Bench performance and improves generalization across challenging GUI benchmarks.

  • Problem

    Long-horizon GUI scenarios remain challenging because agents suffer from memory degradation, progress confusion, and math hallucination.

  • Method

    UI-Copilot decouples persistent observations from transient execution context and selectively invokes a lightweight copilot as Retriever or Calculator, while TIPO separately trains tool selection and task execution.

  • Results

    UI-Copilot-7B achieves state-of-the-art performance on MemGUI-Bench and consistently strong performance across challenging GUI benchmarks.

  • Takeaways & Limitations

    The framework supports long-horizon, memory-intensive GUI tasks and generalizes across several challenging GUI benchmarks.

  • Takeaways & Limitations

    The current tool set is limited to Calculator and Retriever, while real-world GUI scenarios may require broader tools such as web search and visual cropping.

Abstract

from arXiv · show

MLLM-based GUI agents have demonstrated strong capabilities in complex user interface interaction tasks. However, long-horizon scenarios remain challenging, as these agents are burdened with tasks beyond their intrinsic capabilities, suffering from memory degradation, progress confusion, and math hallucination. To address these challenges, we present UI-Copilot, a collaborative framework where the GUI agent focuses on task execution while a lightweight copilot provides on-demand assistance for memory retrieval and numerical computation. We introduce memory decoupling to separate persistent observations from transient execution context, and train the policy agent to selectively invoke the copilot as Retriever or Calculator based on task demands. To enable effective tool invocation learning, we propose Tool-Integrated Policy Optimization (TIPO), which separately optimizes tool selection through single-turn prediction and task execution through on-policy multi-turn rollouts. Experimental results show that UI-Copilot-7B achieves state-of-the-art performance on challenging MemGUI-Bench, outperforming strong 7B-scale GUI agents such as GUI-Owl-7B and UI-TARS-1.5-7B. Moreover, UI-Copilot-7B delivers a 17.1% absolute improvement on AndroidWorld over the base Qwen model, highlighting UI-Copilot's strong generalization to real-world GUI tasks.

1 Introduction

Long-horizon GUI tasks expose memory, progress-tracking, and numerical-reasoning failures in existing agents. UI-Copilot addresses these issues by decoupling memory and computation, selectively invoking tools, and separately training tool use and task execution.

  • Challenges: Existing 7B GUI models suffer an average performance drop of 90.90% on MemGUI-Bench.
  • Challenges: Long-horizon failures involve memory degradation, progress confusion, and math hallucination.Overloaded context can cause information loss or misrecall, obscure task state, and propagate incorrect intermediate calculations.
  • UI-Copilot: UI-Copilot lets the GUI agent focus on task execution while a lightweight copilot provides on-demand memory retrieval and numerical computation.The framework decouples persistent observations from transient execution context and invokes the copilot as Retriever or Calculator when needed.
  • TIPO: TIPO separately trains tool invocation and action generation for more effective tool-integrated learning.Tool selection uses single-turn supervision, while action generation learns through multi-turn rollouts conditioned on self-generated histories.
  • Results: UI-Copilot-7B achieves state-of-the-art performance on MemGUI-Bench and attains 39.1% accuracy on AndroidWorld.The benchmark results are reported for tasks involving persistent memory and generalization to multi-turn GUI performance.

2 Related Work

Prior GUI-automation research has advanced multimodal modeling and reinforcement learning, but external tool calling during GUI-agent training remains unexplored. Existing memory-oriented approaches include multi-agent workflows and retrieval augmentation, with scalability and deployment-cost limitations.

  • GUI-agent training: Recent work applies Group Relative Policy Optimization to GUI automation.
  • GUI-agent training: External tool calling for GUI-agent training remains unexplored in the cited prior work.
  • Memory enhancement: Multi-agent workflows and few-shot retrieval-augmented generation aim to improve GUI-agent memory without model fine-tuning.
  • Memory enhancement: These memory-enhancement approaches often suffer from limited scalability and high deployment costs.

3 Method

UI-Copilot formulates GUI automation as sequential action generation and combines memory decoupling with selective copilot assistance. TIPO separately trains tool selection and multi-turn action execution to align learning with deployment.

  • Problem Definition: GUI automation is modeled as sequential decision-making over screenshots, action histories, and coordinate-, text-, and system-based operations.The environment transitions between screen states after each action until task completion or failure.
  • UI-Copilot: The rollout paradigm lets a policy agent invoke a copilot as Retriever, Calculator, or none during multi-turn execution.The Retriever accesses stored knowledge and summaries, while the Calculator generates executable Python code for numerical tasks.
  • UI-Copilot: Memory decoupling keeps concise progress summaries in dialogue history while storing detailed reasoning and observations locally for on-demand retrieval or computation.This separates progress tracking from detailed reasoning to reduce context overload while preserving information needed later.
  • Dataset Curation: The training pipeline constructs expert and tool-call datasets, merges them into D0, and splits the result for SFT and RL training.The GUI agent is trained on the pseudo-labeled D0 without involving the copilot model during this training stage.
  • Tool-Integrated Policy Optimization: Training begins with SFT cold-start initialization, then applies distinct tool-call and action-prediction learning objectives with reward-based optimization.TIPO enforces minimum advantage variance during dynamic sampling, with the threshold η set to 0.3.
  • Tool-Integrated Policy Optimization: TIPO separates tool-calling rollouts and learning from action-generation rollouts and learning, using single-turn tool prediction and self-generated multi-turn histories.Tool-call learning uses off-policy histories, whereas action prediction follows on-policy histories to better match multi-turn evaluation.

4 Experiment

Experiments evaluate UI-Copilot across memory-intensive, dynamic, and tool-calling GUI benchmarks, showing strong performance and improvements from its training and inference design. Ablations indicate that selective tools, summarized histories, and decoupled on-policy training improve execution efficiency and task performance.

  • Main Results: UI-Copilot-7B achieves 16.4% pass@1 and 20.3% pass@3 accuracy on MemGUI-Bench, outperforming strong 7B baselines reaching up to 10.2%.It also solves some Hard tasks requiring over 40 steps or 4 Apps.
  • Main Results: 39.1% accuracy on AndroidWorld and 61.2% on MiniWob++ place UI-Copilot-7B among advancing 7B models on dynamic benchmarks.The results are reported as comparable to closed-source GPT-4o.
  • Training Dynamics: Training improves accuracy, reduces tool-call frequency, and decreases average execution steps, with MemGUI-Bench using nearly 13% tool calls versus approximately 6% on AndroidWorld.Accuracy converges after approximately 40 training steps, while longer training is needed to stabilize tool invocation on more complex tasks.
  • Inference Ablations: The full Calculator-and-Retriever tool set reaches 51.5% average accuracy with 16.6 average steps, while Multi-turn Summary outperforms alternative rollout histories.The full collaborative set remains competitive with a workflow invoking the copilot at every step.
  • Copilot Ablations: Qwen3-4B is the strongest copilot among the tested models, outperforming smaller Qwen variants and Qwen2.5VL-7B.The analysis links this result to context understanding and summarization capability.
  • TIPO Ablations: SFT provides the cold start, tool-call RL and action-prediction RL support distinct capabilities, and on-policy histories outperform expert-collected histories for multi-turn evaluation.A 600:2000 data ratio balances tool-call and action-generation learning, with diminishing returns from additional samples.

5 Conclusion

The paper concludes that UI-Copilot combines external-tool assistance with TIPO to address complex, long-horizon GUI tasks. UI-Copilot-7B shows consistently strong performance across challenging GUI benchmarks.

  • UI-Copilot incorporates the Copilot Model as an external tool and proposes TIPO for tool-integrated learning.
  • UI-Copilot-7B achieves consistently strong performance across several challenging GUI benchmarks.

Limitations

The paper identifies a scope boundary in its current tool set: UI-Copilot supports Calculator and Retriever, while broader integrations remain future work.

  • The current tool set is limited to Calculator and Retriever, excluding broader tools such as web search and visual cropping.Extending the framework to more diverse tool integrations is identified as future work.

C.2 Multi-turn Action Prediction Learning

This section explains why multi-turn, on-policy action prediction better matches deployment than single-turn training with off-policy histories. The approach reduces training–inference mismatch and supports more consistent optimization toward deployment-time performance.

  • Setup: The training objective evaluates task rewards over action and tool trajectories generated under the rollout policy.The setup distinguishes the training policy from the rollout policy and defines the deployment objective over instruction-conditioned trajectories.
  • Single-turn Training Mismatch: Single-turn training conditions each step on off-policy histories, creating biased gradients and a deployment gap.The deployment objective is defined over rollout trajectories, whereas single-turn training uses expert or otherwise off-policy histories.
  • Multi-turn Training Alignment: Multi-turn training conditions action prediction on self-generated histories from the rollout policy.This aligns training histories with the histories encountered during deployment.
  • Multi-turn Training Alignment: On-policy multi-turn rollouts reduce train–inference mismatch by approximating the deployment objective more closely.The resulting optimization is described as more consistent and stable toward deployment-time performance.
  • Policy Gradient: With a fixed tool, concentrated action distributions make the action-level advantage small, so the action-related gradient contributes negligibly.For no-tool actions, the policy reduces to action generation and rewards depend solely on task progress.

D.1 Training Dataset DRL

The DRL training dataset combines memory-intensive, calculation-required, and no-tool GUI tasks, with varied applications and interaction demands. Its action distributions are dominated by clicks, while most trajectories contain 5–9 interaction steps.

  • Action Composition: CLICK is the most prevalent action type, followed by TERMINATE and SWIPE across 2,000 training trajectories.TERMINATE consistently serves as the final action in successfully completed trajectories.
  • Action Composition: Most DRL trajectories comprise between 5 and 9 interaction steps.The trajectory-length distribution is reported alongside action-type distributions for the 2,000 training trajectories.
  • Task Examples: The dataset includes tasks involving navigation, traffic, calendars, email, shopping, media, and offline file access.Examples cover apps and tasks such as TickTick, Gmail, Nike, Google Drive, and music or travel queries.
  • Dataset Composition: DRL contains 170 memory-intensive queries, 80 calculation-required queries, and 350 tasks requiring no tool usage.The dataset is constructed using GPT-4o and includes these task categories to vary tool demands.
  • Training Details: The training setup uses Qwen2.5VL-7B, 50 training steps, 8 A100 GPUs, 16 prompts per batch, and 8 rollouts per prompt.The maximum response length is defined as 12,288 tokens.

D.2 Evaluation Dataset

The evaluation datasets are designed to test tool use and long-horizon execution across different difficulty profiles. MemGUI-Bench is substantially harder and longer-horizon than AndroidWorld.

  • Tool-call-Test: Tool-call-Test contains 1,000 GPT-4o-generated tasks aligned with DRL in task-type distribution.The subset is intended to evaluate tool-call behavior under a distribution matching the training dataset.
  • AndroidWorld vs MemGUI-Bench: More than half of AndroidWorld tasks are Easy, whereas MemGUI-Bench has nearly uniform Easy, Medium, and Hard proportions.The two benchmarks therefore differ substantially in difficulty composition.
  • AndroidWorld vs MemGUI-Bench: 36.2 optimal steps are required on average in MemGUI-Bench, compared with 8.4 steps in AndroidWorld.The longer optimal trajectories characterize MemGUI-Bench as the more long-horizon evaluation setting.
  • AndroidWorld vs MemGUI-Bench: The evaluation includes golden-step comparisons between AndroidWorld and MemGUI-Bench.This comparison complements the reported difficulty and average trajectory-length differences.

E Supplementary Results

Supplementary results show competitive single-turn performance, stronger pass@k scaling on MemGUI-Bench, more deliberate tool use, and fewer tool invocation errors for UI-Copilot-7B. The results also connect multi-turn training dynamics with long-horizon execution reliability.

  • Single-turn Performance: +19.1% on AC-High SR and +4.8% on GUI Odyssey SR are reported for UI-Copilot-7B over the base model.These gains occur while maintaining competitive single-turn performance.
  • Single-turn Performance: Single-turn RL models can excel on single-turn tasks yet reach only 16.4% on AndroidWorld.The reported gap is attributed to training–evaluation dynamics mismatch and overfitting to local reward signals.
  • Pass@k Validation: Increasing k consistently improves pass@k performance, and UI-Copilot-7B scales more strongly than Qwen2.5VL-7B on MemGUI-Bench.The evaluation uses k=1,2,3,4 without cross-session long-term memory.
  • Tool Distribution: Higher tool usage correlates with greater task difficulty, Retriever is used more often than Calculator, and TIPO slightly reduces overall invocation frequency.The distributions compare Qwen2.5VL-7B and UI-Copilot-7B across AndroidWorld-Verified, MemGUI-Bench-Verified, and MiniWob++.
  • Tool Error Type Analysis: UI-Copilot-7B exhibits fewer tool type errors and execution errors than the base model on MemGUI-Bench-Verified.The comparison is presented as evidence of improved tool invocation reliability.

F.1 Successful Cases

UI-Copilot demonstrates coherent tool-free execution and selective delegation for numerical and memory-intensive GUI tasks, while its failed cases expose remaining reasoning, progress-tracking, and action-consistency limitations.

  • Vanilla Execution: Over six interaction steps, vanilla rollout completes a search task coherently without external tools.The agent sequentially opens the target application, navigates to the correct section, and accesses the desired view.
  • Numerical Calculation: At step 12, the Calculator computes stock price × (1 + growth_rate) as 306.89, exactly matching ground truth.The agent first retrieves the stock price and revenue growth rate before invoking the Calculator.
  • Memory Retrieval: In two memory-intensive AndroidWorld and MemGUI-Bench cases, the agent retrieves essential early-step information for later form filling.The Memory Retriever is invoked with the task goal and locally stored history.
  • Reasoning Hallucination: UI-Copilot-7B fails a maze task by repeatedly executing Down against a black barrier, revealing limitations in visual perception and spatial reasoning.The repeated action creates a loop of incorrect behavior.
  • Progress Hallucination and Action Inconsistency: Other failures involve changing an unrelated country and terminating despite an intention to continue reviewing transactions, exposing progress confusion and action inconsistency.The unrelated action causes premature task termination, while the terminate action conflicts with the internal plan.
Loading 2604.13822v1…