PaperlayerView exact revision on arXiv

Source-linked AI summary

UI-MOPD: Multi-Platform On-Policy Distillation for Unified GUI Agents

Niu Lian, Tongbo Chen, Zhehao Yu, Chengzhen Duan, Fazhan Liu, Hui Liu, Pei Fu, Jian Luan, Heng Qu, Shu-Tao Xia, Jinpeng Wang

arXiv:2607.04425v2cs.CLcs.AIcs.CVcs.LGcs.MM

TL;DR

Unified GUI agents must transfer shared capabilities across desktop and mobile while preserving platform-specific interaction conventions, but high-quality cross-platform trajectories are scarce. UI-MOPD addresses this with Uni-GUI and platform-routed multi-teacher on-policy distillation, achieving 38.2% OSWorld and 12.0% MobileWorld task success rates.

  • Problem

    Unified GUI learning lacks abundant high-quality cross-platform trajectories and must reconcile shared capabilities with conflicting desktop and mobile interaction conventions.

  • Method

    UI-MOPD builds on nearly 10K Uni-GUI trajectories to train one shared student with platform-specialized teachers routed as behavioral anchors during on-policy learning.

  • Results

    38.2% on OSWorld and 12.0% on MobileWorld task success demonstrate effective cross-platform capability preservation and adaptation.

  • Takeaways & Limitations

    Multi-teacher on-policy distillation provides an effective approach to integrating desktop and mobile expertise without averaging their distinct interaction conventions.

Abstract

from arXiv · show

Recent advances in multimodal foundation models and agent systems have driven GUI agents from single-platform task execution toward cross-platform interaction. However, unified multi-platform GUI learning remains challenging: high-quality cross-platform trajectories remain scarce, while platforms share transferable capabilities but differ in action semantics and interaction conventions. Naively mixing supervision or merging specialized models can blur native behaviors and produce imbalanced performance. To address these challenges, we construct Uni-GUI, a high-quality dataset containing nearly 10K executable cross-platform interaction trajectories collected through a unified desktop-mobile harness. Building on Uni-GUI, we propose UI-MOPD, the first framework to introduce multi-teacher on-policy distillation (MOPD) into unified multi-platform GUI agent training. UI-MOPD trains a shared student on its own rollouts and dynamically routes each rollout to the corresponding platform-specialized teacher. At student-visited states, teacher guidance serves as a platform-conditioned behavioral anchor, enabling the integration of complementary desktop and mobile expertise without averaging their distinct interaction conventions. On OSWorld and MobileWorld, UI-MOPD achieves task success rates of 38.2% and 12.0%, respectively, outperforming parameter-matched integration strategies while preserving general GUI grounding. These results demonstrate that multi-teacher on-policy distillation provides an effective approach to building unified cross-platform GUI agents. Project page: https://elispectre.github.io/UI-MOPD/.

1 Introduction

UI-MOPD addresses the conflict between shared GUI capabilities and platform-specific interaction conventions by combining a unified cross-platform dataset with multi-teacher on-policy distillation. It achieves 38.2% task success on OSWorld and 12.0% on MobileWorld while preserving cross-platform capabilities.

  • Motivation: Desktop and mobile platforms share visual understanding, GUI grounding, planning, and tool use but differ in action semantics, affordances, and navigation conventions.For example, returning may close a computer window but press mobile back.
  • Motivation: Naive heterogeneous supervision and static model merging can average away platform-specific behaviors, while single-platform specialization risks degrading performance on the other platform.The paper frames unified GUI learning as integrating complementary but partially divergent specialist knowledge.
  • Uni-GUI: Approximately 10K high-quality cross-platform interaction trajectories were collected from computer and mobile environments using a unified data-collection harness.The harness addresses scarce multi-platform data and dataset issues such as invalid actions or inaccurate state-action alignment.
  • UI-MOPD: UI-MOPD introduces multi-teacher on-policy distillation for unified GUI agent learning, routing each rollout to a platform-specific teacher.The teachers act as behavioral anchors that preserve native interaction patterns and mitigate convention mixing during optimization.
  • Results: 38.2% task success on OSWorld and 12.0% on MobileWorld demonstrate effective cross-platform capability preservation and adaptation.These results are reported for computer and mobile environments, respectively.

2 Related Work

Related work spans GUI agents evolving from single-platform automation toward web and mobile evaluation, alongside distillation methods that address capability trade-offs. On-policy distillation has progressed from single-teacher supervision to multi-teacher approaches for unifying specialized capabilities in foundation models.

  • GUI Agents: GUI agents interpret natural-language instructions and visual interface states to automate tasks, with early research largely focused on single-platform settings.Web benchmarks evaluate browser navigation, while mobile benchmarks cover executable app control, cross-app workflows, and personalization.
  • On-Policy Distillation: Post-training specialization can create a seesaw effect in which improving one capability impairs another, motivating on-policy distillation and on-policy self-distillation.The passage illustrates this trade-off through mathematical RLVR shortening reasoning traces while impairing open-ended writing.
  • On-Policy Distillation: OPD samples student trajectories and matches teacher distributions along them using reverse KL divergence, providing dense token-level supervision.Multi-teacher on-policy distillation extends OPD by assigning the strongest checkpoint for each capability dimension as a teacher.
  • Multi-Teacher Distillation: Recent foundation-model post-training systems use MOPD to distill unified models from multiple experts or recover forgotten capabilities between specialized reinforcement-learning stages.MiMo-V2-Flash and GLM-5 use MOPD as a final post-training step, while Nemotron-Cascade 2 uses it between specialized RL stages.
  • Multi-Teacher Distillation: DeepSeek-V4 further scales MOPD with full-vocabulary logits, more than ten teachers, and dedicated infrastructure for scheduling and fault-tolerant trajectory generation.These design choices support teacher coordination and robust trajectory production in multi-teacher post-training.

3 Method

UI-MOPD first trains platform-specific teachers, then integrates them into one shared student through platform-routed multi-teacher on-policy distillation. Online teacher guidance is applied to student-visited states, with adaptive masking and structured rewards supporting task improvement while preserving platform-specific interaction behaviors.

  • Training Stages: UI-MOPD uses two stages: supervised fine-tuning produces desktop and mobile teachers, followed by MOPD training of a unified student policy πθ.Stage 1 uses high-quality platform trajectories; Stage 2 integrates the specialized capabilities.
  • On-Policy Distillation: The student samples online rollouts, and teacher supervision is applied only to states it actually visits, aligning distillation with its current errors.This contrasts with directly merging expert models or distilling from static offline trajectories.
  • Adaptive KL Masking: Adaptive KL masking removes teacher penalties for prompt groups with sufficient reward while preserving guidance on low-reward rollouts.Teacher constraints are intended to remain active when task feedback is weak and avoid unnecessarily restricting exploration when feedback is already sufficient.
  • Platform-Conditioned Routing: Platform-conditioned routing evaluates each rollout with its corresponding teacher while retaining a single shared student policy and no teacher models at inference.Mixed-platform batches are partitioned by platform, evaluated by routed teachers, and restored to their original batch order.
  • Reward and Objective: Stage-2 optimization combines clipped policy optimization, structured outcome rewards, and the platform-conditioned MOPD penalty to improve long-horizon completion while preserving behavioral anchors.The structured reward distinguishes fully matched, partially valid, and invalid actions with values 1.0, −0.5, and −1.0, respectively.

4 Experiments

Experiments show that UI-MOPD delivers balanced desktop–mobile performance through routed specialist supervision, while preserving general GUI grounding and transferring complementary platform-specific behavior.

  • Main results: UI-MOPD reaches 38.2% on OSWorld and 12.0% on MobileWorld, with an 18.3% H-Mean that is highest among 8B integration strategies.Its H-Mean matches the much larger Qwen3-VL-235B-A22B-Thinking model.
  • Ablations: Neither single-platform specialist matches UI-MOPD, and Unified-Teacher OPD reaches only 35.9%/8.7% despite using a strong mixed-data 32B teacher.Random teacher selection and applying KL to all rollout groups both reduce balanced performance relative to platform-conditioned routing and adaptive masking.
  • Teacher analysis: Specialist teachers provide complementary supervision: the desktop teacher corrects more desktop states, while the mobile teacher corrects more mobile states.Useful disagreement concentrates in low-reward rollout groups, where adaptive masking retains teacher guidance.
  • Main results: UI-MOPD improves both platforms by 4.3 points over the Qwen3-VL-8B-Thinking base, achieving the largest MinGain in Table 2.Desktop-only SFT collapses MobileWorld from 7.7% to 0%, whereas mobile-only SFT reaches 12.8% on MobileWorld but remains below UI-MOPD’s 38.2% on OSWorld.
  • Generalization: UI-MOPD improves AndroidControl⋆ from 78.73% to 80.05% and OSWorld-G from 52.13% to 52.84%, while staying within 0.6 points of the base model on ScreenSpot-Pro and ScreenSpotV2.The evaluation reports success rates for interactive benchmarks and accuracy for static understanding and grounding.

5 Conclusions

The work integrates specialist knowledge into a unified GUI agent for heterogeneous desktop and mobile environments. It contributes a unified data-collection harness, the nearly 10K-trajectory Uni-GUI dataset, and UI-MOPD for multi-platform GUI training.

  • The work targets specialist knowledge integration for a unified GUI agent across heterogeneous desktop and mobile environments.
  • Uni-GUI contains nearly 10K high-quality, executable cross-platform interaction trajectories collected with a unified data-collection harness.
  • UI-MOPD is introduced as the first multi-teacher on-policy distillation framework for unified multi-platform GUI agent training.

Appendix · A Dataset Construction and Composition

The appendix describes Uni-GUI’s cross-platform trajectory composition and the AndroidControl⋆ subset used for static GUI-understanding evaluation. It also records the shared mobile action space, grounding metadata, and preprocessing applied to public trajectories.

  • A Dataset Construction and Composition: AndroidControl⋆ contains 4,260 step-level records from 781 Android trajectories for static GUI-understanding evaluation.Each JSONL record includes trajectory and step identifiers, task and step instructions, normalized action, screenshot path, and screenshot resolution.
  • A Dataset Construction and Composition: AndroidControl⋆ preserves Uni-GUI’s normalized mobile action space, including click, scroll, input_text, open_app, wait, navigate_back, long_press, and navigate_home.The subset also includes grounding metadata for actions matched to UI elements.
  • A Dataset Construction and Composition: Grounding metadata includes target bounding boxes, widget class, visible text or content description, resource id, package name, ancestor information, and matching-node counts.Steps without directly groundable targets, such as waiting or app-related actions, are handled separately in the supplied description.
  • A Dataset Construction and Composition: Uni-GUI combines four trajectory groups spanning desktop and mobile platforms.Each platform mixes trajectories collected with the Unified Cross-Platform Data Collection Harness and cleaned open-source trajectories.
  • A Dataset Construction and Composition: 95K desktop interaction steps come from self-collected GUI environments, while 13K cleaned steps come from OpenCUA.The counts are approximate, as reported in the composition description.
  • A Dataset Construction and Composition: OpenCUA and OpenMobile are not used as raw trajectories; they undergo action-space compatibility checking, trajectory filtering, and format normalization.These cleaning and post-processing steps are described in Appendix B.
  • A Dataset Construction and Composition: Table 4 summarizes Uni-GUI’s approximate composition, with counts rounded for readability.The supplied table passage provides the table’s scope and rounding convention but no additional row values.
  • A Dataset Construction and Composition: Including filtered and normalized OpenCUA and OpenMobile data broadens GUI-state, application, and task-type coverage while adding supervision for cross-platform generalization.These sources complement the desktop and mobile trajectories collected by the unified harness.

B Unified Cross-Platform Data Collection Harness

The unified harness constructs cross-platform GUI trajectories through four stages: query generation, trajectory collection, trajectory cleaning, and post-processing. It produces successful, executable, action-compatible, reasoning-normalized, and visually grounded trajectories for desktop and mobile platforms.

  • Harness Pipeline: The harness organizes data construction into query generation, trajectory collection, trajectory cleaning, and post-processing.This four-stage structure is designed to reduce noise in GUI agent trajectories.
  • Query Generation: Queries are grounded in realistic, executable functionalities supported by the target desktop or mobile environments.Kimi-K2.6 assists desktop query generation, while Gemini-3.1-Pro assists mobile query generation.
  • Trajectory Collection: Teacher models collect trajectories through a shared high-level interface while preserving native desktop and mobile action forms before normalization.Each trajectory records observations, actions, intermediate reasoning, and task states during rollout.
  • Trajectory Cleaning: Cleaning removes malformed or student-incompatible trajectories and retains only rollouts whose ordered sub-tasks are all judged complete.Gemini-3.1-Pro performs sub-task-level adjudication of task completion.
  • Post-Processing: Post-processing normalizes reasoning into structured chain-of-thought and re-annotates visual grounding boxes for UI-element actions.The resulting trajectories support subsequent rule-based evaluation of whether actions target the correct region.

C Training and Implementation Details · C.1 Action Space and Trajectory Format

The trajectory format uses platform-specific tool interfaces and normalized episode records for unified training. Raw generation artifacts are retained separately to preserve traceability while training uses cleaned trajectories.

  • C.1 Action Space and Trajectory Format: Desktop trajectories use computer_use with mouse and keyboard actions, whereas mobile trajectories use mobile_use with touchscreen actions.Post-processing converts actions from different data sources into these platform-specific tool-call formats.
  • C.1 Action Space and Trajectory Format: Every retained trajectory is converted into a platform-specific tool-call format during post-processing.This conversion standardizes trajectories across data sources before retention and training.
  • C.1 Action Space and Trajectory Format: Each trajectory is stored as an episode directory containing normalized records and associated screenshots.The mobile episode structure includes task.json, task_raw.json, and step-indexed screenshots such as 0.jpg and 1.jpg.
  • C.1 Action Space and Trajectory Format: The normalized task.json file stores metadata including task source, application, package, resolution, query, episode identifier, device type, and split.Its data field contains the step trajectory records.
  • C.1 Action Space and Trajectory Format: Each step records the trajectory state in the normalized episode representation.The provided description identifies the data field as the container for step trajectory records.
  • C.1 Action Space and Trajectory Format: The raw task_raw.json file preserves the original prompt, model response, raw action, converted action, and labeled screenshot reference for each step.This separation retains generation evidence while training uses only the cleaned, normalized trajectory representation.

C.2 Training Hyperparameters

Training initializes a smaller student and larger platform-specific teachers, then runs one-epoch teacher SFT and student GRPO-based DAPO training with multi-teacher on-policy distillation. Training and inference use different visual and action-history contexts.

  • The student policy is initialized from Qwen3-VL-8B-Thinking, while platform-specific teachers are initialized from Qwen3-VL-32B-Thinking.
  • Both teacher SFT and student training run for one epoch using a GRPO-based DAPO objective with multi-teacher on-policy distillation.Each prompt samples 8 rollouts, and the OPD auxiliary KL loss uses coefficient 0.01.
  • Training uses only the current screenshot, whereas inference uses four historical screenshots, the current screenshot, and the full text action history.

D Fine-Grained Static GUI and Grounding Results

UI-MOPD improves fine-grained AndroidControl⋆ metrics over the base model while preserving grounding performance across benchmarks. Model Merge instead degrades mobile GUI understanding and grounding more broadly.

  • Static GUI evaluation: UI-MOPD improves all reported AndroidControl⋆ metrics, including action type prediction, target grounding, ancestor grounding, and overall accuracy.AndroidControl⋆ is the evaluated subset; the base model is Qwen3-VL-8B-Thinking.
  • Static GUI evaluation: Model Merge consistently degrades AndroidControl⋆ metrics, indicating less stable preservation of mobile GUI understanding.Model Merge refers to the TIES-merging checkpoint.
  • Grounding benchmarks: UI-MOPD largely preserves base-model grounding performance on ScreenSpot-Pro and ScreenSpotV2 while improving Creative, Scientific, and desktop grounding subcategories.These results contrast with the larger grounding degradation observed for Model Merge.
  • Grounding benchmarks: UI-MOPD achieves the best overall OSWorld-G score and improves layout understanding, demonstrating retained fine-grained GUI grounding ability.The grounding results indicate that multi-teacher on-policy distillation can preserve detailed GUI understanding.

E Additional Case Studies · F Prompt Templates

UI-MOPD is qualitatively demonstrated on a multi-step desktop task requiring dense-interface localization and precise mouse-and-keyboard control. The prompt templates define platform-specific Qwen3-VL interfaces and separate desktop/mobile trajectory-collection protocols with Kimi-K2.6 and Gemini-3.1-Pro.

  • E Additional Case Studies: UI-MOPD follows a multi-step desktop instruction by locating task-relevant regions in a dense interface and executing state-dependent mouse-and-keyboard actions.The case study highlights larger layouts, window-level operations, and precise cursor-based grounding relative to mobile tasks.
  • E Additional Case Studies: The desktop case transfers Calc data into a Writer table, preserves the original format, and saves the document as "price.docx" on the desktop.This is the concrete instruction illustrated by the desktop execution example.
  • F Prompt Templates: Four system prompts support data construction and policy training, separating normalized platform interfaces from trajectory-collection prompts.The first two prompts define desktop and mobile interfaces for Qwen3-VL-based policies, while the other two collect platform-specific trajectories.
  • F Prompt Templates: The desktop Qwen3-VL interface uses computer_use for screenshot-guided mouse-and-keyboard interaction at 1000x1000 resolution, with centered cursor clicks and explicit waiting when needed.The tool supports desktop actions including key presses, typing, cursor movement, clicks, dragging, scrolling, waiting, and termination.
  • F Prompt Templates: Desktop policy responses must contain one brief Action followed by a single JSON tool_call, with termination represented by action=terminate.The prompt forbids additional output outside the required Action and tool_call order.
  • F Prompt Templates: The mobile Qwen3-VL interface uses mobile_use for touchscreen actions at 999x999 resolution, including centered clicks, long presses, swipes, typing, waits, and system-button operations.Its action vocabulary includes click, long_press, swipe, type, answer, system_button, wait, ask_user, and terminate.
  • F Prompt Templates: Mobile policy responses must provide one concise Thought, one Action, and one JSON tool_call, while termination uses mobile_use with action=terminate.The prompt requires this exact order and prohibits output outside the three specified parts.
Loading 2607.04425v2…