Source-linked AI summary
Towards Omni-dimensional GUI Agent Navigation with Masked Trajectory Prediction
Yan Zhang, Pei Fu, Daiqing Wu, Huawen Shen, Ruoceng Zhang, Shaojie Zhang, Jiahui Yang, Yu Zhou, Can Ma, Zhenbo Luo, Jian Luan
TL;DR
GUI navigation requires step-wise decisions, state-action alignment, and long-horizon planning, but mixing these tasks is hindered by inconsistent objectives and heterogeneous data. MaP uses masked trajectory prediction and role-aware adapters to unify them. The paper reports that MaP outperforms direct mixture training across GUI navigation benchmarks.
Problem
GUI navigation needs complementary capabilities, but direct mixture training is limited by inconsistent objectives across paradigms and heterogeneous corpora.
Source: §1 · p. 1Method
MaP models multi-turn interactions as trajectories, predicts masked components under a unified objective, and uses role-aware adapters to route tokens to specialized representation spaces.
Source: §1 · p. 2Results
MaP significantly outperforms direct mixture training across multiple public benchmarks, and outperforms pairwise combinations of the navigation paradigms.
Source: §1 · pp. 2–3Takeaways & Limitations
The paper concludes that MaP provides a unified framework for multi-task GUI navigation over heterogeneous data.
Source: §5 · p. 9The study mainly uses moderate-scale backbones and publicly available GUI corpora, and does not fully explore scaling to larger LVLMs or more diverse training data.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Graphical User Interface (GUI) Agents autonomously interact with software to fulfill user requests, where GUI navigation stands out as the most critical and challenging capability. Mastering this capability demands a complex synergy of step-wise decision-making, state-action alignment, and long-horizon planning. While directly mixing these corresponding navigation tasks seems intuitive to simultaneously acquire these skills, such a direct combination is severely bottlenecked by inconsistent optimization objectives and profound data heterogeneity. To overcome these barriers, we propose the MaP (stands for ``\textbf{M}asked Tr\textbf{a}jectory \textbf{P}rediction''), a unified framework that seamlessly harmonizes divergent GUI navigation tasks. By modeling multi-turn GUI interactions as a trajectory and defining training objectives through component masking and prediction, MaP shifts the optimization from task-specific marginal distributions to a consistent objective. Furthermore, to handle the data heterogeneity across multiple navigation tasks, we design a role-aware adapter learning module that dynamically routes each token to a specialized representation space. Extensive experiments on five representative GUI navigation benchmarks demonstrate that MaP effectively mitigates gradient conflicts and significantly outperforms the direct mixture training, establishing a robust paradigm for multi-task GUI navigation.
1 Introduction
GUI navigation combines visual-semantic reasoning, state-action alignment, and long-horizon planning. MaP unifies these paradigms through masked trajectory prediction and role-aware adapters, addressing mismatched objectives and heterogeneous data; the paper reports improvements over existing methods across multiple benchmarks.
1 Introduction
- Direct mixture training yields only marginal improvements because navigation paradigms use inconsistent objectives and their corpora differ in reasoning styles, annotation protocols, and interface domains.Step-wise decision focuses on the current state, state-action alignment neglects broader task context, and long-horizon planning can lack fine-grained visual grounding.Source: §1 · pp. 1–2
- MaP models multi-turn GUI interactions as trajectories, masks arbitrary components, and trains the agent to predict the masked parts autoregressively.This casts different navigation paradigms into a unified trajectory-masking formulation.Source: §1 · p. 2, Figure 2 · p. 2
- MaP unifies the three navigation paradigms and outperforms their pairwise combinations, with significant improvements over existing methods across multiple public benchmarks.Experiments span five representative benchmarks: AndroidWorld, AndroidControl, GUI-Odyssey, AITZ, and Mind2Web.Source: §1 · pp. 2–3
- A role-aware adapter dynamically routes each token to a specialized representation space to address heterogeneity across navigation-task data.MaP shifts optimization from task-specific marginal dependencies to a unified objective over complete trajectories.Source: §1 · pp. 2–3
2 Related Work
Related work covers vision-based GUI agents and the varied corpora used to train them, including mobile, web, and desktop interactions.
2.1 GUI Agent
- Vision-based GUI agents use LVLMs to interpret screenshots and predict actions, with work spanning visual grounding, deliberate reasoning, and imitation of human actions.Earlier source-code-based approaches relied on internal APIs, limiting applicability to commercial software.Source: §2.1 · p. 3
2.2 GUI Corpus
- GUI corpora include Android sequences, CoT annotations, cross-app navigation, and synthetic or video-derived trajectories that expand training data beyond manually annotated interactions.Examples include Mobile3M's 20 million synthetic interactions and MONDAY's 313k annotated frames from instructional videos.Source: §2.2 · p. 3
3 Method
MaP unifies GUI navigation training through masked trajectory prediction and role-aware adapters: the former aligns task objectives, while the latter routes heterogeneous trajectory tokens to specialized representations.
3.2 Mask Trajectory Prediction
- Training randomly masks a fixed proportion of trajectory components, using [Mask] placeholders to prompt prediction of the missing parts.The trajectory contains instructions, screenshots, reasoning, and actions.Source: §3.2 · p. 4
- MaP treats multi-turn interactions as trajectories and masks selected components, shifting optimization from task-specific marginals to a joint objective across navigation paradigms.The objective predicts masked components from the remaining trajectory context, with loss computed only at masked positions.Source: §3.2 · pp. 3–4
- An 80% masking ratio gives the best reported trade-off between task difficulty and model performance; extreme masking such as 100% could harm learning stability.The authors cite data heterogeneity and low-quality screenshots as reasons to avoid extreme masking.Source: §3.2 · p. 4
- At inference, MaP replaces current-step reasoning and action with masks, then predicts them from the contextual trajectory.The inference masking configuration mirrors training.Source: §3.2 · p. 4, Figure 3 · p. 5
3.3 Role-aware Adapter Learning
- Role-aware learning routes each trajectory token to a specialized adapter by semantic role, addressing corpora with differing reasoning, annotations, and screenshot quality.GUI trajectories include high- and low-level instructions, screenshots, thoughts, observations, and actions.Source: §3.3 · pp. 4–5
4 Experiments
Across five GUI-navigation benchmarks, MaP generally outperforms direct mixture training and existing methods. The experiments cover zero-shot capability, downstream and online evaluation, trade-offs, and two design ablations.
4.1 Datasets and Evaluation Metrics
- Evaluation covers five datasets, with global benchmarks testing end-to-end tasks and AndroidControl-Low testing step-wise actions.The reported metrics are action-type exact match, click-coordinate accuracy, and step-wise success rate.Source: §4.1 · pp. 5–6
4.2.1 Zero Shot Performance.
- MaP outperforms direct mixture training by 2.8%, 5.4%, and 2.1% on step-wise decision, state-action alignment, and long-horizon planning, respectively.These are zero-shot results on AndroidControl without dataset-specific adaptation.Source: Figure 5 · p. 6, §4.2.1 · p. 6
4.2.3 Trade-off Analysis.
- MaP achieves 89.9% on AndroidControl-Low and 67.9% on AndroidControl-High, while direct mixture checkpoints fail to surpass specialized baselines on both benchmarks.Figure 5(b) plots SR on AndroidControl-Low against SR on AndroidControl-High.Source: Figure 5 · p. 6, §4.2.3 · pp. 6–7
4.2.2 Post Training Performance.
- MaP's downstream SR increases are 3.0% on AndroidControl-High and 2.7% on AndroidControl-Low, while direct mixture training yields negligible baseline improvements.The reported pattern holds across Qwen2.5-VL model scales including 3B and 7B.Source: §4.2.2 · p. 6
4.3.2 Offline Evaluation
- MaP exceeds UI-TARS-7B by 1.7% SR on AndroidControl-High and 1.5% SR on GUI-Odyssey, and surpasses Falcon-UI by 4.8% on AITZ.On Mind2Web, MaP achieves state-of-the-art performance and surpasses UI-TARS despite its 50B tokens of in-house pretraining.Source: Table 2 · p. 7, §4.3.2 · p. 8
4.3.3 Local Evaluation.
- For local AndroidControl-Low evaluation, MaP on Qwen2.5-VL-7B has higher SR than UI-TARS, despite UI-TARS being pretrained on 50B tokens.The authors attribute the result to training on diverse prior navigation strategies, particularly single-step instructions.Source: §4.3.3 · p. 8
4.3.1 Online Evaluation.
- MaP gains 2.6% Pass@1 and 5.4% Pass@4 over direct mixture training on AndroidWorld with Qwen2.5-VL-7B.The evaluation involves online interactions with real-world applications.Source: §4.3.1 · pp. 7–8, Figure 6 · p. 8
4.4.1 Mask Ratio.
- Performance rises as the mask ratio increases from 0.2 to 0.8, then drops at 1.0; adapter performance peaks at four, above the single-adapter LoRA baseline.At a mask ratio of 1.0, reasoning relies only on visual context across screenshots; the adapter comparison uses AndroidControl-High.Source: §4.4.1 · p. 8, Table 3 · p. 8, Table 4 · p. 8
5 Conclusion
MaP addresses inconsistent objectives and data heterogeneity in multi-task GUI-agent mixture training. It uses masked trajectory prediction and role-aware adapters, and evaluations report gains over direct mixture training.
5 Conclusion
- MaP targets inconsistent objectives and data heterogeneity in multi-task GUI-agent mixture training.These are identified as challenges for mixture training.Source: §5 · p. 9
- MaP casts diverse navigation strategies as masked trajectory prediction and uses a role-aware adapter to consolidate objectives across heterogeneous data.Source: §5 · p. 9
- Extensive evaluations report that MaP significantly outperforms direct mixture training across GUI navigation benchmarks.Source: §5 · p. 9
Limitations
The study reports a scope boundary in model scale and training data: its experiments mainly use moderate-scale backbones and publicly available GUI corpora.
Limitations
- The study mainly evaluates moderate-scale backbone models and publicly available GUI corpora.Source: Limitations · p. 9
- The scaling potential of MaP with larger LVLMs and more diverse training data remains unexplored because of computational constraints.Source: Limitations · p. 9
- Future work will examine whether larger model capacities and expanded GUI trajectory corpora further benefit MaP.Source: Limitations · p. 9
Ethical Considerations
The paper identifies GUI interaction as the source of potential risks, including sensitive screenshot information and unintended operations. It recommends standard safeguards and describes an action space for GUI navigation.
Ethical Considerations
- Potential deployment risks come from GUI interaction, including sensitive information in screenshots and unintended operations.The paper recommends data anonymization, user authorization, and confirmation for sensitive actions.Source: Ethical Considerations · p. 9
Appendix
- The appendix lists implementation details, evaluation benchmarks, theoretical and gradient-optimization analyses, and qualitative analysis.Source: Appendix · p. 12
A Unified Action Space
- The unified action space distinguishes Ground actions requiring targets and coordinates, Content actions requiring text, and Func actions requiring only an action type.It adopts the Qwen2.5-VL base model's native action set.Source: A Unified Action Space · p. 12
B Implementation Details
The implementation specifies the model inputs, training configuration, and compute setup used for MaP.
B Implementation Details
- MaP uses Qwen2.5-VL models with 1280 × 720 inputs, 16,384-token limits, and batches of 32 with eight gradient-accumulation steps.Training uses AdamW, a cosine learning-rate schedule with 5% warm-up, DeepSpeed, BF16, and gradient checkpointing on H100-80G GPUs.Source: B Implementation Details · p. 12
C Evaluation Benchmarks
The evaluation benchmarks cover mobile and web GUI tasks, from individual action grounding to long-horizon and zero-shot navigation. The theoretical analysis contrasts summed task gradients with MaP’s trajectory-level masked prediction objective.
C.1 AndroidWorld.
- AndroidWorld evaluates 116 tasks across 20 Android apps, using state-based rewards to verify completion and reporting Success Rate.Its tasks range from system settings to multi-step workflows such as shopping and information retrieval.Source: §C.1 · p. 12
C.2 AndroidControl.
- AndroidControl contains 15,226 tasks across 120 applications and evaluates action grounding and exact predicted-step matches.The reported metrics are Grounding and SR, respectively.Source: §C.2 · p. 12
C.3 GUI-Odyssey.
- GUI-Odyssey provides 8,334 long-horizon episodes across 212 applications for cross-app navigation, evaluated with Ground Acc and SR.The episodes require agents to manage long-term history across multiple apps.Source: §C.3 · p. 12
C.4 AITZ.
- AITZ contains 18,643 smartphone screen-action pairs and is distinguished by Chain-of-Action-Thought annotations.Source: §C.4 · p. 12
C.5 Mind2Web.
- Mind2Web covers over 2,000 open-ended tasks across 137 websites and 31 domains, emphasizing zero-shot navigation of unseen tasks, sites, and domains.Source: §C.5 · pp. 12–13
D Theoretical Analysis
- Direct mixture optimization sums gradients from step-wise decision, state-action alignment, and long-horizon planning to balance their optimization directions.The mixture gradient is represented by ∇θLmix.Source: D Theoretical Analysis · p. 13
- MaP randomly masks trajectory components such as reasoning, actions, or interaction history, then predicts them in the context of the full sequence.This trains the components toward a shared trajectory-level objective rather than treating tasks in isolation.Source: D Theoretical Analysis · p. 13
- The analysis compares gradient cosine similarity across samples from existing navigation paradigms, defining its mean and variance to characterize alignment.The cosine is between gradient vectors; the mean is taken over sample-pair similarities.Source: D Theoretical Analysis · p. 13
E Gradient Optimization Analysis.
The gradient analysis compares MaP with direct mixture training using cosine-similarity distributions. MaP shows higher mean similarity and lower variance at the examined layers.
E Gradient Optimization Analysis.
- The comparison examines optimization-direction consistency between MaP and direct mixture training through gradient cosine similarity.Source: E Gradient Optimization Analysis · p. 13
- Approximately 0.17 higher mean cosine similarity and about 0.02 lower variance: MaP versus direct mixture training at the LM head and last attention k-projection layer.The authors interpret the higher similarity as more harmonized objectives and the lower variance as greater training stability.Source: Figure 7 · p. 14, E Gradient Optimization Analysis · p. 14
F Qualitative Analysis
The qualitative comparisons contrast MaP with the Qwen2.5-VL-3B base model on long-horizon GUI-Odyssey navigation and local AndroidControl-Low actions. The examples show failures by the base model that MaP avoids.
F Qualitative Analysis
- On GUI-Odyssey’s calendar task, MaP completes the intended sequence while the base model opens Chrome instead of Calendar and selects the wrong date.The figure marks MaP tap positions in red and base-model positions in green.Source: F Qualitative Analysis · p. 14
- On AndroidControl-Low, MaP correctly taps the send icon in a representative case where the base model taps “multiple options.”The comparison uses four representative examples of step-wise decision capability.Source: F Qualitative Analysis · p. 14, Figure 9 · p. 15