Source-linked AI summary
Mobile GUI Agent Privacy Personalization with Trajectory Induced Preference Optimization
Zhixin Lin, Jungang Li, Dongliang Xu, Shidong Pan, Yibo Shi, Yuchi Liu, Yuecong Min, Yue Yao
TL;DR
Mobile GUI agents commonly optimize task success or efficiency while overlooking privacy personalization, although personas can induce structurally heterogeneous trajectories. TIPO uses preference-intensity weighting and padding gating to align such trajectories, achieving strong reported performance while preserving task executability. The method improves persona adherence and distinction, but primarily aligns trajectories rather than improving the base agent’s grounding, planning, or execution capabilities.
Problem
Existing systems often optimize task success or efficiency while overlooking privacy preferences that affect how mobile GUI tasks are executed.
Method
TIPO stabilizes preference optimization for heterogeneous trajectories using preference-intensity weighting and padding gating to emphasize relevant steps and suppress alignment noise.
Results
TIPO achieves the best overall performance among compared methods, reaching 65.60% SR, 46.22 Compliance, and 66.6% PD while preserving strong task executability.
Takeaways & Limitations
Trajectory-level persona alignment improves persona adherence and distinction while maintaining strong task executability across diverse mobile tasks.
Takeaways & Limitations
TIPO mainly improves trajectory-level persona alignment and does not substantially improve the base agent’s grounding, planning, or task execution abilities.
Abstract
from arXiv · showhide
Mobile GUI agents powered by Multimodal Large Language Models (MLLMs) can execute complex tasks on mobile devices. Despite this progress, most existing systems still optimize task success or efficiency, neglecting users' privacy personalization. In this paper, we study the often-overlooked problem of agent personalization. We observe that personalization can induce systematic structural heterogeneity in execution trajectories. For example, privacy-first users often prefer protective actions, e.g., refusing permissions, logging out, and minimizing exposure, leading to logically different execution trajectories from utility-first users. Such variable-length and structurally different trajectories make standard preference optimization unstable and less informative. To address this issue, we propose Trajectory Induced Preference Optimization (TIPO), which uses preference-intensity weighting to emphasize key privacy-related steps and padding gating to suppress alignment noise. Results on our Privacy Preference Dataset show that TIPO improves persona alignment and distinction while preserving strong task executability, achieving 65.60% SR, 46.22 Compliance, and 66.67% PD, outperforming existing optimization methods across various GUI tasks. The code and dataset will be publicly released at https://github.com/Zhixin-L/TIPO.
1 Introduction
Mobile GUI personalization requires selecting persona-consistent trajectories, not merely completing tasks, because privacy preferences can produce structurally different execution paths. TIPO addresses this heterogeneity with weighted preference learning and padding gating, improving alignment while preserving task executability.
- Privacy preferences can induce conservative actions and systematically different trajectory structures, lengths, and exposure trade-offs.
- Existing systems often optimize Success Rate or interaction efficiency while assuming each task has a single optimal trajectory.
- 65.60% SR, 46.22 Compliance, and 66.6% PD are reported for TIPO while preserving strong task executability.
- 31.94 to 38.93 Compliance from preference-intensity weighting and 46.22 with the full model show complementary gains from the two components.
- Mobile GUI personalization selects a persona-consistent trajectory among multiple paths that complete the same task.
- TIPO stabilizes preference optimization for variable-length, structurally heterogeneous trajectories through preference-intensity weighting and padding gating.
2 Related Work
Prior alignment and personalization research largely addresses homogeneous outputs, dialogue, or web-agent behavior. Mobile GUI privacy personalization instead changes concrete actions and the organization of entire trajectories, motivating trajectory-specific methods.
- 2 Related Work: DPO and related preference methods were mainly developed for relatively homogeneous compared outputs, unlike variable-length GUI trajectories.
- 2.2 Personalization and User Modeling: Under the same goal, Utility-first and Privacy-first personas can share initial steps before diverging at privacy-sensitive decisions.
- 2 Related Work: Existing personalization frameworks primarily target dialogue systems or web agents, where personalization appears in response content or high-level decisions.
- 2 Related Work: Mobile GUI preferences affect permission handling, account states, privacy exposure, and risk-related action choices throughout execution.
- 2 Related Work: Recent mobile GUI systems and benchmarks improve autonomous operation, grounding, longer-horizon execution, and practical deployment.
- 2 Related Work: Privacy research in mobile agents has focused mainly on task success, privacy awareness, or information protection rather than user-specific trajectory selection.
3 Problem Definition
Mobile GUI personalization is framed as selecting feasible execution trajectories that match a user's privacy persona while preserving task feasibility. Privacy preferences can produce systematically different strategies and trajectory lengths, so training compares persona-aligned and less-aligned trajectories under the same task context.
- Personalized trajectory selection: Privacy-first and utility-first personas can induce different execution strategies and systematically different trajectory structures and lengths.Privacy preference may add defensive actions or suppress utility-oriented actions.
- Personalized trajectory selection: Mobile GUI personalization selects a trajectory conditioned on the task, initial UI state, interaction history, and privacy persona.The agent input is x = (g, o, h, p), and the output is an action sequence y.
- Preference formulation: Persona-conditioned preference samples pair a more privacy-aligned trajectory with a less-aligned alternative under the same task context.The samples use triplets (x, y+, y−), where y+ is more aligned with the specified persona.
- Preference formulation: The preference relation concerns privacy alignment rather than task success because both trajectories are assumed to be feasible task solutions.
4 Method
Standard DPO is poorly suited to privacy-personalized GUI trajectories because alignment introduces placeholders and treats all positions uniformly. TIPO addresses these issues with step-wise preference weighting and padding gating for structurally heterogeneous trajectory pairs.
- Preliminaries: Standard DPO compares preferred and less-preferred trajectories under the same context by increasing the policy's relative likelihood of the preferred trajectory against a reference policy.
- Limitations of DPO: Variable-length persona-induced trajectories make uniform aligned-position supervision problematic because padding placeholders add semantic noise.Aligned positions may represent genuine persona decisions or positions introduced only to equalize trajectory lengths.
- TIPO: The two TIPO mechanisms are complementary: weighting highlights persona-critical steps while gating suppresses alignment-induced placeholder noise.Figure 3 presents this contrast against step-DPO, which treats aligned positions uniformly.
- Limitations of DPO: Padding alignment introduces no_action placeholders that carry no genuine preference information but still enter the loss calculation.
- TIPO: TIPO decomposes trajectory comparison into step-wise preference signals using the planning context and interaction history at each aligned step.This is a decomposition tailored to aligned-action trajectories rather than the standard DPO formulation itself.
- Preference intensity weighting: TIPO uses preference-intensity weighting to emphasize persona-critical aligned positions, such as denying permissions or disabling tracking.Neutral task-execution steps receive less preference emphasis because they carry little persona information.
- Preference intensity weighting: The preference-intensity weights are derived from rule-based action scores with LLM assistance for semantic normalization and equivalent-action cases.The detailed scoring rules are provided in the appendix.
- Padding gating: TIPO uses padding gating to suppress optimization noise from no_action placeholders introduced by variable-length alignment.The gate is zero when the chosen action is no_action and one otherwise.
5 Experiments and Results
The experiments use a binary-persona Privacy Preference Dataset and evaluate trajectory-level personalization across executability, persona adherence, and persona distinction. The protocol compares TIPO with preference-optimization baselines under controlled task splits and metrics.
- Dataset Construction: The dataset pairs Privacy-first and Utility-first trajectories for the same tasks, with human role-play, verification, and alignment producing preference pairs.The collection includes 151 task instances, 302 trajectories, and approximately 8.2k annotated steps across eight task categories.
- Dataset Construction: Privacy preferences create structurally heterogeneous trajectory pairs because protective actions can add, remove, or reorganize steps.The resulting trajectories are often length-mismatched, making direct sequence-level comparison difficult.
- Evaluation Metrics: The evaluation measures task executability, persona adherence, and persona distinction under Privacy-first and Utility-first conditions.PAS-S and PAS-U quantify normalized similarity to persona-relevant trajectory dimensions, while Compliance aggregates persona-consistent directions.
- Experimental Settings: TIPO is evaluated against Frozen, SFT, DPO, ORPO, IPO, SimPO, and CPO using a task-level split that keeps persona branches of each task together.All methods share the same backbone and controlled training and evaluation protocol, except for their optimization objectives.
6 Results
TIPO achieves the strongest overall balance across task success, persona adherence, compliance, and distinction, with performance varying by task category. Ablations attribute the gains to complementary weighting and padding-gating components.
- Comparison Against Baselines: 65.60% Overall SR is the best among compared methods, while TIPO also reaches 42.85 PAS-S under Privacy-first and 66.67% PD.The results indicate strong executability alongside persona alignment and distinction.
- Category-wise Results: TIPO maintains relatively stable persona distinction across Browsing & Interaction, Account & File Management, and Transactional Tasks, with PD scores of 66.67%, 80.00%, and 62.50%.It also achieves 53.31 Compliance in Browsing & Interaction and 72.17% Overall SR in Transactional Tasks.
- Category-wise Results: Browsing and interaction tasks concentrate persona differences at localized privacy decisions, producing relatively high Compliance.Examples include incognito mode, tracking acceptance, and clearing browsing traces.
- Category-wise Results: Account and file-management tasks show larger action-level differences between local privacy-preserving choices and cloud-oriented utility choices, supporting stronger distinction.Transactional tasks instead share most of the main flow and express preferences through softer trade-offs.
- Ablation Study: 46.22% Compliance and 66.67% PD are achieved by full TIPO, while removing either preference-intensity weighting or padding gating degrades performance.Weighting amplifies sparse persona-critical steps, whereas padding gating suppresses noise from alignment placeholders and invalid gradients.
7 Discussion
The discussion positions TIPO as a trajectory-level personalization framework extending beyond privacy and offers paired-trajectory supervision as a practical development pattern. Its scope is preference alignment rather than fundamental GUI-agent capability improvement.
- Extending TIPO Beyond Privacy: TIPO’s formulation can extend from privacy to efficiency, cost sensitivity, risk tolerance, and accessibility-related interaction needs.Privacy is presented as a representative, high-stakes entry point for user-specific trajectory preferences.
- Implications for Agent Developers: Developers can collect paired trajectories for different user profiles under the same task and use them as preference supervision without redefining the task objective.This targets acceptable interaction processes when user trust depends on how tasks are completed, not only whether they succeed.
- Limitations: TIPO mainly improves trajectory-level persona alignment rather than the base agent’s grounding, planning, or fundamental task-execution capabilities.The authors characterize it as a preference-alignment framework built on an existing agent.
8 Conclusion
The paper shows that privacy preferences induce structurally heterogeneous trajectories, making standard preference optimization less effective. TIPO improves persona alignment and distinction while preserving strong task executability.
- TIPO addresses structurally heterogeneous trajectories caused by different privacy personas through preference-intensity weighting and padding gating.These mechanisms improve preference learning for variable-length trajectory pairs.
- Experiments on the Privacy Preference dataset show improved persona adherence and persona distinction with strong task executability.
- Trajectory-level persona alignment is presented as an important step toward more practical and user-aware Mobile GUI agents.The paper identifies potential for extending personalization beyond privacy.