PaperlayerView exact revision on arXiv

Source-linked AI summary

PersonalAlign: Hierarchical Implicit Intent Alignment for Personalized GUI Agent with Long-Term User-Centric Records

Yibo Lyu, Gongwei Chen, Rui Shao, Weili Guan, Liqiang Nie

arXiv:2601.09636v2cs.AIcs.CVcs.HCcs.LG

TL;DR

Existing GUI agents often assume complete instructions and lack mechanisms for long-term, user-centric records needed to resolve implicit intent. PersonalAlign introduces a hierarchical task, AndroidIntent benchmark, and HIM-Agent memory framework; HIM-Agent achieves the best reported performance in alleviating vague instructions, with a CSR score of 42.3.

  • Problem

    Existing GUI agents are commonly evaluated in simulated environments, assume complete instructions, and struggle to manage long-term user records needed to infer implicit intent.

  • Method

    PersonalAlign evaluates hierarchical implicit-intent alignment through AndroidIntent and HIM-Agent, which hierarchically organizes preferences and routines from long-term user records.

  • Results

    HIM-Agent achieves the best performance in alleviating vague instructions, obtaining a CSR score of 42.3.

  • Takeaways & Limitations

    The benchmark and memory framework support evaluating and improving personalized GUI agents that resolve implicit intent in daily interactions.

  • Takeaways & Limitations

    Evaluation is restricted to the Fingertip because suitable large-scale public datasets of long-term interaction records remain limited, and cold-start effects may impair intent inference.

Abstract

from arXiv · show

While GUI agents have shown strong performance under explicit and completion instructions, real-world deployment requires aligning with users' more complex implicit intents. In this work, we highlight Hierarchical Implicit Intent Alignment for Personalized GUI Agent (PersonalAlign), a new agent task that requires agents to leverage long-term user records as persistent context to resolve omitted preferences in vague instructions and anticipate latent routines by user state for proactive assistance. To facilitate this study, we introduce AndroidIntent, a benchmark designed to evaluate agents' ability in resolving vague instructions and providing proactive suggestions through reasoning over long-term user records. We annotated 775 user-specific preferences and 215 routines from 20k long-term records across different users for evaluation. Furthermore, we introduce Hierarchical Intent Memory Agent (HIM-Agent), which maintains a continuously updating personal memory and hierarchically organizes user preferences and routines for personalization. Finally, we evaluate a range of GUI agents on AndroidIntent, including GPT-5, Qwen3-VL, and UI-TARS, further results show that HIM-Agent significantly improves both execution and proactive performance by 15.7% and 7.3%.

1 Introduction

PersonalAlign addresses the gap between reactive GUI execution and users’ implicit intent by using long-term records to resolve omitted preferences and anticipate routines. It reframes personalized GUI assistance around hierarchical intent alignment rather than complete instructions alone.

  • Existing GUI agents are largely evaluated in simulated environments under the assumption that user instructions are complete.
  • Users may omit recurring preferences because they expect agents to infer shared context from historical records.
  • PersonalAlign requires agents to infer preferences in vague instructions and separate frequent, state-consistent routines for proactive assistance.
  • The task aims to move GUI agents from independent reactive executors toward personalized partners that co-evolve through interaction.

1. Lack of long-term, user-centric annotated

The paper identifies missing long-term, user-centric annotations and inadequate memory organization as barriers to implicit intent alignment. It responds with AndroidIntent and HIM-Agent, then evaluates their effectiveness across GUI agents.

  • AndroidIntent annotates daily preferences and routines from long-term records while simulating implicit intent by removing recoverable personal preferences.
  • A hierarchical filtering-verifying strategy converts ambiguous personalization concepts into quantifiable scores for human verification.
  • HIM-Agent incrementally updates user behavior and separates preference and routine memories through execution-based and state-based filters.
  • Vague instructions can cause fine-grained execution failures when specific personal requirements are absent at individual steps.
  • The paper reports superior performance for HIM-Agent across experiments with various GUI agents.

2 Related Work

Related work has improved GUI execution, personalization, and memory, but existing benchmarks and methods leave long-term user-centric intent alignment insufficiently evaluated.

  • Many GUI datasets and methods emphasize execution success rate rather than matching users’ true intent.
  • Prior personalization work can extract explicit and implicit intention flows from short-term instruction traces.
  • Agent-memory research compresses dialogues, organizes memory through reinforcement learning, or stores execution skills to improve GUI performance.
  • These related approaches motivate memory mechanisms for persistent user interactions and personalized GUI assistance.

3 PersonalAlign Task Definition

PersonalAlign defines a hierarchy from reactive execution to preference inference and routine anticipation. Agents use chronological long-term records to fill omitted preferences or proactively suggest actions under similar user states.

  • Each user record contains an instruction, interaction time, scenario, action trajectory, and screen observations, split chronologically into historical and executing records.
  • Reactive Intent Alignment: Reactive alignment executes explicit instructions without leveraging long-term historical records or personalization.
  • Preference Intent Alignment: Preference alignment infers omitted preferences from historical intent and action sequences so each action step satisfies the user’s intent.
  • Routine Intent Alignment: Routine alignment uses repetitive historical intents under similar user states to generate proactive suggestions without an instruction.
  • AndroidIntent uses a two-stage filtering-verification pipeline that combines objective criteria with subjective judgment for hierarchical intent annotation.

4 AndroidIntent

AndroidIntent is constructed from long-term Android interaction records to identify preferences and routines, quantify their stability, and evaluate vague instructions and proactive behaviors. Its filtering and verification process yields a diverse, user-centric benchmark while remaining currently scoped to Android environments.

  • Dataset construction: AndroidIntent is built from 91 users’ two months of Android interaction records to annotate preference and routine intents from long-term histories.The source records come from Fingertip20K and contain approximately 20k interactions.
  • Hierarchical filtering: The analysis distinguishes moment intents from recurring preferences and routines using occurrence frequency and consistency within historical records.The strategy aims to validate stable, modelable personal patterns before annotation.
  • Hierarchical filtering: Semantic similarity, temporal offsets, and scenario offsets quantify whether historical interactions recur under similar intents and user states.Lower temporal and scenario entropy indicates more stable states suitable for proactive assistance.
  • Hierarchical filtering: Three approximately Gaussian modes emerge in large-scale intent scores, corresponding to moment, preference, and routine intents.This structure supports fast candidate filtering for preference and routine annotation.
  • Annotation: Human verification broadens sampling near overlapping score regions, rechecks intent types against user histories, and cross-validates annotations.GPT-generated alternatives omit potential preferences to create vague instructions, which annotators select and verify.
  • Benchmark statistics: The benchmark contains 775 preference interactions and 215 proactive interactions across 190 applications, with richer user coverage and temporal depth than prior datasets.Preference data include 7,915 GUI actions, while proactive behaviors span 60 apps; vague instructions show reduced lexical entropy and simplified language.
  • Scope: AndroidIntent currently targets Android mobile GUI interaction, although its filtering and annotation pipeline is described as adaptable to other platforms and languages.The authors identify privacy-preserving, scalable collection of user-centric GUI data as an important future direction.

5 HIM-Agent

HIM-Agent builds a continuously evolving personal memory from long-term GUI records, then separates stable preferences from state-consistent routines for hierarchical intent alignment.

  • 5 HIM-Agent: HIM-Agent updates user records incrementally and hierarchically organizes aggregated prototypes into preference and routine intent memories.The Streaming Aggregation Module updates records daily, while the resulting prototypes support personalized preference and routine intent.
  • 5 HIM-Agent: Raw GUI records are fragmented and noisy, so Streaming Aggregation Module replaces static logs with continually evolving Record Prototypes.The module aggregates similar records rather than retaining only individual low-level interactions.
  • 5 HIM-Agent: Execution-based filtering combines semantic intent similarity with action-trajectory consistency to aggregate GUI interactions more comprehensively.Semantic similarity combines dense cosine and sparse Jaccard measures, while action consistency uses Dynamic Time Warping.
  • 5 HIM-Agent: Record prototypes become preference memories whose center intent and action are supplied when the agent infers user preferences.Prototype centers are updated daily using instructions and actions with minimum average distance to assigned records, based on GUI action success rate.
  • 5 HIM-Agent: State-based filtering separates passive preferences from proactive routines using state stability, recurrence length, and consistency-derived confidence.Prototypes exceeding the proactive confidence boundary enter Routine Intent Memory and provide their center intent and most frequent state for proactive decisions.

6 Experiment

Experiments show that vague instructions substantially impair fine-grained GUI execution and that current agents struggle with proactive identification. HIM-Agent improves implicit-intent alignment through preference and routine memories, while ablations support its component design.

  • 6 Experiment: Vague instructions reduce SSR and CSR by approximately 20% and 45%, respectively, despite only a 3% drop in type accuracy.Agents often identify the high-level operation but fail at fine-grained steps when personalized preference information is missing.
  • 6 Experiment: Most GUI agents struggle to balance proactive false alarms and recall, often defaulting to overly proactive suggestions.The evaluation measures proactive service using false-alarm and alignment-related criteria.
  • 6 Experiment: HIM-Agent achieves a CSR score of 42.3 and improves proactive balance to 81.4% recall and 49% false-alarm, with Intent Alignment scores of 53.3% and 36.3%.It is built on Qwen3-VL and compared with retrieval-based and LLM-based alternatives across execution and proactive evaluations.
  • 6 Experiment: The execution-based preference filter’s full configuration improves CSR by 9.1%, while removing state-related filtering increases false alarms.The ablation attributes gains to the three preference-filter components and highlights the importance of time and scenario information for proactive behavior.
  • 6 Experiment: Case studies show HIM-Agent infers omitted preferences for execution and combines historical records with current state for proactive assistance.Reactive agents remain inactive without explicit instructions in the reported comparisons.

7 Conclusion

PersonalAlign defines hierarchical personalization as a challenge for resolving implicit intents in daily GUI interactions. AndroidIntent and HIM-Agent are introduced to evaluate and support this capability.

  • PersonalAlign requires hierarchical personalization to resolve implicit intents in daily interactions.The task shifts GUI agents beyond simple execution toward aligning with users’ implicit intent.
  • AndroidIntent is introduced as a user-centric benchmark for evaluating implicit-intent resolution.The benchmark is curated with a filter-verify strategy and uses long-term user records.
  • HIM-Agent is proposed as a memory framework that enables hierarchical personalization from long-term user records.The framework is presented as support for the PersonalAlign task.

Limitations

The evaluation is constrained by limited suitable datasets and insufficient validation across diverse real-world settings. HIM-Agent may also face cold-start difficulties when historical interaction data is insufficient.

  • Evaluation is currently restricted to the Fingertip dataset because large-scale public datasets of long-term user interaction records are limited.The annotation strategy and method therefore cannot yet be broadly validated across diverse real-world datasets.
  • Insufficient historical interaction data may limit accurate intent inference and proactive suggestions through a cold-start issue.The paper identifies addressing this challenge as future research.
  • Future validation could extend data collection to other GUI environments, including operating systems and desktop applications.

Ethical considerations

Personalized GUI agents rely on historical interaction data, creating privacy risks that require explicit safeguards. The evaluation uses the Fingertip dataset without adding personal information and relies on participant consent.

  • Historical interaction data used for user-intent modeling may create privacy risks.
  • The study uses the Fingertip dataset without introducing additional personal information.
  • Fingertip participants were informed about data use, signed data usage agreements, and were instructed not to upload private or sensitive information.
  • The paper identifies on-device deployment, federated learning, and simulated user agents as possible privacy-preserving directions.

A.3 Online Evaluation

The appendix discusses online evaluation and more proactive forms of personalization, while explaining why PersonalAlign currently uses offline, instruction-level assessment. These extensions require real-device or online-like environments and richer evaluation of user responses.

  • Online Evaluation: AndroidIntent requires real physical devices via ADB because many daily apps cannot run on emulators under privacy and security restrictions.
  • Online Evaluation: Online evaluation is difficult because successful intent completion cannot be automatically verified and real-device results vary across apps, devices, and runtime environments.Executions must instead be manually inspected, motivating primarily offline evaluation for objective and reproducible assessment.
  • C.2 Proactive Triggering: Trigger-based proactive agents activate recommendations only when predefined temporal and contextual conditions are met.This approach avoids continuously reasoning over the current state and is described as more efficient than real-time reasoning.
  • C.2 Proactive Triggering: Evaluating trigger-based or proactive-executing agents requires an online-like environment with a simulated user agent that provides acceptance or rejection feedback.
  • Online Evaluation: PersonalAlign restricts proactive behavior to instruction-level intent prediction rather than autonomous action execution.The current setting primarily uses interaction time and scenario, leaving richer signals for future work.
Loading 2601.09636v2…