PaperlayerView exact revision on arXiv

Source-linked AI summary

Act2Intention: A Benchmark For Developing Active Mobile Agents Through Inferring User Intention from GUI Actions

Xiaokai Yan, Jingtao Ding, Yong Li, Zhiwen Yu

arXiv:2608.14132v1cs.HCcs.AI

TL;DR

Existing mobile GUI-agent research mainly addresses reactive task execution, leaving continuous user-intention modeling insufficiently studied. Act2Intention introduces a benchmark and proactive agent for understanding, predicting, and executing intentions, achieving competitive intent-understanding and prediction performance.

  • Problem

    Existing GUI-agent research primarily performs reactive task execution, while continuous intention modeling from action streams remains insufficiently studied.

  • Method

    Act2Intention constructs a continuous intention-action benchmark and develops an agent that understands, predicts, and executes user intentions.

  • Results

    The Act2Intention Agent achieves competitive performance in intent understanding and prediction, supporting the benchmark’s value for training and evaluation.

  • Takeaways & Limitations

    Act2Intention provides a benchmark for evaluating proactive mobile agents through continuous intention-action trajectories.

  • Takeaways & Limitations

    The benchmark covers 52 apps and common usage patterns from 90 participants, limiting coverage of diverse users and unseen apps or interfaces.

Abstract

from arXiv · show

Mobile GUI Agents powered by multimodal large language models (MLLMs) show promise in human-computer intelligence. However, current research primarily focuses on reactive task execution while lacking a comprehensive understanding-prediction-execution process for user intentions, which are the core requirements of active agents. In this paper, we propose the Act2Intention framework that builds an active mobile agent by integrating understanding, predicting user intentions, and executing decisions. First, we construct the Act2Intention Bench through data collection and validated generation, comprising 72,511 intentions and over 700,000 actions across 52 apps, thereby establishing the first benchmark for evaluating proactive agents via continuous intention-action trajectories. We further develop the Act2Intention Agent, achieving proactive services through Proactive-oriented Intention Understanding, Personalized Proactive Intention Prediction, and Experience-guided Intention Execution. Experimental results show that supervised fine-tuning on Act2Intention Bench yields absolute improvements of +32.0 Acc-S, +10.25 Acc-S, and +6.9 SSR points over non-fine-tuned counterparts under the same agent framework for intention understanding, prediction, and execution, respectively. This success underscores the necessity and value of the Act2Intention Bench, which establishes a standardized platform for developing and evaluating proactive agents and consequently paves the way for research on intention-driven human-computer interaction.

1 INTRODUCTION

Act2Intention addresses the gap between reactive GUI task execution and proactive assistance by inferring user intentions from continuous GUI-action trajectories. It introduces an anticipatory Understanding → Predicting → Executing paradigm, benchmark, and multi-agent framework for user-confirmed mobile assistance.

  • Background: Mobile GUI agents have progressed from rule-based systems to LLM-powered architectures that perceive multimodal observations and autonomously reason and act through GUI interactions.These developments motivate new forms of interaction with mobile and ubiquitous systems.
  • Motivation: Act2Intention targets the practical gap between instruction-dependent reactive GUI agents and users’ often-unspoken intentions requiring foresight and autonomous decision-making.Current GUI research primarily focuses on explicit-instruction task execution, whereas Act2Intention studies the underlying purpose behind action sequences.
  • Benchmark: Act2Intention Bench captures continuous multi-intention and action trajectories across mobile interactions, containing 360 personas, 72,511 intentions, and over 700,000 actions.The benchmark is designed to address existing datasets’ focus on discrete task execution rather than continuous intention-action trajectories.
  • Agent: The Act2Intention Agent uses specialized modules for intent understanding, intent prediction from historical intentions and characteristics, and experience-guided intention execution.The framework analyzes historical GUI actions and supports user-confirmed execution to assist users in accomplishing their goals.
  • Framework: The proposed paradigm understands intentions from atomic action sequences, predicts future intentions from trajectories and personas, and executes user-confirmed intentions through GUI actions.This defines the anticipatory workflow: Understanding → Predicting → Executing.

2 RELATED WORKS

Prior work spans general-purpose GUI agents, mobile GUI datasets, and active agents that infer intentions or identify opportune moments for proactive assistance. Recent GUI-agent studies extend this direction through proactive need inference, task initiation, and benchmark construction.

  • GUI Agents: MLLM-based GUI agents have progressed from rule-based automation toward generalized systems requiring perception, planning, execution, reflection, and memory [48].Some multi-agent frameworks use precisely designed workflows without Supervised Fine-Tuning (SFT) [63].
  • GUI Datasets: Mobile GUI datasets such as AiTW, Android Control, AMEX, GUI Odyssey, and AiTZ provide instructions, environmental information, and task trajectories [21] [5].These components support development of GUI agents by specifying objectives, interface states, and action sequences.
  • Active Agents: Active agents have been studied in dialogue and embodied tasks to recognize long-term goals or potential intentions and assist users without explicit instructions [8] [35].Embodied active bots use observed actions and emotions to support human operations [3] [10] [11].
  • Proactive Interaction Timing: Mobile and ubiquitous-system research predicts opportune moments for proactive interaction using contextual factors including phone-use behavior, user mood, activity, and social presence.One notification model achieved higher success rates by incorporating phone-use behavior, while other studies examined interruptibility in smart-speaker interactions.

3 TASK DEFINITION

Act2Intention defines a user intention as a coherent objective motivating a contiguous GUI-action segment, expressed in natural language and mapped to an executable trajectory. Its benchmark organizes agents into intention understanding, prediction, and execution tasks spanning observed behavior, proactive suggestion, and confirmed-task completion.

  • Task Definition: The benchmark comprises three agent-level tasks: Intention Understanding, Intention Prediction, and Intention Execution.
  • Task Definition: A user intention is a coherent, self-contained objective motivating contiguous GUI actions within one mobile session, described by app, operation, and purpose.Each intention maps directly to a short, executable device-action trajectory, such as ordering a no-ice Starbucks Latte via Meituan.
  • Intention Understanding: Intention Understanding parses intentions from observed action sequences through session segmentation and semantic intention description.
  • Intention Prediction: Intention Prediction proactively proposes the next potential intention from historical intentions, current time, and user persona.
  • Intention Execution: Intention Execution iteratively selects the next device action for a user-confirmed intention using the explicit task, action history, current observation, persona, and similar retrieved trajectories.Retrieved trajectories from the user’s historical intentions provide operating knowledge for decision-making.

4 BENCH CONSTRUCTION

Act2Intention Bench combines real interaction logs with persona-conditioned synthetic trajectories to support intention understanding, prediction, and personalized execution. Its construction uses continuous intention-action sequences, executable GUI trajectories, and dual filtering for persona alignment and action validity.

  • Benchmark Construction: Act2Intention Bench comprises RR real-persona-to-real-trajectory, RG real-persona-to-generated-trajectory, and GG generated-persona-to-generated-trajectory subsets.RR preserves real user trajectories, whereas RG and GG synthesize trajectories from real or generated personas to expand behavioral diversity.
  • Real-World Data: The benchmark is built from consented, anonymized mobile interaction logs containing timestamps, app and activity information, GUI observations, actions, parameters, and participant intentions.The source dataset was collected by a major smartphone manufacturer and underwent anonymization and security screening.
  • Synthetic Trajectory Generation: Behavior-derived personas summarize historical intention trajectories from 90 participants, while LLMs iteratively generate timestamped, app-grounded intention chains conditioned on personas and scenarios.Generated intentions are required to be objective, feasible, and grounded in realistic app operations.
  • Verification and Execution: Synthetic trajectories undergo persona and action verification, filtering sequences below 0.7 cosine similarity and removing incomplete, infeasible, incorrect, or insufficiently evidenced executions.UI-TARS-1.5-7B [37] executes synthesized intentions across parallel Android emulators to collect corresponding GUI action trajectories.
  • Dataset Statistics: The benchmark contains 360 personas, 72,511 intentions, and 700,000+ actions spanning 52 apps.Its schema also includes fine-grained event categories and coarser app-level domains.

5 ACT2INTENTION AGENT

Act2Intention Agent is a modular proactive-agent architecture that converts raw GUI interactions into structured intentions, predicts subsequent needs from user context, and executes accepted suggestions using prior experience. Its pipeline combines VLM-based action description, LLM-based segmentation and intention understanding, background intention prediction, and trajectory-guided GUI execution.

  • Architecture: The modular Act2Intention Agent uses raw GUI interactions to infer intentions, incorporates user persona context to identify latent requirements, and provides proactive assistance when needed.The architecture is organized around understanding, prediction, and execution components.
  • Intention Understanding: Action understanding converts atomic actions into descriptions, segments continuous sessions into intention spans, and interprets each span as a structured intention.VLMs generate descriptions from pre-action and post-action observations, while supervised fine-tuned LLMs perform segmentation and intention understanding in a chained process.
  • Intention Prediction: The agent predicts the next potential intention from historical intentions, user persona, and time information.During deployment, prediction runs in the background and the resulting task suggestion is shown upon device wake-up for user confirmation; interruption timing is outside the benchmark.
  • Intention Execution: After user acceptance, experience-guided execution retrieves similar intention-action trajectories and incorporates them with the user profile into GUI-agent prompts.A GUI agent iteratively performs perception, decision-making, and execution until task completion or the maximum step limit.

6 EXPERIMENTS

Experiments evaluate Act2Intention across intention understanding, prediction, and execution using closed-source models, fine-tuned open-source models, and GUI agents with experience guidance. Fine-tuning substantially improves benchmark-specific understanding and prediction, while autonomous end-to-end completion remains limited by intention prediction and long-horizon execution.

  • Evaluation Metrics: SSR measures the proportion of successfully executed steps using conservative same-step action matching after action-space adaptation.Functionally equivalent GUI paths can count as mismatches, so SSR diagnoses trajectory-level execution alignment rather than all functional success.
  • Intention Understanding: 100% Acc-G and 0.49 Acc-S are achieved by Qwen-2.5-7B-SFT, improving from 12.14% and 0.24, respectively, on the ID understanding test set.Fine-tuned models improve intent segmentation and descriptive capabilities relative to closed-source and non-fine-tuned models.
  • Intention Prediction: 48.5% category accuracy is achieved by Qwen-max and 0.36 Acc-S by Gemini-1.5-pro among closed-source predictors, but fine-tuned open-source models perform better overall.SFT improves prediction across open-source models, while larger non-fine-tuned models can misalign with the benchmark’s predefined category taxonomy.
  • Intention Understanding and Prediction: SFT reverses the semantic-metric trend: without fine-tuning, understanding generally trails prediction in Acc-S and BLEU-4, whereas after SFT, understanding outperforms prediction.The paper attributes this change to the difficulty of segmentation–summarization in intention understanding.
  • Intention Execution: 9.9 and 3.2 SSR-point gains result from adding relevant action trajectories to UI-TARS-2B-SFT and UI-TARS-7B-SFT prompts, respectively, on OOD benchmark data.The guided 2B model reaches SSR comparable to the stronger executor, although both models begin with relatively low step accuracy on OOD data.
  • End-to-End Evaluation: 22.7 end-to-end SR is achieved with Llama-3.1-8B-SFT for understanding/prediction and UI-TARS-7B-SFT as executor, while ground-truth intentions raise UI-TARS-7B-SFT SR to 47.6.The gap indicates that predicted intentions remain inaccurate and long-horizon GUI execution remains difficult.

7 LIMITATIONS AND FUTURE DIRECTIONS

The work establishes the feasibility of the Act2Intention benchmark and framework but leaves important gaps in intention prediction, intervention timing, user-centered evaluation, generalization, and real-world deployment. Future directions include richer memory and context modeling, opportune intervention prediction, user studies, broader data coverage, and addressing privacy and efficiency challenges.

  • Framework scope and core limitations: Act2Intention primarily contributes a benchmark and proactive-agent framework, while its modules largely rely on existing techniques and intention prediction and execution remain open for improvement.The stated goal is to examine framework feasibility and provide an evaluation setting for proactive agents.
  • Intervention timing: The benchmark evaluates anticipatory intention prediction but omits opportune intervention timing, including when an agent should interrupt or engage the user.The current prototype surfaces predicted intentions through predefined system events and requires user confirmation before execution.
  • User-centric evaluation: User-centric evaluation remains absent: quantitative metrics do not establish whether proactive suggestions are useful, timely, non-intrusive, or accepted by real users.Future work plans to conduct in-situ user evaluation, while prior HCI studies identify relevance, timing, and user control as determinants of acceptance.
  • Coverage and generalization: Dataset coverage and generalization are limited because the benchmark spans 52 apps and common usage patterns from 90 anonymous participants without demographic information.Collected behaviors may reflect specific user groups or habits, and synthetic trajectories cannot fully replace real-world data.
  • Deployment challenges: Real-world deployment faces privacy risks from sensitive mobile data and efficiency costs from large-model UI parsing, intention prediction, and GUI execution on devices.Although the collected data were anonymized and desensitized, experiments were conducted in emulator-based settings rather than fully representative real-device deployment.

8 CONCLUSION · A ETHICS STATEMENT · C ACT2INTENTION FRAMEWORK DETAILS

The work centers proactive mobile agents on understanding, predicting, and executing user intentions, supported by Act2Intention Bench and the Act2Intention Agent. The dataset contains 72,511 intentions and over 700,000 actions across 52 apps, while collection and release procedures address participant awareness, privacy, safety, and harmful content.

  • 8 CONCLUSION: Act2Intention Bench contains 72,511 intentions and over 700,000 actions across 52 apps.It was constructed through data collection and validated automated generation.
  • 8 CONCLUSION: The benchmark evaluates proactive agents through continuous intention-action trajectories.
  • 8 CONCLUSION: The proposed agent follows an Understanding → Predicting → Executing process for user intentions.
  • 8 CONCLUSION: The Act2Intention Agent was developed using Act2Intention Bench for proactive mobile-agent research.
  • A ETHICS STATEMENT: Data-collection participants were informed about the study’s intention-inference purpose and potential privacy risks.
  • A ETHICS STATEMENT: The released dataset was processed with safety filters to exclude harmful content and private information.

C.1 Data Collection … D SUPPLEMENTARY EXPERIMENTS

The paper constructs intention-action data through monitored smartphone usage, structured generation, and human realism checks, then operationalizes an agent that understands action sequences and predicts intentions from history. Its procedures combine persona-aware trajectory generation, strict intention segmentation, and confidence-gated proactive execution.

  • C.1 Data Collection: The data collection process records foreground app switches during normal smartphone use and prompts participants to describe their current application intentions.A recording application monitors app-switching events and requests brief intention descriptions when switches occur.
  • C.2 Data Generation: The generation pipeline builds personalized profiles by analyzing historical intention trajectories and merging semantically similar, high-frequency events.The profile-generation procedure takes time, app, intention, and event records as input and organizes related events according to their semantics and frequency.
  • C.2 Data Generation: Generated intention trajectories specify a time, app, and single concrete action while enforcing logical, coherent progression without added reasoning or interpretation.The trajectory-generation instructions require realistic user behavior, connected intentions, and one brief action per intention.
  • C.3 Human Realism Assessment Details: Ten independent annotators assessed 90 sampled trajectories from RR, RG, and GG subsets while remaining blind to each sample’s source subset.The annotators were graduate students or senior undergraduates, and 30 intention segments were sampled from each subset.
  • C.3 Human Realism Assessment Details: Annotators rated intention realism, action achievability, and persona consistency on five-point Likert scales, providing preliminary rather than conclusive plausibility evidence.The three dimensions evaluate plausible goals, whether actions can accomplish intentions, and consistency with behavior-derived personas.
  • Action Description: Action descriptions are generated in strict JSON with a concise app-action-purpose format, while a confidence-based fallback evaluates whether predicted intentions are reliable enough to push.The fallback considers persona, recent history, current context, and the predicted intention before proactive suggestion.
  • Action Description: The agent groups ordered action descriptions into intention segments, retains completed groups in intention memory, and uses recent history to predict the next intention.Its predictor can trigger a proactive notification when the device screen is on, and execution occurs only after user acceptance.

D.1 Error Analysis of Semantic Accuracy under Correct Segmentation · D.2 Confidence Statistics for Fallback · D.3 Robustness Analysis

The analyses show that semantic inaccuracies under correct segmentation mainly reflect over-generalization and omitted details, while fallback evaluation uses confidence thresholds to decide which intentions are executed. No substantive passage describing D.3 Robustness Analysis was supplied.

  • D.1 Error Analysis of Semantic Accuracy under Correct Segmentation: Low-Acc-S cases are dominated by over-generalization and detail omission despite correct intention-boundary segmentation.The analysis examined 100 randomly sampled trajectories with correctly segmented intention boundaries and categorized semantic errors into four types.
  • D.1 Error Analysis of Semantic Accuracy under Correct Segmentation: The semantic-accuracy gap reflects failures to preserve app-specific, object-specific, and contextual details, not only broad semantic misunderstanding.Figure 6 summarizes semantic cosine similarity, Acc-S distributions, and error-category proportions for correctly segmented cases.
  • D.2 Confidence Statistics for Fallback: Fallback pushes only predictions whose confidence exceeds a threshold, while withholding the remainder from execution.The experiment uses Qwen-2.5-7B-SFT† for understanding, Qwen-2.5-7B-SFT‡ for prediction, and UI-TARS-7B-SFT for execution; Push Rate measures the proportion pushed.
  • D.2 Confidence Statistics for Fallback: The fallback experiment evaluates pushed predictions using semantic accuracy and execution success metrics.Acc-Spush and SRpush are reported for predictions selected for execution, while Push Rate is reported as a percentage.
  • D.2 Confidence Statistics for Fallback: The default fallback threshold is θ=0.60, with θ=0.80 used as a stricter test setting.Figure 7 displays the confidence-score distribution and marks the default threshold with a dashed green line; Table 13 reports the confidence-based fallback settings.

D.3.1 Generalization Capability.

The Act2Intention Agent’s SFT model outperforms non-fine-tuned counterparts on both ID and OOD intent understanding and prediction, but exhibits substantial OOD degradation. Generalization was also examined across varying intention and trajectory lengths.

  • Generalization evaluation: SFT-trained models outperform corresponding non-fine-tuned models on both ID and OOD sets, but Llama-3.1-8B-SFT loses 12.73 Acc-G and 0.15 Acc-S in understanding and 10.65 Acc-C and 0.05 Acc-S in prediction OOD evaluation.These results come from the generalization evaluation summarized in Table 14.
  • Generalization evaluation: SFT can reduce OOD generalization relative to non-fine-tuned and open-source models, with Deepseek-7B-SFT dropping 11.37 Acc-G versus 0.25 for Deepseek-7B in understanding.The passage also notes that Qwen-2.5-7B-SFT has the lightest degradation in the reported comparison, although the supplied text is truncated before its value.
  • Length generalization: The study additionally evaluated intention understanding across lengths 1, 2, 3, 5, and 10, and prediction across trajectory lengths 10, 20, 30, 40, and 50.LLaMA-3.1-8B and Qwen-2.5-7B were fine-tuned using sequence length 5 for understanding and trajectory length 50 for prediction.

D.3.2 Impact of Intention Length.

Intention length affects understanding and prediction differently: understanding remains highly accurate for lengths 1–5 but declines at length 10, whereas prediction improves as histories approach the training length of 50. Longer prediction histories also improve category accuracy and BLEU-4 across both fine-tuned models.

  • Intention Understanding: At sequence lengths 1–5, both fine-tuned models maintain at least 95% understanding accuracy, but group accuracy falls to 91.94% at length 10.The decline is attributed to fine-tuning on length-5 sequences, with overly long inputs potentially exceeding attention and information-integration capacity.
  • Intention Prediction: From lengths 10 to 40, LLaMA-3.1-8B-SFT improves from 48.6 to 50.6 Acc-C and from 32.4 to 32.9 BLEU-4, while Qwen-2.5-7B-SFT improves from 45.2 to 47.3 Acc-C and from 28.5 to 31.8 BLEU-4.Both models show a consistent upward trend as input length increases toward the length-50 sequences used during training.

D.3.3 Impact of Training Data. · D.3.4 Impact of Persona. · E CASE STUDY

The ablations show that combining real and generated training data produces the strongest overall performance, while persona information mainly improves categorical intention understanding rather than fine-grained semantic prediction. A case study illustrates how action-purpose descriptions from before-and-after screenshots support intention inference over action sequences.

  • D.3.3 Impact of Training Data.: For intention understanding, GG reaches Acc-G 95.7 but Acc-S 0.35, respectively the highest and lowest values among RR, RG, and GG.The larger synthetic dataset supports better intention grouping, whereas purely synthetic data lacks complete semantic information.
  • D.3.3 Impact of Training Data.: All four training sets contribute positively to model performance, with their combination yielding the best results.The compared sets are RR, RG, GG, and RR+RG+GG.
  • D.3.4 Impact of Persona.: Adding persona improves every evaluated model’s ability to identify the correct intention type, as measured by category accuracy.The ablation compares Qwen-max, Llama-3.1-8B-SFT, Deepseek-7B-SFT, Mistral-7B-SFT, and Qwen-2.5-7B-SFT with and without persona; the SFT models were trained without persona information.
  • D.3.3 Impact of Training Data.: “RR+RG+GG” achieves the highest overall performance, indicating that real and generated data are complementary for model generalization.The experiment fine-tunes Qwen-2.5-7B on four training sets for intention understanding and prediction.
  • D.3.4 Impact of Persona.: Persona mainly supports categorical user-goal understanding, influencing high-level preferences more than detailed semantic information.For example, persona can indicate a preference to open shopping apps but not necessarily a preference to buy women’s jeans.
  • E CASE STUDY: The case study generates natural-language descriptions of each action and its purpose by analyzing screenshots before and after execution.These action descriptions enable the agent to infer underlying intentions from the resulting action sequence.
Loading 2608.14132v1…