PaperlayerView exact revision on arXiv

Source-linked AI summary

AndroTMem: From Interaction Trajectories to Anchored Memory in Long-Horizon GUI Agents

Yibo Shi, Jungang Li, Linghao Zhang, Zihao Dongfang, Biao Wu, Sicheng Tao, Yibo Yan, Chenxi Qin, Weiting Liu, Zhixin Lin, Hanqian Li, Yu Huang, Song Dai, Yonghua Hei, Yue Ding, Xiang Li, Shikang Wang, Chengdong Xu, Jingqi Liu, Xueying Ma, Zhiwen Zheng, Xiaofei Zhang, Bincheng Wang, Nichen Yang, Jie Wu, Lihua Tian, Chen Li, Xuming Hu

arXiv:2603.18429v1cs.CV

TL;DR

Long-horizon GUI agents need memory that preserves dependency-critical intermediate states, yet existing replay and summary approaches do not adequately support this requirement. AndroTMem introduces a causal-anchor benchmark and ASM, finding that anchored memory improves performance across GUI agents while leaving cross-session and dynamic-environment evaluation as an open boundary.

  • Problem

    Long-horizon GUI benchmarks and history mechanisms inadequately evaluate or preserve sparse intermediate states required by later dependent actions.

  • Method

    AndroTMem combines a dependency-critical Android GUI benchmark with diagnostic evaluation and Anchored State Memory, which organizes trajectories into causally linked intermediate-state anchors.

  • Results

    Across multiple settings and 12 GUI agents, ASM consistently outperforms full-sequence replay and summary-based baselines, improving TCR by 5%–30.16% and AMS by 4.93%–24.66%.

  • Takeaways & Limitations

    Anchored, structured memory mitigates the interaction-memory bottleneck and supports more reliable long-horizon GUI-agent execution.

  • Takeaways & Limitations

    The framework does not fully capture cross-session tasks and uses curated trajectories with fixed goals and relatively stable app states.

Abstract

from arXiv · show

Long-horizon GUI agents are a key step toward real-world deployment, yet effective interaction memory under prevailing paradigms remains under-explored. Replaying full interaction sequences is redundant and amplifies noise, while summaries often erase dependency-critical information and traceability. We present AndroTMem, a diagnostic framework for anchored memory in long-horizon Android GUI agents. Its core benchmark, AndroTMem-Bench, comprises 1,069 tasks with 34,473 interaction steps (avg. 32.1 per task, max. 65). We evaluate agents with TCR (Task Complete Rate), focusing on tasks whose completion requires carrying forward critical intermediate state; AndroTMem-Bench is designed to enforce strong step-to-step causal dependencies, making sparse yet essential intermediate states decisive for downstream actions and centering interaction memory in evaluation. Across open- and closed-source GUI agents, we observe a consistent pattern: as interaction sequences grow longer, performance drops are driven mainly by within-task memory failures, not isolated perception errors or local action mistakes. Guided by this diagnosis, we propose Anchored State Memory (ASM), which represents interaction sequences as a compact set of causally linked intermediate-state anchors to enable subgoal-targeted retrieval and attribution-aware decision making. Across multiple settings and 12 evaluated GUI agents, ASM consistently outperforms full-sequence replay and summary-based baselines, improving TCR by 5%-30.16% and AMS by 4.93%-24.66%, indicating that anchored, structured memory effectively mitigates the interaction-memory bottleneck in long-horizon GUI tasks. The code, benchmark, and related resources are publicly available at [https://github.com/CVC2233/AndroTMem](https://github.com/CVC2233/AndroTMem).

1. Introduction

Long-horizon GUI agents must preserve and reuse sparse intermediate states, but existing benchmarks and history representations inadequately expose memory failures. AndroTMem addresses this gap with a dependency-critical benchmark, diagnostic evaluation, and Anchored State Memory.

  • Motivation: Long-horizon GUI tasks require carrying forward intermediate results across many steps, not merely perceiving the current interface or selecting the next action.Realistic workflows include extracting values, verifying prerequisites, handling exceptions, and reusing earlier results later.
  • Motivation: Existing GUI-agent benchmarks often emphasize short or weakly coupled tasks, allowing later decisions to succeed without faithfully reusing earlier information.This obscures agents’ long-horizon memory ability.
  • Motivation: Full-trace replay introduces redundant UI transitions, while summaries can omit dependency-critical intermediate states and traceability.These limitations motivate structured history representations centered on task-relevant states.
  • Contributions: AndroTMem-Bench contains 1,069 tasks and 34,473 interaction steps, averaging 32.1 steps per task with a maximum of 65.The benchmark spans 50 applications and enforces strong step-to-step causal dependencies, making sparse intermediate states decisive for downstream actions.
  • Contributions: Across open- and closed-source GUI agents, longer interaction sequences primarily expose within-task memory failures rather than isolated perception or local action errors.AndroTMem evaluates this behavior with Task Complete Rate on tasks requiring critical intermediate state to be carried forward.
  • Contributions: ASM organizes history into causally linked intermediate-state anchors for targeted retrieval and improves TCR by 5%–30.16% and AMS by 4.93%–24.66%.The comparison is against full-sequence replay and summary-based baselines across multiple settings and 12 GUI agents.

2. Related Work

Prior GUI-agent research has improved perception, grounding, and step-level action prediction, but long-horizon execution remains difficult when later actions depend on earlier intermediate results. AndroTMem targets this gap by explicitly diagnosing interaction memory and structuring history around causal state anchors.

  • Prior GUI agents: GUI agents have advanced from single-screen operations toward longer, more realistic multi-step tasks across web and mobile applications.Recent work also increasingly uses multimodal and pure-vision observations.
  • Memory in GUI agents: Long-horizon workflows remain challenging because later decisions can depend on intermediate values, completed subgoals, or environment changes obtained several steps earlier.Effective history utilization and intermediate-state management are therefore central to execution.
  • Memory in GUI agents: Existing systems commonly use raw traces, compressed summaries, or generic context aggregation, without explicitly preserving and retrieving sparse causally critical states.Raw histories can become unwieldy, while compressed representations may omit information needed later.
  • AndroTMem’s position: AndroTMem differs by enforcing strong cross-step causal dependencies to systematically evaluate how agents preserve and reuse intermediate states.It also introduces ASM, which organizes trajectories around causally linked state anchors.

3. Dataset Construction and Statistics

AndroTMem-Bench constructs intent-driven, cross-app tasks with explicit causal dependencies and sparse state-anchor annotations, using a semi-automatic pipeline with quality control. The resulting dataset contains 1,069 diverse mobile GUI tasks averaging 32.1 steps.

  • 3.1. Long-Horizon Task Formulation: AndroTMem tasks are constructed around substantial cross-step causal dependencies, with sparse State Anchors marking task-relevant state changes or intermediate outcomes that constrain later steps.These states are not recoverable solely from the initial instruction or current GUI state.
  • 3.1. Long-Horizon Task Formulation: Each task receives one primary intent, with eight classes including Lookup, Compare & Decide, Purchase / Order, and Communicate / Coordinate.Intent classification captures differences in interaction patterns and dependencies that coarse application-domain labels miss.
  • 3.2. Data Pipeline: A semi-automatic pipeline combines expert-specified intent-driven tasks with automated closed-loop execution, synchronized UI-state collection, and structured annotations.This design aims to produce causally grounded, fine-grained annotations more cheaply than full post-hoc re-annotation.
  • 3.2. Data Pipeline: The benchmark pipeline collects and groups mobile apps, generates dependency-aware cross-app instructions, then executes and annotates tasks on Android devices or emulators.The overview identifies these as the three main construction stages.
  • 3.3. Dataset Statistics: The dataset includes diverse cross-app workflows and reports app combinations, task-type step lengths, trajectory lengths, app frequencies, action proportions, and per-task action diversity.These statistics are summarized in the benchmark overview.
  • 3.3. Dataset Statistics: AndroTMem-Bench contains 1,069 tasks, 34,473 interaction steps, 50 mobile applications, an average of 32.1 steps per task, and a maximum of 65 steps.It provides raw trajectories plus reasoning traces, summaries, and step-aligned sparse state anchors.

4. AndroTMem-Bench

AndroTMem-Bench evaluates long-horizon GUI agents on causally dependent Android tasks using step-level action accuracy and anchor-based task completion. Results show substantial difficulty across agents and intents, with performance degradation primarily reflecting failures to preserve and reuse task-critical intermediate states.

  • Benchmark setup: AndroTMem-Bench evaluates out-of-the-box GUI agents on long-horizon, cross-app tasks with strong step-to-step dependencies.The benchmark uses TCR for anchor-defined task completion and AMS for step-level action accuracy.
  • Overall benchmark results: All evaluated agents show relatively low absolute performance, indicating that long-horizon Android GUI tasks remain highly challenging.Among closed-source models, Gemini-3-Flash leads with 46.14% AMS and 55.21% TCR; UI-TARS-1.5-7B leads open-source agents with 35.62% AMS and 34.55% TCR.
  • Performance by task type: Performance varies substantially across intents, with non-local state reuse and cross-app coordination particularly challenging.Compare & Decide tasks require gathering and reconciling information across multiple sources, while the broader pattern implicates preservation and reuse of intermediate information.
  • Evaluation metrics: AMS measures action accuracy, while TCR measures completion only when agents reach the final anchor and satisfy preceding causal dependencies.The two metrics distinguish local action correctness from preservation and reuse of task-critical intermediate states.
  • Diagnosis: Performance consistently degrades as interaction sequences lengthen, and the primary bottleneck is representing and retrieving task-critical intermediate states.Failures often arise when agents must reuse earlier prices, selected items, or contacts; raw trajectories and coarse summaries may not preserve these states as histories grow.
  • Memory design implication: ASM organizes past interactions around causally linked state anchors to support targeted retrieval and attribution-aware decision making.The design follows the diagnosis that accurate perception and local action prediction alone are insufficient for long-horizon interaction.

5. Anchored State Memory

Anchored State Memory (ASM) represents long-horizon interaction history as sparse, causally linked intermediate states, addressing the dilution and omission problems of raw traces and coarse summaries. Across history ablations, ASM improves robustness, efficiency, and task performance while targeting recurring state-management failures.

  • Motivation: Raw traces dilute task-relevant states as horizons grow, whereas coarse summaries can omit dependency-critical intermediate information.Both representations degrade significantly with longer interaction horizons, motivating explicit modeling of task-relevant states and causal roles.
  • ASM Representation: ASM stores decision-critical intermediate states and their causal relations instead of replaying full trajectories or using free-form summaries.Anchors may represent extracted values, identified entities, completed subgoals, persistent changes, or exceptions, with evidence and dependency links.
  • ASM Operation: ASM retrieves relevant anchors for the current UI state and instruction, uses them with current context to predict an action, then updates the memory bank.This retrieve–reason–update process supports non-local state reuse without relying on extremely long raw histories.
  • Evaluation Setup: ASM is compared with Raw History and Coarse Summary in a history ablation using automatically generated representations under shared rules.The ablation isolates differences in history representation and utilization rather than manually controlling information strength.
  • Results: ASM consistently achieves the best AMS and TCR across closed- and open-source agents, while maintaining higher AMS as interaction step ranges increase.It also improves performance across task categories, especially those requiring cross-step reasoning or cross-app coordination.
  • Results: ASM improves AMS and TCR while keeping token usage and inference time comparable to summary history and far below raw trajectory replay.The resulting efficiency–effectiveness trade-off complements its performance gains over both baselines.
  • Failure Modes: Long-horizon failures include state loss, state mis-binding, context drift, unverified progress, and interruption handling failure.ASM addresses these patterns by organizing task-relevant intermediate states into structured anchors with causal relations for more stable decisions.

6. Limitations

AndroTMem isolates within-task interaction memory in long-horizon Android trajectories but does not yet cover cross-session persistence or highly dynamic environments.

  • The framework does not fully capture cross-session tasks requiring state persistence across interruptions, days-long gaps, or multi-episode workflows.
  • The benchmark uses curated trajectories with fixed goals and relatively stable app states.Future benchmarks should expose UI drift, content updates, stochastic outcomes, and evolving external conditions.

7. Conclusion

AndroTMem-Bench studies long-horizon mobile GUI tasks with cross-app workflows and complex dependency structures, while ASM organizes their histories around sparse causal state anchors. The paper concludes that this approach improves action accuracy, task completion, and robustness in long-horizon scenarios.

  • AndroTMem-Bench targets long-horizon mobile GUI tasks with cross-app workflows and complex dependency structures.
  • ASM organizes interaction trajectories into sparse intermediate-state anchors and their causal relations.
  • Experiments across diverse GUI agents show improved action accuracy, task completion, and robustness in long-horizon scenarios with ASM.
  • The analysis identifies common history-related failure modes and shows that structured intermediate-state memory mitigates them.

A.2. Task Types

AndroTMem-Bench categorizes long-chain tasks by primary user intent and supports precise interaction annotation through a unified Android action space. Its data collection protects privacy through anonymization.

  • Each long-chain task is mapped to one primary intent representing the sequence’s core objective, rather than categorized by application domain.
  • The benchmark defines a parameterized action space covering low-level touch interactions and high-level Android system commands.
  • Task categories and representative examples are documented according to simulated daily operations and primary user intent.
  • Personally identifiable information is anonymized by removing or replacing addresses, usernames, phone numbers, contact lists, and payment details.

A.5. Task Template Examples

The appendix describes templated cross-app task construction, semi-automatic trajectory annotation, and ASM’s causal anchor taxonomy and generation protocol. It also specifies fairness controls for comparing raw history, summaries, and anchors.

  • A.5. Task Template Examples: Task instructions use templates with predefined application-group and contact slots to preserve structural consistency while maintaining diversity.An example compares products across shopping apps, adds the higher-priced item to a cart, and shares it through a social app.
  • A.6. Annotation Platform: The annotation platform uses Android Debug Bridge to capture screens, execute labeled actions, and support fine-grained trajectory collection on devices or emulators.It also records UI XML and configurable auxiliary annotations alongside screenshots and actions.
  • B.1. State Anchor Categories: ASM represents history with six anchor types covering subgoal completion, state changes, causal dependencies, exceptions, global context, and task completion.These anchors capture progress, execution context, prerequisites, disruptions, persistent constraints, and terminal success conditions.
  • B.1. State Anchor Categories: Causal dependency anchors make prerequisites explicit for downstream validation, such as copied content needed later or filters affecting subsequent results.
  • B.2. Generation Protocol for Summary and ASM: Raw history, summaries, and ASM representations are generated automatically from the same trajectory using a model-agnostic rule-based protocol.
  • B.2. Generation Protocol for Summary and ASM: Anchors are generated online from current observations, prior anchors, and interaction history, with optional causal links to earlier anchors.
  • B.2. Generation Protocol for Summary and ASM: Unified prompting, schema enforcement, and retries reduce the influence of model-specific output style on anchor generation.
  • B.3. Fairness of History Representation: The comparison is designed to avoid implicit supervision differences by using common trajectories and prompting across history representations.

C. Experiment Details

The experiment uses structured prompts to represent both compact history summaries and Anchored State Memory for long-duration GUI reasoning. ASM prompts emphasize semantically significant transitions that affect long-term planning, with detailed structures specified separately.

  • Structured prompts support both high-level history summarization and Anchored State Memory during long-duration GUI tasks.The prompts define the action space, normalized coordinate system, and core interaction decision principles.
  • ASM records semantically significant transitions, including subgoal achievements and task dependencies, that influence long-term planning.

C.2. Evaluation Protocol and Success Criteria

AndroTMem-Bench evaluates success through dependency-critical intermediate anchors rather than final UI state alone. TCR requires reaching the final anchor while preserving the causal dependencies established by preceding anchors.

  • Task Success Definition: Task success requires reaching the final anchor while preserving required causal dependencies among preceding task-relevant anchors.Each task contains a sparse sequence of intermediate state anchors and a final completion anchor.
  • Dependency Satisfaction: Dependency satisfaction is judged by whether downstream behavior matches the intermediate state required by the task.Evaluation combines annotated anchor semantics with the observed action trajectory and UI state.
  • Matching and Tolerance: Task-critical anchors use exact semantic matching for discrete states and task-dependent rules for structured values.Examples include page transitions, subgoal completion, item selection, prices, contact names, and message recipients.
  • Handling UI Drift and Interruptions: Interruption-related UI states are included in evaluation rather than treated as ignorable noise.The prerecorded trajectories may include advertisements, permission dialogs, loading screens, and other unexpected states.
  • Task-Level Metrics: TCR measures the proportion of tasks reaching the final anchor while satisfying dependency constraints from earlier anchors.By incorporating intermediate-state consistency, TCR tests whether agents preserve and reuse critical information across long horizons.

C.3. Agent Implementation and Model Behavior

Agents operate in a shared execution framework with constrained actions and standardized inputs, while their behavior is examined under long-horizon interaction demands. The implementation uses structured prompts and anchored state representations to organize durable, causally relevant history.

  • Unified Agent Framework: Each model receives the current UI observation, user instruction, and selected history representation before producing one structured action.The shared framework compares raw trajectories, summaries, and ASM under the same execution process.
  • Action-Schema Alignment: Normalized coordinates, fixed action types, JSON schemas, and retries reduce failures caused by formatting or interface mismatches.
  • Long-Horizon Interaction Difficulty: Long-horizon tasks commonly span 30–60 steps and require cross-application dependencies and reuse of earlier intermediate information.Observed difficulties include recalling previously seen values, selected entities, or copied content.
  • Failure Patterns Observed in Large Models: Large models often lose extracted values, misbind entities across applications, or drift from intended subgoals after multiple steps.These failures can occur even when individual perception or reasoning steps appear correct.
  • Interpretation of Absolute Performance: AndroTMem-Bench exposes a gap between strong multimodal reasoning and reliable long-horizon interaction memory under strict action-execution constraints.The benchmark is designed to reveal this interaction-memory difficulty rather than simply reflect model capability in other settings.
  • Anchored State Memory: Anchors capture durable task-relevant events and organize history as a structured dependency graph instead of a flat event list.The generation guidelines require concise semantic content, explanations of downstream relevance, and links only for decision-critical dependencies.
  • Anchor Categories: The anchor taxonomy includes subgoals, state changes, dependencies, exceptions, contextual information, and final completion.
  • Causal Link Types: Causal links encode prerequisite, enables, result_of, and blocks relations between anchors.
Loading 2603.18429v1…