PaperlayerView exact revision on arXiv

Source-linked AI summary

Reuse, Rewrite, or Reject: Evidence-Constrained Trace Reuse for Repeated Mobile GUI Control

Bingyu Bai, Cheng Zhang, Erhu Feng

arXiv:2610.31712v1cs.SEcs.AIcs.LG

TL;DR

Repeated mobile GUI tasks offer opportunities to reduce redundant model inference, but changed instructions and interfaces can invalidate previously successful actions. The paper reuses, rewrites, or rejects individual transitions through graph–text templates and live checks. Evaluations report faster execution on AndroidWorld and real devices, alongside substantial real-device action reuse.

  • Problem

    Mobile GUI agents repeat similar tasks, but changes in instructions or execution environments can make prior actions inappropriate, so the challenge is sustaining reuse under changing conditions.

    Source: §1 · p. 1

  • Method

    A record–template–replay framework stores verified executions as graph–text procedures and uses current instructions and live UI evidence to reuse, rewrite, or reject individual transitions.

    Source: §1 · p. 3

  • Results

    1.2–1.9× execution-time speedup on AndroidWorld and 70.7–79.8% action reuse with 1.8–3.9× execution-time speedup in real-device evaluations.

    Source: §6 · pp. 9–10

  • Takeaways & Limitations

    Selective trace reuse can reduce repeated model inference and execution time in the evaluated settings while retaining applicable transitions and checking reused actions against current conditions.

    Source: §4.3 · p. 9

    Reuse depends on fallback coverage and UI observability; novel free-form content, inaccessible metadata, rapid visual changes, and unsupported workflows limit it, while combined variations and cross-system trajectory transfer need further study.

    Source: §5 · p. 9

Abstract

from arXiv · show

Mobile GUI agents frequently perform repetitive tasks with similar interaction procedures. Reusing previous executions can reduce redundant LLM inference and execution overhead, but variations in user instructions and execution environments may invalidate previously successful actions. We systematically characterize these variations into six representative categories and observe that such changes do not necessarily invalidate entire procedures: individual transitions can remain reusable when validated against the current task context and interface state. Based on this insight, we propose graph-text templates for selective execution-trace reuse. The graph captures UI states and action transitions, while the text encodes task intent, procedural structure, and typed slots for variable content. A record-template-replay framework constructs templates from verified executions and leverages the current instruction and live UI state to reuse, adapt, or discard individual transitions. Given an execution trace, our framework reduces LLM inference token consumption by approximately 80% on average across five base agents on AndroidWorld, with most agents achieving at least 1.5-fold execution speedup over LLM-only execution. In real-device evaluations on Android and HarmonyOS, our framework achieves action reuse ratios of 70.7-79.8% while maintaining over 2-fold execution speedup in most evaluated configurations.

1 INTRODUCTION

Repeated mobile GUI tasks could reduce redundant model calls, but changes in instructions and interface conditions can make previously successful actions inappropriate. The paper proposes evidence-constrained reuse of individual transitions and evaluates it on AndroidWorld and real devices.

1 INTRODUCTION

  • Repeated familiar tasks create opportunities to reduce redundant LLM calls and user waiting time, but the benefit must hold as task conditions change.An agent that invokes an LLM before every action continues to pay for decisions it has already made successfully.Source: §1 · p. 1
  • Six variation categories cover background-state, layout/workflow, and modal changes, plus parameter, entity, and goal-expression changes.The categories guide analysis of when historical steps can be retained, adapted, or abandoned.Source: §1 · p. 1
  • The record–template–replay framework stores checked action transitions, represents procedures as graph–text templates with typed slots, and reuses, rewrites, or rejects steps using the current instruction and live interface.Admitted transitions receive pre- and post-checks; insufficient evidence or failed checks return control to the base agent.Source: §1 · p. 2
  • Across five base agents on AndroidWorld, verified records yield 82.0–87.5% executed reuse and 1.2–1.9× execution-time speedup.The benchmark supports parameter and initial-state variation, while the paper notes that it alone cannot establish performance across real-use conditions.Source: §1 · p. 2
  • 1.8–3.9× execution-time speedup across real-device conditions accompanies studies of repeated execution under environment and instruction changes.The framework is evaluated on four phone models spanning Android and HarmonyOS.Source: §1 · pp. 2–3
  • The contributions combine the variation analysis, selective graph–text replay, and AndroidWorld and four-phone evaluation with targeted ablations.The ablations examine adaptation to state, parameter, and wording changes.Source: §1 · p. 3

2 RELATED WORK

Prior mobile GUI work addresses automation, efficient execution, memory and skill learning, trace replay, and action verification through complementary approaches. The paper focuses its comparison on evidence for admitting or rewriting transitions under changed intent and state, and on the cost of obtaining that evidence.

2 RELATED WORK

  • Mobile automation and efficiency work uses app knowledge, demonstrations, perception-action modeling, compact policies, overlapping inference with exploration, and app graphs.The approaches address model and planning costs through complementary mechanisms (Wen et al., 2024; Zhang et al., 2025b; Qin et al., 2025; Wen et al., 2025; Yang et al., 2025; Huang et al., 2026; Chai et al., 2026).Source: §2 · p. 3
  • Memory and skill systems organize workflows, experience, demonstrations, hierarchical procedures, executable skills, or evolving persistent procedures.Examples include Agent Workflow Memory (Wang et al., 2025b), SkillWeaver (Zheng et al., 2025), and WebXSkill (Wang et al., 2026).Source: §2 · p. 3
  • GUI trace-replay work already includes selective replay, parameter adaptation, checks, and fallback, while the comparison here concerns evidence under changed intent and state and its acquisition cost.Examples include MobiMem (Liu et al., 2025), AgentRR (Feng et al., 2025), and SkillDroid (Chen et al., 2026b).Source: §2 · p. 3
  • Action-verification systems check intent, postconditions, or pre/postconditions and provide context for comparing pre-reuse checks and post-action verification.Examples include VeriSafe Agent, SenseAct, and ReUseIt (Liu et al., 2026c).Source: §2 · p. 3
  • The proposed comparison centers on evidence-constrained transition reuse under changing instructions and UI states, with reuse, rewriting, rejection, and fallback decisions.The graph–text contracts select among these decisions using current conditions.Source: §2 · p. 4

3 METHOD

The method separates experience formation from use: verified executions become graph–text procedures, and replay admits one transition at a time under the current instruction and UI evidence. Each step is reused, rewritten, or rejected, with failed checks returning control to the fallback policy.

3 METHOD

  • Environment and instruction changes can invalidate individual transitions without invalidating an entire trajectory.Background-state and TOCTOU changes suppress duplicate or inverse actions; layout drift may require fallback, and blocking dialogs remain blocking unless separately verified as benign. Changed values are rewritten only when explicitly supplied in a matching typed slot; entity changes require re-grounding.Source: §3 · p. 4
  • Recording stores each action with its target, procedure step, typed value, and observable before-and-after states and resulting progress or completion signal.The memory unit is a transition with a checked outcome.Source: §3 · p. 4
  • Template extraction merges compatible states and actions, keeps conflicting states as separate branches, and represents variable values as typed slots while preserving stable navigation as reusable edges.Transitions lacking evidence, an identifiable target, or a successful terminal outcome are excluded from the released template.Source: §3 · p. 5
  • The transition contract pairs source state, action, target state, and evidence with procedure intent, typed slots, and selectors.The graph represents concrete UI change; the text represents intended progress.Source: §3 · p. 5
  • Replay requires agreement with the current instruction and live support for source-state conditions and the target; a changed value can be rewritten only when explicitly bound to a compatible typed slot.If the required value or target cannot be established, the controller rejects the step even when navigation context matches; high-risk entity/value pairs require stronger evidence.Source: §3 · p. 5
  • Each admitted edge receives a refreshed pre-check, one execution, and a semantic post-check; a failed post-check blocks that edge for the episode and returns control to fallback.A page change alone is not sufficient evidence for every action.Source: §3 · p. 5
  • In the milk-tea example, valid navigation is reused, quantity is rebound to 2 and checked, and adding the item is separately verified against the requested item and quantity.If the quantity control cannot be grounded, the transition is rejected and control returns to the base agent.Source: §3 · p. 6

4 EXPERIMENTS

The experiments assess repeated execution on AndroidWorld and real phones, measuring success and efficiency as environments and instructions change. The full controller improves success over replay alone across tested real-device conditions, while the evaluation also identifies overheads and comparison limits.

4.1 OVERALL RESULTS ON ANDROIDWORLD AND REAL-WORLD TASKS

  • AndroidWorld uses successful LLM-only executions to construct each of five agents’ templates, while record–replay task pairs test reuse across task details or parameter values.Its original suite has 116 tasks across 20 Android applications, with parameterized initialization and programmatic success checks.Source: §4.1 · p. 6
  • Success rate combines base success with replay-and-fallback success, while efficiency comparisons require both systems to succeed on the same task instance with matched initial states and criteria.Failed executions remain in the task-success denominator but are excluded from efficiency comparisons; token reductions include visual tokens and model-based checking or recovery.Source: §4.1 · pp. 6–7
  • AndroidWorld results span 19.7–67.8% success, 82.0–87.5% executed reuse, and 1.2–1.9× speedup; real-device conditions span 40.9–58.8% success and 1.8–3.9× speedup.Token consumption and LLM calls decrease across all reported settings.Source: Table 1 · p. 7, §4.1 · p. 7

4.2 REAL-WORLD EVALUATION AND ABLATIONS

  • AndroidWorld preparation and post-action state acquisition can account for 40.02% of execution time, limiting the speedup available from avoiding model calls.Real-device evaluation optimizes UI-structure acquisition and reduces its frequency.Source: §4.2 · p. 8
  • For parameter changes, the full configuration achieves 93.0–98.0% success, a 42.0–50.0 percentage-point gain over replay only.Parameter generalization and instruction–outcome agreement support value binding and verification.Source: §4.2 · p. 8
  • For paraphrased instructions, the full configuration reaches 92.0–97.0% success, a 35.0–46.0 percentage-point gain over replay only.App-level semantics and soft task matching support procedure retrieval.Source: §4.2 · p. 8
  • Direct comparisons with closely related reuse systems are constrained by unavailable official implementations and the lack of established evaluation pipelines matching this record–live protocol.The authors also note that PreAct’s controlled AndroidWorld replay experiments use a 15-task subset.Source: §4.2 · p. 8
  • Across all five agents and three real-device conditions, the full controller improves success over replay only; the figure compares cumulative component additions with PreAct.The evaluated conditions are background-state changes, parameter changes, and paraphrased instructions.Source: §4.2 · p. 8, Figure 4 · p. 9

5 LIMITATIONS AND BROADER IMPACT

Reuse is bounded by fallback coverage and UI observability, and the real-device evaluation covers selected changes rather than combined variations or long-term adaptation. Reusing private actions also carries residual risks despite safeguards.

5 LIMITATIONS AND BROADER IMPACT

  • Reuse is limited by fallback coverage and UI observability, including novel free-form content, inaccessible metadata, rapid visual changes, and unsupported workflows.These conditions can limit reuse.Source: §5 · p. 9
  • The evaluation covers five AndroidWorld base agents and selected real-device changes; combined variations, cross-system trajectory transfer, and long-term adaptation remain for further study.Source: §5 · p. 9
  • Execution-time ratios omit the amortized cost of recording and template construction.Source: §5 · p. 9
  • Reusing private actions can expose stale or sensitive choices; safeguards mitigate but do not eliminate this risk, and learned preferences are not authorization.The described safeguards include single-step admission, high-risk slot checks, post-state validation, fallback, audit logs, and permission boundaries.Source: §5 · p. 9

6 CONCLUSION

The paper presents selective reuse of verified executions as a way to accelerate repeated mobile GUI tasks under changing conditions. It reports speedups on AndroidWorld and real devices spanning Android and HarmonyOS.

6 CONCLUSION

  • 1.2–1.9× execution-time speedup is reported across five base agents on AndroidWorld given a verified execution record.Source: §6 · p. 9
  • 70.7–79.8% action reuse ratios and 1.8–3.9× execution-time speedup are reported in real-device evaluations on Android and HarmonyOS.Source: §6 · pp. 9–10
  • The findings support selective reuse as a means of translating past execution into fewer repeated model decisions and faster execution under changing task conditions.Source: §6 · p. 10

AI USE STATEMENT

The authors report using generative AI tools for manuscript-related tasks and state that they reviewed the resulting content and take responsibility for the final manuscript.

AI USE STATEMENT

A IMPLEMENTATION DETAILS

The implementation records verified executions, extracts graph–text templates, and checks candidate transitions against the live instruction and interface. Its subsections define the transition record, template construction, retrieval and grounding, and one-transition admission and recovery.

A.1 RECORD AND TRANSITION SCHEMA

  • A recorded transition associates source and target states, an action, procedure step, typed slots, target, risk, and required pre- and post-action evidence.A failed terminal action is not a successful demonstration, while an explicitly verified prefix may be retained with its truncation boundary.Source: §A.1 · p. 14, Table 3 · p. 14

A.2 TEMPLATE EXTRACTION, MERGING, AND CONFLICT BRANCHES

  • Template extraction preserves stable procedure structure, exposes changeable quantities, times, and entities as typed slots, and merges observations only when their conditions, action semantics, and checked outcomes are compatible.Visual resemblance alone does not justify merging; conflicting conditions remain separate branches.Source: §A.2 · p. 14

A.3 RETRIEVAL, SLOT BINDING, AND LIVE TARGET GROUNDING

  • Retrieval proposes procedures and candidate transitions, but similarity alone does not authorize action; slot values must be bound to their actions and expected outcomes.An ambiguous entity reference does not justify importing a historical value.Source: §A.3 · pp. 14–15
  • Current observations ground candidate actions to controls; displaced controls may remain usable when their semantics and state are supported, while missing or conflicting controls require rejection or fallback.Source: §A.3 · p. 15

A.4 PRECHECK, POSTCHECK, AND FALLBACK

  • Figure 5 presents control flow for admitting and recovering from one transition; the timer example contrasts a recorded 10-minute instruction with a live 15-minute instruction.Source: Figure 5 · p. 15, §A.4 · p. 15

B EXPERIMENTAL SETUP AND EVALUATION PROTOCOL

The protocol evaluates record–live reuse across AndroidWorld and physical devices, using matched task conditions and explicit outcome judgments. It also defines cumulative controller configurations and documents the tested application coverage.

B.1 ANDROIDWORLD AND BASE AGENTS

  • AndroidWorld uses parameterized initialization and programmatic completion checks; successful LLM-only runs provide each agent’s template records, and assisted runs use that agent as fallback.The benchmark contains 116 tasks across 20 applications, but those figures do not specify the evaluated subset for an individual model.Source: §B.1 · p. 15

B.2 PHYSICAL DEVICES AND LANGUAGE SETTINGS

  • Real-device testing spans four phones across Android and HarmonyOS, with multiple languages including Chinese interfaces and task instructions.The study includes 100–110 tasks; English task descriptions are for readability and do not imply English execution or prior translation.Source: §B.2 · p. 16

B.3 RECORD–LIVE COMPARISONS AND OUTCOME JUDGMENT

  • Record–live comparisons vary background state, requested parameters, or paraphrased wording while matching the live instruction, initial state, and completion criterion within each comparison.An observer judges whether the resulting object, value, and final state satisfy the live request without intervening in action selection.Source: §B.3 · p. 16

B.4 CUMULATIVE COMPONENT CONFIGURATIONS

  • C0 uses replay alone, C1 and C2 add cumulative mechanisms without fallback, and only full configuration C3 enables base-model fallback.Each Table 5 row is a separate comparison; additions accumulate within that row.Source: §B.4 · pp. 16–17, Table 5 · p. 17

C.1 APPLICATIONS INCLUDED IN TESTING

  • The real-device application inventory covers communication, media, information access, shopping, travel, and productivity, but does not imply every app was tested on every device or model.AI-assistant apps in the inventory are target interfaces, distinct from the five evaluated base agents.Source: §C.1 · p. 17, Table 6 · p. 17

C.2 REPRESENTATIVE TASK DESCRIPTIONS

The examples show how task descriptions preserve relevant constraints while neutralizing identifying details, and how compatible task paths can share individually checked transitions. They distinguish illustrative variation examples from the three quantitatively studied conditions.

C.2 REPRESENTATIVE TASK DESCRIPTIONS

  • Representative task descriptions use neutral aliases while retaining task structure, including requested-category constraints and check-in and check-out slots.The examples cover direct search, constrained product selection, date-conditioned retrieval, creator lookup, and search within a store.Source: §C.2 · pp. 17–18, Table 7 · p. 18

C.3 SHARED SUFFIXES AFTER DIFFERENT NAVIGATION PATHS

  • Different navigation paths can reach a compatible state for a shared suffix, but reuse still requires checking the live target and state.For the cart example, an additional-unit request requires an increase of one for the intended product/configuration; a current selection mismatch can invalidate a suffix transition.Source: §C.3 · pp. 18–19

C.4 RELATION TO THE VARIATION TAXONOMY

  • Illustrative examples distinguish background-state changes, parameter changes, and paraphrases; layout/workflow changes, modal interruptions, and entity substitutions complete the six-category taxonomy but are not quantitative conditions.For example, a changed timer duration can leave navigation reusable while requiring the live value for duration entry and its expected effect.Source: §C.4 · p. 19

D METRIC DEFINITIONS AND SUPPLEMENTARY RESULTS

The paper defines success and efficiency measures with distinct evaluation scopes, including task-level reuse aggregation and paired successful executions. It also explains why replay rate does not directly determine token or runtime savings.

D.1 TASK SUCCESS AND EVALUATION SCOPE

  • Task success is verified successful episodes divided by eligible evaluated episodes, with failures and safety terminations counted as failures.AndroidWorld’s reported success product uses base success multiplied by success conditional on records from successful base executions; real-world ablations use the same practical task set across configurations.Source: §D.1 · p. 19

D.2 EXECUTED REUSE AND ZERO-ACTION COMPLETION

  • Executed reuse counts replay-supplied actions without a model call, while correctly stopping at an already-satisfied goal is success-eligible but has no action-based reuse ratio.Reported replay rates average each episode’s replayed-action fraction equally, excluding zero-action episodes from that aggregation.Source: §D.2 · pp. 19–20

D.3 PAIRED COSTS, COVERAGE, AND TIMING BOUNDARIES

  • Efficiency compares matched LLM-only and full-controller executions and includes only pairs where both succeed; reported speedup is the arithmetic mean of per-task time ratios.Timing includes online acquisition, inference, actions, waits, checks, retrieval, binding, and recovery, but excludes setup and record/template construction.Source: §D.3 · p. 20

D.4 DERIVED COST RELATIONSHIPS

  • Derived cost relationships distinguish replay rate, model-call savings, token reduction, and runtime savings, and rely on simplifying assumptions that need not hold in reported runs.Replay rate is therefore not substituted for measured LLM-step or token reduction, and the idealized runtime limit excludes device execution, waits, extra calls, and construction costs.Source: §D.4 · p. 20

D.5 EXECUTION EFFICIENCY

  • Table 8 reports efficiency ranges across five base agents per condition, with speedups referenced to LLM-only execution and averaged over successful pairs.AndroidWorld’s repeated UI-structure acquisition reduces the share of runtime attributable to avoidable inference; the real-device implementation reduces acquisition frequency, so inference savings need not translate proportionally into speedups.Source: §D.5 · p. 21, Table 8 · p. 21

E TASK EXAMPLE PRESENTATION AND DATA PRIVACY

Task examples use stable aliases and neutralized descriptions to preserve relationships and constraints while reducing identifying detail. Those edits apply to textual examples, not to sharing other execution artifacts.

TASK EXAMPLE PRESENTATION AND DATA PRIVACY

  • Stable aliases preserve product, store, creator, and destination relationships while avoiding unnecessary personal identifiers and preference-specific details.Numeric constraints, including the distinction between “one unit” and “one additional unit,” remain explicit.Source: E TASK EXAMPLE PRESENTATION AND DATA… · p. 21
  • Neutral wording and anonymization are presentation choices, not a fairness evaluation.They do not change the reported aggregate results.Source: E TASK EXAMPLE PRESENTATION AND DATA… · p. 21
  • Sharing screenshots, UI trees, action payloads, or logs requires separate review for personal and payment information.Masking must preserve identity and value relationships needed to interpret a decision.Source: E TASK EXAMPLE PRESENTATION AND DATA… · p. 21

Related papers

Loading 2610.31712v1…