Source-linked AI summary
PastForward: Faster On-Device GUI Agents via Computational Experience Reuse
Taehwan Park, Changmin Lee, Hayeon Lee, Taesik Gong
TL;DR
On-device GUI agents can keep sensitive inputs local, but repeated VLM inference is costly, even as token sequences and screen transitions recur across executions. PastForward validates reuse of prior computation, achieving 1.63–2.36× on-device action-step speedups while maintaining task success rates.
Problem
On-device execution keeps sensitive inputs local but shifts costly, repeated VLM inference onto resource-constrained devices, despite recurrence in prior outputs and transitions.
Source: §1 · p. 1Method
PastForward reuses execution experience through validated adaptive token proposals, next-screen-guided early inference, and cross-step KV-state carryover.
Source: §1 · p. 2Results
1.63–2.36× on-device action-step speedups are achieved while maintaining task success rates.
Source: §1 · p. 2, §5.2 · p. 9Takeaways & Limitations
PastForward reduces action-step latency across evaluated VLMs and platforms without task-specific offline exploration or changes to the agent policy.
Source: §8 · p. 13Practical gains may vary by workload: infrequent decoding recurrence, variable resulting screens, or short A2O intervals can reduce reuse opportunities.
Source: §6 · p. 12
Abstract
from arXiv · showhide
Running GUI agents on edge devices can keep sensitive screens and interaction histories local, but the computational cost of inference at every action step makes deployment challenging. Existing GUI agent systems either perform full vision-language model (VLM) inference at each action step or reuse coarse-grained knowledge matched to prior tasks. However, dynamic mobile environments and user tasks make it difficult to fully utilize prior task executions without additional fine-tuning or task-specific offline exploration. To address this challenge, we present PastForward, a system that accelerates GUI agents through validated, fine-grained reuse of computational experience accumulated during ordinary task execution. During decoding, PastForward retrieves prior output sequences as device-adaptive multi-token proposals and verifies them in a single VLM forward pass. Across action steps, it uses prior GUI transitions to begin next-step inference while the device executes the current action, retains the early computation only when the predicted screen matches the observed screen, and carries reusable KV states forward. We evaluate PastForward on AndroidWorld workloads derived from real mobile usage patterns using multiple VLM backbones across server and edge platforms. On device, PastForward achieves action-step latency speedups of 1.63-2.36$\times$ while maintaining task success rates.
1 Introduction
On-device GUI agents can keep sensitive screens and interaction histories local, but repeated VLM inference makes execution costly. PastForward reuses prior computation only when validated against current predictions or observed screens, accelerating action steps while maintaining task success.
1 Introduction
- On a Galaxy S26 Ultra, VLM inference takes 8.57 s per action step and accounts for 77.26% of action-step latency.Inference cost accumulates along the critical path of multi-step tasks.Source: §1 · p. 1
- Prior executions show recurrence in token sequences and GUI transitions, even when no complete task or trajectory repeats.At minimum match length four, prior sequences cover 64.1% of generated tokens and transitions recur in 15.1% of action steps under the user-calibrated workload.Source: §1 · p. 1
- Existing systems reuse coarser app knowledge, subtasks, workflows, or action sequences, while model/runtime techniques do not reuse execution records across task executions.Fine-grained records cannot simply be replayed because outputs or resulting screens may diverge; reuse must preserve the current agent decision.Source: §1 · pp. 1–2
- PastForward reuses execution records as computational experience through adaptive token proposals, next-screen-guided early inference, and cross-step KV-state carryover.Reuse is validated against current VLM predictions or the observed screen; the system requires no offline exploration or additional fine-tuning and preserves the VLM backbone and agent policy.Source: §1 · p. 2
- 1.40–1.68× server and 1.63–2.36× edge action-step speedups are achieved while maintaining task success rates.On the Jetson Orin Nano, peak RAM usage increases by only 0.1% (2.4 MiB).Source: §1 · p. 2
2 Characterization and Motivation
On-device action steps spend most of their time in VLM inference, while token sequences and GUI transitions recur across task executions. PastForward’s opportunity depends on validating that prior computation remains applicable and adapting reuse to device constraints.
2.1 GUI Agent Action-Step Latency
- 2.62 s of prefill and 5.95 s of decoding account for 77.26% of mean action-step latency; the A2O interval is 2.52 s.On the Galaxy S26 Ultra, VLM inference latency has a median of 8.34 s and a 90th percentile of 10.45 s.Source: §2.1 · p. 2
2.2 Reuse Opportunities Across Task Executions
- At Lmin = 4, prior sequences cover 59.6%, 61.4%, and 64.1% of tokens; transitions recur in 11.2%, 16.1%, and 15.1% across the three workloads.A prior (screen, action) match does not guarantee the same next screen: the next screen reappears in 52.3%, 59.2%, and 57.2% of matched cases.Source: Figure 2 · p. 3, §2.2 · p. 3
- The characterization uses live AndroidWorld executions and separates task-once, task-uniform, and user-calibrated workloads; repeated-use instances receive new parameter values.AndroidWorld comprises 20 applications and 116 programmatic tasks.Source: §2.2 · p. 3
2.3 Design Challenges and Implications
- Reuse must capture local recurrence, tolerate divergence from prior executions, and account for runtime overhead and device-specific compute and memory constraints.A retrieved token sequence may diverge, and a previously observed transition may lead to a different screen.Source: §2.3 · p. 3
3 PastForward Design
PastForward reuses computational experience only when validated against current VLM predictions or the observed screen. Its design combines adaptive decoding proposals, early inference from predicted next screens, and KV-state carryover across action steps.
3.1 Overview
- PastForward addresses VLM inference and A2O latency by reusing experience only when validated against current predictions or the observed screen.Its three design components are adaptive decoding, next-screen early inference, and cross-step state carryover.Source: §3.1 · p. 4
3.2 Experience-Guided Adaptive Decoding
- PastForward retrieves prior output sequences as multi-token proposals, verifies them against the current VLM, and adapts proposal length to acceptance behavior and device costs.Only the matching proposal prefix is accepted, preserving the greedy-decoding output sequence.Source: §3.2 · pp. 4–5
3.3 Early Inference via Next-Screen Prediction
- PastForward retrieves prior GUI transitions to start next-step inference during the current action’s A2O interval, then uses that computation only if the observed screen matches.Direct screen-and-action lookup can be supplemented by latent retrieval; a miss skips early inference.Source: §3.3 · pp. 5–6
3.4 Cross-Step State Carryover
- Cross-step state carryover reuses KV states for unchanged prompt prefixes and retained screenshots to reduce repeated prefill computation.Retained screenshot keys are adjusted for their new positions, but reuse remains approximate because prior causal context may have changed.Source: §3.4 · pp. 6–7
4 Implementation
The implementation integrates PastForward into a shared inference runtime and uses isolated runtime state for early inference. Figure 6 depicts task execution counts across two workloads.
4 Implementation
- PastForward is implemented as a common inference runtime supporting multiple VLM-based GUI agents in AndroidWorld.The integration preserves each agent’s prompt construction, image processing, output parsing, and action interface.Source: §4 · p. 7
- Figure 6 counts executions for each task under two workloads, with M=348 task instances.Source: Figure 6 · p. 7
- The runtime isolates Early Inference’s KV cache, decoding memory, and proposal-controller state until screen validation completes.It uses a contiguous BF16 cache for prompt-prefix and rolling-image KV-state carryover.Source: §4 · p. 7
5 Evaluation
PastForward is evaluated on AndroidWorld across varied workloads, VLM backbones, and server and edge platforms. It reduces action-step latency while maintaining task success, with the evaluation also examining its mechanisms, accumulated experience, design choices, and computational overhead.
5.1 Evaluation Setup
- The evaluation uses AndroidWorld's 116 tasks across 20 applications, with task-uniform and user-calibrated workloads, three VLM backbones, and four platforms.The user-calibrated workload uses α=1.0 to match app-level concentration in FingerTip20K.Source: §5.1 · pp. 7–8
5.2 Overall System Performance
- 1.63–2.36× edge-device speedups over Vanilla maintained SR; server speedups were 1.40–1.61× under task-uniform and 1.47–1.68× under user-calibrated workloads.The edge result covers three devices; server figures are action-step speedups.Source: §5.2 · pp. 8–9, Table 3 · p. 9, Table 2 · p. 9
- PastForward also achieves 1.48–2.31× and 1.23–1.52× action-step speedups over FastV and Agent-X, respectively, across Jetson CUDA and smartphone NPU runtimes.Source: §5.2 · p. 9
5.3 Latency Reduction Breakdown
- Adaptive decoding supplies the largest incremental latency reduction, while early inference and cross-step state carryover reduce additional action-step and prefill work.Adaptive decoding reduces decoding latency by 70.3–81.6% across three backbones. Carryover cuts prefilled token counts by 70.7–77.9% and prefill latency by 31.7–40.4% for MAI-UI and GUI-Owl; Qwen3-VL's prefill latency changes only from 0.452 s to 0.447 s.Source: §5.3 · pp. 9–10, Figure 7 · p. 10, Figure 8 · p. 10
5.4 Effect of Experience Accumulation
- As experience accumulates, adaptive-decoding retrieval rises from 67.0% to 91.1%, and the proposal-acceptance rate reaches 59.4%.Adaptive decoding's mean action-step latency reduction grows from 1.037 s over the first 10% of task instances to 1.606 s over the last 10%; Early Inference's matched-screen rate converges to 20.2%, with latency reductions growing from 0.272 s to 0.577 s. The larger late-run reduction is consistent with more action steps benefiting from inference performed during the A2O interval.Source: Figure 9 · p. 10, §5.4 · pp. 10–11
5.5 Analysis of Design Alternatives
- The online proposal-length controller tracks acceptance and verification latency, selecting device-specific lengths that outperform fixed-length policies.Fixed-length decoding latency is minimized at ℓ=16 on Jetson Orin Nano and ℓ=8 on Galaxy S26 Ultra; longer proposals increase latency beyond those lengths.Source: Figure 10 · p. 11, §5.5 · p. 11
- The two-stage direct-plus-latent next-screen lookup raises matched-screen rate from 17.6% to 23.0% and launch rate from 43.1% to 57.3% over direct lookup.Action-only and screen-only lookups launch on 81.6% and 55.3% of steps, but match only 18.2% and 13.7%, respectively.Source: Figure 12 · p. 11, §5.5 · p. 11
5.6 System Computational Overhead
- PastForward increases peak RAM by 2.4 MiB (0.1%) over Vanilla, while online operations take at most 2.5% of action-step latency.Retrieval remains below 5 ms per lookup across budgets, but reducing the decoding-memory budget from uncapped to 4 MiB raises per-step decoding latency from 4.111 s to 5.137 s.Source: Figure 13 · p. 11, §5.6 · p. 12
6 Discussion
PastForward is compatible with other VLM inference optimizations, but its latency benefit depends on workload opportunities for reuse and early inference.
6 Discussion
- Other VLM inference optimizations are largely complementary to PastForward, but changes to hidden-state representations or KV layouts require rebuilding affected records.Examples include quantization, optimized attention kernels, visual-token pruning, and KV-cache compression.Source: §6 · p. 12
- PastForward’s latency benefit depends on workload opportunities for token reuse and early inference.Infrequent decoding-context recurrence, variable resulting screens, or a short A2O interval can reduce proposals, discard more early computation, or hide less computation.Source: §6 · p. 12
- The observed gains appear to arise mainly from recurring decoding contexts and predictable screen transitions.Source: §6 · p. 12
7 Related Work
Prior work reuses interaction history as behavioral knowledge, optimizes particular model or pipeline stages, or gathers UI context; PastForward instead reuses fine-grained execution records online.
7 Related Work
- PastForward reuses fine-grained execution records online, unlike prior systems that organize interaction history into app knowledge, subtasks, workflows, or executable procedures.Many prior systems rely on offline exploration, model training, or memory construction; PastForward needs neither task-specific offline preparation nor changes to the VLM backbone or agent policy.Source: §7 · pp. 12–13
- Prior systems derive app-specific knowledge or train models from exploration and documentation, [24], [25], while others reuse procedural units or compose executable paths, [23], [3], [15], [17].LLM-Explorer [34] focuses on improving exploration itself.Source: §7 · p. 12
- Other work accelerates agent pipelines or VLM inference through V-Droid’s batched verification over a discrete action space, Agent-X’s N-gram speculative decoding [4], and FastV’s low-importance visual-token pruning.V-Droid requires a separately trained 8B verifier and still invokes a cloud LLM to update working memory.Source: §7 · p. 12
8 Conclusion
PastForward turns prior GUI-agent executions into fine-grained inference reuse, reducing action-step latency while maintaining task success across multiple VLMs and platforms.
8 Conclusion
- PastForward uses validated token-sequence proposals, next-screen-guided early inference, and cross-step KV-state carryover to reuse computational experience.Across multiple VLMs and server, edge, and mobile platforms, it reduces action-step latency while maintaining task success, without task-specific offline exploration or changes to the agent policy.Source: §8 · p. 13