Source-linked AI summary
CausalCache: Conditional High-Fidelity Restoration for Long-Horizon GUI Agents
Jiaxuan Luo, Zhanfeng Liao, Jiayao Teng, Yuan Wang
TL;DR
Long-horizon GUI agents must decide which archived screenshots to restore when only a few historical images fit in active context. CausalCache reallocates this fixed visual budget across the complete action history, improving zero-shot MobileWorld success over recency from 30.2% to 36.8%, especially on cross-app memory-critical tasks.
Problem
Long-horizon GUI agents retain complete action histories as text but must decide which recorded events should regain high-fidelity pixels under a limited visual budget.
Method
CausalCache keeps every event summarized and reallocates a fixed budget of restored screenshots across the complete trace, promoting older events when their predicted utility exceeds recent alternatives.
Results
On zero-shot MobileWorld, CausalCache reaches 36.8% versus 30.2% for HGKV+Recent-B, with gains concentrated on memory-critical cross-app tasks.
Takeaways & Limitations
Selecting which past events regain pixels is more effective where tasks depend on distant visual information, while single-app controls show no detectable effect.
Abstract
from arXiv · showhide
Long-horizon GUI agents can retain complete action histories as compact text, but only a few historical screenshots fit in active context. We formulate this as budgeted fidelity restoration: every event remains summarized, while a fixed budget $B$ determines which events regain their archived screenshots. Recent-$B$ assigns all visual slots to the latest events. CausalCache instead scores the complete history and swaps in an older event only when its predicted utility exceeds that of a recent event. A history-gated key/value adapter modifies only restored history-image tokens and is exactly bypassed when no history image is active, preserving current-screen processing. The adapter and selector are trained with matched-budget interventions on desktop trajectories and evaluated zero-shot on mobile. On OSWorld-Verified, activating historical screenshots improves success by about $13$ percentage points over summary-only memory. Under the official $15$-step limit, CausalCache and Recent-$4$ are statistically indistinguishable; in a $30$-step diagnostic, CausalCache achieves $46.7\%$ success versus $42.4\%$ ($+4.3$ points). Zero-shot on $117$ MobileWorld tasks, CausalCache improves over Recent-$4$ from $30.2\%$ to $36.8\%$. The gain is concentrated on a pre-defined cross-app memory-candidate split ($30.6\%$ vs. $19.4\%$, $+11.2$ points), while single-app controls show no detectable difference ($43.6\%$ vs. $42.4\%$). These results show that selecting which past events regain pixels is more effective than spending a fixed visual budget entirely on recency.
Introduction
CausalCache reframes GUI memory as conditional restoration of event fidelity: all events remain summarized, while a fixed visual budget determines which archived screenshots return to active context. It uses a budget-aware selector and history-gated interface to restore useful pixels without altering current-screen processing.
- Problem: GUI agents must decide not only which events remain recorded, but which recorded events should be exposed again as high-fidelity pixels.This frames long-task memory as a fidelity-restoration problem rather than simple event retention.
- Core idea: CausalCache controls event fidelity, recognizing that an event’s value depends on which historical images are already active.A distant screenshot may lack local grounding without recent images, while another image may be redundant when the recent window is saturated.
- Method: Every event remains in a complete low-fidelity action trace, while only promoted events regain archived screenshots within the budget B.Recent-B promotes the latest B events; CausalCache instead restores selected older events without reconstructing pixels from text.
- Policy preservation: The history-gated KV interface modifies only restored history-image tokens and is structurally bypassed when no history image is active.This design preserves the base agent’s current-screen understanding, action syntax, and grounding.
- Contributions: CausalCache combines conditional fidelity restoration, matched-budget HGKV training, exact no-history parity, drift caps, and a selector trained over the complete summarized trace.The listed contributions define the system’s allocation, adaptation, and preservation mechanisms.
Related Work
Prior work spans visual GUI benchmarks, reusable agent memory, long-horizon GUI memory, context compression, and interventional memory value. CausalCache differs by restoring selected archived event images while keeping compressed summaries and the remaining prompt fixed.
- Visual GUI agents and benchmarks: Visual GUI benchmarks measure long-horizon interaction across grounded shopping, websites, enterprise workflows, visual web interaction, desktop systems, and mobile workflows.MobileWorld specifically evaluates reproducible, long-horizon cross-application mobile workflows.
- Agent memory and reusable experience: RAG, ReAct, reflection-based agents, evolving stores, and ICAL preserve or retrieve reusable artifacts across interactions or tasks.CausalCache instead holds a within-episode summary trace.
- Long-horizon GUI memory: GUI-Odyssey, MementoGUI, and AndroTMem vary which prior screenshots, memory items, or causally linked anchor states are retained or retrieved.CausalCache keeps the compressed trace fixed, conditions event fidelity on active visual context, and trains the interface consuming restored pixels.
- Context and visual-token compression: Prompt, KV-cache, and visual-token compression methods operate on textual redundancy, cache states, or tokens within images, whereas CausalCache preserves the summary trace.Their unit is a token or KV position.
- Interventional memory value: Interventional memory research studies controlled interventions or active state influence, while CausalCache restores one archived post-action image with the remaining prompt fixed.This yields a policy-specific behavioral surrogate rather than an environment-level causal effect.
Method
CausalCache treats screenshot restoration as a fixed-budget fidelity allocation over summarized history, then conditionally adapts only the reattached historical-image tokens. The method defines utility through interventions on allocations and preserves the original computation when no historical event is promoted.
- Budgeted fidelity allocation: Promotion and demotion move each retained event’s verbatim policy response and archived post-action screenshot together, reallocating retained turns rather than images alone.The official retained-turn form pairs response rj with archived screenshot vj.
- Budgeted fidelity allocation: A fixed visual-context budget B promotes exactly min(B, t−1) summarized events by reattaching archived pixels, while all textual action summaries and the current screenshot remain present.The budget limits active visual context, not cold archival storage, and reallocates which events occupy the slots.
- Allocation scoring: Causal utility is measured by intervening on fidelity allocations while holding the policy, current observation, summarized trace, and active-image budget fixed.This operational definition is policy- and prompt-specific rather than an environment-level structural causal effect.
- History-gated adapter: In the last eight language-model layers, a rank-8, α=16 low-rank residual modifies only key and value projections for reattached historical-image tokens.The historical-token mask excludes instruction, summary-text, current-observation, and action tokens, and is constructed fail-closed from token types and image-patch geometry.
- History-gated adapter: When no historical event is promoted, the adapter context is absent and the projection hook returns the original module output, yielding πHG(· | S=∅) = π0(· | S=∅) up to bitwise equality.This bypass condition is implemented exactly when no historical image is active.
Per-B Fidelity-Restoration Supervision
Per-B supervision constructs matched-budget restoration interventions that isolate target-relevant historical screenshots from recency and wrong-history controls. CausalCache then learns a single-slot restoration interface and a policy-grounded selector over summarized events.
- Matched-budget interventions: Training states are decision points in successful desktop trajectories where the target action recurs earlier under full-action equivalence, identifying a target-specific archived screenshot.The screenshot preceding the earlier equivalent action becomes the restoration candidate.
- Matched-budget interventions: Each budget B receives exactly B history images under Recent-B, relevant restore, and wrong restore allocations with identical summaries, current screen, and target.Relevant restore replaces the oldest recent visual slot with the target-specific archived screenshot; wrong restore uses an age-matched screenshot whose following action is not equivalent.
- Single-slot interface: k=1 supervision is used because matched-budget probes find the first replacement valuable, while paired second- and third-replacement increments are indistinguishable from zero or negative.Deployment can still express larger-k allocations by composing exact-B sets with recent fallback.
- Learned components: The history-gated K/V adapter modifies only restored history-image tokens, leaving current-screen, text, and response tokens unchanged and vanishing at B=0.At B=0, the policy is bitwise-identical to the frozen base.
- Selector supervision: After the interface is frozen, the selector learns policy utilities by rescoring complete prompts and predicts set-conditional marginal gains over candidate events.Its training uses singleton utilities, matched Q(Recent-B) anchors, teacher-path set utilities, joint regression, and within-state ranking losses.
Experiments
Experiments isolate same-budget fidelity allocation and test it across desktop and zero-shot mobile settings. CausalCache’s gains are strongest when tasks require distant visual evidence, while controls and adaptive-budget analyses support selective restoration rather than recency or similarity alone.
- Experimental design: B=4 is the primary deployment-like setting, with every allocation restoring exactly B history images at identical resolutions, summaries, and prompt structure.The realized number of non-recent promotions k is measured rather than preset.
- Mobile closed loop: 36.8% is CausalCache’s overall mobile success rate, against 30.2% for HGKV+Recent-B and 28.2% for the reported comparison arm.The 117-task mobile benchmark contributes no training or selection signal and is evaluated zero-shot.
- Mobile closed loop: 30.6% versus 19.4% is CausalCache’s cross-app memory-critical success contrast with Recent-B, while controls show 43.6% versus 42.4%.The split is constructed so earlier-application information is needed after switching; the split-by-method interaction is +10.1pp.
- Selector analysis: 19% of B=4 states keep the entire recent window, while 26% replace all of it, demonstrating state-dependent allocation rather than fixed recency.Random exact-B and RGB-similarity restoration are indistinguishable from Recent-B, whereas inverting the learned score is significantly harmful.
- Deployment cost: 2.0/2.3/2.9 s is the added model time per step at B=1/2/4, corresponding to 8.1/8.7/10.5% of measured per-step wall time.The selector costs under 40 ms, and desktop median end-to-end task overhead is 7%.
Limitations and Conclusion
The paper frames long-horizon GUI memory as allocating a fixed visual budget between text-preserved events and restored pixels. HGKV learns to identify useful distant screenshots without altering matched-recent behavior, while the selector reallocates visual capacity where distant dependencies exist.
- Problem formulation: The method treats memory as fidelity allocation: every event remains text, while only B events regain pixels.This defines the fixed visual-budget setting underlying the approach.
- History-gated adaptation: HGKV learns to distinguish useful distant screenshots from recent frames they would displace within a drift envelope.Matched-recent behavior remains untouched under this mechanism.
- Conclusion: The learned selector reallocates visual capacity where distant visual dependency exists, particularly in memory-critical tasks.The supplied passage ends while describing the resulting improvement, so no quantitative result is reported here.