Source-linked AI summary
Learning How to Remember: Memory Modality Selection for GUI Agents
Taeyun Roh, Suhyeong Park, Gijin Kim, Wonjune Jang, Jaewoo Kang
TL;DR
GUI memory systems typically use a fixed representation, leaving unclear whether one modality suits every interaction step. The paper studies modality preference and introduces SwitchMem, which routes between text and image memory at each step. Across benchmarks, SwitchMem outperforms static memory baselines, supporting step-dependent memory selection.
Problem
Existing GUI memory systems generally use fixed representations, and it remains unclear whether one modality is suitable for every interaction step.
Source: §1 · p. 1Method
The paper studies memory-modality preference and introduces SwitchMem, which uses internal representation signals to select textual or visual memory at each step.
Source: §1 · p. 2Results
SwitchMem consistently outperforms static memory baselines across evaluation tasks, while preferred memory modality varies across interaction steps.
Source: §1 · p. 2Takeaways & Limitations
Memory modality selection is a step-dependent decision rather than a fixed design choice.
Source: §7 · p. 8SwitchMem requires actor hidden states for preference-target construction and inference, limiting direct use with APIs that do not expose them.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Recent advances in vision-language models (VLMs) have substantially improved the capabilities of GUI agents, while memory augmentation has emerged as a promising approach for reusing knowledge from past interactions. However, existing memory augmented GUI agents typically adopt a fixed representation strategy, providing retrieved experience as a static modality. It remains unclear whether a single memory modality is consistently optimal, or how modality preference changes across interaction steps. In this work, we conduct a systematic study of memory modality in VLM based GUI agents. Across GUI benchmarks, we find that the preferred memory representation is strongly step dependent. Motivated by the finding, we introduce \textbf{SwitchMem}, a routing framework that dynamically selects between textual and visual memory representations at each step. Across three GUI benchmarks, SwitchMem outperforms memory augmented baselines that rely on static memory representations. Our findings validate our claim that memory modality is a step dependent decision rather than a fixed design choice. Code and project resources are available at https://txxnrd.github.io/switchmem
1 Introduction
GUI agents face long-horizon tasks where errors can propagate, while existing memory systems generally use a fixed representation for retrieved experience. This work studies step-dependent modality preference and introduces SwitchMem to route between textual and visual memory.
1 Introduction
- GUI tasks often require long-horizon decisions, and errors at one step can propagate to later decisions.Source: §1 · p. 1
- Existing GUI memory systems generally present retrieved experience in a predefined, unchanged representation across interaction steps.Source: §1 · p. 1
- Text can compactly abstract visual content, while visual memory preserves fine-grained information that textual surrogates may lose.Source: §1 · pp. 1–2
- Figure 2 reports that neither Text nor Image memory consistently dominates across benchmarks.Source: Figure 2 · p. 2
- Preferred memory modality varies across interaction steps, and the preference is reflected in agents’ internal representations.Source: §1 · p. 2
- SwitchMem adaptively selects Text or Image memory at each step and consistently outperforms static memory baselines across evaluation tasks.Source: §1 · p. 2
2 Related Work
Mobile GUI agents repeatedly interpret screenshots and act, with errors potentially propagating through later decisions. Prior memory approaches reuse past interactions in several forms, but typically keep a fixed modality after retrieval.
2 Related Work
- Mobile GUI agents act through repeated screenshot-based decisions, and errors in perception or action selection can propagate through subsequent decisions.Source: §2 · p. 2
- GUI memory approaches reuse episodic histories, textual traces, structured procedures, executable knowledge, or visual observations, but keep a fixed modality after retrieval.Source: §2 · p. 2
3 Understanding Memory Modality Preference
The section compares textual and visual memory while holding retrieved demonstrations fixed, then finds modality utility varies by interaction step. It also probes whether preference is reflected in the actor’s hidden representations.
3.2 Step-wise Memory Modality Preference
- Across AITW and LearnGUI, both memories are preferred on substantial step subsets, while the step-wise oracle beats the strongest fixed strategy across benchmarks.The oracle compares both modalities at each evaluation step under the same fixed seed and takes the higher-scoring action.Source: Figure 3 · p. 3, §3.2 · p. 3
3.1 Study Setup
- The setup holds retrieval quality constant by using one text-based retriever to select a demonstration instantiated as either Text or Image memory.The study compares expected action utility for both modalities using sampled actions from the frozen actor.Source: §3.1 · p. 3
3.3 Modality Preference in Hidden States
- On AITW, modality preference is predictable from representation shifts, with probe AUROC increasing in deeper layers for examples with at least five-of-seven agreement.The analysis reduces each layer’s shift representation with PCA and trains a linear probe.Source: Figure 4 · p. 3, §3.3 · p. 4
4 SwitchMem
SwitchMem trains a router to select Text or Image memory at each interaction step, using modality-dependent actor representations to construct preference targets. The router compares candidate scores and provides only the selected memory to the frozen actor.
4 SwitchMem
- SwitchMem uses two stages: a preference estimator maps actor representation shifts to targets, then a pairwise router learns to predict those targets.The representation shift is transformed into a continuous preference target for router training.Source: §4 · p. 4, §4.1 · p. 4
4.1 Preference Target Construction
- The preference estimator uses final-layer D-CoE shifts and non-tied states, reducing 4096-dimensional representations to 64 dimensions with PCA before fitting a logistic probe.Positive targets favor Image memory; negative targets favor Text memory.Source: §4.1 · p. 5
4.2 Preference-Supervised Router Training
- A pairwise router uses the current-state hidden representation and retrieved-memory embeddings to score Text and Image candidates; their score difference represents relative preference.A shared MLP scorer processes each candidate feature, and the router regresses the continuous preference target.Source: §4.2 · p. 5
- At inference, the router selects Image when its relative score meets threshold τ; otherwise it selects Text, and only that memory reaches the actor.Table 2 summarizes routing behavior using dual-modality trajectories, Image-routed steps, modality switches, and contiguous run length.Source: §4.2 · pp. 5–6, Table 2 · p. 6
5 Experiments
Across three GUI benchmarks, SwitchMem selects one memory modality per step and achieves the strongest performance. The experiments compare this approach with fixed memory representations and other GUI agents.
5 Experiments
- Experiments use a frozen Qwen3.5-9B actor, a Qwen3-VL-Embedding-2B retriever, and benchmarks MobileMiniWob++, AITW, and LearnGUI.Because MobileMiniWoB++ has no official training or validation splits, the router trained on LearnGUI is transferred with weights fixed.Source: §5 · p. 6
5.1 Results
- Image memory tends to perform well on AITW and LearnGUI, while providing both modalities is more effective on MobileMiniWoB.Advanced memory formulations do not consistently outperform raw context injection.Source: §5.1 · p. 6
- SwitchMem achieves the strongest performance across all three benchmarks by selecting a single memory modality at each interaction step.The authors attribute its performance to exploiting complementary Text and Image strengths across steps while avoiding degradation from providing both modalities together.Source: §5.1 · p. 6
- Across benchmarks, SwitchMem frequently uses both modalities within a trajectory, with moderate transition frequency and consecutive runs in the selected modality.This routing pattern is reported across all three benchmarks.Source: §5.1 · p. 6
6 Analysis
The analysis examines when Image memory is more useful, whether SwitchMem’s choices track those utility patterns, and how its routing compares with alternatives in performance, efficiency, and transfer.
6.1.1 When Is Each Memory Modality Useful?
- Image memory has a larger utility advantage on denser screens, Function rather than Spatial actions, and post-initial states.Element density is positively associated with the Image-over-Text utility gap; visual memory is also reported as useful for fine-grained layouts and interface state or functionality.Source: §6.1.1 · p. 7
6.1.2 Understanding SwitchMem’s Memory Modality Selection
- SwitchMem routes Image more often on visually dense screens, Function-action steps, and post-initial states, aligning its choices with observed utility shifts across all three dimensions.The authors report that this alignment helps explain its gains over static memory strategies across all three benchmarks.Source: §6.1.2 · p. 7, Figure 6 · p. 7
6.2 Comparison with Alternative Routing Strategies
- Conventional learned routing baselines offer minimal gains over static Image Memory, while SwitchMem improves performance consistently.Utility and Binary use action-utility supervision, while Basic routing uses current instruction, screenshot, and retrieved-memory embeddings rather than actor hidden states.Source: §6.2 · p. 7, Figure 7 · p. 8
- SwitchMem outperforms ratio-matched random routing at every evaluated intermediate Image-selection rate, supporting the value of assigning modalities to particular steps.The random policy matches SwitchMem’s Image-selection rate.Source: §6.2 · pp. 7–8, Figure 7 · p. 8
6.3 Efficiency Analysis
- On LearnGUI, SwitchMem achieves the highest action score with modest latency, without processing both memory-conditioned prompts or adding an autoregressive decision stage.Procedural baselines add annotation, document-construction, or guideline-generation stages and have higher latency despite lower performance.Source: §6.3 · p. 8, Figure 7 · p. 8
6.4 Generalization to a GUI specialized Actor
- With the UI-TARS-7B-SFT actor, SwitchMem outperforms static memory baselines across all three benchmarks.Table 3 reports the best performance on each benchmark and improvement over the strongest fixed memory condition.Source: Table 3 · p. 8, §6.4 · p. 8
7 Conclusion
The paper finds that memory-modality preference varies across interaction steps and is reflected in the agent’s internal representations. SwitchMem uses that signal to choose between textual and visual memory at each step, outperforming fixed-modality and memory-augmentation baselines across benchmarks.
7 Conclusion
- Neither Text nor Image memory consistently dominates, and their utility varies across interaction steps.D-CoE analysis indicates that modality preference is reflected in the agent’s internal state.Source: §7 · p. 8
- SwitchMem uses modality-conditioned representation shifts to train a router that selects a memory modality at each step, while providing only one modality to the actor.The paper reports improvement over fixed-modality and memory-augmentation baselines.Source: §7 · p. 8
Limitations
The evaluation has three stated scope constraints: Android-centered controlled benchmarks, emulated rather than physical devices, and dependence on actor hidden states.
Limitations
- Evaluation is limited to Android-centered mobile GUI benchmarks in controlled interaction settings; generalization to iOS and newer Android versions remains unverified.Those platforms may introduce different visual layouts and interaction conventions.Source: Limitations · p. 9
- Online evaluation uses emulated environments, which do not fully capture device rendering differences, system interruptions, network variability, or hardware constraints.Evaluation on physical devices is identified as future work.Source: Limitations · p. 9
- SwitchMem needs actor hidden states to construct preference targets and route at inference, so it cannot directly serve models whose APIs conceal those representations.Extending routing to such models remains open.Source: Limitations · p. 9
A Dataset and Baseline Details
The evaluation spans three mobile GUI benchmarks with complementary interaction settings and standardizes the actor and retrieval setup across memory-augmented methods. Baselines differ in how they represent or deliver retrieved experience.
A.1 Dataset Details
- AITW tests next-action prediction on heterogeneous human-demonstrated Android trajectories spanning diverse applications and multi-step goals.The benchmark-specific action scorer is used.Source: §A.1 · p. 11
- LearnGUI tests action-level execution accuracy in demonstration-augmented navigation, retrieving support traces distinct from each query trajectory.Its setting requires agents to use retrieved historical execution traces.Source: §A.1 · p. 11
- MobileMiniWoB++ measures task-completion success in deterministic, online closed-loop tasks, providing immediate feedback for step-wise actions.The controlled setting isolates routing from real-world rendering anomalies and ambiguous rewards.Source: §A.1 · p. 11
A.2 Baseline Details
- Across all three benchmarks, Text and Image conditions hold the actor, current observation, retrieved demonstration, and decoding configuration fixed, varying only retrieved-memory representation.Methods share a frozen Qwen3.5-9B actor, retrieval bank, and decoding protocol; all baselines are training-free component-level memory methods.Source: §A.1 · p. 11, §A.2 · pp. 11–12
- Fixed TEXT serializes the retrieved instruction, state description, and action; IMAGE substitutes the screenshot for the state description, while BOTH supplies both.The instruction and action information are retained in IMAGE.Source: §A.2 · p. 11
- Other baselines supply procedural guidelines from tutorials (RAG-GUI) [Xu et al., 2025a], structured UI context (M3A), or retrieved app-operation documents (AppAgent).EchoTrail reuses compact textual trajectory traces, while AGMem provides cropped visual evidence, falling back to full frames when regions cannot be identified [Choi et al., 2026].Source: §A.2 · pp. 11–12
B.1 SwitchMem Training and Evaluation Details
SwitchMem trains a router from sampled Text-versus-Image action preferences and evaluates it using benchmark-specific data protocols. Probing finds preference signals in later layers, while the method uses the final layer as a consistent extraction point.
B.1 SwitchMem Training and Evaluation Details
- For AITW and LearnGUI, training compares seven sampled actor actions per modality, excludes utility ties, and freezes a utility-selected routing threshold before testing.The router is a two-layer 1024 →256 →1 MLP with GELU and 0.1 dropout, trained using AdamW at learning rate 0.001.Source: §B.1 · p. 12
- AITW training separates 20,000 steps by trajectory into 10,019 preference-query steps and 9,981 memory steps, with no trajectory overlap.Testing uses 5,000 steps from 759 held-out trajectories and permits retrieval from the full 20,000-step training bank.Source: §B.1 · p. 12
- LearnGUI uses 101 successful Online demonstrations comprising 1,500 steps for preference supervision, then evaluates in its offline one-shot setting with the router and threshold fixed.No Offline-Test actions, utilities, or outcomes are used for fitting or calibration.Source: §B.1 · p. 12
- MobileMiniWoB++ transfers the LearnGUI router without weight updates and calibrates its threshold using only initial states from 92 task templates.The memory bank is fixed before calibration and evaluation; different suite seeds instantiate distinct task instances, and no seed-42 observations are used for training or calibration.Source: §B.1 · p. 12
- Text memories pair task instructions and demonstrated actions with serialized GUI observations, using dataset annotations for AITW and LearnGUI and UIAutomator accessibility hierarchies for MobileMiniWoB++.No additional captioning model is used.Source: §B.1 · p. 12
B.2 Modality-preference Probing Details
- Preference probes use signed D-CoE shifts from the prompt token before action generation, standardized and reduced to 64 PCA dimensions before logistic classification.A preference label is retained only when at least five of seven sampled generations agree.Source: §B.2 · p. 12
- Preference signals are strongest in later layers, peaking at layer 31 on AITW and layer 23 on LearnGUI; SwitchMem uses layer 31 for both.The final block supplies a single benchmark-independent extraction rule; Figure 8 reports AUROC on the 5-of-7 stable-preference subset, with chance marked.Source: §B.2 · pp. 12–13, Figure 8 · p. 13
B.3 Additional Routing Strategy Comparisons
SwitchMem outperforms simpler and ratio-matched routing alternatives, with reported gains over fixed memory at modest added latency. Case studies show different modalities preferred in distinct interaction states, while the MobileMiniWoB++ gain is not statistically significant.
B.3 Additional Routing Strategy Comparisons
- Utility, Binary, and Basic routing improve over fixed Image but remain below SwitchMem; Binary is close on AITW, while SwitchMem’s advantage is more pronounced on LearnGUI.SwitchMem has the highest point estimates across both benchmarks; Figure 9 compares simpler routing baselines and notes a truncated y-axis.Source: §B.3 · p. 13, Figure 9 · p. 13
- SwitchMem outperforms ratio-matched random routing at every evaluated non-trivial modality budget, supporting step-specific assignment rather than the global Text–Image ratio as the source of adaptive-routing gains.The random baseline controls aggregate Image usage but does not condition selections on interaction state.Source: §B.3 · p. 13, Figure 10 · p. 13
B.4 Efficiency Details
- SWITCHMEM adds 0.128 s, 0.190 s, and 0.304 s per step over Image on AITW, LearnGUI, and MobileMiniWoB++, respectively.RAG-GUI takes 8.815 s and 11.118 s per step on AITW and LearnGUI, versus SWITCHMEM’s 1.407 s and 1.848 s; Table 4 reports mean latency per step and excludes retrieval and input preprocessing.Source: §B.4 · pp. 13–14, Table 4 · p. 14
B.5 Text and Image Memory Examples
- Text and Image memories preserve the same retrieved successful step and demonstrated action, differing only in whether the past GUI observation is rendered as text or a screenshot.The paired prompts also hold the current task, screenshot, action schema, actor, and decoding configuration fixed.Source: §B.5 · p. 14, Figure 11 · p. 15
B.6 Statistical significance.
- Improvements are significant on AITW (p < .001) and LearnGUI (p = .016), but not on MobileMiniWoB++ (p = .688).The AITW and LearnGUI gains remain significant after Holm correction; the MobileMiniWoB++ result is a positive but nonsignificant transfer gain from LearnGUI, with 92 evaluation tasks.Source: §B.6 · p. 14, Table 5 · p. 15
B.7 Case Studies of When Each Memory Modality Is Useful
- In a Sparse initial Spatial step, Text memory yields the correct address-bar tap (score 1 versus 0), and SwitchMem selects Text.The contrasting Image condition produces a typing action before selecting the address bar.Source: §B.7 · p. 16, Figure 12 · p. 17
- In a Dense post-initial Function step, Image memory completes the Walmart search (score 1 versus 0), while Text memory produces an unnecessary Enter action; SwitchMem selects Image.The current screen already displays the requested search results.Source: §B.7 · p. 16, Figure 13 · p. 17
- The two examples illustrate contextual variation in memory usefulness but do not isolate the causal contribution of individual contextual factors..Source: §B.7 · p. 16