PaperlayerView exact revision on arXiv

Source-linked AI summary

ReMAP: Restoring the Perceptual Cycle with Reasoning-Time Latent Visual Memory

Hao Jiang, Zhanyu Guo, Chenwei Wu, Yichen Guo, Qizhe Zhang, Junchi Yao, Jixian Wu, Jinhao You, Kai Tang, Jiajun Cao, Tinghao Wang, Mengyu Wang, Leo Anthony Celi, Shanghang Zhang

arXiv:2610.05097v1cs.CVcs.AIcs.CL

TL;DR

Longer multimodal reasoning makes relevant visual evidence harder to retain, while visual-memory approaches vary in how they select, represent, and access it. ReMAP pairs question-conditioned Global context with reasoning-state-guided Local retrieval in compact latent tokens and learns access decisions through branched rollouts. It outperforms prior visual-memory methods on all four multi-image benchmarks, including by 8.38 and 14.84 percentage points on MuirBench and MIMIC.

  • Problem

    During extended generation, attention to visual inputs can diminish, and visual-memory methods differ in evidence selection, representation, and access timing.

    Source: §1 · pp. 1–2

  • Method

    ReMAP combines question-conditioned Global context with reasoning-state-guided Local region retrieval in compact latent tokens and uses branched rollouts to learn memory access.

    Source: §1 · p. 2

  • Results

    ReMAP outperforms prior visual-memory methods on all four multi-image benchmarks, exceeding the strongest prior results on MuirBench by 8.38 percentage points and MIMIC by 14.84 percentage points.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The conclusion frames perception as part of reasoning, with each reasoning step's evidence guiding the next visual inspection.

    Source: §6 · p. 11

    ReMAP's fixed memory and access budgets may omit evidence for inputs with many images or tasks requiring extended visual search.

    Source: §5 · p. 10

Abstract

from arXiv · show

As multimodal large language models (MLLMs) reason for longer, attention to the initial visual input diminishes, weakening visual grounding. Visual memory reintroduces visual evidence during reasoning. We conduct a controlled analysis of visual memory along three axes: curation, organization, and access. We find that local evidence benefits from global context, compact latent representations balance accuracy and visual-context cost, and the utility of memory access depends on the reasoning state. Guided by these findings, we propose ReMAP (Reasoning-Time Memory-Augmented Perception), which couples two complementary latent memories: a static, question-conditioned Global memory that preserves scene and cross-image context, and a dynamic Local memory that uses this context as an anchor while selecting and re-encoding region-level evidence according to the current reasoning state. Both memories return compact latent tokens inserted into the reasoning sequence, and a reinforcement-learning access policy trained with branched rollouts decides when to continue reasoning or invoke Global or Local memory. On ten benchmark families, ReMAP outperforms prior visual-memory methods on all four multi-image benchmarks, exceeding the strongest prior results on MuirBench and MIMIC by 8.38 and 14.84 percentage points. Across four backbone families, enabling memory access improves over the same trained model with memory disabled, and on shared V*Bench, CV-Bench-2D, and MuirBench questions ReMAP reduces the visual tokens entering the reasoning sequence by 51.0-76.8% relative to the native-resolution backbone. Further analyses show that Global and Local memory form distinct yet complementary latent representations. Together, these components restore the perceptual cycle by letting the reasoning state trigger targeted visual retrieval, with the retrieved evidence guiding subsequent reasoning.

1 INTRODUCTION

Longer multimodal reasoning makes sustained access to relevant visual evidence difficult, while existing visual-memory approaches make different choices about evidence, representation, and access. ReMAP combines question-conditioned Global context, reasoning-state-guided Local evidence, compact latent returns, and learned memory access; it outperforms prior visual-memory methods on all four multi-image benchmarks.

1 INTRODUCTION

  • Longer visual tasks require evidence at different reasoning stages, but attention to visual inputs can diminish during extended generation.The introduction frames visual forgetting as a challenge to sustaining visual grounding throughout reasoning.Source: §1 · p. 1
  • Existing visual-memory approaches differ in evidence selection, representation, and delivery timing, motivating a controlled analysis of which design choices improve visual reasoning.The cited approaches have distinct trade-offs, including evidence coverage, processing overhead, and interpretability.Source: §1 · pp. 1–2
  • Controlled analyses identify three design findings: local evidence can need global context, latent representations trade accuracy against overhead, and memory utility depends on access timing.The findings motivate contextualizing local evidence, compressing representations, and adapting access to the evolving reasoning state.Source: §1 · pp. 2–3
  • ReMAP pairs Global memory for scene and cross-image context with Local memory that selects and re-encodes regions according to the reasoning state.Both return compact latent tokens inserted into the reasoning sequence; branched rollouts train whether, when, and which memory to invoke.Source: §1 · pp. 2–3
  • 8.38 and 14.84 percentage points separate ReMAP from the strongest prior results on MuirBench and MIMIC, respectively; it outperforms prior visual-memory methods on all four multi-image benchmarks.Across ten benchmark families, ReMAP achieves state-of-the-art overall performance across the benchmark suite.Source: §1 · pp. 2–3

2 RELATED WORK

Related work spans different ways to select, represent, and access visual evidence, each with trade-offs. ReMAP combines pixel reacquisition with compact latent memory and trains access by comparing memory requests with continued reasoning.

2 RELATED WORK

  • Visual-memory approaches span pixel reacquisition, feature replay, and learned memory formation, with trade-offs between evidence coverage and processing overhead.Memory organization also varies across images, text, encoded features, and latent tokens, with interpretability and context cost as trade-offs (Hu et al., 2024; Zhang et al., 2026d; Huang et al., 2026; Yu et al., 2026).Source: §2 · p. 3
  • Memory access ranges from heuristic triggers and uncertainty criteria to learned policies, while related policy optimization uses training feedback for tool use and visual reasoning.The passage cites Sun et al. (2025), Liu et al. (2026b), and Peng et al. (2026) for access approaches, and Dong et al. (2026a) and Li et al. (2026c) for policy optimization.Source: §2 · p. 3
  • ReMAP bridges these dimensions by pairing pixel reacquisition with compact latent memory, preserving scene and cross-image context to interpret newly acquired local evidence.Its access training branches before invocation to compare requesting memory with continuing from the same reasoning state.Source: §2 · p. 3

3 REASONING-TIME MEMORY-AUGMENTED PERCEPTION

ReMAP interleaves reasoning with question-conditioned Global memory and state-guided Local memory, returning latent visual evidence to the reasoning sequence. It separates memory formation from access-policy learning, then uses branched rollouts to learn when to request memory.

3.1 MEMORY CURATION AND ORGANIZATION

  • Global memory summarizes each image and their relations conditioned on the question, then remains cached and fixed throughout reasoning.Image Weaver aggregates individual-image content; Relation Weaver integrates the summaries into cross-image context.Source: §3.1 · pp. 3–4

3 REASONING-TIME MEMORY-AUGMENTED PERCEPTION

  • Global and Local returns are inserted as continuous latent blocks after their request markers, followed by end markers before reasoning resumes.The framework also begins reasoning from a coarse visual prefix.Source: §3 · p. 3, Figure 2 · p. 4

3.1 MEMORY CURATION AND ORGANIZATION

  • Local memory selects images and crops from the current reasoning state, then compresses region features with cached Global context into K Local latents.The Global context anchors crops in image-level and cross-image information that local evidence alone can miss.Source: §3.1 · p. 4

3.2 MEMORY ACCESS

  • Stage I trains memory formation and region localization with supervised signals, then warms up memory-token handling while the formation modules are fixed.Training uses teacher CoTs and annotated target regions, followed by continuation and ROI targets reviewed on student rollouts.Source: §3.2 · p. 5
  • Stage II freezes the formation modules and learns whether and when to request Global or Local memory by branching from shared reasoning prefixes.Invocation gain compares a requesting sibling’s final-answer reward with the mean reward of siblings that continue without a request at that branch.Source: §3.2 · p. 5

4 EXPERIMENTS

The experiments evaluate ReMAP across ten benchmark families and analyze how memory access, latent organization, component choices, and visual-context cost relate to performance. ReMAP’s strongest overall result is on multi-image reasoning, while the analyses characterize distinct memory roles and state-dependent access.

4.1 EXPERIMENTAL SETUP

  • The evaluation covers ten benchmark families spanning single-image perception, multi-image reasoning, and broader visual understanding.Baselines include pixel reinspection, encoded-feature reuse, and latent-representation methods, alongside Native and Coarse backbone controls.Source: §4 · p. 6, §4.1 · p. 6

4.2 OVERALL PERFORMANCE

  • ReMAP outperforms prior visual-memory methods on all four multi-image benchmarks, beating the strongest prior results by 8.38 points on MuirBench and 14.84 on MIMIC.It also improves over the native-resolution backbone on all five single-image metrics.Source: §4.2 · p. 7

4.3 INTERPRETING LATENT MEMORY ORGANIZATION

  • Figure 4 shows benchmark-specific Local representations and cross-benchmark-mixed Global representations; Global is more dispersed in the multi-image view.The joint t-SNE projections show mean-pooled memory vectors, with benchmark groups distinguished by color and Global and Local by shape.Source: §4.3 · p. 7, Figure 4 · p. 8

4.4 MEMORY ACCESS AND THE PERCEPTUAL CYCLE

  • Figure 5 illustrates Local refinement and Global–Local interaction, while ReMAP inserts memory returns into ongoing reasoning and changes the subsequent reasoning state.Figure 6(b) shows Local calls concentrated early and in the middle of the trajectory, with Global calls persisting later.Source: Figure 5 · p. 8, §4.4 · pp. 8–9

4.5 COMPONENT, CAPACITY, AND ACCESS ABLATIONS

  • Global is stronger on CV-Bench-3D, MuirBench, and MMVet, while Local is stronger on V⋆Bench and BLINK; combining them performs best on every benchmark.The combined variant exceeds the better single-memory variant by 3.72 points on MMVet.Source: §4.5 · p. 9
  • ReMAP’s learned invocation policy is highest on every benchmark among learned, Fixed, Random, and Entropy schedules, but Fixed falls below w/o Memory on V⋆Bench.The learned policy exceeds the strongest alternative by 3.38 points on MuirBench and 4.64 on MM-Vet.Source: §4.5 · p. 9
  • Figure 7 shows broadly stable performance across tested latent capacities, with no consistent gains from returning more tokens; the selected settings are K = 32 Local tokens and 4N + 4 Global tokens.The figure reports benchmark scores and their equal-weight macro average on 20% holdout splits.Source: §4.5 · p. 10, Figure 7 · p. 10

4.6 VISUAL-CONTEXT COST

  • On matched V*Bench, CV-Bench-2D, and MuirBench questions, ReMAP remains competitive while using fewer visual tokens than Native.Compact memory returns refresh visual evidence without repeatedly expanding the decoder context.Source: Figure 6 · p. 9, §4.6 · p. 10

5 LIMITATIONS

ReMAP’s fixed request and memory-layout budgets, limited Local localization training, and fixed backbone attachment layers constrain its operation.

5 LIMITATIONS

  • Each trajectory permits at most five memory requests, each Local request re-encodes at most three selected images, and each Global block uses a fixed 4N + 4 layout.These limits keep computation low but may omit evidence for inputs with many images or tasks requiring extended visual search.Source: §5 · p. 10
  • Twig is trained only in Stage I on teacher-reviewed ROI targets and receives no further localization-specific training.When its box misses the relevant region, the zoom-in contributes less useful detail; the authors expect dedicated grounding training could improve Local memory.Source: §5 · p. 10
  • The Source Selector and Twig are attached at fixed backbone layers 12 and 18, respectively.The authors selected these taps based on internal representation-quality sweeps but have not studied other placements’ end-to-end effects.Source: §5 · p. 10

6 CONCLUSION

ReMAP coordinates Global scene and cross-image context with Local fine-grained evidence through compact latent returns and a learned access policy. The paper reports gains over prior methods on all four multi-image benchmarks and over the Native backbone on all five single-image metrics, while reducing profiled visual context.

6 CONCLUSION

  • ReMAP couples Global context with Local region-level evidence and uses a learned policy to coordinate compact latent returns according to reasoning state.The conclusion frames this coordination as restoring a perceptual cycle in which current evidence guides subsequent visual inspection.Source: §6 · pp. 10–11
  • ReMAP improves over prior methods on all four multi-image benchmarks and over the Native backbone on all five single-image metrics while reducing profiled visual context.The conclusion presents these outcomes as support for following the reasoning state in the scope and timing of visual evidence.Source: §6 · pp. 10–11

A.1 RELATED WORK AND POSITIONING

The related-work discussion organizes visual-memory methods by evidence curation, representation, and access. ReMAP is positioned as connecting active visual evidence acquisition with latent memory and learned access.

A.1 RELATED WORK AND POSITIONING

  • Curation methods range from pixel reacquisition and region selection to replaying encoded tokens and forming learned memories.The cited approaches include crop requests, image and video-frame selection, state-conditioned re-encoding, and explicit perceptual descriptions.Source: §A.1 · p. 16
  • Memory organization spans sketches, annotated images, text, pooled or compressed features, and latent token sequences.The cited methods use differing representations, including cached cross-image visual features and continuous hidden states.Source: §A.1 · pp. 16–17
  • Memory access methods use reflection events, self-diagnosis, uncertainty criteria, learned triggers, or separate perception modules; policy-training approaches also differ in rollout branching.The cited VisMem approach uses memories differing in abstraction level rather than spatial scope, and does not compare a request with continuing from the same state.Source: §A.1 · p. 17
  • ReMAP connects active evidence acquisition with latent memory and learned access, emphasizing complementary local detail and scene or cross-image context.Its positioning also highlights comparing invocation with continuation to guide selective access at a given reasoning state.Source: §A.1 · p. 17

A.2 RUNTIME AND MEMORY INTERFACE

ReMAP begins with coarse visual input and inserts Global or Local memory only when the decoder requests it. Global memory preserves cached image and cross-image context, while Local memory uses the current reasoning state to select and encode regions.

Memory transactions

  • Reasoning continues by default; a Global or Local request token triggers a memory transaction.After the returned latent block and end marker are inserted, generation resumes after the marker.Source: §A.2 · pp. 17–18
  • Inserted latents and end markers occupy decoder positions and KV-cache capacity, although they do not count toward the ordinary-text budget.The trajectory permits at most five requests; once the budget is exhausted or an answer is committed, further request tokens are masked.Source: §A.2 · p. 18

Global memory

  • Global memory allocates four tokens per image plus four for cross-image relations, forming a question-conditioned block before decoding.The block is cached throughout the trajectory, so later requests reuse it rather than recomputing it.Source: §A.2 · p. 18

Local memory

  • Local memory uses the current reasoning state to select sources and regions, then combines crop features with that state and cached Global memory.Selection stops when cumulative probability reaches 0.85 or three sources have been selected; a frozen vision encoder maps crops to region features.Source: §A.2 · p. 18

Memory interface

  • Each Local request returns 32 latent tokens, while Global context informs compression without being concatenated into that output.The interface uses a 4N + 4 Global layout and caps Local selection at three sources.Source: §A.2 · p. 18

A.3 MEMORY FORMATION AND ACCESS OBJECTIVES

The appendix specifies how Global and Local memories are formed and how the decoder learns to access them. It details teacher- and student-state supervision, interface warm-up, and branched rollouts that compare memory requests with continuation.

A.3.1 GLOBAL LAYOUT AND CACHING

  • Global memory uses four latent queries per image and four relation queries over image summaries, with both weavers conditioned on the question.The resulting 4N + 4 block is computed from prompt-time images and the question, then cached and reused at later requests.Source: §A.3.1 · p. 18

A.3.2 LOCAL SELECTION AND COMPRESSION

  • Local selection ranks image sources from the request state, keeps the shortest prefix reaching cumulative score 0.85, and caps selection at three sources.Twig localizes crops from selected images, and the frozen vision encoder produces their region features.Source: §A.3.2 · pp. 18–19
  • The Local Weaver conditions its compression queries on the reasoning state and cached Global memory, using crop features as fine-grained evidence.Global memory supplies image summaries and cross-image context; the default Local return contains 32 latent tokens.Source: §A.3.2 · p. 19

A.3.3 FORMATION AND WARM-UP OBJECTIVES

  • Continuation distillation trains each memory weaver through the frozen LLM to encode evidence used by the teacher’s subsequent reasoning.The frozen LLM passes gradients only to the weaver that produced the inserted memory.Source: §A.3.3 · p. 19
  • ROI localization trains Twig against teacher-reviewed boxes using weighted BCE and a soft-IoU term that matches heatmap extent to the target.Positive cells are weighted to prevent small target regions from being dominated by background cells.Source: §A.3.3 · p. 19
  • Student-state adaptation applies localization and continuation objectives to states from autoregressive student trajectories, with teacher-reviewed ROI and continuation targets.The weavers receive suffix-loss training conditioned on the student prefix.Source: §A.3.3 · p. 20
  • Interface warm-up uses next-token cross-entropy on supervised positions, including sequences without transactions so the decoder does not request memory at every state.The request token, first text token after the return, and later reasoning and answer tokens define supervised positions; prompt tokens, latents, and end markers are excluded.Source: §A.3.3 · p. 20

A.3.4 ACCESS LEARNING OBJECTIVE

  • Access learning branches from shared prefixes to compare requests with continuation, using answer rewards and a same-state continuation baseline.Four root trajectories are sampled per prompt, eligible roots branch into two siblings, and the policy uses dual-clipped updates; the request interface’s preference loss raises request probability when a request improves reward and lowers it otherwise.Source: §A.3.4 · pp. 20–22

A.4 TRAINING CONFIGURATION

Training proceeds through five steps: four Stage I formation steps followed by Stage II access learning. The vision encoder and pretrained LLM remain frozen, and checkpoint selection uses a fixed development set.

Optimization

  • The vision encoder and pretrained LLM weights stay frozen in every step, and only modules marked trainable are updated.The decoder LoRA is attached to the top four decoder layers with rank 16 and α = 32.Source: §A.4 · p. 22

Training pipeline

  • Figure 8 organizes training into five steps, with steps 1–4 in Stage I and step 5 in Stage II access learning.The decoder LoRA receives gradients only from Ldec, while the request interface receives gradients only from Laccess.Source: Figure 8 · p. 22

Rollouts and training length

  • Stage II samples 64–192 natural trajectories per step across 16 prompts, with at most five memory requests per rollout.The 40,000-example schedule contains 2,500 steps.Source: §A.4 · p. 22

Checkpoint selection

  • The selected checkpoint maximizes the equally weighted mean of per-task rewards on a fixed development set, with ties resolved in favor of the earlier checkpoint.Final benchmark scores do not enter checkpoint selection.Source: §A.4 · p. 22

Optimization

  • Table 5 lists the optimizer settings for memory formation and access learning.The ratio-clipping width is 0.2 and the dual-clip constant is 3.Source: §A.4 · p. 22, Table 5 · p. 23

A.5 TRAINING DATA

Training data are drawn from source-dataset training splits, with separate allocations for Local formation, Global formation, and access learning. Stage I annotations supervise memory content and regions, while Stage II uses source answers for rewards and post-call answer loss.

Sources and allocation

  • The training set comprises 50,000 Local-memory formation examples, 30,000 Global-memory formation examples, and 40,000 Stage II access-learning examples.All examples come from training splits and are deduplicated against evaluation units.Source: §A.5 · p. 23
  • Table 6 lists the source datasets and indicates whether they provide question–answer pairs, region annotations, or instance annotations used to construct additional questions.Source: Table 6 · p. 23

Stage I annotations

  • Stage I examples include reference CoT, per-image facts, cross-image relations, answers, and target regions.Facts and relations supervise Global memory; region annotations supervise Local source selection and localization; answers supervise task completion.Source: §A.5 · p. 23

Student-state targets

  • Student-state adaptation records student text-and-latent prefixes, source selections, crops, and memory returns for teacher review.The teacher returns ROI and continuation targets rather than reusing a full teacher CoT that may not follow from the student prefix.Source: §A.5 · p. 23

Access-learning data

  • Stage II data cover single-image perception, multi-image reasoning, and general visual question answering, using source answers for rewards and post-call answer loss.Stage I memory annotations are not used in Stage II.Source: §A.5 · p. 24

A.6 EVALUATION SPLITS AND SCORING

The appendix defines paired evaluation and scoring protocols, then tests how visual influence, access timing, memory content, and branch selection vary across reasoning. It also reports the visual-token cost of ReMAP on shared benchmark questions.

A.6 EVALUATION SPLITS AND SCORING

  • The suite covers ten benchmark families in fifteen evaluation units (19,913 examples), with paired same-example comparisons and benchmark-specific aggregation and judging.Baselines use released checkpoints and are not retrained on the study data; the main table also includes backbone and same-data controls.Source: Table 7 · p. 24, §A.6 · p. 24

A.7 VISUAL INFLUENCE AND STATE-CONDITIONED UTILITY

  • Under patch shuffling, mean JS divergence fell from 0.0320 to 0.0039, while identical evidence was more useful at later states on V*Bench (+0.1771) and CV-Bench-3D (+0.0726).The pooled signed change in generated-token log probability was not consistent across cohorts, so removing the initial images did not uniformly lower that probability.Source: Figure 9 · p. 26, §A.7.1 · p. 26, Table 10 · p. 27

A.8.1 SAME-BOX PIXEL RE-ENCODING

  • At K = 32, pixel re-encoding raised gold-answer log probability by 0.2243 and accuracy from 62.30% to 71.73%, versus feature pooling of the same gold RoI.The high-resolution RoI pass used 278.4 visual tokens on average, compared with 1,244.4 for the high-resolution full image.Source: Figure 10 · p. 27, §A.8.1 · p. 27

A.8.2 RELATIONAL-ROLE PROBE

  • Role changes shifted predictions more than appearance changes (+0.0482); oracle relational context repaired 41 of 71 selected late failures, an accuracy of 57.7%.Those states were selected as failures, and the relational context was an oracle; the probe measures the contribution of correct relational content, not learned Global memory.Source: Figure 11 · p. 28, §A.8.2 · p. 28, Table 11 · p. 28

A.9 ADDITIONAL ABLATIONS

  • The access variants share ReMAP weights and evaluation inputs; w/o Memory masks requests at inference, isolating the contribution of reasoning-time memory access without changing trained parameters.The curation variants separately enable Global or Local memory, while Fixed, Random, and Entropy change only request timing at inference.Source: §A.9 · p. 29

A.10 BRANCH-POINT SELECTION DIAGNOSTIC

  • 75.91% routing yield for u(st, at) exceeded 68.36% for route entropy, 38.54% for random selection, and 32.16% for the ARPO-style signal.Joint EAY was 0.144 for u(st, at), close to route entropy at 0.143 and above random at 0.084 and the ARPO-style signal at 0.067.Source: Figure 12 · p. 30, §A.10 · p. 30

A.11 COMPLETE QUALITATIVE CASE TRACES

  • Three Local calls changed HR-Bench-4K #0347 from 40 km/h/B to 30 km/h/D; two Local plus one Global call changed MMSI #0377 from east/C to northeast/B.These are paired memory-disabled and memory-enabled reasoning traces.Source: §A.11 · p. 30, Figure 13 · p. 31

A.12 VISUAL-CONTEXT ACCOUNTING

  • 76.8%, 51.0%, and 52.3% fewer visual tokens than Native on V*Bench, CV-Bench-2D, and MuirBench, respectively.ReMAP adds 16.0, 18.7, and 29.3 tokens per question through memory returns on those benchmarks, respectively.Source: Table 12 · p. 32, §A.12 · p. 32
Loading 2610.05097v1…