PaperlayerView exact revision on arXiv

Source-linked AI summary

VisMem: Latent Vision Memory Unlocks Potential of Vision-Language Models

Xinlei Yu, Chengming Xu, Guibin Zhang, Zhangquan Chen, Yudong Zhang, Yongbo He, Peng-Tao Jiang, Jiangning Zhang, Xiaobin Hu, Shuicheng Yan

arXiv:2511.11007v2cs.CVcs.AIcs.LG

TL;DR

VLMs can lose visual grounding and contextualized visual experience during prolonged generation, limiting complex visual understanding, reasoning, and generation. VisMem introduces dynamically invoked short- and long-term latent vision memories, and experiments report an average improvement of 11.0% over the vanilla model across diverse benchmarks.

  • Problem

    VLMs face a visual processing bottleneck that impairs fine-grained perception, multi-step reasoning, and fidelity over long generative sequences.

  • Method

    VisMem equips VLMs with dynamically invoked short-term latent memory for fine-grained visual evidence and long-term latent memory for abstract semantic knowledge.

  • Results

    11.0% average improvement over the vanilla model is reported across benchmarks spanning visual understanding, reasoning, and generation.

  • Takeaways & Limitations

    VisMem improves comprehensive visual capabilities while supporting cross-domain generalization, catastrophic forgetting mitigation, compatibility, and efficient inference.

Abstract

from arXiv · show

Despite the remarkable success of Vision-Language Models (VLMs), their performance on a range of complex visual tasks is often hindered by a "visual processing bottleneck": a propensity to lose grounding in visual evidence and exhibit a deficit in contextualized visual experience during prolonged generation. Drawing inspiration from human cognitive memory theory, which distinguishes short-term visually-dominant memory and long-term semantically-dominant memory, we propose VisMem, a cognitively-aligned framework that equips VLMs with dynamic latent vision memories, a short-term module for fine-grained perceptual retention and a long-term module for abstract semantic consolidation. These memories are seamlessly invoked during inference, allowing VLMs to maintain both perceptual fidelity and semantic consistency across thinking and generation. Extensive experiments across diverse visual benchmarks for understanding, reasoning, and generation reveal that VisMem delivers a significant average performance boost of 11.0% relative to the vanilla model and outperforms all counterparts, establishing a new paradigm for latent-space memory enhancement. The code will be available: https://github.com/YU-deep/VisMem.git.

1. Introduction

VLMs face a visual processing bottleneck during complex tasks, losing visual grounding and contextualized visual experience over prolonged generation. VisMem addresses this with dynamically invoked short- and long-term latent vision memories.

  • VLMs struggle with fine-grained perception, multi-step reasoning, and fidelity over long generative sequences because textual context can displace initial visual evidence.
  • Short-term memory is visually dominant for current-scene perception, whereas long-term memory is semantically dominant for generalized knowledge and contextualized semantics.
  • VisMem belongs to the latent-space paradigm, which introduces continuous latent contexts during sequential inference.
  • VisMem adds dynamically invoked latent vision memories to preserve fine-grained visual evidence and abstract semantic knowledge during generation.A query builder formulates context-aware requests, while specialized memory formers generate latent tokens that are inserted into the generation stream.
  • A two-stage reinforcement-learning training paradigm first optimizes memory formation and then learns effective memory invocation.
  • The framework is evaluated across visual understanding, reasoning, and generation, with reported improvements in cross-domain generalization, catastrophic forgetting mitigation, and base-model compatibility.

2. Related Work

Prior visual-capability methods span direct training, image-level, token-level, and latent-space paradigms, each with distinct efficiency, generalization, or expressiveness trade-offs. VisMem extends latent-space methods with short- and long-term latent vision memory integrated into generation.

  • Existing methods are organized into direct training, image-level, token-level, and latent-space paradigms for alleviating the visual processing bottleneck.
  • Direct training can sacrifice generalization through catastrophic forgetting, while image-level interventions incur high computational cost, latency, and external-tool dependence.
  • Token-level methods are more efficient but cannot modify visual evidence, whereas existing latent-space methods may rely only on language space or substantial manual labeling.
  • VisMem differs from prior latent-space methods by integrating short- and long-term latent vision memory directly within the generation process.

3. Methodology

VisMem adds dynamically invoked short- and long-term latent vision memories to VLM inference, preserving fine-grained visual evidence and contextual semantic knowledge without modifying the core VLM. A query builder and specialized memory formers create memories, while two-stage training optimizes their formation and invocation.

  • 3.1. Preliminary: VisMem separates visual retention from semantic consolidation to address VLMs’ tendency to prioritize accumulated text over initial visual evidence during long generation.Short-term memory supports immediate fine-grained perception, whereas long-term memory abstracts experiences for flexible reuse.
  • 3.2. Memory Invocation: During autoregressive generation, special invocation tokens trigger memory formation, insertion, and continued token-by-token decoding.Paired invocation and end tokens distinguish short- and long-term memory and structure the inserted latent tokens.
  • 3.3. Memory Formation: VisMem integrates short-term memory for rich visual evidence and long-term memory for high-level visual semantics, preserving the core VLM and its general abilities.The two pathways target advanced visual perception and semantic generalization during reasoning.
  • 3.3. Memory Formation: A shared query builder encodes multimodal hidden states with learnable queries, while dedicated memory formers construct short- or long-term latent memory tokens.The query builder uses masked attention from queries to hidden states; the resulting representations are processed by the corresponding memory former.
  • 3.4. Training Recipe: Two-stage GRPO training first improves memory formation and then optimizes invocation efficiency, including memory-type selection and invalid-invocation penalties.The second stage freezes memory-formation components while updating policy parameters using type and negative-return penalties.

4. Experiments

Across 12 benchmarks and additional analyses, VisMem improves visual capabilities, transfers knowledge across domains, retains knowledge during continual learning, and maintains efficiency across base models.

  • 4.2. Main Results: 11.0% average improvement over the vanilla model across 12 benchmarks establishes VisMem’s strongest overall result.It also improves over Vision-R1, VLM-R1, and OpenThinkImg by 3.0%, 4.2%, and 4.9%, respectively.
  • 4.2. Main Results: +8.9% in visual understanding, +14.4% in reasoning, and +10.6% in generation show gains across all three core task domains.Additional evaluations report gains in counting, visual retrieval, grounding, inductive learning, and deductive learning.
  • 4.2. Main Results: VisMem retains knowledge more effectively during four-stage continual learning, whereas SFT loses over 10% performance.Its performance gap relative to original full-data training is the smallest among the compared baselines, with gains at stages 1 and 3 without degradation.
  • 4.3. Additional Analyses: VisMem remains compatible across nine base models ranging from 3B to 38B parameters, while short- and long-term memories specialize differently across tasks.Short-term memory is invoked more for fine-grained detail retrieval, whereas long-term memory is more important for reasoning.
  • 4.3. Additional Analyses: VisMem achieves the best performance with limited latency overhead, while both memory components contribute and their combination produces optimal performance.Latency increases by 8.2% to 43.8% relative to the vanilla model, and performance generally improves as memory sequence lengths increase within a reasonable range.

5. Conclusion

VisMem addresses the visual processing bottleneck with cognitively inspired dynamic latent vision memory and a non-intrusive invocation mechanism. Experiments show broad performance improvements and additional robustness properties.

  • 5. Conclusion: VisMem embeds dynamic latent vision memory through dual specialized memory formers and a non-intrusive invocation mechanism.The framework is designed to address advanced visual-capability limitations in VLMs.
  • 5. Conclusion: Experiments validate improvements across visual benchmarks, alongside cross-domain generalization, catastrophic forgetting mitigation, compatibility, and efficient inference.
  • 5. Conclusion: VisMem is presented as a latent-space memory enhancement paradigm for unlocking comprehensive visual capabilities.

6. Theoretical Foundations

The theoretical foundation distinguishes short-term and long-term memory by their functional and neural underpinnings. VisMem translates these roles into dual latent memories invoked during autoregressive generation.

  • 6. Theoretical Foundations: Short-term and long-term memory are distinct storage systems differentiated by functional and neural underpinnings.The Dennis Norris Theory associates short-term memory with processing new visual information and temporary token storage.
  • 6. Theoretical Foundations: VisMem instantiates these cognitive roles as short- and long-term latent memory tokens integrated into token-by-token autoregressive generation.The framework combines latent memory formation with latent memory invocation in a VLM backbone.

7. Methodology Details

The methodology uses a masked transformer query builder and a two-stage GRPO training pipeline. Stage I optimizes memory formation, while Stage II optimizes memory invocation.

  • 7. Methodology Details: A lightweight transformer encoder receives concatenated memory queries and vision-output hidden states to produce memory hooks.The builder contains L encoder layers and encodes the query for subsequent memory formation.
  • 7. Methodology Details: The attention mask permits memory queries to attend to hidden states while blocking reverse-direction attention.This mask imposes the intended information-flow direction in the query builder.
  • 7. Methodology Details: The two-stage GRPO pipeline first optimizes memory formation and then optimizes memory invocation.For each instruction-vision pair, the policy generates grouped candidate trajectories whose performance determines a relative baseline.
  • 7. Methodology Details: Group-relative advantages are formulated after averaging and standardizing trajectory performance within each candidate group.
  • 7. Methodology Details: Stage I applies reinforcement learning to optimize the memory formation objective.
  • 7. Methodology Details: Stage II applies reinforcement learning to optimize the memory invocation objective.

8. Experiment Details

The experiments use shared training procedures and evaluate VisMem against diverse baselines on 12 benchmarks spanning understanding, reasoning, and generation. The setup includes transferred methods and documented configuration choices.

  • 8. Experiment Details: The two-stage training uses benchmark training splits and additionally incorporates Visual CoT and Mullberry data.Benchmarks without training phases are used only for evaluation.
  • 8. Experiment Details: The evaluation covers 12 benchmarks: five for understanding, four for reasoning, and three for generation.The benchmark suite includes tasks involving visual understanding, hallucination, trustworthiness, and generation quality.
  • 8. Experiment Details: The comparison includes 16 baselines across direct-training, image-level, token-level, and latent-space paradigms.
  • 8. Experiment Details: VisMem is trained on Qwen-2.5-VL-7B, while methods developed for other backbones are transferred for fair comparison.Most counterparts use identical training datasets, with exceptions for methods requiring specially curated data.
  • 8. Experiment Details: Experiment configurations cover core hyperparameters, LoRA-adapter parameters, and training parameters.The configurations are listed in Table 4.

9. Additional Results

VisMem improves fine-grained visual understanding and reasoning across selected benchmark subsets, generalizes across domains, and mitigates catastrophic forgetting during continual learning.

  • Capability boundaries: VisMem achieves the best results on 7 MuirBench subsets and second-best on the remaining two, with at least 5% generalized improvement over the base model.It improves counting, grounding, geographic, and retrieval by 16.7%, 18.2%, 11.8%, and 13.7%, respectively, over the vanilla model.
  • Capability boundaries: VisMem ranks first on 8 LogicVista subsets and improves over the base model by at least 7%, including gains of 14.8% on inductive and deductive reasoning.Its improvements reach 18.4% on graph-based and 21.1% on table-based subtasks.
  • Cross-domain generalization: With training limited to Visual CoT and Mulberry, VisMem shows smaller performance drops than the fully trained model across all four evaluated benchmarks.Its 2.3% drop compares with 5.3% for VLM-R1, 2.1% for OpenThinkImg, and 1.1% for MINT-CoT.
  • Continual learning: VisMem consistently achieves the best performance at every continual-learning stage and retains 72.1% performance on MMVet after four-stage sequential training.This exceeds DeepEyes at 68.4% and Mirage at 67.0%.

9.4. Versatility across Various Base Models

VisMem transfers across diverse VLM families and model sizes, while its memory invocation strategy yields a performance–efficiency trade-off and its full dual-memory design performs best overall.

  • Versatility across base models: VisMem consistently enhances visual capabilities across 9 base models spanning 3B to 38B parameters and three VLM families.For widely used 7B or 8B models, it provides substantial performance gains.
  • Versatility across base models: Smaller base models generally obtain larger gains, while larger models show more significant improvements on dense reasoning benchmarks.The authors attribute this pattern to baseline-score differences and reasoning bottlenecks.
  • Invocation and dual-memory ablations: The complete VisMem framework achieves the highest performance across all benchmarks, whereas excessive memory invocation reduces efficiency without further performance benefits.Random invocation performance peaks at a 75% probability before declining as invocation probability increases toward 100%.

9.6. Analysis of Latent Vision Memory

The two memory components serve different task needs, and VisMem dynamically adjusts when and how often memory is invoked to balance performance with computational cost.

  • Component roles: Short-term memory supports counting, grounding, and retrieval, while long-term memory strengthens complex visual reasoning through contextualized semantic knowledge.The component advantages vary across visual subtasks and benchmarks.
  • Invocation behavior: VisMem dynamically adjusts memory invocation position and frequency according to task characteristics, balancing efficiency and performance.Invocation ratio and relative position vary across memory components, output locations, and benchmarks.
  • Sequence-length sensitivity: Increasing the memory-query, short-term-memory, and long-term-memory sequence lengths from 2 to 32 improves performance but increases computational cost.This sensitivity analysis evaluates K, N_s, and N_l on MMVet.
  • Efficiency: VisMem achieves an optimal performance–efficiency balance with minimal additional time overhead compared with methods such as image-level interventions.Image-level paradigms require nearly twice the vanilla model’s inference time.
Loading 2511.11007v2…