PaperlayerView exact revision on arXiv

Source-linked AI summary

Persistent Visual Memory: Sustaining Perception for Deep Generation in LVLMs

Siyuan Huang, Xiaoye Qu, Yafu Li, Tong Zhu, Zefeng He, Muxin Fu, Daizong Liu, Wei-Long Zheng, Yu Cheng

arXiv:2605.00814v2cs.CVcs.AI

TL;DR

Autoregressive LVLMs suffer Visual Signal Dilution as growing textual histories attenuate visual attention, motivating sustained, on-demand visual access. PVM adds a parallel retrieval pathway, yielding 4.8% average accuracy improvement on Qwen3-VL 8B and 4.4% on 4B across multimodal benchmarks.

  • Problem

    Growing textual histories progressively attenuate visual signals in autoregressive LVLMs, creating a need for sustained, on-demand visual perception.

  • Method

    PVM adds a lightweight parallel branch beside the FFN that uses gated cross-attention for distance-agnostic retrieval from visual embeddings.

  • Results

    4.8% average accuracy improvement over the baseline is achieved on Qwen3-VL 8B, with a consistent 4.4% improvement on 4B across eight multimodal benchmarks.

  • Takeaways & Limitations

    PVM helps decouple visual memory retention from growing autoregressive context, improving access to visual evidence during extended generation with negligible parameter overhead.

  • Takeaways & Limitations

    Theoretical guarantees assume average textual attention mass is bounded below by µ > 0 and condition structural mitigation on a fixed local hidden state.

Abstract

from arXiv · show

While autoregressive Large Vision-Language Models (LVLMs) demonstrate remarkable proficiency in multimodal tasks, they face a "Visual Signal Dilution" phenomenon, where the accumulation of textual history expands the attention partition function, causing visual attention to decay inversely with generated sequence length. To counteract this, we propose Persistent Visual Memory (PVM), a lightweight learnable module designed to strengthen sustained, on-demand access to visual evidence. Integrated as a parallel branch alongside the Feed-Forward Network (FFN) in LVLMs, PVM establishes a distance-agnostic retrieval pathway that directly provides visual embeddings for enhanced visual perception, thereby structurally mitigating the signal suppression inherent to deep generation. Extensive experiments on Qwen3-VL models demonstrate that PVM brings notable improvements with negligible parameter overhead, delivering consistent average accuracy gains across both 4B and 8B scales, particularly in complex reasoning tasks that demand persistent visual perception. Furthermore, in-depth analysis reveals that PVM shows improved robustness in longer generations and accelerates internal prediction convergence.

1 Introduction

The introduction identifies Visual Signal Dilution as a structural loss of visual attention during autoregressive generation and proposes Persistent Visual Memory (PVM) as a parallel retrieval pathway for sustained visual perception. Across eight multimodal benchmarks, PVM improves average accuracy on Qwen3-VL 4B and 8B models, with greater benefits in longer generations and faster internal prediction convergence.

  • Problem: Visual Signal Dilution arises because accumulating textual history redistributes attention over an expanding context while visual tokens cannot regenerate intrinsically.This creates a structural conflict in standard autoregressive LVLMs.
  • Motivation: PVM addresses the need for sustained, on-demand perception by enabling models to dynamically re-examine visual memory during deep generation.This shifts visual processing from passive retention toward sustained access to visual evidence.
  • Method: PVM is a lightweight learnable module integrated alongside the FFN, creating a dedicated parallel channel for retrieving raw visual evidence.The original FFN preserves reasoning logic while PVM provides distance-agnostic visual retrieval without serially perturbing generation.
  • Results: PVM is more beneficial in longer generations and accelerates internal prediction convergence.These analyses indicate improved robustness when visual information must remain accessible over extended generation.

2 Related Work

Related work develops LVLMs by connecting visual encoders to language models and addresses visual-signal volatility through architectural and context-management strategies.

  • General LVLMs and Challenges in Visual Persistence: LVLMs bridge pretrained visual encoders with LLMs through learnable interfaces such as linear layers, Q-Formers, and advanced variants.These models have achieved proficiency across diverse tasks, including visual perception.
  • Visual Injection and Context Management: Recent research addresses volatile visual signals in extended contexts through hierarchical systems, retrieval-augmented frameworks, token compression, and bottleneck mechanisms.Hierarchical and retrieval-augmented approaches decouple storage from processing, while compression methods reduce feature granularity via resampling.

3 Analysis of Visual Signal Dilution

Standard self-attention structurally dilutes visual attention as textual history grows, with empirical analyses confirming inverse-length decay, intermediate-layer attenuation, and a persistent low-attention equilibrium. These findings motivate an isolated visual-memory pathway with independent normalization for on-demand retrieval.

  • 3.1 Theoretical Analysis: A fixed visual context competes with an expanding textual history because ZV remains bounded while ZT aggregates over t tokens.The partition function separates visual and textual mass, making textual-history growth the structural driver of dilution.
  • 3.1 Theoretical Analysis: Theorem 3.1 shows that visual attention mass is bounded by β/(β+µ·t), yielding ΩV(t) = O(t^-1) as textual history grows.The fixed visual mass is upper-bounded by β, while textual mass grows at least linearly under the persistent textual attention condition.
  • 3.1 Theoretical Analysis: Even after textual attention saturates, textual mass remains dominant because the effective window Weff is orders of magnitude larger than the visual context M.The resulting low-attention equilibrium leaves visual signals structurally outweighed and increases vulnerability when precise visual grounding is required.
  • 3.2 Empirical Validation: Empirical analysis on Qwen3-VL-8B-Instruct shows visual attention mass closely follows the predicted O(t^-1) decay.The Blind Painter stress test requires active visual retrieval throughout detailed image reconstruction rather than only at the beginning.
  • 3.2 Empirical Validation: Attenuation is pronounced in Layers 8–27, the primary locus for multimodal reasoning, despite these layers retaining the highest relative visual relevance.As text expands, these layers operate on a progressively weakening visual foundation.
  • Design Implications: The analysis motivates a parallel visual-memory pathway structurally isolated from the reasoning backbone and independently normalized within the visual domain.Independent normalization prevents textual dominance in the partition function.
  • 3.2 Empirical Validation: TVR initially grows linearly and then reaches a plateau where textual mass exceeds visual mass by nearly two orders of magnitude.This two-phase trajectory corresponds to active textual competition followed by a low-attention equilibrium.

4 Method: Persistent Visual Memory

PVM adds a parallel visual-memory branch beside the frozen FFN, enabling hidden-state queries to retrieve visual details through an independently normalized visual pathway. Its gated, selectively activated bottleneck design structurally decouples visual retrieval from growing textual history, mitigating visual signal dilution under a fixed local-query assumption.

  • Parallel Architecture: PVM is integrated as a parallel Transformer-decoder branch beside the frozen FFN, functioning as an active perception channel for the current visual context.The hidden state bifurcates into an original reasoning path through the FFN and a PVM looking path that queries visual memory.
  • PVM Computation: The PVM looking path uses the hidden state as Query and restricts Cross-Attention Keys and Values to projected visual features, creating independent visual-only normalization.A lightweight latent FFN follows retrieval, and the refined representation is restored to the original model dimension.
  • Gated Fusion and Activation: PVM injects retrieved visual features through a learnable gate and text-token silencing mask, with λ initialized to 0 and the output preserving the hidden-state shape.This selectively activates visual injection during text generation while preserving pretrained capabilities and enabling seamless backbone integration.
  • Theoretical Analysis: Under a fixed local hidden state, PVM’s retrieval representation is locally independent of sequence length because its partition function sums only over the fixed visual set.Theorem 4.1 contrasts this with the standard backbone, where visual relevance decays as ΩV(t) ∈ O(t−1).
  • Theoretical Analysis: The parallel pathway and fixed visual-domain normalization structurally isolate visual grounding from probability competition with growing textual history.These mechanisms realize the two design principles identified in Section 3.2: a parallel memory pathway and independent attention normalization.

5 Experiments

The experiments evaluate PVM on Qwen3-VL 4B and 8B models across eight multimodal benchmarks, using staged visual-memory alignment and policy refinement. PVM is compared with internal training baselines and visual-injection methods, with PVM (SFT + GRPO) achieving the best overall performance.

  • Models and Data: PVM adds 27.92M trainable parameters, approximately 0.32% of the 8B model, through modules inserted into selected intermediate Transformer layers.The integration follows the configuration in DeepStack.
  • Training Details: Training uses two stages: frozen-backbone SFT for visual-memory alignment, followed by GRPO policy refinement with the LLM and PVM unfrozen and the Vision Encoder frozen.The stages establish query-to-visual-key mapping and encourage active visual retrieval during complex reasoning.
  • Baselines: Baselines include original Qwen3-VL, SFT, LoRA, RL-enhanced variants, and representative visual-injection methods such as MemVR and ICoT.This design separates architectural gains from improvements attributable to data quality or reinforcement-learning alignment.
  • Evaluation Benchmarks: Evaluation covers eight multimodal benchmarks spanning general understanding, mathematics, and science, reporting 4-run average accuracy at temperature 0.7.The benchmarks include MMMU, MMBench-CN, MMBench-EN, MMStar, MMT, MathVerse, MathVision, and AI2D, evaluated with lmms-eval.
  • Main Results: PVM (SFT + GRPO) achieves the best overall performance, with consistent gains across comprehensive understanding and specialized reasoning benchmarks.Table 1 reports 4-run average accuracy (%) across eight benchmarks and compares changes against the original model baseline.

6 Results

PVM achieves strong benchmark performance, with PVM-SFT reaching 70.6% on the 8B backbone and outperforming several baselines and existing methods. Its gains increase with generation length, accelerate prediction convergence, and depend on raw visual retrieval and broad layer coverage.

  • Overall Results: 70.6% overall score: PVM-SFT on the 8B backbone outperforms vanilla SFT, LoRA-SFT, CoMemo, and ICoT.It also mitigates degradation on perception tasks such as MMT observed with standard fine-tuning.
  • Extended-Generation Robustness: +27.3% relative improvement: PVM’s gain on MathVerseV “Long” samples exceeds its +6.1% gain on “Very Short” samples.The comparison uses PVM-8B (SFT + GRPO) against Qwen3-VL-8B-Instruct, with samples grouped by output length.
  • Prediction Convergence: PVM produces a lower KL-divergence trajectory than baselines, with an Improvement Gap emerging after Layer 8 and widening in deeper layers.The analysis uses LogitLens on the “Blind Painter” test to examine prediction readiness under high visual dependency.
  • Retrieval-Source Ablation: Replacing raw visual embeddings with processed hidden states causes catastrophic collapse across reasoning benchmarks.The ablation re-trains the replacement variant under the identical two-stage pipeline, isolating raw visual retrieval as the source of improvement.
  • Injection-Strategy Ablation: 70.6%: the Strided Strategy outperforms Max Decay at 69.3% and Peak Attention at 68.4% on the 8B model.Its Layers 8, 16, and 24 configuration spans the network’s depth and yields a +1.8% reasoning gain over the decay-focused approach.

7 Conclusion

Persistent Visual Memory (PVM) addresses Visual Signal Dilution by providing a dedicated parallel pathway for active visual retrieval during extended generation. It improves access to visual evidence across diverse benchmarks with negligible parameter overhead.

  • Conclusion: PVM addresses Visual Signal Dilution through a dedicated parallel pathway for active visual retrieval.This pathway helps decouple visual memory retention from the growing autoregressive context.
  • Conclusion: PVM improves access to visual evidence during extended generation by decoupling retention from the growing autoregressive context.
  • Conclusion: PVM delivers notable improvements across diverse benchmarks with negligible parameter overhead.

A Visual Attention Mass Heatmap Analysis · B Discussion on the Fixed Local Query Assumption

Figure 7 shows that visual attention decays most severely in intermediate reasoning layers as generation length grows, motivating PVM placement there. The fixed-local-query analysis isolates textual partition growth as the structural source of dilution while acknowledging query drift during real decoding.

  • A Visual Attention Mass Heatmap Analysis: The heatmap tracks visual attention mass ΩV across all 36 Qwen3-VL-8B-Instruct layers and generated-token positions, with darker regions indicating lower attention.ΩV is measured during the Blind Painter long-form description stress test as visual-token probability mass divided by total visual-plus-textual mass.
  • A Visual Attention Mass Heatmap Analysis: Visual attention decay is nonuniform across depth, forming three architectural zones rather than a uniform decline.The heatmap’s layer-wise dynamics distinguish shallow, intermediate, and deep regions.
  • A Visual Attention Mass Heatmap Analysis: ΩV < 0.05 in shallow layers 0–7 throughout generation, consistent with their focus on local syntax and shallow textual features.These layers therefore require minimal multimodal integration according to the passage.
  • A Visual Attention Mass Heatmap Analysis: Intermediate layers 8–27 begin with ΩV > 0.10 but collapse to negligible visual mass as sequence length increases, identifying the critical dilution zone.This observation motivates injecting PVM at layers 8, 16, and 24.
  • A Visual Attention Mass Heatmap Analysis: Deep layers 28–35 revert to text-dominated processing, likely emphasizing output formatting and next-token prediction distributions.Their late-layer behavior differs from the intermediate layers’ semantic-grounding role.
  • B Discussion on the Fixed Local Query Assumption: During autoregressive decoding, the PVM input hidden state xt depends on the entire preceding visual and textual context, so it drifts as textual history grows.Consequently, the final PVM output hpvm cannot be absolutely invariant to t across the full global trajectory.
  • B Discussion on the Fixed Local Query Assumption: Conditioning on a fixed local query x and constant visual keys makes ∂∥hpvm∥/∂t = 0, isolating textual partition expansion from semantic query evolution.The result proves structural isolation of PVM’s partition function from expanding textual mass, shielding the visual branch from long-text probability competition.

C Implementation Details

The implementation uses a two-stage training strategy for PVM-enhanced Qwen3-VL models, supported by stage-specific system, model, data, and hyperparameter configurations. PVM is inserted into selected intermediate layers with a 512-dimensional bottleneck and zero-initialized gating for stable warm-up.

  • System Infrastructure: Experiments run on 8 × NVIDIA H200 GPUs with PyTorch, Hugging Face transformers and trl, using ZeRO-2 for SFT and ZeRO-3 for GRPO.The infrastructure is configured for training efficiency and long-context processing.
  • Model Configuration: PVM modules are inserted at layers {8, 16, 24} for 8B and {5, 11, 17} for 4B models, with bottleneck dimension d′ = 512 and gating scalar α initialized to 0.The zero initialization supports stable warm-up by gradually incorporating the visual memory branch.
  • Data Curation: The SFT dataset contains 526k visually centered, clear-answer samples filtered from OpenMMReasoner-SFT-874K, while GRPO uses 3.6k queries with 8 reasoning rollouts each.GRPO retains samples exhibiting the strongest learning signals for policy optimization.
  • Training Strategy: Training uses Stage I Visual Memory Alignment to initialize PVM parameters and Stage II Policy Refinement with GRPO for complex reasoning.Table 4 reports the detailed hyperparameters for both stages.

D Decoupling Sequence Length from Problem Difficulty … G Impact of Latent Dimension Size

Controlled analysis shows PVM’s gains increase with generation length even at matched difficulty, while LogitLens, attention profiling, and latent-dimension ablations characterize its convergence, injection layers, and capacity limits.

  • D Decoupling Sequence Length from Problem Difficulty: Within matched difficulty tiers, PVM’s absolute gain increases with sequence length, rising from +4.00% on short to +15.71% on long Medium sequences.Samples were stratified into Medium and Hard difficulty tiers, then subdivided into Short, Medium, and Long token-length groups.
  • D Decoupling Sequence Length from Problem Difficulty: The upward trend after controlling for difficulty supports PVM’s benefit in longer generations by mitigating length-associated visual dilution.The analysis was designed to separate sequence length from the intrinsic difficulty of tasks requiring longer reasoning outputs.
  • E LogitLens Analysis Formulation: LogitLens projects each layer’s hidden state into vocabulary space and compares its distribution with the final output using KL divergence.A lower KL divergence indicates that an intermediate layer more closely approximates the model’s final decision.
  • F Detailed Analysis of Injection Layer Selection: Visual attention profiling reveals a Rise-Peak-Decay pattern across Qwen3-VL-8B-Instruct layers, motivating data-driven PVM injection strategies.Mean Visual Mass averages visual attention mass across samples and generation steps.
  • F Detailed Analysis of Injection Layer Selection: The Peak Attention strategy selects layers 13, 17, and 18 to reinforce locations where the model already seeks visual information.Layer 18 is the global maximum, while Layers 13 and 17 form secondary peaks.
  • F Detailed Analysis of Injection Layer Selection: The Max Decay strategy selects layers 14, 19, and 22 to compensate for bottlenecks where visual context is rapidly discarded.These layers follow significant attention drops after peaks at Layers 13, 18, and 21.
  • G Impact of Latent Dimension Size: A latent dimension of d′ = 512 provides optimal performance and parameter efficiency, whereas increasing it to 1024 or 2048 slightly regresses performance.The regression is attributed to a mismatch between the larger parameter space and the available fine-tuning supervision.

H Iso-Parameter Control Analysis

The iso-parameter control shows that PVM’s gains arise from active visual retrieval rather than increased parameter capacity. With matched parameters and identical SFT+GRPO training, PVM-8B outperforms the MLP baseline across all eight benchmarks.

  • Experimental Design: The experiment tests whether PVM’s improvements stem from active visual retrieval rather than increased parameter capacity.The control is designed to isolate the contribution of PVM’s visual retrieval mechanism.
  • Baseline Design: The parallel MLP baseline exactly matches PVM’s parameter count but removes visual cross-attention and retrieves no raw visual signals.It relies solely on processed hidden states.
  • Experimental Design: Both models share the exact same parameter count and identical SFT+GRPO training pipeline.The MLP baseline is trained from scratch under the same two-stage procedure as the default PVM model.
  • Results and Analysis: 2.5% higher average score is achieved by PVM-8B (SFT+GRPO) across all 8 evaluation benchmarks despite equal capacity and identical RL optimization.The MLP baseline consistently underperforms PVM-8B (SFT+GRPO).
  • Results and Analysis: The gains are attributed to PVM’s dynamic retrieval and integration of preserved visual signals during reasoning, not passive capacity expansion or regularization.The comparison supports the active visual retrieval mechanism as the source of the performance improvement.

I Computational Overhead Analysis … M Limitations and Future Work

The merged sections specify PVM’s modest inference overhead, its parallel visual-retrieval computation, evaluation prompts, societal-impact framing, and current validation limitations. Together, they describe implementation details and boundaries for interpreting the method’s empirical evidence.

  • I Computational Overhead Analysis: PVM adds only 1.18 ms TPOT and reduces throughput by 4.6% through its parameter-efficient parallel branch.The added computation follows a Projection → Cross-Attention → Fusion graph expansion.
  • J PVM Inference Algorithm: The inference algorithm bifurcates after self-attention into a frozen FFN reasoning path and an active PVM visual-retrieval path before unified fusion.The final output combines the residual state, FFN output, and gated visual injection.
  • J PVM Inference Algorithm: PVM compresses queries and keys into a latent space and applies a textual-token mask so only text triggers visual retrieval.The latent path performs cross-attention, latent FFN processing, restoration, and gated injection.
  • K Prompt Templates: The Blind Painter prompt tests visual signal dilution by demanding exhaustive, extended descriptions that maintain continuous dependence on the image.It requests microscopic detail and generation until the token limit.
  • K Prompt Templates: The structured reasoning template requires an internal monologue in <think> tags before the final answer in <answer> tags.This format follows the OpenMMReasoner-SFT-874K setting for policy refinement and evaluation.
  • L Societal Impacts: PVM is presented as improving visual fidelity and mitigating visual hallucinations without inherently introducing societal risks beyond those associated with base LVLMs.The stated application contexts include scientific reasoning and visual assistants.
  • M Limitations and Future Work: Empirical validation currently covers Qwen3-VL 4B and 8B models, motivating evaluation across broader LVLM architectures and larger parameter scales.The authors describe PVM’s parallel design as theoretically backbone-agnostic.
Loading 2605.00814v2…