PaperlayerView exact revision on arXiv

Source-linked AI summary

Just MLPs: Efficient Visual State Reconstruction for Multimodal Language Models

Jingdi lei, Junxian Li, Di Zhang, Zhanqiu Zhang, Yiwen Guo, Soujanya Poria

arXiv:2609.34972v1cs.CVcs.AI

TL;DR

Long visual-token sequences add computation because tokens are processed at every Transformer layer, while pruning can permanently remove useful visual evidence. δ-Vision predicts layer-wise visual memories with low-rank adapters while preserving all visual tokens for text retrieval. Across benchmarks it achieves a favorable accuracy–efficiency trade-off relative to pruning, including a 9.1-point average-score gain over DivPrune on Qwen3-VL-4B.

  • Problem

    Long visual-token sequences add MLLM computation, while pruning reduces sequence length by irreversibly discarding visual evidence that later layers may need.

    Source: §1 · p. 1

  • Method

    δ-Vision replaces repeated visual-token Transformer updates with low-rank adapters that construct layer-wise visual memories while keeping all tokens available for text retrieval.

    Source: §1 · pp. 2–3

  • Results

    Across MLLM backbones and image, multi-image, and video benchmarks, δ-Vision achieves a favorable accuracy–efficiency trade-off; on Qwen3-VL-4B its 74.4 average score exceeds DivPrune by 9.1 points.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    δ-Vision offers an alternative to pruning: reduce visual-token computation while preserving every visual token for text retrieval.

    Source: §1 · p. 2, §6 · p. 10

    δ-Vision assumes each layer-specific visual-memory displacement can be represented within an at-most-r-dimensional hidden-channel subspace.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

Long visual token sequences often account for a substantial fraction of the computational overhead in multimodal large language models~(MLLMs). Existing approaches reduce this cost by pruning redundant visual tokens, but permanently discard visual evidence that may become useful in subsequent layers. We instead ask whether all visual tokens can be preserved while reducing the cost of repeatedly evolving the representations through the Transformer. To answer this question, we perform low-rank interventions on visual-to-text information flow. We find that, after visual-to-text attention is blocked, restoring only a few directions recovers most of the lost accuracy, suggesting the relevant visual influence is concentrated in a low-dimensional subspace. We further observe strong predictability in layer-specific visual states: lightweight MLPs approximate them with high cosine similarity and low reconstruction error. Motivated by these findings, we propose $δ$-Vision, which replaces repeated Transformer evolution of visual tokens with lightweight low-rank adapters that construct layer-wise visual memories while preserving all visual tokens for text retrieval. Across image and video benchmarks, $δ$-Vision achieves higher accuracy than visual token pruning baselines at comparable or lower computation, while delivering competitive inference efficiency without discarding visual tokens.

1 Introduction

Long visual-token sequences raise MLLM computation, while pruning can irreversibly remove evidence. The introduction motivates δ-Vision, which preserves visual tokens while reducing their repeated Transformer computation, and reports favorable accuracy–efficiency results.

1 Introduction

  • Pruning shortens visual sequences but irreversibly removes evidence that later layers could use, motivating preservation of all tokens while reducing their processing cost.The introduction asks whether visual tokens need full computation at every Transformer layer or whether their necessary information can be integrated into the text stream more efficiently.Source: §1 · pp. 1–2
  • Low-rank projections retain much original performance, and lightweight per-layer MLPs predict Transformer-evolved visual states with high cosine similarity and low reconstruction error.Figure 1 presents downstream accuracy across retained ranks and layer-wise reconstruction quality using cosine similarity and reconstruction MSE.Source: §1 · p. 2, Figure 1 · p. 2
  • δ-Vision replaces visual-token self-attention and feed-forward updates with low-rank adapters that construct layer-wise visual memories while keeping all tokens available for text retrieval.The language-model backbone stays frozen; predicted memories pass through the original frozen key and value projections.Source: §1 · pp. 2–3
  • On Qwen3-VL-4B, δ-Vision scores 74.4 on average under a compute budget comparable to methods discarding 95% of visual tokens, beating the strongest baseline by 9.1 points and matching the best result retaining 20%.Source: §1 · p. 2
  • Across MLLM backbones and image, multi-image, and video benchmarks, δ-Vision achieves stronger accuracy–efficiency trade-offs than visual-token pruning under comparable computational budgets.On Video-MME, it uses 17.90% of uncompressed-model FLOPs, with 1.30× total and 1.50× prefill speedup.Source: §1 · pp. 2–3

2 Preliminaries

Visual and textual embeddings are concatenated and processed together through the Transformer. Visual tokens provide context to text but are also repeatedly updated as active Transformer states, adding computation for long sequences.

2 Preliminaries

  • A vision encoder and multimodal projector convert visual input into visual embeddings, which are concatenated with text embeddings for processing by an L-layer Transformer.The input contains Nv visual embeddings; d denotes the language model's hidden dimension.Source: §2 · p. 3
  • At each layer, visual tokens serve as context through keys and values for textual queries and as active states that generate queries, receive attention, and pass through feed-forward computation.The resulting visual representations evolve across layers, rather than exposing only the initial visual embeddings to text.Source: §2 · p. 3
  • Because all visual tokens undergo attention and feed-forward computation at every layer, long image or video sequences make repeated visual-state evolution a substantial inference cost.The stated goal is reducing the computation needed to obtain layer-specific visual representations, not reducing the number of visual tokens.Source: §2 · p. 3

3 δ-Vision

δ-Vision replaces visual-token Transformer updates with layer-wise low-rank memory predictors while leaving textual computation unchanged and retaining visual memories as text-accessible key–value context. It trains those adapters by distilling from the frozen MLLM.

3 δ-Vision

  • δ-Vision constructs layer-wise visual memories with lightweight adapters instead of full visual-token Transformer updates, while keeping textual computation unchanged.The memories are projected through the original key–value modules and accessed by text queries.Source: §3 · p. 3

3.2 Visual Memory as Context

  • Text queries retrieve from predicted visual key–value memories, while visual queries, visual attention outputs, and visual feed-forward updates are omitted.Each layer’s memory is external context: text computation does not update it, and the next layer’s memory is constructed separately.Source: §3 · p. 3, §3.2 · p. 5

3.1 Layer-Wise Visual Memory Prediction

  • The memory remains full-dimensional; only each residual correction is restricted to a hidden-channel subspace of dimension at most r.The adapter initializes as an identity mapping because its up-projection is zero-initialized.Source: §3.1 · pp. 3–4
  • The embedding adapter predicts each layer’s memory independently from initial embeddings, whereas the recurrent adapter propagates memory through successive low-dimensional corrections.The recurrent variant preserves an explicit layer-to-layer state; the embedding variant can construct layer memories without propagating through preceding Transformer layers.Source: §3.1 · p. 4

3.3 Training Objective

  • Supervised-KD trains only the visual-memory adapters to match the frozen teacher’s next-token distributions under teacher forcing, using forward KL on answer positions.It achieves the best average performance among the compared training objectives while avoiding the substantial rollout cost of on-policy training.Source: §3.3 · p. 5

4 Experiments

Experiments evaluate δ-Vision against token-pruning methods across image, multi-image, and video benchmarks, as well as across model backbones. They also analyze the dimensionality and layer-wise importance of visual computation.

4.1 Experimental Setup

  • The experiments cover Qwen3-VL-4B-Instruct by default, single-image, multi-image, and video benchmarks, and comparisons with training-free and training-based compression methods.The overall score is the unweighted mean of benchmark scores; primary pruning comparisons use 20% and 5% visual-token retention.Source: §4.1 · pp. 6–7

4.2 Main Results

  • 17.90% of uncompressed FLOPs: δ-Vision achieves 1.30× end-to-end and 1.50× prefill speedups on Video-MME while retaining all visual tokens.Its computation is comparable to methods retaining 5% of visual tokens, whose FLOPs range from 16.58% to 23.90% of the uncompressed model.Source: Table 2 · p. 7, §4.2 · p. 7
  • Across three additional backbones, δ-Vision retains approximately 94–96% of vanilla performance; on Qwen3-VL-30B-A3B it scores 78.6 versus DivPrune’s 71.3 at 5% retention.On Qwen3.5-4B, the corresponding scores are 71.6 for δ-Vision and 62.7 for DivPrune.Source: §4.2 · p. 7, Table 3 · p. 8
  • 74.4 average score: δ-Vision outperforms the strongest 5%-retention pruning baseline by 9.1 points and slightly exceeds the best 20%-retention pruning result on Qwen3-VL-4B.The embedding adapter reaches 74.4, and the recurrent adapter raises the average to 75.4.Source: §4.2 · pp. 7–8

Qwen3.5-4B

  • Across multi-image and video benchmarks, the embedding adapter’s average rises from 49.5 with single-image-only training to 52.0 with multi-image/video data, surpassing the strongest 5%-retention pruning baseline (50.2).Largest gains were on MuirBench (40.7 to 45.5) and MVBench (57.4 to 59.5).Source: Qwen3.5-4B · p. 8, Table 4 · p. 8

4.3 How Much Visual Computation Is Actually Necessary?

  • Visual attention outputs need only 38.0–103.5 directions on Qwen3-VL-4B and 29.6–37.6 on LLaVA-1.5-7B to retain 95% of spectral energy.The study characterizes visual queries, per-head QKᵀ matrices, and visual attention outputs using r95 and effective rank.Source: §4.3 · p. 8, Table 5 · p. 9
  • Restoring 128 directions after blocking visual information across Qwen3-VL-4B’s layers recovers RWQA from 46.1 to 70.7 and MMStar from 25.6 to 64.6.Visual-information dependence varies by depth and task: middle layers matter most for Qwen3-VL-4B, while LLaVA-1.5-7B shows a more distributed, task-dependent pattern.Source: §4.3 · p. 9, Table 6 · p. 10

5 Related Work

Related work reduces visual computation through token compression or changes to layer-wise visual processing; δ-Vision instead predicts visual states while retaining every token.

Visual Representation Redundancy

  • Methods including EE-MLLM (Ma et al., 2024) and ShortV (Yuan et al., 2025) remove visual self-attention or estimate layer importance, while δ-Vision learns an alternative pathway for visual states.LRCP (Lu et al., 2026) uses PCA to identify dominant visual subspaces and retains tokens poorly represented by them.Source: §5 · p. 10
  • Unlike KV Prediction (Horton et al., 2024), which predicts a larger model’s KV cache, δ-Vision predicts visual memory before KV projection and reuses frozen key and value projections.Source: §5 · p. 10

Token Reduction for Efficient MLLMs

  • Token-reduction methods select, merge, or compress visual tokens, whereas δ-Vision constructs key-value context for all visual tokens at every layer.SwiftVLM (Qian et al., 2026) lets removed tokens re-enter through cross-layer bypasses and residual connections; δ-Vision changes how visual states are generated, not which tokens are retained.Source: §5 · p. 10

6 Conclusion

δ-Vision replaces repeated visual-token Transformer computation with lightweight low-rank adapters while preserving tokens for text retrieval. Across image, multi-image, and video benchmarks, it achieves a favorable accuracy–efficiency trade-off relative to pruning.

6 Conclusion

  • Across multiple MLLM backbones and image, multi-image, and video benchmarks, δ-Vision achieves a favorable accuracy–efficiency trade-off relative to visual-token pruning while preserving all visual tokens.The conclusion presents efficient construction of visual states as a complementary axis to sequence-length reduction.Source: §6 · p. 10

A Additional Implementation and Training Details

  • Training freezes the vision encoder, multimodal projector, and language-model backbone, optimizing only the visual-memory adapters.The default adapter bottleneck rank is 128; single-image training uses 2,000 steps, while multi-image and video training uses 4,000 steps.Source: A Additional Implementation and… · p. 15

B Additional Token-Retention Results

At 15% and 10% visual-token retention, DivPrune is the strongest pruning baseline, while δ-Vision scores 74.4 with all visual tokens preserved. Additional results cover multi-image and video benchmarks at 20% token retention.

B Additional Token-Retention Results

  • δ-Vision scores 74.4 while preserving all visual tokens, above DivPrune’s 72.7 at 15% retention and 71.1 at 10% retention.DivPrune is the strongest pruning baseline at both retention levels, and performance decreases as fewer visual tokens are retained.Source: B Additional Token-Retention Results · p. 15, Table 7 · p. 15

C Additional Result on Multi-Image and Video

D Additional Efficiency Results

On Video-MME, δ-Vision uses 17.90% of the uncompressed model’s FLOPs and achieves 1.30× total and 1.50× prefill speedups while retaining all visual tokens.

D Additional Efficiency Results

  • 17.90% of uncompressed-model FLOPs yields δ-Vision 1.30× total and 1.50× prefill speedups on Video-MME while preserving all visual tokens.Pruning baselines at 20% retention use 29.05–36.36% of FLOPs and achieve 1.13×–1.25× total speedups.Source: Table 9 · p. 16, D Additional Efficiency Results · p. 16
  • Table 9 compares visual-token pruning methods at 20% retention; Total Speedup includes both prefill and decoding.Pruning-baseline prefill speedups range from 1.15× to 1.34×, and decode speedups range from 1.06× to 1.18×.Source: Table 9 · p. 16, D Additional Efficiency Results · p. 16

E Training Objective Ablation

Among three embedding-adapter training objectives, Supervised-KD achieves the highest average score with substantially less training time than on-policy distillation. All compared variants optimize only adapter parameters.

E Training Objective Ablation

  • The ablation compares SFT, OPD, and Supervised-KD, with every variant optimizing only adapter parameters.The shared-representation embedding-adapter initialization is the lower-bound baseline and has no learned layer-specific adaptation.Source: Table 10 · p. 17, E Training Objective Ablation · p. 17
  • Supervised-KD scores 74.4 in 34m53s, versus OPD’s 74.0 in 4h38m and SFT’s 68.9 average score.Supervised-KD reduces training time by approximately 8× relative to OPD.Source: E Training Objective Ablation · p. 17

F Additional Cross-Backbone Results

Across backbones and retention settings, δ-Vision remains competitive with token-pruning methods; combining adapter-based visual-state compression with pruning preserves much of the adapter’s performance at moderate pruning. Skipping adapters in selected layers further trades a small accuracy reduction for lower computation and faster inference.

F Additional Cross-Backbone Results

  • At 20% retention, δ-Vision scores 73.8, 78.6, and 71.6 on three Qwen models, close to DivPrune’s respective scores of 75.0, 78.8, and 70.8.On LLaVA-1.5-7B, LLaVA-1.5-13B, and LLaVA-v1.6-Mistral-7B, δ-Vision’s average scores are 61.9, 62.9, and 60.9.Source: Qwen3.5-4B · p. 17, Table 11 · p. 17

Qwen3.5-4B

  • At 5% retention, δ-Vision improves over DivPrune by 6.9 points on LLaVA-1.5-13B and 8.4 points on Qwen3-VL-8B; on LLaVA-v1.6-Mistral-7B, it scores 60.9 versus DivPrune’s 61.5 and DART’s 50.4.Table 12 reports these results for visual-token compression methods, with DART and DivPrune retaining 5% of visual tokens.Source: Qwen3.5-4B · p. 18, Table 12 · p. 18

G Adapter Rank Ablation

  • Increasing adapter rank generally yields moderate downstream-accuracy improvements, consistent with a larger bottleneck providing more capacity for layer-specific visual-state corrections.Figure 4 compares embedding-adapter performance across ranks, with dashed lines marking vanilla-model performance.Source: G Adapter Rank Ablation · p. 18, Figure 4 · p. 18

H Hidden-State Compression

  • The hidden-state compression intervention fits a separate uncentered PCA basis per layer from 1,024 images, then projects and reconstructs each visual state using its leading r directions.The same layer basis is shared across visual-token positions, input samples, and evaluation benchmarks; token count and positions remain unchanged.Source: H Hidden-State Compression · p. 18

I Adapter Compatibility

  • Combining the embedding adapter with 50% token retention yields average scores of 73.2 with DART and 73.1 with DivPrune, only 1.2–1.3 points below the adapter’s 74.4.At 20% retention, DART/DivPrune score 69.8/70.3; at 5%, they score 61.4/61.0, supporting the complementary use of state and sequence compression.Source: I Adapter Compatibility · p. 19, Table 13 · p. 19

J Layer-wise Adapter Skipping

  • Skipping adapters in the first 5 and last 10 layers lowers the average score from 74.4 to 73.3 while reducing FLOPs from 17.90% to 15.64% and raising speedup from 1.30× to 1.35×.Skipping the first and last 10 layers yields an average score of 71.9, 14.89% of vanilla-model FLOPs, and 1.37× total inference speedup.Source: J Layer-wise Adapter Skipping · p. 19, Table 14 · p. 19, Table 15 · p. 20

K Linear Attention Analysis

The analysis separates visual influence through recurrent-state writes in Linear Attention (LA) layers from visual-key/value reads in Full Attention (FA) layers. In the tested hybrid backbone, canceling visual writes across LA layers has limited effect, while blocking reads at FA layers 11 and 15 substantially degrades RWQA and MMStar.

Hybrid-backbone adapter path

  • In the embedding-adapter variant, predicted visual tokens still enter the original Gated DeltaNet recurrent-state update, and later textual tokens retrieve that visual information through their queries.The visual representation at each layer is constructed from the initial visual embedding rather than propagated from the preceding layer.Source: K Linear Attention Analysis · p. 20

Intervention setup

  • The analysis intervenes on two routes for visual information to affect later text computation: writing into LA recurrent states and reading visual key–value states in FA layers.The figure defines Mask LA as restoring the post-visual recurrent state to its pre-visual value, and compares two sets of blocked FA layers.Source: K Linear Attention Analysis · p. 20, Figure 5 · p. 21

Blocking visual write in Linear Attention

  • Under the rank-0 LA intervention, the recurrent state is restored at the visual boundary; visual tokens are processed, but their net contribution to that state is removed before later text tokens.All 24 LA layers receive this intervention, while FA layers remain unmodified and can still access visual keys and values.Source: K Linear Attention Analysis · p. 20

Visual write and read interventions

  • Blocking visual reads at FA layers 11 and 15 substantially degrades RWQA and MMStar, whereas blocking reads at layers 3, 7, 19, 23, 27, and 31 causes a considerably smaller drop.Canceling visual writes to recurrent states across all 24 LA layers has only a limited effect; the interventions indicate visual information is not used uniformly across these layers.Source: K Linear Attention Analysis · p. 21, Figure 5 · p. 21
Loading 2609.34972v1…