DeCo: Decoupling Token Compression from Semantic Abstraction in Multimodal Large Language Models

1 Introduction

• Visual token sequences grow with image resolution and image count, making non-compressive linear projectors costly for high-resolution images and video.The passage notes that video sequences can reach up to a million tokens.Source: §1 · p. 2
• R-GAE traces relevance from generated text to patches and separates Text-to-Patch relevance into Text-to-Query and Query-to-Patch flows.These flows respectively indicate visual context used by the LLM and patterns learned by query tokens from patches.Source: §1 · p. 2
• Compressive query tokens abstract visual concepts but can lose fine-grained attributes and spatial locality, creating a redundant second abstraction by the LLM.The projector’s semantic losses can propagate to the LLM, while training an effective semantic abstractor adds complexity.Source: §1 · pp. 2–3
• DeCo uses adaptive average pooling to reduce visual tokens at the patch level, leaving the LLM to abstract visual semantics.The pooling is parameter-free, supports a flexible output token count, and preserves spatial locality through neighboring-patch integration.Source: §1 · p. 3
• The paper reports that DeCo improves performance and efficiency over prevailing compressed projectors and strengthens spatial understanding across MLLM configurations.Its reported robustness spans vision backbones, image resolutions, and LLMs.Source: §1 · p. 3

2 Related Work

• Vision-language models combine encoders and LLMs using approaches including single projection layers, LoRA tuning, and instruction-integrated alignment.The related-work discussion also cites Flamingo, IDEFICS, Q-Former, Qwen-VL-Chat, and Fuyu-8 as examples of broader architectural directions.Source: §2 · p. 3
• Projector modules have often been used empirically or evaluated through downstream MLLM performance rather than analyzed in depth.This paper examines projector behavior by tracking vision-language semantic flow and visualizing learned patterns.Source: §2 · p. 3
• Transformer interpretation methods include rollout, relevance propagation, and GAE, but in-depth explainability of existing MLLMs is described as rare.Figure 2 frames the paper’s analysis as tracing language-to-vision semantic flow during image-to-text generation.Source: §2 · p. 3, Figure 2 · p. 4

3.1 R-GAE: Relevance Maps in MLLMs Derived from GAE

• R-GAE decomposes Text-to-Patch relevance into Text-to-Query and Query-to-Patch maps, propagating relevance across arbitrary layers and averaging steps for sentence-level visual relevance.The maps respectively show which query tokens relate to generated text and which visual patterns the queries capture.Source: §3 · p. 4, §3.1 · pp. 4–5

3.2 A Redundant Double-Abstraction Phenomenon Resulting from Compressive Projectors

• With patch representations, LLMs can localize fine-grained visual attributes directly, whereas QFormer’s query map drops “purple and red” and misaligns final patch relevance.The comparison holds the MLLM architecture constant except for the projector.Source: §3.2 · pp. 5–6, Figure 3 · p. 5
• Compressive projectors pre-abstract patches into limited semantic concepts, so LLM re-extraction compounds losses in fine-grained attributes and spatial locality.The resulting misalignment can carry through to the final Text-to-Patch map.Source: §3.2 · p. 6
• QFormer’s fixed query tokens can be visually repetitive and semantically sparse, while training the projector as an effective semantic abstractor increases training cost and complexity.The paper gives Qwen-VL-7B’s resampler as an example, requiring 1.4B pretraining and 50M fine-tuning data across three stages.Source: §3.2 · p. 6

• DeCo compresses visual-token numbers at the patch level and leaves semantic abstraction to the LLM, removing the intermediate semantic bottleneck and simplifying training.The method uses AdaptiveAvgPool as a natural patch-level downsampler.Source: §4 · p. 6
• AdaptiveAvgPool reshapes patch tokens as a 2D grid, averages spatial neighbors into fewer tokens, flattens them, and linearly projects them to the LLM dimension.Adaptive pooling automatically sets its stride and kernel size in a parameter-free mode.Source: §4 · pp. 6–7
• AdaptiveAvgPool uniformly downsamples spatial patches while retaining fine-grained attributes; its Text-to-Patch maps are nearly identical to the linear projector’s.The comparison describes a combination of preserved visual context and fewer vision tokens.Source: §4 · p. 7

5.1 Experiment Setting

• The experiments evaluate MLLM benchmarks, visual localization, and open-ended visual question answering using the LLaVA v1.5 framework.The default setup uses a two-stage schedule; generalization experiments use one-stage instruction tuning.Source: §5.1 · pp. 7–8

5.2 Compared with Existing Projectors

• DeCo outperforms existing compressive projectors across most benchmarks, including visual localization and open-ended VQA, at the same 576->144 compression ratio.The linear projector achieves the best overall performance, while C-Abstractor results are comparable to DeCo.Source: §5.2 · p. 8
• AdaptiveAvgPool is more lightweight and DeCo converges faster during pre-training than the compared projectors.DeCo uses parameter-free compression; the cited comparison describes C-Abstractor’s ResNet blocks and a two-layer QFormer.Source: §5.2 · pp. 8–9
• DeCo with AdaptiveAvgPool scores 56.0 on six spatial-understanding tasks, close to the linear projector’s 56.4 and above C-Abstractor’s 54.4 and QFormer’s 40.9.The authors attribute spatial-context preservation to AdaptiveAvgPool’s kernel and stride operation.Source: §5.2 · p. 9

5.3 Generalization Results

• Across six settings varying backbones, image resolutions, and LLMs, DeCo surpasses C-Abstractor in almost all metrics and settings.Table 3 reports one-stage instruction tuning on 665K data.Source: Table 3 · p. 8, §5.3 · p. 9

5.4 Ablation Study

• A quarter compression from 24 × 24 to 12 × 12 gives AdaptiveAvgPool the best balance between visual information deficiency and training cost in localization.The study compares grids of 20 × 20, 16 × 16, 12 × 12, and 8 × 8.Source: §5.4 · p. 9, Figure 6 · p. 9
• Adaptive average pooling performs better than max pooling across almost all metrics, especially visual localization.The authors say averaging integrates patches within each kernel window and serves more visual context.Source: §5.4 · p. 9, Figure 7 · p. 9
• Two-stage training is recommended over one-stage training for DeCo.Figure 8 compares the two schedules.Source: §5.4 · p. 9, Figure 8 · p. 9

6 Limitations

• At increasingly high compression, AdaptiveAvgPool may cause severe visual information deficiency and erase fine-grained context within a pooling kernel.Source: §6 · p. 10
• DeCo’s advantage is framed for settings with limited GPUs for long visual sequences or limited training data for optimizing semantic QFormer-type projectors.Source: §6 · p. 10
• With abundant training resources, projector architecture tends to be insignificant in an MLLM system, as pointed out in MM1 [53].Source: §6 · p. 10

7 Conclusion

• DeCo decouples visual token compression from semantic abstraction, motivated by existing projectors’ double abstraction.Source: §7 · p. 10
• The method simplifies compressive projectors with AdaptiveAvgPool, which downsamples spatial vision tokens directly at the spatial level.Source: §7 · p. 10
• Experiments across diverse configurations demonstrate DeCo’s efficiency, effectiveness, and robustness, with potential to improve in demanding high-compression scenarios.Source: §7 · p. 10

A.1 Background: GAE Explainability for Transformer Layers

• GAE combines attention maps and gradients into layer-level relevance maps, then propagates relevance between selected Transformer layers.The relevance map is initialized as an identity matrix and updated from a start layer to an end layer.Source: §A.1 · p. 16

A.2 R-GAE Propagation for MLLMs

• R-GAE traces relevance from generated text through LLM and projector tokens to the original visual patches.The method caches attention maps in the ViT, projector, and LLM, and obtains gradients by targeting a generated word.Source: §A.2 · p. 16
• The overall text-to-patch relevance map is computed by multiplying the text-to-query and query-to-patch maps.Source: §A.2 · p. 16
• For a complete sentence, R-GAE averages relevance maps across generation steps to obtain sentence-level visual relevance.Source: §A.2 · p. 17
• For non-compressive projectors, the query-to-patch map is an identity mapping because queries correspond one-to-one with original patches.Source: §A.2 · p. 17
• For AdaptiveAvgPool, each merged token receives the average relevance of raw tokens within its pooling window.For a window of size K^2, its score is 1/K^2 times the sum of the window’s raw-token relevance scores.Source: §A.2 · p. 17

• Raw attention maps show only single-layer relevance, whereas R-GAE can track relevance from a target layer back to original patches.Source: B Comparison between R-GAE and Raw… · p. 17
• R-GAE provides a more interpretable representation of MLLM vision-language alignment than raw attention maps on the compared model and image-text pair.Source: B Comparison between R-GAE and Raw… · p. 17
• The raw-map error mainly arises in the LLM’s final Text-to-Query map, where 32 self-attention layers have fused token semantics; the QFormer Query-to-Patch map is relatively similar to R-GAE.The QFormer has a single cross-attention layer.Source: B Comparison between R-GAE and Raw… · p. 17

• AdaptiveAvgPool is parameter-free and is paired with a two-layer MLP to map vision features to the LLM dimension.Source: C Training Hyper-parameters · p. 17
• Main experiments use the LLaVA v1.5 framework, while generalization experiments use a lightweight instruction-tuning setup following PRISM.Source: C Training Hyper-parameters · p. 18
• Tables 4 and 5 are identified as reporting hyperparameters for the main and generalization experiments, respectively.Source: Table 4 · p. 18, Table 5 · p. 18, C Training Hyper-parameters · p. 18

D More R-GAE Relevance Maps

• Figure 10 presents additional R-GAE relevance maps for the caption “small orange Nissan pick up truck in traffic.”The visualization labels Text-to-Patch, Text-to-Query, and Query-to-Patch maps.Source: D More R-GAE Relevance Maps · pp. 18–19
• The linear projector preserves 576 vision tokens, while QFormer and AdaptiveAvgPool compress them to 64.Figure 10 frames highlighted text query tokens in red.Source: Figure 10 · p. 19

E Broader Impacts

• Frozen LLMs may generate hallucinated, ungrounded, or biased outputs, so downstream applications warrant application-specific safety and fairness assessment.Source: E Broader Impacts · p. 20