PaperlayerView exact revision on arXiv

Source-linked AI summary

Chain-of-Thought Compression Should Not Be Blind: V-Skip for Efficient Multimodal Reasoning via Dual-Path Anchoring

Dongxu Zhang, Yiding Sun, Cheng Tan, Wenbiao Yan, Ning Yang, Jihua Zhu, Haijun Zhang

arXiv:2601.13879v4cs.MMcs.CLcs.CV

TL;DR

MLLM chain-of-thought reasoning is costly, and text-centric compression can prune tokens needed for visual grounding, causing Visual Amnesia. V-Skip formulates pruning as a Visual-Anchored Information Bottleneck and combines linguistic surprisal with cross-modal attention flow. It achieves a 2.9× inference speedup with negligible performance loss and outperforms state-of-the-art baselines by over 45% on fine-grained visual tasks.

  • Problem

    Autoregressive multimodal reasoning incurs severe latency and computational overhead, while text-centric pruning can discard visually essential tokens and induce Visual Amnesia.

  • Method

    V-Skip formulates compression as a Visual-Anchored Information Bottleneck and uses linguistic surprisal plus cross-modal attention flow, distilled into a LoRA adapter.

  • Results

    2.9× inference speedup with negligible performance loss, while outperforming state-of-the-art baselines by over 45% on fine-grained visual tasks.

  • Takeaways & Limitations

    V-Skip preserves visual grounding during multimodal CoT compression and significantly mitigates object hallucination.

  • Takeaways & Limitations

    V-Skip depends on the base model’s cross-modal alignment and requires offline distillation, adding training overhead.

Abstract

from arXiv · show

While Chain-of-Thought (CoT) reasoning significantly enhances the performance of Multimodal Large Language Models (MLLMs), its autoregressive nature incurs prohibitive latency constraints. Current efforts to mitigate this via token compression often fail by blindly applying text-centric metrics to multimodal contexts. We identify a critical failure mode termed Visual Amnesia, where linguistically redundant tokens are erroneously pruned, leading to hallucinations. To address this, we introduce V-Skip that reformulates token pruning as a Visual-Anchored Information Bottleneck (VA-IB) optimization problem. V-Skip employs a dual-path gating mechanism that weighs token importance through both linguistic surprisal and cross-modal attention flow, effectively rescuing visually salient anchors. Extensive experiments on Qwen2-VL and Llama-3.2 families demonstrate that V-Skip achieves a $2.9\times$ speedup with negligible accuracy loss. Specifically, it preserves fine-grained visual details, outperforming other baselines over 30\% on the DocVQA.

1 Introduction

MLLM chain-of-thought improves multimodal reasoning but creates substantial autoregressive overhead, while text-centric compression can remove visually essential tokens. V-Skip addresses this failure by combining linguistic and visual signals to preserve grounding while compressing reasoning.

  • Motivation: Autoregressive reasoning chains create severe latency and computational overhead, while linguistic fillers often add little information to final predictions.The growing KV cache also restricts batch sizes and throughput.
  • Motivation: Text-centric pruning can cause Visual Amnesia by discarding tokens essential for visual grounding and inducing object hallucinations.Linguistic predictability does not reliably indicate visual redundancy in multimodal reasoning.
  • Approach: V-Skip reformulates multimodal compression as a Visual-Anchored Information Bottleneck optimization problem balancing linguistic efficiency with cross-modal grounding.The framework explicitly incorporates visual anchoring into the pruning objective.
  • Approach: V-Skip uses dual-path scoring to combine linguistic surprisal with cross-modal attention flow and rescue tokens that are linguistically redundant but visually salient.The method preserves visual grounding while compressing multimodal chain-of-thought sequences.
  • Approach: V-Skip distills its pruning policy into a lightweight LoRA adapter, avoiding explicit scoring or online filtering during inference.This produces an efficient reasoner that generates concise, visually grounded rationales directly.
  • Results: 2.9× speedup with minimal accuracy trade-offs, while outperforming other baselines over 30% on DocVQA.The reported result is based on extensive experiments on Qwen2-VL.

2 Related Work

Prior multimodal reasoning methods improve accuracy but often produce computationally expensive reasoning chains. V-Skip targets this sequence-length bottleneck through token compression that preserves visual anchors rather than relying only on linguistic redundancy.

  • Multimodal reasoning: MLLMs project visual features into language-model embedding spaces, enabling interleaved image-text processing.Examples include Llama, LLaVA, and Qwen-VL.
  • Multimodal reasoning: Multimodal reasoning methods such as PICa, Multimodal-CoT, PromptCoT, and T-SciQ emphasize visual-linguistic alignment or rationale quality.These approaches include caption-based prompting and human-annotated rationale fine-tuning.
  • Efficient reasoning: Extensive reasoning chains introduce significant computational overhead, motivating methods that prune redundant computation while preserving accurate prediction.V-Skip focuses on efficient utilization of reasoning chains rather than generation-centric improvements.
  • Efficient reasoning: System-level methods reduce memory footprints but do not address the fundamental bottleneck of sequence length in autoregressive generation.The cited approaches include quantization, structured compression, and KV-cache optimization.
  • Token compression: Token pruning methods such as TokenSkip, ASCoT, and LLMLingua-2 reduce unnecessary generation using structural or linguistic-probability criteria.Their compression decisions target redundant layers or tokens with low linguistic information gain.
  • Token compression: V-Skip distinguishes itself by explicitly incorporating Visual Anchoring into the pruning metric because linguistic compression can discard critical visual anchors.Its VA-IB formulation guides a selection mechanism that separates linguistic redundancy from visual necessity.

3 Method

V-Skip formulates multimodal chain-of-thought compression as a Visual-Anchored Information Bottleneck problem and preserves tokens needed for both linguistic coherence and visual grounding. Its dual-path gating combines linguistic information with cross-modal attention, then distills the pruning policy for efficient inference.

  • 3.1 Preliminaries and Problem Formulation: Visual Amnesia occurs when text-centric pruning discards visually grounded tokens that are linguistically unpredictable or redundant.Such pruning can sever image connections and induce object hallucinations.
  • 3.2 Visual-Anchored Information Bottleneck: V-Skip reformulates compression as a VA-IB objective balancing answer sufficiency, visual grounding, and a length budget.The objective retains semantic content predictive of the answer while maintaining dependence on the visual input.
  • 3.3 Visual Anchoring Score via Attention: V-Skip scores each token through linguistic self-information and visual attention flow, using cross-modal dependence to identify visual anchors.Visual attention is aggregated over selected layers and heads to capture grounding signals while avoiding dilution by syntactic noise.
  • 3.4 The V-Skip Gating Mechanism: The V-Skip Gate retains a token when either its textual or visual saliency exceeds its dynamic threshold.This union-of-saliency rule allows visual saliency to override linguistic redundancy and preserve visually essential descriptors.
  • 3.4 The V-Skip Gating Mechanism: The pruning policy is distilled offline into the base model with LoRA, eliminating explicit dual-path scoring during inference.Instruction tuning on masked reasoning chains enables concise, visually anchored rationale generation without online attention analysis.

4 Experiments

Experiments evaluate V-Skip across Qwen2-VL and Llama-3.2 models using multimodal reasoning, document understanding, visual-attribute retention, and hallucination benchmarks. Results show that visual anchoring preserves performance and visual grounding under compression, while robustness generally improves with model size.

  • Experimental Setup: Experiments use Qwen2-VL variants and Llama-3.2-11B-Vision-Instruct, evaluating MMMU and DocVQA with Accuracy, latency, and actual compression ratio.DocVQA targets fine-grained OCR and spatial grounding, while MMMU measures multidisciplinary reasoning.
  • Main Results: V-Skip retains 83.7% ANLS on DocVQA at γ = 0.5 and outperforms LLMLingua-2 by 45.2%.Text-centric LLMLingua-2 drops over 53% on this fine-grained task.
  • Main Results: V-Skip limits MMMU accuracy loss to 5.9%, while baselines degrade by over 20%, and achieves a 1.8× DocVQA speedup with 2.71s latency.It is faster than LLMLingua-2 at 2.93s despite generating longer sequences.
  • Robustness: At γ = 0.5, Llama-3.2 with V-Skip retains over 91% of original performance on both MMMU and DocVQA across compression ratios.The reported curves remain flat as compression varies.
  • Scaling: V-Skip robustness increases with model size: accuracy drops 8.5% for Qwen2-VL-2B but only 3.2% for the 72B model after removing 50% of tokens.The authors relate this pattern to more verbose and redundant reasoning in larger models.
  • Visual Grounding and Hallucination: V-Skip preserves visual attributes and reduces hallucination-related degradation by retaining visual anchors during compression.The analysis reports that LLMLingua-2 retains only 42.5% of color tokens, whereas V-Skip rescues visual anchors; V-Skip also maintains a 51.2% Yes-Ratio and achieves F1 88.9 on POPE.
  • Ablation: The union strategy achieves 48.2% accuracy, outperforming the intersection strategy at 30.1% by combining linguistic structure with visual anchors.Text-only pruning minimizes syntax errors but sacrifices visual recall, whereas vision-only pruning disrupts syntactic fluency.

5 Conclusion

V-Skip addresses the efficiency bottleneck in multimodal CoT compression by combining visual grounding with token pruning. It achieves substantial acceleration, reduces hallucination, and is positioned for extension beyond textual CoT.

  • 2.9× inference speedup is achieved with negligible performance loss through V-Skip’s visual-anchored compression strategy.
  • V-Skip outperforms state-of-the-art baselines by over 45% on fine-grained visual tasks while significantly mitigating object hallucination.
  • V-Skip is primarily validated on textual CoT, motivating future exploration of alignment-based compression for dynamic video and audio-visual interactions.

Limitations

V-Skip’s pipeline uses dual-path token utilities and a union-of-saliency gate before LoRA-based student training. Its main limitations concern dependence on cross-modal alignment and additional offline training overhead.

  • Limitations: V-Skip may degrade on architectures with weaker cross-modal alignment than the base model assumes.
  • Limitations: Unlike training-free methods, V-Skip requires offline distillation, introducing additional training overhead.
  • Method: Each token receives textual information and visual anchoring scores before an Union-of-Saliency gate determines retention.

A.2 Qualitative Analysis: What is Skipped?

The qualitative analysis compares retention masks at 50% compression and shows that V-Skip preserves visually essential answer tokens that text-only pruning removes. The resulting rationale remains concise while retaining visual grounding.

  • Retention-mask comparison: At γ = 0.5, the comparison marks deleted tokens with strikethroughs to expose differences between LLMLingua-2 and V-Skip retention masks.
  • Retention-mask comparison: LLMLingua-2 removes the answer $45.20, whereas V-Skip retains it while skipping the same linguistic fillers.
  • Interpretation: V-Skip’s compressed CoT preserves the minimal sufficient statistics needed for the final answer while remaining visually grounded.

B.1 Datasets

The evaluation uses MMMU for complex multimodal reasoning and DocVQA for document visual question answering. It measures accuracy, end-to-end latency, and controllability, with qualitative analysis highlighting visually salient answer tokens.

  • MMMU: MMMU provides collegiate-level Art, Science, and Engineering tasks requiring multimodal reasoning and expert knowledge application.
  • DocVQA: DocVQA complements MMMU by testing document questions whose answers can depend on high-entropy visual tokens.
  • Qualitative analysis: Figure 5 qualitatively illustrates the Information Entropy Mismatch on DocVQA through the retention of token $45.20.
  • Metrics: The reported evaluation metrics are Accuracy, End-to-End Latency, and ActRatio.

B.2 Detailed Evaluation Metrics

The evaluation reports task accuracy alongside efficiency measures covering reasoning length, wall-clock latency, and realized compression. It also defines DocVQA’s ANLS metric and evaluates on official validation splits.

  • Efficiency metrics: Average Generated Tokens measures reasoning-chain verbosity using decoded tokens excluding the input prompt.Lower token counts with maintained accuracy indicate a more concise reasoning path.
  • Efficiency metrics: Latency measures wall-clock inference time, including method-specific overheads rather than only theoretical FLOPs.This protocol targets real-world computational cost.
  • Efficiency metrics: ActRatio measures realized compressed-chain length relative to the original and can exceed or fall below the target compression ratio γ.V-Skip may retain more tokens for extensive visual grounding and compress more aggressively for simple images.
  • Task accuracy: Accuracy is reported for MMMU using standard Top-1 classification accuracy.MMMU consists primarily of expert-level multiple-choice tasks.
  • Task accuracy: ANLS evaluates DocVQA text extraction by normalizing Levenshtein distance and averaging scores from 0–100.A threshold τ = 0.5 determines when the normalized score becomes zero.
  • Evaluation protocol: All main-text benchmarking experiments use official MMMU and DocVQA validation splits to support consistent baseline comparisons and avoid test-set leakage.

B.4 Training Configurations

V-Skip is fine-tuned with LoRA adapters applied throughout self-attention projections, using AdamW and a short, resource-bounded training schedule. Supplementary sensitivity analysis examines which visual-layer signals best support pruning.

  • Adapter configuration: LoRA adapters are applied to Wq, Wk, Wv, and Wo with rank r = 16, scaling α = 32, and dropout 0.05.The adapters fine-tune attention mechanisms while limiting computational overhead.
  • Optimization setup: AdamW optimization uses a 2 × 10^-4 learning rate, cosine scheduling with 0.03 warm-up, three epochs, and global batch size 16.Training uses an 8×RTX 3090 GPU infrastructure.
  • Sensitivity analysis: The visual-layer sensitivity study evaluates how Transformer depth affects the cross-attention signal used to compute the Visual Anchoring Score.The analysis is conducted on MMMU to identify an effective signal source.
  • Sensitivity analysis: Shallow layers cause a 41.5% performance drop, while final layers reach 46.8% because their attention signals are respectively noisy or affected by abstraction drift.The passage attributes these outcomes to low-level feature encoding in shallow layers and output-layer representation drift in deep layers.

C.2 Hyperparameter Robustness: LoRA Rank Analysis

The LoRA-rank study frames adapter size as a trade-off between pruning performance and resource use, with rank 16 serving as the practical reference point. Qualitative evidence illustrates why visual anchoring preserves critical invoice details while adapting compression to visual complexity.

  • Experimental setup: LoRA-rank sensitivity evaluates MMMU accuracy, trainable-parameter proportion, and peak training VRAM at γ = 0.5.The study varies adapter sizes to examine capacity-resource trade-offs.
  • Rank-performance trade-off: At r = 4, accuracy reaches only 46.5%, indicating a representational bottleneck under the 0.04% parameter budget.The constrained adapter cannot fully learn the cross-modal alignment patterns needed for precise pruning.
  • Rank-performance trade-off: Ranks above 16 provide diminishing returns, while the selected configuration maintains a minimal memory footprint of 18.5 GB.The reported result supports parameter-efficient fine-tuning rather than extensive adapter capacity.
  • Visual-layer comparison: The visual-layer study compares accuracy and deviation from Middle Layers across Transformer-layer ranges at γ = 0.5.
  • Qualitative evidence: In the invoice case, LLMLingua-2 prunes the crucial $45.20 value, whereas V-Skip retains it because its Visual Anchoring Score is high.The example concerns extracting a small-print invoice total.
  • Future directions: Future work proposes extending VA-IB to 3D geometric reasoning, including point-cloud modules and geometry-aware skip attention.
Loading 2601.13879v4…