PaperlayerView exact revision on arXiv

Source-linked AI summary

DUET-VLM: Dual stage Unified Efficient Token reduction for VLM Training and Inference

Aditya Kumar Singh, Hitesh Kandala, Pratik Prabhanjan Brahma, Zicheng Liu, Emad Barsoum

arXiv:2602.18846v2cs.CVcs.AI

TL;DR

Dense visual tokenization makes VLMs computationally expensive, while existing compression methods can trade accuracy for speed. DUET-VLM combines vision-to-vision redundancy-aware merging with layer-wise text-guided pruning, retaining near-baseline performance across compressed token budgets.

  • Problem

    Dense visual tokenization makes VLMs computationally expensive because images require many visual tokens that scale quadratically with resolution.

  • Method

    DUET-VLM first merges correlated vision tokens into compact embeddings, then progressively prunes visual tokens using text-to-vision attention scores across language-backbone stages.

  • Results

    Across five LLaVA-1.5-7B benchmarks, DUET-VLM retains 99.0% of baseline average accuracy with 67% fewer tokens and 95.4% with 89% fewer tokens, outperforming prior methods.

  • Takeaways & Limitations

    DUET-VLM sustains strong cross-modal reasoning while reducing visual-token budgets and also reduces training time by up to 36% across evaluated settings.

Abstract

from arXiv · show

Vision-language models (VLMs) have achieved remarkable multimodal understanding and reasoning capabilities, yet remain computationally expensive due to dense visual tokenization. Existing efficiency approaches either merge redundant visual tokens or drop them progressively in language backbone, often trading accuracy for speed. In this work, we propose DUET-VLM, a versatile plug-and-play dual compression framework that consists of (a) vision-only redundancy aware compression of vision encoder's output into information-preserving tokens, followed by (b) layer-wise, salient text-guided dropping of visual tokens within the language backbone to progressively prune less informative tokens. This coordinated token management enables aggressive compression while retaining critical semantics. On LLaVA-1.5-7B, our approach maintains over 99% of baseline accuracy with 67% fewer tokens, and still retains >97% even at 89% reduction. With this dual-stage compression during training, it achieves 99.7% accuracy at 67% and 97.6% at 89%, surpassing prior SoTA visual token reduction methods across multiple benchmarks. When integrated into Video-LLaVA-7B, it even surpasses the baseline -- achieving >100% accuracy with a substantial 53.1% token reduction and retaining 97.6% accuracy under an extreme 93.4% setting. These results highlight end-to-end training with DUET-VLM, enabling robust adaptation to reduced visual (image/video) input without sacrificing accuracy, producing compact yet semantically rich representations within the same computational budget. Our code is available at https://github.com/AMD-AGI/DUET-VLM.

1. Introduction

Dense visual tokenization makes VLMs memory- and latency-bound, despite improving grounding and reasoning accuracy. DUET-VLM addresses this with coordinated vision-side merging and language-side pruning, retaining near-baseline accuracy under heavy compression.

  • LLaVA-1.5 processes 576 visual tokens per image, while LLaVA-NeXT exceeds 2,800 at 672×672 resolution.
  • Visual tokens scale quadratically with image resolution, compounding quadratic attention costs and limiting VLM memory and latency.
  • Scaling visual tokens improves grounding and reasoning accuracy but incurs high computational cost.
  • DUET-VLM combines Vision-to-Vision merging with Text-to-Vision pruning to remove redundancy before and during language processing.
  • 99% baseline performance is retained at 67% token reduction, while above 97% is retained at 89% reduction during inference.
  • Video-LLaVA surpasses baseline performance with 53.1% token reduction and retains 97.6% accuracy at 93.4% reduction.

2. Related Works

VLM efficiency research primarily compresses visual representations before cross-modal interaction or prunes visual tokens during language processing. DUET-VLM unifies these directions through redundancy-aware merging and text-guided, layer-wise pruning.

  • Token-efficiency methods target redundant visual computation and quadratic attention costs through vision-side or language-side compression.
  • Vision-side methods compact visual embeddings before cross-modal interaction, whereas language-side methods prune visual tokens during or after interaction.
  • The framework extends early-stage merging and progressive pruning under a unified, differentiable design.
  • DUET-VLM merges correlated patches before fusion and dynamically prunes visual tokens using text-to-vision attention across successive language layers.

3. Methodology

DUET-VLM first compresses vision tokens through attention-guided dominant-token selection and local residual clustering, then progressively drops tokens using salient text-to-vision attention across language stages.

  • 3.1. Clustering of vision tokens: Vision tokens are divided into influential dominant tokens and residual tokens that are merged into contextual tokens via clustering.
  • 3.1. Clustering of vision tokens: The clustering algorithm selects dominant indices with TopK attention scores and forms residual indices from the remaining tokens.
  • 3.1. Clustering of vision tokens: Local cluster aggregation partitions residual tokens into k2 small clusters, each producing one contextual token by local averaging.
  • 3.1. Clustering of vision tokens: The method keeps k2 contextual tokens while enforcing a small cluster width w instead of increasing contextual-token count.
  • 3.2. Hierarchical visual tokens dropping: Only salient text tokens compute text-to-vision saliency, while the final text token is always included as a sink token.
  • 3.2. Hierarchical visual tokens dropping: At each language stage, visual tokens are ranked by cross-attention and reduced to ⌊(1 − λ).Nl⌋ tokens.

4. Experiment

Experiments evaluate DUET-VLM across image, video, inference-only, and training settings, showing strong accuracy–efficiency trade-offs under substantial visual-token compression. Results also examine generalization, training cost, local clustering, and cluster-width choices.

  • Experimental setup: The evaluation covers LLaVA-1.5-7B, LLaVA-NeXT-7B, Qwen-2.5-VL-7B, and Video-LLaVA-7B across image and video benchmarks.Image benchmarks include POPE, GQA, TextVQA, MME, SQAI, and SeedI; video benchmarks include TGIF-QA, MSVD-QA, and MSRVTT-QA.
  • Inference-only results: 99.0%, 98.1%, and 95.4% relative average accuracy are retained at 67%, 78%, and 89% token reduction on inference-only LLaVA-1.5-7B.These settings use 192, 128, and 64 visual tokens, respectively.
  • Cross-architecture generalization: 99.9%, 99.8%, and 98.4% relative average accuracy are retained by DUET-VLM (C) at 640, 320, and 160 average tokens on Qwen-2.5-VL-7B.The corresponding speedups are about 1.3×, 1.4×, and 1.5×, and DUET-VLM (C) outperforms VisionZip on relative average accuracy at each budget.
  • Training results: 99.7% average accuracy is achieved at 67% token reduction in trained LLaVA-1.5-7B experiments, closely matching the full-token baseline.At 89% reduction, DUET-VLM retains 95.4% average accuracy in the reported training-time trade-off.
  • Training efficiency: 26%, 31%, and 36% training-time reductions accompany 192, 128, and 64 tokens, respectively, on LLaVA-1.5-7B.The reported average accuracies at 192 and 128 tokens are 99.7% and 99.1%.
  • Video results: 100.8% average accuracy is achieved at 53.1% token reduction on Video-LLaVA-7B, while 93.4% reduction retains 97.6% accuracy.The 53.1% setting uses 960 tokens and slightly surpasses the full model baseline; the extreme setting uses 136 tokens.
  • Local clustering: DUET-VLM (C) consistently surpasses VisionZip, DUET-VLM (Vanilla), and VisionZip (C) across token budgets in the local-clustering ablation.Cluster widths {4:6} yield the best TextVQA performance by balancing preservation and redundancy reduction.

M POPE SQAI VQAT MME GQA SeedI Avg

DUET-VLM retains strong accuracy across LLaVA-NeXT-7B token budgets, including more than 91% accuracy at 94.4% token reduction. Salient text guidance and clustering help preserve query-relevant visual information during pruning.

  • DUET-VLM maintains strong accuracy above 91% even at 94.4% token reduction on LLaVA-NeXT-7B.
  • 98.9%, 97.8%, and 94.8% of baseline accuracy are retained at 192, 128, and 64 tokens on LLaVA-1.5-7B.
  • Salient text tokens provide sharper contextual cues than using all text tokens equally, improving visual-token importance estimation.
  • Attention from salient text tokens narrows token dropping toward query-relevant image regions and helps remove irrelevant backgrounds and distracting objects.

5. Conclusion

DUET-VLM combines spatial redundancy removal with contextual, text-guided pruning to compress visual tokens while retaining semantically critical information. It maintains near-baseline accuracy in inference-only settings and can surpass baseline performance after compression-aware fine-tuning.

  • DUET-VLM combines vision-encoder token merging with saliency-based, text-guided pruning in the language backbone.
  • Experiments on image and video benchmarks show near-baseline accuracy for inference-only compression and baseline-surpassing performance after fine-tuning under compression.
  • Exact inference and training times are not reported, although runtimes are described as comparable to VisionZip at the same token budget.
  • Future work includes detailed speed analysis, video-model training, longer-horizon videos, stronger region proposals, and extension to additional modalities.

A. Sensitivity Analysis

The sensitivity analyses examine when visual tokens can be dropped and how dominant and contextual token allocations affect compression. They support later pruning and budget-dependent allocation choices.

  • 100% pruning after the 24th layer remains competitive, whereas pruning all tokens from the middle stage degrades performance.
  • Late-layer visual tokens become redundant after image knowledge has transferred into hidden states, while finer layer-by-layer analysis remains future work.
  • At fixed token budgets, k1 and k2 vary symmetrically because their sum is constrained by the budget.
  • Larger budgets favor scaling k1 upward, while aggressive 128-token compression favors an intermediate dominant–contextual split.

B. Additional Experiments

Additional experiments show that local clustering and text-token selection improve compression-aware visual-token pruning across LLaVA variants and token budgets. The analyses also examine dropping configurations and cluster-width trade-offs.

  • The clustering and text-selection experiments evaluate performance across token budgets, benchmarks, and alternative visual-token dropping configurations.
  • Effect of Text Token Selection in PyramidDrop: Using salient or all query text tokens improves PyramidDrop accuracy across benchmarks compared with relying only on the last text token.
  • Effect of Text Token Selection in PyramidDrop: All query tokens achieve 99.5% average accuracy versus 99.4% for salient tokens, with only a marginal gap between strategies.

C. Hyperparameters

The hyperparameter configurations specify target vision-token budgets for local clustering across four VLM settings. For Qwen2.5-VL-7B, the language-model-side rank-and-drop uses layers 14 and 21 with image-token ratios of 0.5 and 0.25.

  • LLaVA-1.5-7B: Local clustering configurations target different vision-token budgets for LLaVA-1.5-7B.Table 12 describes the vision-token configuration used to achieve these targets.
  • LLaVA-NeXT-7B: Local clustering configurations target different vision-token budgets for LLaVA-NeXT-7B.Table 13 presents the corresponding configurations across three settings.
  • Video-LLaVA-7B: Local clustering configurations target different vision-token budgets for Video-LLaVA-7B processing eight video frames.Table 14 summarizes dominant and contextual token counts for these configurations.
  • Configuration procedure: For the first three models, dominant and contextual token counts are selected to meet target budgets after pruning on the language-backbone side.The configurations follow trends reported in Figures 7 and the associated implementation tables.
  • Qwen2.5-VL-7B: Qwen2.5-VL-7B uses language-side rank-and-drop at layers 14 and 21 with image-token ratios of 0.5 and 0.25.These settings accompany local clustering configurations for different target token budgets.
Loading 2602.18846v2…