PaperlayerView exact revision on arXiv

Source-linked AI summary

HAP: Head-Adaptive Visual Token Pruning via Cross-Modal Alignment

Yuanhao Sun, Huawei Ji, Yuan Jin, Cheng Deng, Luoyi Fu, Xinbing Wang

arXiv:2608.23921v1cs.CV

TL;DR

High-resolution VLMs produce long visual-token sequences that make prefill costly, and uniform head averaging can dilute prompt-relevant attention because heads align differently. HAP uses PAQ to weight heads and allocate pruning budgets under a target FLOPs budget. Across 18 benchmarks, it achieves state-of-the-art performance-efficiency trade-offs; on LLaVA-1.5-7B, retaining 5.6% of visual tokens preserves 99.1% of original performance and surpasses AutoPrune by 4.2 points.

  • Problem

    Long visual-token sequences create substantial prefill computation and memory overhead, while existing uniform head averaging assumes comparable prompt alignment across heads.

  • Method

    HAP is a training-free pruning framework that uses PAQ-weighted attention aggregation across heads and layers to score prompt-relevant visual tokens under a target FLOPs budget.

  • Results

    Across five architectures and 18 benchmarks, HAP achieves state-of-the-art performance-efficiency trade-offs; at 5.6% token retention, it preserves 99.1% of original performance and beats AutoPrune by 4.2 points.

  • Takeaways & Limitations

    PAQ-guided aggregation provides a prompt-aware alternative to uniform head averaging for training-free visual-token pruning.

  • Takeaways & Limitations

    HAP operates during prefill, assumes paired text queries for its attention-based proxy, and is evaluated mainly on static-image and short-video benchmarks.

Abstract

from arXiv · show

Recent Vision-Language Models encode high-resolution images into long visual token sequences, incurring prohibitive prefill costs. To compress them, existing methods score each visual token by averaging text-to-visual attention uniformly across all heads, which assumes every head matches the query. However, our empirical analysis shows that misaligned heads dominate the average, amplifying background tokens and drowning out fine-grained cues. To address this, we propose PAQ (Prompt-Grounded Attention Quality), a metric quantifying how well each head aligns the prompt with image regions. Built on PAQ, our pruning proceeds in three stages. Given a target FLOPs budget, we first partition the transformer layers into groups and allocate a visual token budget to each. Within each group, we then aggregate per-head attention maps via PAQ-weighted softmax into a group-level matrix. Finally, we score visual tokens by this matrix's magnitude and retain the allocated budget per group. By weighting heads with PAQ, our method scores tokens by attention signals that more faithfully reflect prompt relevance, rather than diluting them through uniform averaging. Across 18 benchmarks, our method delivers state-of-the-art trade-offs. Specifically, on LLaVA-1.5-7B (9 tasks), retaining only \textbf{5.6\%} tokens preserves \textbf{99.1\%} of the original performance, surpassing the strongest baseline AutoPrune by 4.2 points. Code is available in https://github.com/baokou-fw2/HAP.

1 Introduction

High-resolution VLMs face prefill costs from long visual-token sequences, while uniform head averaging can dilute prompt-relevant signals because attention heads exhibit different alignment patterns. HAP addresses this with PAQ-guided head aggregation and budget allocation, achieving strong performance-efficiency trade-offs across benchmarks.

  • Long visual-token sequences create substantial computational and memory overhead during autoregressive prefill, limiting real-time and resource-constrained deployment.
  • Attention heads exhibit sink, dispersed, and optimal alignment modes, so uniform averaging can select tokens worse than the best-aligned head.Sink heads fixate on fixed regions, dispersed heads spread attention broadly, and optimal heads track task-relevant regions.
  • PAQ distinguishes reliable prompt-grounded attention from attention-sink and dispersed patterns using intrinsic post-softmax attention statistics.Higher PAQ corresponds to concentrated attention on semantically prompt-relevant regions, while lower PAQ reflects unreliable patterns.
  • HAP is a training-free pruning strategy that uses PAQ-guided softmax aggregation and target FLOPs budgets to score and retain visual tokens.It groups layers, allocates token budgets, and replaces coarse uniform head averaging with quality-aware aggregation.
  • Across five VLM architectures and 18 benchmarks, HAP achieves state-of-the-art performance-efficiency trade-offs.

2 Related Work

VLM visual-token pruning addresses dense visual sequences through training-based or training-free approaches, while attention-head heterogeneity motivates more selective aggregation.

  • Dense visual encoders generate hundreds to thousands of visual tokens per image, creating computational and memory bottlenecks during prefill.
  • Visual-token pruning methods include training-based approaches that require retraining and training-free approaches based on visual or cross-modal attention.
  • VLM attention heads can show heterogeneous semantics, including visual sinks focused on backgrounds or edges and specialized routing behavior.

3 Method

HAP addresses the limitations of uniform head aggregation and fixed-layer pruning by measuring prompt-grounded attention quality, weighting reliable heads, and allocating token budgets across layer groups. Its pipeline computes PAQ-guided attention scores and retains the highest-scoring visual tokens under a FLOPs-derived geometric budget.

  • 3.1 Rethinking Token Pruning: HAP formulates visual token pruning as query-conditioned relevance ranking, retaining the top-K tokens according to cross-modal attention scores.Previous methods aggregate text-to-visual attention uniformly across all heads in a fixed layer.
  • 3.1 Rethinking Token Pruning: Uniform head aggregation is suboptimal because attention heads exhibit heterogeneous sink, dispersed, and optimal patterns across layers and architectures.Experiments show tokens selected by average attention trail those selected by the best single head on MME and RefCOCO evaluations.
  • 3.2 Prompt-Grounded Attention Quality: PAQ quantifies how much a head’s visual attention distribution is driven by the text query using the ratio I(V; T) / H(V), bounded between 0 and 1.The score is computed from post-softmax intrinsic attention statistics, with high values indicating concentrated prompt-relevant attention and low values reflecting sink or dispersed patterns.
  • 3.3 HAP: HAP partitions the L transformer layers into M consecutive groups and deterministically assigns each group a geometrically decaying visual-token budget from the target FLOPs budget.The group budget is Nv(g) = V/(1 + g)^2, and the stage count is derived from a closed-form cost function without per-task tuning.
  • 3.3 HAP: Within layers and groups, HAP applies PAQ-guided softmax weights to aggregate attention maps, amplifying optimal-head signals while down-weighting suboptimal candidates.The resulting group attention is averaged over text positions to score visual tokens, which are then sorted and pruned according to the allocated group budget.
  • 3.3 HAP: HAP scores visual tokens with cross-modal attention signals that more faithfully reflect prompt relevance than uniform head averaging.The framework aggregates heads within layers and layers within groups before producing final token scores.

4 Experiments

HAP is evaluated across five VLM architectures and 18 benchmarks, showing strong accuracy retention under aggressive visual-token compression. Ablations attribute the gains to PAQ-guided aggregation and demonstrate stable performance across layers and prompt variations.

  • HAP is evaluated on five VLM architectures across 18 benchmarks, including grounding, OCR, and comprehensive VQA tasks.
  • 0.9% overall performance drop results when LLaVA-1.5-7B is compressed from 576 to 32 visual tokens.HAP uses 0.89 TFLOPs at this compression level, while several attention-based baselines degrade by more than 15%.
  • 103.0% relative average is reported on DeepSeek-VL2 Small-16B with 50% token retention across six general VQA benchmarks.On Qwen3-VL-8B, HAP reaches 81.5% on the REC task under 75% pruning, exceeding FastV and PDrop by over 30% absolute margins.

5 Visualization Analysis

Visualization and efficiency analyses show that PAQ-weighted aggregation sharpens attention toward semantically coherent, task-critical regions while reducing latency and KV-cache usage. HAP remains robust across prompt styles and supports substantial speedups under compression.

  • PAQ-weighted head aggregation sharpens noisy attention mixtures into concentrated regions such as jersey numbers and pedestrian silhouettes.Layer aggregation further refines these maps into compact regions including signature blocks and digit strings.
  • Ablation Study: PAQ-selected tokens consistently achieve higher RefCOCO IoU than random and bottom-PAQ tokens across four evaluated VLMs.
  • Prompt Robustness: 1.2% is the maximum variation across standard, detailed, and corrupt prompt styles.Scores are reported as relative performance against the original prompt.
  • Efficiency Analysis: 2.18× speedup results when reducing visual tokens from 576 to 128, while latency falls from 0.48 s to 0.22 s and KV cache usage reaches 36%.
  • Efficiency Analysis: 2.82× speedup is obtained at 32 tokens with 20% KV-cache usage and 99.1% accuracy.At 64 tokens, HAP reaches 2.53× speedup with 25% KV-cache usage and 99.7% performance.

6 Conclusion

The paper concludes that HAP addresses heterogeneous cross-modal alignment across heads and layers using PAQ-guided soft aggregation and adaptive pruning budgets. Experiments across five architectures and 18 benchmarks report state-of-the-art performance-efficiency trade-offs without hyperparameter tuning.

  • HAP is a training-free visual-token pruning framework that assesses heterogeneous cross-modal alignment across attention heads and layers.
  • PAQ-guided soft aggregation dynamically fuses attention heads and layer consensus while allocating pruning budgets by alignment quality.
  • State-of-the-art performance-efficiency trade-offs are demonstrated across five architectures and 18 benchmarks without hyperparameter tuning.

7 Limitations

The paper identifies scope boundaries around decoding, causal validation, modality, and evaluation settings, while reporting supporting evidence for PAQ's alignment behavior.

  • Scope boundaries: HAP operates during prefill and leaves the visual KV cache untouched during decoding.Extending token consolidation to autoregressive decoding is identified as a future direction.
  • Method assumptions: PAQ assumes paired text queries because it is an attention-based proxy derived from cross-modal attention.The paper provides correlational evidence linking high PAQ to correct visual grounding, while fully causal validation remains future work.
  • Open extensions: Applying head-adaptive criteria to unimodal compression remains future work.The paper distinguishes this open direction from its cross-modal VLM setting.
  • Evaluation scope: Evaluation focuses on static-image and shortvideo benchmarks, leaving streaming and ultra-long multimodal contexts for additional study.The paper states that head-level alignment patterns in those settings warrant further study.
  • Alignment evidence: PAQ-softmax produces strongly non-uniform aggregation weights rather than a mild perturbation of uniform averaging.The effective number of heads averages 4.2 and 5.1, with KL divergence of 0.83 and 1.12 nats on the two reported models.
  • Alignment evidence: High-PAQ heads are concentrated mainly in middle-to-deep layers, while early layers contain more sink and dispersed heads.This depth-wise pattern is consistent with retaining more tokens in early layers under the geometric pyramid schedule.

B.4 Factorial Ablation of HAP Components

The factorial ablation shows that PAQ weighting is the primary contributor to HAP's performance, while the geometric budget schedule provides a smaller complementary gain.

  • Experimental setup: 32 tokens retained corresponds to 5.6% retention in the LLaVA-1.5-7B ablation.Table 7 varies scoring rule and budget schedule independently and reports relative score.
  • Component contributions: ∼9.7 points separate PAQ from uniform scoring under the same pyramid budget, making PAQ weighting the largest single gain.The comparison uses LLaVA-1.5-7B with 32 tokens retained, or 5.6%.
  • Component contributions: ∼2.3 points are added by the geometric schedule as a complementary gain over a uniform budget.The reported ranking is PAQ+Pyramid > PAQ+Uniform > Uniform+Pyramid > Uniform+Uniform.

C Compatibility with FlashAttention

HAP avoids materializing the full text-by-vision attention matrix by fusing PAQ reductions into FlashAttention, preserving memory efficiency with low measured overhead.

  • Avoiding materialization: HAP computes PAQ through reductions over vision tokens, producing a scalar per layer or a small vector instead of storing the full attention map.The inner softmax normalizes over text queries and the outer sum collapses vision keys.
  • Fused aggregation: Block-wise reductions are fused into the FlashAttention kernel and aggregated across sequence partitions.The reduction may use sum, max, or higher-order moments, and associative and commutative reductions fit parallel block scheduling.
  • Overhead: < 2% wall-clock overhead is measured relative to base FlashAttention.For a 32-layer model, the extension adds only a few kilobytes through L × C auxiliary scalars, versus hundreds of megabytes for a full attention matrix.
  • Compatibility: HAP is compatible with FlashAttention and other optimized attention implementations without architectural modifications to the VLM backbone.The design inherits their memory efficiency and hardware affinity.
  • Budget schedule: Nv(l) = V/(1 + ml/L)^2 reduces the visual-token count progressively across the m stages.The factor L/m in the closed-form cost reflects per-stage layer depth and makes total cost scale inversely with m for fixed L.
  • Complexity: O(HNtV(l)) PAQ computation is incurred per layer, while token selection adds attention aggregation and sorting costs.Because Nt is much larger than log Nv(l) in practice, sorting is dominated by attention aggregation.
  • Complexity: O(HNtV L/m) is the total PAQ-and-pruning overhead, strictly below the unpruned attention cost under the stated schedule.Relative to full forward-pass FLOPs, the ratio is O(HD/(Am)), with typical configurations making the overhead negligible.

F Statistical Robustness across Retention Ratios

Across retention ratios and architectures, HAP's gains are stable, with small run-to-run variation and additional evidence of strong compression performance on DeepSeek-VL2.

  • Retention-ratio sweep: HAP outperforms the strongest baseline at every retention level on every evaluated architecture.The evaluation averages 10 runs across a systematic retention-ratio sweep on three architectures.
  • Statistical stability: ≤0.7 standard deviations are small relative to the performance gaps between methods.The reported stability comes from mean ± standard deviation across 10 runs.
  • Retention-ratio sweep: 102.0% relative performance is observed at 20% retention on LLaVA-1.5-7B.The paper attributes scores above 100% to pruning background and hallucination-inducing tokens, with tight confidence intervals.
  • Cross-architecture validation: 103.0% relative average is achieved at 50% visual-token retention on DeepSeek-VL2 Small-16B across six general VQA benchmarks.HAP outperforms the unpruned Vanilla model and other pruning methods on every benchmark in this group.

H Robustness under Diverse Prompt and Input Settings

HAP maintains strong performance across corrupted-image and multi-turn settings, while retention-ratio results show stable gains over the strongest available baseline.

  • Robustness under Diverse Prompt and Input Settings: VLM-RobustBench evaluates robustness across 49 augmentation types and graded severity levels, while MultiVerse contains 647 dialogues averaging four turns.The two evaluations cover image corruptions and multi-turn conversations derived from 12 VLM datasets.
  • Robustness under Diverse Prompt and Input Settings: HAP outperforms the strongest available baseline at every retention level, with small standard deviations across 10 runs.The retention-ratio sweep reports relative scores as percentages with mean and standard deviation.
  • Robustness under Diverse Prompt and Input Settings: Table 9 reports DeepSeek-VL2 Small-16B performance across six benchmarks.The supplied passage identifies the evaluation scope but does not report the benchmark values.
  • Robustness under Diverse Prompt and Input Settings: HAP attains the strongest retention on both VLM-RobustBench and MultiVerse, with overall degradation around 3%.This is only marginally higher than the approximately 1% degradation observed in the single-turn prompt robustness study.

I Efficiency in the Decoding Stage and Long-Generation Regimes

HAP’s decoding-stage analysis models end-to-end cost as prefill plus cache-dependent decoding, showing that token pruning continues to improve efficiency and quality in long-generation settings. Its measured end-to-end speedup increases with batch size, while reported FLOPs savings remain positive at long contexts.

  • Decoding Cost Model: HAP prunes visual tokens during prefill, so its efficiency advantage depends on how decoding contributes to total inference cost.The analysis explicitly quantifies how the efficiency gain evolves when decoding dominates.
  • Decoding Cost Model: Per-layer, per-step decode FLOPs equal 4D^2 + 4N_cacheD + 3Dd_ffn, depending on model width, cache size, and feed-forward dimension.Summing over decode steps and layers yields the decoding cost; total end-to-end cost is prefill plus decoding.
  • Long-Generation Efficiency: 75% visual token pruning reduces visual KV-cache memory-bandwidth pressure during decoding, improving throughput for long sequences.The gain limit point S∗ is defined where end-to-end FLOPs reduction falls to 1%.
  • Long-Generation Efficiency: 6.9% end-to-end FLOPs reduction occurs at LLaVA-1.5-7B’s 2K-token maximum context, versus 1.6% at Qwen2.5-VL-7B’s 128K-token maximum context.The PAQ overhead is below 0.5% and is amortized within approximately 100 decode steps.
  • Long-Generation Quality: HAP scores 59.6 on M3CoT, compared with AutoPrune’s 54.7 and the unpruned model’s 60.1.M3CoT averages approximately 1K decoded tokens per example, evaluating quality when decoding dominates total cost.
  • Latency and Batch Scaling: HAP’s end-to-end speedup grows from 1.9× at batch size 1 to 2.4× at batch size 8.The latency breakdown covers prefill, decoding, and end-to-end components on Qwen3-VL-8B with 25% token retention.
Loading 2608.23921v1…