PaperlayerView exact revision on arXiv

Source-linked AI summary

STAR-Pro: Stage-Wise Token Adaptive Reduction with Progressive Refinement for Efficient Large Vision-Language Models

Yichen Guo, Tinghao Wang, Qizhe Zhang, Lingbei Meng, Yuan Zhang, Jiajun Cao, Hao Jiang, Chenwei Wu, Jixian Wu, Sixiang Chen, Tao Luo, Hongyang Cheng, Kai Tang, Chenxi Li, Renyuan Li, Xiande Huang, Wenya Wang, Shanghang Zhang

arXiv:2609.05916v1cs.CVcs.AI

TL;DR

LVLMs process large numbers of visual tokens, creating computational overhead, while pruning can discard feature information and token importance changes across decoder layers. STAR-Pro combines pre-fusion feature coverage with progressive attention-based refinement; across the reported evaluations it shows accuracy–efficiency trade-offs, including on LLaVA-Video-7B, where it reduces visual tokens by 90.5%, retains 92.7% of baseline performance, and achieves a 2.24× inference speedup.

  • Problem

    LVLMs process hundreds to thousands of visual tokens, creating substantial computational overhead, while aggressive pre-LLM pruning loses complementary visual information and token importance changes across decoder layers.

    Source: §1 · pp. 1–2

  • Method

    STAR-Pro is a training-free two-stage framework that uses pivoted QR to build a pre-fusion feature-coverage pool and evolving text-to-visual attention to progressively prune survivors in the decoder.

    Source: §1 · p. 2

  • Results

    STAR-Pro shows accuracy–efficiency trade-offs across models, benchmarks, and pruning ratios; on LLaVA-Video-7B it reduces visual tokens by 90.5%, retains 92.7% of baseline performance, and achieves a 2.24× inference speedup.

    Source: §2 · p. 3, §5 · p. 8

  • Takeaways & Limitations

    The findings support combining broad pre-fusion feature coverage with progressive refinement as visual-token importance evolves across decoder layers.

    Source: §1 · p. 2

Abstract

from arXiv · show

Large vision-language models (LVLMs) achieve strong multimodal understanding, but the hundreds to thousands of visual tokens they process impose substantial computational overhead, motivating training-free visual token pruning. In this work, we conduct two complementary analyses of visual token pruning. First, we measure the feature-space coverage of tokens retained before cross-modal fusion and find that aggressive pruning discards substantial visual information. Second, we track text-to-visual attention across decoder layers and find that the visual tokens considered important change substantially with depth, making one-shot pruning decisions unreliable. Together, these findings show that effective pruning should preserve broad visual coverage before fusion and progressively refine the retained tokens as cross-modal evidence evolves during fusion. We therefore propose STAR-Pro (STage-Wise Adaptive Token Reduction with Progressive Refinement), a training-free two-stage framework. Its Adaptive Stage applies pivoted QR to construct an over-budget feature-coverage candidate pool, while its Progressive Stage uses evolving text-to-visual attention at selected decoder layers to prune a nested survivor set under a target layer-average token budget. Extensive experiments across seven LVLMs spanning multiple architectures and 18 image and video benchmarks demonstrate the effectiveness of STAR-Pro under aggressive pruning. On LLaVA-Video-7B, STAR-Pro reduces visual tokens by 90.5%, retains 92.7% of baseline performance, and achieves a $2.24\times$ measured inference speedup. Code is available at https://github.com/EasonAI-5589/starpro.

1 Introduction

LVLMs process hundreds to thousands of visual tokens, creating substantial computational overhead, while aggressive pre-LLM pruning loses visual information and token importance changes across decoder depth. STAR-Pro addresses these observations with pre-fusion feature coverage and progressive refinement using evolving text-to-visual attention.

1 Introduction

  • LVLMs commonly generate hundreds to thousands of visual tokens per image, with token counts escalating further for high-resolution images and video.This visual-token volume imposes substantial computational overhead.Source: §1 · p. 1
  • Across decoder-layer gaps of 1–5, top-K attended-token continuity stays well below 1, indicating that visual-token importance changes with depth.The measurement uses LLaVA-1.5-7B, K=64, and the mean over 500 MME images.Source: §1 · p. 2
  • Aggressive pre-LLM pruning loses complementary visual information before cross-modal reasoning, as feature retention and coverage of distinct feature groups decline with shrinking token budgets.The study measures feature retention with η and uses PCA to visualize whether selected tokens span distinct feature groups.Source: §1 · p. 2, Figure 2 · p. 2
  • On LLaVA-Video-7B, STAR-Pro reduces visual tokens by 90.5%, retains 92.7% of baseline performance, and achieves a 2.24× inference speedup.The framework is training-free and requires no architectural modifications.Source: §1 · p. 2
  • STAR-Pro combines an over-budget, pivoted-QR feature-coverage pool before fusion with progressive pruning based on evolving text-to-visual attention inside the decoder.The two stages address pre-fusion coverage and changing token importance across decoder depth.Source: §1 · p. 2

2. We propose STAR-Pro, a training-free two-stage frame-

The paper proposes STAR-Pro as a training-free two-stage visual-token pruning framework and evaluates its accuracy–efficiency trade-offs across models and pruning ratios.

2. We propose STAR-Pro, a training-free two-stage frame-

  • STAR-Pro is evaluated across multiple model architectures, benchmarks, and pruning ratios, with strong accuracy–efficiency trade-offs especially under aggressive pruning.Ablation studies also demonstrate the benefit of each stage.Source: §2 · p. 3
  • The Adaptive Stage constructs an over-budget feature-coverage candidate pool, and the Progressive Stage refines a nested survivor set using evolving text-to-visual attention.The approach is presented as a training-free two-stage framework.Source: §2 · p. 3
  • Ablation studies demonstrate the benefit of each of STAR-Pro’s two stages.The passage does not specify the ablation values.Source: §2 · p. 3

2 Related Work

LVLMs face growing visual-token costs, particularly in high-resolution and video settings. Training-free reduction methods prune at different points in the vision-language pipeline, while STAR-Pro combines pre-LLM and in-LLM pruning with distinct feature-space coverage and attention-based refinement.

2.1 Large Vision-Language Models (LVLMs)

  • Video LVLMs can process sequences exceeding 10K or even 100K tokens, imposing substantial computational overhead and limiting inference speed.High-resolution and dynamic-resolution image processing also increases visual-token counts.Source: §2.1 · p. 3

2.2 Token Reduction for LVLMs

  • Training-free pruning methods select tokens before the LLM using attention, feature diversity, reconstruction, or spatial-coverage strategies, or prune within decoder layers using cross-modal attention.Some learned compressors require additional training and, in several cases, architectural modifications.Source: §2.2 · p. 3
  • STAR-Pro combines pre-LLM and in-LLM pruning, using feature-subspace coverage for its candidate pool and evolving text-to-visual attention to refine survivors across decoder layers.The paper distinguishes its feature-subspace coverage from spatial or set-level coverage used by some baselines.Source: §2.2 · p. 3

3 Method

STAR-Pro first builds a feature-coverage candidate pool from vision-side representations, then progressively prunes survivors using text-to-visual attention at selected decoder layers. Its pruning schedule targets a visual-token budget averaged across decoder depth.

3 Method

  • STAR-Pro uses a vision-side candidate pool followed by attention-based refinement at selected decoder layers.The two stages are described as complementary.Source: §3 · p. 3, Figure 3 · p. 4

3.1 Preliminaries

  • The layer-average budget T is the visual-token count averaged over all L decoder layers, allowing the retained count to vary across depth.Visual tokens dominate per-layer cost because the visual-token count n greatly exceeds the text-token count n_q.Source: §3.1 · p. 3

3.2 Adaptive Stage: Feature-Coverage Candidate Pool Construction

  • Pivoted QR builds an over-budget candidate pool by selecting visual tokens that preserve complementary directions in the feature space.The selection approximates maximum-volume subset selection greedily, choosing each remaining token by its residual orthogonal to the selected span.Source: §3.2 · pp. 3–4

3.3 Progressive Stage: Refinement via Evolving Text-to-Visual Attention

  • At selected decoder layers, the Progressive Stage averages text-to-visual attention over heads and text tokens to score surviving tokens, then retains the highest-scoring tokens in nested sets.Non-increasing pruning budgets are chosen so that the token count averages to T across decoder layers; removed tokens are not reintroduced.Source: §3.3 · p. 4

4 Experiments

The experiments evaluate STAR-Pro across seven LVLMs and 18 image and video benchmarks, reporting strong performance retention under aggressive pruning alongside inference-efficiency gains. Ablations examine the contribution of both stages and key configuration choices.

4.1 Experimental Setup

  • Experiments cover seven LVLMs, 15 image and three video benchmarks, and nine training-free baselines.For LLaVA-1.5/NeXT-7B, the settings use α=2 and pruning layers {12, 20}; NeXT budgets count tokens from five image crops.Source: §4.1 · pp. 4–5

4.2 Main Results

  • At 94.4% compression, STAR-Pro retains 94.9% and 96.1% of baseline performance on LLaVA-1.5-7B/13B, and 97.0% and 97.5% on LLaVA-NeXT-7B/13B.It is best or tied for best in 11 of 12 model–compression settings, trailing SparseVLM by 0.1 points in the remaining setting.Source: §4.2 · p. 5, Table 1 · p. 5

4.3 STAR-Pro for Video Understanding

  • At 90.5% compression, STAR-Pro retains 92.7% of baseline performance and ranks first on all three reported video benchmarks.At 81.1% compression, it retains 96.7% of baseline performance and outperforms the second-best method by 1.5 percentage points.Source: §4.3 · p. 5, Table 2 · p. 5

4.4 STAR-Pro for Advanced Architectures

  • On Qwen3-VL, STAR-Pro retains 95.6% and 91.7% of baseline performance at budgets 256 and 128; on InternVL3, it retains 97.7% and 92.2%.It leads all eight benchmarks on InternVL3 at both budgets; margins over runners-up increase as budgets tighten.Source: §4.4 · p. 5, Table 3 · p. 6

4.5 Efficiency Studies

  • At 16 tokens per video frame, STAR-Pro retains 92.7% of baseline performance with a 2.24× speedup; decoder prefilling accounts for 86.2% of net latency savings.The efficiency study also compares latency, operator-counted TFLOPs, peak GPU memory, speedup, and relative performance across LLaVA-NeXT and LLaVA-Video budgets.Source: §4.5 · pp. 6–7, Figure 4 · p. 7

4.6 Ablation Studies

  • Removing either stage hurts performance; α=2 performs best at tighter budgets, and the deployed (12, 20) schedule ties the best Mean-4 score of 68.6.The deployed schedule remains among the high-scoring configurations at nearby pruning depths.Source: §4.6 · pp. 7–8, Figure 5 · p. 7

5 Conclusion

The conclusion identifies broad pre-fusion coverage and progressive refinement as requirements for visual token pruning, then presents STAR-Pro as a training-free framework for them. It highlights results on LLaVA-Video-7B under 90.5% token reduction.

5 Conclusion

  • STAR-Pro combines pivoted-QR coverage selection before fusion with decoder refinement using evolving text-to-visual attention.On LLaVA-Video-7B, it reduces visual tokens by 90.5%, retains 92.7% of baseline performance, and achieves a 2.24× inference speedup.Source: §5 · p. 8

Appendix Roadmap

  • The appendix is organized into five categories covering experimental setup, implementation, additional results and efficiency, extended ablations, and empirical-study protocols..Source: Appendix Roadmap · p. 10

A Experimental Setup

  • Appendix A documents the model architectures, evaluation benchmarks, and baseline methods used in the experiments.Source: A Experimental Setup · p. 10

A.1 Model Architectures

This section describes the evaluated model and benchmark settings, comparison methods, evaluation protocol, and STAR-Pro’s two algorithm stages. Across these components, the method selects a feature-coverage candidate pool before decoder refinement under a layer-average token budget.

A.1 Model Architectures

  • LLaVA-1.5 uses 576 tokens at 336 × 336, LLaVA-NeXT uses 2,880 tokens at 672 × 672, and LLaVA-Video uses 10,816 tokens from 64 pooled frames.Qwen3-VL’s single-image reference uses 1,296 tokens, while InternVL3 uses 1,280 tokens at 448 × 448.Source: §A.1 · p. 10

A.2 Evaluation Benchmarks

  • The benchmark suite spans image QA, visual reasoning, OCR, object hallucination, multilingual evaluation, diagram understanding, and long-video understanding.The descriptions include low-quality user-captured images, text in images, perception and cognition, and video splits by duration.Source: §A.2 · pp. 10–11

A.3 Baseline Methods

  • Baselines cover attention, staged dropping, feature diversity, clustering, saliency and coverage, spatial scanning, and crop-adaptive holistic-context selection.The listed methods include FastV, PyramidDrop, SparseVLM, VisionZip, DivPrune, CDPruner, SCOPE, VScan, and HoloV.Source: §A.3 · p. 11

A.4 Evaluation Protocol

  • Accuracy runs use batch size 1 and deterministic greedy decoding, with each harness retaining its task-specific metric and benchmark scorer.For multi-layer methods, the reported budget is the decoder-layer average; for one-shot pre-LLM methods, it is the number entering the decoder.Source: §A.4 · pp. 11–12

B.1 Algorithm Details

  • Pivoted QR selects tokens with the largest residual norms to build the feature-coverage candidate pool, then restores selected indices to original token order.The pivot rule determines membership, not the decoder’s spatial sequence order.Source: §B.1 · p. 12
  • The pruning schedule constrains survivor counts to satisfy the target layer-average token budget across decoder layers.The schedule uses nested budgets and pruning at specified layers.Source: §B.1 · p. 12
  • At selected decoder layers, STAR-Pro recomputes text-to-visual scores over current survivors and retains the top-scoring tokens within non-increasing budgets.The survivor sets are nested, so removed tokens do not reenter later candidate sets.Source: §B.1 · p. 12

B.2 Implementation Details

Implementation uses architecture-specific pruning checkpoints and a validated layer-average budget schedule. The candidate pool and attention scoring are adapted to model interfaces and input groupings while retaining the two-stage procedure.

B.2 Implementation Details

  • At a fixed target budget T, the Adaptive Stage uses candidate multiplier α, defaulting to 2, while the Progressive Stage uses specified pruning layers and integer survivor counts.The implementation applies residual pivoted QR to final vision-side embeddings and carries selected indices into the decoder without token merging.Source: §B.2 · p. 13
  • Pruning-layer locations and survivor counts are calibrated jointly for each architecture rather than transferred by a fixed fraction of decoder depth.Earlier pruning changes the sequence seen by later blocks and thus the effective depth at which cross-modal evidence stabilizes.Source: Architecture-Specific Pruning Layers · p. 13, Table 5 · p. 14
  • InternVL3 allocates four times as many candidate-pool tokens to its global thumbnail as to each local crop, preserving 2T candidates in total.For four local crops, quotas are [64, 64, 64, 64, 256] at T=256 and [32, 32, 32, 32, 128] at T=128.Source: InternVL3-Specific Group Allocation · p. 13
  • The standard two-layer design starts with αT candidates and selects survivor counts so the average across all decoder layers equals T.Integer-valued survivor counts obey a front-heavy constraint that retains more tokens in earlier layers.Source: Token Budget Schedule Derivation · p. 13
  • Latency and speedup measurements use a controlled single-A800-80GB protocol, and code, evaluation configurations, and schedule-validation utilities are available.Accuracy-worker timing is not used for efficiency comparisons.Source: Token Budget Schedule Derivation · p. 13

C.1 LLaVA Series

Across the twelve LLaVA model-budget settings, STAR-Pro has the highest aggregate in ten, ties one, and trails the best reported aggregate by 0.1 points in one. Its benchmark-level results vary, so aggregate leadership does not imply winning every task.

C.1 LLaVA Series

  • Across twelve LLaVA model-budget settings, STAR-Pro leads ten aggregates, ties one, and trails the best by 0.1 relative-performance points in one.The reported margins over the strongest alternative at the tightest budget are 0.8, 2.1, 0.4, and 0.6 points across the four models.Source: §C.1 · p. 14, Table 6 · p. 15, Table 7 · p. 16
  • Benchmark-level results are not uniform: GQA and MMBench-CN recur as strengths, while POPE, MME, and reasoning-benchmark winners vary by model and budget.The reported pattern is an aggregate balance across complementary tasks, not uniform per-column dominance.Source: §C.1 · pp. 14, 16
  • 98.9%, 97.1%, and 94.9% of baseline performance are retained by LLaVA-1.5-7B STAR-Pro at 128, 64, and 32 tokens, respectively.STAR-Pro has the highest aggregate at all three budgets; its benchmark performance varies, including lower-budget MMMU values that do not lead.Source: §C.1 · p. 14
  • 97.9% baseline aggregate is retained by LLaVA-1.5-13B STAR-Pro at 128 tokens, then it leads at 64 tokens and reaches 96.1% at 32 tokens, 2.1 points above CDPruner.At 32 tokens, STAR-Pro leads seven of ten individual benchmarks.Source: §C.1 · pp. 14–15
  • 99.6% of baseline performance is retained by LLaVA-NeXT-7B at 640 tokens, tied at reported precision for best aggregate; it then leads at 320 and 160 tokens.At 160 tokens, its 97.0% relative performance is 0.4 points above CDPruner.Source: §C.1 · p. 15
  • 100.3% and 97.5% of baseline aggregate are retained by LLaVA-NeXT-13B STAR-Pro at 640 and 160 tokens, the highest reported aggregates at both budgets.At 320 tokens, STAR-Pro leads the aggregate by 0.7 relative-performance points over CDPruner and SCOPE.Source: §C.1 · p. 15

C.2 LLaVA-Video

On LLaVA-Video-7B, STAR-Pro’s aggregate advantage is strongest at the tighter video-token budget, and its measured end-to-end speedup is supported by large prefill savings relative to one-time selection cost.

C.2 LLaVA-Video

  • At 64 × 16 retained tokens, STAR-Pro retains 92.7% of baseline performance, 2.8 percentage points above the runner-up, and ranks first on all eight reported benchmark and subtask columns.The aggregate combines MLVU m-avg, LongVideoBench val, and Video-MME without subtitles.Source: §C.2 · pp. 16–17
  • At 64 × 32 retained tokens, STAR-Pro preserves 96.7% of baseline performance, 1.5 percentage points above VScan, while LongVideoBench ranks second and relation and long-duration evaluation remain weaker.At this budget, STAR-Pro leads MLVU and Video-MME, but the detailed results show its remaining weaknesses in relation reasoning and long-duration evaluation.Source: §C.2 · pp. 16–17
  • At 16 tokens per frame, STAR-Pro leads all four Video-MME columns, with margins of 2.7, 3.0, 1.6, and 2.0 points over the strongest competing values, respectively.Its LongVideoBench validation score also leads by 1.3 points at this budget, following a reversal in the relation-subset comparison.Source: §C.2 · p. 17

C.3 Efficiency Protocol and Adaptive-Stage Overhead

  • The measured 2.24× end-to-end speedup accompanies an 872.9 ms decoder-prefill reduction versus a 40.6 ms Adaptive-Stage selection cost.In the separate 16-token diagnostic, cached decoding takes 47.2 ms across 15 steps, while Adaptive Stage and QR execute only once.Source: §C.3 · p. 18

D.1 Pruning-Schedule Robustness at a Fixed Average-Token Budget

A matched-budget sweep finds that the deployed (12, 20) pruning schedule performs competitively within a broad neighboring region, while the evidence is scoped to one model and budget.

D.1 Pruning-Schedule Robustness at a Fixed Average-Token Budget

  • The 100 tested schedules vary two pruning-layer positions while recomputing integer survivor counts to preserve the same layer-average visual-token budget.Mean-4 averages POPE, GQA, TextVQA, and MME/20.Source: §D.1 · pp. 18–19
  • Mean-4 68.6 for the deployed (12, 20) schedule ties the best measured score and is about 0.5 points above CDPruner’s 68.1 reference; roughly 61% of the search domain meets or exceeds that reference.The competitive region is contiguous around the deployed schedule, rather than consisting of isolated peaks.Source: §D.1 · p. 19, Figure 6 · p. 20
  • The schedule-robustness evidence is limited to LLaVA-1.5-7B at T=64 over the displayed search range and does not establish global optimality across models or budgets.The quantitative claims use the 100 measured configurations; interpolation is only for rendering and partitioning the plotted grid.Source: §D.1 · p. 19

D.2 Hyperparameter Analysis of the Adaptive and Progressive Stages

The candidate-pool multiplier has a stable performance neighborhood around its deployed setting; moderate per-layer token reallocation also produces little change, alongside additional empirical-study protocol details.

D.2 Hyperparameter Analysis of the Adaptive and Progressive Stages

  • Across four models and three budgets, α=2 matches or exceeds the strongest prior method in every setting, while α values from 1.5 to 2.5 stay within 1.4 Rel. points per model and budget.Performance generally peaks near α=2 and degrades at α=1 and α=3.Source: §D.2 · p. 20, Figure 7 · p. 21
  • Random and Stride trail the complete configuration on POPE, GQA, and TextVQA at every budget; Stride is higher on MME at T=128, and pruning-layer controls compare P=0, P=3, and P=4 against deployed P=2.The P=3 and P=4 placements preserve the same layer-average token budget.Source: §D.2 · p. 22

D.3 Robustness to the Per-Layer Token Allocation

  • Across twelve model–budget pairs, moderate front-heavy token reallocation changes Rel. by only −0.3 to +0.5 percentage points; nine variants improve, one is unchanged, and two decline by at most 0.3 points.The reallocation keeps pruning layers and α=2 fixed and preserves the exact layer-average token budget.Source: §D.3 · p. 22

E Additional Details for the Empirical Study

E.1 Feature-Retention Metric and PCA Protocol

The appendix specifies feature-space coverage and selection-continuity diagnostics, including their computation and the limits on interpreting or comparing them.

E.1 Feature-Retention Metric and PCA Protocol

  • The feature-retention ratio η measures the fraction of visual feature energy recoverable from the span of selected tokens, using raw, uncentered CLIP features before the multimodal projector.Coverage is averaged over 30 MME images and is a within-representation diagnostic, not an accuracy metric; comparisons require the same feature layer, preprocessing, image set, and budget.Source: §E.1 · pp. 22–23
  • The qualitative PCA insets use one image, a basis shared across all 576 tokens, and mark each method’s first six selected tokens; they use normalized, centered features rather than η’s raw features.The insets visualize how quickly selectors reach distinct feature groups and do not enter the η computation.Source: §E.1 · p. 23

E.2 Selection-Continuity Metric and Protocol

  • Selection continuity tracks set membership for the 64 highest-attention visual tokens across layers of LLaVA-1.5-7B, averaging pairwise IoU over 500 MME images.The analysis reports per-layer curves and means for layer gaps 1–5; values below one indicate changing membership, not changes in attention magnitude or task accuracy.Source: §E.2 · p. 23
Loading 2609.05916v1…