PaperlayerView exact revision on arXiv

Source-linked AI summary

Stage-adaptive Token Selection for Efficient Omni-modal LLMs

Zijie Xin, Jie Yang, Ruixiang Zhao, Tianyi Wang, Fengyun Rao, Jing Lyu, Xirong Li

arXiv:2605.20035v1cs.CV

TL;DR

Dense visual and audio tokens make om-LLM inference expensive, while existing training-free selection methods do not adapt to changing cross-modal importance across layers. SEATS introduces training-free, stage-adaptive pruning across pre-LLM and inner-LLM stages, and at 10% retention achieves 9.3× FLOPs reduction and 4.8× prefill speedup while preserving 96.3% of original performance.

  • Problem

    Dense visual and audio tokens create substantial computation and memory overhead, while existing om-LLM selection methods use fixed pre-LLM pruning that overlooks layer-wise importance changes.

  • Method

    SEATS uses attention-weighted diversity selection before the LLM, progressive block-wise pruning with query-guided window-to-modality allocation inside it, and late-layer non-textual removal.

  • Results

    At a 0.1 token-retention ratio, SEATS achieves a 9.3× FLOPs reduction and 4.8× prefill speedup while preserving 96.3% of original performance.

  • Takeaways & Limitations

    Experiments across five audio-visual benchmarks and two representative om-LLMs show that SEATS provides a strong efficiency-performance trade-off and is applicable as a plug-and-play module.

  • Takeaways & Limitations

    SEATS relies on heuristic hyperparameters tuned per backbone and does not yet address streaming inference when the full sequence is unavailable at prefill time.

Abstract

from arXiv · show

Omni-modal large language models (om-LLMs) achieve unified audio-visual understanding by encoding video and audio into temporally aligned token sequences interleaved at the window level. However, processing these dense non-textual tokens throughout the LLM incurs substantial computational overhead. Although training-free token selection can reduce this cost, existing methods either focus on visual-only inputs or prune om-LLM tokens only before the LLM with fixed per-modality ratios, failing to capture how cross-modal token importance evolves across layers. To address this limitation, we first analyze the layer-wise token dependency of om-LLMs. We find that visual and audio dependencies follow a block-wise pattern and gradually weaken with depth, indicating that many late-layer non-textual tokens become redundant after cross-modal fusion. Motivated by this observation, we propose SEATS, a training-free, stage-adaptive token selection method for efficient om-LLM inference. Before the LLM, SEATS removes spatiotemporal redundancy via attention-weighted diversity selection. Inside the LLM, it progressively prunes tokens across blocks and dynamically allocates the retention budget from temporal windows to modalities using query relevance scores. In late layers, it removes all remaining non-textual tokens once cross-modal fusion is complete. Experiments on Qwen2.5-Omni and Qwen3-Omni demonstrate that SEATS effectively improves inference efficiency. Retaining only 10% of visual and audio tokens, it achieves a 9.3x FLOPs reduction and a 4.8x prefill speedup while preserving 96.3% of the original performance.

1 Introduction

om-LLMs enable unified audio-visual understanding but incur high computational costs from dense visual and audio tokens. SEATS addresses this with training-free, stage-adaptive pruning that follows layer-wise dependency and cross-modal importance, achieving substantial efficiency gains while preserving performance.

  • Dense visual and audio tokens can reach tens of thousands, making full-sequence processing costly because self-attention scales quadratically with sequence length.
  • Existing om-LLM token-selection methods prune only before the LLM with fixed retention ratios, overlooking token-importance changes across layers.
  • Fixed per-modality budgets are inadequate because redundancy, layer dependence, and audio-visual importance vary across stages and temporal windows.
  • SEATS combines pre-LLM attention-weighted diversity selection with progressive inner-LLM pruning and query-guided allocation across temporal windows and modalities.
  • At a 0.1 token-retention ratio, SEATS reduces FLOPs by 9.3× and improves prefill speed by 4.8× while preserving 96.3% of original performance.
  • Experiments on five audio-visual benchmarks and Qwen2.5-Omni-7B and Qwen3-Omni-30B show an efficiency-performance trade-off comparable to full-token performance at reduced cost.

2 Related Work

Training-free token selection spans image-, video-, and omni-modal LLMs, but om-LLM coverage remains limited. Existing om-LLM methods largely operate before the LLM rather than selecting tokens within it.

  • Training-free token-selection methods are surveyed across image-, video-, and omni-modal LLMs.
  • Image-LLM methods use pre-LLM saliency or diversity signals and inner-LLM attention-based pruning.
  • Video-LLM methods exploit inter-frame redundancy and increasingly combine pre-LLM and inner-LLM token reduction.
  • Among the few om-LLM methods, OmniZip is identified as the only training-free approach, and it operates exclusively before the LLM.

3 Token Selection for Om-LLM: Preliminaries and Observations

Om-LLM inputs combine temporally windowed visual and audio tokens with text, creating a large non-textual sequence for LLM processing. Layer-removal experiments reveal block-wise dependence: strong in shallow layers, declining through middle layers, and negligible in late layers.

  • 3.1 Preliminaries: An om-LLM encodes video, audio, and text as visual, audio, and textual token sequences concatenated for autoregressive LLM generation.
  • 3.1 Preliminaries: Visual and audio tokens are grouped into T chronological, non-overlapping windows before being concatenated with intact textual tokens.
  • 3.1 Preliminaries: Because non-textual tokens greatly outnumber textual tokens, efficient prefill primarily reduces visual and audio tokens while preserving text tokens.
  • 3.1 Preliminaries: Layer-wise token retention ratios determine the input length and allocate an overall token budget across visual and audio tokens.
  • 3.2 Observations: Removing non-textual tokens in specific layers measures how strongly those layers rely on visual and audio information.
  • 3.2 Observations: The first 50% of layers are shallow and critically depend on non-textual tokens, middle layers support fusion with reduced dependence, and layers beyond roughly 80% need little non-textual information.
  • 3.2 Observations: SEATS follows this pattern with three stages: pre-LLM selection, inter-LLM selection, and complete non-text removal in late layers.

4 Proposed Method

SEATS is a three-stage, training-free token-selection method that removes pre-LLM redundancy, progressively decays retention ratios inside the LLM, and allocates budgets by query relevance across windows and modalities.

  • Method overview: SEATS uses three stages: pre-LLM selection, inter-LLM selection, and late-layer removal of all non-text tokens.The late stage removes non-text tokens, while the first two stages perform selection before and inside the LLM.
  • Stage I: Pre-LLM selection: Pre-LLM window-based selection uses attention-weighted diversity to retain a compact, diverse subset of visual and audio tokens.The method extends DivPrune to omni-modal inputs, addressing spatiotemporal redundancy within temporal windows.
  • Stage I: Pre-LLM selection: The pre-LLM visual and audio retention ratios are set to rs,v = λRv and rs,a = λRa, with λ > 1, before further pruning inside the LLM.This keeps the pre-LLM retention ratio larger than the target LLM ratio R because retention progressively decreases with depth.
  • Stage II: Block-wise TRR decay: SEATS divides LLM layers into shallow, middle, and late blocks and applies a block-wise decay schedule to token retention ratios.Shallow layers preserve their pre-LLM ratios for cross-modal fusion, whereas middle layers progressively increase pruning strength with depth.
  • Stage II: Block-wise TRR decay: The proposed decay schedule uses three middle-block sub-blocks whose TRRs decrease with exponentially increasing steps.For Qwen2.5-Omni-7B with R=0.3 and λ=1.4, the stated setting yields δ=0.029.
  • Stage II: Query-guided allocation: Middle-layer budgets are allocated first across temporal windows using query relevance, then between visual and audio tokens using modality-specific relevance and layer-wise budgets.Within each window, the method retains the highest-scoring visual and audio tokens according to query-to-token attention.

5 Experiments

Experiments evaluate SEATS on two om-LLMs using multiple audio-visual benchmarks and training-free baselines. SEATS maintains strong accuracy while substantially reducing computation, latency, and memory, and ablations support its stage-adaptive design.

  • Experimental Setup: SEATS is evaluated on five audio-visual test sets using Qwen2.5-Omni-7B and Qwen3-Omni-30B.Baselines include six recent training-free methods adapted to the omni-modal setting.
  • SEATS versus SOTA: At 10% retention, SEATS preserves 96.3% of full-token performance with only 11% of FLOPs on Qwen2.5-Omni-7B.At 35% retention, it scores 49.3 versus 48.7 for full tokens.
  • Efficiency Analysis: At 10% retention, prefill speedup reaches 4.8× and TTFT is reduced by 1.9×.At 35% retention, SEATS achieves 2.1× prefill speedup, 1.4× TTFT reduction, and 18.68 GB peak GPU memory.
  • SEATS versus SOTA: On Qwen3-Omni-30B, SEATS reaches 55.4 versus 55.5 for full tokens at 35% retention and preserves 95.5% performance at 10%.At 10% retention, it uses only 8.3% FLOPs.
  • Pre-LLM Token Selection: Ablations show that saliency and diversity selection are complementary, with their combination scoring 49.3 versus 48.9 for saliency-only and 48.6 for diversity-only selection.Random selection causes the largest degradation among the pre-LLM variants.
  • Inner-LLM Token Selection: Inner-LLM ablations support progressive decay, joint cross-modal allocation, window-local selection, and both inter-window and intra-window budget re-allocation.Removing Stage II yields 48.7, uniform decay yields 49.0, decoupled modalities yield 48.5, and global ranking yields 48.6.
  • Late-block Non-textual Token Removal: Removing late-block token removal changes accuracy only from 49.3 to 49.2 but increases prefill time from 436 ms to 668 ms.The reported increase is 53%.

6 Conclusions

SEATS is presented as a training-free, stage-adaptive token selection module for efficient omni-modal inference. Across two omni-modal models and five audio-visual benchmarks, it offers an efficiency-performance trade-off while preserving task performance.

  • Conclusion: SEATS is a training-free, stage-adaptive token selection method for efficient om-LLM inference.The method is described as a plug-and-play module for existing om-LLMs.
  • Conclusion: Experiments on two om-LLMs and five audio-visual benchmarks show reductions in FLOPs, prefill latency, and memory consumption while preserving task performance.The paper characterizes this as a state-of-the-art efficiency-performance trade-off.

A Derivation of the Scale Factor δ

The derivation expresses the overall token-retention ratio as a layer-count-weighted average of shallow and middle-block retention ratios because late-block non-textual tokens are removed.

  • TRR Derivation: The overall TRR R is the layer-count weighted average of the per-block TRRs.Only the shallow and middle blocks contribute because the shallow block retains tokens at λR and the late block has TRR 0.
  • TRR Derivation: The weighted sum combines the shallow retention ratio with the three middle sub-block ratios across their respective layer ranges.The expression includes rs, rm1, rm2, and rm3 multiplied by the corresponding block lengths.
  • Scale-Factor Derivation: The constant C collects the layer-count terms used in solving the retention constraint.It is defined from the shallow, middle-boundary, and late-layer indices.
  • Numerical Example: For Qwen2.5-Omni-7B with L=28, Ls=16, Lm1=19, Lm2=21, Ll=24, and λ=1.4, C ≈−42.759 and R=0.3 gives δ ≈0.02947.These values instantiate the derived scale factor for the stated configuration.

B Experimental Details

The experiments use diverse audio-visual benchmarks, two omni-modal model configurations, and modality-aware retention budgets. Baselines are adapted to either audio-intact or both-selected token compression settings for consistent comparison.

  • Benchmarks: The five benchmarks span varied temporal scales and differ in their reliance on audio and visual cues.Together they assess compression robustness across duration and modality dependencies.
  • Benchmarks: WorldSense contains 1,662 synchronized audio-visual videos and 3,172 questions across 26 cognitive tasks, with each question requiring both modalities.Its average video duration is approximately 141 seconds.
  • Benchmarks: Daily-Omni uses 684 real-world videos lasting 30–60 seconds and 1,197 questions requiring audio-event and visual-action correspondence.Its defining property is temporal precision rather than global semantic matching.
  • Benchmarks: OmniVideoBench includes 628 videos from several seconds to 30 minutes and 1,000 questions with multi-step reasoning chains.The chains record the modalities and evidence involved.
  • Benchmarks: Video-MME contains 900 videos from 11 seconds to 1 hour and 2,700 annotated question-answer pairs, evaluated without subtitles.Removing subtitles isolates the effect of token compression from external textual cues.
  • Benchmarks: LVOmniBench contains 275 videos lasting 10–90 minutes and 1,014 questions, stressing multimodal retention and temporal localization over extended sequences.Its average duration is 34.5 minutes, totaling 140 hours.
  • Retention Budgets: The overall budget satisfies Rv · nv + Ra · na = R · (nv + na), with audio-intact and both-selected evaluation modes.Each 2-second window contains 288 visual tokens and 50 or 26 audio tokens depending on the model.
  • Baselines: Baselines include image, video, and omni-modal methods adapted to the omni-modal setting under matched retention ratios.The compared methods include FastV, VisionZip, DivPrune, DyCoke, FastVID, OmniZip, and Random.

C More Experimental Results

Table 8 provides full Qwen3-Omni-30B results, complementing the Qwen2.5-Omni-7B results reported earlier.

  • Table 8 reports the full comparison on Qwen3-Omni-30B-A3B-Instruct.
  • The Qwen3-Omni-30B results complement the Qwen2.5-Omni-7B results in Section 5.2.
  • The section extends evaluation across the two om-LLM backbones described in the paper.

C.1 Detailed Hyperparameter Analysis

The hyperparameter analysis reports robustness across tested λ values at 35% retention on Qwen2.5-Omni, alongside supporting experimental and reproducibility details.

  • Hyperparameter sensitivity: Performance exhibits small variance across tested λ values at a 35% retention ratio on Qwen2.5-Omni.
  • Practical implications: The work frames efficient om-LLM inference as reducing computational cost and memory consumption for resource-constrained settings.
  • Reproducibility: The paper lists licenses and repository links for several baseline, evaluation, and model resources.
  • Hyperparameter sensitivity: Table 9 reports the effect of the scale hyperparameter λ under the unified experimental setup.

D.2 Limitations

The framework depends on backbone-specific heuristic settings and does not yet automatically adapt to new om-LLMs or streaming inference.

  • Current limitations: The framework relies on heuristic hyperparameters tuned separately for each backbone.
  • Current limitations: Automatically adapting these configurations to new om-LLMs remains future work.
  • Current limitations: Extending SEATS to streaming inference is also identified as a future direction because the full sequence is unavailable at prefill time.
Loading 2605.20035v1…