PaperlayerView exact revision on arXiv

Source-linked AI summary

OmniPack: Unified Token Compression for Efficient Omni-modal Large Language Models

Wanshun Su, Yang Shi, Feihu Liu, Ziwen Yu, Yan Min, Zhuoran Zhang, Qixun Wang, Haotian Wang, Shixuan Liu, Yuanxing Zhang, Peng Wu, Chengfu Huo, Liang Ding

arXiv:2608.03812v1cs.CV

TL;DR

Omni-LLM compression under aggressive token budgets remains limited in modeling long-range audio-visual structure and exploiting query-conditioned cross-modal evidence. OmniPack coordinates pre-LLM structural compression with inner-LLM semantic refinement, and across five benchmarks and three backbones achieves the best reported performance–efficiency trade-off, preserving 95.6% of original performance while reducing FLOPs by 10.0× at 7.5% token retention.

  • Problem

    Under aggressive token budgets, existing Omni-LLM compression methods remain limited in modeling long-range audio-visual structure and exploiting query-conditioned cross-modal evidence.

    Source: §1 · p. 2

  • Method

    OmniPack is a training-free framework that combines modality-specific structural compression before the LLM with task-conditioned semantic refinement within it.

    Source: §1 · p. 2, §3.2 · p. 3

  • Results

    Across five benchmarks and three backbones, OmniPack consistently achieves the best performance–efficiency trade-off; at 7.5% token retention, it preserves 95.6% of original performance while reducing FLOPs by 10.0×.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The reported results show that substantial token compression can retain multimodal understanding performance across the tested settings.

    Source: §1 · p. 2, §5 · p. 8

    All text tokens remain unchanged throughout compression.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Omni-modal large language models (Omni-LLMs) have achieved remarkable performance on audio-visual understanding tasks, but processing long and highly redundant visual and audio token sequences incurs substantial computational overhead, demanding aggressive token compression for efficient deployment. Existing methods often degrade at low token budgets: pre-LLM compression may discard structurally important and globally distributed evidence, whereas inner-LLM compression often underexploits query-conditioned audio-visual collaboration. To address these limitations, we propose OmniPack, a training-free framework that coordinates structural compression before the LLM with task-relevant semantic refinement within the LLM. Before the LLM, OmniPack removes structural redundancy through modality-specific importance, global coverage, and similarity-aware merging. After sufficient multimodal interaction, it further consolidates diverse, task-relevant representations through textual guidance and audio-visual collaboration. Extensive experiments on five benchmarks with three Omni-LLM backbones demonstrate that OmniPack consistently achieves the best performance-efficiency trade-off across diverse retention ratios, outperforming all existing methods. Notably, on Qwen2.5-Omni-7B, OmniPack preserves 98.0% of the original performance while reducing FLOPs to 16.7%, and still retains 92.9% of the original performance with only 6.8% of the original FLOPs.

1. Introduction

OmniPack addresses information loss and weak audio-visual coordination under aggressive token compression with a training-free, two-stage framework. Its experiments report strong performance and efficiency across Omni-LLM backbones and benchmarks.

1. Introduction

  • At a final token retention ratio of 7.5%, OmniPack reduces FLOPs by 10.0× and achieves a 4.5× prefill speedup while preserving 95.6% of original performance.Experiments show it generalizes across three Omni-LLM backbones and five benchmarks and offers the best performance–efficiency trade-off among existing token compression methods.Source: §1 · p. 2
  • Dense audio-visual tokens inflate inference compute and memory, while local scoring can miss sparse long-range evidence and discarding tokens can lose complementary context.Existing cross-modal guidance may use weakly aligned features before the LLM, while inner-LLM compression primarily uses textual guidance without explicitly modeling audio-visual collaboration.Source: §1 · pp. 1–2
  • OmniPack uses modality-specific pre-LLM compression to preserve important evidence and global coverage, followed after sufficient multimodal interaction by query-conditioned inner-LLM refinement using textual guidance and audio-visual collaboration.The framework merges removed information into retained representatives.Source: §1 · p. 2

2. Related Work

Omni-LLMs unify multimodal processing, while prior compression methods leave a gap in preserving audio-visual structure and exploiting task-relevant semantics under aggressive compression.

2.1. Omni-modal Large Language Models

  • Omni-LLMs extend visual-centric VLMs [1] [37] [39] [51] to jointly process text, images, video, and audio, but dense audio-visual inputs raise inference costs.Their cross-modal reasoning supports understanding complex, dynamic scenarios.Source: §2.1 · p. 2

2.2. Token Compression in Omni-LLMs

  • Prior work compresses audio-visual tokens via audio-guided video compression (OmniZip [34]), vision-guided audio selection (OmniSIFT [7]), contextualization [15], recoverability [19], aligned units [6], and query-guided pruning [26] [42].Token compression has also been studied for image [2] [31] [43] [47], video [8] [10] [28] [33], and audio [20] [23] inputs.Source: §2.2 · p. 2
  • Existing methods introduce cross-modal guidance before sufficient audio-visual interaction or prune inside the LLM without explicit collaboration, limiting structural and task-semantic preservation under aggressive compression.This gap motivates coordinating structure-aware pre-LLM compression with query-conditioned audio-visual compression inside the LLM.Source: §2.2 · pp. 2–3

3. Method

OmniPack combines modality-specific structural compression before the LLM with query-conditioned semantic refinement inside it, while leaving text tokens unchanged. Its pre-LLM stage preserves salient and representative information, and its inner-LLM stage uses textual relevance and audio-visual collaboration to refine retained tokens.

3.1. Preliminary

  • All text tokens remain unchanged throughout compression.Source: §3.1 · p. 3

3.2. Overview of OmniPack

  • OmniPack pairs pre-LLM structural compression with query-conditioned inner-LLM refinement, shifting from structural cues to task-conditioned semantics with minimal information loss.Source: §3.2 · p. 3

3.3. Modality-Specific Pre-LLM Compression

  • Pre-LLM compression independently scores each modality using encoder attention and modality-specific variation cues to preserve salient tokens.Video cues capture adjacent-frame changes and spatial distinctiveness; audio variation indicates stronger acoustic boundaries or transient events.Source: §3.3 · pp. 3–4
  • To broaden coverage beyond salient regions, pre-LLM compression uses feature and positional distances with DPC-KNN to select representative tokens.The joint distance accounts for representation similarity and temporal or spatiotemporal proximity, so distant tokens with similar features are not treated as equivalent.Source: §3.3 · p. 4
  • Rather than discard unselected tokens, similarity-aware merging assigns them to retained representatives and aggregates their information using feature similarity, positional proximity, and representative importance.Importance-weighted aggregation lets representatives absorb nearby redundant information while mitigating information loss.Source: §3.3 · p. 4

3.4. Query-Conditioned Inner-LLM Compression

  • After the first ℓ Transformer blocks, inner-LLM compression scores tokens by textual relevance, audio-visual collaboration, and within-modality representativeness, then selects a relevant and diverse subset.Unselected hidden states are assigned to their most cosine-similar retained representative and incorporated through relevance-modulated similarity-weighted averaging.Source: §3.4 · pp. 4–5

4. Experiment

Across five audio-visual benchmarks and three backbones, OmniPack combines pre-LLM structural compression with query-conditioned inner-LLM refinement. Experiments report strong performance-efficiency results and show the two stages work effectively together.

4.1. Experimental Settings

  • Evaluations cover five audio-visual benchmarks, three Omni-LLM backbones, and pre-LLM retention ratios of 25%, 20%, 15%, and 10%.Benchmarks span audio- and video-centric understanding, short- and long-form videos, and perception-to-reasoning tasks; models are Qwen2.5-Omni-3B/7B and MiniCPM-o-2.6.Source: §4.1 · p. 5

4.2. Main Results

  • At 15%/7.5% retention, OmniPack preserves 95.6% of original performance at 10.0% FLOPs; at 10%/5%, it preserves 92.9% at 6.8% FLOPs.On Qwen2.5-Omni-7B, OmniPack (w/o M) preserves 98.2% at 25% retention and 93.0% at 10% retention using 8.2% of original FLOPs.Source: §4.2 · p. 5, Table 1 · p. 6
  • On Qwen2.5-Omni-3B, OmniPack preserves 92.7% at 15%/7.5% retention using 9.0% of FLOPs; on MiniCPM-o-2.6, it reaches 100.8% at the same setting using 10.0% of FLOPs.The MiniCPM-o-2.6 result outperforms existing compression methods at that retention setting.Source: §4.2 · pp. 5–6, Table 2 · p. 7

4.3. Efficiency Analysis

  • At 15%/7.5% retention, OmniPack preserves 95.6% of performance, reduces FLOPs 10.0×, and achieves a 4.5× prefill speedup.At 25% retention, OmniPack (w/o M) preserves 98.2% of performance while reducing FLOPs by 4.9×.Source: §4.3 · p. 6, Table 3 · p. 7

4.4. Ablation Studies

  • Coverage selection led individual pre-LLM components on WorldSense and LVOmniBench, while combining all three components achieved the best performance across three benchmarks.Coverage improved WorldSense and LVOmniBench by approximately 3.4% and 6.1% over importance selection, and by 3.4% and 3.0% over token merging; adding importance to coverage improved WorldSense and VideoMME by approximately 3.0% and 1.9%.Source: Table 4 · p. 7, §4.4 · pp. 7–8
  • Pairing OmniPack's inner-LLM strategy with VisionZip-om, OmniSIFT◦, or SEATS† underperformed full OmniPack; replacing it with SEATS⋆ lowered average score and degraded three benchmarks.The results support the complementarity of the pre-LLM and inner-LLM strategies.Source: Table 7 · p. 8, §4.4 · p. 8
  • Layer 18 gave the best overall inner-LLM compression performance; audio-visual collaboration beat independent compression, and text-aware guidance outperformed question-agnostic guidance.Compression applied too early limits task-conditioned multimodal interaction, while applying it too late reduces computational benefit; text-aware guidance aggregates semantic cues from the full textual query.Source: Table 5 · p. 7, Table 6 · p. 7, Figure 3 · p. 8

5. Conclusion

OmniPack is a training-free compression framework that combines pre-LLM structural compression with post-interaction, task-conditioned semantic compression. Experiments across five audio-visual understanding benchmarks and three Omni-LLM backbones show the best performance–efficiency trade-off among existing methods across diverse token budgets.

  • Across five audio-visual understanding benchmarks and three Omni-LLM backbones, OmniPack achieves the best performance–efficiency trade-off among existing methods across diverse token budgets.Source: §5 · p. 8
  • OmniPack progressively uses modality-specific structural information before the LLM and task-conditioned semantic information after sufficient multimodal interaction within it.Source: §5 · p. 8
  • OmniPack is a training-free token compression framework designed for efficient Omni-LLMs.Source: §5 · p. 8

Supplementary Material · A. Benchmark Details

The evaluation uses five benchmarks spanning diverse real-world scenarios, temporal scales, modality dependencies, and reasoning requirements. Together, they contain 4,212 videos and 9,817 QA pairs.

A. Benchmark Details

  • 4,212 videos and 9,817 QA pairs across five benchmarks provide varied durations, modality dependencies, and reasoning requirements.The benchmarks differ substantially in their dependence on audio and visual evidence.Source: A. Benchmark Details · p. 11
  • AVUT [48] evaluates audio understanding and audio-visual alignment; following OmniZip [34], evaluation uses its human-annotated subset of 691 videos and 1,734 QA pairs.The subset’s videos average 69.1 seconds in duration.Source: A. Benchmark Details · p. 11
  • WorldSense [17] contains 1,662 synchronized audiovisual videos and 3,172 expert-annotated questions across 26 tasks, requiring joint audio-visual evidence integration.The videos span 8 domains and 67 fine-grained categories, averaging 140.7 seconds.Source: A. Benchmark Details · p. 11
  • DailyOmni [52] covers 684 daily-life videos and 1,197 questions across 6 tasks, testing temporally aligned audio-visual reasoning from event ordering to contextual reasoning.Videos average 43.2 seconds and contain rich auditory and visual events.Source: A. Benchmark Details · p. 11
  • VideoMME [12] includes 900 videos and 2,700 human-annotated questions, with short, medium, and long subsets and support for audio inputs.Durations range from 11 seconds to 1 hour, with a 1,017.9-second average video duration.Source: A. Benchmark Details · p. 11
  • LVOmniBench [35] contains 275 long-form videos and 1,014 questions, emphasizing long-range audio-visual correspondence, multi-event retention, and temporal localization.Videos range from 10 to 90 minutes and average 2,069.7 seconds.Source: A. Benchmark Details · p. 11

B. More Implementation Details · B.1. DPC-KNN Formulation

The implementation matches visual and audio budgets to each model’s native tokenization, while DPC-KNN selects coverage tokens from candidates left after importance selection. The formulation uses modality-specific positional distances and ranks representative, well-separated candidates.

B. More Implementation Details

  • Qwen2.5-Omni produces about 50 audio and 288 visual tokens per temporal window, while MiniCPM-o-2.6 produces about 750 audio tokens per 4,096 visual tokens.Source: B. More Implementation Details · p. 11
  • Across models, modality-specific retention ratios preserve comparable visual-audio balance under the same overall pre-LLM retention budget; for all models, each sampled frame has maximum pixel budget 128×28×28.Table 9 entries give retained visual and audio token percentages whose token-count-weighted average approximately matches R. Audio-intact keeps all audio tokens, whereas Both-selected compresses both modalities.Source: Table 9 · p. 11, B. More Implementation Details · p. 12

B.1. DPC-KNN Formulation

  • DPC-KNN selects coverage tokens from candidates remaining after the top Kimp tokens are chosen by importance.The coverage stage selects the top Km − Kimp candidates, then combines them with the importance-selected tokens.Source: §B.1 · p. 12
  • Audio distance uses temporal position, video distance uses spatiotemporal position, and positional distances are normalized to [0, 1] within each modality.Source: §B.1 · p. 12
  • The method sets k=7 and defines each candidate’s local density using its k nearest candidates, excluding itself.Source: §B.1 · p. 12
  • A high representative score indicates local representativeness and separation from higher-density regions.The formulation defines separation relative to higher-density candidates and ranks candidates by a representative score.Source: §B.1 · p. 12

C. Reproduction Details

This section defines a unified reproduction protocol and documents baseline-specific adaptations and constraints for comparing pre-LLM and in-LLM compression methods.

C. Reproduction Details

  • All methods use model-specific visual–audio allocations from Table 9, with audio-intact, both-selected, or visual-only budget handling as appropriate.Visual-only methods retain all audio tokens and reallocate the remaining budget to visual tokens.Source: C. Reproduction Details · p. 12
  • FastV, VisionZip [47], FastVID [28], and VidCom2 [24] are adapted to pooled visual tokens or specified attention-based selections for comparison.FastV uses K=2; VisionZip is applied after pooling with no contextual tokens; FastVID uses c=8, τ=0.84, p=4, and a 0.7 nonanchor-frame scaling; VidCom2 uses a 0.5 low-variance channel ratio and temperature 0.01. FastV-om is not reported at R=10% for Qwen2.5-Omni-7B or MiniCPM-o-2.6 because target visual retention falls below the minimum imposed by the first two unpruned blocks.Source: C. Reproduction Details · p. 12
  • OmniZip [34] uses audio attention and contextual representatives to guide group-wise video discarding, with DPC-KNN on alternating frames and similarity selection on others.Its contextual ratio is 0.05 with g=3; the 75% maximum video discard ratio prevents Rv below 25%, so at R=25% both Rv and Ra are set to 25%.Source: C. Reproduction Details · p. 12
  • OmniSIFT [7] is evaluated without alignment training, while SEATS [42] is compared in original and late-layer-removal-disabled variants.Both SEATS variants match the proposed method’s final modality-specific retention ratios and are evaluated only on the two 28-layer models because the official layer schedule has no 36-layer adaptation.Source: C. Reproduction Details · p. 13
  • All methods are reproduced under a unified evaluation setting using NVIDIA H20 GPUs.Source: C. Reproduction Details · p. 13

D. Computing Cost Evaluation

The section estimates token-compression efficiency by counting vision- and audio-token FLOPs during LLM prefilling and decoding, with layer-specific sequence lengths accounted for when compression is progressive.

D. Computing Cost Evaluation

  • Prefilling FLOPs per layer are approximated as 4nd^2 + 2n^2d + 2ndm, and decoding cost assumes R=100 generated tokens.Here n is the total number of vision and audio tokens; one multiply–accumulate operation counts as one FLOP.Source: D. Computing Cost Evaluation · p. 13
  • For methods that change sequence length across layers, the cost is computed per layer using its effective multimodal sequence length and summed across layers.Pre-LLM methods use the compressed sequence throughout; FastV variants retain the full sequence for two layers, while SEATS variants progressively compress or discard tokens and OmniPack compresses at layer 18 or 26 depending on backbone.Source: D. Computing Cost Evaluation · p. 13
  • The FLOPs accounting excludes vision and audio encoders, modality projectors, textual tokens, token-selection operations, and the language-model head.FastVID’s nonkey-frame tokens are additionally reduced by a factor of 0.7 in the accounting.Source: D. Computing Cost Evaluation · p. 13

E. More Experimental Details

OmniPack’s supplementary experiments report strong efficiency–performance trade-offs across backbones and benchmarks, explain how its pre-LLM and inner-LLM compression complement each other, and show robustness to parameter variation.

Fine-Grained Results

  • At 25%/12.5% retention, OmniPack performs particularly well on AVUT audio-event localization and character matching, and DailyOmni audio-visual alignment, inference, and context understanding.Consistent performance across WorldSense categories indicates that the benefit generalizes across video domains.Source: E. More Experimental Details · pp. 13–14, Table 11 · p. 15, Table 12 · p. 15

Different Backbones

  • At 25%, 20%, and 10% pre-LLM retention, Qwen2.5-Omni-3B retains 95.0%, 93.8%, and 89.5% performance using 15.5%, 12.2%, and 6.1% of original FLOPs.On MiniCPM-o-2.6, OmniPack achieves 103.8% and 103.4% of original performance at 25% and 20% retention, and 96.4% at 10% retention with 6.8% of original FLOPs.Source: E. More Experimental Details · p. 13, Table 10 · p. 14

Fine-Grained Results

  • At 15%/7.5% retention, OmniPack performs best on short- and long-duration VideoMME videos and remains competitive on medium-duration videos.It retains 107.4% of original long-video performance; the passage attributes this to removing redundant audio-visual tokens from long sequences.Source: E. More Experimental Details · p. 14, Table 14 · p. 16

Compression Strategies Collaboration

  • Pre-LLM compression removes modality-specific redundancy, while inner-LLM compression refines tokens using task-conditioned representations after audio-visual interaction.Audio contributions vary more across temporal groups because acoustic events are sparse, while video contributions remain stable with continuous visual content.Source: E. More Experimental Details · pp. 14–15

Case Study

  • At the same 25% pre-LLM retention, OmniPack preserves task-relevant evidence and removes redundancy where SEATS⋆ can discard critical evidence and retain redundant tokens.The comparison reports an incorrect SEATS⋆ prediction and a correct OmniPack prediction under aggressive compression.Source: E. More Experimental Details · pp. 15–16, Figure 5 · p. 16

Parameter Ablation

  • The tested parameter configurations all outperform existing methods; varying λ changes performance by at most 1.0 point, (τv, τa) by 0.5, and ζ by 0.2.The analysis finds (ηv, ηa)=(0.25,0.35) favorable and adopts λ=0.20, (τv, τa)=(0.10,0.05), and ζ=0.10 as a balanced configuration.Source: E. More Experimental Details · p. 16, Figure 6 · p. 16, Table 15 · p. 16
Loading 2608.03812v1…