PaperlayerView exact revision on arXiv

Source-linked AI summary

OmniPack: Unified Token Compression for Efficient Omni-modal Large Language Models

Wanshun Su, Yang Shi, Feihu Liu, Ziwen Yu, Yan Min, Zhuoran Zhang, Qixun Wang, Haotian Wang, Shixuan Liu, Yuanxing Zhang, Peng Wu, Chengfu Huo, Liang Ding

arXiv:2608.03812v1cs.CV

TL;DR

Omni-LLMs require aggressive compression because long, redundant audio-visual token sequences impose substantial inference costs while existing methods can miss globally distributed or query-relevant evidence. OmniPack combines structural pre-LLM compression with task-conditioned semantic refinement within the LLM, achieving the best performance–efficiency trade-off across five benchmarks and three backbones.

  • Problem

    Long, redundant audio-visual token sequences create substantial computation and memory overhead, while aggressive compression can overlook globally distributed task-relevant evidence.

  • Method

    OmniPack is a training-free two-stage framework combining modality-specific structural compression before the LLM with query-conditioned semantic refinement within it.

  • Results

    At a 7.5% final token retention ratio, OmniPack reduces FLOPs by 10.0× and achieves a 4.5× prefill speedup while preserving 95.6% of original performance across five benchmarks and three backbones.

  • Takeaways & Limitations

    OmniPack provides a strong performance–efficiency trade-off for efficient Omni-LLM inference across diverse token budgets.

  • Takeaways & Limitations

    The compression framework leaves all text tokens unchanged throughout compression.

Abstract

from arXiv · show

Omni-modal large language models (Omni-LLMs) have achieved remarkable performance on audio-visual understanding tasks, but processing long and highly redundant visual and audio token sequences incurs substantial computational overhead, demanding aggressive token compression for efficient deployment. Existing methods often degrade at low token budgets: pre-LLM compression may discard structurally important and globally distributed evidence, whereas inner-LLM compression often underexploits query-conditioned audio-visual collaboration. To address these limitations, we propose OmniPack, a training-free framework that coordinates structural compression before the LLM with task-relevant semantic refinement within the LLM. Before the LLM, OmniPack removes structural redundancy through modality-specific importance, global coverage, and similarity-aware merging. After sufficient multimodal interaction, it further consolidates diverse, task-relevant representations through textual guidance and audio-visual collaboration. Extensive experiments on five benchmarks with three Omni-LLM backbones demonstrate that OmniPack consistently achieves the best performance-efficiency trade-off across diverse retention ratios, outperforming all existing methods. Notably, on Qwen2.5-Omni-7B, OmniPack preserves 98.0% of the original performance while reducing FLOPs to 16.7%, and still retains 92.9% of the original performance with only 6.8% of the original FLOPs.

1. Introduction

OmniPack addresses the computational burden of long audio-visual sequences with a training-free, two-stage compression framework that combines structural pre-LLM reduction with query-conditioned inner-LLM refinement. Across three Omni-LLM backbones and five benchmarks, it achieves strong performance–efficiency trade-offs under aggressive token budgets.

  • Motivation: Aggressive compression is needed because densely tokenized visual and audio inputs create long sequences with substantial inference computation and memory overhead.The challenge is removing redundancy without losing task-relevant multimodal evidence.
  • Limitations: Existing methods inadequately model long-range audio-visual structure and insufficiently exploit query-conditioned cross-modal evidence under aggressive token budgets.Relevant evidence may be sparsely distributed across distant temporal segments, abrupt visual changes, and transient acoustic events.
  • OmniPack: OmniPack is a training-free two-stage framework that progressively compresses tokens before and within the LLM.Pre-LLM compression uses modality-specific spatiotemporal structure to remove redundancy while preserving important evidence and global coverage; inner-LLM compression refines retained representations using textual guidance and audio-visual collaboration.
  • Results: At a final token retention ratio of 7.5%, OmniPack reduces FLOPs by 10.0×, achieves a 4.5× prefill speedup, and preserves 95.6% of original performance.These results are reported across three Omni-LLM backbones and five benchmarks.
  • Results: OmniPack consistently offers the best performance–efficiency trade-off among existing token compression methods.Figure 1 compares performance on Qwen2.5-Omni-7B across five benchmarks and shows OmniPack achieving the best overall performance as token retention decreases from 15% to 7.5%.

2. Related Work

Related work traces Omni-LLMs’ expansion beyond vision-centric models and the development of token compression across individual and joint audio-visual inputs. Existing approaches still struggle under aggressive compression because they inadequately preserve modality-specific structure or exploit task-relevant audio-visual semantics.

  • Omni-LLMs: Omni-LLMs extend visual-language models to unified processing of text, images, videos, and audio.This broader capability supports audio-visual understanding, real-time conversational assistants, and multimodal agents.
  • Token Compression: Token compression has been studied for image, video, and audio inputs to reduce redundancy and improve inference efficiency.Recent work extends compression to joint audio-visual inputs as Omni-LLMs develop.
  • Audio-Visual Compression: OmniZip uses audio-guided video token compression, while OmniSIFT combines spatio-temporal video compression with vision-guided audio selection.These methods exemplify cross-modal compression for joint audio-visual inputs.
  • Limitations and Motivation: Existing methods struggle under aggressive compression because they fail to jointly preserve modality-specific structure and exploit task-relevant semantics.The paper motivates coordinating structure-aware pre-LLM compression with query-conditioned audio-visual compression inside the LLM.

3. Method

OmniPack coordinates modality-specific structural compression before the LLM with query-conditioned audio-visual semantic compression inside it. The framework preserves salient evidence, broad coverage, and representative information while keeping all text tokens unchanged.

  • Overall Framework: OmniPack combines pre-LLM structural compression with inner-LLM query-conditioned compression using textual relevance, audio-visual collaboration, and within-modality representativeness.Pre-LLM compression operates independently on each modality before sufficient cross-modal semantics emerge; inner-LLM compression follows the first ℓ Transformer blocks.
  • Pre-LLM Compression: Pre-LLM importance selection combines encoder attention with modality-specific structural variation to identify salient visual and audio tokens.Video cues capture inter-frame changes and spatial distinctiveness, while audio variation indicates stronger acoustic boundaries or transient events.
  • Pre-LLM Compression: Coverage selection supplements salient tokens with representatives chosen using joint feature and positional distances, maintaining broad spatiotemporal coverage.DPC-KNN identifies candidates representative of local neighborhoods and separated from other representative regions; λ controls positional-distance contribution.
  • Pre-LLM Compression: Similarity-aware merging transfers unselected-token information to retained representatives using feature similarity, positional proximity, and representative importance.Importance-weighted aggregation enables representatives to absorb nearby redundant information while mitigating information loss.
  • Inner-LLM Compression: Inner-LLM compression scores tokens by textual relevance, audio-visual collaboration, and within-modality representativeness, then selects diverse tokens and merges the rest.Unselected hidden states are assigned to their most cosine-similar retained representatives through relevance-modulated similarity-weighted averaging; text tokens remain uncompressed.

4. Experiment

OmniPack is evaluated across five audio-visual benchmarks, three Omni-LLM backbones, and multiple retention settings, achieving strong performance-efficiency trade-offs. Ablations show that its pre-LLM components and inner-LLM multimodal compression are complementary and work best together.

  • Benchmarks: OmniPack is evaluated on five benchmarks spanning audio- and video-centric understanding, short- and long-form videos, and perception-to-reasoning tasks.The benchmarks are AVUT, WorldSense, DailyOmni, VideoMME, and LVOmniBench.
  • State-of-the-Art Performance: At 25% retention, OmniPack (w/o M) preserves 98.2% of original performance; at 15%, it retains 95.2%, exceeding VisionZip-om’s 91.4% and OmniSIFT◦’s 89.7%.Under 10% retention, OmniPack (w/o M) still preserves 93.0% of original performance.
  • Different Backbones: On Qwen2.5-Omni-3B, OmniPack preserves 92.7% of original performance at 15%/7.5% retention using 9.0% of original FLOPs.On MiniCPM-o-2.6, it achieves 100.8% of original performance under the same retention setting while using 10.0% of original FLOPs.
  • Efficiency Analysis: At 15%/7.5% retention, OmniPack preserves 95.6% of original performance, reduces FLOPs by 10.0×, and achieves a 4.5× prefill speedup.This outperforms all pre-LLM compression methods evaluated at 25% retention despite retaining substantially fewer tokens.
  • Pre-LLM Compression Strategy: Coverage selection outperforms individual pre-LLM alternatives on WorldSense and LVOmniBench, while combining importance and coverage adds approximately 3.0% and 1.9% on WorldSense and VideoMME.Adding all three components yields the best performance across the three benchmarks, including a further 4.8% gain on LVOmniBench.
  • Inner-LLM Compression Strategy: Layer 18 provides the best overall inner-LLM performance, audio-visual collaboration outperforms independent compression, and combining both compression stages outperforms alternative pairings.Replacing OmniPack’s inner-LLM compression with SEATS⋆ lowers the average score and degrades three of five benchmarks.

5. Conclusion

OmniPack is a training-free compression framework that progressively combines modality-specific structural compression before the LLM with task-conditioned semantic compression after multimodal interaction. This design supports more aggressive compression while preserving downstream performance.

  • OmniPack is a training-free token compression framework for efficient Omni-LLMs.
  • Before the LLM, OmniPack exploits modality-specific structural information; within the LLM, it uses task-conditioned semantic information after sufficient multimodal interaction.
  • OmniPack’s progressive design enables substantially more aggressive compression while effectively preserving downstream performance.

OmniPack: Unified Token Compression for Efficient

This section introduces Omni-modal Large Language Models.

  • The section focuses on Omni-modal Large Language Models.

A. Benchmark Details

The evaluation uses five benchmarks totaling 4,212 videos and 9,817 QA pairs, spanning diverse scenarios, temporal scales, modalities, and reasoning requirements. AVUT and VideoMME provide complementary tests of audio-visual reasoning and token compression across varied video durations and modality configurations.

  • AVUT: 691 videos and 1,734 QA pairs form the human-annotated AVUT subset used for evaluation, following OmniZip’s evaluation setting.The full AVUT benchmark contains 2,662 videos and 11,609 QA pairs.
  • Benchmark Coverage: 4,212 videos and 9,817 QA pairs comprise the five evaluation benchmarks, covering diverse real-world scenarios and temporal scales.The benchmarks differ in their reliance on audio and visual evidence, supporting evaluation across modalities and reasoning requirements.
  • AVUT: AVUT evaluates audio-content understanding, audio-event localization, audio-visual alignment, temporal ordering, and cross-modal contextual reasoning.Its videos contain rich auditory and visual events and average 43.2 seconds in duration.
  • VideoMME: 900 videos and 2,700 human-annotated multiple-choice QA pairs comprise VideoMME across six major domains, with durations from 11 seconds to 1 hour.Its short-, medium-, and long-duration subsets and support for audio inputs enable evaluation under varied duration and modality configurations.

B. More Implementation Details · B.1. DPC-KNN Formulation · C. Reproduction Details

The paper specifies model-aware multimodal retention budgets, a DPC-KNN coverage-selection formulation, and a unified reproduction protocol for comparing compression methods. These details align token allocation with native tokenization and standardize evaluation across models and methods.

  • B. More Implementation Details: Approximately 50 audio tokens and 288 visual tokens per temporal window guide Qwen2.5-Omni’s modality-specific budget allocation, while MiniCPM-o-2.6 uses a different native tokenization ratio.The allocations preserve comparable audio–visual balance under the same overall pre-LLM retention budget.
  • B. More Implementation Details: 128×28×28 is the maximum pixel budget per sampled frame for all models.
  • B.1. DPC-KNN Formulation: DPC-KNN coverage selection operates on candidates remaining after the top Kimp tokens are chosen by importance selection.The method uses a joint distance for coverage selection and combines the selected coverage tokens with importance-selected tokens.
  • B.1. DPC-KNN Formulation: The joint distance incorporates temporal distance for audio and spatiotemporal distance for video, with positional distances normalized independently to [0, 1] within each modality.
  • B.1. DPC-KNN Formulation: k=7 is used in all experiments when defining each candidate’s local density.Representative scores favor tokens that represent local neighborhoods and remain separated from other high-density regions.
  • C. Reproduction Details: FastV, VisionZip, FastVID, and VidCom2 use Audio-intact evaluation, whereas FastV-om, VisionZip-om, and OmniZip use Both-selected evaluation.Visual-only methods retain Ra = 100% and reallocate the remaining token budget to visual tokens for each example.
  • C. Reproduction Details: K=2 is used for FastV, which prunes visual tokens after the K-th LLM block using attention scores from the last text token.FastV-om applies the same attention-based selection to both visual and audio tokens.
  • C. Reproduction Details: All methods are reproduced under a unified evaluation setting and evaluated using NVIDIA H20 GPUs.OmniSIFT is additionally evaluated without its alignment training for a fair comparison with training-free methods.

D. Computing Cost Evaluation

The paper evaluates token-compression efficiency using FLOPs from vision and audio tokens during prefilling and decoding. Costs are accumulated layer by layer when multimodal sequence lengths change across the transformer.

  • Cost formulation: FLOPs from vision and audio tokens quantify computational efficiency during both prefilling and decoding.The analysis models each transformer layer as multi-head attention plus a feed-forward network, with total multimodal tokens n = n(v) + n(a).
  • Layerwise accounting: For changing sequence lengths, Eq. (18) is applied at each layer using effective length n_i, then summed across layers.Pre-LLM methods use their compressed sequence throughout the transformer, while other methods may retain or progressively reduce tokens later.
  • Layerwise accounting: FastV and FastV-om retain the full multimodal sequence for the first two layers before compressing it in subsequent layers.SEATS⋆ progressively reduces tokens after layers 16, 18, and 20, while SEATS† applies additional discarding beyond those reductions.

E. More Experimental Details

OmniPack maintains strong performance-efficiency trade-offs across additional backbones and retention ratios. Fine-grained results further show particular strengths on tasks requiring audio-visual evidence association.

  • Different Backbones: 95.0%, 93.8%, and 89.5% of original performance are retained on Qwen2.5-Omni-3B at 25%, 20%, and 10% retention, using 15.5%, 12.2%, and 6.1% FLOPs.On MiniCPM-o-2.6, OmniPack achieves 103.8% and 103.4% performance at 25% and 20% retention, and 96.4% at 10%.
  • Fine-Grained Results: At 25%/12.5% retention, OmniPack performs particularly well on AVUT audio-event localization and character matching, and DailyOmni event alignment, inference, and context understanding.These tasks require associating acoustic evidence with relevant visual content.

FLOPs

OmniPack combines structural pre-LLM compression with delayed, task-conditioned inner-LLM refinement to preserve complementary audio-visual evidence under aggressive compression. This complementary strategy supports strong performance across video durations and enables correct predictions despite reduced tokens.

  • VideoMME: 107.4% of the original long-video performance is retained on VideoMME, showing particular benefit from removing redundant audio-visual tokens in long sequences.OmniPack performs best on short- and long-duration videos and remains competitive on medium-duration videos at the 15%/7.5% retention setting.
  • Compression Strategies Collaboration: OmniPack combines modality-specific redundancy removal before the LLM with task-conditioned token refinement after audio-visual interaction.Pre-LLM compression targets structural redundancy, while inner-LLM compression exploits semantic representations formed through multimodal interaction.
  • Case Study: At the same 25% pre-LLM retention ratio, OmniPack preserves task-relevant evidence and produces the correct prediction where SEATS⋆ fails.Its structure-aware selection, delayed query-conditioned audio-visual compression, and redundancy removal yield a more compact and informative representation.
Loading 2608.03812v1…