PaperlayerView exact revision on arXiv

Source-linked AI summary

OmniZip: Audio-Guided Dynamic Token Compression for Fast Omnimodal Large Language Models

Keda Tao, Kele Shao, Bohan Yu, Weiqiang Wang, Jian liu, Huan Wang

arXiv:2511.14582v2cs.CV

TL;DR

OmniLLM inference is bottlenecked by long audio-video token sequences, while existing compression methods do not jointly address both modalities. OmniZip provides training-free audio-guided multimodal compression, achieving faster, lower-memory inference while maintaining accuracy, with limitations for streaming and task-dependent audio-video compression balance.

  • Problem

    OmniLLM inference is constrained by the computational and memory costs of large audio-video token sequences, while joint multimodal token compression remains underexplored.

  • Method

    OmniZip identifies salient audio tokens, computes per-window audio-retention scores to guide video pruning, and applies cross-modal audio merging with interleaved spatio-temporal video compression.

  • Results

    OmniZip consistently surpasses prior single-modality compression methods while maintaining accuracy and improving inference speed and memory efficiency across audio-video understanding tasks.

  • Takeaways & Limitations

    Training-free joint audio-video token compression can accelerate OmniLLMs while preserving their audio-video understanding performance.

  • Takeaways & Limitations

    The optimal audio-video compression balance varies across tasks, and the method primarily targets offline inference rather than online or arbitrary-length streaming input.

Abstract

from arXiv · show

Omnimodal large language models (OmniLLMs) have attracted increasing research attention of late towards unified audio-video understanding. However, the high computational cost of processing longer joint audio-video token sequences has become a key bottleneck. Existing token compression methods have not addressed the emerging need to jointly compress multimodal tokens. To bridge this gap, we present OmniZip, a training-free, audio-guided audio-visual token-compression framework that optimizes multimodal token representation and accelerates model inference. Specifically, OmniZip first identifies salient audio tokens, then computes an audio retention score for each time group to capture information density, thereby dynamically guiding video token pruning and preserving cues from audio anchors enhanced by cross-modal similarity. For each time window, OmniZip compresses the video tokens using an interleaved spatio-temporal scheme. Extensive results demonstrate the merits of OmniZip: it achieves a 3.42X inference speedup and a 1.4X memory reduction over other top-performing counterparts, while maintaining the performance of OmniLLMs without training.

1. Introduction

OmniZip addresses the computational and memory burden of long audio-video token sequences by jointly compressing multimodal tokens. Its audio-guided pruning preserves performance while accelerating inference and reducing memory use.

  • Audio-token attention analysis motivates compression because audio tokens receive consistently higher attention while many video tokens receive little attention.Attention concentrates within time windows and declines across windows, indicating short-range contextual structure.
  • Joint audio-video compression is needed because existing approaches largely reduce visual tokens, while audio tokens inflate sequence length and audio-video streams differ in temporal scale and sparsity.Their redundancy and complementarity make multimodal pruning sensitive and challenging.
  • OmniZip identifies dominant audio tokens and computes per-window retention rates interpreted as information density and event-boundary cues.High-retention windows receive less video pruning, whereas sparse windows receive more pruning.
  • The method combines uniformly sampled audio anchors and cross-modal merging with interleaved spatio-temporal video compression to suppress redundancy without excessive reduction along one dimension.This design targets both temporal redundancy between frames and spatial redundancy within frames.
  • 2.51× to 3.42× inference speedup, 10G lower GPU memory, and highest accuracy are achieved by OmniZip on Qwen2.5-Omni-7B while remaining training-free.The method significantly outperforms single-modality token compression methods on audio-video understanding tasks.
  • OmniZip jointly compresses audio-video tokens through audio-guided pruning and a video token compression module while preserving cross-modal semantic and temporal alignment.The framework is presented as a training-free approach for accelerating OmniLLM inference.

2. Related Work

OmniLLMs extend video understanding with audio but introduce substantial multimodal-token efficiency challenges. Token compression offers a tuning-free way to reduce redundancy, yet prior work has largely treated modalities separately.

  • OmniLLMs incorporate audio alongside visual data for more comprehensive video understanding, but their large multimodal token sequences impede practical deployment.Balancing model performance and computational efficiency remains a central challenge.
  • Token compression is a tuning-free post-processing approach that targets redundancy in image, video, and audio inputs to improve multimodal inference efficiency.Existing methods establish token-importance metrics before reducing tokens.

3. Proposed Method

OmniZip compresses omnimodal inputs window-by-window by selecting salient audio tokens, using audio retention to guide video pruning, and applying interleaved spatio-temporal compression. Its design targets redundant multimodal tokens while preserving cross-modal and temporal information, without attention-matrix access or training.

  • Overall Architecture: OmniZip processes audio-video tokens in fixed-length time windows and compresses them before the LLM, reducing downstream token count and inference overhead.The method preserves the chronological organization of co-temporal multimodal blocks while pruning within each window.
  • Audio Token Selection: Audio tokens are ranked by mean attention from the audio encoder, pooled for alignment, and the highest-scoring ρa% are retained as representatives.The importance scores are derived from the last audio-encoder layer and used to select salient audio features.
  • Audio Anchor Consolidation: Uniformly sampled audio anchors merge top-G non-salient tokens chosen by audio-video similarity, while remaining non-salient audio tokens are discarded.This consolidation preserves context coverage and multimodal consistency through cross-modal similarity.
  • Audio-Guided Video Compression: Per-window audio retention scores treat high-retention windows as information-dense, conservatively pruning their video tokens while pruning low-retention windows more aggressively.The resulting video rates are normalized to satisfy the global pruning budget, with video pruning remaining more time-adaptive than audio pruning.
  • ISTC Block: ISTC uses adjacent-frame cosine similarity to remove temporally redundant tokens and density-based clustering to retain salient tokens while pruning spatial redundancy.Compression is performed independently within each time window, with temporal and spatial strategies interleaved across frames.
  • Design Remarks: OmniZip avoids accessing LLM attention-score matrices, remains compatible with FlashAttention and multi-round dialogue, and takes less than 40 ms for pruning.The design also avoids the GPU memory overflow associated with extracting attention matrices in some ViT-based visual encoders.

4. Experimental Results

Across audio-video benchmarks, OmniZip preserves accuracy while reducing token-related inference cost through dynamic, audio-guided compression. Its efficiency gains are supported by adaptive pruning, audio-anchor consolidation, and interleaved video token selection.

  • Dynamic Pruning: Low-audio-retention windows receive higher video pruning ratios, while the overall pruning rate remains constant across time windows.This allocation shifts pruning toward information-sparse windows without changing the total pruning budget.
  • Main Results: 99.1% average accuracy remains at a 60% computational-FLOPs reduction, while OmniZip maintains optimal performance with the fewest tokens across diverse benchmarks.The comparison uses normalized accuracy across multiple benchmarks and contrasts OmniZip with random pruning and FastV.
  • Sensitivity Analysis: Excessive pruning of either audio or video significantly degrades performance, indicating that compression requires a suitable balance between modalities.The sensitivity analysis motivates dynamically adapting pruning rates to the task.
  • Efficiency Analyses: 3.42× prefilling speedup and 10G lower memory consumption are achieved on the 7B model while maintaining approximately 97% accuracy on WorldSense.The efficiency comparison reports results across four benchmarks and identifies larger gains for the 7B model.
  • Ablation Study: Removing dynamic video pruning lowers accuracy, and additionally removing audio anchor consolidation causes further degradation.The ablation evaluates dynamic pruning and audio anchor consolidation as core OmniZip components.
  • Token Selection: Interleaved spatio-temporal token selection outperforms DyCoke for video tokens, while global selection is suboptimal for the omnimodal setting.The comparison attributes the design rationale to the need to preserve semantic alignment across audio and video.

5. Conclusion

OmniZip is presented as a training-free, audio-guided method for jointly compressing audio-video tokens in OmniLLMs. Across audio-video understanding tasks and 3B and 7B models, it surpasses prior single-modal methods while reducing memory and prefilling time with nearly unchanged performance.

  • Conclusion: OmniZip dynamically reduces audio-video tokens using audio guidance and spatio-temporal video compression without training.The method identifies salient audio tokens, uses per-window audio retention to guide video pruning, and jointly compresses multimodal tokens.
  • Conclusion: OmniZip consistently surpasses prior single-modal methods across audio-video understanding tasks on 3B and 7B OmniLLMs.The conclusion summarizes benchmark and analysis results across two model scales.
  • Conclusion: Up to 10G memory reduction and 2.7–3.8× prefilling speedup are achieved while maintaining nearly identical performance.These results summarize the reported efficiency gains and accuracy preservation.

A. Dynamic Pruning Rate Allocation Algorithm

The dynamic pruning algorithm converts per-window audio retention ratios into adaptive video pruning rates. It then normalizes those rates so their total matches the global pruning budget.

  • Algorithm Inputs and Outputs: The algorithm takes the audio-retention ratio for each time window as input and returns a vector of dynamic video-pruning rates.The procedure defines minimum and maximum pruning parameters alongside the target video pruning ratio.
  • Initial Allocation: Initial video-pruning ratios are computed separately for each time window from the audio-retention signal.Algorithm 1 iterates over all N time windows before global normalization.
  • Budget Normalization: The initial ratios are normalized to match the global budget T_budget = ρ_v × N before the final rates are returned.Normalization preserves the adaptive allocation while enforcing the specified overall pruning rate.

B.1. Adaptivity of OmniZip

OmniZip’s adaptivity is motivated by local temporal structure in audio-visual streams, but its effectiveness may depend on how future OmniLLMs organize tokens. The method uses temporal locality and modality-specific information density to guide compression.

  • B.1. Adaptivity of OmniZip: OmniZip exploits temporal locality because short audio-visual segments contain correlated, synchronized, and redundant information.The authors argue this principle remains feasible beyond explicit time-window concatenation.
  • B.1. Adaptivity of OmniZip: Explicit time-window concatenation is a potential scope boundary for OmniZip as OmniLLM architectures evolve.The paper asks whether efficacy could decline in models that no longer use this organization.
  • B.1. Adaptivity of OmniZip: Audio and video differ in task-specific contribution and compressibility, complicating a single compression balance across OmniLLM tasks.The paper contrasts high visual reduction rates with the greater difficulty of compressing audio-visual inputs.

C. Computing Cost Evaluation

The computing-cost analysis models transformer FLOPs for multimodal token sequences and situates OmniLLMs within broader video-language and token-compression research. It frames token count as a central efficiency variable for audio-visual inference.

  • C. Computing Cost Evaluation: Transformer prefilling cost is approximated as 4nd^2+2n^2d+2ndm, with n tokens, d hidden dimension, and m FFN dimension.The analysis considers MHA and FFN computation in OmniLLMs.
  • C. Computing Cost Evaluation: FastV failed on the 7B model because of an A6000 out-of-memory error, so its evaluation used a single H100 GPU.The table identifies ρa and ρv as audio-token and video-token pruning ratios.
  • C. Computing Cost Evaluation: OmniLLMs jointly process audio and visual information, increasing the multimodal token burden relative to video-language systems.This section places audio-video processing within the broader development of VideoLLMs and OmniLLMs.
  • C. Computing Cost Evaluation: Token compression is a tuning-free post-processing strategy for reducing redundancy in multimodal inputs.Prior work applies compression to image, video, and audio tokens, motivating efficiency analysis for joint audio-video inputs.

E. More Experimental Results

Supplementary experiments examine pruning-ratio robustness, dynamic allocation, and the audio-merging hyperparameter G. Results indicate that OmniZip remains competitive across settings while G should reflect the relative role of audio.

  • E. More Experimental Results: OmniZip achieves better performance than other methods under a 50% overall compression rate with balanced audio and video pruning.The supplementary comparison tests whether its advantage depends on aggressively pruning audio relative to video.
  • E. More Experimental Results: Low-audio-retention windows receive higher video pruning ratios while the total pruning rate remains constant.This visualization illustrates dynamic allocation based on audio retention.
  • E. More Experimental Results: At 45% retention, G controls how many tokens each audio anchor merges in the ablation on Qwen2.5-Omni-7B.The experiment evaluates accuracy as G varies.
  • E. More Experimental Results: Dynamic pruning-ratio allocation receives additional visualization evidence in the supplementary experiments.The supplied passage identifies Fig. 7 as the source of these visualizations.
  • E. More Experimental Results: Higher G is appropriate for audio-centric AVUT, whereas G = 3 performs best when audio is balanced with or supplementary to video.Larger G values introduce noise and slightly degrade performance in those other benchmarks.

F. Limitations and Future Work

The paper identifies unresolved limitations in choosing modality-specific compression and supporting deployment settings. Future work targets streaming inference and combining compression with other efficiency techniques.

  • F. Limitations and Future Work: Optimal audio-video compression balance remains difficult because modality importance varies across tasks and contexts.The authors identify this as a central limitation of the current approach.
  • F. Limitations and Future Work: OmniZip is designed primarily for offline inference and does not natively support online or arbitrary-length streaming audio-visual input.The authors propose developing a streaming video-inference framework that incorporates audio.
  • F. Limitations and Future Work: Larger models’ substantial parameter counts continue to impede practical deployment.The paper suggests combining token compression with quantization and pruning as a future direction.
Loading 2511.14582v2…