Source-linked AI summary
OmniSIFT: Modality-Asymmetric Token Compression for Efficient Omni-modal Large Language Models
Yue Ding, Yiyan Ji, Jungang Li, Xuyang Liu, Xinlong Chen, Junfei Wu, Bozhou Li, Bohan Zeng, Yang Shi, Yushuo Guan, Yuanxing Zhang, Jiaheng Liu, Qiang Liu, Pengfei Wan, Liang Wang
TL;DR
Long multimodal token sequences make Omni-LLM inference computationally expensive, while compression methods tailored to audio-video reasoning remain limited. OmniSIFT uses modality-asymmetric compression by pruning spatio-temporal video redundancy before selecting audio tokens with visual guidance. Across five benchmarks, it remains effective with aggressive compression, including 25% token retention, while adding few parameters and reducing latency against training-free baselines.
Problem
Long interleaved audio-video token sequences create substantial computational overhead, while token compression methods for Omni-LLMs remain limited.
Method
OmniSIFT first prunes spatial and temporal redundancy from video tokens, then uses the resulting visual anchors to select informative audio tokens.
Results
Across five benchmarks, OmniSIFT consistently outperforms compression baselines with 25% of the original tokens retained and surpasses full-token models on several settings.
Takeaways & Limitations
OmniSIFT provides an effective and efficient approach for reducing Omni-LLM token counts while preserving key audio-visual information for downstream tasks.
Takeaways & Limitations
Precise audio-visual alignment remains challenging, while STVP models temporal redundancy only within each multimodal chunk and uses a fixed, query-agnostic token budget.
Abstract
from arXiv · showhide
Omni-modal Large Language Models (Omni-LLMs) have demonstrated strong capabilities in audio-video understanding tasks. However, their reliance on long multimodal token sequences leads to substantial computational overhead. Despite this challenge, token compression methods designed for Omni-LLMs remain limited. To bridge this gap, we propose OmniSIFT (Omni-modal Spatio-temporal Informed Fine-grained Token compression), a modality-asymmetric token compression framework tailored for Omni-LLMs. Specifically, OmniSIFT adopts a two-stage compression strategy: (i) a spatio-temporal video pruning module that removes video redundancy arising from both intra-frame structure and inter-frame overlap, and (ii) a vision-guided audio selection module that filters audio tokens. The entire framework is optimized end-to-end via a differentiable straight-through estimator. Extensive experiments on five representative benchmarks demonstrate the efficacy and robustness of OmniSIFT. Notably, for Qwen2.5-Omni-7B, OmniSIFT introduces only 4.85M parameters while maintaining lower latency than training-free baselines such as OmniZip. With merely 25% of the original token context, OmniSIFT consistently outperforms all compression baselines and even surpasses the performance of the full-token model on several tasks.
1. Introduction
Omni-LLMs face rapidly growing computational costs from long interleaved audio-video token sequences, while existing compression paradigms inadequately model cross-modal dependencies. OmniSIFT addresses this gap with vision-guided, modality-asymmetric compression and reports strong efficiency and accuracy under aggressive token reduction.
- Motivation: A typical 20-second multimodal clip can yield more than 20K tokens, substantially increasing computational cost for joint audio-video reasoning.Video uses densely sampled frames, while audio requires high temporal resolution to capture acoustic dynamics.
- Compression paradigms: Existing approaches include modality-decoupled compression, modality-symmetric compression, and modality-asymmetric compression.The latter first prunes visual redundancy and then performs visually guided audio compression.
- Design rationale: Audio saliency is often context dependent, whereas video redundancy can be estimated from the visual stream, motivating vision-guided rather than symmetric compression.Visual scenes can provide semantic anchors such as visible speakers or visually grounded events.
- Results: 25% of the original tokens retained: OmniSIFT consistently outperforms all compression baselines and surpasses the full-token model on several settings.The result is reported across five representative benchmarks.
- Results: 4.85M additional parameters: OmniSIFT achieves lower latency than training-free baselines such as OmniZip on Qwen2.5-Omni-7B.This demonstrates an efficiency advantage under the reported model and comparison.
- OmniSIFT: OmniSIFT first removes spatial and temporal redundancy in video tokens, then uses the resulting visual anchors to select informative audio tokens.This implements the paper’s modality-asymmetric compression framework.
2. Related Works
Omni-LLMs extend unified autoregressive language models to heterogeneous audio, video, and language inputs, while prior compression work mainly targets visual redundancy. Related approaches include saliency-based, training-free, variation-aware, hierarchical, and curvature-aware video-token reduction, alongside multimodal compression paradigms.
- Omni-modal Large Language Models: Omni-LLMs incorporate audio signals alongside visual and textual inputs within a unified autoregressive framework.This distinguishes them from conventional video-language models focused primarily on visual sequences and textual instructions.
- Video token compression: Prior video compression methods estimate token importance through saliency or similarity metrics and explore several mechanisms for reducing visual redundancy.The cited mechanisms include training-free filtering, variation-aware dropping, hierarchical compression, and curvature-aware spatio-temporal pruning.
- Video token compression: Video-token reduction should preserve holistic context and balance temporal gains with spatial preservation under aggressive pruning.These considerations are identified in recent video compression studies.
3. Method
OmniSIFT compresses interleaved audio-video tokens through chunk-level, modality-asymmetric selection: it prunes spatial and temporal video redundancy, then uses compressed visual context to select audio tokens. A straight-through estimator enables end-to-end optimization through discrete audio-token selection.
- Chunked multimodal processing: OmniSIFT processes synchronized audio-video tokens in aligned chunks, with each chunk serving as the basic processing unit.The implementation uses 2-second chunks, and the interleaved chunk sequence is provided to the LLM with textual instructions.
- Spatio-Temporal Video Pruning: STVP removes spatial and temporal redundancy by computing saliency for visual tokens and retaining the highest-scoring tokens under the visual retention ratio.Spatial saliency measures divergence from a frame-level mean, while temporal saliency captures changes between consecutive frames.
- Spatio-Temporal Video Pruning: Spatial saliency identifies visually distinctive patches by measuring each token’s cosine distance from the global frame representation.Tokens with larger divergence from the global frame context are treated as more informative.
- Spatio-Temporal Video Pruning: Temporal saliency compares corresponding patches across two consecutive frames to capture motion dynamics and appearance changes that contribute new information.A higher temporal saliency score indicates stronger deviation over time.
- Vision-Guided Audio Selection: VGAS uses compressed visual tokens to guide audio-token selection through lightweight cross-attention, making each audio representation incorporate scene-aligned visual information.Audio tokens act as queries, while pruned video tokens provide keys and values; a residual connection preserves original audio semantics.
- Vision-Guided Audio Selection: A sigmoid-scored TopK mask selects audio tokens under the audio retention ratio, while a straight-through estimator propagates gradients through the discrete selection during end-to-end training.The forward pass uses a binary top-k mask, and the backward pass uses an identity surrogate gradient.
4. Experiment
OmniSIFT is evaluated across five audio-visual benchmarks, multiple model sizes and retention ratios, with ablations testing efficiency, robustness, and its modality-asymmetric architecture.
- Experimental Setting: OmniSIFT is evaluated on four audio-visual QA benchmarks and the video-SALMONN-2 captioning test set.Experiments use Qwen2.5-Omni-7B and 3B, with 35% and 25% token retention ratios.
- Experimental Setting: The baselines are OmniZip, DyCoke, and Random Pruning, covering Omni-LLM, videocentric, and uniform-random compression strategies.Compression baselines are applied after fine-tuning the Qwen2.5-Omni backbone under the same setting.
- Main Results: Across all tested settings, OmniSIFT achieves the highest accuracy among compression methods and matches or exceeds the full-token baseline on multiple benchmarks.At 35% retention on Qwen2.5-Omni-7B, it scores 50.0 on WorldSense versus 49.7 for the full-token model.
- Fine-Grained Results: At 25% retention on Qwen2.5-Omni-7B, OmniSIFT scores 66.7 on DailyOmni Event Sequence and 68.9 on AV Event Alignment, exceeding OmniZip’s 61.8 and 59.7.These categories require temporal or cross-modal reasoning under aggressive compression.
- Robustness Across Compression Ratios: As audio compression increases from 0.3 to 0.9, OmniSIFT remains above 49.3% accuracy while OmniZip declines from over 48.9% to approximately 44.0%.The comparison demonstrates greater robustness across audio compression ratios.
- Structural Ablation: Removing either spatial or temporal video selection reduces accuracy, while replacing visual guidance with audio-only selection causes declines of 3.9% on DailyOmni and 2.9% on WorldSense.The spatial and temporal components are complementary, and audio importance is context-dependent.
- Token Compression Paradigm Ablation: OmniSIFT consistently outperforms the modality-symmetric variant across retention ratios, with larger gaps at lower retention.The modality-asymmetric design uses vision-guided audio selection and explicitly models cross-modal dependencies.
- Case Study: A case study shows modality-symmetric compression pruning scoreboard patches when audio and video have unequal importance, whereas OmniSIFT preserves salient video patches and informative audio cues.The example concerns a score-changing scene on OmniVideoBench.
5. Conclusion
OmniSIFT reduces Omni-LLM token counts through modality-asymmetric compression while preserving audio-visual information for downstream reasoning. Across five benchmarks, it outperforms compression baselines and sometimes exceeds full-token models, while improving inference speed and memory usage.
- Conclusion: OmniSIFT is a modality-asymmetric token compression framework for Omni-LLMs.It decouples spatial and temporal video redundancy before using compact visual cues to guide audio selection.
- Conclusion: The two-stage design preserves key audio-visual information while reducing token counts for downstream tasks.Visual cues guide selection of contextually informative audio tokens.
- Conclusion: Across five audio-visual benchmarks, OmniSIFT consistently outperforms compression baselines and exceeds full-token models in several settings.The conclusion also reports substantial gains in inference speed and memory usage.
6. Limitations.
OmniSIFT’s aggressive compression retains strong performance–efficiency tradeoffs but remains limited by localized audio-visual alignment and incomplete long-range temporal modeling. Its fixed, query-agnostic token budget also constrains adaptation to varying information density and task-specific needs.
- Precise audio-visual alignment remains challenging because reasoning can depend on highly localized evidence sensitive to token pruning.
- Temporal redundancy across multimodal chunks and long-range audio-visual structure remain unexplored.
- OmniSIFT uses a fixed, query-agnostic token budget rather than adapting compression to task-specific information demands.
- Future work may explore adaptive budgeting and query-guided pruning while preserving multiturn context and audio-triggered visual evidence.
Impact Statement
OmniSIFT reduces redundant tokens while preserving or enhancing performance, supporting broader deployment of Omni-modal LLMs in resource-constrained or real-time settings.
- OmniSIFT improves efficiency by reducing redundant tokens while preserving or enhancing performance.
- The framework may support applications such as audio-visual question answering and video captioning.
A. Expanded Benchmark Details
The evaluation uses five representative benchmarks spanning major audio-visual understanding capabilities, including temporal reasoning, cross-modal understanding, fine-grained comprehension, and captioning.
- Five representative benchmarks cover long-horizon temporal reasoning, cross-modal alignment and fusion, fine-grained multi-dimensional comprehension, and generative captioning.
- The benchmark suite is presented as the evaluation set for this work.
B. Expanded Implementation Details
The implementation standardizes video preprocessing, calibrates modality-specific compression ratios, and follows established prompt protocols for evaluation and caption judging.
- Videos are sampled at 2 FPS, capped at 256 frames, with each frame configured at a maximum resolution of 320 × 28 × 28 pixels.
- Visual and audio compression ratios are dynamically calibrated across methods and total retention levels.
- QA evaluation prompts follow the protocol established by Fu et al. (2025), while Video-SALMONN-2 follows its original methodology.
- Generated captions are evaluated using an LLM-as-a-judge framework with GPT-4.1.
C. Computing Cost Evaluation
OmniSIFT adds lightweight compression modules whose costs remain below the backbone’s self-attention overhead while substantially reducing computation and resource growth. Ablations show that shallow, modality-specific selection preserves performance more effectively than random or deeper alternatives.
- Computational overhead: The VGAS module adds approximately 4.85M parameters for Qwen2.5-Omni-7B, representing less than 0.1% of the 7B-class backbone.It uses 512-dimensional projections, single-layer cross-attention, and a compact MLP score head.
- Computational overhead: At 25% retention, OmniSIFT requires 250.83T FLOPs versus 555.74T for the full-token baseline, reducing computation by over 50%.The comparison is reported for WorldSense with Qwen2.5-Omni-7B.
- Scalability: OmniSIFT limits resource growth as video duration increases, whereas full-token latency escalates with the backbone’s quadratic self-attention complexity.The scalability analysis covers videos from 0s to 120s and evaluates both latency and memory consumption.
- Selector-depth ablation: At 35% retention, the single-layer selector scores 68.3, 73.2, and 50.0 on VideoMME, DailyOmni, and WorldSense, exceeding the 3-layer variant’s 67.2, 72.3, and 49.0.The deeper variant also raises peak GPU memory from 22.62 GB to 22.67 GB.
- Architecture ablation: Random video or audio pruning and an SSM selector underperform the proposed modules, with the SSM-based selector yielding scores of 67.3 and 47.4.The ablation evaluates the Qwen2.5-Omni-7B model at a 35% retention ratio and supports the modality-asymmetric design.
D.5. Additional Rebuttal Experiments
Additional experiments evaluate OmniSIFT against a recent pruning baseline, across backbones, under adaptive budgets, and with alternative audio selectors. OmniSIFT remains competitive across these settings, while fixed allocation and the proposed VGAS design are supported as practical choices.
- Comparison with FASTAV: OmniSIFT consistently outperforms FASTAV on DailyOmni and WorldSense under a similar token budget.FASTAV uses 50% global pruning followed by 20% fine pruning.
- Transfer to Qwen3-Omni: When transferred to Qwen3-Omni, OmniSIFT incurs only minor performance degradation while reducing prefilling latency and GPU memory usage.The transfer results are summarized in Table 10, where latency is measured in seconds and memory in GB.
- Adaptive budget allocation: Under the same 35% retention budget, adaptive video-token allocation provides no clear performance gains and increases prefill latency.The additional latency comes from the budget-estimation step, supporting fixed allocation as a practical design choice.
- Alternative VGAS designs: The proposed VGAS design achieves the best overall performance among the evaluated audio-compression variants at a 35% token-retention budget.The alternatives remove either the score head or the cross-attention module.