PaperlayerView exact revision on arXiv

Source-linked AI summary

OmniScope: Modality-Decoupled Token Compression for Omnimodal Large Language Models

Jinsen Su, Yongdong Luo, Yuexiao Ma, Yibo Hu, Meiguang Jin, Xiawu Zheng

arXiv:2607.23193v3cs.CV

TL;DR

Omnimodal token compression can lose important information when one modality guides another despite differing audio-video relevance patterns. OmniScope instead uses a shared query anchor with independent modality compression, achieving the best average performance across compression settings while reducing inference costs.

  • Problem

    Prior omnimodal compression methods rely on unidirectional cross-modal guidance even though audio and video relevance can peak at different moments, risking loss of critical tokens.

  • Method

    OmniScope independently scores audio and visual tokens against a shared query, then applies anchor-delta visual pruning and per-second audio merging.

  • Results

    Across four benchmarks and two model scales, OmniScope achieves the best average performance at all compression ratios; at 25% retention, the 7B model drops only 0.35 points.

  • Takeaways & Limitations

    The results support sharing the query across modalities while keeping their salience estimates separate during omnimodal inference.

Abstract

from arXiv · show

Existing token compression methods for omnimodal large language models typically rely on one modality to determine what to retain in the other. We show that this assumption often breaks down: for the same query, audio and video relevance often peaks at different moments. This cross-modal salience mismatch makes unidirectional guidance prone to discarding answer-critical cues under aggressive compression. We propose OmniScope, a training-free token compression framework that uses the query as a shared semantic anchor while estimating relevance separately for audio and video. OmniScope allocates modality-specific token budgets, prunes visual tokens with an anchor-delta strategy that preserves both global context and temporal changes, and merges audio tokens within each second to reduce redundancy while maintaining temporal continuity. Across four audio-video benchmarks and two Qwen2.5-Omni model scales, OmniScope achieves the best average accuracy across all compression settings. At 25% overall token retention, it delivers up to 3.53x prefill speedup and more than 15% GPU memory reduction, with only a 0.35-point drop in average accuracy. These results suggest a simple design principle for OmniLLM inference: share the query across modalities, but not the salience estimates. The code is available at https://github.com/MAC-AutoML/OmniScope.

B Corresponding Author · 1 Introduction

OmniScope addresses the efficiency and reliability limits of omnimodal token compression by decoupling audio and visual salience estimation while sharing the query as a semantic anchor. Across four benchmarks and two model scales, it achieves the best average performance across compression ratios while substantially reducing inference cost.

  • 1 Introduction: OmniLLMs jointly process visual, auditory, and textual modalities, enabling end-to-end audio-video understanding but creating substantial deployment demands.Separate audio and visual encoders produce rapidly growing token sequences as video duration increases, increasing computation and memory overhead.
  • 1 Introduction: Existing vision-only compression methods cannot directly compress audio and video jointly, while omnimodal methods commonly use unidirectional cross-modal guidance.This approach assumes one modality can determine which tokens should be retained in the other.
  • 1 Introduction: Cross-modal salience mismatch makes unidirectional guidance unreliable because visually decisive moments can be acoustically unremarkable, and audio-critical segments can show little visual change.Thus, low-salience regions in one modality may still contain critical cues for the other.
  • 1 Introduction: Compression amplifies cross-window modal interaction, but discarded critical tokens can redirect that attention toward secondary cues under salience mismatch.Retaining query-relevant tokens independently in each modality avoids this failure mode.
  • 1 Introduction: OmniScope is a training-free framework that shares the query as a semantic anchor while independently estimating modality relevance and allocating audio and visual compression budgets.Its visual branch uses AD-STC for global coverage and temporal changes, while its audio branch merges tokens per second to reduce redundancy and preserve continuity.
  • 1 Introduction: 45% retention yields nearly lossless accuracy, while 25% retention delivers up to 3.53× prefilling speedup and over 15% GPU memory reduction.Across four benchmarks and 7B and 3B model scales, OmniScope achieves the best average performance at all compression ratios and remains more robust at aggressive retention.
  • 1 Introduction: AD-STC addresses lost global semantic coverage in video compression through anchor-delta spatio-temporal selection.The strategy balances global semantic coverage with temporal increments.
  • 1 Introduction: Per-second bipartite soft matching compresses post-encoder audio embeddings at finer granularity while preserving temporal continuity.This addresses the coarse granularity of existing audio compression methods.

2 Related Work

OmniLLMs unify visual, auditory, and textual modalities for end-to-end audio-video understanding, extending beyond vision-language models that handle only visual and textual inputs. Related token compression work spans similarity-based merging, query-guided compression, attention-based pruning, and video spatio-temporal methods exploiting interframe redundancy.

  • Omnimodal large language models: OmniLLMs integrate visual, auditory, and textual modalities within one model for end-to-end audio-video understanding.They target applications including video question answering and content moderation.
  • Omnimodal large language models: Unlike vision-language models limited to visual and textual modalities, OmniLLMs model synergistic semantics between visual scenes and acoustic signals.The passage attributes significant advantages in applications such as video question answering and content moderation to this multimodal integration.
  • Token compression: Visual token compression methods include similarity-based merging, query-based text-guided compression, and attention-based pruning.Video methods additionally exploit interframe redundancy through spatio-temporal compression to reduce token counts.

3 Method

OmniScope is a training-free, query-aware token compression framework that independently estimates visual and audio relevance while sharing the query as a semantic anchor. It combines per-position budget allocation with anchor–delta visual pruning and per-second audio merging to reduce redundant tokens while preserving task-relevant information.

  • Motivation: Redundant tokens in temporal windows dilute attention available to important cross-window tokens and increase computational and memory overhead as video duration grows.OmniLLMs interleave co-temporal visual and audio tokens into chronological cross-modal blocks before LLM processing.
  • Overview: OmniScope independently scores visual and audio tokens against the input query, allocates modality-specific per-position budgets, and compresses each modality separately.This modality-decoupled design addresses the loss of critical non-guiding-modality tokens caused by cross-modal guidance.
  • Query-Aware Scoring: CLIP independently scores video-frame relevance using cosine similarity between frame and query features, while audio relevance is computed directly in the LLM token embedding space.CLAP is considered ill-suited because its clip-level several-second granularity is too coarse for per-second or per-token scoring; the visual side therefore retains external CLIP scoring.
  • Query-Aware Scoring: Audio token similarities are gated with query embedding norms, aggregated by averaging top-k scores within each second, and propagated to nearby high-scoring seconds with distance decay.The propagation uses high-scoring seconds above the P-th percentile and parameters β, γ, and R.
  • Vision Pruning: AD-STC alternates anchor and delta frames, preserving spatially distinctive tokens in anchors and using complementary temporal-spatial selection to retain coverage under compression.At high retention ratios, the method uses a joint irreplaceability score accounting for spatial and temporal dimensions, with source tokens paired to their most similar destination tokens.

4 Experiments

Experiments evaluate OmniScope across diverse audio-video benchmarks and Qwen2.5-Omni model scales under fixed compression settings. OmniScope consistently leads compressed methods in accuracy, robustness, and inference efficiency, while ablations support independent query-guided modality compression.

  • Benchmarks: OmniScope is evaluated on WorldSense, DailyOmni, OmniVideoBench, and Video-MME, covering varied task types, video durations, and modality balances.WorldSense spans eight domains, while the other benchmarks target everyday omnimodal QA, comprehensive video understanding, and audio-enhanced video understanding.
  • Experimental setup: Experiments use Qwen2.5-Omni at 7B and 3B scales with FlashAttention-2 enabled throughout, and report 45% and 25% overall token retention.The retention settings correspond to (ρv=0.6, ρa=0.25) and (ρv=0.8, ρa=0.35), respectively, with fixed hyperparameters across benchmarks and model scales.
  • Accuracy: Across all evaluated settings, OmniScope achieves the highest average accuracy among compression methods and the smallest accuracy drop versus Full Tokens at 25% retention.At 45% retention, accuracy is described as nearly lossless at both model scales.
  • Fine-grained results: OmniScope achieves the highest average WorldSense accuracy at both retention levels and model scales, remaining robust across eight domains and modality balances.Competing methods show uneven domain performance, including OmniZip’s sharp drops on Performance and Games at 25% retention on 7B.
  • Efficiency: 3.53× prefilling speedup is achieved at 25% retention on the 7B model, while OmniScope also has the lowest GPU memory footprint at both model scales.At 45% retention on 7B, the prefilling speedup is 2.87×; CLIP visual scoring adds one-time overhead outside the LLM inference pipeline.
  • Ablations: Uniform compression causes a 1.65-point average drop, while random-drop and energy-based audio filtering degrade most by disrupting temporal continuity and discarding information.Energy-based filtering performs worst because audio-token energy reflects acoustic loudness rather than semantic importance.

5 Conclusion · Appendix

OmniScope is a training-free, modality-decoupled token compression framework for omnimodal large language models. It uses a shared query anchor while independently assessing visual and audio token importance to allocate compression budgets.

  • 5 Conclusion: OmniScope provides training-free token compression for omnimodal large language models.
  • 5 Conclusion: The framework uses the query as a shared semantic anchor across modalities.
  • 5 Conclusion: Visual and audio tokens independently assess importance and receive separate compression budgets.This modality-decoupled allocation addresses cross-modal salience mismatch.

A Per-Domain Results on WorldSense · B Hyperparameter Settings

On WorldSense, OmniScope achieves the strongest average accuracy across retention levels and model scales while remaining competitive across domains. Its fixed hyperparameter configuration is shared across benchmarks and model scales without per-dataset tuning.

  • A Per-Domain Results on WorldSense: OmniScope achieves the highest average accuracy on WorldSense at both 45% and 25% retention across both model scales.Table 6 reports the best result among compression methods for each metric.
  • A Per-Domain Results on WorldSense: OmniScope remains competitive across the eight diverse WorldSense domains.WorldSense covers eight domains, with per-domain results summarized in Table 6.
  • A Per-Domain Results on WorldSense: On the 7B model, OmniZip performs well on Culture & Politics but drops sharply on Performance and Games at 25% retention.The example illustrates uneven cross-domain performance among competing methods.
  • A Per-Domain Results on WorldSense: The WorldSense results further validate independently compressing each modality based on its own relevance.The supplied passage presents this design as supported by the per-domain findings.
  • B Hyperparameter Settings: Table 7 lists all hyperparameters used in OmniScope.The audio scoring module uses a temperature coefficient, top-k aggregation count, and neighbor boost parameters; AD-STC uses a switching threshold and DPC-KNN neighbor count.
  • B Hyperparameter Settings: The same OmniScope configuration is used across all benchmarks and both model scales without per-dataset tuning.This configuration includes parameters for both the audio scoring and AD-STC modules.

C CLIP Scoring Overhead Analysis

The CLIP visual scorer adds a fixed one-time overhead whose share of end-to-end latency decreases as generation length increases, indicating progressive amortization.

  • C CLIP Scoring Overhead Analysis: The declining latency share confirms that the fixed CLIP scoring cost is progressively amortized in longer generation scenarios.

D Visual Scoring Strategy Ablation

Table 8 shows that external CLIP scoring outperforms OmniLLM Self-Embed visual scoring across model scales and compression ratios, with smaller performance losses for 7B than 3B.

  • Visual scoring comparison: CLIP scoring achieves higher overall average accuracy than Self-Embed across both model scales and compression ratios.Self-Embed replaces the external CLIP scorer with OmniLLM’s visual embeddings; ∆ denotes the difference relative to CLIP scoring.
  • Visual scoring comparison: Approximately 0.5 points is the average 7B performance drop for Self-Embed relative to CLIP scoring.The 7B model’s Self-Embed degradation is relatively small across the ablation settings.
  • Visual scoring comparison: Approximately 0.9 points is the average 3B performance drop for Self-Embed relative to CLIP scoring.The 3B model shows a more pronounced Self-Embed degradation than the 7B model.

E Ratio Threshold Sensitivity Analysis

The ratio threshold τr in AD-STC controls switching between two Delta frame scoring strategies, with DailyOmni accuracy following an inverted U-shaped curve and peaking in the middle region. This indicates robustness to threshold selection across a wide optimal range without fine-grained tuning.

  • Ratio Threshold Sensitivity Analysis: On DailyOmni, accuracy versus τr exhibits a clear inverted U-shaped trend, reaching its optimal peak in the middle region.The analysis uses Qwen2.5-Omni-3B with ρv=0.6 and ρa=0.25.
  • Ratio Threshold Sensitivity Analysis: The broad middle-region optimum shows that AD-STC is robust to threshold selection and does not require fine-grained tuning.τr determines the switching point between the two Delta frame scoring strategies.

F Cross-Modal Salience Mismatch Analysis

OmniScope’s analysis measures cross-modal salience mismatch by correlating query-conditioned visual and audio importance scores on DailyOmni. Across 1,197 query-video pairs, approximately 78.3% show only weak correlation, indicating that mismatch is pervasive.

  • Measurement: OmniScope quantifies mismatch using Spearman correlation between visual CLIP and audio CLAP query-similarity scores on DailyOmni.CLIP scores each frame against the query, while CLAP scores each one-second audio segment; both serve as external probes independent of OmniScope’s scoring pipeline.
  • Findings: Approximately 78.3% of 1,197 DailyOmni query-video pairs exhibit weak visual-audio correlation (|ρ| < 0.3).The distribution confirms that cross-modal salience mismatch is pervasive rather than anecdotal.
Loading 2607.23193v3…