PaperlayerView exact revision on arXiv

Source-linked AI summary

OmniDelta: Skill-Driven Budget Allocation for Token Compression in OmniLLMs

Haoyang Huang, Wenjie Huang, Tianqi Xu, Hongyaoxing Gu, Kang Tan, Yikai Fu, Yuhao Shen, Tianyu Liu, Baolin Zhang, Jun Zhang, Xinyi Hu, Jun Dai, Shuang Ge, Lei Chen, Yue Li, Mingchen Wang, Meng Zhang

arXiv:2607.25669v1cs.AI

TL;DR

OmniDelta addresses the underexplored problem of allocating limited tokens across audio, video, and temporal regions in costly OmniLLM inference. Its training-free, skill-driven allocation improves the accuracy-efficiency tradeoff across benchmarks, including 22.0% lower GPU memory and 1.64× speedup at 25% retention on Qwen2.5-Omni-7B.

  • Problem

    Long audio-video sequences make OmniLLM inference costly, while fixed or uniform compression budgets leave query- and content-dependent budget allocation underexplored.

  • Method

    OmniDelta uses query-skill routing for inter-modal allocation and local complexity and temporal redundancy for audio-segment and video-frame budgets under a fixed retention ratio.

  • Results

    OmniDelta advances the accuracy-efficiency Pareto frontier across four benchmarks and two Qwen2.5-Omni models, reducing GPU memory by 22.0% and achieving 1.64× speedup at 25% retention on 7B.

  • Takeaways & Limitations

    Budget allocation before token selection is an effective direction for efficient omni-modal inference while remaining compatible with existing pruning strategies.

Abstract

from arXiv · show

Emerging Omni-modal Large Language Models (OmniLLMs) enable unified understanding of text, audio, and video, but their long audio-video token sequences introduce substantial memory and inference costs. Existing compression methods mainly focus on selecting important tokens under fixed budgets, leaving the preceding budget-allocation problem underexplored. We show that direct query-to-audio/video similarity is unreliable for inter-modal budget allocation, and that uniform intra-modal budgets can miss key evidence while retaining redundant content. To address these limitations, we propose OmniDelta, a training-free, skill-driven framework that couples intent-aware inter-modal allocation with content-aware intra-modal allocation. OmniDelta first constructs audio and video skill pools to shift the fixed retained-token budget according to query demand, then reallocates modality budgets over audio segments and video frames using local complexity and temporal redundancy. The resulting local budgets can be combined with existing pruning strategies, preserving the total retained-token ratio while changing where the budget is spent. Experiments on four audio-video benchmarks with two Qwen2.5-Omni models show that OmniDelta establishes a new accuracy-efficiency Pareto frontier across pruning ratios. At 25% token retention on Qwen2.5-Omni-7B, OmniDelta reduces GPU memory by 22.0% and achieves a 1.64x end-to-end speedup over full-token inference.

1. Introduction

OmniLLMs face substantial inference costs because unified audio-video inputs create long multimodal token sequences, while existing compression methods underexplore how to allocate a fixed retained-token budget. OmniDelta addresses this with training-free, skill-driven inter- and intra-modal allocation, improving the accuracy-efficiency tradeoff.

  • Motivation: A 60-second video at 2 FPS with 72 visual tokens per frame and 25 audio tokens per second yields roughly 10K audio-video tokens before text and special tokens.Such long multimodal contexts substantially increase memory usage and inference cost.
  • Budget-allocation diagnostics: Uniform intra-modal budgets can miss critical evidence because information is unevenly distributed across audio segments and video frames.Videos contain information-rich key frames and low-information transitional frames, while audio likewise alternates between informative and less informative content.
  • Skill-driven budget allocation: OmniDelta is a training-free hierarchical allocator that redistributes a fixed retained-token budget using query-skill routing, local complexity, and temporal redundancy.It constructs modality-specific audio and video skill pools from representative keywords expanded with semantically related terms, then allocates tokens across modalities, audio segments, and video frames.
  • Accuracy–efficiency gains: 22.0% GPU memory reduction and 1.64x end-to-end speedup are achieved at 25% token retention on Qwen2.5-Omni-7B.Across four audio-video benchmarks and two Qwen2.5-Omni model sizes, OmniDelta consistently advances the accuracy-efficiency Pareto frontier.

2. Motivation

OmniDelta is motivated by two weaknesses in existing budget assignment: direct query-to-modality similarity may misidentify the required evidence source, and uniform within-modality budgets may waste capacity while missing key evidence. Diagnostics show that skill-pool routing better reflects query intent, while nonuniform budget allocation is necessary beyond pruning alone.

  • Motivation: Existing compression methods usually fix retained budgets across queries or assign them with predefined temporal policies, leaving budget allocation underexplored.The motivation distinguishes deciding which tokens to remove from deciding where the retained budget should be spent.
  • Inter-modal allocation: Direct query-to-audio/video similarity can fail because queries encode task intent, whereas modality tokens represent sample-specific content.This mismatch motivates evaluating whether similarity is a reliable modality-level routing signal.
  • Inter-modal allocation: Deeper contextualization can increase cross-modal mixing, causing acoustically salient terms in an audio-oriented query to favor video.A representative Top-3 pooling failure shows every word receiving a positive video-minus-audio margin after the Thinker stage.
  • Inter-modal allocation: 60.8% is the best accuracy reached by direct similarity routing, which remains close to random guessing across representation stages and pooling rules.Skill-pool routing outperforms matched direct variants by 25.5–28.6 percentage points, supporting task prototypes as better intent representations.
  • Intra-modal allocation: Uniform within-window budgets can preserve redundant units while underallocating informative ones, and answer-aware pruning cannot fully compensate for an unsuitable split.Across audio and video, Uniform–Oracle remains far below Oracle–Oracle and performs close to First-only, demonstrating that allocation and content selection are distinct.

3. Method

OmniDelta treats omni-modal compression as hierarchical budget allocation under a fixed retained-token budget. It first shifts budget between audio and video using query-driven skills, then reallocates each modality’s budget across temporal units using complexity and redundancy before token pruning.

  • Hierarchical Budget Allocation: OmniDelta decomposes compression into global budget allocation, modality-level transfer, local redistribution, and subsequent token pruning.The total retained budget remains fixed; the method changes where that budget is spent.
  • Query-Driven Modality Allocation: Offline skill pools identify audio demands such as speech, speakers, music, and sound events, alongside video demands including objects, OCR, and actions.GPT-5.5-xhigh extracts diagnostic keywords and semantically related terms from WorldSense task types.
  • Query-Driven Modality Allocation: Signed query modality bias transfers budget toward the more relevant modality, while ambiguous queries cause only a small shift and preserve Ka + Kv = K.Positive bias indicates stronger video demand; negative bias indicates stronger audio demand.
  • Content-Aware Local Allocation: Within each modality, OmniDelta allocates budget to one-second audio segments and individual video frames using local complexity and temporal redundancy.Complex, diverse units receive more tokens, whereas units repeating the previous temporal unit can be compressed more aggressively.
  • Content-Aware Local Allocation: The local allocation rule prioritizes units with high complexity and low redundancy while enforcing feasible keep intervals and the modality budget.The same rule applies to audio segments and video frames.
  • Token Pruning: Following OmniZip, the pruning backend selects concrete audio and video tokens under allocated local budgets using encoder attention and ISTC.Audio masks use attention scores and aligned chunk priors; video ISTC alternates temporal similarity pruning with spatial DPC-KNN selection.

4. Experiments

OmniDelta is evaluated on four audio-video QA benchmarks against full-token inference and three compression baselines. Across retained-token budgets, ablations, sensitivity analyses, and efficiency measurements, it improves accuracy while reducing inference cost.

  • Experimental Setup: OmniDelta is evaluated on WorldSense, AVUT, VideoMME with audio, and DailyOmni using Qwen2.5-Omni-7B and Qwen2.5-Omni-3B.WorldSense contains 3,172 QA pairs across eight real-world domains, while AVUT covers audio-centric tasks including localization, OCR matching, extraction, counting, and character matching.
  • Comparison with Baselines: OmniDelta improves WorldSense averages over OmniZip at both retained ratios and model sizes: 43.2 to 44.2 at 25% for 7B, and 42.3 to 43.2 at 20% for 3B.The corresponding 7B 20% result is 42.7 to 43.0, while the 3B 25% result is 44.1 to 44.7.
  • Shift-Ratio Sensitivity: Across 27 shift-coefficient configurations, average accuracy ranges from 43.28%–44.29%, with query-level shifting most sensitive and intra-modal shifts more stable.Moderate video shifts peak around 𝜆ᵥ=0.30 on average.
  • Budget Allocation Visualization: OmniDelta assigns non-uniform budgets across audio segments and video frames, concentrating tokens on task-relevant temporal regions and compressing redundant regions more aggressively.The video branch redistributes visual budgets at the frame level rather than keeping a fixed budget within each time window.
  • Ablation Study: OmniDelta’s three allocation levels each contribute improvements, and their combination gives the best overall average.The levels adjust the audio-video modality split, redistribute audio evidence across temporal segments, and refine visual budgets across frames.
  • Inference Efficiency: On Qwen2.5-Omni-7B, GPU memory falls from 30.0G to 23.4G, with 1.69× TTFT and 1.64× end-to-end latency speedups versus full-token inference.On Qwen2.5-Omni-3B, OmniDelta also achieves the lowest memory usage and latency among compressed methods.

5. Related Work

Related work develops token pruning for individual modalities and increasingly extends compression to unified audio-video LLMs. These approaches include heuristic, learned, and pipeline-specific strategies for reducing perceptual tokens, caches, and semantic chunks.

  • Omni-modal LLMs: OmniLLMs unify text, vision, speech, and audio processing through a single autoregressive interface.Recent video and audio-video LLMs have strengthened temporal reasoning and audio-aware video understanding, while Omni-series and open-source omni-modal models demonstrate broader integration.
  • Token Pruning: Token pruning reduces multimodal inference cost by removing redundant perceptual tokens before LLM processing.Image methods use token merging, early-layer filtering, and sparsification; video methods exploit temporal saliency, frame similarity, and hierarchical spatio-temporal structure; audio methods are also studied.
  • OmniLLM Compression: OmniLLM compressors extend pruning across audio and video through cross-modal guidance, learned selectors, and predefined compression pipelines.OmniZip uses audio attention for video pruning, OmniSIFT and EchoingPixels learn cross-modal selectors, and FastAV, AccKV, and DASH compress tokens, KV caches, or semantic chunks.

6. Conclusion … B. Theoretical Properties of Dynamic Budget Allocation

OmniDelta is a training-free hierarchical allocator that determines where a fixed retained-token budget is spent across modalities and temporal units. Its analysis establishes smooth, bounded transfer, directional local redistribution, and exact budget conservation independently of downstream pruning.

  • 6. Conclusion: OmniDelta allocates a fixed retained-token budget using query-skill similarity across modalities and local redundancy or complexity within audio and video.The framework remains compatible with existing pruning strategies.
  • Appendix: The appendix organizes supporting material into dynamic-allocation pseudocode, theoretical properties, complexity analysis, diagnostic studies, and modality skill-pool construction.These materials are presented in Sections A–E.
  • A. Detailed Algorithm: Algorithm 1 distributes the fixed retained-token budget first between modalities and then among their temporal units.Its inputs include the query, audio/video skill pools, projected tokens, target retained ratio, prior, and shift coefficients.
  • A. Detailed Algorithm: The algorithm’s retained-token target is computed as K = round(r(N_a + N_v)) before query-aware inter-modal allocation.This fixed total is subsequently redistributed across modalities and temporal units.
  • A. Detailed Algorithm: The allocator partitions audio into one-second segments and video into individual frames for fine-grained local allocation.These temporal units receive local keep counts after modality-level allocation.
  • B. Theoretical Properties of Dynamic Budget Allocation: The theoretical analysis treats the two-stage allocator independently of its downstream pruning operator.It establishes smooth and bounded query-aware transfer, directionally consistent local redistribution, and exact retained-token conservation.

B.1. Bounded and Monotonic Inter-Modal Transfer · B.2. Directional and Bounded Intra-Modal Redistribution

OmniDelta’s inter-modal transfer is bounded, monotonic, and budget-preserving, while intra-modal redistribution shifts capacity directionally according to local complexity and redundancy within bounded limits.

  • B.1. Bounded and Monotonic Inter-Modal Transfer: Δq = λq K tanh(g/2) is odd, strictly increasing in the video–audio skill gap g, and bounded before range projection and integer correction.Here, g = rv − ra and K > 0, λq > 0.
  • B.1. Bounded and Monotonic Inter-Modal Transfer: The transfer shifts the audio and video priors oppositely: eKa = K0a − Δq and eKv = K0v + Δq.The resulting modality budgets preserve the total retained budget, eKa + eKv = K.
  • B.1. Bounded and Monotonic Inter-Modal Transfer: ProjectExact enforces feasible modality ranges and integer counts while preserving the transferred allocation’s total budget.The operation follows the continuous transfer and maintains eKa + eKv = K.
  • B.1. Bounded and Monotonic Inter-Modal Transfer: For any two score gaps g1 and g2, query routing remains stable as the score-gap change is bounded by the transfer’s tanh-based sensitivity.This stability result follows from the mean value theorem.
  • B.1. Bounded and Monotonic Inter-Modal Transfer: Near g = 0, Δq = λqK(g/2) + O(g^3), so ambiguous queries induce small transfers while clear preferences receive larger bounded adjustments.The local expansion distinguishes weak from strong modality preferences without changing the total retained budget.
  • B.2. Directional and Bounded Intra-Modal Redistribution: A unit with positive centered clipped score can only lose budget, by at most λmLi, whereas a negative score can only gain budget by at most λmLi.The original interval [ℓm, Li] bounds each local shift.
  • B.2. Directional and Bounded Intra-Modal Redistribution: High temporal redundancy relative to internal complexity yields positive zm and more aggressive compression, while complex, less repetitive units have negative zm and receive capacity.If original intervals cannot contain the exact modality budget, boundary relaxation minimally repairs feasibility before integer allocation.

B.3. Exact Integer Budget Conservation … C.3. Intra-Modal Signal and Allocation Overhead

OmniDelta guarantees exact integer and sample-level budget conservation while motivating heterogeneous local budgets through marginal-gain analysis. Its training-free allocation overhead is bounded and lightweight, with cached skill routing and local signal computation added around shared encoder features.

  • B.3. Exact Integer Budget Conservation: The bounded largest-remainder procedure returns integer local allocations satisfying lower and upper bounds while conserving the prescribed total budget.This follows after interval construction and feasibility repair.
  • B.3. Exact Integer Budget Conservation: Different queries and local redundancy patterns change budget placement without changing the sample-level retained ratio.Exact local allocation combined with inter-modal projection provides the retained-ratio guarantee.
  • B.4. Why Heterogeneous Budgets Are Necessary: Uniform allocation is not optimal when feasible units have heterogeneous marginal gains under decreasing, convex information-loss functions.At an interior optimum, active units instead require equal marginal gains.
  • B.4. Why Heterogeneous Budgets Are Necessary: OmniDelta uses high local complexity and low temporal redundancy as training-free proxies for larger marginal gains, while repetitive units can absorb more compression.The proxies motivate non-uniform allocation but are not assumed to recover the unknown loss functions exactly.
  • C.1. Notation: The notation defines N = N_a + N_v as pre-pruning audio-video tokens, N_t as unchanged text and special tokens, and d and L as shared hidden dimension and layer count.It also defines full and compressed input lengths, total skills S, query length L_q, routing Top-k, local units U, and maximum local length L_max.
  • C.2. Skill-Pool Routing: Skill embeddings are sample-independent and cacheable, while query routing costs O(L_qd) for pooling, O(Sd) for similarities, and O(Slog k) for Top-k selection.The online routing overhead is incurred once per query and is linear in skill-bank size.
  • C.3. Intra-Modal Signal and Allocation Overhead: Intra-modal means, complexity, and adjacent-unit redundancy each require O(Nd), whereas normalization, centering, and interval construction require O(U).These computations visit modality tokens once or perform linear work over local units.
  • C.3. Intra-Modal Signal and Allocation Overhead: Bounded largest-remainder allocation costs O(L_maxUlogU), effectively O(UlogU) for Qwen2.5-Omni because audio segments contain 25 tokens and video frames approximately 72.The allocator stores the cached skill bank and constant statistics per local unit, excluding encoder features shared with the base model.

C.4. Overall Inference and Memory Complexity · D. Additional Details of the Motivation Studies

OmniDelta reduces sequence-dependent inference and KV-cache costs by retaining a fraction of audio-video tokens while preserving the fixed retained-token count. The motivation studies additionally document routing-set construction and controlled budget-allocation diagnostics.

  • C.4. Overall Inference and Memory Complexity: The full-sequence Transformer cost is dominated by self-attention and linear projection/feed-forward operations over the token sequence.The self-attention term scales with the squared sequence length, while projections and feed-forward layers scale with sequence length and model width.
  • C.4. Overall Inference and Memory Complexity: After retaining rN audio-video tokens, the LLM’s sequence-dependent computation is recalculated using the compressed sequence length.The complete inference cost also includes pruning-backend computation, represented by T_prune.
  • C.4. Overall Inference and Memory Complexity: The quadratic attention term shrinks by (N̄r/N̄)^2, approaching r^2 when audio-video tokens dominate the shorter text sequence.At retained ratios of 25% and 20%, this corresponds to approximately 6.25% and 4% of the full pairwise-attention term, respectively.
  • C.4. Overall Inference and Memory Complexity: End-to-end speedup is smaller than the attention reduction because encoder computation, projections, and feed-forward layers remain part of inference.The pruning backend also contributes T_prune, which is not introduced by the budget allocator.
  • C.4. Overall Inference and Memory Complexity: Sequence-dependent KV-cache memory changes from O(LN̄d) to O(LN̄rd), while OmniDelta adds only O(Sd + U) allocator memory.OmniDelta preserves the same rN token count as any fixed-budget method at the same retained ratio.
  • C.4. Overall Inference and Memory Complexity: At a fixed budget, OmniDelta leaves theoretical downstream cost unchanged and instead improves accuracy by placing the budget more effectively.Its allocation overhead is linear or near-linear.
  • D. Additional Details of the Motivation Studies: The supplementary motivation studies describe construction of the balanced modality-routing set and test whether routing quality affects downstream question answering.They also provide complete protocols for controlled video and audio budget-allocation diagnostics.

D.1. Inter-Modal Budget Allocation Diagnostic

The diagnostic constructs a balanced, high-confidence WorldSense subset to test whether queries indicate audio or video evidence needs. Downstream results show skill-pool routing outperforms direct embedding routing and the fixed prior under identical pruning conditions.

  • Dataset construction: The subset contains exactly 400 queries: 200 strongly audio-relevant and 200 strongly video-relevant.Selection uses WorldSense questions and answer options, excluding mixed or ambiguous queries and ground-truth or model-prediction information.
  • Annotation criteria: Queries are classified individually from their questions and answer candidates, with the most diagnostic modality-relevant phrase recorded.The procedure prioritizes unambiguous, high-confidence cases and excludes questions where both modalities are necessary or either modality is plausible alone.

D.2. Intra-Modal Budget Allocation Diagnostics

Controlled video and audio diagnostics compare budget placement with retained-content selection under fixed total budgets. Across both modalities, reallocating budget improves evidence retention even with random pruning, while strong local selection cannot compensate for insufficient unit budgets.

  • Video protocol: Video diagnostics sample 50 WorldSense clips, retain the same 25% total visible budget across five policies, and score answers on a 0–5 scale.Each clip contains four evenly spaced visual views; masked image regions proxy removed visual tokens.
  • Video protocol: Video policies separate uniform, weighted, first-only, and oracle allocation effects while keeping the total retained visual budget fixed.Weighted–Random uses [0.70, 0.15, 0.10, 0.05], whereas Uniform–Oracle preserves uniform allocation with answer-aware region selection.
  • Audio protocol: Audio diagnostics sample 50 WorldSense files, retain exactly 12 of 24 chunks (50%), and replace removed 0.5-second chunks with silence.Each 12-second clip is divided into four 3-second segments, with five policies evaluated against reference answers.
  • Controls and interpretation: Both modalities show that changing only budget distribution improves over Uniform–Random, while Uniform–Oracle remains below Oracle–Oracle despite answer-aware pruning.The result indicates that local content selection cannot recover evidence from a unit receiving insufficient budget.

E. Modality Skill-Pool Details · E.1. Skill-Pool Construction · E.2. Representative Skill-Pool Entries

OmniDelta constructs offline audio and video skill pools from WorldSense task types and lexical cues for query-aware modality routing. The pools combine compact modality keywords with task-level phrases, which are jointly embedded and retrieved to match query intent across granularities.

  • E. Modality Skill-Pool Details: OmniDelta extracts modality-specific task types and lexical cues from WorldSense to support query-aware inter-modal allocation.The construction uses WorldSense questions and answer candidates to identify tasks predominantly requiring acoustic or visual evidence.
  • E.1. Skill-Pool Construction: The offline construction uses GPT-5.5-xhigh to identify representative audio- or video-dominant tasks and expand compact keywords with related words and short phrases.Mixed or ambiguous tasks requiring both modalities are excluded.
  • E.1. Skill-Pool Construction: The pools are designed to describe modality and task requirements rather than sample-specific video content, enabling embedding-based comparison with queries.The resulting lists are intended for cosine-similarity routing after embedding with the model’s text embedding layer.
  • E.1. Skill-Pool Construction: The construction outputs compact skill lists and task-level phrase lists, targeting approximately 60–120 compact entries and 10–30 phrases per modality.The JSON fields include version, description, audio_skills, video_skills, audio_skill_phrases, and video_skill_phrases, with validation for modality assignment, concision, and coverage.
  • E.1. Skill-Pool Construction: The stored bank contains 73 compact audio entries and 105 compact video entries, plus 17 task-level phrases per modality, yielding inference banks of 90 audio and 122 video entries.Each entry is encoded once with the OmniLLM thinker input embedding layer, mean-pooled, ℓ2-normalized, and cached.
  • E.2. Representative Skill-Pool Entries: Compact keywords and longer phrases share one flat modality bank, allowing joint Top-k retrieval to match query intent at different granularities without hand-crafted task-family weights.The implementation applies retrieval over all compact skills and task-level phrases jointly.
  • E.2. Representative Skill-Pool Entries: Representative audio entries cover listening, speech and speakers, music and rhythm, sound events and sources, acoustic changes, loudness, silence, and related task-level phrases.Examples include “who is speaking,” “identify the sound,” “count audio events,” and “listen to the soundtrack.”
  • E.2. Representative Skill-Pool Entries: Representative video entries cover scenes and objects, attributes and spatial relations, on-screen text and diagrams, actions and motion, temporal interactions, and explicit visual task phrases.Examples include “read on-screen text,” “visual appearance,” “spatial layout,” and “track visual changes.”
Loading 2607.25669v1…