Source-linked AI summary
Small Vision-Language Models are Smart Compressors for Long Video Understanding
Junjie Fei, Jun Chen, Zechun Liu, Yunyang Xiong, Chong Zhou, Wei Wen, Junlin Han, Mingchen Zhuge, Saksham Suri, Qi Qian, Shuming Liu, Lemeng Wu, Raghuraman Krishnamoorthi, Vikas Chandra, Mohamed Elhoseiny, Chenchen Zhu
TL;DR
Long videos exceed downstream context limits, while query-agnostic compression can discard decisive evidence. Tempo uses an SVLM for query-conditioned cross-modal distillation and ATA for zero-shot, budget-aware routing, achieving state-of-the-art long-video performance while often compressing videos below available limits.
Problem
Hour-long videos produce massive visual streams that overwhelm bounded LLM contexts, while existing query-agnostic methods risk discarding query-critical evidence.
Method
Tempo combines an SVLM local compressor with an LLM global decoder and uses ATA to route tokens according to zero-shot relevance and semantic front-loading.
Results
Tempo-6B achieves state-of-the-art long-video performance, scoring 52.3 on LVBench under an 8K visual budget and 53.7 when scaling to 2048 frames.
Takeaways & Limitations
Tempo’s scaling analysis indicates that optimal resource allocation depends on video duration, with hour-long understanding benefiting from larger contextual capacities while compression remains below practical limits.
Takeaways & Limitations
ATA currently relies on the SVLM’s zero-shot relevance capability, while autoregressive adaptive extraction remains an unresolved latency challenge.
Abstract
from arXiv · showhide
Adapting Multimodal Large Language Models (MLLMs) for hour-long videos is bottlenecked by context limits. Dense visual streams saturate token budgets and exacerbate the lost-in-the-middle phenomenon. Existing heuristics, like sparse sampling or uniform pooling, blindly sacrifice fidelity by discarding decisive moments and wasting bandwidth on irrelevant backgrounds. We propose Tempo, an efficient query-aware framework compressing long videos for downstream understanding. Tempo leverages a Small Vision-Language Model (SVLM) as a local temporal compressor, casting token reduction as an early cross-modal distillation process to generate compact, intent-aligned representations in a single forward pass. To enforce strict budgets without breaking causality, we introduce Adaptive Token Allocation (ATA). Exploiting the SVLM's zero-shot relevance prior and semantic front-loading, ATA acts as a training-free $O(1)$ dynamic router. It allocates dense bandwidth to query-critical segments while compressing redundancies into minimal temporal anchors to maintain the global storyline. Extensive experiments show our 6B architecture achieves state-of-the-art performance with aggressive dynamic compression (0.5-16 tokens/frame). On the extreme-long LVBench (4101s), Tempo scores 52.3 under a strict 8K visual budget, outperforming GPT-4o and Gemini 1.5 Pro. Scaling to 2048 frames reaches 53.7. Crucially, Tempo compresses hour-long videos substantially below theoretical limits, proving true long-form video understanding relies on intent-driven efficiency rather than greedily padded context windows.
1 Introduction
Tempo addresses long-video context bottlenecks by using query-aware semantic compression and Adaptive Token Allocation to preserve relevant evidence while controlling visual tokens.
- Long videos overwhelm bounded LLM contexts, diluting attention and making sparse evidence difficult to retrieve.
- Existing sparse sampling and query-agnostic compression can miss decisive moments, blur query-critical details, and waste bandwidth on irrelevant backgrounds.
- Tempo uses an SVLM local compressor and an LLM global decoder for query-conditioned cross-modal distillation in one forward pass.
- ATA is training-free and allocates dense bandwidth to query-critical segments while retaining minimal temporal anchors for redundant contexts and causal continuity.
- Scaling to 2048 frames reaches 53.7, while standard long-video tasks favor a 4K budget and extreme-long videos benefit from larger capacities.
- 52.3 on LVBench under an 8K visual budget establishes strong performance for the compact 6B model against proprietary and open-source baselines.
2 Related Work
Prior long-video approaches extend context or reduce visual tokens, but many remain vision-driven or query-agnostic; Tempo instead integrates query-aware routing into multimodal compression.
- Short-video MLLMs commonly map sampled frames directly into LLM contexts, a strategy that becomes difficult at extended temporal horizons.
- Context-extension methods use extrapolation, parallelization, or hybrid architectures to support massive visual token sequences and mitigate memory constraints.
- Token-reduction methods exploit redundancy through hierarchical compression or temporal-spatial pooling, but remain agnostic to textual queries.
- Dual-pathway and similarity-based methods balance resolutions or frames, yet allocate resources according to visual signals rather than user intent.
- Tempo differs by unifying query-aware compression and multimodal decoding instead of relying on disjoint auxiliary routing modules.
3 TEMPO
Tempo compresses each video segment into query-conditioned memory tokens with an SVLM, then uses zero-shot relevance and semantic front-loading to allocate bounded budgets efficiently.
- Tempo targets the mismatch between massive visual streams and limited downstream visual-token capacity through early cross-modal distillation.
- Its two-level hierarchy combines an SVLM local compressor with an LLM global decoder that generates answers from temporally ordered segment memories.
- Adaptive Token Allocation (ATA): ATA uses the SVLM’s zero-shot relevance prior from the same forward pass to assign per-segment budgets between kmin and kmax.
- Each segment combines visual tokens, the query, and final learnable memory tokens, whose causal attention produces a fixed-capacity compressed representation.
- Temporal tags preserve segment identity and causal order, stabilizing long-range attribution in the global decoder.
- Adaptive Token Allocation (ATA): Min-max normalization and capacity-aware allocation convert relevance scores into integer budgets satisfying the global Bmax constraint.
- Adaptive Token Allocation (ATA): Head truncation selects Hi[1:ki], exploiting semantic front-loading as an O(1) operation without lossy spatiotemporal pooling.
4 Experiments
Tempo combines a compact SVLM–LLM architecture with progressive training and Adaptive Token Allocation to achieve strong long-video performance under strict visual-token budgets. Experiments show query-aware routing, semantic front-loading, temporal anchors, and duration-dependent scaling are central to its efficiency and robustness.
- Training and setup: Tempo uses a progressive four-stage curriculum, expanding training from 8-frame temporal perception to 384-frame long-context adaptation.Stage 3 freezes the SVLM and fine-tunes the global LLM on approximately 384K samples.
- Training and setup: Tempo evaluates long-video understanding across LongVideoBench, MLVU, Video-MME, and extreme-long LVBench against proprietary, general open-weight, and specialized MLLMs.The evaluation uses videos spanning standard long-form tasks to hour-long stress tests.
- Benchmark results: 52.7 on LVBench under a strict 4K visual budget exceeds VideoChat-Flash by 4.5 points and surpasses GPT-4o and Gemini 1.5 Pro.The reported baselines score 48.2, 30.8, and 33.1, respectively.
- Benchmark results: 67.8 on Video-MME, 57.8 on Video-MME Long, 75.6 on MLVU, and 65.1 on LongVideoBench demonstrate performance across temporal scales and tasks.LongVideoBench is reported under an 8K budget.
- Budget behavior: 52.7 versus 52.3 on LVBench and 57.8 versus 57.0 on Video-MME Long show that the 4K budget can match or exceed 8K.The paper attributes this “Less is More” pattern to filtering background distractors and mitigating lost-in-the-middle effects.
- Ablations: 65.1 versus 61.9 on LongVideoBench shows that head truncation outperforms tail truncation under Dynamic allocation.The result supports semantic front-loading in the causal SVLM.
- Ablations: 67.7 on Video-MME shows that Tempo’s SVLM relevance prior can provide routing logits during the compression pass without an extra per-segment forward pass.An external dense retriever instead requires a redundant isolated forward pass per segment.
- Ablations: 65.1 versus 63.9 on LongVideoBench shows that retaining 4 tokens per segment improves performance over dropping irrelevant segments to 0 tokens.Minimal temporal anchors preserve continuous temporal orientation and causal tracking.
5 Conclusion
Tempo addresses bounded LLM context windows by distilling long-video evidence through a local SVLM and global LLM, then routing tokens with ATA. Its results support duration-dependent, intent-driven allocation rather than uniformly expanding visual context.
- Conclusion: Tempo is a 6B-parameter framework that unifies a local SVLM and global LLM through single-pass, query-conditioned cross-modal distillation.The resulting representations are compressed and aligned with user intent.
- Conclusion: ATA performs O(1) dynamic head truncation, sending dense bandwidth to query-critical semantic beats while retaining minimal temporal anchors for the global storyline.Its routing uses the SVLM’s zero-shot relevance prior and semantic front-loading.
- Conclusion: Tempo achieves state-of-the-art performance across diverse benchmarks, including results that outperform specialized long-video MLLMs and proprietary baselines on LVBench.The conclusion frames this performance together with substantial compression below theoretical token limits.
- Conclusion: Optimal resource allocation depends on video duration: 4K efficiently denoises standard long-video tasks, whereas hour-long narratives require scaled contextual capacity.The conclusion distinguishes efficient filtering from the larger capacities needed for extreme temporal spans.
6 Discussion and Future Works
Tempo resolves hour-long video context bottlenecks through query-aware compression, while its discussion identifies three directions for improving relevance estimation, adaptive extraction, and multi-turn reuse.
- Tempo’s query-aware multimodal compression addresses context bottlenecks in hour-long videos.
- Eliciting Inherent Relevance Priors via Post-Training: Post-training could sharpen the SVLM’s zero-shot relevance judgments, although supervised fine-tuning may introduce inductive bias or overfit heuristic labels.
- Autoregressive, Reasoning-Driven Compression: Autoregressive compression could let the SVLM stop after gathering sufficient evidence, but avoiding severe inference-latency bottlenecks remains challenging.
- Hierarchical On-Demand Distillation for Multi-Turn Dialogue: Multi-turn dialogue currently requires re-extracting visual features from the entire video as user intents shift.
- Hierarchical On-Demand Distillation for Multi-Turn Dialogue: Hierarchical on-demand routing could preserve query-agnostic global context while invoking high-fidelity distillation only for targeted temporal segments.
A Statistical Analysis of Adaptive Token Allocation
The ATA analysis examines token allocations across video segments under fixed 4K and 8K budgets, finding heavy-tailed sparsity and stable query-driven allocation patterns.
- ATA analysis evaluates allocation distributions across video lengths and token budgets using 8-frame inference segments.
- Token allocations are measured per segment, with average tokens per frame obtained by dividing by 8.
- Heavy-Tailed Sparsity: 4K and 8K budgets produce strongly right-skewed, long-tailed distributions dominated by highly compressed segments.
- Budget Robustness: Allocation distributions remain stable as the global budget changes, preserving extreme sparsity for backgrounds while adding tokens selectively to query-aligned segments.
A.2 Dynamic Budget Utilization and Compression Efficiency
ATA generally consumes less than dataset-level theoretical capacity while respecting hard global limits under extreme long-context pressure.
- Theoretical capacity is compared with actual average token consumption per segment using a dataset-level average capacity line.
- Shorter videos can exceed the dataset-wide average line because their individual theoretical limits are higher without violating the global budget.
- Query-Driven Adaptability: Actual consumption clusters below dataset-level capacity on LongVideoBench, MLVU, and Video-MME, preserving bandwidth when content is irrelevant to the query.
- Hard-Boundary Reliability: Under extreme long-context pressure such as LVBench, actual consumption forms a ceiling at the theoretical limit, demonstrating reliable global-capacity enforcement.
B Qualitative Analysis of Query-Aware Allocation
Qualitative examples show that ATA changes temporal allocation according to query scope, concentrating tokens on localized evidence while maintaining broader coverage for holistic summaries.
- Figure C visualizes how ATA adjusts temporal context budgets according to semantic query requirements in LVBench examples.
- Under 4K and 8K budgets, most segments receive very low-token representations while a small fraction receives higher allocations for query-aligned content.
- Precise Action Retrieval: Localized action retrieval concentrates a sharp allocation peak on the brief target-action segment while compressing unrelated background scenes.
- Targeted Object Grounding: Object grounding assigns high capacities to segments containing semantically aligned apparatuses and suppresses manual food-preparation scenes.
- Global Video Summarization: Global summarization maintains relatively dense, fluctuating allocation across the sequence to retain distributed thematic cues.
- Overall, the SVLM performs interpretable query-aware cross-modal distillation that prioritizes semantically relevant temporal segments.
C Detailed Training Configurations
The appendix documents Tempo’s staged training setup, hyper-parameters, and budget-utilization behavior, including strict global-budget adherence on extremely long videos.
- Training Curriculum: Tempo uses a progressive four-stage training curriculum, with configurations spanning alignment, pre-training, broad SFT, and long-context SFT.The supplied passages identify the curriculum and its reproducibility-oriented configuration tables.
- Budget Utilization: On diverse-length datasets, token consumption remains below theoretical capacity, whereas LVBench reaches the theoretical ceiling and respects the global budget.The figure reports adaptability on LongVideoBench and Video-MME and strict-limit behavior on LVBench.
- Optimization Setup: 64 GPUs support Stages 1–3, while FSDP, gradient checkpointing, and bfloat16 mixed precision accommodate long sequences and the 6B architecture.Stage 0 uses 32 NVIDIA H100 GPUs; later stages use 64 GPUs.
- Context Scaling: The training schedule increases maximum sampled frames from 1 for images to 384 in Stage 3 while keeping segment capacity at 128.The listed maximum sampled frames are 1, 8, 128, and 384 across Stages 0–3.
- Strategic Freezing: During Stage 3, the local compressor, memory tokens, and projector are frozen so the global LLM learns long-range temporal dependencies without degrading cross-modal alignment.Stage 3 targets temporal extrapolation up to 12K visual tokens.
D Dataset Statistics
Tempo’s data and prompting setup transitions from spatial alignment toward spatiotemporal understanding while conditioning the local compressor on query relevance.
- Dataset Statistics: The staged data mixture transitions from spatial alignment on static images to complex spatiotemporal understanding on videos.The supplied passage describes this progression across the four training stages.
- Prompting: Tempo uses system prompts that instruct the local SVLM to store minimal query-relevant visual information and ignore irrelevant details.An explicit routing variant additionally asks for a binary relevance decision before compression.
- Zero-Shot Routing: The SVLM produces relevance decisions zero-shot at inference despite training exclusively with the standard compression prompt.The explicit routing prompt is applied only at inference, not used to train binary relevance labels.
E.3 Ablation Analysis on Relevance Source (Ablation-D)
The ablation examines explicit relevance routing and contextualizes Tempo’s budget calculations for long-video evaluation.
- Explicit Routing: Replacing the standard prompt with the explicit routing prompt generally improves performance for both the Base Model Prior and Tempo SVLM Prior.The ablation attributes the improvement to explicitly deciding relevance before context aggregation.
- Base Model Prior: The Base Model Prior still performs strongly without explicit routing, suggesting Qwen-VL models may implicitly assess visual-query relevance.This observation concerns the standard prompt using Qwen-VL’s default assistant instruction.
- Budget Calculation: For LVBench, fmax denotes the maximum sampled frames and Bmax the global visual token budget used to calculate theoretical token-per-frame limits.The cited definitions establish the two quantities used in the bound calculation.
- Budget Calculation: With fmax = 1024 and a 4K budget, the theoretical upper bound is 4 tokens per frame; with fmax = 2048 and a 12K budget, it is 6.These bounds assume the full global budget is consumed.