Source-linked AI summary
AdaReTaKe: Adaptive Redundancy Reduction to Perceive Longer for Video-language Understanding
Xiao Wang, Qingyi Si, Jianlong Wu, Shiyu Zhu, Li Cao, Liqiang Nie
TL;DR
Long-video MLLM understanding is limited by context length and by compression methods that treat visual redundancy uniformly, although redundancy varies across time and layers. AdaRETAKE is a training-free method that adaptively allocates compression ratios across both dimensions, increasing capacity from 256 to 2048 frames. Across four datasets, it improves results by 2.3% for 7B models and 2.8% for 72B models, with the authors reporting limitations inherited from underlying MLLMs.
Problem
MLLMs require hundreds of tokens per image, limiting long-video processing, while visual redundancy varies across timestamps and model layers.
Method
AdaRETAKE is a training-free method that adaptively allocates compression ratios across video time and MLLM layers.
Results
2.3% and 2.8% average improvements are achieved across four datasets for 7B and 72B models, respectively.
Takeaways & Limitations
AdaRETAKE enables integrated MLLMs to process up to 2048 frames while reducing visual redundancy within the same computational budget.
Takeaways & Limitations
AdaRETAKE may inherit underlying MLLM limitations, including factual inaccuracies, biases, and hallucinations.
Abstract
from arXiv · showhide
Multimodal Large Language Models (MLLMs) have revolutionized video understanding, yet are still limited by context length when processing long videos. Recent methods compress videos by leveraging visual redundancy uniformly, yielding promising results. Nevertheless, our quantitative analysis shows that redundancy varies significantly across time and model layers, necessitating a more flexible compression strategy. We propose AdaReTaKe, a training-free method that flexibly reduces visual redundancy by allocating compression ratios among time and layers with theoretical guarantees. Integrated into state-of-the-art MLLMs, AdaReTaKe improves processing capacity from 256 to 2048 frames while preserving critical information. Experiments on VideoMME, MLVU, LongVideoBench, and LVBench datasets demonstrate that AdaReTaKe outperforms existing methods by 2.3% and 2.8% for 7B and 72B models, respectively, with even greater improvements of 5.9% and 6.0% on the longest LVBench. Our code is available at https://github.com/SCZwangxiao/video-FlexReduc.git.
1 Introduction
Long-video MLLM understanding is constrained by visual-token context demands, motivating AdaRETAKE’s adaptive compression across time and layers. The method expands processing capacity and improves benchmark performance.
- MLLMs often need hundreds of tokens per image, limiting video processing to less than 10 minutes.
- Quantitative analysis finds visual redundancy varies across timestamps and LLM layers, challenging fixed compression ratios.
- AdaRETAKE is a training-free method with temporal- and layer-adaptive allocation modules for reducing visual redundancy.Temporal allocation adjusts compression by video chunk, while layer allocation manages KV-cache states across layers.
- AdaRETAKE expands MLLM processing capacity from 256 to 2048 frames.
- 2.3% and 2.8% average improvements are achieved across four datasets for 7B and 72B models, respectively.
2 Related Work
Long-video methods extend usable context through agents, extrapolation, parallelism, or compression, but each retains important constraints. AdaRETAKE advances compression by adapting ratios across timestamps and layers.
- Video-agent systems segment videos and retrieve or aggregate clips, but remain constrained by single-model capabilities.
- Length extrapolation extends context windows beyond training lengths, while GPU memory still limits context size.
- Q-Former compression methods condense videos with language or query tokens but may be suboptimal because Q-Former is trained from scratch.
- Token-compression methods reduce sequence length by evicting less important tokens, typically with some performance loss.
- AdaRETAKE advances MLLM token compression by adaptively adjusting compression ratios across timestamps and layers.
3 Preliminary Analysis
The analysis measures visual redundancy with heavy-hitter ratios and finds substantial unevenness across both video time and LLM layers. These patterns motivate adaptive compression allocation.
- Heavy-hitter ratios quantify redundancy using influential video tokens identified from accumulated attention.A token is a heavy-hitter when its accumulated attention exceeds 0.01 times the maximum attention value.
- Temporal redundancy is uneven, with heavy-hitter ratios varying by up to 3x within a video.
- Layer redundancy is uneven: deeper layers generally have lower heavy-hitter ratios, but substantial local fluctuations remain.
- Local minima occur at layers 2, 14, and 21, while maxima occur at layers 7 and 18.
- These distributions make monotonic deeper-layer compression assignments suboptimal and motivate adaptive ratios across timestamps and layers.
4 Methods
AdaRETAKE processes long videos through chunked, temporal-adaptive and layer-adaptive compression, assigning budgets according to redundancy and token significance while maintaining a refined context limit. Its layer-wise allocation has a theoretical near-optimal guarantee for minimizing the upper bound of token compression loss.
- General Pipeline: AdaRETAKE divides visual features into frame chunks and allocates each chunk a compression ratio under a total context-length budget.The final KV-cache sequence length is constrained to Cmax, with prompt tokens included in the budget.
- Layer-adaptive Allocation: Layer-adaptive allocation uses prompt-query attention to determine per-layer compression ratios for each video chunk.It measures token significance from head-averaged accumulated attention and layer significance from the number of highly significant tokens.
- Temporal-adaptive Allocation: Temporal-adaptive allocation estimates each chunk’s redundancy from adjacent-frame cosine similarities and assigns compression according to mean feature distances.Chunk-based processing improves allocation robustness and reduces memory overhead compared with single-frame processing.
- Token Compression: Token compression retains the top-significance visual tokens and updates each layer’s visual key-value cache after chunk prefilling.The selected token indices are obtained with ArgTopK, which returns the indices of the K largest significance values.
- Theoretical Guarantee: Theoretical analysis shows that the layer-wise allocation can achieve near-optimal minimization of the upper bound of token compression loss.The method provides a theoretical guarantee for its layer-wise budget allocation strategy.
5 Experiments
Experiments evaluate AdaRETAKE across long-video benchmarks, model scales, token-compression baselines, and component ablations. The results show consistent gains, with larger improvements on longer videos and benefits from adaptive allocation and extended context.
- Main Results: AdaRETAKE achieves average improvements of 1.2%, 2.8%, and 6.2% on VideoMME, MLVU, and LVBench, respectively.The largest gain occurs on LVBench, which has the longest average video duration among these datasets.
- Main Results: AdaRETAKE delivers consistent gains across MLLMs, including average improvements of 2.3% for 7B and 1.5% for 72B QWen2.5-VL models.These results support generalization across model sizes.
- Main Results: Compared with other token-compression methods, AdaRETAKE uses theoretically grounded budget distribution rather than heuristic or suboptimal allocation strategies.PyramidDrop’s monotonic layer-wise allocation conflicts with observed layer importance, while VL-Cache relies on heuristic dynamic allocation.
- Ablation Studies: Adaptive allocation across frames and layers improves performance by 1.0% and 0.8% on average, respectively, in ablation comparisons.These comparisons isolate the contributions of temporal- and layer-wise compression allocation.
- Ablation Studies: Extending context length produces an additional average gain of 3.4% after adaptive allocation components are applied.The ablation compares rows 4 and 5 while retaining the preceding compression strategy.
- Ablation Studies: On fine-grained temporal tasks, improvements are higher for Action Order than Needle QA and for Key Information Retrieval than Temporal Grounding.The reported average gains are 8% versus 0.8% for Action Order and Needle QA, and 11.2% versus 4.3% for Key Information Retrieval and Temporal Grounding.
6 Conclusion
AdaRETAKE is a training-free method that adaptively reduces redundancy across frames and model layers. Integrated into state-of-the-art MLLMs, it processes up to 2,048 frames and outperforms existing methods on long-video benchmarks.
- Conclusion: AdaRETAKE is a training-free method for adaptive redundancy reduction in MLLMs.It dynamically allocates compression ratios across frames and model layers.
- Conclusion: Dynamic compression allocation enables processing up to 2,048 frames while extracting valuable information within the same computational budget.The method is integrated into state-of-the-art MLLMs.
- Conclusion: AdaRETAKE outperforms existing methods on VideoMME, MLVU, LongVideoBench, and LVBench.The conclusion reports large-margin improvements across these long-video understanding benchmarks.
7 Limitations
AdaRETAKE can be integrated into most MLLMs, but it may inherit their factual inaccuracies, biases, and hallucinations.
- AdaRETAKE may inherit factual inaccuracies, biases, and hallucinations from the MLLMs into which it is integrated.
A.1 Preliminaries
The preliminaries describe autoregressive generation and the attention computations that update each layer’s key-value cache and produce its output.
- At each autoregressive step, the last token is used to predict the next token.
- In layer l, the input token is mapped into query, key, and value states during attention.
- The previous key-value cache is then updated through vector concatenation.
- The layer output y^(l) is computed from the updated attention process, with A^(l) denoting attention weights.
A.2 Attention Output after Compression
Token compression is represented by a binary preservation choice that removes dropped tokens from softmax attention, yielding a compressed layer output.
- The binary vector I^(l) specifies which tokens are preserved during compression in layer l.
- The compressed attention output uses the retained tokens’ keys and values to produce the layer output ˆy^(l).
- The derivation relies on the softmax definition to rewrite attention after token compression.
- Dropped tokens are represented by −∞ in the softmax calculation, so their compressed attention weights become zero.
A.3 Upper Bound of Token Compression Loss
The analysis bounds compression error across layers and frames token selection as a submodular optimization, enabling near-optimal minimization of the final loss bound under a token budget.
- Layer compression error D^(l) measures the L1 distance between compressed and uncompressed outputs at layer l.
- Under assumptions including 4C^(l) > 1, each layer’s compression error is bounded by ϵ^(l), with perturbations propagated through subsequent layers.
- The final compression loss L is the L1 distance between outputs before and after compression in the last layer.
- The final compression loss is bounded by ϵ_L for given token compression choices across layers.
- With a token sequence budget P, the proposed selection achieves near-optimal minimization of the upper bound on token compression loss.
- Token selection forms a monotone submodular objective with diminishing returns, so greedy selection achieves a (1 − 1/e)-approximation under a cardinality constraint.