Source-linked AI summary
Adaptive Keyframe Sampling for Long Video Understanding
Xi Tang, Jihao Qiu, Lingxi Xie, Yunjie Tian, Jianbin Jiao, Qixiang Ye
TL;DR
Long videos exceed MLLMs’ visual-token capacity, forcing systems to sample frames and risking loss of key information. The paper introduces AKS, which balances prompt relevance with video coverage when selecting a fixed number of keyframes. Across two long-video benchmarks, AKS consistently improves accuracy over baselines, while the paper notes that its optimization lacks direct supervision for keyframe selection.
Problem
Long videos contain more visual tokens than MLLMs can process, so sampling frames may lose key information and lead to incorrect answers.
Method
AKS is a plug-and-play keyframe-selection module that balances prompt-frame relevance with coverage of useful information across the video.
Results
AKS delivers consistent accuracy gains across LongVideoBench and VideoMME, including a 5.0% improvement on LongVideoBench over Qwen2VL.
Takeaways & Limitations
The results support pre-filtering visual data before MLLM processing to provide more informative visual contexts for long-video understanding.
Takeaways & Limitations
Keyframe selection lacks direct supervision, and the resulting optimization is mathematically intractable because candidate sets are exponential and MLLM confidence is difficult to estimate.
Abstract
from arXiv · showhide
Multimodal large language models (MLLMs) have enabled open-world visual understanding by injecting visual input as extra tokens into large language models (LLMs) as contexts. However, when the visual input changes from a single image to a long video, the above paradigm encounters difficulty because the vast amount of video tokens has significantly exceeded the maximal capacity of MLLMs. Therefore, existing video-based MLLMs are mostly established upon sampling a small portion of tokens from input data, which can cause key information to be lost and thus produce incorrect answers. This paper presents a simple yet effective algorithm named Adaptive Keyframe Sampling (AKS). It inserts a plug-and-play module known as keyframe selection, which aims to maximize the useful information with a fixed number of video tokens. We formulate keyframe selection as an optimization involving (1) the relevance between the keyframes and the prompt, and (2) the coverage of the keyframes over the video, and present an adaptive algorithm to approximate the best solution. Experiments on two long video understanding benchmarks validate that Adaptive Keyframe Sampling improves video QA accuracy (beyond strong baselines) upon selecting informative keyframes. Our study reveals the importance of information pre-filtering in video-based MLLMs. Code is available at https://github.com/ncTimTang/AKS.
1. Introduction
Long videos exceed MLLMs’ visual-token capacity, making keyframe quality central to video understanding. AKS addresses this by selecting frames using both prompt relevance and video coverage, improving results across long-video benchmarks.
- Long videos contain richer, more complex visual content that challenges MLLMs with information retrieval, summarization, and logical inference.
- MLLMs cannot process all visual tokens from long videos, so existing systems sample a small portion of frames.
- Uniform sampling can select irrelevant frames, lose important information, and produce incorrect video-understanding outputs.
- AKS formulates keyframe selection around prompt relevance and video coverage, using a plug-and-play module before the MLLM’s visual encoder.Relevance is estimated with a vision-language model, while coverage measures how selected frames cover useful information across the video.
- AKS achieves consistent accuracy gains across LongVideoBench and VideoMME and sets new records for 7B models with LLaVA-Video-7B.The study attributes the gains to higher-quality keyframes and identifies visual-data pre-filtering as important for long-video and higher-dimensional understanding.
2. Related Work
Related work extends multimodal language models from vision-language alignment toward video understanding. These systems motivate selecting a limited set of video frames because long videos cannot be fully represented within MLLM context capacity.
- MLLMs align visual and language data through internal adaptation or external modules to support language-guided visual understanding.
- Video-based MLLMs extend this paradigm to video understanding through systems including VideoChat, Video-ChatGPT, Video-LLaMA, Video-LLaVA, LanguageBind, and Valley.
3. Method
The method inserts a plug-and-play keyframe-selection module into video MLLMs and optimizes frame relevance to the prompt alongside temporal coverage. Its adaptive hierarchical procedure approximates this objective by balancing relevance-focused and distribution-aware sampling under a fixed keyframe budget.
- 3.1. Preliminaries: Video MLLMs select a fixed number M of keyframes because their visual-context capacity cannot accommodate all frames, making selection quality crucial.The selection function outputs an index set I with |I| = M, whose frames provide the MLLM context.
- 3.2. Principles of Keyframe Selection: AKS formulates keyframe selection as maximizing useful information through prompt-frame relevance and keyframe-set coverage over the video.The method places this selection module before the MLLM’s visual encoder and uses relevance and coverage as its two guiding principles.
- 3.2. Principles of Keyframe Selection: Relevance scores are estimated with a cheaper vision-language model, while coverage is modeled over the time axis using a balancing hyper-parameter λ.The relevance quantity measures the relationship between prompt Q and frame F_t; c(I) measures temporal coverage.
- 3.2. Principles of Keyframe Selection: Coverage uses recursive temporal binning, penalizing uneven keyframe counts across bins and stopping at a maximum depth L ≤ ⌈log2 M⌉.Each time interval is repeatedly split into two non-overlapping bins, with imbalance penalties encouraging broader temporal coverage.
- 3.3. Adaptive Keyframe Sampling: TOP selects the highest-scoring frames without coverage constraints, BIN selects high-scoring frames across bins, and UNI is BIN with constant relevance scores.TOP can concentrate frames in a small time range, whereas BIN enforces temporal distribution; UNI is the resulting uniform-sampling degeneration.
- 3.3. Adaptive Keyframe Sampling: For intermediate trade-offs, ADA hierarchically allocates keyframes across bins using relevance-score statistics and a threshold, combining the advantages of TOP and BIN.The procedure recursively judges whether to split bins further or sample high-scoring frames, while the threshold replaces explicit tuning of λ.
4. Experiments
Experiments evaluate AKS as an off-the-shelf keyframe-selection module across long-video benchmarks, baseline MLLMs, sampling strategies, scorer choices, and downstream tasks. Results show consistent accuracy gains, with performance depending on temporal coverage, prompt relevance, sampling frequency, and task characteristics.
- Comparison to the State-of-the-Art: AKS improves accuracy over Qwen2VL, LLaVA-OV, and LLaVA-Video baselines on LongVideoBench and VideoMME.The evaluation uses unchanged baseline MLLMs and inserts AKS to improve keyframe quality.
- Diagnostic on Keyframe Selection: TOP concentrates frames around high-scoring moments, BIN distributes frames across temporal bins, and ADA combines both strategies.TOP can miss information outside a small time range, whereas BIN can lose information when useful content is concentrated in one moment.
- Ablative Studies: 0.1 fps remains above baseline on LongVideoBench, reaching 57.4%, 57.9%, and 58.9% with 16, 32, and 64 frames.VideoMME shows a similar trend, with 0.25 fps described as a safe option for surpassing the baseline.
- Ablative Studies: BLIP performs better on LongVideoBench, whereas CLIP performs better on VideoMME because their training data align differently with benchmark question types.LongVideoBench questions focus more on objects, while VideoMME questions emphasize global perception.
- Generalization to Other Tasks: AKS also improves qualitative video referring and captioning results by locating keyframes used as visual contexts.The demonstrations use LLaVA-Video-7B with task-specific prompts and no answer options.
5. Conclusions
The paper introduces AKS to improve long-video understanding by selecting informative visual tokens, and finds benefits across baselines and benchmarks. Its studies support information pre-filtering as beneficial for video understanding.
- AKS uses a vision-language model to estimate frame relevance and an adaptive optimization algorithm to improve coverage of selected keyframes.
- AKS is evaluated across different baselines and benchmarks using quantitative and qualitative studies.
- The study reports that information pre-filtering brings considerable benefit to video understanding and motivates further investigation.
A. Details of the ADA Algorithm
The ADA implementation pre-processes one-frame-per-second video frames with their questions, then stores frame–question matching scores for adaptive selection.
- Video frames sampled at 1 frame per second are paired with corresponding questions and passed through a vision-language scorer.
- The resulting frame–question scores are stored in a list called matching score for subsequent processing.
B. More Visualization Results
The appendix presents ADA’s recursive frame-selection procedure and additional AKS visualizations. The algorithm selects high-scoring frames while allocating selections across adaptively identified segments.
- More Visualization Results: Figure 8 provides additional AKS video-understanding examples using LLaVA-Video-7B, Qwen2-VL-7B, and LLaVA-OV-7B baselines.
- A. Details of the ADA Algorithm: ADA takes matching scores, recursion parameters, a threshold, and a target frame count as input, returning selected frame indices.
- A. Details of the ADA Algorithm: ADA computes the mean score and the mean of the top M scores, using their difference to decide whether a segment should be split.
- A. Details of the ADA Algorithm: When the splitting condition is met, ADA divides matching scores into two bins around the center and appends the resulting subsets to new scores.
- A. Details of the ADA Algorithm: The recursive procedure merges split and deeper segments before returning the selected frames.
- A. Details of the ADA Algorithm: SelectFrames assigns each segment floor(M × length(segment)/total length) frames and chooses the highest-scoring frame indices within it.