Source-linked AI summary
M-LLM Based Video Frame Selection for Efficient Video Understanding
Kai Hu, Feng Gao, Xiaohan Nie, Peng Zhou, Son Tran, Tal Neiman, Lingyun Wang, Mubarak Shah, Raffay Hamid, Bing Yin, Trishul Chilimbi
TL;DR
Uniform frame sampling can omit crucial question-relevant context in long videos while operating under limited M-LLM context. The paper introduces a lightweight, question-aware M-LLM frame selector trained with spatial and temporal pseudo-labels, and reports improved video QA across medium- and long-context benchmarks.
Problem
Uniform sampling may select irrelevant or redundant frames and ignore important ones, limiting visual information available for video QA.
Method
A lightweight question-aware frame selector uses spatial M-LLM scores and temporal LLM caption-based pseudo-labels before a frozen downstream video M-LLM answers questions.
Results
The selector improves video QA performance across medium-length and long-video benchmarks and works with multiple downstream M-LLMs without additional downstream fine-tuning.
Takeaways & Limitations
Question-aware adaptive frame selection can provide an efficient plug-and-play alternative to uniform sampling for video question answering.
Takeaways & Limitations
The selector assumes that far fewer visual tokens per frame suffice to determine frame importance than to capture visual details for video QA.
Abstract
from arXiv · showhide
Recent advances in Multi-Modal Large Language Models (M-LLMs) show promising results in video reasoning. Popular Multi-Modal Large Language Model (M-LLM) frameworks usually apply naive uniform sampling to reduce the number of video frames that are fed into an M-LLM, particularly for long context videos. However, it could lose crucial context in certain periods of a video, so that the downstream M-LLM may not have sufficient visual information to answer a question. To attack this pain point, we propose a light-weight M-LLM -based frame selection method that adaptively select frames that are more relevant to users' queries. In order to train the proposed frame selector, we introduce two supervision signals (i) Spatial signal, where single frame importance score by prompting a M-LLM; (ii) Temporal signal, in which multiple frames selection by prompting Large Language Model (LLM) using the captions of all frame candidates. The selected frames are then digested by a frozen downstream video M-LLM for visual reasoning and question answering. Empirical results show that the proposed M-LLM video frame selector improves the performances various downstream video Large Language Model (video-LLM) across medium (ActivityNet, NExT-QA) and long (EgoSchema, LongVideoBench) context video question answering benchmarks.
1. Introduction
Video QA must balance understanding long sequences with limited M-LLM context, while uniform sampling can omit question-relevant frames. The paper proposes a lightweight, question-aware selector trained with spatial and temporal pseudo-labels and usable with multiple downstream M-LLMs.
- Long video QA requires models to comprehend many frames, creating a critical video-context-length challenge.
- Uniform sampling covers the time axis but may select irrelevant or redundant frames while ignoring important ones.This can leave downstream models with insufficient visual information for particular questions.
- The proposed selector adaptively selects question-relevant frames, aiming to use fewer frames without sacrificing video-understanding quality.It uses a fine-tuned LLM, compressed per-frame tokens, and M-LLM multimodal relevance estimation.
- Spatial pseudo-labeling scores individual frames with an M-LLM, while temporal pseudo-labeling identifies top-k relevant frames from their captions.The two strategies address limited frame-level annotations and temporal-context constraints.
- The plug-and-play selector requires no downstream M-LLM fine-tuning and improves video QA across medium- and long-context benchmarks.The reported benchmarks include ActivityNet, NExT-QA, EgoSchema, and VideoMME.
2. Related Work
Prior video-LLM frame-selection methods either assess frames independently, incur high inference costs, or lack text awareness. The proposed method addresses these limitations by producing question-aware importance scores for all frames in one pass using compressed visual tokens.
- SeViLA and related methods score frames with an M-LLM, but evaluate frames independently without temporal context and become more expensive as videos lengthen.
- The proposed method outputs importance scores for all frames in a single pass with compressed visual tokens, reducing computational cost while enabling temporal reasoning.
- Koala and ViLA lack text awareness during inference, unlike the proposed question-aware frame-selection approach.
3. Method
The method replaces question-agnostic uniform sampling with a lightweight, query-aware frame selector that scores candidate frames and chooses informative, temporally diverse inputs. It trains the selector using spatial and temporal pseudo labels, then applies greedy NMS sampling before downstream video-LLM reasoning.
- Motivation: Uniform sampling can be inefficient or omit crucial short-duration actions, motivating question-dependent adaptive frame selection.Dense sampling increases the number of visual tokens, while sparse sampling risks missing relevant events.
- Frame Selector: The selector consumes sampled frames and a question, then outputs an n-dimensional vector whose elements represent frame importance scores.A learnable score query aggregates visual and text information, and an MLP converts its hidden representation into the score vector.
- Frame Selection: Greedy NMS selects the highest-scoring frames sequentially and suppresses nearby frames to avoid redundant information.After each selection, neighboring indices within the configured gap are excluded, and the selected indices are sorted.
- Frame Selector: The method reduces per-frame tokens through spatial pooling so the lightweight selector can ingest many candidate frames efficiently.The design keeps dense initial sampling while reducing each frame to substantially fewer visual tokens before selector inference.
- Pseudo Labels: Spatial pseudo labels estimate each frame’s relevance independently by prompting an M-LLM and normalizing its True-versus-False probabilities.The score is computed as p_True/(p_True + p_False), with scores normalized by the maximum frame score.
4. Experiments
The experiments evaluate the selector across multiple M-LLMs, benchmarks, frame-selection strategies, frame budgets, and selector configurations. Results show that question-aware selection improves efficiency and often preserves or improves video QA performance, including on long videos.
- Experimental Setup: The selector is evaluated as a plug-and-play component without additional downstream M-LLM fine-tuning.The experiments examine whether selecting better frames improves well-trained M-LLMs while keeping their parameters unchanged.
- Experimental Setup: The evaluation compares uniform sampling with selector-based frame selection across ActivityNet-QA, NExT-QA, EgoSchema, VideoMME, and LongVideoBench.The study uses video and multi-image M-LLMs, including PLLaVA, LLaVA-NeXT-video, Idefics, and Qwen2-VL.
- Ablation Studies: 55.5 ANet-QA and 63.9 NExT-QA are achieved by spatial-and-temporal pseudo-label selection, versus 53.5 and 62.4 with uniform sampling.The trained selector reaches 55.1 ANet-QA and 63.4 NExT-QA, while pseudo-label selection is computationally expensive because it densely prompts an M-LLM.
- Ablation Studies: A selector using n frames can achieve similar video QA performance to uniform sampling with 2n frames, while adding only modest frame-selection latency.For example, the 128 →4 configuration outperforms 8-frame uniform sampling with faster inference speed.
- Long-Context Evaluation: On LongVideoBench, selector sampling with n frames surpasses uniform sampling with 2n frames for both LLaVA-NeXT-Video 34B and Qwen2-VL 7B.LongVideoBench videos average 473 seconds, providing a long-context evaluation setting.
- Qualitative Results: Qualitative examples show that the selector identifies frames containing answers when questions require reasoning over two events.The authors also report QVHighlights results of 43.9% R1@5 and 32.3% R1@7, below SeViLa’s 54.5% and 36.5% on those metrics.
5. Conclusion
The paper presents a lightweight, question-aware M-LLM frame selector for efficient video QA and evaluates it across medium- and long-video benchmarks.
- The selector takes dense video frames and a question as input, then selects the most relevant frames for downstream question answering.
- Spatial and temporal pseudo-labeling address limited public annotations for frame-level importance during selector training.
- Experiments cover ActivityNet QA, NExT-QA, EgoSchema, LongVideoBench, and VideoMME, demonstrating the method’s effectiveness across medium- and long-video QA.
A. Prompt for Pseudo Label Generation
The appendix describes prompt-based procedures for generating spatial and temporal pseudo-labels and provides tables for their prompts and related evaluations.
- A. Prompt for Pseudo Label Generation: Spatial pseudo-labels are generated by uniformly sampling n = 128 frames and scoring each frame independently with an M-LLM prompt.The score uses the logit associated with “True” or “False” after “Evaluation:”.
- A. Prompt for Pseudo Label Generation: Tables 10 and 11 provide the spatial and temporal pseudo-label prompt templates, while Table 12 evaluates spatial labels from different prompting M-LLMs.
- A. Prompt for Pseudo Label Generation: Temporal pseudo-labels are generated by captioning n = 128 uniformly sampled frames and prompting for the indexes of the most helpful frames.
- A. Prompt for Pseudo Label Generation: Table 13 reports performance when selecting different numbers of frames on EgoSchema and LongVideoBench with LLaVA-NeXT-Video 34B.
B. Additional Results
Additional experiments examine prompting-model choice and the size of the frame candidate pool, finding that stronger prompting models and larger pools improve selection-related results.
- B. Additional Results: A stronger prompting M-LLM produces higher-quality spatial pseudo-labels in evaluations with LLaVA-Next-Video 7B on ActivityNet and NExT-QA.
- B. Additional Results: The method samples 128 candidate frames before selection, compared with the 16–32 frames typically sampled by existing frame-selection methods.
- B. Additional Results: On EgoSchema and LongVideoBench, selecting 4 frames from 128 candidates improves QA performance significantly compared with selecting 4 frames from 16 candidates.
C. More Visualization Results
The appendix contains additional visualizations of frame-selection results, including zoomed-in views and examples comparing selected frames with uniform sampling.
- C. More Visualization Results: Figures 5 and 6 provide zoomed-in views of Figure 4’s frame-selection results.
- C. More Visualization Results: Figures 7 and 8 present additional visualizations of the frame-selection results.
- C. More Visualization Results: The appendix includes an example showing four frames obtained through uniform sampling.