PaperlayerView exact revision on arXiv

Source-linked AI summary

Q-Frame: Query-aware Frame Selection and Multi-Resolution Adaptation for Video-LLMs

Shaojie Zhang, Jiahui Yang, Jianqin Yin, Zhenbo Luo, Jian Luan

arXiv:2506.22139v3cs.CV

TL;DR

Video-LLMs struggle with long videos because frame volume, temporal redundancy, query-agnostic sampling, and limited context make relevant spatiotemporal information difficult to preserve. Q-Frame uses training-free CLIP-guided query-aware selection and multi-resolution adaptation, and experiments report superior performance across MLVU, LongVideoBench, and Video-MME. The method remains limited by pretrained-model dependence, absent explicit temporal modeling, a fixed token budget, and benchmark-centered evaluation.

  • Problem

    Video-LLMs face excessive frame volume, temporal redundancy, restricted context, and uniform sampling that is sparse, static, and query-agnostic.

  • Method

    Q-Frame is a training-free, plug-and-play framework using CLIP-like cross-modal relevance, Gumbel-Max frame selection, and query-based multi-resolution assignment.

  • Results

    Q-Frame outperforms traditional uniform sampling and keyframe selection techniques across MLVU, LongVideoBench, and Video-MME; its QFS-and-MRA combination reaches 58.4% accuracy in ablation.

  • Takeaways & Limitations

    Q-Frame provides a model-agnostic, scalable approach for long-form video understanding by focusing processing on query-relevant visual content.

  • Takeaways & Limitations

    Q-Frame depends on pretrained models, lacks explicit temporal modeling, operates within a fixed token budget, and is evaluated primarily on benchmarks.

Abstract

from arXiv · show

Multimodal Large Language Models (MLLMs) have demonstrated significant success in visual understanding tasks. However, challenges persist in adapting these models for video comprehension due to the large volume of data and temporal complexity. Existing Video-LLMs using uniform frame sampling often struggle to capture the query-related crucial spatiotemporal clues of videos effectively. In this paper, we introduce Q-Frame, a novel approach for adaptive frame selection and multi-resolution scaling tailored to the video's content and the specific query. Q-Frame employs a training-free, plug-and-play strategy generated by a text-image matching network like CLIP, utilizing the Gumbel-Max trick for efficient frame selection. Q-Frame allows Video-LLMs to process more frames without exceeding computational limits, thereby preserving critical temporal and spatial information. We demonstrate Q-Frame's effectiveness through extensive experiments on benchmark datasets, including MLVU, LongVideoBench, and Video-MME, illustrating its superiority over existing methods and its applicability across various video understanding tasks.

1. Introduction

Video understanding is constrained by massive, redundant frame sequences and limited MLLM context, while uniform sampling is sparse, query-agnostic, and inefficient. Q-Frame addresses this with query-adaptive selection and resolution scaling without additional training, and is evaluated across three benchmarks.

  • Motivation: A 3-minute video at 24 fps contains approximately 4,320 frames, exceeding the practical context limits of current Video-LLMs.VideoLLaMA2 supports 2,000 tokens, while VILA-V1.5 supports roughly 4,000 tokens.
  • Motivation: Uniform sampling can disrupt temporal continuity, omit critical transitions and subtle details, and treat all frames equally regardless of the query.These limitations particularly affect fine-grained tasks such as object counting and temporal grounding.
  • Q-Frame: Q-Frame dynamically selects frames and adjusts their resolutions according to video content and the textual query.Its selection distribution is parameterized by a text-image matching model such as CLIP, with the Gumbel-Max trick producing a frame combination without additional training.
  • Q-Frame: Q-Frame enables Video-LLMs to process more informative frames within the computational budget while preserving visual details that uniform sampling may lose.The approach is model-agnostic and plug-and-play across open-source and closed-source Video-LLMs.
  • Evaluation: The framework combines query-aware frame selection with multi-resolution scaling and is evaluated on MLVU, LongVideoBench, and Video-MME.The paper reports superiority over previous methods and practical usability across video understanding tasks.

2. Related Work

Related work spans Video-LLMs built from instruction data, efficient feature compression, adaptive keyframe selection, and variable-resolution visual processing. Existing approaches still face temporal fragmentation and rigid resolution constraints.

  • Video-LLMs: Video-LLMs use synthesized video-instruction data to improve fine-grained action understanding and narrative tracking.Examples include VideoChat, LLaVA-Hound, InternVideo2, ShareGPT4Video, Video-LLaVA, and LLaVA-Video.
  • Video-LLMs: Other Video-LLMs develop efficient feature-compression methods to reduce the visual information processed by the language model.The supplied passage introduces this category but does not enumerate its methods completely.
  • Keyframe Selection: Keyframe selection methods range from reinforcement-learning and motion-guided sampling to query-conditioned retrieval and loss-driven combinatorial optimization.Examples include AdaFrame, MGSampler, PMGSampler, KeyVideoLLM, and Frame-Voyager.
  • Keyframe Selection: Most Video-LLMs use query-agnostic uniform sampling, while newer methods seek query-aware selection but continue to face temporal fragmentation and rigid resolution constraints.The related-work discussion motivates selection and representation strategies that account for both temporal structure and resolution flexibility.
  • Any-Resolution Processing: Native Dynamic Resolution frameworks adapt visual-token generation across varying input resolutions, extending beyond fixed-resolution visual encoders and fixed-grid patching.These approaches address native-resolution preservation but retain structural limitations described in the passage.

3. Method

Q-Frame retrieves query-relevant candidate frames, samples a query-aware subset with Gumbel-Max perturbations, and assigns resolutions according to relevance. This combines semantic selection with token-budget-aware visual processing.

  • Pipeline: The Video-LLM pipeline takes a video V and textual query q as inputs and produces an answer a.The video is represented as a sequence of D frames, while uniformly sampled candidates form a shorter sequence of T frames.
  • Framework: Q-Frame combines Cross-modal Query Retrieval, Query-Aware Frame Selection, and Multi-Resolution Adaptation.Together, these modules select query-relevant frame sequences and scale their resolutions while maintaining a fixed computational budget.
  • Cross-modal Query Retrieval: Uniformly sampled candidate frames are embedded with the textual query by a pretrained CLIP-like model in a shared semantic space.The method then computes frame relevance using the inner product between query and frame embeddings.
  • Query-aware Frame Selection: Temperature-scaled softmax converts frame-query matching intensities into selection probabilities, with τ controlling distribution sharpness.The resulting probability distribution determines how likely each candidate frame is to be selected.
  • Query-aware Frame Selection: Gumbel noise perturbs log-probabilities, after which frames ranked among the top K are selected.This sampling strategy supports diversified selection and balances exploration with exploitation.
  • Multi-Resolution Adaptation: Multi-Resolution Adaptation assigns higher resolution to more relevant frames and lower resolution to less important frames using relevance thresholds K, M, and N.High-resolution frames preserve fine details, while lower-resolution frames reduce visual-token costs and allow more frames within the same budget.
  • Multi-Resolution Adaptation: The resolution levels are ordered as r(1) = 4r(2) = 16r(3), corresponding to low, medium, and high resolutions, respectively.The method assigns r(3), r(2), or r(1) according to whether a frame falls in the high-, medium-, or low-relevance index set.
  • Inference: During inference, higher-resolution frames produce more visual tokens and lower-resolution frames produce fewer tokens before textual and visual tokens are processed by the Video-LLM.The MRA component is not applicable to every model because Video-LLM preprocessing strategies differ.

4. Experiments

Q-Frame is evaluated across three long-video benchmarks and multiple Video-LLM backbones, with experiments covering overall performance, video duration, task types, and component ablations. Results generally show gains from query-aware selection and multi-resolution adaptation, while short-video and unusual API-model cases require qualification.

  • Experimental Setup: Q-Frame is evaluated with VILA-V1.5, Qwen2-VL, and GPT-4o on MLVU, LongVideoBench, and Video-MME.The setup uses 128 uniformly sampled candidate frames and selects 8 frames, or an equivalent token budget through multi-resolution assignment.
  • Comparison with SOTA Methods: Across fixed-frame and fixed-token settings, Q-Frame consistently outperforms uniform sampling across the three evaluated benchmarks.Qwen2-VL with Q-Frame achieves the highest overall performance on MLVU, while GPT-4o achieves the best result on the other benchmarks.
  • Performance Across Video Tasks: Q-Frame improves performance across most video durations, with particularly strong gains on longer videos where uniform-sampling baselines decline as duration increases.VILA-V1.5 shows a slight decrease in the (8s, 15] range, where uniform sampling already provides about one frame per second.
  • Performance Across Video Tasks: Q-Frame improves Reasoning, Recognition, and Counting Problem performance in Video-MME by selecting semantically relevant frames and adapting their resolutions.The figure covers six tasks: Reasoning, Recognition, Perception, Counting Problem, OCR Problems, and Information Synopsis.
  • Ablation Study: 58.4% is achieved when QFS and MRA are combined, exceeding direct CLIP matching at 56.0%, QFS at 57.6%, and MRA alone at 52.58%.The ablation identifies both query-aware frame selection and multi-resolution adaptation as essential components.
  • Ablation Study: 58.4% is achieved with high-, medium-, and low-resolution frames, compared with 57.6% using only high resolution, 55.9% using only medium resolution, and 49.0% using only low resolution.The optimal frame allocation is 4 high-resolution, 8 medium-resolution, and 32 low-resolution frames; increasing low-resolution frames to 48 reduces accuracy to 57.4%.

5. Conclusion

Q-Frame is presented as a training-free, efficient approach that combines query-aware frame selection with multi-resolution adaptation for long-form video understanding. Experiments across MLVU, LongVideoBench, and Video-MME report improved performance and stronger results on fine-grained temporal and spatial tasks.

  • Q-Frame dynamically selects query-relevant frame sequences and adjusts frame resolutions while minimizing computational costs.
  • The method outperforms traditional uniform sampling and keyframe selection techniques on MLVU, LongVideoBench, and Video-MME.
  • Q-Frame’s query-aware selection and multi-resolution adaptation improve frame relevance and resolution optimization.
  • Its model-agnostic, plug-and-play design provides a scalable solution for integrating Q-Frame with Video-LLMs.

A. Limitations

Q-Frame depends on pre-trained models, lacks explicit temporal modeling, and operates within a fixed token budget. Its benchmark-focused evaluation leaves real-world validation as an open need.

  • Q-Frame depends on pre-trained models and lacks explicit temporal modeling.
  • The method operates within a fixed token budget, limiting its processing setting.
  • The evaluation primarily relies on benchmarks, motivating validation in real-world applications.

B.1. Dataset Details

The paper evaluates Q-Frame using long-video benchmarks covering diverse durations, domains, genres, tasks, and question-answer formats. The supplied passages also document GPT-4o configuration and repeated MLVU result verification.

  • LongVideoBench: LongVideoBench contains 3,763 web-collected videos up to one hour, with 6,678 annotated multiple-choice questions across 17 categories.The paper uses a validation subset without subtitles containing 1,337 question-answer pairs and averaging 12 minutes per video.
  • MLVU: MLVU contains 2,593 tasks across nine categories, with videos averaging 12 minutes and spanning movies, surveillance footage, and cartoons.
  • Video-MME: Video-MME comprises 900 expert-annotated videos across six visual domains, with durations from 11 seconds to one hour and 2,700 question-answer pairs.
  • GPT-4o evaluation: The appendix includes GPT-4o parameter configuration and comparisons with and without Q-Frame on MLVU.
  • GPT-4o evaluation: Multiple experiments were conducted to verify unusual GPT-4o results on MLVU, with underlined values adopted in the manuscript.

B.3. Detailed experimental results on subtasks

Q-Frame improves performance across most evaluated subtasks, with especially notable gains on query-dependent, temporal, object-centric, perception, recognition, and OCR tasks. Some results remain task-dependent, including minor drops and variation in retrieval-based subtasks.

  • Q-Frame consistently enhances performance across most subtasks, particularly query-dependent tasks requiring adaptive frame selection.
  • LongVideoBench: On LongVideoBench, Q-Frame improves most categories, with notable gains in temporal and object-centric tasks.Some subtasks show minor performance drops.
  • MLVU: On MLVU, Q-Frame significantly boosts nonuniform video understanding tasks such as NQA and PQA.Performance varies in some retrieval-based tasks, likely reflecting the pre-trained vision-language model guiding frame selection.
  • Video-MME: On Video-MME, Q-Frame significantly improves perception, recognition, and OCR, while also enhancing reasoning and counting accuracy.Performance gains tend to be task-dependent.
  • The strongest overall impact occurs in tasks requiring fine-grained temporal and spatial understanding.

B.4. Additional Ablation Study

The ablations examine the vision-language backbone, temperature, candidate-frame count, and inference overhead in Q-Frame. Long-CLIP and moderate temperature settings perform best, while candidate-frame increases remain stable and preprocessing adds limited cost relative to Video-LLM processing.

  • CLIP-like Model: Long-CLIP achieves the highest CQR accuracy at 58.4%, exceeding standard CLIP at 57.8% and SigLIP at 57.9%.The baseline without a dedicated retrieval module reaches 53.5%.
  • Temperature Parameter: A temperature of τ = 1.2 produces the maximum QFS accuracy of 58.4%, while τ = 0.5 reaches 57.6%.The passage attributes the difference to the balance between exploration and exploitation.
  • Candidate Frames: With eight sampled frames, Q-Frame achieves stable improvement across different candidate-frame counts.The candidate-frame count is set to T = 128 for comparability with FRAME-VOYAGER.
  • Overhead: Q-Frame adds inference-time preprocessing for embeddings and sampling, but candidate embeddings can be computed in parallel and MRA adds almost negligible burden as candidates increase.Compared with Video-LLM processing time, Q-Frame can introduce about five times more effective frames with minimal time impact.

B.5. Additional Case Analysis

Case analyses show that Q-Frame can identify query-relevant keyframes for visual and action questions. However, its sparse semantic selection does not preserve the sequential and causal structure required for some temporal reasoning tasks.

  • Case Analysis: Q-Frame more accurately identifies frames containing answers, providing Video-LLMs with a more reliable information source than uniform sampling.The comparison visualization reports this outcome across Figures 6, 7, and 8.
  • Case Analysis: The Gumbel-Max trick makes QFS sampling continue over time, partially addressing the sparsity problem caused by sampling.This mechanism is reported alongside the case-analysis comparison.
  • Bad Case Analysis: Neither Q-Frame nor uniform sampling correctly handles the illustrated temporal reasoning task.The failure is attributed to Q-Frame’s sparse semantic selection, which can omit transitional moments and event boundaries needed to reconstruct temporal logic.
  • Case Analysis: Q-Frame captures keyframes for car-color, dunking-player, and subway-interaction questions, enabling Video-LLMs to answer accordingly.The listed cases concern object attributes, an action, and interacting people.
Loading 2506.22139v3…