PaperlayerView exact revision on arXiv

Source-linked AI summary

HFS: Holistic Query-Aware Frame Selection for Efficient Video Understanding

Yiqing Yang, Yun Li, Daiqing Qi, Lehan Yang, Tianlong Wang, Wenhao Zhang, Sheng Li, Kin-man Lam

arXiv:2512.11534v3cs.CVcs.CLcs.MM

TL;DR

Long videos contain dense, redundant frames that can overwhelm model context, while existing frame-selection methods do not fully capture set-level interactions or adapt criteria to downstream tasks. HFS combines task-conditioned query extraction, differentiable set-quality optimization, and student-teacher mutual learning. It outperforms existing approaches across four benchmarks, with reported gains including up to 4.0 percentage points on MLVU M-AVG.

  • Problem

    Long videos can overflow model context, while frame-selection methods may ignore inter-frame dependencies, use fixed set criteria, or rely on offline supervision detached from task goals.

    Source: §1 · pp. 1–2

  • Method

    HFS uses CoT-conditioned task-specific queries, a differentiable objective for relevance, coverage, and redundancy, and student-teacher mutual learning to train frame selection with downstream reasoning.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    HFS outperforms existing approaches across four benchmarks, improving MLVU M-AVG by up to 4.0 percentage points and Video-MME overall accuracy by 2.6 percentage points over the baseline.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    Reported gains hold across both teacher backbones and all four benchmarks.

    Source: §5 · p. 8

    When questions are ambiguous or have minimal semantic content, latent query vectors may fail to capture sufficiently discriminative task intent.

    Source: F Limitations and Future Work · p. 16

Abstract

from arXiv · show

Key frame selection is essentially a set-level optimization problem: the quality of the selected subset depends on the interactions among frames, rather than the score of any single frame. Existing methods generally exhibit three major limitations. Point-wise methods score each frame independently and ignore inter-frame dependencies. Although the training-free set-level methods explicitly model the inter-frame relationships, their selection criteria are fixed and cannot be adapted through downstream task feedback. Learnable methods can leverage data-driven training; however, they lack an explicit, differentiable set-quality objective and rely on offline-generated supervision signals. To address these limitations, we propose an end-to-end trainable and task-adaptive framework for frame selection. A Chain-of-Thought prompt conditions a Small Language Model (SLM) to extract task-specific latent query vectors, which are combined with multimodal features to enable dynamic, query-aware frame scoring. We further formulate a continuous set-level objective function that jointly accounts for relevance, coverage, and redundancy, enabling differentiable set-level optimization via Gumbel-TopK for selecting optimal frame combinations. Finally, we employ a student-teacher mutual learning strategy, in which the student selector (SLM) and teacher reasoner (MLLM) are trained to align their frame-importance distributions via KL divergence. Combined with cross-entropy loss, this design enables fully end-to-end optimization, eliminating reliance on static pseudo-labels. Experiments across multiple benchmarks, including Video-MME, LongVideoBench, MLVU, and NExT-QA, demonstrate that our method significantly outperforms existing frame-selection approaches.

1 Introduction

Long videos can overwhelm model context with redundant frames, making informative frame selection essential. HFS addresses shortcomings in independent scoring, fixed set-level criteria, and offline supervision through task-aware queries, differentiable set optimization, and student-teacher mutual learning.

1 Introduction

  • Video’s dense temporal information and redundant frames can overflow model context windows, making informative key-frame subsets important for video understanding.Even moderately sized clips may exceed model processing capabilities.Source: §1 · p. 1
  • Frame selection is a set-level problem, but existing approaches score frames independently, use fixed set criteria, or rely on offline supervision disconnected from task goals.A relevance-based method can select redundant frames while omitting the decisive event.Source: §1 · p. 2
  • HFS combines a CoT-conditioned SLM selector, a differentiable coverage-based set-quality objective, and student-teacher mutual learning to jointly optimize frame selection and downstream reasoning.The student and teacher align frame-importance distributions through KL divergence, alongside downstream cross-entropy loss.Source: §1 · p. 2
  • CoT-conditioned latent query extraction produces task-specific vectors that guide frame scoring, while mutual learning removes dependence on static offline supervision.The SLM integrates the query vectors with multimodal features through a LoRA adapter.Source: §1 · p. 2
  • HFS significantly outperforms existing approaches across four benchmarks, improving MLVU M-AVG by up to 4.0 percentage points and Video-MME overall accuracy by 2.6 percentage points over the baseline.It also surpasses the strongest training-free method by up to 2.0 points.Source: §1 · p. 2

2 Related Work

Video-understanding research spans long-form processing strategies and adaptive frame selection. Existing selection methods include query-aware and set-level approaches, but none jointly optimizes a differentiable set-quality objective with the reasoner.

2 Related Work

  • Long-form video understanding has been approached with sparse memory, temporal modeling, context-length extension, and adaptive compression.Other work explores fine-grained spatial understanding and regional-level video comprehension.Source: §2 · p. 3
  • Frame-selection methods include relevance-and-coverage balancing, spatial-temporal pseudo-labels, variable frame counts, hierarchical clustering, and query-aware selection.Examples include AKS [32], M-LLM [11], FFS, VideoTree [38], Frame-Voyager [45], and Q-Frame [48].Source: §2 · p. 3
  • No reviewed method jointly optimizes a differentiable set-quality objective with the reasoner; prior approaches use independent scoring, frozen set criteria, or detached offline supervision.MDP3 [31] uses determinantal point processes for training-free list-wise frame selection.Source: §2 · p. 3

3 Holistic Query-Aware Frame Selection

HFS frames selection as end-to-end set-level optimization: a student extracts task-aware queries and selects frames, while a teacher supports downstream reasoning and mutual learning. Its differentiable objective balances relevance, coverage, and temporal redundancy, with task supervision and distribution alignment guiding training.

3.1 Overview

  • The student scores video frames using visual and question context, selects a representative subset, and passes it with the question to a teacher reasoner.The teacher also produces a frame-importance distribution for alignment with the student during training.Source: §3.1 · p. 3

3.2 Adaptive Query Generation

  • A CoT prompt conditions the SLM to extract task-specific latent queries from the question and answer options, with uniformly sampled hidden states intended to capture diverse semantic aspects.The prompt instructs the model to analyze the question and candidate options and identify key concepts that distinguish them.Source: §3.2 · p. 3
  • The SLM fuses video and text embeddings with the task queries through an aggregator token, whose final hidden state becomes a context-aware query vector.That vector is concatenated with each frame embedding to produce context-dependent relevance scores.Source: §3.2 · p. 4
  • Gumbel-TopK relaxes discrete top-K selection into a differentiable soft mask, while the set objective acts as a training regularizer whose learnable relevance and redundancy scores receive downstream gradients.The paper describes the objective as a differentiable regularizer rather than an independent inference-time objective.Source: §3.2 · pp. 4, 6
  • The continuous set objective combines task relevance and coverage while penalizing temporal redundancy, with proximity measured between normalized indices of uniformly sampled frames.The coverage term encourages at least one selected frame to have high relevance; the temporal kernel assigns greater redundancy to nearby frames.Source: §3.2 · p. 5

3.3 End-to-End Training with Mutual Learning

  • Training combines downstream answer cross-entropy, KL alignment between teacher and student frame-importance distributions, the set objective, and query separation loss.The KL loss sends gradients to both teacher and student scorers, while the set objective is maximized through its negative contribution to total loss.Source: §3.3 · p. 6

4 Experiments

HFS is evaluated on four benchmarks against frame-selection baselines, with ablations testing its components, query count, set objective, redundancy kernel, supervision strategy, and efficiency. Across the main results, HFS improves benchmark performance; ablations also show gains from the full framework and a latency advantage over uniform sampling at a lower frame budget.

4.1 Experimental Setup

  • HFS is evaluated on Video-MME, MLVU, LongVideoBench, and NExT-QA against frame-selection methods under identical settings, using specified training datasets and teacher models.The setup uses 250K VideoChat2-IT and 196K LLaVA-Video-178K samples; Qwen2.5-1.5B-Instruct is the student, with Qwen2.5-VL-7B-Instruct and InternVL3-8B-hf as teachers.Source: §4.1 · pp. 6–7

4.2 Main Results

  • InternVL3 with HFS leads on Video-MME’s short and long subsets, raises MLVU M-AVG by 4.0 points, and reaches 60.2% on LongVideoBench; it scores 83.1% on NExT-QA, surpassing state-of-the-art methods.On Video-MME, InternVL3 with HFS scores 63.3 versus VideoTree’s 62.6 without subtitles; on LongVideoBench, it scores 60.2 versus 59.3.Source: §4.2 · p. 7

4.3 Ablation Studies

  • HFS with all components outperforms the end-to-end LCE-only baseline across three benchmarks; among query counts, Video-MME peaks at 66.1% with K=3.The baseline uses static queries and top-k selection from MLP_s; performance declines slightly to 65.9% at K=4, so K=3 is used thereafter.Source: §4.3 · pp. 7–8
  • Adding relevance, then coverage, then redundancy to the set objective improves MLVU performance to 50.0%, while alternative semantic redundancy kernels do not outperform the temporal prior.Coverage provides the greatest marginal gain; four weight configurations vary Video-MME accuracy by at most 1.4 points.Source: Table 5 · p. 7, §4.3 · p. 8, Table 6 · p. 8
  • End-to-end cross-entropy training without distillation beats static pseudo-label supervision, while adding KL distillation yields state-of-the-art results on NExT-QA and Video-MME.The comparison covers offline Qwen2.5-VL-7B-Instruct pseudo-labels, downstream-task cross-entropy alone, and the full online distillation framework.Source: §4.3 · p. 8, Table 7 · p. 8
  • HFS with k_sel=8 exceeds uniform sampling with k_sel=16 while reducing latency from 0.59s to 0.49s per sample; at k_sel=16, it runs 2.4–3.5 times faster than other query-aware selectors.The selector scores all candidates in one pass, so its overhead is constant in k_sel; Figure 3 compares accuracy and latency across selection counts.Source: Figure 3 · p. 8, §4.3 · p. 8, Table 8 · p. 8

5 Conclusion

HFS combines task-adaptive latent queries, a set-level objective, and mutual learning to train a video frame selector jointly with its reasoner. The paper reports consistent gains across benchmarks and describes a fixed, automatically applied prompt design that requires no human annotation.

5 Conclusion

  • HFS uses CoT-conditioned latent queries to adapt frame selection to task intent, combines relevance, coverage, and redundancy in a set-level objective, and uses mutual learning to avoid offline pseudo-labels.The conclusion reports consistent gains across both teacher backbones and all four benchmarks.Source: §5 · p. 8

Overview of the Appendix

  • The appendix outlines supporting analyses on training cost, stability, sensitivity, reliability, query diversity, robustness, candidate-pool size, and per-category results.It also includes qualitative frame-selection comparisons and a limitations discussion.Source: Overview of the Appendix · p. 11

A Detailed Prompts

  • The student prompt is a fixed template applied automatically to each question-option pair, and its hidden states yield latent query vectors without human annotation.The prompt’s three reasoning dimensions are question semantics, option discrimination, and anticipated visual or temporal evidence; the authors report that the concise version best balances query informativeness and conditioning cost for a 1.5B-parameter student.Source: A Detailed Prompts · p. 11

B Implementation Details

The implementation uses option-restricted answer supervision and a mutual alignment objective whose gradients update both teacher and student scoring. The appendix also specifies the set objective and the teacher’s frame-importance scorer used for alignment.

B.1 Answer Supervision

  • Cross-entropy is computed over the teacher’s logits for answer-option letters A–F, keeping supervision aligned with multiple-choice benchmarks rather than free-form answer wording.Batches are partitioned by option count; an effective batch size of 16 uses micro-batches of 2 with accumulation over 8 steps.Source: §B.1 · pp. 11–12

B.2 Optimization Dynamics

  • Fresh Gumbel re-sampling and straight-through gradients update unselected candidates, countering the risk that KL alignment over selected frames locks in early choices.The KL weight is warmed up from 0.1 to 1.0 during the first epoch while the selector explores.Source: §B.2 · p. 12

B.3 Status of the Set Objective

  • The set objective combines modular relevance, log-sum-exp coverage with diminishing returns, and pairwise dispersion, making subset quality depend on interactions among frames.The term weights are not learned, and the reported sweep indicates the objective is not delicately tuned.Source: §B.3 · p. 12

B.4 Teacher Scorer

  • The teacher scores each selected frame by combining its pooled representation with global question-option context, then normalizes scores over the selected subset for KL alignment.The teacher and student distributions share the same selected-frame indices; the teacher scorer is used only during training, so it adds no inference latency.Source: §B.4 · p. 12

C.1 Training Cost

HFS adds training cost over fine-tuning only the teacher LoRA adapter, but the reported overhead is incurred once. Its training loss decreases smoothly in the reported setting.

C.1 Training Cost

  • HFS uses 20.98M parameters, 34.4G peak memory, and 0.766s per step, versus 2.52M, 24.9G, and 0.649s for teacher-LoRA-only fine-tuning.The per-step increase is 18%, with data, schedule, and batch size held fixed.Source: §C.1 · p. 12, Table 10 · p. 13
  • The total training loss decreases smoothly over the full epoch, indicating stable Gumbel-TopK training in this setting.Source: Figure 4 · p. 13
  • Training takes one epoch of about 40 GPU-h on a single H100, so the reported 18% per-step overhead is paid once before deployment.Source: §C.1 · pp. 12–13

C.2 Training Stability and Hyperparameter Sensitivity

The sensitivity analyses favor an annealed Gumbel-TopK temperature and a fixed temporal-kernel bandwidth across video durations. Additional analyses report bootstrap uncertainty and show that separation loss diversifies latent queries while HFS selects less redundant frames.

C.2 Training Stability and Hyperparameter Sensitivity

  • Annealing temperature from 2.0 to 0.5 reaches 63.3 on Video-MME, ahead of every fixed temperature, whose best is 62.8 at 𝜏=1.0.The authors attribute the result to early exploration followed by late commitment.Source: §C.2 · p. 13, Table 11 · p. 13
  • Across the temporal-kernel bandwidth sweep, no duration split moves by more than 1.1 points, and 𝛾=10 is best overall.The kernel uses a normalized candidate-index grid, so one bandwidth transfers across short, medium, and long videos.Source: §C.2 · p. 13, Table 12 · p. 13

C.3 Statistical Reliability

  • Bootstrap 95% confidence-interval widths range from ±0.8 on NExT-QA to ±4.5 on MLVU; paired system differences carry less variance than marginal intervals suggest.The comparisons use the same questions, teacher, and frame budget.Source: §C.3 · p. 13, Table 13 · p. 13

C.4 Query Diversity and Selection Redundancy

  • Adding separation loss reduces shared top-16 frames from 12.84 to 6.65, importance correlation from 0.85 to 0.56, and squared query cosine similarity from 0.59 to 0.12.The three latent queries attend to complementary candidate-frame regions rather than collapsing onto one intent.Source: §C.4 · p. 13, Table 14 · p. 14, Figure 5 · p. 14
  • HFS has the lowest average pairwise CLIP cosine similarity on both splits: 0.672 and 0.683, versus MDP3's 0.680 and 0.702.The selected-frame comparison covers the MLVU test and LongVideoBench validation splits; HFS remains query-aware and is 2.4 times faster at inference.Source: §C.4 · pp. 13–14, Table 15 · p. 14

C.5 Robustness to Query Paraphrasing

HFS accuracy changes little when NExT-QA questions are paraphrased while preserving intent. The candidate-pool analysis finds 128 frames a balanced default: fewer candidates lose performance, while a larger pool adds no gain.

C.5 Robustness to Query Paraphrasing

  • On 2K paraphrased NExT-QA questions, accuracy moves by 0.2 points with Qwen2.5-VL and 0.6 points with InternVL3, within the reported intervals.A separate paraphraser applies word-order reshuffling, synonym substitution, or active-passive conversion while preserving intent.Source: §C.5 · p. 14, Table 16 · p. 14

C.6 Initial Candidate Pool Size

  • Reducing the candidate pool to 64 costs 1.2 and 1.1 points on the two benchmarks, while 256 candidates yield no further gain after retraining.The authors identify the downstream selection budget of 16 frames, rather than candidate density, as the binding constraint; 128 is the best-balanced default.Source: §C.6 · p. 14, Table 17 · p. 14

D Detailed Benchmark Results

Detailed benchmark results show HFS improving many LongVideoBench categories and all three NExT-QA question types for both backbones. The reported gains are not uniform: InternVL3 has declines concentrated in categories involving fine-grained attribute recognition.

LongVideoBench

  • HFS improves Qwen2.5-VL in 15 of 17 LongVideoBench categories, with only a marginal decline in T2A.The table reports category-level changes on the validation set; red marks improvements and blue marks declines.Source: D Detailed Benchmark Results · pp. 14–15, Table 18 · p. 15
  • With InternVL3, HFS improves reasoning categories but shows slight declines in five categories, primarily those involving fine-grained attribute recognition.The reported interpretation is that frame selection helps object localization and event understanding, while attribute-level discrimination depends on what the reasoner resolves from the selected frames.Source: D Detailed Benchmark Results · p. 15

NExT-QA

  • Both Qwen2.5-VL and InternVL3 improve across causal, temporal, and descriptive NExT-QA question types.Table 19 reports category-level results on the test set; the table caption states that both backbones improve on all three types.Source: D Detailed Benchmark Results · p. 15, Table 19 · p. 15

Detailed benchmark results

  • Across both benchmarks, HFS is reported to outperform uniform sampling through query-aware frame selection.The authors describe this as mitigating information loss associated with fixed-interval sampling.Source: D Detailed Benchmark Results · p. 15

E Qualitative Analysis

The qualitative examples show that HFS selects frames according to the question’s specific intent, including task-relevant states and events that other sampling strategies may miss. The examples span different evidence patterns, question types, and video sources.

E Qualitative Analysis

  • For the television-status question, HFS selects a frame showing the TV on, while relevance-based top-K selects the TV off and uniform sampling spends its budget on unrelated scenes.The comparison illustrates why locating the right object is not always enough: the frame must also show the task-relevant state.Source: E Qualitative Analysis · p. 15, Figure 6 · p. 15
  • Five further examples vary the evidence demand, from a single-instant event and accumulating evidence to redundant actions, brief collisions, and pre-action object locations.The examples use 128 candidate frames, with 16 selected by each method; together they span question types and sources from synthetic renderings to first-person video.Source: E Qualitative Analysis · pp. 15–16
  • Across these cases, HFS’s selection pattern adapts to question intent and differs qualitatively from uniform sampling and relevance-based top-K.The examples include object localization, action recognition, and causal reasoning over rare events.Source: E Qualitative Analysis · p. 16

F Limitations and Future Work

HFS has limitations when questions provide weak task intent and when events fall between its uniformly sampled candidates. The paper identifies adaptive candidate generation as a complementary direction.

F Limitations and Future Work

  • Ambiguous or minimally informative questions may produce latent queries that fail to capture sufficiently discriminative task intent.The queries are extracted from student-model hidden states conditioned on the question and its answer options.Source: F Limitations and Future Work · p. 16
  • The fixed pool of 128 uniformly sampled frames can miss events shorter than the spacing between candidates before selection begins.Among tested pool sizes 64, 128, and 256, 128 performs best; the downstream selection budget of 16 frames is the binding constraint, and adaptive candidate generation is proposed as complementary future work.Source: F Limitations and Future Work · p. 16

Related papers

Loading 2512.11534v3…