PaperlayerView exact revision on arXiv

Source-linked AI summary

Divide, then Ground: Adapting Frame Selection to Query Types for Long-Form Video Understanding

Jialuo Li, Bin Li, Jiahao Li, Yan Lu

arXiv:2512.04000v2cs.CVcs.AIcs.LG

TL;DR

Limited context lengths and dense video tokens constrain long-form video understanding, while query-aware selection can carry substantial computational overhead. DIG adapts selection to global versus localized queries, and experiments report consistent baseline improvements across benchmarks and frame counts from 8 to 256.

  • Problem

    Limited LLM context and dense video tokens make processing all video frames impractical, while query-aware selection methods incur significant computational overhead.

    Source: §1 · pp. 1–2

  • Method

    DIG is a training-free framework that uses uniform sampling for global queries and a specialized multi-stage selection pipeline for localized queries.

    Source: §1 · p. 2

  • Results

    Across long-form video benchmarks, DIG consistently outperforms baselines; at 32 frames, it boosts Qwen2.5-VL-7B accuracy by 7.68% on MLVU compared with uniform sampling.

    Source: §5.2 · p. 6, §7 · p. 11

  • Takeaways & Limitations

    Global queries can use uniform sampling, while localized queries call for targeted selection where uniform sampling fails.

    Source: §1 · p. 2

Abstract

from arXiv · show

The application of Large Multimodal Models (LMMs) to long-form video understanding is constrained by limited context lengths and the computationally prohibitive cost of processing dense video tokens. Consequently, recent research has focused on query-aware frame selection, methods that often incur significant computational overhead. This paper challenges the assumption that such complex search mechanisms are universally necessary. We first identify and validate a query typology distinguishing between global query and localized query. We demonstrate that while uniform sampling is both effective and efficient for global queries, localized queries indeed necessitate query-aware selection for optimal performance. Building on this insight, we propose DIG, a training-free frame selection framework that adapts its strategy based on the query type. Specifically,DIG employs efficient uniform sampling for global queries while activating a specialized pipeline to extract query-relevant frames for localized queries. Experiments on three long-form video understanding benchmarks demonstrate that DIG consistently outperforms existing baselines and robustly improves LMM performance, even when scaling the input frame count to 256.

1 Introduction

Limited context and dense video tokens make processing every frame impractical, while query-aware selection adds computational overhead. DIG adapts frame selection to query type, using uniform sampling for global queries and targeted selection for localized queries.

1 Introduction

  • Uniform sampling maximizes temporal coverage but can select redundant frames and omit query-relevant moments.Limited LLM context length and the volume of video tokens make it impractical to input all frames directly.Source: §1 · p. 1
  • Global queries remain stable as sampled frames increase, whereas localized-query performance degrades substantially as irrelevant frames enter the context.Uniform sampling is robust for global queries; localized queries benefit from query-aware selection where uniform sampling fails.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • DIG classifies each query, uses uniform sampling for global queries, and applies a multi-stage targeted pipeline for localized queries.For localized queries, the pipeline selects representative frames, scores their relevance with the LMM, refines the video, then uniformly samples the refined representation.Source: §1 · p. 2
  • Across three long-form video benchmarks, DIG consistently outperforms existing baselines and improves LMM performance with inputs from 8 to 256 frames.The introduction also reports that query-based sampling is necessary for high performance on localized queries.Source: §1 · p. 2

2 Related Work

Video LMMs face token-volume and context-length constraints, motivating frame and token reduction. Related approaches trade compactness against fine-grained detail or rely on query-relevant frame selection.

2.1 Video-based Large Multimodal Models

  • LMMs process visual features from video frames, but high video-token volume and limited context length constrain video understanding.The passage identifies a need for efficient frame selection that captures key visual content without overloading the model.Source: §2.1 · p. 2

2.2 Video Token Reduction for the VQA Task

  • Uniform sampling is a standard VQA token-reduction method, while adaptive reduction methods are divided into two categories.The accompanying performance-trend figure shows accuracy peaking at an optimal frame count and declining beyond it.Source: §2.2 · p. 2, Figure 2 · p. 3
  • Token compression creates compact representations through memory banks, temporal redundancy reduction, or hierarchical compression, but can lose fine-grained or query-related details.Over-compression may generalize highly query-related segments and compromise answers.Source: §2.2 · p. 3
  • Query-based selection scores sampled frames or segments for relevance, but sparse candidates can miss clip-level detail and relevance metrics can be unreliable for complex reasoning.The paper's stated responses are content-adaptive candidates, LMM-based relevance assessment, and retrieval of continuous clips before final selection.Source: §2.2 · p. 3

3 Revisiting Inference Mechanism of LMM in Video Understanding

Increasing the number of uniformly sampled frames does not guarantee better VQA performance. The paper distinguishes global from localized queries and finds that their performance trends differ as frame counts rise.

3 Revisiting Inference Mechanism of LMM in Video Understanding

  • Across three LMMs and three long-form benchmarks, performance initially improves with more uniformly sampled frames but declines beyond a certain point.The evaluated models are Qwen2.5-VL-7B, InternVL3-8B, and LLaVA-OneVision-7B; Figure 2 presents the frame-count trends.Source: §3 · p. 3, Figure 3 · p. 4
  • At high frame counts, performance degradation is predominantly attributed to localized queries, while global-query performance remains relatively stable.Figure 3 plots relative accuracy as deviation from the initial baseline.Source: Figure 3 · p. 4, §3 · p. 4
  • Localized queries target specific video segments, while global queries require comprehensive understanding of the entire video.Examples are asking what kind of bike a man is riding and what title best summarizes a video, respectively.Source: §3 · p. 4
  • Global-query performance remains relatively stable with more frames, while localized-query performance drops significantly, which the paper attributes to excess frames introducing noise for localized tasks.The authors conclude that global queries can use uniform sampling without the computational overhead of key-frame search.Source: §3 · p. 4

4 Method: DIG

DIG is a training-free frame-selection framework that routes global queries to uniform sampling and localized queries through a specialized pipeline before sampling frames for LMM inference.

4.1 Query Type Identification

  • Global queries use uniform sampling across the video, while localized queries enter DIG’s specialized frame-selection pipeline.The selected frames are ultimately processed by the LMM for inference.Source: §4.1 · p. 4, §4 · p. 4, Figure 4 · p. 5

4.2 Content-Adaptive Frame Selection (CAFS)

  • CAFS samples at 2 fps, uses DINOv2 feature-distance peaks to segment the video, and selects each segment’s midpoint as an r-frame.Aggregating r-frames provides a compact representation of the video’s essential visual content.Source: §4.2 · p. 5

4.3 Reward Assignment

  • DIG uses the LMM to score r-frame relevance, including whether adjacent frames may contain context useful for answering the query.This two-factor assessment is intended to address queries, such as “why” or “how,” that a single frame may not answer fully.Source: §4.3 · p. 5

4.4 Video Refinement

  • Rather than fixed Top-K selection, DIG repeatedly thresholds rewards below their current mean until the positive-index set stabilizes.The selected r-frames are those with positive rewards in the final iteration.Source: §4.4 · p. 6
  • For each selected r-frame, DIG unions nearby video segments within a window of wlen, then uniformly samples frames from the refined video.The segment range includes the selected frame’s neighboring r-frames, providing additional fine-grained temporal content.Source: §4.4 · p. 6

5 Experiment

DIG is evaluated on three long-form video benchmarks with two Qwen2.5-VL models. It consistently improves over the compared frame-selection methods across frame counts, including settings up to 256 frames.

5.1 Experiment Settings

  • The evaluation uses MLVU, LVB, and the medium and long VideoMME splits, with no subtitles and two Qwen2.5-VL model sizes.The setup also compares against AKS, Q-Frame, and uniform sampling.Source: §5.1 · p. 6

5.2 Main Results

  • Across 8–256 frames, DIG improves performance over uniform sampling, Q-Frame, and AKS on both Qwen2.5-VL sizes; at 32 frames, Qwen2.5-VL-7B accuracy gains 7.68% on MLVU and 4.51% on LongVideoBench versus uniform sampling.The comparison is reported in Table 1, which covers Qwen2.5-VL-32B and Qwen2.5-VL-7B.Source: §5.2 · p. 6, Table 1 · p. 7
  • At 128 frames, AKS and Q-Frame underperform uniform sampling by 1–2% on LongVideoBench with Qwen2.5-VL-7B, while DIG gains over uniform in most tested settings.The evaluation extends to 256 frames, unlike prior work cited in the passage that validates in regimes below 64 frames.Source: §5.2 · p. 7

6 Discussion and Analysis

The analysis compares DIG’s query-dependent sampling with uniform sampling, then examines CAFS, reward assignment, temporal window length, and computational efficiency.

6.1 Frame Selection Strategy vs. Query Type

  • For global queries, uniform sampling is comparable to or sometimes better than DIG’s pipeline, while the pipeline consistently outperforms it on localized queries.Figure 5 compares the strategies across query types using Qwen2.5-VL-7B and Qwen2.5-VL-32B.Source: §6.1 · p. 8, Figure 5 · p. 8

6.2 Analysis of CAFS Effectiveness

  • CAFS is assessed with Global Coverage and Localized Coverage, alongside FPS and uniform sampling at a consistent average frame count.GlC measures how well r-frames represent sampled video content; LoC measures their similarity to neighboring frames. The benchmarks are MLVU and VideoMME.Source: §6.2 · pp. 8–9
  • Uniform-sampling performance declines as videos get longer, whereas CAFS consistently outperforms FPS, particularly beyond 10 minutes.The analysis attributes uniform sampling’s decline to redundancy in short videos and inadequate semantic coverage in long ones.Source: §6.2 · pp. 9–10
  • Within DIG, CAFS outperforms uniform sampling across benchmarks, and the performance gap widens as more input frames are used.The authors report that CAFS adapts to video length and provides better coverage for subsequent reasoning.Source: §6.2 · p. 10

6.3 Reward Assignment: LMM vs. CLIPScore

  • LMM-based rewards outperform CLIPScore in most benchmark cases, and Qwen2.5-VL-32B outperforms the 7B reward assigner, including on VideoMME-short.Table 2 compares the reward assigners with Qwen2.5-VL-7B as the base LMM.Source: Table 2 · p. 9, §6.3 · p. 10

6.4 Impact of Window Length

  • A window length of 0 gives the lowest performance, while a length of 8 degrades results relative to 2 and 4; length 2 performs best across benchmarks.The authors attribute the short-window deficit to missing surrounding context and the long-window degradation to irrelevant context noise.Source: §6.4 · p. 10

6.5 Efficiency of DIG

  • 62.5% is uniform sampling’s peak accuracy; DIG surpasses it above 720 TFLOPs and continues improving at higher computation.The comparison measures average FLOPs per question-answer pair on LongVideoBench.Source: §6.5 · p. 11

7 Conclusion

The conclusion presents DIG as a training-free framework that adapts frame selection to query type and reports consistent baseline gains across benchmarks and frame counts. The appendix supplies benchmark, annotation, prompt, and extended-analysis details.

7 Conclusion

  • DIG uses uniform sampling for global queries and a multi-stage targeted pipeline for localized queries, and experiments show baseline gains across diverse benchmarks and LMMs.The conclusion reports robust scaling for inputs from 8 to 256 frames.Source: §7 · p. 11

Appendix

  • The appendix adds benchmark statistics, query-annotation protocols, prompt designs, component analyses, per-task results, and runtime profiling.It also includes extended experiments on Qwen3-VL-8B and analysis of query-identification efficiency gains.Source: Appendix · p. 17

A Benchmark Details

The evaluation uses selected portions of three long-video benchmarks and documents how queries are classified and prompts guide DIG’s stages.

A Benchmark Details

  • Evaluation uses MLVU dev-set multiple-choice questions, LongVideoBench validation questions, and VideoMME medium- and long-duration video-only questions.MLVU has 3,102 questions across nine categories; LongVideoBench has 3,763 videos and 6,678 questions; VideoMME has 900 videos and 2,700 question-answer pairs.Source: A Benchmark Details · p. 17

B Query Identification by Human Annotator

  • Query labels map MLVU holistic tasks to global and detail tasks to localized, classify all LongVideoBench queries as localized, and majority-vote VideoMME annotations.The authors identify 462 global and 1,708 localized MLVU queries, and 479 global and 2,221 localized VideoMME queries.Source: B Query Identification by Human… · p. 18

C Prompt Design

  • DIG’s three core prompts handle query identification, reward assignment, and direct inference; the identification prompt uses explicit reasoning steps to classify query type.The reward prompt asks an LMM to describe query-relevant frame content and assign a score from 0 to 100; direct inference uses multiple-choice options.Source: C Prompt Design · p. 18

D More Details about Query Identification

Off-the-shelf LLMs align well with human query-type labels, especially for localized queries. CAFS then selects representative frames by detecting prominent changes in frame-to-frame distance and choosing midpoints between them.

D More Details about Query Identification

  • Human annotations provide the ground-truth query labels used to evaluate LLM classification accuracy on MLVU, LongVideoBench, and VideoMME.The authors use Qwen3-Next-80B-A3B-Instruct in their main experiments as a trade-off between compute cost and final performance.Source: D More Details about Query… · p. 19
  • Nearly all evaluated LLMs exceed 80% overall query-classification accuracy, and almost all exceed 90% accuracy on localized queries.Global-query accuracy is lower, but the authors report only minor computational overhead and choose Qwen3-Next-80B-A3B-Instruct for the main experiments.Source: D More Details about Query… · p. 19

E.1 Detailed Algorithm of CAFS

  • CAFS takes frame-to-frame distances and original frame indices as input, then outputs selected r-frame indices.Its algorithm has three sequential stages: initial peak detection, prominence filtering, and r-frame selection.Source: §E.1 · p. 19
  • CAFS marks a distance value as an initial peak when it exceeds both immediate neighbors, treating local maxima as candidate content boundaries.The indices of these local maxima form the initial peaks set.Source: §E.1 · p. 19
  • CAFS retains peaks whose prominence exceeds 0.1, measuring prominence as peak height minus the higher of the left and right base levels.This filtering discards minor fluctuations before selection.Source: §E.1 · p. 19
  • CAFS selects frames at the temporal midpoints between consecutive prominent peaks, using their original frame indices.The midpoints represent centers of stable segments and are aggregated into the final r-frame set.Source: §E.1 · pp. 19–20

E.2 More Results of CAFS

CAFS frame counts vary nonlinearly with video duration, while DIG remains effective as input frame counts scale into ultra-long-context settings. The extended experiments report gains over uniform sampling on all three benchmarks, including at 768 frames.

E.2 More Results of CAFS

  • On LongVideoBench, CAFS averages 47.9 r-frames for videos of 0–10 minutes and 226.4 for videos of 10–20 minutes.The authors report that this nonlinear scaling challenges fixed-rate sampling, which risks information loss when sparse and redundancy when dense.Source: §E.2 · p. 20, Figure 10 · p. 21
  • On MLVU, videos averaging 12.7 minutes are reduced to 180.8 r-frames on average, about one r-frame every 4.22 seconds.The authors describe this as sparse compression of prolonged video context.Source: §E.2 · p. 20

F.1 Detailed Experiment Settings

  • The AKS baseline uses candidate frames sampled at 1 fps and excludes budgets of 8 and 16 because it sometimes returns null at those settings.Q-Frame uses a fixed-frame-count strategy and is limited to budgets up to 128 because its initial candidate pool is capped there.Source: §F.1 · p. 21

F.2 Extended Experiments with DIG

  • The extended DIG evaluation uses Qwen3-VL-8B and scales input counts from 8 to 768 frames, with about 150 tokens per frame.DIG and AKS use Qwen3-VL-8B as the unified backbone for reward assignment and final inference.Source: §F.2 · p. 21
  • At 768 frames, DIG surpasses uniform sampling by 4.7% on MLVU, 3.7% on LongVideoBench, and 3.5% on VideoMME-Medium.At 256 frames, DIG achieves an 8.6% performance boost on MLVU compared with uniform sampling; AKS degrades as context length increases.Source: §F.2 · p. 21

F.3 Detailed Experiment Results & More Analysis

Uniform sampling performs comparably on global-query tasks, whereas query-aware selection improves results on localized perception tasks. Even with relevant visual information, temporal reasoning performance remains stagnant across methods.

F.3 Detailed Experiment Results & More Analysis

  • On MLVU Anomaly Recognition and Topic Reasoning, evaluated methods perform comparably to uniform sampling regardless of input frame count.The authors identify uniform sampling as the preferred strategy for these global queries because it maintains efficiency while providing sufficient performance.Source: §F.3 · p. 21
  • Query-aware selection significantly and consistently outperforms uniform sampling on localized perception tasks including PlotQA, NeedleQA, and L1-Perception.The authors attribute the gap to uniform sampling including irrelevant content, while query-aware selection supplies relevant frames.Source: §F.3 · p. 22, Table 6 · p. 23, Table 8 · p. 24
  • Performance remains stagnant across methods on temporal-logic tasks such as Action Order and L2-Relation, even with query-relevant visual information.The authors identify temporal reasoning and sequencing as a limitation that persists independently of visual information retrieval quality.Source: §F.3 · p. 22

G More Efficiency Analysis of DIG

DIG separates key-frame selection from LMM inference and reduces selection overhead relative to AKS while maintaining downstream performance. Its query-identification module saves time on mixed-query benchmarks by avoiding specialized selection for global queries.

G.1 Detailed Runtime Profiling

  • Runtime profiling separates key-frame selection overhead from the LMM inference time for processing selected frames.Experiments ran on a node with 8 NVIDIA A100 GPUs; Tables 9 and 10 report inference latency and selection overhead, respectively.Source: §G.1 · p. 25, Table 9 · p. 25, Table 10 · p. 25
  • DIG reduces computational overhead by an order of magnitude versus AKS while maintaining superior downstream performance.Compared with Q-Frame, DIG incurs a marginal processing-time increase, while Q-Frame fails to beat uniform sampling above 32 frames.Source: §G.1 · p. 25, Table 10 · p. 25

G.2 Efficiency Gains from Query Identification

  • Query identification saves 19.9% of frame-selection time on VideoMME and 13.3% on MLVU, but adds 3.6% overhead on predominantly localized LongVideoBench.The adaptive approach applies resource-intensive selection only to localized queries and uses uniform sampling for global ones.Source: §G.2 · p. 25, Table 11 · p. 26
Loading 2512.04000v2…