PaperlayerView exact revision on arXiv

Source-linked AI summary

CueKFS: Agentic Cue-Driven Keyframe Selection for Long Video Understanding

Weitai Kang, Hanieh Deilamsalehy, Yumo Xu, Dewang Sultania, Serdar Cellat, Yan Yan

arXiv:2609.31873v1cs.CVcs.AIcs.CL

TL;DR

Question-level frame matching can miss evidence for questions combining subjects or moments, while decompositions based on static initial context may be inaccurate. CUEKFS dynamically generates and revises visual cues using retrieved evidence, then allocates frames across surviving cues. It reports state-of-the-art results in all 27 evaluated settings with prior results, with budget-averaged gains up to +4.54% over the previous baseline.

  • Problem

    Whole-question frame similarity can miss questions spanning multiple subjects or moments, requiring relations, or relying on implicit information; decompositions based on a question or coarse preview may also be biased by limited initial context.

    Source: §1 · p. 1

  • Method

    CUEKFS uses an embedding model to retrieve video-wide evidence for visual cues, then a reasoning VLM revises cues against that evidence before budget allocation.

    Source: §1 · pp. 1–2

  • Results

    CUEKFS achieves state-of-the-art results in all 27 settings with available prior results, with budget-averaged gains of up to +4.54% over the previous baseline.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Behavioral analyses show that iterative evidence-guided retrieval can sharpen underspecified cues, separate conflated targets, and suppress distractors before final selection.

    Source: §5 · p. 9

    In the described failure case, frame-wise embeddings do not capture the event stages needed to distinguish a completed lift from an unsuccessful attempt.

    Source: §4.5 · p. 9

Abstract

from arXiv · show

Keyframe selection (KFS) has long produced compact video summaries for browsing and retrieval, and representative frames for thumbnails. More recently, when conditioned on a question, KFS provides an alternative to uniform sampling for long-video question answering by selecting frames that are more relevant to the question. Most methods rank frames by similarity to the question. Yet a relevant frame may score poorly when the question combines subjects or moments that no single frame shows, or requires implicit information absent from its wording. Other methods try to break down the question into subqueries, but suffer from inaccurate decomposition due to their static initial context. Therefore, we propose CueKFS, a training-free method that reformulates question--frame matching as comparing frames against a set of dynamically generated visual cues. From an initial set of salient frames, we decompose the question into cues. Each cue concurrently probes the video to navigate to its own evidence. A reasoning VLM then agentically revises the cue set against its evidence to re-explore the video. CueKFS then allocates the budget across the surviving cues. Across three benchmarks, CueKFS establishes state-of-the-art results in all 27 evaluated settings with available prior results, achieving budget-averaged gains of up to +4.54% over the previous baseline and a median of only two VLM calls. We further provide a detailed behavioral analysis of CueKFS, showing that agentic cue refinement drives active re-exploration of the video, yielding relative similarity gains of up to 92% over the initial context.

1 INTRODUCTION

CUEKFS replaces whole-question frame matching with dynamically revised visual cues, using embedding-based retrieval and reasoning-VLM guidance. It reports state-of-the-art results across evaluated settings and analyzes how refinement changes video exploration.

1 INTRODUCTION

  • Whole-question similarity can miss relevant frames when questions combine subjects or moments, ask about relations, or rely on implicit information absent from their wording.Question decompositions formed from a question alone or coarse preview may remain biased because much of the video is unseen before retrieval targets are set.Source: §1 · p. 1
  • CUEKFS uses an embedding model for video-wide retrieval and a reasoning VLM to dynamically revise what should be retrieved, without training.The embedding model scores frames efficiently but has limited reasoning capacity, while applying a VLM across an entire video is costly.Source: §1 · p. 1
  • Initial cues are grounded in a compact video overview, retrieve evidence across the video, and are revised against that evidence before the frame budget is allocated.Cue-Wave-Guided Refinement uses peaks and plateaus to surface evidence, while Batched Agentic Refinement revises cues efficiently.Source: §1 · p. 2
  • CUEKFS achieves state-of-the-art results in all 27 settings with available prior results, with budget-averaged gains of up to +4.54% over the previous baseline.It uses a median of only 2 reasoning-VLM calls per selection.Source: §1 · p. 2

2 RELATED WORK

Prior KFS methods commonly rank frames using question-level similarity or decompose questions into targets fixed by an initial context. The paper also describes an evaluation protocol that scores selected frames through downstream VLM question answering.

2.1 SELECTION BY QUESTION RELEVANCE

  • Question-relevance methods mostly rank frames by a single question-level similarity signal, whereas CUEKFS scores frames against separate cues so a frame relevant to any cue can surface.The related methods include methods that condition or verify the single question-level signal, but do not replace it with cue-specific matching.Source: §2.1 · p. 2

2.2 SELECTION BY QUESTION DECOMPOSITION

  • Other decomposition methods form target descriptions from an initial question or preview, while CUEKFS lets each cue evolve against its own retrieved evidence.The cited approaches may update search locations or remove found targets, but do not update the targets themselves.Source: §2.2 · p. 2

2.3 EVALUATING KEYFRAME SELECTION

  • KFS is evaluated through downstream VLM QA accuracy, a score that can conflate frame-selection quality with the VLM’s answering capability.The paper retains this protocol, adds GPT-5.5 to reduce the influence of downstream-VLM limitations, and averages accuracy across frame budgets.Source: §2.3 · pp. 2–3

3 METHOD

CUEKFS builds initial visual cues from a compact video overview, retrieves cue-specific evidence, revises cues using that evidence, and allocates a frame budget across the surviving cues.

3 METHOD

  • CUEKFS takes a video, task context, and frame budget, samples candidate frames, and uses a frozen embedding model before applying cue decomposition, refinement, and budget allocation.The method returns at most k frames for a downstream VLM.Source: §3 · p. 3

3.1 CUE DECOMPOSITION

  • Rapid inter-frame transitions define clips whose representative frames form a compact overview, and a reasoning VLM uses that overview and the question to create initial cues.The overview is built using adjacent-frame similarity, wavelet-based transition detection, clip medoids, and farthest-point sampling; cues should be concrete, distinct, and separately represent distinct instances or stages.Source: §3.1 · pp. 3–4

3.2 CUE-WAVE-GUIDED REFINEMENT

  • Each cue is compared with every candidate frame to form a separately normalized, smoothed similarity wave, and evidence is drawn from both sustained plateaus and brief peaks.Plateau samples use interval medoids and farthest-point sampling; retained peak centers and plateau samples form the cue-specific evidence set.Source: §3.2 · p. 4
  • A reasoning VLM reviews each cue with its own retrieved evidence in batches, keeping, rewriting, splitting, or dropping cues before rewritten or split cues re-explore all candidate frames.Batched review reduces the need for individual VLM calls, while kept cues retain their existing waves and evidence.Source: §3.2 · pp. 4–5

3.3 CUE BUDGET ALLOCATION

  • Cue Budget Allocation ranks refined cues by peak similarity and removes frames only when they are both visually similar and temporally nearby.When two such frames come from different cues, the higher-ranked cue’s frame is retained.Source: §3.3 · p. 5
  • After de-duplication, the frame budget is divided among cues in proportion to their remaining evidence counts, prioritizing coverage when possible.Each cue contributes its highest-scoring retained evidence frames, and the final frames are returned in temporal order.Source: §3.3 · p. 5

4 EXPERIMENTS

Across three long-video QA benchmarks, CueKFS is evaluated at three frame budgets with multiple downstream VLMs. The experiments report benchmark results, component ablations, cue-refinement behavior, and qualitative cases.

4.1 EXPERIMENTAL SETUP

  • Evaluation uses 1-fps candidate frames, budgets of 8, 16, or 32, and accuracy averaged across budgets on Video-MME, MLVU, and LongVideoBench without subtitles.A frozen downstream VLM answers using only selected frames; experiments include GPT-5.5, Qwen2.5-VL-7B, and InternVL3-8B.Source: §4.1 · p. 5

4.2 MAIN RESULTS

  • All 27 settings achieve state-of-the-art accuracy; with GPT-5.5, budget-average gains over WFS-SB are +4.06 on Video-MME, +2.42 on MLVU, and +4.06 on LongVideoBench.At k=32, CueKFS also outperforms AKS, MDP3, and A.I.R. on every benchmark.Source: Table 1 · p. 6, §4.2 · p. 6

4.3 ABLATION STUDIES

  • Batched refinement maintains 81.04 accuracy while reducing median VLM calls from 11 to 2; removing allocation or cue-wave evidence reduces accuracy by 2.38 or 2.94 points, respectively.Per-cue refinement raises average accuracy from 79.26 to 80.95 but increases median calls to 11; the batched variant uses B=5.Source: §4.3 · pp. 6–7, Table 2 · p. 6

4.4 ANALYSIS OF CUE BEHAVIOR

  • Refinement rewrites at least one cue on 73% of questions, drops one on 71%, and splits one on 33%; these actions become more frequent as videos grow longer.Refinement gains are positive for all three action groups at both reported budgets and are larger at k=8 than at k=32.Source: §4.4 · p. 7, Table 4 · p. 7
  • Refinement increases relative full-video match gains over the overview from 0.05 to 0.07 on short videos and from 0.43 to 0.53 on long videos; split cues reach 0.92 on long videos.The table reports median relative gains for cue-to-frame similarity, with larger gains indicating stronger matches beyond the overview.Source: §4.4 · p. 8, Table 5 · p. 8
  • Initial and final cue coverage overlap by 0.76 overall and 0.65 on long videos; frames outside final cue coverage score 7.44 to 9.63 points below CueKFS and below uniform sampling at every budget.Split-cue overlap falls from 0.52 on short videos to 0.31 on long videos.Source: §4.4 · p. 8, Table 6 · p. 8

4.5 QUALITATIVE RESULTS

  • In the illustrated success, rewriting and splitting cues sharpens evidence while dropping distractors avoids wasted budget; the failure case shows frame-wise embeddings can miss event completion.Irrelevant cue retrieval and review consume computation, and the lift outcome is lost because the embedding similarity does not represent the local temporal transition.Source: §4.5 · p. 9, Figure 3 · p. 9

5 CONCLUSION

CUEKFS combines a reasoning VLM and embedding model for evidence-guided video exploration, with results across 27 settings and behavioral analyses of cue refinement. The conclusion also identifies temporal context for event transitions and outcomes as a future direction.

5 CONCLUSION

  • Across three benchmarks, downstream VLMs, and frame budgets, CUEKFS achieves state-of-the-art results in all 27 evaluated settings with available prior results.Ablations support its component designs, and behavioral analyses show retrieval beyond the overview and redirected exploration, particularly in longer videos.Source: §5 · p. 9
  • The authors identify adding temporal context to the embedding model to better capture event transitions and outcomes as a promising direction.Source: §5 · p. 9

AI USE STATEMENT

  • Generative AI assisted with language revision and LaTeX typesetting; the authors determined the scientific content, verified claims and values, and take responsibility for the manuscript.Source: AI USE STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

  • Section 3 describes the method, Section 4.1 specifies evaluation, Table 3 reports reasoning-VLM and hyperparameter selection, and Appendices A–B give implementation details and prompt templates.Source: REPRODUCIBILITY STATEMENT · p. 10

A ADDITIONAL IMPLEMENTATION DETAILS

The implementation details specify image handling, overview segmentation and sampling, cue-evidence selection, budget allocation, and the score transformation used in cue analyses.

Image preprocessing and decoding

Cue Decomposition

  • Overview segmentation uses a db4 wavelet detail band, thresholded boundary peaks, and clip durations of 4–12 seconds.Overview slots are allocated approximately by candidate-frame count; medoids initialize within-clip farthest-point sampling, with global sampling filling or reducing the set to budget.Source: A ADDITIONAL IMPLEMENTATION DETAILS · p. 13

Cue-Wave-Guided Refinement

  • Cue evidence retains separated prominent peaks and plateau samples; each cue receives one image per refinement batch before remaining slots go to cues with the largest evidence-count share shortfall.At most five peaks are retained per cue, with a minimum three-frame separation; plateau samples span at least three frames.Source: A ADDITIONAL IMPLEMENTATION DETAILS · p. 13

Cue Budget Allocation

  • Cue budget allocation targets shares proportional to retained evidence counts, gives nonempty cues an initial slot when possible, and fills unused budget with farthest-point sampling if evidence is insufficient.Source: A ADDITIONAL IMPLEMENTATION DETAILS · p. 13

Scores for cue analysis

B FULL PROMPTS

The prompt templates cover initial cue decomposition, batched cue refinement, and downstream question answering. The decomposition prompt uses task context and a video overview, with candidate answers only for multiple-choice inputs.

B FULL PROMPTS

  • Figures 4, 5, and 6 provide templates for cue decomposition, batched agentic refinement, and downstream QA prompts.Source: B FULL PROMPTS · p. 13
  • The Cue Decomposition prompt gives the reasoning VLM task context and a video overview to generate initial visual cues.Source: Figure 4 · p. 14
  • Candidate answers appear in the Cue Decomposition prompt only for multiple-choice inputs.Source: Figure 4 · p. 14

System

The prompts pair each cue with its retrieved video evidence for an independent keep, rewrite, split, or drop decision. Downstream QA receives the selected frames in temporal order with dataset-specific prompts.

System

  • A cue is kept when useful, rewritten when imprecise or retrieving the wrong thing, split when it bundles unrelated visual elements, and dropped when unhelpful to context.Source: System · p. 15
  • Each cue is paired with its own evidence frames, and the reasoning VLM returns one keep, rewrite, split, or drop action per cue.Source: Figure 5 · p. 15
  • Downstream VLMs receive selected frames in temporal order with the corresponding dataset-specific user prompt.Source: Figure 6 · p. 16
Loading 2609.31873v1…