PaperlayerView exact revision on arXiv

Source-linked AI summary

RACER: Reflective Agent Coupling Query Interpretation and Tool-Based Retrieval for Frame Selection in Long Video Understanding

Yiyang Huang, Yitian Zhang, Yizhou Wang, Jianglin Lu, Qihua Dong, Hailing Wang, Huimin Zeng, Mingyuan Zhang, Yun Fu

arXiv:2610.08954v1cs.CVcs.AI

TL;DR

Long-video frame selection must retrieve scattered evidence for complex queries, yet similarity-based and judgment-based methods leave interpretation and localization coupled. RACER uses a lightweight Vid-LLM to reformulate queries and an embedding retrieval tool to localize evidence, refining sub-queries from retrieved frames. It consistently improves long-video understanding benchmarks, including when limited-capability components support more capable Vid-LLMs.

  • Problem

    Long-video frame selection must retrieve relevant evidence scattered across segments in large candidate pools, while complex queries may leave required visual evidence implicit.

    Source: §1 · pp. 1–2

  • Method

    RACER separates query interpretation by a lightweight Vid-LLM from evidence localization by an embedding retrieval tool, using retrieved frames to refine sub-queries.

    Source: §1 · p. 2

  • Results

    RACER consistently improves performance on long-video understanding benchmarks, including Video-MME, MLVU, and LongVideoBench, across multiple Vid-LLM families.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    Agentic integration enables limited-capability components to enhance more capable Vid-LLMs in long-video understanding.

    Source: §1 · pp. 2–3

    Benchmark improvements do not guarantee reliable real-world performance, and human oversight remains important in safety-sensitive applications.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Video large language models (Vid-LLMs) excel at diverse video-language tasks by reasoning over selected frames. However, frame selection for long videos remains challenging, as it requires retrieving relevant frames distributed across segments from a large candidate pool given complex queries. This paper investigates dominant approaches to long-video frame selection from a task-decomposition perspective, identifying two key challenges: the Query Comprehension Gap in similarity-based methods and the Interpretation--Selection Gap in judgment-based methods. To address them, we propose RACER, a training-free reflective agentic framework that decomposes long-video frame selection into query interpretation driven by a lightweight Vid-LLM and evidence localization supported by an embedding model serving as a retrieval tool. Specifically, the Vid-LLM is responsible solely for reformulating the complex query into sub-queries that make implicit information requirements explicit, mitigating the Query Comprehension Gap. Meanwhile, the retrieval tool leverages these sub-queries to localize relevant evidence, relieving the Vid-LLM of direct frame selection and thus addressing the Interpretation--Selection Gap. Finally, the retrieved frames are fed back to the Vid-LLM for sub-query refinement, forming a reflection loop that iteratively improves query interpretation and frame selection. Experiments across multiple benchmarks show that RACER consistently improves long video understanding. Notably, RACER achieves effective frame selection even with limited-capability components, demonstrating that agentic integration enables these components to enhance more capable Vid-LLMs.

1 INTRODUCTION

Long-video frame selection must find scattered evidence for complex queries, but current approaches couple interpreting what evidence is needed with locating it. RACER separates these tasks and consistently improves benchmark performance.

1 INTRODUCTION

  • Long-video evidence is scattered across segments in large candidate pools, and complex queries may not specify the required visual evidence.The paper notes reliability and safety risks in autonomous driving, robotics, and security.Source: §1 · p. 1
  • Similarity-based methods can miss implicit temporal or causal requirements, while judgment-based methods may understand them but fail to select the needed frames reliably.Figure 1 illustrates these gaps with a query about why milk spilled: retrieval misses the cat tipping the glass, while judgment selects milk pouring.Source: §1 · p. 2, Figure 1 · p. 2
  • RACER assigns query reformulation to a lightweight Vid-LLM and evidence localization to an embedding retrieval tool, then refines sub-queries from retrieved frames.This training-free reflection loop separates interpretation from direct frame selection.Source: §1 · p. 2
  • RACER consistently improves long-video understanding across multiple benchmarks and Vid-LLM families, with limited-capability components enhancing more capable models.The reported benchmarks include Video-MME, MLVU, and LongVideoBench.Source: §1 · pp. 2–3

2 RELATED WORK

Prior work develops Vid-LLMs for video-language tasks and long-context processing, while frame-selection methods use similarity scores or model judgments. The paper frames its approach as addressing the coupling of evidence interpretation and localization.

2.1 VIDEO LARGE LANGUAGE MODELS (VID-LLMS)

  • Long-video Vid-LLMs use memory or compact-token compression, as in MovieChat (Song et al., 2024) and LLaMA-VID (Li et al., 2024), or adaptive spatiotemporal compression, as in LongVA (Zhang et al., 2025b) and LongVU (Shen et al., 2025).Earlier and newer Vid-LLMs use approaches including query-based adapters, spatiotemporal pooling, interleaved pretraining, and unified image-video modeling.Source: §2.1 · p. 3

2.2 KEY-FRAME SELECTION FOR VID-LLMS

3 METHOD

Long-video frame selection must interpret complex queries and locate scattered evidence in large candidate pools. RACER separates interpretation from retrieval and refines its sub-queries using retrieved frames.

3.1 CHALLENGES IN DOMINANT FRAME SELECTION APPROACHES

  • Similarity-based methods struggle with complex queries requiring multiple evidence targets linked by temporal, causal, or relational dependencies.They remain effective for simple queries requiring a single salient evidence target.Source: §3.1.1 · p. 3, §3.1 · p. 3

3.1.1 QUERY COMPREHENSION GAP IN SIMILARITY-BASED METHODS

  • On multi-evidence L2-Relation queries, decomposed selection gains 3.2 accuracy points over uniform sampling, versus 0.8 for direct selection.Both approaches achieve equally substantial gains over uniform sampling on single-evidence L1-Perception queries.Source: §3.1.1 · p. 4

3.1.2 INTERPRETATION–SELECTION GAP IN JUDGMENT-BASED METHODS

  • For the longer-duration group, tool-augmented VLM selection adds 3.5 accuracy points and runs 5.4× faster than VLM judgment-based selection.Both VLM variants improve accuracy over uniform sampling across the two duration groups.Source: §3.1.2 · p. 4

3.2 RACER: REFLECTIVE AGENT COUPLING QUERY INTERPRETATION AND TOOL-BASED RETRIEVAL

  • RACER assigns query interpretation to a lightweight Vid-LLM, evidence localization to an embedding retrieval tool, and sub-query refinement to a reflection loop.Retrieved frames are fed back to refine the retrieval targets.Source: §3.2 · p. 5

3.2.1 TASK FORMULATION: FRAME SELECTION IN LONG VIDEO UNDERSTANDING

  • Frame selection chooses N key frames K from T candidate frames V for query q, then provides K and q to a Vid-LLM to generate an answer.The frame budget is much smaller than the candidate pool.Source: §3.2.1 · p. 5

3.2.2 QUERY INTERPRETATION

  • A lightweight Vid-LLM decomposes q using a uniformly sampled video overview into complementary, visually grounded sub-queries targeting distinct implicit information requirements.The overview contains N frames sampled uniformly from the candidate pool.Source: §3.2.2 · p. 5

3.2.3 TOOL-BASED EVIDENCE LOCALIZATION

  • An embedding model scores the original query and sub-queries against candidate frames, then entropy-adaptive fusion and DPP sampling select relevant, visually diverse frames.Lower-entropy sub-query signals receive larger adaptive weights; DPP balances relevance with visual diversity.Source: §3.2.3 · p. 6

3.2 RACER: REFLECTIVE AGENT COUPLING QUERY INTERPRETATION AND TOOL-BASED RETRIEVAL

  • RACER achieves the best or tied-best performance across three benchmarks and four answering models.The comparison reports accuracy for Video-MME and LongVideoBench and M-Avg for MLVU.Source: Table 3 · p. 7

4 EXPERIMENTS

RACER is evaluated across three long-video benchmarks and four answering Vid-LLMs, with consistent gains over uniform sampling and prior frame-selection methods. Ablations assess module contributions, component choices, computational overhead, and qualitative examples of evidence recovery.

4.1 EXPERIMENTAL SETUP

  • RACER’s setup uses frozen components, a 32-frame budget, and two reflection rounds, evaluating four answering Vid-LLMs on Video-MME, MLVU, and LongVideoBench.The default setup uses Qwen2.5-VL-3B for query interpretation and CLIP ViT-L/14 for retrieval.Source: §4.1 · p. 7

4.2 COMPARISON TO THE STATE-OF-THE-ART

  • RACER gains 4.3–6.6 over uniform sampling and consistently outperforms prior frame-selection methods across three benchmarks, with especially large gains on MLVU.On MLVU, which includes videos up to 120 minutes, gains over uniform sampling are 7.3–11.6.Source: §4.2 · pp. 7–8

4.3 ABLATION STUDIES

  • Under cumulative module ablation, accuracy rises from 58.5 to 60.6 with Tool-Based Retrieval, 61.5 after Query Interpretation, and 61.9 with Reflection.The full framework performs best across all three video-duration groups, and Qwen2.5-VL-3B improves LLaVA-OV-7B when assigned Query Interpretation despite lower direct answering accuracy.Source: §4.3 · p. 8, Table 4 · p. 8
  • Replacing CLIP ViT-L/14 with SigLIP-L/16 or EVA02-CLIP-L/14 yields accuracies of 62.26 and 61.81, versus 61.93 for the default; Qwen3-VL-4B yields 62.04.Comparable performance across configurations supports applicability to different retrieval tools and Query Interpretation models.Source: §4.3 · p. 8, Table 5 · p. 8
  • On Video-MME, Tool-Based Retrieval takes 15.1 seconds; adding Query Decomposition raises runtime to 22.4 seconds, and Reflection brings the total to 35.4 seconds and 62.9 distinct Vid-LLM input frames.Candidate-pool scoring is delegated to CLIP, while Vid-LLM input remains limited to a compact set of distinct frames.Source: §4.3 · p. 8, Table 6 · p. 8

4.4 QUALITATIVE RESULTS

  • Qualitative examples show query interpretation recovering the explanation that water expands upon freezing, while reflection retrieves a close-up that corrects “clipper” to “dremel.”Full RACER retains the freezing explanation with less redundancy; refined sub-queries target the blue object and its sanding attachment.Source: §4.4 · p. 9, Figure 4 · p. 9

5 CONCLUSION

RACER separates query interpretation from evidence localization and uses reflection to refine sub-queries. Experiments show consistent gains, while future work will explore broader tasks and more efficient reflection.

5 CONCLUSION

  • RACER uses a lightweight Vid-LLM to make implicit evidence requirements explicit, an embedding-based retrieval tool to select frames, and reflection to refine sub-queries.The paper identifies Query Comprehension and Interpretation–Selection gaps in existing approaches and presents the framework as addressing them.Source: §5 · p. 10
  • Experiments across three benchmarks and four answering Vid-LLMs demonstrate consistent gains in long-video understanding.The findings support assigning query interpretation and evidence localization to models with suitable capabilities, including limited-capability components paired with more capable Vid-LLMs.Source: §5 · p. 10
  • Future work will explore open-ended video understanding and more efficient reflection strategies to reduce computational overhead.Source: §5 · p. 10

AI USE STATEMENT

AI assistants supported editing and coding tasks, while the authors retained responsibility for reviewing and verifying the work.

AI USE STATEMENT

  • AI assistants edited author-written drafts for grammar, clarity, and readability.Source: AI USE STATEMENT · p. 10
  • AI coding assistants supported method implementation and debugging based on author-defined ideas and designs.Source: AI USE STATEMENT · p. 10
  • The authors reviewed AI-assisted edits, tested code, verified outputs, and took responsibility for the final text, claims, and artifacts.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

The ethics statement notes that RACER may inherit biases and inaccuracies, and that omitted evidence can yield incomplete or incorrect answers. It calls for privacy protections and human oversight in safety-sensitive use.

ETHICS STATEMENT

  • RACER may inherit biases and inaccuracies from its pretrained models, and omitted evidence may produce incomplete or incorrect answers.Source: ETHICS STATEMENT · p. 10
  • Responsible use should respect privacy and dataset and model usage conditions.Source: ETHICS STATEMENT · p. 10
  • Human oversight remains important in safety-sensitive applications because benchmark improvements do not guarantee reliable real-world performance.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

RACER’s reproducibility materials include model and evaluation details, while Video-MME results report gains across question types and video durations with improved efficiency.

REPRODUCIBILITY STATEMENT

  • RACER uses public datasets and pretrained models without additional training, and the paper specifies configurations, protocols, frame budgets, hyperparameters, and selection procedures.The authors state that they will release code, experimental configurations, and per-question outputs.Source: REPRODUCIBILITY STATEMENT · p. 10

A COMPARISON ACROSS QUESTION TYPES

  • RACER achieves the highest accuracy across all six Video-MME categories, over all 2700 questions.The comparison includes uniform sampling, direct CLIP selection, and VLM judgment-based selection.Source: A COMPARISON ACROSS QUESTION TYPES · p. 14

B EFFICIENCY ACROSS VIDEO DURATIONS

  • Overall, RACER improves accuracy from 60.1 to 61.9 with a 3.5× speedup, and on long videos improves accuracy from 51.7 to 52.7 with a 5.2× speedup.Across durations, RACER analyzes 57.6–67.1 distinct frames in 22.4–54.9 seconds, versus 157.7–2465.9 frames and 22.8–286.2 seconds for VLM judgment-based selection.Source: B EFFICIENCY ACROSS VIDEO DURATIONS · p. 14

C ADDITIONAL HYPERPARAMETER ABLATION

The hyperparameter ablation evaluates RACER on Video-MME with LLaVA-OV-7B; the α sweep favors balancing relevance and visual diversity in DPP sampling.

C ADDITIONAL HYPERPARAMETER ABLATION

  • The α sweep evaluates DPP sampling on Video-MME with LLaVA-OV-7B.The ablation varies one parameter at a time, with default settings marked and the best observed result identified.Source: C ADDITIONAL HYPERPARAMETER ABLATION · p. 14

C.1 EFFECT OF α IN DPP SAMPLING

  • Accuracy rises from 60.41% at α = 0.55 to a peak of 61.93% at α = 0.75.Larger α values yield no further improvement.Source: §C.1 · p. 14
  • The results favor balancing relevance and visual diversity over placing excessive emphasis on relevance.Source: §C.1 · p. 14

C.2 EFFECT OF β IN ENTROPY-BASED FUSION

The ablations examine entropy sensitivity, original-query weighting, and Vid-LLM generation temperature on Video-MME with LLaVA-OV-7B.

C.2 EFFECT OF β IN ENTROPY-BASED FUSION

  • Accuracy varies only from 61.89% to 62.11% over β ∈[2.0, 2.4], and the default β = 2.2 achieves a comparable 61.93%.Although β = 2.0 yields the highest accuracy, the default indicates limited sensitivity within the tested range.Source: §C.2 · pp. 14–15

C.3 EFFECT OF THE ORIGINAL-QUERY WEIGHT w0

  • The default w0 = 0.6 achieves the highest accuracy of 61.93%, while accuracy drops to 60.48% at w0 = 0.8.The weight balances the original-query score against combined sub-query scores.Source: §C.3 · p. 15

C.4 EFFECT OF THE VID-LLM GENERATION TEMPERATURE

  • The default τ = 0.8 achieves 61.93% accuracy, compared with 60.30% at τ = 0.6 and 60.89% at τ = 1.0.Neither lower nor higher generation temperature consistently improves frame selection.Source: §C.4 · p. 15

D ADDITIONAL FRAMEWORK DESIGN ABLATIONS

Framework ablations assess sub-query aggregation, reflection depth, and reflection inputs; the discussion contrasts RACER’s division of interpretation and evidence localization with prior frame-selection approaches.

D.1 EFFECT OF SUB-QUERY AGGREGATION

  • Entropy-based weighting improves accuracy from 61.67% to 61.93% over uniform weighting.This supports adaptive sub-query weighting based on retrieval-score distributions.Source: §D.1 · p. 16

D.2 EFFECT OF THE NUMBER OF REFLECTION STEPS

  • Accuracy rises from 61.52% without reflection to 61.89% after one step, 61.93% after two, and 61.96% with three or four steps.The default two steps capture most of the gain and require three retrieval rounds, including the initial round.Source: §D.2 · p. 16

D.3 EFFECT OF THE REFLECTION INPUT

  • Hybrid overviews using retrieved feedback reach 61.70% with temporal-distance pruning and 61.93% with visual-similarity pruning, versus 61.56% for a uniform overview.The default retains all feedback frames and removes the least visually similar overview frames.Source: §D.3 · p. 16

E ADDITIONAL DISCUSSION OF RELATED WORK

  • Unlike similarity-based methods [Liu et al., 2025; Zhang et al., 2025c; Tang et al., 2025; Sun et al., 2025] and judgment-based methods [Wang et al., 2025b; Zhang et al., 2024a; Wang et al., 2024a; Zou et al., 2026], RACER assigns frame selection to retrieval and query refinement to a lightweight Vid-LLM.The Vid-LLM does not score frames, select indices, or specify temporal regions; retrieved frames inform only sub-query refinement.Source: E ADDITIONAL DISCUSSION OF RELATED WORK · p. 16
Loading 2610.08954v1…