PaperlayerView exact revision on arXiv

Source-linked AI summary

VideoSearch-R1: Iterative Video Retrieval and Reasoning via Soft Query Refinement

Seohyun Lee, Seoung Choi, Dohwan Ko, Jongha Kim, Hyunwoo J. Kim

arXiv:2607.00446v1cs.CVcs.AI

TL;DR

Existing video systems often separate retrieval from fine-grained reasoning or assume the relevant video is already known, limiting end-to-end video corpus moment retrieval. VideoSearch-R1 addresses this with iterative retrieval, verification, latent-space query refinement, and intra-video reasoning, achieving state-of-the-art performance across three VCMR benchmarks in video retrieval and temporal grounding.

  • Problem

    Existing methods often treat retrieval as preprocessing, while video agents typically assume the query-relevant video is already known, leaving integrated retrieval and fine-grained reasoning underexplored.

  • Method

    VideoSearch-R1 iteratively retrieves and verifies candidate videos, refines queries through continuous latent-space Soft Query Refinement, and performs intra-video reasoning in a multi-turn loop.

  • Results

    VideoSearch-R1 achieves state-of-the-art performance across three VCMR benchmarks in both video retrieval and temporal grounding.

  • Takeaways & Limitations

    SQR provides efficient, fine-grained query adjustment with substantially fewer generated tokens while unifying inter-video retrieval and intra-video reasoning.

Abstract

from arXiv · show

As video corpora continue to expand in both scale and task complexity, there is increasing demand for approaches that retrieve relevant videos from large-scale corpora (inter-video reasoning) and subsequently perform fine-grained, query-conditioned tasks (intra-video reasoning) within the retrieved content, such as temporal grounding. However, existing approaches typically treat retrieval as a preprocessing step, and consequently, when the initial retrieval fails, there is no mechanism to refine the search, leading to the failure of subsequent fine-grained intra-video reasoning. Moreover, while recent agentic frameworks have advanced video understanding, they typically assume that the query-relevant video is already given, focusing exclusively on intra-video reasoning tasks. To address these limitations, we propose VideoSearch-R1, an agentic framework for iterative video retrieval and reasoning through multi-turn interaction with a video search engine. Specifically, we introduce Soft Query Refinement (SQR) to refine search query tokens in a continuous latent space rather than rewriting queries in the discrete text space, enabling more efficient and fine-grained adjustments. SQR and its reasoning process are trained using Group Relative Policy Optimization (GRPO), guided by task-level reward signals derived from retrieval and downstream tasks. Building upon this, VideoSearch-R1 achieves state-of-the-art performance across three datasets on Video Corpus Moment Retrieval (VCMR), iteratively retrieving videos from large-scale corpora, refining search queries, and performing precise query-conditioned temporal grounding within the retrieved content. Our analyses show that SQR effectively refines the original query, requiring significantly fewer generated tokens than explicit text-level query refinement. Code and model checkpoints are publicly available at mlvlab.github.io/VideoSearch-R1.

1 Introduction · 2 Related Works · 3 Method

VideoSearch-R1 addresses the failure of decoupled video retrieval and intra-video reasoning by iteratively retrieving, verifying, refining, and temporally grounding videos through multi-turn interaction. Its Soft Query Refinement operates in continuous latent space, while SFT and GRPO jointly optimize retrieval and reasoning, yielding state-of-the-art results on three VCMR benchmarks.

  • 1 Introduction: VideoSearch-R1 jointly performs corpus-level video retrieval and query-conditioned intra-video reasoning, including precise temporal grounding within retrieved videos.The framework addresses the limitation that retrieving a relevant video alone is insufficient for fine-grained applications, while retrieval failures in decoupled pipelines propagate to later reasoning.
  • 2 Related Works: Existing video agentic frameworks commonly assume the query-relevant video is already available, unlike VideoSearch-R1’s explicit integration of an external video search engine.Related retrieval methods range from efficient dual encoders with coarse alignment to VLM-based reranking of fixed top-K candidates.
  • 3 Method: VideoSearch-R1 iteratively retrieves candidate videos, verifies query-video matching, refines the query, and performs intra-video reasoning through multi-turn interaction.The system uses an external video search engine and unifies inter-video retrieval with intra-video reasoning.
  • 1 Introduction: VideoSearch-R1 achieves state-of-the-art performance on three VCMR benchmarks for both video retrieval and temporal grounding.The reported result covers the jointly optimized inter-video retrieval and intra-video reasoning framework.
  • 3.1 VideoSearch-R1 with Soft Query Refinement: When retrieval mismatches the query, SQR autoregressively generates N continuous latent soft query tokens, appends them to the original query, and re-invokes the search engine.Unlike explicit text-level rewriting, SQR enables fine-grained representation adjustments with fewer generated tokens; verification precedes refinement, and a match triggers temporal grounding.
  • 3.2 Training Procedure: The two-stage training procedure combines SFT for structured reasoning and soft-query generation with GRPO for exploring improved retrieval-and-reasoning trajectories.SFT supervises verification and temporal grounding on positive pairs, verification on negative pairs, and trains soft queries with InfoNCE against ground-truth and negative videos; GRPO propagates rewards across retrieval and reasoning.
  • 3.2 Training Procedure: GRPO’s reward design combines format compliance with verification, retrieval, and conditional temporal-grounding rewards to holistically optimize the iterative framework.Temporal rewards apply when the model predicts a match, and format reward requires the prescribed reasoning, answer, start, and end structures.
  • 3.3 Inference via Multi-Turn Interaction: At inference, each turn assesses semantic alignment, generates soft tokens for mismatches, reranks candidates with the refined query, and incorporates the new video into subsequent context.The process consists of external search, video verification, SQR, and temporal grounding within the selected video, continuing until a match or the maximum number of turns is reached.

4 Experiments

Experiments evaluate VideoSearch-R1’s joint corpus-level video retrieval and temporal grounding on VCMR across Charades-FIG, DiDeMo-FIG, and ActivityNet-FIG. Results show that iterative SQR improves retrieval, GRPO strengthens temporal reasoning, and a small number of refinement turns is sufficient.

  • Main results: 6.0 R@1 improvement on ActivityNet-FIG demonstrates that SQR substantially improves video retrieval over baselines using the same search engine.Qwen3-VL-2B zero-shot and fine-tuned baselines leave query representations unchanged during multi-turn inference, whereas VideoSearch-R1 iteratively refines them.
  • Effect of training stages: GRPO substantially improves temporal grounding beyond SFT, while SFT mainly establishes reasoning structure and basic SQR capabilities.SFT produces strong VR gains but only marginal VCMR grounding gains; adding RL yields pronounced improvements.
  • Reward design: Adding retrieval, verification, and temporal-grounding rewards progressively improves query alignment, verification accuracy, retrieval decisions, and temporal reasoning.The retrieval reward encourages representations aligned with video embeddings, while verification improves semantic consistency assessment and retrieval reliability.
  • Analysis of SQR: 7.2 R@1 gain from SQR exceeds HQR’s 3.7 gain, while SQR uses eight soft tokens versus HQR’s 26.8-token refined queries.Continuous latent refinement therefore provides finer adjustments with substantially fewer generated tokens than explicit text-level refinement.
  • Analysis of SQR: Performance improves from the first to second inference turn and saturates at T = 3, while appended soft tokens progressively improve retrieval specificity and ground-truth rank.A qualitative case reaches IoU 0.89 after correcting an initially mismatched retrieval and localizing the target from 0.0s to 9.86s.

5 Conclusion

VideoSearch-R1 unifies inter-video retrieval and intra-video reasoning in an iterative multi-turn loop, using Soft Query Refinement to optimize queries in continuous latent space instead of rewriting tokens explicitly.

  • VideoSearch-R1 unifies inter-video retrieval and intra-video reasoning within an iterative multi-turn loop.
  • The model autonomously retrieves candidate videos, verifies semantic alignment with user intent, refines search queries, and reasons over retrieved content.
  • Soft Query Refinement optimizes search queries in continuous latent space rather than explicitly rewriting tokens.

(Supplement) · S1 Effect of InfoNCE objective in Stage 1

Ablating InfoNCE from Stage 1, while keeping the rest of training unchanged, noticeably reduces retrieval performance—especially VR—and consequently degrades VCMR. The supplement also reports analyses of query refinement, scaling, generalization, implementation, and cross-task performance.

  • (Supplement): The supplement includes additional analysis of Hard Query Refinement.
  • (Supplement): The supplement evaluates the effect of scaling Stage 1 training data.
  • (Supplement): The supplement analyzes scaling for the video search engine and Soft Query Refinement.
  • (Supplement): The supplement reports HQR training implementation details and cross-dataset generalization analysis.
  • (Supplement): The supplement includes cross-task VideoQA evaluation and qualitative comparisons between HQR and SQR.
  • S1 Effect of InfoNCE objective in Stage 1: Removing InfoNCE from Stage 1 noticeably lowers retrieval performance, especially VR, and consequently degrades VCMR.The ablation keeps the remainder of the training pipeline unchanged, indicating that retrieval-aware representation learning improves cold-start initialization.

S2 Analysis on Hard Query Refinement (baseline)

The analysis finds that scaling hard query refinement (HQR) improves retrieval only modestly before quickly saturating, while instruction-based augmentation is the strongest HQR variant but still provides limited gains. These findings motivate soft query refinement (SQR) as a way to address ambiguity beyond simply increasing reasoning capacity.

  • HQR scaling: Scaling the HQR rewriter from 2B to 8B yields modest retrieval improvements that quickly saturate, indicating that reasoning capacity alone is insufficient to resolve query ambiguity.The evaluation uses a retrieval-retry setting in which refinement occurs after an incorrect initial top-1 retrieval.
  • HQR variants: HQR-InstructionAug performs best among the tested HQR variants on initially mismatched queries.The variants differ in whether they rewrite the query or preserve it while adding instruction-based guidance; the comparison is illustrated in Fig. 2.
  • HQR variants: Effective HQR selectively downweights already matched query aspects and upweights missing target-video aspects, rather than rewriting the query from scratch.This redistribution of semantic emphasis can recover the ground-truth video when the retriever has overemphasized only part of the query.
  • Motivation for SQR: Even the optimized HQR-InstructionAug design produces only limited gains, supporting the motivation for developing SQR.The authors adopt HQR-InstructionAug as the default hard-refinement baseline because it is strongest among the tested variants.

S3 Analysis of Scaling Stage 1 Training Data

Scaling Stage 1 training data with additional synthetic reasoning examples improves the supervised baseline but does not recover Stage 2’s performance gains.

  • S3 Analysis of Scaling Stage 1 Training Data: The scaling experiment augments Stage 1 with additional synthetic reasoning data to test whether data volume alone explains Stage 2’s improvement.Stage 1 uses supervised imitation and InfoNCE.
  • S3 Analysis of Scaling Stage 1 Training Data: A 10× increase in Stage 1 data strengthens the supervised baseline but fails to reproduce Stage 2’s gains.The added synthetic reasoning data, generated with Qwen3-VL-30B, matches the volume used in Stage 2.

S4 Effect of Scaling the Video Search Engine

This section examines whether HQR’s limited effectiveness stems from insufficient responsiveness to textual refinement by scaling Qwen3-VL-Embedding from 2B to 8B and evaluating retrieval and Stage 1 training.

  • S4 Effect of Scaling the Video Search Engine: The 8B search engine is substantially stronger than the 2B model in zero-shot retrieval and Stage 1 evaluation.The comparison scales Qwen3-VL-Embedding-2B to Qwen3-VL-Embedding-8B.
  • S4 Effect of Scaling the Video Search Engine: The analysis tests whether HQR’s limited effectiveness is caused by the video search engine’s inadequate response to textual refinement instructions.Both zero-shot retrieval and Stage 1 training are evaluated after scaling the search engine.

S5 Effect of Scaling the SQR Module

Scaling VideoSearch-R1’s SQR module from 2B to 4B consistently improves VCMR, VER, and VR performance. The advantage is less evident zero-shot but clearer after task-specific training, suggesting a more effective refinement policy.

  • S5 Effect of Scaling the SQR Module: The 4B model’s advantage is less evident zero-shot, likely because it behaves more conservatively.
  • S5 Effect of Scaling the SQR Module: Scaling the SQR module from 2B to 4B consistently improves performance across VCMR, VER, and VR on DiDeMo-FIG.Table 5 evaluates the effect of scaling VideoSearch-R1 on DiDeMo-FIG.
  • S5 Effect of Scaling the SQR Module: After task-specific training, the larger SQR module shows clearer gains, indicating a more effective refinement policy and stronger overall performance.

S6 Implementation Details for HQR Training

HQR is trained with cold-start targets from hard-query rewriting because it lacks a unique ground-truth rewritten query, while otherwise matching SQR’s training setup for fair comparison.

  • S6 Implementation Details for HQR Training: HQR uses refined queries generated by Qwen3-VL-30B, together with their reasoning paths, as cold-start training targets.The approach addresses HQR’s lack of a unique ground-truth rewritten query for supervision.
  • S6 Implementation Details for HQR Training: HQR follows the hard-query rewriting scheme that produced the strongest retrieval improvement in the analysis.
  • S6 Implementation Details for HQR Training: For fair comparison, HQR keeps SQR’s overall training setup but replaces soft query tokens with explicit textual query rewrites.

S7 Cross-dataset Generalization

VideoSearch-R1 learns transferable query refinement across video corpora, improving zero-shot transfer performance over Qwen3-VL-2B in both evaluation directions without target-dataset fine-tuning.

  • Cross-dataset transfer: VCMR 0.5/R@1 rises from 10.6 to 17.3 when transferring from Charades-FIG to DiDeMo-FIG.This comparison is against the zero-shot Qwen3-VL-2B baseline.
  • Cross-dataset transfer: VCMR 0.5/R@1 rises from 7.2 to 9.3 when transferring from DiDeMo-FIG to Charades-FIG.This comparison is against the zero-shot Qwen3-VL-2B baseline.
  • Cross-dataset transfer: VideoSearch-R1 consistently improves over the zero-shot Qwen3-VL-2B baseline in both transfer directions, with corresponding VR gains indicating effective cross-corpus refinement.The model is trained on one video corpus and evaluated on another without target-dataset fine-tuning.

S8 Cross-task Evaluation: VideoQA

VideoSearch-R1 transfers its iterative retrieval-and-reasoning framework beyond temporal grounding to retrieval-conditioned VideoQA on IntentQA, where it retrieves a relevant video before answering the question. On this evaluation, VideoSearch-R1 raises VideoQA accuracy from 39.4 to 57.3.

  • Cross-task Evaluation: VideoQA: 57.3 VideoQA accuracy, up from 39.4, demonstrates VideoSearch-R1’s performance on IntentQA.The evaluation uses retrieval-conditioned VideoQA: the model first retrieves a relevant video from the corpus, then answers the question from the retrieved content.
  • Cross-task Evaluation: VideoQA: VideoSearch-R1 applies iterative retrieval and reasoning to VideoQA beyond its temporal-grounding setting.The cross-task evaluation tests whether the framework transfers to a different intra-video reasoning task.
  • Cross-task Evaluation: VideoQA: On IntentQA, the model retrieves a relevant corpus video before answering the given question from its content.

S9 Comparison between HQR and SQR: A Case Study

The section qualitatively compares SQR with HQR across video-retrieval cases involving fine-grained actions and attributes. The examples report differing ground-truth video ranks while showing how hard query refinement focuses rewritten queries on selected details.

  • Comparison between HQR and SQR: A Case Study: For the crowd query, hard refinement explicitly focuses retrieval on a woman with a short ponytail while maintaining the stage-gazing context.The example contrasts presence and missing-attribute assessments for crowd, stage, ponytail, rising, and gazing details.
  • Comparison between HQR and SQR: A Case Study: For the trolley-train query, hard refinement focuses on the train while retaining its emergence from the right and stopping at the station.The assessments distinguish whether the train, direction, emergence, and stopping-at-station details are present or missing.
  • Comparison between HQR and SQR: A Case Study: Additional examples apply hard refinement to a lady in red turning toward a gate and a baby standing, touching a bench, and smiling.The corresponding assessments separately track contextual people, clothing, turning, standing, touching, and smiling attributes.
  • Comparison between HQR and SQR: A Case Study: A wedding-scene example concerns a groom embracing a priest while two other individuals are present, with assessments indicating which components are retrieved.The hard-refinement query instead focuses on the two additional individuals watching the embrace.
Loading 2607.00446v1…