PaperlayerView exact revision on arXiv

Source-linked AI summary

RANKVIDEO: Reasoning Reranking for Text-to-Video Retrieval

Tyler Skow, Alexander Martin, Benjamin Van Durme, Rama Chellappa, Reno Kriz

arXiv:2602.02444v2cs.IRcs.CV

TL;DR

Video retrieval remains challenging because effective systems require strong multimodal representations and audiovisual-text alignment, while extracted text can omit critical information or be unavailable and costly to generate. RANKVIDEO uses video-native query–video reasoning and a two-stage training process for reranking, improving MultiVENT 2.0 retrieval performance by an average of 31% on nDCG@10 while scoring faster than existing reasoning-based baselines.

  • Problem

    Video retrieval is challenging because systems need strong multimodal representations and audiovisual-text alignment, while extracted text may omit critical information, be unavailable, or be expensive to generate.

  • Method

    RANKVIDEO predicts query–video relevance from audiovisual inputs using a two-stage process combining perception-grounded SFT with pointwise, pairwise, and distillation-based reranking objectives.

  • Results

    31% average improvement on nDCG@10 was achieved on MultiVENT 2.0 across various first-stage retrievers, with faster scoring than existing reasoning-based reranking baselines.

  • Takeaways & Limitations

    RANKVIDEO is especially effective for queries and videos containing visually anchorable events, such as mining dump trucks and SpaceX operational missions.

  • Takeaways & Limitations

    List-wise reranking was not explored because multivideo inference was too computationally costly for the available training setup.

Abstract

from arXiv · show

Reranking is a critical component of modern retrieval systems, which typically pair an efficient first-stage retriever with a more expressive model to refine results. While large reasoning models have driven rapid progress in text-centric reranking, reasoning-based reranking for video retrieval remains underexplored. To address this gap, we introduce RANKVIDEO, a reasoning-based reranker for video retrieval that explicitly reasons over query-video pairs using video content to assess relevance. RANKVIDEO is trained using a two-stage curriculum consisting of perception-grounded supervised fine-tuning followed by reranking training that combines pointwise, pairwise, and teacher confidence distillation objectives, and is supported by a data synthesis pipeline for constructing reasoning-intensive query-video pairs. Experiments on the large-scale MultiVENT 2.0 benchmark demonstrate that RANKVIDEO consistently improves retrieval performance within a two-stage framework, yielding an average improvement of 31% on nDCG@10 and outperforming text-only and vision-language reranking alternatives, while more efficient.

1 Introduction

Text-to-video retrieval must scale to large multimodal collections, yet video reranking remains underexplored and often depends on incomplete extracted text. RANKVIDEO addresses this with video-native reasoning, curriculum training, and synthesized reasoning-intensive data.

  • Video retrieval must rank large collections while handling multimodal representations, cross-modal alignment, and scalability challenges.
  • Two-stage retrieval pairs an efficient first-stage retriever with a more expressive reranker that refines a small candidate set.This reduces the number of expensive query-candidate comparisons at query time.
  • Extracted text can omit critical visual or audio information, may be unavailable, and can be computationally expensive to generate.
  • RANKVIDEO directly uses audiovisual inputs to score query-video relevance by comparing answer-token log-probabilities, without requiring reasoning traces.
  • The model combines perception-grounded SFT with pointwise classification, pairwise ranking, and teacher distillation, supported by synthesized reasoning-intensive query-video pairs.
  • 31% average improvement on nDCG@10 is achieved across diverse first-stage retrievers, while RANKVIDEO remains faster than existing reasoning-based reranking baselines.The model also allocates deeper reasoning only when necessary, improving efficiency without sacrificing performance.

2 Related

The related work situates RANKVIDEO at the intersection of large reasoning models, two-stage reranking, and large-scale video retrieval. Existing video retrieval datasets and methods have struggled to represent real-world, reasoning-intensive retrieval at scale.

  • Large Reasoning Models: Large reasoning models perform multistep reasoning by generating intermediate rationales or allocating additional inference computation.Their test-time computational cost remains a recognized concern.
  • Reranking: Neural information retrieval commonly separates a fast, high-recall first-stage retriever from a more expressive reranker operating on a small candidate subset.
  • Text-to-Video Retrieval: MULTIVENT 2.0 introduced a large-scale, reasoning-intensive video retrieval dataset because prior caption-derived datasets were small and state-of-the-art methods struggled with 100k+ videos or real-world queries.

3 Data Synthesis

The data synthesis pipeline constructs challenging video-retrieval supervision by extracting multimodal representations, filtering for difficult and discriminative queries, and retaining a large set of positive and negative pairs.

  • The pipeline generates video representations from captions, transcribed audio, OCR, and additional video-content features to create high-level reasoning-intensive queries.
  • Queries are filtered when their positive video is absent from the first 1000 OMNIEMBED candidates or when hard-negative separation is insufficient.
  • The final dataset contains 35684 records, including 9267 unique positive query-video pairs and 26258 negative query-video pairs.Each query has 3.85 candidates on average, with most queries containing three negative samples.

4 RANKVIDEO Two-Stage Training

RANKVIDEO uses a perception-grounded cold-start stage before ranking-specific fine-tuning, which combines hard-negative mining with pointwise, pairwise, and teacher-guided objectives. Candidates are scored from yes/no logits within query-grouped batches to improve relevance ranking.

  • Stage 1: Perception Cold Start SFT: RANKVIDEO first learns to generate captions grounded in video content, producing a perception-grounded initialization for ranking fine-tuning.Captioning supplies dense supervision for salient video entities before the model learns the retrieval objective.
  • Stage 2: Ranking Finetuning: Hard-negative mining retains trusted and ambiguous negatives while dropping suspected positives identified by high-margin teacher judgments.Ambiguous candidates are retained because they resemble positives under the first-stage retriever and contribute to reranking errors.
  • Stage 2: Ranking Finetuning: Stage 2 trains on query-grouped batches containing one labeled positive and negatives from the same query.The candidate set is organized around each query so the model learns within-query relevance distinctions.
  • Stage 2: Ranking Finetuning: The reranker uses the difference between yes and no logits as a scalar relevance score instead of decoding generated text.This score provides a stable, monotonic ranking signal and enables fast scoring without long rationales.
  • Stage 2: Ranking Finetuning: Stage 2 combines pointwise accuracy, pairwise ranking, and teacher-probability distillation in a weighted training objective.Teacher distillation transfers calibrated confidence, while pointwise and pairwise losses support calibration and within-query ordering.

5 Experiments

Experiments on MULTIVENT 2.0 show that RANKVIDEO improves reranking across first-stage retrievers, with strong retrieval gains, efficient inference, and performance tied to visually anchorable events.

  • RANKVIDEO and Baselines: RANKVIDEO achieves state-of-the-art retrieval results across all reported metrics when reranking OMNIEMBED candidates.Both training stages improve reranking performance, while the other video-native baselines generally fail to improve the first-stage results.
  • Score Distribution Shift: Stage-2 training separates relevant and non-relevant score distributions by increasing positive margins for relevant pairs and negative margins for non-relevant candidates.The resulting logit-delta scores provide a monotonic ranking signal without decoding long rationales and better target early-rank quality.
  • Stability Across First-Stage Models: RANKVIDEO benefits first-stage retrievers of varying quality and can refine rankings even when the initial candidate list is weaker.This enables faster, less accurate first-stage models to handle large indices while RANKVIDEO refines their ranked lists.
  • Efficiency and Dynamic Reasoning: RANKVIDEO is 2.67s faster than QVL-T in median query latency and operates within 0.15s of ReasonRank, at 0.87s versus 1.02s.Inference uses video frames directly and avoids ReasonRank’s preprocessing latency, with only a small latency gap versus text-based reranking.
  • Query Analysis: Coarse metadata explains only a small fraction of per-query nDCG@10 variance, with held-out Random Forest regression achieving R2 = 0.093.Query length is the most informative individual feature, consistent with shorter or underspecified queries being more difficult.
  • Query Analysis: RANKVIDEO’s score variance is driven more by query–video interaction than by video-level priors, with R2 values of 0.139 for query-only and 0.090 for video-only effects.QVL-R exhibits a much stronger video prior, with R2 = 0.755.
  • Qualitative Results Discussion: RANKVIDEO performs best on queries with visually anchorable events, such as mining dump trucks and SpaceX operational missions, but can struggle on natural disasters.Storm footage may contain generic visuals rather than distinctive cues needed for high-quality reranking.

6 Conclusion

RANKVIDEO is a video-native reasoning reranker trained in two stages for text-to-video retrieval. On MULTIVENT 2.0, it consistently improves retrieval across first-stage retrievers while being faster than existing reasoning-based reranking baselines.

  • RANKVIDEO introduces video-native reasoning reranking for text-to-video retrieval.Its two-stage process uses perception-grounded SFT before reranking fine-tuning with pairwise, pointwise, and distillation objectives.
  • 31% average improvement on nDCG@10 was achieved across various first-stage retrievers on MULTIVENT 2.0.
  • RANKVIDEO was significantly faster than existing reasoning-based reranking baselines.

Limitations

The study does not explore list-wise reranking because multivideo inference is computationally expensive. Training the pairwise objective also required reducing batch size and maximum frames to fit the available hardware.

  • List-wise reranking was not explored because multivideo inference incurs substantial computational costs.
  • The pairwise objective required at most three videos per query and reduced batch size and maximum frames for training on eight 80GB A100s.
  • Making multivideo inference more computationally feasible could reduce this training burden.

A RANKVIDEO Details

The appendix provides implementation settings and prompts for RANKVIDEO and related models, including hard-negative mining thresholds and a summary table of training configurations.

  • Additional training configurations are provided in Table 6.The table summarizes RANKVIDEO training settings for stages 1 and 2.
  • System and user prompts are provided for RANKVIDEO, QVL-I/T, and ReasonRank.
  • Hard-negative mining uses thresholds α1 = −6 and α2 = −8 based on the distribution of query-video pair logit scores.

B Training Loss Ablation

The ablation evaluates the three-part reranking loss and metadata slices to identify which training signals contribute to gains and where retrieval performance varies.

  • The ablation separates pairwise ranking, pointwise calibration, and teacher probability distillation objectives.Results are reported on a 50-query subset of the MULTIVENT 2.0 test set.
  • The pointwise objective produces substantial gains in the ablation.
  • Teacher probability distillation improves nDCG when included.
  • Retrieval performance is broken down by video language, query event type, video type, and video modality.The breakdown reports per-query nDCG@10 aggregated across these metadata slices.

D Disconnect Between Binary Classification and Reranking

Binary relevance-classification quality can diverge from second-stage reranking quality because retrieval reranking is dominated by difficult near-miss negatives rather than easy negatives. RANKVIDEO therefore curates candidate-pool negatives and aligns supervision with the reranking objective.

  • Disconnect Between Binary Classification and Reranking: Strong accuracy and recall as relevance classifiers do not necessarily indicate strong second-stage retrieval results.The observed disconnect reflects that classifier metrics largely capture performance on easy negatives.
  • Disconnect Between Binary Classification and Reranking: Top-k reranking exposes models to semantically or visually plausible hard negatives that can become high-scoring false positives.Accuracy and reranking performance are slightly positively correlated, but higher accuracy does not always produce stronger reranking.
  • Hard-Negative Curation: RANKVIDEO mines negatives from each query’s candidate pool and filters them using teacher confidence to better match reranking supervision.Suspected positives are removed, while trusted and ambiguous negatives are retained under confidence-based criteria.
  • Hard-Negative Curation: The training design shifts learning from separating irrelevant videos toward suppressing retrieval hard negatives while avoiding noisy supervision from unlabeled positives.At least one trusted negative is required per query.
  • Reasoning Efficiency: Reasoning traces show RANKVIDEO can use substantially fewer tokens than ReasonRank and Qwen3-VL-8B-Thinking in example reranking decisions.The examples include both a case where reasoning was unnecessary and a case where RANKVIDEO needed to reason.

F Use of AI Assistants

AI assistants were used to improve writing fluency and support some code development. The section also presents example reasoning traces and prompts for RANKVIDEO and comparison models.

  • Use of AI Assistants: AI assistants were used to improve the fluency of the writing and in some of the code development.
  • Examples: The appendix presents reasoning-trace examples for relevance judgments involving the queries “Emergency response Notre-Dame fire” and “Super Bowl 2023 Philadelphia Eagles.”The examples include predictions, gold labels, and model reasoning outputs.
  • Examples: For the Super Bowl query, all models correctly classify the video as not relevant, while ReasonRank and Qwen3-VL-8B-Thinking use substantially more tokens than RANKVIDEO.
  • Prompts: Prompt figures are provided separately for RANKVIDEO’s two stages, QwenVL models, and REASONRANK.
Loading 2602.02444v2…