PaperlayerView exact revision on arXiv

Source-linked AI summary

X-CoT: Explainable Text-to-Video Retrieval via LLM-based Chain-of-Thought Reasoning

Prasanna Reddy Pulakurthi, Jiamian Wang, Majid Rabbani, Sohail Dianat, Raghuveer Rao, Zhiqiang Tao

arXiv:2509.21559v1cs.CV

TL;DR

Embedding-based text-to-video retrieval provides rankings without explaining them, while low-quality text-video pairs can be difficult to identify. X-CoT adds video annotations and LLM pairwise reasoning to produce explained rankings; the authors report improved retrieval performance and support for model and data-quality analysis.

  • Problem

    Embedding-based retrieval can be undermined by flawed videos or crude captions, which are difficult to track, and cosine-similarity rankings do not explain why candidates rank as they do.

    Source: §1 · p. 1, §3.1 · p. 2

  • Method

    X-CoT expands benchmarks with video annotations and uses LLM pairwise comparisons with textual justifications to rank video candidates.

    Source: §1 · p. 1, §3.3 · p. 3

  • Results

    X-CoT improves retrieval performance while providing explanations, and its explanations support analysis of model behavior and text-video data quality.

    Source: §5 · pp. 4–5, §4.4 · p. 4

  • Takeaways & Limitations

    Explanations can help diagnose semantic factors missed by embedding models and assess whether retrieval errors relate to flaws in text-video data.

    Source: §4.4 · p. 4

    X-CoT’s reasoning and ranking depend heavily on LLM capacity, which may be less effective for domain-specific or highly noisy data, including very long videos.

    Source: Limitations · p. 5

Abstract

from arXiv · show

Prevalent text-to-video retrieval systems mainly adopt embedding models for feature extraction and compute cosine similarities for ranking. However, this design presents two limitations. Low-quality text-video data pairs could compromise the retrieval, yet are hard to identify and examine. Cosine similarity alone provides no explanation for the ranking results, limiting the interpretability. We ask that can we interpret the ranking results, so as to assess the retrieval models and examine the text-video data? This work proposes X-CoT, an explainable retrieval framework upon LLM CoT reasoning in place of the embedding model-based similarity ranking. We first expand the existing benchmarks with additional video annotations to support semantic understanding and reduce data bias. We also devise a retrieval CoT consisting of pairwise comparison steps, yielding detailed reasoning and complete ranking. X-CoT empirically improves the retrieval performance and produces detailed rationales. It also facilitates the model behavior and data quality analysis. Code and data are available at: https://github.com/PrasannaPulakurthi/X-CoT.

1 Introduction

Embedding-based retrieval can be undermined by flawed video-caption pairs and offers rankings without reasons. X-CoT uses LLM reasoning and added video annotations to produce rankings with rationales and support retrieval analysis.

1 Introduction

  • Embedding retrieval is vulnerable to flawed videos and crude captions, which can undermine retrieval and are hard to track.Source: §1 · p. 1
  • Explanations can help users understand rankings, assess retrieval systems, and examine input data quality.The authors also report analyzing embedding-model behavior and identifying inferior text-video data.Source: §1 · pp. 1–2
  • X-CoT replaces cosine-similarity ranking with LLM judgment and pairwise chain-of-thought reasoning, producing rankings with detailed rationales.The framework also expands existing benchmarks with video annotations to support reasoning and reduce raw-video data bias.Source: §1 · p. 1

2 Related Work

Related work centers on embedding-based text-video retrieval and newer multimodal language-model approaches to retrieval and reasoning.

2 Related Work

  • Text-video retrieval has chiefly used joint video-text embedding models, including X-CLIP, Clip4clip, and UMT.Source: §2 · p. 2
  • Multimodal language models support retrieval and reasoning, with work spanning VLM2Vec (Jiang et al., 2024), MM-REACT (Yang et al., 2023), and reasoning-focused benchmark BRIGHT.Source: §2 · p. 2

3 Method

The method augments embedding-based retrieval with structured video annotations and an LLM pairwise-comparison pipeline. It uses the comparisons to produce a ranked list with explanations.

3.1 Preliminaries

  • Embedding systems represent a text query and video candidate as vectors and rank them by cosine similarity, whose scores do not explain why a candidate ranks high or low.Source: §3.1 · p. 2

3.2 Video Annotation Collection

  • The annotation pipeline samples and filters video frames, then uses an MLLM to generate captions that become structured annotations of objects, actions, scenes, and a video summary.Post-processing includes filtering and deduplicating tags, proofing them, and normalizing text.Source: §3.2 · pp. 2–3

3.3 Retrieval CoT

  • X-CoT optionally starts from an embedding model’s small top-K candidate pool, has an LLM compare candidate pairs with binary preferences and textual justifications, then aggregates judgments with Bradley–Terry scores to rank candidates.The aggregation uses maximum likelihood estimation and sorts ability scores in descending order.Source: §3.3 · p. 3

4 Experiment

X-CoT is evaluated on four text-to-video benchmarks using three embedding models, with experiments also examining its ablations, sensitivity to candidate-pool size, and explanations of retrieval behavior.

4.1 Experimental Settings

  • The evaluation covers MSR-VTT, MSVD, LSMDC, and DiDeMo, reporting R@1, R@5, R@10, Median Rank, and Mean Rank.The coarse top-20 list comes from CLIP-ViT-B/32, Qwen2-VL via VLM2Vec, or X-Pool.Source: §4.1 · p. 3

4.2 Performance Comparison

  • X-CoT improves embedding-model retrieval on all reported metrics across four datasets, including +5.6% R@1 for CLIP on MSVD and +1.9% R@1 for X-Pool on MSVD.The comparison uses three embedding models across MSR-VTT, MSVD, DiDeMo, and LSMDC.Source: §4.2 · p. 3, Table 1 · p. 4, Table 2 · p. 4

4.3 Ablation Study

  • Directly ranking top-K candidates without CoT reduces R@1 by 2.9%, while the proposed joint CoT and Bradley–Terry method improves the baseline by 4.5% on R@1.The ablation uses CLIP on DiDeMo with K = 20.Source: §4.3 · pp. 3–4, Table 3 · p. 4

4.4 Model Discussion

  • As K grows, X-CoT identifies and ranks relevant candidates, indicating adaptability to different candidate-pool scales.The discussion reports CLIP performance on DiDeMo.Source: §4.4 · p. 4, Figure 5 · p. 4
  • Explanations can reveal semantic factors missed by embedding models and help identify caption flaws that contribute to incorrect retrievals.The paper illustrates both retrieval-model behavior analysis and text-video data-quality assessment.Source: §4.4 · p. 4

5 Conclusion

The paper concludes that X-CoT replaces cosine-similarity ranking with LLM chain-of-thought reasoning and pairwise comparisons, improving retrieval while producing explanations. Its examples show how explanations can also support analysis of model behavior and text-video data quality.

5 Conclusion

  • X-CoT combines additional benchmark video annotations with pairwise chain-of-thought reasoning to produce explanations and rankings.The framework replaces embedding-model cosine-similarity ranking.Source: §5 · p. 4
  • In a highly similar-video example, X-CoT identifies the relevant video and explains subtle differences where X-Pool fails to rank it.The comparison illustrates analysis of retrieval-model behavior.Source: Figure 6 · p. 5
  • Jointly inspecting the caption, relevant video, and reasoning can expose ambiguous or minor caption claims that introduce retrieval noise.The example identifies object ambiguity and a minor stop-sign claim.Source: Figure 7 · p. 5
  • The paper reports that X-CoT improves retrieval performance while providing explanations, supporting its potential for interpretable multimodal retrieval.The conclusion frames this as a potential, not a demonstrated general guarantee.Source: §5 · p. 5

Limitations

The paper notes that X-CoT depends on LLM capability and that binary Bradley–Terry outcomes omit uncertainty or graded reasoning strength. The supplied appendix passages also describe frame filtering and a comparison of annotation-only CLIP input with X-CoT.

Limitations

  • LLM capacity may limit X-CoT in domain-specific or highly noisy data, including very long-video comprehension.The authors identify these as scenarios where modern LLMs may be less effective.Source: Limitations · p. 5
  • Bradley–Terry aggregation uses binary win/loss outcomes and does not capture uncertainty or nuanced LLM reasoning strength.The authors suggest soft confidence scores or learnable aggregation strategies as possible future directions.Source: Limitations · p. 5

A Similar Frame Filtering

  • Frame filtering retains a frame when its maximum cosine similarity to previously retained frames is below a threshold such as 0.95.The sequential procedure aims to produce non-duplicate frames and promote annotation diversity.Source: A Similar Frame Filtering · p. 8

B Structured Video Annotations as Input: CLIP vs. X-CoT

  • Using structured annotations as CLIP input yields lower performance than X-CoT, suggesting LLM reasoning is needed to exploit long, verb-rich context.The comparison is reported for Table 4; the passage does not provide its numerical results.Source: B Structured Video Annotations as… · p. 8

C Robustness to Noisy Annotations

On MSR-VTT, X-CoT shows only a small performance decline after 20% of structured annotation tags are perturbed, while complete annotations improve performance. Figure 8 provides an example of the collected annotations.

C Robustness to Noisy Annotations

  • Perturbing 20% of structured annotation tags on MSR-VTT causes a small performance decline, indicating robustness to annotation noise in this test.The authors also report improved performance with the complete annotations.Source: C Robustness to Noisy Annotations · p. 8
  • Complete annotations improve performance relative to the perturbed-annotation condition.The passage reports this result for the MSR-VTT experiment.Source: C Robustness to Noisy Annotations · p. 8
  • Figure 8 shows an example of collected annotations.Source: Figure 8 · p. 8

D Additional Qualitative Video Annotation Examples

Structured annotations reveal mismatches in dataset captions and add fine-grained details about people, objects, actions, and scene attributes. The paper presents these annotations as foundational to X-CoT’s reasoning and retrieval reliability.

E Quantitative Evaluation of Video Annotations

The annotation evaluation compares explanation embeddings with baseline and X-CoT-ranked video embeddings, while the ranking algorithm uses local pairwise judgments and global aggregation. Its description also reports caching savings and identifies established frameworks for future explanation-quality evaluation.

E Quantitative Evaluation of Video Annotations

F X-CoT Pairwise Ranking Algorithm

G Efficiency and Scalability

The paper describes engineering strategies for reducing X-CoT’s runtime and reports that runtime falls as GPU parallelism increases. It also illustrates a query whose matching protest video is recovered through pairwise reasoning and aggregation.

G Efficiency and Scalability

H X-CoT Ranking Examples

  • For the Brazil protest query, pairwise reasoning and global aggregation re-rank a video of a protester speaking to a reporter as the accurate match.Source: H X-CoT Ranking Examples · p. 10

I Embedding Model Details and Complete Benchmarking Results

The benchmarking section identifies three retrieval models and four video-text datasets, with results organized across two tables. A separate example depicts a successful X-CoT ranking through stepwise pairwise comparisons and natural-language justifications.

Loading 2509.21559v1…