PaperlayerView exact revision on arXiv

Source-linked AI summary

ReTrack: Evidence-Driven Dual-Stream Directional Anchor Calibration Network for Composed Video Retrieval

Zixu Li, Yupeng Hu, Zhiwei Chen, Qinlei Huang, Guozhi Qiu, Zhiheng Fu, Meng Liu

arXiv:2604.17898v1cs.CV

TL;DR

CVR must combine a reference video with modification text to retrieve a target, yet information-density differences can bias composed features toward the reference. ReTrack calibrates this directional bias through modality disentanglement, composition geometry calibration, and evidence-driven alignment, achieving state-of-the-art performance across CVR and CIR benchmarks.

  • Problem

    CVR combines reference-video and modification-text semantics, but existing methods can bias composed features toward the reference because the modalities differ substantially in information density.

  • Method

    ReTrack disentangles modality contributions, builds calibrated directional anchors, reconstructs composed features, and derives bidirectional evidence for reliable composed-to-target similarity estimation.

  • Results

    ReTrack achieves state-of-the-art performance across three benchmark datasets covering both CVR and CIR tasks.

  • Takeaways & Limitations

    Calibrating directional bias and using evidence-driven alignment improves multimodal query understanding across video and image retrieval settings.

Abstract

from arXiv · show

With the rapid growth of video data, Composed Video Retrieval (CVR) has emerged as a novel paradigm in video retrieval and is receiving increasing attention from researchers. Unlike unimodal video retrieval methods, the CVR task takes a multi-modal query consisting of a reference video and a piece of modification text as input. The modification text conveys the user's intended alterations to the reference video. Based on this input, the model aims to retrieve the most relevant target video. In the CVR task, there exists a substantial discrepancy in information density between video and text modalities. Traditional composition methods tend to bias the composed feature toward the reference video, which leads to suboptimal retrieval performance. This limitation is significant due to the presence of three core challenges: (1) modal contribution entanglement, (2) explicit optimization of composed features, and (3) retrieval uncertainty. To address these challenges, we propose the evidence-dRivRn dual-sTream diRectionAl anChor calibration networK (ReTrack). ReTrack is the first CVR framework that improves multi-modal query understanding by calibrating directional bias in composed features. It consists of three key modules: Semantic Contribution Disentanglement, Composition Geometry Calibration, and Reliable Evidence-driven Alignment. Specifically, ReTrack estimates the semantic contribution of each modality to calibrate the directional bias of the composed feature. It then uses the calibrated directional anchors to compute bidirectional evidence that drives reliable composed-to-target similarity estimation. Moreover, ReTrack exhibits strong generalization to the Composed Image Retrieval (CIR) task, achieving SOTA performance across three benchmark datasets in both CVR and CIR scenarios. Codes are available at https://github.com/Lee-zixu/ReTrack

1 Introduction

CVR retrieves target videos using reference-video and modification-text queries, but existing composition methods can be biased toward the reference video. ReTrack addresses this bias by disentangling modality contributions, calibrating composition geometry, and using evidence-driven alignment.

  • CVR retrieves a target video from a large database using a reference video and modification text as a multimodal query.
  • Existing methods often overweight the reference video because video contains richer information than concise modification text.This produces high similarity to the reference and low similarity to the modification text.
  • Directional bias can make the positive target’s similarity resemble that of negative candidates, degrading retrieval accuracy.
  • ReTrack calibrates composed features with dual-stream directional anchors to integrate cross-modal semantics and improve candidate discriminability.
  • ReTrack combines Semantic Contribution Disentanglement, Composition Geometry Calibration, and Reliable Evidence-driven Alignment to address contribution entanglement, feature optimization, and retrieval uncertainty.The framework separates modality contributions, reconstructs the composed feature, and uses bidirectional evidence for reliable similarity estimation.

2 Related Work

Prior work established CVR through pretrained visual-linguistic models and enriched captions, while uncertainty-estimation research developed alternative approaches with different computational trade-offs.

  • CVR research adapts pretrained visual-linguistic models such as BLIP and BLIP-2 with simple composition mechanisms for multimodal query understanding.
  • Enriched captions were later used to enhance CVR query semantics.
  • Uncertainty-estimation research includes Bayesian Neural Networks, but their posterior approximations can incur high computational costs and slow inference.
  • Evidential Deep Learning models uncertainty through network outputs as an alternative to Bayesian approaches.

3 ReTrack

ReTrack calibrates directional bias in composed CVR features through disentangled modality contributions, geometric directional anchors, and evidence-driven alignment. Its pipeline explicitly models contributions, aligns composition geometry toward targets, and estimates similarity reliability.

  • ReTrack comprises Semantic Contribution Disentanglement, Composition Geometry Calibration, and Reliable Evidence-driven Alignment.
  • 3.1 Problem Formulation: CVR embeds a reference video, modification text, and target video, optimizing a shared metric space that brings the multimodal query close to its target.
  • 3.2 Semantic Contribution Disentanglement: Semantic Contribution Disentanglement extracts reference, modification, and composed features, then separately interacts the composed feature with each modality to isolate contributions.
  • 3.3 Composition Geometry Calibration: Composition Geometry Calibration learns channel-wise point weights, constructs reference and modification anchors, and aligns composition direction toward the target feature.
  • 3.3 Composition Geometry Calibration: The true directional vector At = (Ft − Fc) guides the composition anchor toward the target, reducing directional bias in the composed feature.
  • 3.4 Reliable Evidence-driven Alignment: Reliable Evidence-driven Alignment computes bidirectional evidence between directional anchors and target features, weighting credible samples to improve composed-to-target similarity reliability.

4 Experiments

Experiments evaluate ReTrack across CVR and CIR benchmarks, ablations, hyperparameter sensitivity, and qualitative retrieval cases. ReTrack achieves leading results and benefits from disentanglement, calibration, and evidence-driven alignment.

  • Performance Comparison: ReTrack achieves the best performance across all evaluation metrics on both CVR datasets, including a 2.50% improvement in WebVid-CoVR’s mean metric.The evaluation uses standard Recall@k metrics across WebVid-CoVR, FashionIQ, and CIRR.
  • Performance Comparison: ReTrack attains the best performance across both CIR datasets, with relative R@10 gains of 1.54%, 7.7%, and 0.88% on FashionIQ categories and a 0.73% R@1 gain on CIRR.These results support generalization from video retrieval to image retrieval.
  • Ablation Study: Removing either modality’s semantic contribution reduces performance, while removing Semantic Contribution Disentanglement produces the largest decline within its ablation group.The findings indicate that both visual and textual contributions are needed for effective calibration and retrieval.
  • Ablation Study: Removing distance guidance, either directional anchor, or evidence-driven regularization reduces performance, with full evidence regularization causing the sharpest decline in its group.Replacing exponential evidence computation with ReLU or Softplus also shows that the adopted evidence-theory-compliant exponential method is optimal.
  • Sensitivity Analysis: Performance on WebVid-CoVR and CIRR first increases and then decreases as κ and λ grow, indicating that balanced calibration and evidence weighting are required.Excessively large λ values can overemphasize channels lacking reliable semantic information.
  • Case Study: ReTrack retrieves the target at rank 1 in the WebVid-CoVR case and ranks the CIRR target first, whereas CoVR-2 ranks it third in the latter case.The cases associate ReTrack’s behavior with reduced background interference and more complete capture of low-uncertainty modification requirements.

5 Conclusion

The conclusion presents ReTrack as a CVR framework that corrects directional bias through modality-specific contributions and evidence-based similarity estimation. It reports state-of-the-art performance across CVR and CIR benchmarks and identifies multi-turn retrieval as future work.

  • Conclusion: ReTrack corrects directional bias by computing modality-specific semantic contributions and using calibrated directional anchors to generate bidirectional evidence.This supports reliable composed-to-target similarity estimation.
  • Conclusion: ReTrack is compatible with CIR and achieves state-of-the-art performance on three benchmark datasets covering both CVR and CIR tasks.The conclusion frames this as the method’s reported cross-task outcome.
  • Conclusion: Future work will extend ReTrack to multi-turn interactive Composed Multi-modal Retrieval.

A.1 CVR Datasets

WebVid-CoVR is a large-scale benchmark designed specifically for Composed Video Retrieval. It contains diverse CVR triplets and a curated test set for evaluation.

  • WebVid-CoVR: WebVid-CoVR contains approximately 1.6 million CVR triplets spanning around 131k unique videos and 467k distinct modification texts.Each video averages 16.8 seconds, and each modification text averages approximately 4.8 words.
  • WebVid-CoVR: The dataset’s test set contains 2,500 high-quality triplets selected after annotation and noise removal.The test set is derived from WebVid-10M.

A.2 CIR Datasets

FashionIQ and CIRR provide complementary Composed Image Retrieval benchmarks spanning fashion-specific and complex natural-scene domains. Together they support evaluation across varied multimodal retrieval settings.

  • FashionIQ: FashionIQ contains 77,684 online images paired into 30,134 annotated triplets across dresses, shirts, and T-shirts.It evaluates multimodal image retrieval in the fashion domain through image–modification-text relationships.
  • CIRR: CIRR contains 36,554 annotated triplets and 21,552 images from real-world scenes derived from NLVR2.It emphasizes interactions among multiple objects and includes a fine-grained contrastive subset.
  • Dataset Scope: The datasets collectively evaluate ReTrack across CVR and CIR tasks, including web-sourced video, fashion-specific images, and complex natural scenes.

B Derivation of Evidence Theory

ReTrack models matching uncertainty by representing modality-channel evidence and fusing it across matching hypotheses. Dirichlet, Dempster-Shafer, and Subjective Logic formulations support reliable similarity estimation between directional anchors and target features.

  • Dirichlet-based hypothesis probability estimation: Dirichlet parameters accumulate belief masses from multiple evidence sources to represent hypothesis probability distributions.α0 provides an initial balanced belief, while each αi incorporates support assigned to hypothesis Ai.
  • Matching hypothesis space: The matching hypothesis space contains possible configurations between composed features and candidate target features.Each hypothesis Ai represents a possible matching configuration, and its basic probability assignment reflects support from available evidence.
  • Evidence fusion: ReTrack decomposes composed-to-target matching into two anchor-based spaces and treats Q channels as independent evidence sources.Dempster’s rule fuses evidence from the directional-anchor channels, which encode distinct semantic information.
  • Evidence fusion: Dempster’s rule combines channel evidence while tracking conflict among sources through K.K=1 denotes complete conflict, in which evidence cannot be fused for the matching hypothesis.
  • Subjective Logic: Subjective Logic represents channel-level confidence and uncertainty through belief vectors derived from similarity-based evidence.The evidence is the similarity between a directional-anchor channel’s semantic vector and the target feature; belief mass and uncertainty are complementary.
  • Reliable alignment: Reliable evidence computed from directional anchors enhances similarity-matrix reliability and reduces alignment uncertainty.The resulting alignment objective uses evidence across the two anchor sets and target features.

C.1 Comprehensive Performance Comparison on CIR and CVR Tasks

ReTrack is compared with representative models across CIR and CVR using a joint performance view. It occupies the upper-right region, indicating leading R@1 performance on both tasks.

  • Cross-task comparison: ReTrack lies in the upper-right region of the R@1 comparison, indicating leading performance on both CIRR and WebVid-CoVR.The horizontal axis reports CIRR R@1 and the vertical axis reports WebVid-CoVR R@1.
  • Cross-task comparison: ReTrack’s advantage over CoVR Enrich is larger on CIRR than on WebVid-CoVR.The comparison concerns models applicable to both Composed Video Retrieval and Composed Image Retrieval.

C.2 Efficiency Evaluation

The efficiency evaluation examines resource consumption, training time, throughput, and inference latency alongside retrieval performance. ReTrack adds training and compute costs but maintains CoVR-2-level inference efficiency while improving retrieval accuracy.

  • Evaluation scope: The efficiency evaluation measures computational resources, training time, processing throughput, and end-to-end response time.The experiments target real-world performance under resource-constrained settings.
  • Parameter count: ReTrack and its reference-anchor ablation add approximately 3M parameters over CoVR-2 without a noticeable performance penalty.The comparison uses parameter counts, training time, and inference latency on WebVid-CoVR.
  • Training time: ReTrack’s standard model incurs about +1.5s per iteration in training, while some variants are almost on par with CoVR-2.The higher training time is attributed to added calibration and regularization steps.
  • Compute resources: ReTrack and its variants consume more compute than CoVR-2 because they process richer multimodal semantics, directional anchors, and evidence-driven regularization.These costs accompany disentanglement, anchor construction and calibration, and evidence-based optimization.
  • Overall trade-off: ReTrack maintains CoVR-2-level inference efficiency while providing markedly stronger retrieval performance.Across parameter count, inference latency, training time, and resource usage, the authors characterize this as a favorable cost–performance trade-off.

D Algorithm of Retrack’s Training Procedure

The training procedure jointly optimizes ReTrack’s disentanglement, calibration, and evidence-driven alignment modules. The pseudocode is intended to make their joint optimization reproducible.

  • Training procedure: The pseudocode provides a full and reproducible description of ReTrack’s training procedure.It complements the methodology by specifying how the modules are jointly optimized.
  • Joint optimization: ReTrack jointly optimizes Semantic Contribution Disentanglement, Composition Geometry Calibration, and Reliable Evidence-driven Alignment.The procedure combines these modules during training rather than presenting them as isolated operations.

E More Case Study

Qualitative case studies across WebVid-CoVR, CIRR, and FashionIQ show ReTrack handling complex semantic and directional changes, while failures mainly reflect dataset ambiguity, missing annotations, or limited coverage.

  • WebVid-CoVR: ReTrack successfully captures seasonal changes from “autumn” to “spring” in WebVid-CoVR, unlike competing models that retain reference-video content.The successful retrieval is attributed to Composition Geometry Calibration and directional anchors.
  • WebVid-CoVR: None of the models retrieves the correct cappuccino result, but ReTrack still returns milk- or coffee-related scenes.The case suggests directional understanding despite annotation inconsistency or visual ambiguity.
  • CIRR: ReTrack adjusts bird orientation and background elements in CIRR, whereas other models retrieve visually similar but directionally incorrect images.The result is attributed to direction-oriented calibration refining the compositional semantic direction.
  • CIRR: CIRR failures arise when highly specific viewpoints or instances may be absent from the dataset, indicating insufficient sample coverage or incomplete annotations.The sliding-doors query failed for all models under this condition.
  • FashionIQ: ReTrack retrieves white evening dresses for FashionIQ queries, reflecting extraction and integration of modification semantics through directional reconstruction.The cited case concerns the repeated modification “is white and more evening and is white.”
  • FashionIQ: FashionIQ lacks detailed annotations for textual graphics and high-resolution logos, causing all models to fail on such differences while ReTrack remains closer in color and style.This indicates partial capability under annotation limitations rather than complete success.
  • Overall findings: Across all three datasets, ReTrack outperforms its ablated variant and baseline on retrieval involving complex semantics or directional changes.The case studies support the reported value of Direction-oriented Calibration and Evidence-driven Alignment for modality bias and semantic ambiguity.
Loading 2604.17898v1…