Source-linked AI summary
Learning Sample-wise Rank-aware Interpolation Weights for Composed Visual Data Retrieval
Boseung Jeong, Taegyu Park, Donghyeon Kwon, Hyunsouk Cho, Suha Kwak
TL;DR
Composed visual retrieval needs to fuse reference visuals and textual modifications, but optimal interpolation weights are difficult to supervise because they depend on discrimination from hard negatives. SRAIN learns sample-specific weights using rank-aware batch estimation, conditioned prediction, and memory-bank negatives, achieving state-of-the-art composed video retrieval and matching the best composed image retrieval models while substantially reducing query-time latency.
Problem
Estimating optimal interpolation weights is difficult because they must balance target proximity with discrimination from hard negatives, and direct evaluation across potential weights is computationally intractable.
Method
SRAIN predicts sample-specific interpolation weights using batch-wise rank-aware estimation, hard-negative-conditioned prediction, and a two-stage training strategy.
Results
SRAIN achieves state-of-the-art performance on WebVid-CoVR, competitive results on CIRCO and FashionIQ, and substantially reduces query-time latency.
Takeaways & Limitations
Efficient linear interpolation can provide strong composed image and video retrieval performance without reliance on external language models.
Abstract
from arXiv · showhide
At the heart of composed visual data retrieval is the fusion of a reference visual input and a textual modification into a single query. While current state-of-the-art methods utilize multimodal large language models for this fusion, their complexity introduces prohibitive querytime latency, limiting their scalability. We instead revisit the efficacy of simple linear interpolation within an embedding space, and introduce SRAIN, the first framework that dynamically predicts query-specific interpolation weights. The key challenge lies in the fact that the quality of an interpolation weight should be measured by the interpolated embedding's discriminability from negatives as well as its proximity to true targets; this makes collecting and predicting optimal weights intractable. We overcome this bottleneck through two key innovations: batch-wise rank-aware weight estimation during training, and a compact memory bank that synthesizes hard negatives during inference. SRAIN achieves the best in composed video retrieval and matches the current state of the art in composed image retrieval, all while substantially reducing querytime latency compared to MLLM-based alternatives.
1 Introduction
SRAIN replaces costly MLLM-based query fusion with sample-specific linear interpolation, while addressing the difficulty of estimating weights that rank true targets above hard negatives. Its rank-aware estimation, conditioned prediction, and two-stage training support efficient retrieval with strong benchmark performance.
- Composed retrieval identifies a target visual instance from a reference visual input and textual modification, improving search specificity and supporting media-editing applications.
- MLLM-based fusion improves query composition but substantially increases query-time latency, while fixed dataset-level interpolation weights are highly sub-optimal for instance-specific edits.
- SRAIN dynamically predicts a per-sample interpolation weight to fuse reference and modification embeddings through efficient linear interpolation without external MLLMs.
- The optimal weight must rank true targets above hard negatives rather than merely maximize similarity to the positive target, making direct ground-truth collection computationally intractable.
- SRAIN estimates near-optimal weights from batch-wise ranking behavior and conditions prediction on hard negatives, with a memory bank supplying representative target prototypes during inference.
- SRAIN uses two-stage training: encoders are first fine-tuned with hard-negative contrastive learning, then frozen while a lightweight predictor regresses interpolation weights.
- SRAIN achieves state-of-the-art CoVR performance, matches the best CoIR model, and substantially reduces query-time latency without external models.
2 Related Work
Prior composed retrieval methods use cross-attention, textual or multimodal language-model fusion, and related training strategies, but large-model approaches incur substantial inference latency. SRAIN instead uses lightweight interpolation-based fusion without heavy inference or cross-modal modules.
- Composed image retrieval methods commonly integrate visual and textual cues with cross-attention mechanisms.
- MLLM- and LLM-based composed image retrieval methods provide strong generalization or query composition but require computationally intensive inference and introduce significant query-time latency.
- Composed video retrieval methods build on video-text-video triplets and increasingly capable multimodal backbones, including BLIP and BLIP-2.
- SRAIN addresses the latency limitation with lightweight interpolation-based fusion that avoids heavy inference steps and cross-modal modules.
3 Proposed Method
SRAIN builds composed queries by separately encoding reference visuals and modification text, then predicting sample-specific interpolation weights. It estimates training targets from batch-wise ranking and uses conditioned prediction with a memory bank at inference.
- Embedding extraction: SRAIN separately encodes reference visuals and modification text in a shared embedding space for efficient interpolation without additional fusion modules.The implementation freezes BLIP-2 for target visual data to reduce training and inference cost.
- Interpolation weight estimation: SRAIN uses spherical linear interpolation between reference and modification embeddings across discrete candidate weights in [0, 1].The interpolation coefficients depend on the angle between the reference and modification embeddings.
- Interpolation weight estimation: For each candidate weight, SRAIN compares the fused query with batch targets using cosine similarity and selects the candidate giving the positive target the best rank.Smaller positive-target rank indicates better retrieval performance; ties are handled when multiple candidates attain the minimum rank.
- Two-stage training strategy: SRAIN fine-tunes the Q-Former with hard-negative contrastive loss, then freezes the encoders while training a lightweight predictor to regress the estimated weights.The first stage aligns fused queries with targets and suppresses hard negatives; the second stage predicts interpolation weights from query inputs.
- Weight prediction: The predictor conditions on reference, modification, and target or hard-negative information, while inference substitutes a frozen memory bank of target prototypes.The bank is updated with exponential moving averages during training and supplies frequently retrieved prototypes across candidate weights during inference.
4 Experiments
SRAIN is evaluated on one composed video retrieval benchmark and three composed image retrieval benchmarks, with quantitative comparisons, qualitative examples, and component ablations. It achieves strong retrieval performance while using lightweight interpolation and memory-bank conditioning.
- Experimental Setup: SRAIN is evaluated on WebVid-CoVR, FashionIQ, CIRCO, and CIRR, covering composed video retrieval and composed image retrieval.WebVid-CoVR contains 1.64M training triplets and 2,556 test triplets; the image benchmarks cover fashion, zero-shot generalization, and supervised retrieval.
- Composed Video Retrieval: SRAIN surpasses all existing methods in WebVid-CoVR R@1, exceeding by 0.95% and CoVR and CoVR-2 by 5.12% and 1.25%.The comparisons use different fusion strategies, including averaging, MLP fusion, cross-attention, and linear interpolation.
- Composed Image Retrieval: SRAIN matches the best model on composed image retrieval and surpasses all methods across retrieval metrics on CIRR.FashionIQ results are reported in both zero-shot and fine-tuned settings, while CIRCO evaluates zero-shot retrieval.
- Qualitative Results: Adaptive weights focus more on discriminative modifications and rely more on the reference when textual modifications are visually ambiguous.A discriminative instruction receives a relatively high predicted weight, whereas an ambiguous instruction receives a lower one.
- Ablation Studies: Stage 2-only training improves over averaging, while end-to-end training destabilizes supervision because encoder updates change the estimated ground-truth weights.The ablation also reports gains over a frozen baseline and notes sensitivity to the dataset-dependent fixed interpolation weight.
- Ablation Studies: The memory bank improves zero-shot CIRCO and FashionIQ performance over both removing it and replacing its conditioning vectors with random vectors.These results support the memory bank as meaningful conditioning under domain shift.
- Ablation Studies: SRAIN achieves 56.91% R@1 with the BLIP backbone and outperforms CoVR using the same backbone.Component-wise ablations with BLIP show consistent improvements from the proposed components.
5 Conclusion
The paper presents SRAIN for efficient multimodal fusion in composed image and video retrieval. It combines sample-wise rank-aware interpolation, hard-negative conditioning, and two-stage training, achieving strong retrieval performance with reduced query-time latency.
- 5 Conclusion: SRAIN predicts instance-specific interpolation weights to fuse reference and modification embeddings for composed image and video retrieval.Its rank-aware estimation uses mini-batch ranking behavior, while conditioned prediction references hard negatives.
- 5 Conclusion: SRAIN achieves state-of-the-art performance on WebVid-CoVR and competitive results on CIRCO and FashionIQ while significantly reducing query-time latency.The conclusion attributes the approach to lightweight sample-wise linear interpolation without computationally expensive architectures.