Source-linked AI summary
Reasoning-Augmented Representations for Multimodal Retrieval
Jianrui Zhang, Anirudh Sundara Rajan, Brandon Han, Soochahn Lee, Sukanta Ganguly, Yong Jae Lee
TL;DR
Universal multimodal retrieval remains brittle when queries require implicit reasoning, such as resolving references or matching compositional constraints. This paper externalizes that reasoning with VLM-generated semantic enhancements and reports consistent, substantial gains across M-BEIR when retrievers are trained on the enhanced representations.
Problem
Existing multimodal retrievers struggle to interpret underspecified queries and often match superficial cues instead of the intended semantics.
Method
A VLM externalizes reasoning by captioning visual evidence, resolving ambiguous references, rewriting instructions, and training retrievers on the resulting dense representations.
Results
The method yields consistent, substantial gains across M-BEIR tasks, while inference-time enhancement alone is insufficient without training on the enhanced distribution.
Takeaways & Limitations
The findings support prioritizing scalable semantic augmentation and alignment over greater architectural complexity for more robust multimodal retrieval.
Takeaways & Limitations
FashionIQ and Fashion200k evaluation can penalize visually correct retrievals when ground truth contains only one specific image ID among many semantically equivalent candidates.
Abstract
from arXiv · showhide
Universal Multimodal Retrieval (UMR) seeks any-to-any search across text and vision, yet modern embedding models remain brittle when queries require latent reasoning (e.g., resolving underspecified references or matching compositional constraints). We argue this brittleness is often data-induced: when images carry "silent" evidence and queries leave key semantics implicit, a single embedding pass must both reason and compress, encouraging spurious feature matching. We propose a data-centric framework that decouples these roles by externalizing reasoning before retrieval. Using a strong Vision--Language Model, we make implicit semantics explicit by densely captioning visual evidence in corpus entries, resolving ambiguous multimodal references in queries, and rewriting verbose instructions into concise retrieval constraints. Inference-time enhancement alone is insufficient; the retriever must be trained on these semantically dense representations to avoid distribution shift and fully exploit the added signal. Across M-BEIR, our reasoning-augmented training method yields consistent gains over strong baselines, with ablations showing that corpus enhancement chiefly benefits knowledge-intensive queries while query enhancement is critical for compositional modification requests. We publicly release our code at https://github.com/AugmentedRetrieval/ReasoningAugmentedRetrieval.
1. Introduction
The paper argues that UMR retrievers are brittle because a single embedding pass must infer latent semantics while compressing inputs, especially for underspecified queries. It proposes externalizing this reasoning with VLM-based enhancement and training retrievers on the resulting dense semantic representations.
- Background: UMR supports text-to-image, image-to-text, and image-to-image search through shared multimodal embedding spaces.Contrastive models such as CLIP, SigLIP, and ImageBind exemplify this paradigm.
- Motivation: M-BEIR exposes a persistent gap between retrieval and information seeking because queries may require identifying entities, inferring attributes, and connecting them to corpus evidence.The paper illustrates this with an unfamiliar-building image and the question “Who designed this?”
- Motivation: A single embedding pass is often too demanding because it must select relevant latent aspects and compress them, causing heuristic and spurious matching when semantics remain implicit.This is especially problematic for indirect references, missing attributes, and verbose instructions.
- Method: The enhancement pipeline uses a high-capacity VLM to caption salient visual details, resolve ambiguous multimodal references, and rewrite verbose instructions into concise retrieval constraints.This shifts interpretive burden away from the embedding model and materializes hidden reasoning as text annotations.
- Results: Training retrieval backbones on enhanced representations produces consistent and substantial gains over strong baselines across M-BEIR, beyond inference-time text-quality improvements alone.Corpus enhancement is performed offline once, while query enhancement is available at test time with modest overhead.
2. Related Work
Prior multimodal retrieval evolved from contrastive dual encoders toward VLM-based universal embeddings, while M-BEIR exposes brittleness from independent compression and underspecified intent. This work complements architectural and training advances by externally increasing input semantic density for explicit matching.
- Foundational multimodal retrieval: Dual-encoder systems align image and text representations in a shared embedding space using contrastive objectives, with CLIP and SigLIP exemplifying this paradigm.UniIR later introduced instruction-guided universal retrieval and established M-BEIR for realistic information-seeking queries across heterogeneous tasks.
- Foundational multimodal retrieval: Independent modality processing and single-vector compression can fail on mixed-modal interpretation and latent reasoning, especially when intent is underspecified.Examples include resolving “this” or inferring the intended attribute in “Who designed this?”, where shallow correlations may dominate.
- VLM-based universal embeddings: VLM-based universal embeddings enable multimodal inputs and richer cross-modal interactions before compression, with E5-V demonstrating transfer from predominantly text-pair supervision.VLM2Vec is described as showing benefits from explicitly multimodal supervision, although the supplied passage is truncated.
- Data-centric and task-specialized retrieval: Data-centric retrieval improves representations through modality-specific mechanisms or altered training data, complementing architectural approaches such as document-centric visual retrieval.The supplied passage identifies ColPali as an example before describing this approach’s lightweight semantic-density refinement.
- Data-centric and task-specialized retrieval: This approach externally resolves ambiguities, captions salient visual evidence, and rewrites verbose instructions into retrieval-oriented constraints for explicit semantic matching.It shifts implicit reasoning from the embedding model to an external VLM and complements prior architectural and training-strategy improvements.
3. Method
The method decouples multimodal reasoning from embedding compression by using a VLM to make visual and query semantics explicit before retrieval. It trains the retriever on these enhanced representations to align the embedding space with the added signal.
- Framework: The pipeline combines corpus enhancement, query enhancement, and retriever training on the resulting semantically dense representations.Corpus enhancement exposes visual evidence, while query enhancement resolves references and extracts retrieval-oriented constraints.
- Corpus enhancement: For image-only corpus entries, the VLM generates roughly 100-word keyword-rich captions covering salient objects, attributes, relations, and distinctive cues.Text-only entries remain unchanged; image–text entries retain their original metadata and append visual context.
- Query enhancement: Image-only queries receive shorter captions of about 50 words designed to preserve discriminative evidence while avoiding noise.Text-only queries are unchanged, while mixed-modal queries require either reference resolution or constraint extraction.
- Query enhancement: For QA-style queries, the VLM resolves image-dependent references and rewrites vague expressions into self-contained text suitable for standard text-based matching.When canonical naming is uncertain, it emits a concise visual descriptor instead.
- Query enhancement: For modification requests, the VLM extracts concise keyword constraints while excluding the reference image to prevent contextual contamination.Instruction-only conditioning yields more consistent retrieval-oriented rewrites.
4. Experiments
Across M-BEIR, reasoning-augmented retrieval consistently improves most benchmarks by making query and corpus semantics explicit, while requiring matched training to avoid distribution shift. Gains are strongest for knowledge-intensive and modification tasks, although fashion benchmarks expose limitations of exact-ID evaluation.
- Overall Results: The enhanced pipeline consistently improves performance across the majority of M-BEIR benchmarks relative to the baseline LamRA implementation.It addresses semantic gaps in both queries and corpus entries.
- Evaluation Limitations: FashionIQ and Fashion200k scores are difficult to interpret because visually correct alternatives are penalized when they differ from the single ground-truth image ID.The reported flat or slightly regressed scores therefore reflect an evaluation limitation rather than necessarily reduced capability.
- Synergy of Components: 30.72 average recall is achieved by the full pipeline, confirming that query and corpus enhancements are complementary.Their combination is essential for MSCOCO-0, VisNews-0/3, WebQA-2, and InfoSeek-6.
- Impact of Query Enhancement: 4.87 points over baseline are obtained by Q-Only on InfoSeek-8, where resolving ambiguous entities is more effective than enriching the corpus alone.Q-Only averages 30.51, compared with 29.96 for C-Only.
- Inference-Only Experiment: Enhanced data degrades retrieval when applied only at inference to a baseline model, because distribution shift causes dense visual context to be treated as noise.The result shows that high-quality enhanced data requires retriever training on the same representations.
- Comparison with Baselines: 52.38 R@5 is achieved on InfoSeek-8, exceeding GME-2B by over 8 points and nearly doubling SigLIP’s 27.40, while CIRR reaches 49.40 R@5.The method also achieves 90.02 on MSCOCO-3 and state-of-the-art R@5 on 6 of 13 reported tasks.
5. Conclusion
The paper identifies semantic ambiguity in raw multimodal inputs as the key bottleneck in universal multimodal retrieval, causing retrievers to conflate reasoning with compression. Its data-centric solution externalizes latent reasoning through VLM-generated semantic augmentation, enabling explicit matching and more robust, interpretable retrieval without relying on architectural complexity.
- 5. Conclusion: Semantic ambiguity, rather than model capacity, is identified as the key bottleneck in universal multimodal retrieval.Raw multimodal inputs force retrievers to collapse reasoning and compression into one embedding pass, encouraging spurious matching.
- 5. Conclusion: A strong VLM externalizes latent reasoning by converting multimodal inputs into reasoning-augmented text for explicit semantic matching.This turns “reasoning-then-retrieve” into a data-centric retrieval framework.
- 5. Conclusion: The method yields consistent, substantial gains across M-BEIR tasks.The conclusion attributes these gains to reasoning-augmented representations rather than increased architectural complexity.
- 5. Conclusion: The results support prioritizing scalable semantic augmentation and alignment over architectural complexity for robust, interpretable multimodal retrieval.Qualitative examples show enhanced semantic cues resolving ambiguous queries and aligning compositional modification requests with the correct results.
A. Prompts Used
The prompts externalize retrieval-relevant semantics through dense corpus captions, concise QA query rewrites, and phrase-based modification constraints. Each prompt suppresses filler while preserving distinctive visual, semantic, or compositional information.
- A.1. Corpus Enhancement: Corpus enhancement asks Qwen to generate a precise, keyword-rich text entry based on the image.The prompt prioritizes the main subject or scene layout and limits outputs to 100 words.
- A.1. Corpus Enhancement: Corpus captions retain specific colors, materials, visible text or logos, unique shapes, and named entities when present.Viewpoint is included only when distinctive, while aesthetic filler and nonexistent details are excluded.
- A.2. QA Query Enhancement: QA query enhancement rewrites questions by integrating the visual subject into an extremely concise search query matching text documents.The added visual description is capped at 3–5 words.
- A.2. QA Query Enhancement: Named entities are represented by name alone, whereas generic objects use dominant color or material plus a broad category.This specific-versus-generic split removes distracting visual adjectives while retaining useful anchors.
- A.2. QA Query Enhancement: QA rewrites ban filler adjectives and environmental details, and they must never answer the question themselves.The model only rewrites the query, preserving zero-leakage behavior.
- A.3. Modification Request Enhancement: Modification enhancement extracts key semantic phrases describing the target image and removes conversational filler and grammar words.Filler verbs and pronouns or articles are deleted, while adjectives, nouns, and spatial prepositions are preserved.
- A.3. Modification Request Enhancement: Modification outputs preserve compositional logic by retaining prepositions such as “with,” “on,” “in,” and “without.”For example, “Make the needle upside down in the hand” becomes “Needle upside down in hand.”
- A.3. Modification Request Enhancement: The modification prompt allows concise original queries to remain unchanged when no rewriting is necessary.Examples include outputs that retain removal instructions such as “Remove all cheetahs” and “Remove green from background.”
B. Experiments on MVRB Composed Image Retrieval
On MVRB composed image retrieval, the method consistently outperforms the baseline across several tasks, with especially pronounced gains on knowledge-based retrieval requiring multi-step reasoning over object relations.
- MVRB Composed Image Retrieval: The method consistently outperforms the baseline on several MVRB composed image retrieval tasks by recall score.These tasks require multi-step reasoning over object relations, such as identifying the phone case corresponding to a given phone.
- MVRB Composed Image Retrieval: Gains are especially pronounced on knowledge-based retrieval tasks.
C. Impact of Hard Negative Mining and Data Enrichment
This section evaluates the Enriched (Full) data within a robust hard-negative training regime, comparing it with a Baseline model under identical M-BEIR hard-negative training.
- C. Impact of Hard Negative Mining and Data Enrichment: The evaluation tests Enriched (Full) data within a robust Hard Negative training regime.The passage frames this as an effectiveness evaluation of the enriched data.
- C. Impact of Hard Negative Mining and Data Enrichment: Both the Baseline and Full models are trained with M-BEIR hard negatives.This controls the hard-negative training setup when comparing the models.
- C. Impact of Hard Negative Mining and Data Enrichment: Table 6 compares the performance of the Baseline model against the Full model.The cited passage identifies Table 6 as the source of this comparison.
C.1. Robustness of Semantic Enrichment
Semantic enrichment remains effective under hard-negative training, delivering additive gains across most benchmarks despite a stronger baseline. Improvements extend from general retrieval to knowledge-intensive and multimodal tasks, although OVEN-8 and EDIS-2 regress under task-specific interference.
- Robustness under Hard Negative Training: The Full model adds performance across most benchmarks against a hard-negative-mined baseline.This stress test used a highly competitive baseline with a significantly elevated performance bar.
- Task-Specific Regressions: Regressions on OVEN-8 and EDIS-2 are attributed to task-specific interference from hard-negative mining.
- General Retrieval: +1.16% R@1 on MSCOCO-0 demonstrates improved general retrieval and scene understanding from detailed captions.
- Knowledge-Intensive and Multimodal Retrieval: Semantic enrichment consistently outperforms the baseline on VisualNews and InfoSeek knowledge-intensive and multimodal tasks.InfoSeek-6 and InfoSeek-8 show particularly significant gains associated with enriched entity descriptions and factual augmentations.
C.2. Analysis of Task-Specific Deviations
The method shows task-specific regressions on OVEN-8 and EDIS-2, attributed to hard-negative-induced dataset interference and bias. Proposed future work replaces fixed-length enrichment with domain-adaptive length curation.
- OVEN-8 and Cross-Task Interference: OVEN-8 regressed by approximately 6 points when hard negatives were introduced, implicating the hard-negative training regime rather than the enrichment process.The baseline’s drop isolates the observed regression to bias induced during hard-negative training.
- OVEN-8 and Cross-Task Interference: OVEN may conflict with shared InfoSeek hard negatives because its coarser training objective does not require the same fine-grained discrimination.The authors hypothesize that InfoSeek-specific discriminative features fail to generalize to OVEN.
- EDIS-2 and Length-Induced Modality Bias: EDIS-2 regression reflects a mismatch between concise corpus entries and uniformly generated captions of approximately 100 words.This length contrast creates a potential modality imbalance during hard-negative discrimination.
- EDIS-2 and Length-Induced Modality Bias: Without hard negatives, the Full model outperformed the Baseline on EDIS-2, indicating that the generated data is intrinsically high-quality and informative.The reversal appears only after hard-negative training.
- Future Work: Dynamic Length Curation: Future work will dynamically match enriched-text length and information density to each target domain instead of systematically generating 100-word descriptions.This aims to address both EDIS length mismatch and OVEN interference.