Source-linked AI summary
Let Multimodal Embedders Learn When to Augment Query via Adaptive Query Augmentation
Wongyu Kim, Hochang Lee, Sanghak Lee, Yoonsung Kim, Jaehyun Park
TL;DR
Earlier LLM-based embedders augmented every query, although this could increase latency or hurt performance, and prior methods had not been explored in multimodal environments. M-Solomon learns when to augment, and MMEB results show it outperformed both no-augmentation and always-augmentation baselines while embedding faster than the latter.
Problem
Earlier LLM-based embedders augmented every query, which could increase embedding latency or degrade performance, and their effectiveness had not been demonstrated in multimodal environments.
Source: §1 · p. 1Method
M-Solomon groups training datasets by augmentation need, synthesizes answer-based augmentations for one group, and learns to generate /augment or /embed for each query.
Source: §1 · p. 2, §3.1 · p. 3, §3.2 · p. 3Results
On MMEB, M-Solomon substantially outperformed the no-augmentation baseline and outperformed the always-augmentation baseline with significantly faster embedding latency.
Source: §1 · p. 2Takeaways & Limitations
Abstract
from arXiv · showhide
Query augmentation makes queries more meaningful by appending further information to the queries to find relevant documents. Current studies have proposed Large Language Model (LLM)-based embedders, which learn representation for embedding and generation for query augmentation in a multi-task manner by leveraging the generative capabilities of LLM. During inference, these jointly trained embedders have conducted query augmentation followed by embedding, showing effective results. However, augmenting every query leads to substantial embedding latency and query augmentation can be detrimental to performance for some queries. Also, previous methods have not been explored in multimodal environments. To tackle these problems, we propose M-Solomon, a universal multimodal embedder that can adaptively determine when to augment queries. Our approach first divides the queries of the training datasets into two groups at the dataset level. One includes queries that require augmentation and the other includes queries that do not. Then, we introduces a synthesis process that generates appropriate augmentations for queries that require them by leveraging a powerful Multimodal LLM (MLLM). Next, we present adaptive query augmentation. Through this step, M-Solomon can conduct query augmentation only when necessary by learning to generate synthetic augmentations with the prefix /augment for queries that demand them and to generate the simple string /embed for others. Experimental results showed that M-Solomon not only surpassed the baseline without augmentation by a large margin but also outperformed the baseline that always used augmentation, providing much faster embedding latency.
1 Introduction
M-Solomon is designed to decide when query augmentation is useful, addressing latency, performance risks, and the lack of prior multimodal evaluation. On MMEB, it outperformed both no-augmentation and always-augmentation baselines, with faster embedding latency than the latter.
1 Introduction
- Prior LLM-based embedders augmented queries before embedding, but this can increase latency, degrade performance for some queries, and had not been demonstrated in multimodal settings.The pilot study compared an embedding-only model with one that always augmented queries.Source: §1 · p. 1
- M-Solomon learns to choose between generating an /augment-prefixed synthetic augmentation and producing the simple string /embed.Training queries are grouped at the dataset level according to whether they require augmentation.Source: §1 · p. 2
- On MMEB, M-Solomon substantially outperformed the no-augmentation baseline and outperformed the always-augmentation baseline with faster embedding latency.The pilot study illustrates a possible failure of always augmenting: an augmentation misread “Paris” as Eiffel Tower-related clothing designs, while the unaugmented model retrieved the relevant image.Source: §1 · pp. 1–2
2 Related Work
Prior work has explored joint embedding and query augmentation, multimodal embedding techniques, and adaptive generation. M-Solomon applies adaptive selection to query augmentation in multimodal embedding.
Joint Training of Embedding and Query Augmentation
- LLM-based embedders jointly learn embedding representations and query augmentation, but augmenting every query can increase latency and reduce performance for some queries.The cited prior studies had not considered multimodal environments.Source: §2 · p. 2
Multimodal Embedders
- Multimodal embedders have used techniques including data synthesis, hard-negative sampling, distillation, and contrastive-loss optimization.M-Solomon proposes adaptive query augmentation as another approach to improving performance.Source: §2 · p. 2
Adaptive Generation
- Adaptive-generation methods select between thinking and non-thinking modes for individual questions, motivating adaptive query augmentation in M-Solomon.The cited work describes this selection as supporting more effective and efficient test-time scaling.Source: §2 · p. 2
3 Methodology 𝐸, . . . , 𝐷𝑒
M-Solomon learns multimodal query representations and adaptive augmentation, using dataset-level labels to identify where augmentation is needed. It synthesizes answer-based augmentations and jointly trains generation and embedding objectives so inference can select augmentation or direct embedding.
3 Methodology 𝐸, . . . , 𝐷𝑒
- M-Solomon learns representations for text, image, or interleaved text-and-image queries and retrieves positive documents from benchmark candidates.Its training also targets adaptive augmentation generation.Source: §3 · p. 2
3.1 Query Augmentation Synthesis
- The method labels training datasets as requiring or not requiring augmentation based on whether augmentation-free performance is better or comparable on their test sets.In the pilot comparison, this criterion split the 20 datasets into 10 in each group.Source: §3.1 · pp. 2–3
- For datasets marked as requiring augmentation, a teacher MLLM generates query answers, which are used as augmentations.The synthesis process uses Qwen2.5-VL-72B-Instruct and retains the answer rather than the generated reasoning process.Source: §3.1 · p. 3
3.2 Adaptive Query Augmentation
- Training teaches M-Solomon to generate /augment and an augmentation for queries that need one, or /embed for queries that do not.At inference, it appends generated augmentations to queries before encoding; generation and encoding use one forward pass.Source: §3.2 · p. 3
- The training objective combines autoregressive generation of augmentation targets with contrastive learning of query and document representations.The contrastive loss brings the augmented-query embedding closer to the positive document and farther from hard negatives; the method omits contrastive loss on original queries used in [25].Source: §3.2 · p. 3
4 Experiments
On MMEB, M-Solomon outperformed both NoAug and AlwaysAug overall and on OOD results, while ablations and dataset-level analyses examined its adaptive augmentation and latency trade-offs.
4.1 Experimental Setup
- MMEB evaluation covered 20 in-distribution and 16 out-of-distribution datasets across four task categories, with P@1 primary and NoAug and AlwaysAug as baselines.Training used 2.5K queries from each of 20 datasets, totaling 50K samples; NoAug used no augmentation and AlwaysAug augmented every query.Source: §4.1 · pp. 3–4
4.2 Main Results
- M-Solomon outperformed NoAug and AlwaysAug overall and on OOD results, while AlwaysAug's Latency and # of Ts were nearly twice as high.The authors attribute the speed difference to M-Solomon augmenting only queries it identifies as needing augmentation.Source: §4.2 · pp. 4–5, Table 2 · p. 5
4.4 Further Analysis of Adaptive Query Augmentation
- Across FashionIQ, GQA, and ImageNet-R, M-Solomon significantly outperformed NoAug; AlwaysAug underperformed NoAug on FashionIQ and improved less than M-Solomon on the other datasets.The analysis presents these as representative datasets from the MMEB benchmark.Source: Table 3 · p. 5, §4.4 · p. 5
4.3 Ablation Study
- M-Solomon-Half had lower overall performance and higher Latency and # of Ts than M-Solomon; forced /augment lowered performance, while forced /embed was quicker but less effective.The authors report that forced prefixes caused conflicts when they did not match a query's augmentation needs.Source: §4.3 · p. 5
4.4 Further Analysis of Adaptive Query Augmentation
- On FashionIQ, M-Solomon beat AlwaysAug on P@1 with approximately five times lower Latency and # of Ts; on GQA, it had higher P@1 but slightly higher Latency and # of Ts.FashionIQ's /embed% was 91.0%; GQA's was 8.3%, and the authors attribute GQA's latency trade-off to longer, more informative augmentations.Source: §4.4 · p. 5
- On ImageNet-R, M-Solomon had notably higher P@1 with similar Latency and # of Ts, and its confidence scores were robust across datasets.Table 4 examples show M-Solomon producing higher-quality augmentations that led to more accurate retrieval results.Source: §4.4 · pp. 5–6, Table 4 · p. 6
5 Conclusion and Future Work
M-Solomon adaptively augments queries using dataset-level identification and synthesized augmentations, then generates /augment or /embed according to whether augmentation is needed. The authors report improved performance and efficiency, and identify query-level decisions and reasoning-based augmentation as future work.
5 Conclusion and Future Work
- M-Solomon identifies queries needing augmentation at the dataset level, synthesizes augmentations for them, and learns to generate /augment or /embed accordingly.The authors say this training enabled the model to understand when to augment queries.Source: §5 · p. 6
- The authors report that M-Solomon significantly outperformed baselines while creating embeddings effectively and efficiently.Source: §5 · p. 6
- Future work will study query-level identification for finer-grained decisions and add reasoning-based augmentation for reasoning-intensive tasks such as BRIGHT and RAR-b.Source: §5 · p. 6
6 GenAI Usage Disclosure
For research, the authors used a multimodal teacher model to generate answer-style augmentations for 50K queries. For writing, they used generative AI for basic language tasks, checked and revised its outputs, and report that the paper was initially written without it.
6.1 Usage in Research Stage
- Qwen2.5-VL-72B-Instruct served as the teacher model and generated answer-style augmentations for 50K queries.The authors constructed prompts using a template.Source: §6.1 · p. 6
6.2 Usage in Writing Stage
- The authors used ChatGPT for basic writing tasks, checked and revised its outputs, and say the paper was initially written without generative AI.Source: §6.2 · p. 6