Source-linked AI summary
AdaptiveEmbed: Sample-Adaptive Multi-Vector Representation for Multimodal Retrieval
Xinze Liu, Lei Yang, Dayan Wu, Hengjie Zhu, Zihao Zhang, Hanqi Wu, Tianzhu Hu, Peng Fu, Zheng Lin, Weiping Wang
TL;DR
Fixed-capacity multi-vector retrieval assigns uniform budgets despite sample-specific retrieval demands. The paper introduces SAMVR and AdaptiveEmbed, which learn candidate sets and allocate capacity per sample using utility-driven selection. Across image–text, video–text, and audio–text benchmarks, adaptive allocation improves retrieval performance with roughly two embeddings per sample, while learned policies remain below oracle performance and gallery storage reduction remains open.
Problem
Existing multi-vector retrieval methods largely use uniform sample-independent budgets, although samples exhibit different retrieval-utility trends and preferred capacities.
Method
AdaptiveEmbed learns structured candidate representations with MGCL and symmetric SetSim, then uses UPO and MUA to construct sample-specific CAES representations.
Results
Across image, text, video, and audio benchmarks, sample-adaptive allocation achieves higher average mAP than fixed-capacity representations while using around two embeddings per sample.
Takeaways & Limitations
SAMVR provides a multimodal retrieval formulation in which representation capacity is allocated at the sample level according to retrieval utility.
Takeaways & Limitations
The learned policy remains below the oracle, while query-dependent activation still requires each gallery item to store its full eight-token embedding set.
Abstract
from arXiv · showhide
Multi-vector representations have emerged as an effective paradigm for multimodal retrieval, representing each sample with multiple complementary embeddings to capture fine-grained cross-modal information. However, existing approaches typically employ a fixed representation capacity, assigning the same number of vectors to all samples regardless of their individual retrieval demands. Such a fixed-capacity formulation overlooks the fact that different samples may require different amounts of representation capacity for effective retrieval. In this work, we introduce \emph{Sample-Adaptive Multi-Vector Representation} (SAMVR), a new problem setting for multimodal retrieval that studies how multi-vector representation capacity can be allocated at the sample level. Under SAMVR, each sample is represented by a \emph{content-adaptive embedding set} (CAES), whose capacity is determined according to the sample-specific retrieval utility of additional representation vectors. To instantiate SAMVR, we propose \emph{AdaptiveEmbed}, a unified framework for learning sample-adaptive multi-vector representations. AdaptiveEmbed learns structured multi-vector representations through \emph{Multi-Group Contrastive Learning} (MGCL) with the symmetric \emph{set-to-set similarity} (SetSim), and further employs \emph{Utility Policy Optimization} (UPO) to determine sample-specific representation capacity via \emph{Marginal Utility Allocation} (MUA). Experiments across multimodal retrieval benchmarks involving image, text, video, and audio show that sample-adaptive capacity allocation achieves overall better retrieval performance than fixed-capacity multi-vector representations, validating the effectiveness of SAMVR for multimodal retrieval. These results establish SAMVR as a viable formulation for adaptive capacity allocation in multi-vector multimodal retrieval.
1 INTRODUCTION
The paper argues that fixed multi-vector budgets overlook sample-level differences in retrieval demand. It formulates sample-adaptive capacity allocation as SAMVR and proposes AdaptiveEmbed to learn and allocate content-adaptive embedding sets.
- Motivation: Fixed-capacity multi-vector methods generally expose the same representation budget across samples under a given operating configuration.Flexible inference operating points are typically selected system-wide rather than independently per sample.
- Motivation: Samples grouped by oracle-optimal k exhibit different capacity–performance trends, while preferred capacities span the available range.The paper characterizes capacity needs through retrieval utility rather than an explicit content-complexity measure.
- Problem formulation: SAMVR treats representation capacity as a sample-dependent decision variable determined by the retrieval utility of additional embeddings.Each sample receives a content-adaptive embedding set whose composition and capacity are determined individually.
- Approach: AdaptiveEmbed separates candidate-representation learning from sample-specific capacity allocation using MGCL, symmetric SetSim, UPO, and MUA.The framework first learns structured candidates, then learns and applies utility-driven capacity decisions.
- Contributions: Adaptive allocation achieves higher average mAP than fixed-capacity representations across image, text, video, and audio benchmarks while using around two embeddings per sample.The reported comparison uses 3–40 embeddings for fixed-capacity baselines.
2 RELATED WORK
Prior multimodal retrieval work developed multi-vector interaction and retrieval-feedback methods, but SAMVR instead makes embedding-set composition and capacity sample-dependent.
- Multi-vector retrieval: Multi-vector retrieval represents samples with multiple embeddings to model richer cross-modal interactions than single-vector retrieval.Dense single-vector representations enable efficient search but retain less fine-grained evidence.
- Retrieval feedback: Retrieval-feedback methods use retrieval responses to refine representations or learn when refinement should be applied.The related work includes pseudo-relevance feedback and selective feedback for adaptive decisions.
- SAMVR: SAMVR treats embedding-set composition and capacity as per-sample variables rather than constructing more vectors, compressing a predefined representation, or choosing a global budget.MaxSim responses from a retrieval bank provide observable feedback for progressive representation selection.
3 METHODOLOGY
AdaptiveEmbed learns structured candidate embedding sets and then selects sample-specific subsets for retrieval. Its design combines symmetric set matching, grouped prefix representations, and utility-driven sequential allocation.
- Problem definition: SAMVR maps each sample to a CAES with sample-specific capacity selected from an available capacity set.The formulation supports bidirectional cross-modal retrieval between two modality spaces.
- SetSim: SetSim finds a maximum-average-similarity bijection between equal-cardinality embedding sets, producing a symmetric score for both retrieval directions.The optimal matching is computed exactly during training and inference.
- Architecture: AdaptiveEmbed uses a two-stage architecture: candidate embedding sets are learned first, then sample-specific CAES subsets are selected for retrieval.Queries and gallery CAES representations are compared with SetSim to rank gallery samples.
- Stage I: Embedding Set Learning: Stage I projects Q-Former outputs into a shared retrieval space, L2-normalizes them, and forms a candidate embedding pool.The projections are modality-specific while the resulting embeddings share the retrieval dimension.
- Stage I: Embedding Set Learning: MGCL organizes candidates into ordered groups and jointly optimizes supported prefixes so shorter representations remain compact and longer ones add retrieval information.Bidirectional InfoNCE is applied to every group-prefix configuration.
- Stage II: CAES Construction: Stage II uses retrieval-bank MaxSim responses to form allocation-state feedback, which a Token Selection Transformer uses to predict the next capacity action.The feedback records responses of selected and candidate tokens without exposing ground-truth identities or retrieval utilities.
- Stage II: CAES Construction: UPO optimizes expected retrieval utility over explicitly evaluated admissible actions instead of estimating utility from sampled policy-gradient trajectories.MUA then selects the highest-probability admissible expansion until the allocation threshold terminates the process.
4 EXPERIMENTS
Experiments evaluate SAMVR across image–text, video–text, and audio–text retrieval, along with ablations, adaptive-allocation analyses, efficiency, and sensitivity. AdaptiveEmbed generally outperforms fixed-capacity alternatives while using substantially fewer average tokens, and analyses attribute gains to allocation, structured multi-group representations, and retrieval-aware matching.
- Benchmark results: 61.0 average mAP is reached by Oracle AdaptiveEmbed at 2.2 average tokens, while AdaptiveEmbed achieves strong image–text performance with 2.1 average tokens.Fixed-capacity budgets from 3 to 40 tokens do not consistently improve retrieval, whereas sample-adaptive allocation performs strongly at low average capacity.
- Benchmark results: AdaptiveEmbed achieves the strongest multi-vector result on 7 of 8 video–text and audio–text directions with an average of 1.9 tokens on ActivityNet.On image–text retrieval, it is strongest on 6 of 8 directions and has the highest average mAP overall, including COCO and zero-shot OpenImages.
- Ablation study: Removing MGCL or bank feedback lowers performance, with bank-feedback removal doing so despite a larger average capacity.The ablation supports complementary group organization and retrieval-context feedback as contributors to adaptive selection.
- Adaptive allocation: AdaptiveEmbed surpasses every fixed-k operating point at lower average capacity when all methods share the same MGCL-trained representation.This isolates per-sample allocation as the source of the gain rather than a stronger encoder or simply a larger budget.
- Representation structure: Multi-group configurations match or exceed single-group structures across token allocations, supporting multiple semantic anchors for adaptive selection.Both structures use the same coarse-to-fine prefix principle; they differ in how many coarse anchors they maintain.
- Similarity analysis: MaxSim distinct-token coverage drops from 100.0% to 80.8% as capacity grows, whereas SetSim enforces one-to-one coverage between equal-cardinality sets.The comparison motivates SetSim for content-adaptive embedding sets and bidirectional retrieval.
- Efficiency: AdaptiveEmbed’s set-to-set matching cost remains far below high-capacity fixed configurations when only about two query tokens are activated.On COCO, high-capacity late-interaction retrieval time and storage grow sharply with the number of vectors.
- Sensitivity: Average mAP varies by at most 0.43 on Flickr30K and 0.16 on Clotho across evaluated δ and Top-L ranges, while average capacity stays near two tokens.Increasing Top-L reduces Clotho mean capacity from 1.90 to 1.63 without degrading retrieval performance.
5 CONCLUSION
The paper introduces SAMVR for sample-level allocation of multi-vector capacity and presents AdaptiveEmbed as one implementation. Experiments across diverse multimodal retrieval benchmarks show that this capacity can be learned and evaluated in practice.
- Conclusion: SAMVR treats representation capacity as a sample-dependent variable and defines content-adaptive embedding sets for multimodal retrieval.Different samples may receive different amounts of embedding capacity.
- Conclusion: AdaptiveEmbed combines MGCL, SetSim, UPO, and MUA to learn candidate representations and allocate capacity according to utility.The framework is presented as one possible implementation of sample-adaptive multi-vector representations.
- Conclusion: Experiments across diverse multimodal retrieval benchmarks verify that sample-adaptive representation capacity can be learned and evaluated in practice.The paper positions SAMVR as a starting point for further investigation into adaptive representation learning.
A DETAILED FORMULATION OF ADAPTIVE REPRESENTATION ALLOCATION
AdaptiveEmbed allocates representation capacity through structured group prefixes and sequential ADD/STOP decisions. A non-deployable oracle exhaustively evaluates finer-grained configurations, while bank feedback informs the learned policy using retrieval responses.
- A.1 GROUP CONFIGURATION: AdaptiveEmbed organizes eight candidate tokens into two ordered four-token groups, with coarse tokens carrying global semantics and fine tokens adding discriminative detail.Each group begins with one coarse token followed by complementary tokens.
- A.2 MULTI-STEP ADAPTIVE DECISION: The deployed policy constructs a content-adaptive embedding set through up to three sequential decisions that either expand an admissible group prefix or terminate early.This avoids predicting directly over the full combinatorial configuration space.
- A.2 MULTI-STEP ADAPTIVE DECISION: Starting from {g0}, the policy may stop, add a second coarse token, add a fine token, merge to four tokens, or complete the eight-token representation.Each expansion corresponds to an ADD action, and allocation terminates with STOP.
- A.2 MULTI-STEP ADAPTIVE DECISION: Five terminal configurations span capacities from one to eight tokens: coarse-sufficient, coarse-expanded, fine-refined, balanced, and full-capacity.The reachable capacities are 1, 2, 2, 4, and 8 tokens.
- A.3 ORACLE UPPER BOUND: The oracle expands one token at a time and preserves all branches, reaching every admissible configuration rather than the deployed policy’s merged five-configuration subset.It measures the potential of sample-adaptive allocation but is not deployable.
- A.3 ORACLE UPPER BOUND: For each sample, the oracle selects the configuration with maximum retrieval AP from a finite set of 20 configurations.Configurations are formed from the Cartesian product of the two groups’ ordered prefixes.
- A.3 ORACLE UPPER BOUND: The oracle is a non-deployable upper bound because it requires per-sample exhaustive search under the target retrieval metric.The gap from AdaptiveEmbed separates SAMVR’s potential from the current allocation model’s capability.
- A.4 BANK FEEDBACK CONSTRUCTION: Bank feedback lets the capacity policy compare candidate and selected token responses on currently top-scoring gallery items before producing STOP, expansion, or branch logits.Candidate responses that differ from selected-token responses are treated as potentially incremental retrieval information, while the policy input excludes ground-truth labels and AP values.
B DATASETS AND EVALUATION PROTOCOLS
The evaluation uses unified one-to-one full-gallery retrieval protocols across image–text, video–text, and audio–text datasets. Training is source-domain specific, with several image datasets evaluated zero-shot and comparisons spanning single-vector and fixed-capacity multi-vector baselines.
- Unified full-gallery retrieval: AdaptiveEmbed is evaluated on eight datasets covering image–text, video–text, and audio–text retrieval under a unified full-gallery protocol.Each query has exactly one positive item in the gallery.
- Unified full-gallery retrieval: Image–text datasets use one official Localized Narratives description per image, while video and audio captions are concatenated in their original order.The protocol converts each dataset into a one-to-one retrieval benchmark.
- Training and zero-shot splits: Training uses COCO for image–text, ActivityNet+DiDeMo for video, and MACS+Clotho for audio, while Flickr30K, ADE20K, and OpenImages are evaluated zero-shot.The representation learner and capacity policy are trained only on each modality’s source-domain split.
- Baselines: Baselines include single-vector models such as CLIP and SigLIP and multi-vector models such as ColQwen2 and MetaEmbed.Additional modality-specific baselines include VLM2Vec-V2, Qwen3-VL-Embedding, LAION-CLAP, and WAVE-7B where applicable.
- Implementation Details: Implementation uses frozen Qwen3-VL-Embedding-2B encoders for image–text and video–text retrieval and a frozen WAVE-7B encoder for audio–text retrieval.Stage I produces eight representation tokens arranged into two four-token groups.
C.1 ROBUSTNESS ACROSS RETRIEVAL METRICS
Across retrieval metrics, modalities, and allocation analyses, AdaptiveEmbed generally improves retrieval quality at lower or comparable representation capacity than fixed-capacity alternatives. Additional analyses show that its hierarchical policy is practical, its multi-group structure is beneficial, and its learned allocations approximate—but do not fully match—the oracle.
- Metric robustness: AdaptiveEmbed leads every Recall and nDCG metric on COCO while activating the fewest tokens on average.The bidirectional R@1 gain is 1.1 points, with T2I R@1 gains of 1.1 to 1.5 points depending on the configuration.
- Metric robustness: AdaptiveEmbed outperforms every MetaEmbed configuration on all Recall, nDCG, and mAP metrics in strict zero-shot ADE20K transfer at the lowest average capacity.The representation, policy, threshold, and retrieval bank are transferred from COCO without ADE20K-specific tuning.
- Bank robustness: Replacing the COCO bank with a Flickr30K bank changes Avg. mAP by only +0.07.This indicates that the allocation policy is largely insensitive to bank domain, and matching the bank to the target domain yields no meaningful additional benefit.
- Utility–cost trade-off: Across image, video, and audio, AdaptiveEmbed sits above the fixed-capacity utility–cost frontier at roughly two activated tokens.The oracle shows further headroom at comparable or lower average capacity, while fixed-capacity retrieval does not improve and often degrades as shared budget grows.
- Representation structure: Multi-group configurations match or outperform single-group counterparts on COCO, DiDeMo, and MACS under matched token allocations.The largest gains occur on MACS, where short audio–text pairs make a single global anchor easier to saturate.
- Policy complexity: On COCO, independent responses required by the hierarchical policy fall from 3.8 to 1.8 after the first decision.The first coarse decision resolves most allocation ambiguity, leaving later stages with a reduced candidate space.
- Allocation distributions: The oracle spreads selections across many of the 20 canonical configurations, whereas AdaptiveEmbed concentrates on a small subset, especially low-capacity masks and the full set.The learned policy’s gap to the oracle is localized in its restricted five-configuration action space and intermediate-mask coverage.
C.2 QUALITATIVE EXAMPLES OF SAMPLE-ADAPTIVE ALLOCATION
Oracle CAES examples show that optimal capacity varies across the full admissible range and can differ by retrieval direction. Capacity follows gallery-specific retrieval utility rather than surface content complexity.
- Capacity variation: Optimal capacity spans k=1 to k=8, so no single operating point serves all samples.The examples characterize sample-wise allocation independently of any learned policy.
- Direction-specific allocation: Several samples require different token sets for I2T and T2I retrieval, making capacity demand direction-specific.This indicates that allocation demand depends on the retrieval task as well as the sample.
- Utility versus complexity: Surface complexity does not determine optimal capacity: a detailed narrative can use one coarse token, while an ambiguous one-sentence caption can require near-full capacity.The relevant criterion is how much representation is needed to distinguish the sample within the gallery.
D DISCUSSION AND LIMITATIONS
AdaptiveEmbed’s current instantiation has limitations involving policy quality, efficiency boundaries, and the absence of a principled content-complexity criterion. These limitations concern AdaptiveEmbed rather than SAMVR itself.
- Policy quality: A sizable gap remains between the learned policy and the oracle, which achieves higher mAP at comparable or lower average capacity.Finer intermediate decisions would require richer utility feedback because raw MaxSim responses weaken across stages.
- Efficiency: Matching-time efficiency depends on the data distribution, since hard-sample-dominated datasets reduce the speed advantage of lower average activated capacity.Gallery items must still store the full eight-token embedding set under query-dependent activation.
- Efficiency: Reducing gallery-side storage under query-dependent activation remains an open problem.
- Content complexity: Allocation is driven by retrieval utility rather than an explicit content-complexity measure, leaving the relationship between semantic richness and capacity only qualitatively examined.A principled complexity-aware formulation is left for future work.