PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Post-Hoc Rewards: Privileged Reasoning Exploration for Multimodal Embeddings

Tianyu Yang, Yijun Tian, Lars Benedikt Kaesberg, Jan Philip Wahle, Bela Gipp, Terry Ruas, Daniel Kurzawe

arXiv:2610.32474v1cs.AIcs.CL

TL;DR

Retrieval supervision supplies pairwise relevance but does not specify which reasoning traces distinguish relevant candidates, limiting exploration from inputs alone. PILOT pairs an input-conditioned prior with a relationally informed posterior and selectively transfers useful traces. It outperforms the primary baseline across all 12 MMEB-V2 task groups at both model scales.

  • Problem

    Retrieval relations do not specify the reasoning traces that distinguish relevant candidates, leaving input-conditioned methods to explore without relational evidence.

    Source: §1 · p. 2

  • Method

    PILOT models reasoning as a latent variable, using an input-conditioned prior and a relationally informed posterior with selective posterior-to-prior transfer.

    Source: §1 · p. 2

  • Results

    PILOT outperforms UME-R1 across all 12 MMEB-V2 task groups at both scales, with overall scores of 63.3 (2B) and 67.6 (7B).

    Source: §4.2 · pp. 7–8

  • Takeaways & Limitations

    Retrieval supervision can guide exploration of reasoning trajectories, not only evaluate their embedding utility after generation.

    Source: §5 · p. 10

Abstract

from arXiv · show

Advances in Multimodal Large Language Models (MLLMs) have driven substantial progress in multimodal embeddings. Recent methods exploit generative capabilities by augmenting embedded inputs with self-generated reasoning to produce more informative embeddings. However, conventional retrieval supervision for training embedding models specifies pairwise relevance relations without explicitly guiding the reasoning used for input augmentation. Such reasoning is neither directly observed nor uniquely determined by the input, allowing multiple plausible traces to introduce semantic cues of varying retrieval utility. Existing approaches largely optimize reasoning conditioned on the input alone, limiting the exploration of retrieval-relevant traces. We introduce PILOT, a framework that models reasoning as a latent variable and instantiates its prior and posterior distributions as distinct reasoning modes of a shared MLLM. The prior conditions only on the input, whereas the posterior uses additional training-time relational evidence to guide reasoning exploration. To translate the posterior's broader exploration into inference-time improvements, we derive an importance-weighted variational objective and instantiate it as a prior-posterior co-evolution framework that improves both modes under a shared inference-compatible reward while selectively transferring posterior trajectories to the prior according to their utility and prior reachability. On the 78-task MMEB-V2 benchmark, PILOT outperforms the primary baseline across all 12 task groups at both 2B and 7B parameters, improving overall scores by 3.2 and 3.3 points, respectively, with the largest gains in visual-document retrieval (+6.1 and +6.4 points). Our results show that relational supervision is most valuable not as a post-hoc reward but as a signal for shaping how reasoning is explored in training reasoning-enhanced embedding models.

1 INTRODUCTION

Retrieval supervision identifies positive–negative relations but does not specify which reasoning traces distinguish relevant candidates, limiting input-only reasoning exploration. PILOT uses a relationally informed posterior during training and selectively transfers useful reasoning to an inference-time prior; it outperforms primary baselines across MMEB-V2 task groups at both scales.

1 INTRODUCTION

  • Positive–negative relations identify relevant matches but leave the reasoning needed to distinguish them unspecified.Multiple plausible traces for one input can differ in how well they distinguish positives from negatives.Source: §1 · p. 2
  • PILOT models reasoning traces as latent variables, using an input-conditioned prior at inference and a relationally informed posterior during training.Both distributions are distinct reasoning modes of a shared MLLM.Source: §1 · pp. 2–3
  • A blind contrastive posterior compares the target with a randomly ordered positive–negative pair, with relevance labels withheld.Posterior traces are retained only when the annotator correctly identifies the positive counterpart.Source: §1 · p. 2
  • An importance-weighted objective co-evolves prior and posterior reasoning, transferring posterior trajectories according to retrieval utility and prior reachability.The updates combine policy improvement for both modes with weighted posterior-to-prior transfer.Source: §1 · p. 2
  • PILOT outperforms primary multimodal embedding baselines across all 12 MMEB-V2 task groups at both 2B and 7B scales.The largest gains are in visual-document retrieval.Source: §1 · pp. 2–3

2 RELATED WORK

Prior multimodal embedding methods often encode inputs directly or generate reasoning separately, while relational signals mainly evaluate or refine traces after generation. PILOT instead uses positive–negative relations as privileged training evidence and co-evolves that posterior with an input-conditioned prior.

2.1 UNIVERSAL MULTIMODAL EMBEDDING

  • UME methods range from direct encoding to reasoning-augmented embeddings, while relational signals in prior approaches primarily evaluate, select, or refine reasoning after generation.PILOT uses relational evidence to guide trajectory exploration while preserving input-only inference.Source: §2.1 · p. 3

2.2 LEARNING WITH PRIVILEGED INFORMATION

  • In embedding, privileged evidence concerns relations between separate inputs, not reconstruction of a target output as in answer-generation tasks.PILOT conditions a privileged posterior on positive–negative relations and co-evolves it with the input-conditioned prior.Source: §2.2 · p. 3

3 METHODOLOGY

PILOT treats reasoning traces as latent variables and uses a privileged posterior conditioned on retrieval relations to explore traces beyond the input-only prior. It initializes both modes, then co-evolves them while selectively transferring useful, prior-reachable posterior trajectories.

3.1 PROBLEM FORMULATION

  • PILOT models reasoning traces as latent variables, with an input-conditioned prior for inference and a privileged posterior that uses relational evidence to explore retrieval-relevant traces.Both distributions are instantiated by the same MLLM under different conditioning prompts.Source: §3.1 · pp. 3–4

3.2 DUAL-MODE INITIALIZATION

  • A stronger offline annotator synthesizes prior traces from the target input and posterior traces from a blind, unlabeled positive-negative pair; only posterior annotations with a correct prediction are retained.Joint generation and contrastive learning initialize both reasoning modes and align reasoning-augmented representations with retrieval discrimination.Source: §3.2 · p. 5

3.3 PRIOR–POSTERIOR CO-EVOLUTION

  • Prior–posterior co-evolution optimizes both modes using inference-compatible rewards and transfers posterior trajectories according to retrieval utility and reachability under the prior.The mixture proposal combines prior and posterior rollouts; posterior trajectories contribute strongly to transfer only when they achieve high reward and remain sufficiently reachable under the prior.Source: §3.3 · pp. 5–6

4 EXPERIMENTS

The experiments evaluate PILOT on MMEB-V2 and analyze its training data, baselines, implementation, component contributions, and posterior exploration and transfer. The reported results cover all 12 task groups at both model scales, with further analyses examining reasoning quality and transfer to the inference-time prior.

4.1 EXPERIMENTAL SETUP

  • The evaluation uses the full MMEB-V2 benchmark of 78 image, video, and visual-document retrieval tasks, reporting Hit@1 for image and video and nDCG@5 for visual documents.The analyzed sample comprises 1,000 examples drawn approximately equally across MMEB-V2 subsets.Source: §4.1 · p. 6
  • The primary baseline UME-R1 shares PILOT’s training data, Qwen2-VL backbones, and unified embedding setup; comparisons also include encoding-only, reasoning-enhanced, specialized, and proprietary systems.Differences in training data, reasoning supervision, backbone size, and inference configuration are considered when interpreting comparisons.Source: §4.1 · p. 7
  • The study uses Qwen2-VL-2B/7B-Instruct backbones, synthesizes posterior traces with GLM-4.1V-thinking, and uses the released RL dataset for co-evolution.Dual-mode initialization adapts the SFT dataset; approximately 676K examples are retained for posterior-mode supervision and 787K original traces for prior-mode supervision.Source: §4.1 · pp. 6–7

4.2 MAIN RESULTS

  • PILOT outperforms UME-R1 across all 12 task groups, reaching overall scores of 63.3 at 2B and 67.6 at 7B using only the inference-time prior.VisDoc gains are 6.1 and 6.4 points; the 2B model scores 69.9 versus UME-R1-7B at 67.1 in VisDoc.Source: §4.2 · pp. 7–8, Table 1 · p. 7

4.3 ABLATION STUDY

  • On MMEB-V2, full PILOT scores 1.9 points above prior-only GRPO, while importance-weighted prior fitting adds 1.1 points overall.Joint GRPO reaches 62.2 overall and 68.1 on visual documents, outperforming prior-only GRPO across all modalities; full PILOT also improves image, video, and visual-document scores by 0.7, 0.8, and 1.8 points over the preceding configuration.Source: §4.3 · p. 8, Table 2 · p. 8

4.4 FURTHER ANALYSIS

  • Blind synthesis has the highest mean retrieval-usefulness margin, 0.772, and the highest candidate-leakage rate, 79.6%.On the shared non-leakage subset, it scores 0.395 versus 0.275 for positive-only and 0.204 for labeled-pair prompting; matched comparisons across 59 queries yield margins of 1.006 for correct versus −0.403 for incorrect blind predictions.Source: §4.4 · pp. 8–9, Figure 3 · p. 9
  • Posterior Pass@K rises from 61.8% to 67.5% as K increases from 1 to 8, exceeding Prior@2K by 1.9–4.9 percentage points.The analysis samples 16 traces per query from each mode at the dual-mode SFT checkpoint and replays them under the prior prompt.Source: §4.4 · p. 9
  • Full PILOT raises prior-solved coverage from 59.3% to 63.8% versus w/o IW and lowers posterior-only coverage from 15.8% to 11.4%, with nearly unchanged coverage by either mode.Against w/o TS, Full improves prior-solved coverage by 1.7 points and reduces both-failed queries by 1.5 points; transfer prioritizes retrieval utility while accounting for prior reachability.Source: §4.4 · pp. 9–10, Figure 4 · p. 9

5 CONCLUSION

The conclusion presents PILOT as a latent-variable approach that uses training-time relational evidence while retaining input-only inference. It reports gains across MMEB-V2 task groups and identifies selective transfer as important given leakage and limited direct-distillation gains.

5 CONCLUSION

  • PILOT uses training-time relational evidence to improve multimodal-embedding reasoning while retaining input-only inference.The conclusion reports outperformance over main baselines across all 12 MMEB-V2 task groups at both 2B and 7B scales.Source: §5 · p. 10
  • The conclusion argues retrieval supervision should shape which reasoning trajectories are explored, alongside evaluating their embedding utility.Source: §5 · p. 10
  • Increased candidate leakage and limited gains from direct distillation motivate selective transfer, with posterior adaptation and importance-weighted prior fitting identified as a practical strategy.The proposed transfer conditions on retrieval utility and prior reachability.Source: §5 · p. 10

AI USE STATEMENT

The authors state that generative AI tools were used to polish manuscript drafts and assist with experimental code implementation.

AI USE STATEMENT

  • Generative AI tools were used for manuscript polishing and experimental code implementation.Source: AI USE STATEMENT · p. 10
  • The authors manually checked generated manuscript paragraphs for semantic deviation and grammar errors.Source: AI USE STATEMENT · p. 10
  • LLM-generated experimental code was manually verified and tested, and the authors take responsibility for the final content and artifacts.Source: AI USE STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

The reproducibility statement describes the reported experimental materials and promises release of code, configurations, and synthesized reasoning data.

REPRODUCIBILITY STATEMENT

  • The experiments use publicly available datasets and backbone models.Source: REPRODUCIBILITY STATEMENT · p. 10
  • The paper describes training data, model configurations, optimization settings, and evaluation protocols, with additional implementation details in Appendix A.1.Source: REPRODUCIBILITY STATEMENT · p. 10
  • The authors will release training and evaluation code, configurations, and synthesized reasoning data to facilitate reproduction.The statement also points to appendices for prompt, reasoning-evaluation, objective-derivation, probe-dataset, and further experimental details.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1.1 TRAINING SETTINGS

Training uses separate supervised fine-tuning and coevolution stages, with retrieval rewards based on cosine similarity between generated embeddings. The settings specify objectives, optimization, and hardware for both stages.

A.1.1 TRAINING SETTINGS

  • SFT freezes the vision encoder, updates the visual merger and language model, and sums two autoregressive and two symmetric contrastive losses with unit coefficients.It runs for 5,000 AdamW steps at learning rate 5 × 10−5, with global batch size 1024.Source: §A.1.1 · p. 14
  • Coevolution uses 192 rollout iterations, eight sampled responses per input under each prompt mode, and separately normalized prior GRPO, posterior GRPO, and weighted SFT objectives.The objectives are combined with coefficients 1.0, 1.0, and 0.3, respectively.Source: §A.1.1 · p. 14
  • Retrieval rewards use cosine similarity at temperature T = 0.2, and training runs on 16 A100 GPUs for 2B models and 32 for 7B models.Training uses FSDP2, bfloat16 mixed precision, gradient checkpointing, and vLLM-based sampling.Source: §A.1.1 · p. 14

A.1.2 EVALUATION SETTINGS

Evaluation uses MMEB-V2 task configurations and candidate pools, encoding each query and candidate from its original input and task instructions. Retrieval metrics are computed separately for each subset using its ground-truth relevance annotations.

A.1.2 EVALUATION SETTINGS

  • Queries and candidates are independently encoded from their original inputs and task instructions, without additional positive or negative examples.Each input receives one reasoning trace from greedy decoding, with a maximum of 1,024 new tokens.Source: §A.1.2 · p. 15
  • Candidates are ranked by cosine similarity against either the full subset-level pool or the query-specific candidate set, as specified by each task configuration.The embedding is the L2-normalized final-layer hidden state at the <gen_emb> token.Source: §A.1.2 · p. 15
  • Retrieval metrics are computed separately for each subset using its ground-truth relevance annotations.Video inputs use eight sampled frames according to the task configuration.Source: §A.1.2 · p. 15

A.1.3 POSTERIOR REASONING TRACE SYNTHESIS

Posterior reasoning traces are synthesized using relational examples during training, while prior-mode prompts use only the focal input. The dual-mode SFT dataset contains query–candidate pairs with reasoning supervision on both sides.

A.1.3 POSTERIOR REASONING TRACE SYNTHESIS

  • The annotator receives a query with a positive and randomly sampled negative candidate, or symmetrically a candidate with positive and negative queries, with relevance labels hidden.The annotator generates content-grounded reasoning and a concise summary, avoiding unsupported focal-input details.Source: §A.1.3 · p. 15
  • Posterior training pairs are retained only when positive-item identification is correct on both sides and outputs pass format validation.The resulting SFT examples use a simplified, role-independent posterior prompt; predicted option labels are excluded from supervised responses.Source: §A.1.3 · p. 15
  • The dual-mode SFT dataset contains 1,463,360 query–candidate pairs: 787,202 prior-mode pairs (53.79%) and 676,158 posterior-mode pairs (46.21%).Each pair provides reasoning supervision for both the query and its positive candidate; posterior prompts additionally include positive and negative auxiliary examples.Source: §A.1.3 · p. 15, Table 3 · p. 15

A.2 PROMPT TEMPLATES

The appendices detail PILOT’s training and annotation prompts, diagnostic workflows, task adapters, and mixture importance-weighted optimization, including procedures for keeping reasoning grounded and annotations blinded. They also describe controlled reasoning audits and report how transfer weights select eligible traces and how transfer SFT affects their reachability by the prior.

A.2 PROMPT TEMPLATES

  • The prompt templates distinguish short embedding-model prompts from detailed expert-annotation prompts with grounding and leakage-control instructions.Template placeholders are explanatory, while literal model tokens are retained.Source: §A.2 · p. 15

A.2.1 EMBEDDING-MODEL TRAINING PROMPTS

  • Training prompts give the prior only the focal input, while the blind posterior adds one positive and one negative in randomized order without revealing their labels.The posterior infers discriminative features but must represent the focal input alone.Source: §A.2.1 · pp. 15–16
  • Both training prompts request a thinking process, a one-word or one-sentence summary, and a final <gen_emb> representation of the multimodal input.Visual placeholders mark where images or videos are inserted.Source: §A.2.1 · p. 16

A.2.2 EXPERT PROMPTS FOR SFT ANNOTATION

  • For multimodal SFT annotation, candidate labels are randomized and incorrect expert choices are discarded; the choice filters data, while reasoning and answer become student targets.Source: §A.2.2 · p. 16
  • Query-side expert reasoning should derive content-rich answers and express discriminative features positively, while avoiding candidate-only information, wording, visual details, or comparisons.The leakage rule allows overlap only when content is independently derivable from the query.Source: §A.2.2 · p. 17

A.2.3 UPDATED CANDIDATE-SIDE EXPERT ANNOTATION PROMPTS

  • Candidate representations must preserve supported details and evidence relationships, using query suggestions only when independently supported and distinguishing visible evidence from unseen content.The instructions prioritize explicit constraints and treat unknown matches as unconfirmed support.Source: §A.2.3 · pp. 18–20
  • Candidate-side annotation first identifies the candidate’s entities, relations, claims, and unknowns; auxiliary-query choice is an intermediate verification task, not evidence about the candidate.The blind posterior presents one positive and one negative query in randomized order without relevance labels.Source: §A.2.3 · pp. 18–19
  • Candidate-side prompts require broad, retrieval-useful summaries that preserve supported entities, relations, evidence, and qualifiers rather than answering only the selected query.They direct annotators to choose the query the candidate satisfies more completely, then synthesize the candidate without narrating the selection process.Source: §A.2.3 · p. 20
  • The final reasoning and answer must stand alone as an analysis of the candidate, preserve its important content, and omit workflow narration or auxiliary-query references.The schema requests a JSON object with a choice of A or B, reasoning, and answer.Source: §A.2.3 · pp. 20–21
  • Candidate summaries must retain supported concrete details and meaningful relationships while excluding query-only assumptions, unavailable evidence, and unreliable visual, OCR, spatial, or temporal specifics.Uncertainty should be localized; visible facts should remain in candidate-supported context, and new values should not be computed solely to satisfy an auxiliary request.Source: §A.2.3 · pp. 20–21

A.2.4 DIAGNOSTIC ANNOTATION PROMPT-VARIANT EXPERIMENT

  • The diagnostic experiment compares input-only, positive-assisted, labeled positive-and-negative, and blind A/B positive-and-negative conditions, with negatives sampled rather than hard-mined.Conditions share the system prompt, query-first introduction, task adapter, and target rules; auxiliary-input blocks, workflows, and separate choice-field requirements vary.Source: §A.2.4 · p. 22
  • The direct workflow analyzes retrieval-relevant query content, while positive-guided analysis uses a positive candidate to suggest details that must be verified against the query.If suggestions cannot be verified, the prompt calls for detailed direct query analysis rather than narrowing the representation to incidental candidate content.Source: §A.2.4 · p. 22
  • The labeled contrastive workflow uses positive and negative labels to inspect meaningful candidate differences, but retains only query-supported distinctions and other important query content.It says not to force a contrast when differences are uninformative or to reduce the representation to features that distinguish only the supplied candidates.Source: §A.2.4 · p. 23
  • The blind workflow evaluates candidates independently against the query, investigates meaningful ambiguities, and synthesizes a query representation without including candidate-selection reasoning.Unresolved candidate details are excluded, and candidate-suggested content is integrated only when independently supported by the query.Source: §A.2.4 · pp. 23–24
  • Shared output rules preserve the query’s retrieval-relevant meaning, evidence relationships, and locally qualified uncertainty while excluding unsupported candidate facts and workflow narration.The answer is a concise semantic embedding summary, and factual clauses must be supported by query text or visuals.Source: §A.2.4 · p. 24
  • Task adapters tailor query representations to retrieval types, including classification, question answering, composed-image retrieval, and visual-document retrieval, without inventing unsupported details.For example, the visual-document adapter preserves the requested fact type and evidence relationship without inventing page answers or values.Source: §A.2.4 · pp. 25–26

A.2.5 BLINDED TRACE-AUDIT INSTRUCTION

  • The trace audit is a model-based qualitative judgment of query-grounded retrieval usefulness, not an observed retrieval-accuracy change, and its expanded review was not fully independently blinded.Variant names and predicted choices are masked, though prose can reveal conditions; the judge evaluates reasoning and answer together.Source: §A.2.5 · p. 26
  • Paired ratings measure grounded usefulness after auxiliary candidates are removed, distinguishing clear gains, smaller meaningful gains, and no meaningful difference.The protocol warns against rewarding candidate guessing or length alone, and directs judges to preserve meaningful uncertainty and explain concrete tradeoffs.Source: §A.2.5 · p. 27
  • Per-trace flags record candidate-only leakage, generic or vacuous output, and unsupported claims, but inform rather than mechanically determine pair ratings.Flags can co-occur, and identical left/right traces receive no-difference ratings.Source: §A.2.5 · pp. 27–28

A.3 DERIVATION OF THE MIXTURE-IWAE GRADIENT

  • The gradient derivation defines prior, posterior, and mixture distributions, uses a reward-induced likelihood, and assumes the retrieval reward is stop-gradient.The mixture coefficient is independent of model parameters, so the reward-induced likelihood has no direct gradient with respect to them.Source: §A.3 · p. 28
  • The derivation combines score-function and importance-weight gradients, permits a leave-one-out baseline independent of the sampled trajectory, and decomposes the mixture score using component responsibilities.The baseline can be formed from the remaining K −1 trajectories.Source: §A.3 · pp. 28–29
  • The practical objective combines prior and posterior GRPO with importance-weighted prior learning, approximating the mixture signal while assigning reachability influence to the weighted-prior term.This GRPO substitution is an approximation; the three-term derivation assumes a detached retrieval reward.Source: §A.3 · pp. 29–30

A.4 PROBE DATASET CONSTRUCTION

  • MMEB-2-PROBE contains 1,000 query–candidate triplets sampled across all 78 subsets, with one official positive and one qrel-safe random negative per instance.The probe covers image, video, and visual-document queries and is reserved for diagnostic analyses, not training, hyperparameter selection, or checkpoint selection.Source: §A.4 · p. 30

A.5 DETAILED RQ1 EXPERIMENT

  • The RQ1 experiment compares input-only reasoning with positive-conditioned, labeled positive/negative, and blind candidate-pair prompts, which treat candidates as cues and require query-supported, self-contained traces.The blind prompt also asks the annotator to predict the positive candidate before generating its trace.Source: §A.5 · pp. 30–31
  • The blinded audit compared candidate-conditioned traces with matched input-only traces across 1,000 queries, plus eight-sample comparisons on an 81-query subset, using randomized presentation and ratings from −2 to +2.Candidate-only leakage was treated as an operational train–test information mismatch, not benchmark contamination.Source: §A.5 · pp. 31–32
  • The oracle best-of-eight analysis selects each query’s highest-scoring conditioned trace retrospectively, so it estimates potential selection gains rather than a deployable policy.The primary repeated-sampling result averages all eight samples; a predefined single-sample estimate is also reported.Source: §A.5 · p. 32
  • Blind-trace results are stratified by positive-candidate prediction correctness, but this is a conditional association because correct and incorrect groups may differ in query difficulty.The 81-query audit set contains 521 prediction-correct and 127 prediction-incorrect blind generations.Source: §A.5 · p. 32

A.6 DETAILED RQ3 EXPERIMENT

  • ESS rose from about 2.1 to 3.6 while eligible traces grew to 6–7.5 per query, with a mean ESS-to-uniform bound ratio of 0.53, indicating selective transfer.Transfer weights therefore concentrate on a subset of eligible posterior traces rather than averaging them uniformly.Source: §A.6 · p. 33, Figure 5 · p. 35
  • Full had a lower posterior–prior log-probability gap than w/o TS on all ten datasets, including 0.30 vs. 0.60 on CIRR, 0.33 vs. 0.68 on Visual7W, and 0.55 vs. 0.83 on ColPali.Under Full, the gap stayed flat or decreased, supporting the role of transfer SFT in keeping posterior traces reachable by the prior.Source: §A.6 · p. 33, Figure 6 · p. 35

Related papers

Loading 2610.32474v1…