Learning Multimodal Embeddings with Evidence-Aligned Readout

1 Introduction

• The controlled comparison finds the Semantic–Mixed advantage grows from 0.65 points with Distributed readout to 2.39 with Boundary readout.The resulting co-design interaction is 1.74 points.Source: Figure 1 · p. 2, §1 · p. 2
• The central question is whether semantic units generated as retrieval evidence can specify where contextualized states are read for embedding construction.Useful evidence can express a composed-image request while preserving visual cues, but generating it alone does not determine its contribution to the embedding.Source: §1 · pp. 1–2
• EviAlign organizes retrieval evidence into Entity, Attribute, Relation, Detail, and Summary units, then pools their contextualized boundary states into one normalized embedding.Generation supervision and contrastive retrieval jointly optimize the shared MLLM.Source: §1 · p. 2
• Across 12 MMEB retrieval tasks, EviAlign reaches 76.9 average Recall@1 with 500K training pairs, and the controlled study yields a 1.74-point co-design interaction.The approach retains conventional single-vector indexing and scoring.Source: §1 · p. 2, Figure 1 · p. 2
• Matched controls show that semantic generation with a trailing readout and additional readout tokens alone do not reproduce the full gain.The introduction frames the contribution as coordinating evidence organization with representation readout, not evidence generation alone.Source: §1 · pp. 2–3

Multimodal Embedding Learning

• Direct multimodal embedders construct retrieval representations from contextualized states, while UniME-V2 and Qwen3-VL-Embedding scale this approach; Qwen3-VL-Embedding reports 80.2 on MMEB-V2 Image RET (Gu et al., 2026a; Li et al., 2026c).SnapBench evaluates snap-and-ask retrieval under paired visual and textual corruptions (Chen et al., 2026).Source: §2 · p. 3

Generation-Enhanced Embedding Learning

• Generation-enhanced methods expose retrieval-relevant information through reasoning traces, context, rewrites, or latent computation, while EviAlign asks whether semantic units can specify embedding readout locations.Examples include UME-R1 and Think-Then-Embed (Lan et al., 2026; Cui et al., 2026), Reasoning Guided Embeddings and RIME (Liu et al., 2025a; Wu et al., 2026a), and PLUME and LaME (He et al., 2026; Wu et al., 2026b).Source: §2 · p. 3

Structured Evidence in Multimodal Decisions

• Structured-evidence systems support verification, ledgers, or streaming decisions; EviAlign instead connects generated evidence units to the boundary states aggregated for retrieval.Examples include EviRank (Du et al., 2026a), LEDGERMIND (Du et al., 2026c), and Omni-Streaming Thinking (Du et al., 2026b).Source: §2 · p. 3

Representation Readout for Retrieval

• Readout alternatives include trailing states, learned aggregation, predefined-span late chunking, and multi-vector interaction; EviAlign pools generated evidence-boundary states into one vector.DRE-SLCL provides a domain-specific example of local-feature aggregation with global representation learning (Jin et al., 2025).Source: §2 · p. 4, Figure 2 · p. 4

3 EviAlign

• EviAlign shares one MLLM across Semantic Evidence Generation and Boundary Readout, converting task-relevant evidence into a single retrieval embedding.Source: §3 · p. 4

3.1 Problem Formulation and Design Principle

• Evidence-aligned readout couples task-relevant evidence selection, evidence-defined readout roles and locations, and joint retrieval integration in one normalized vector.Source: §3.1 · p. 5

3.2 Semantic Evidence Generation

• Generation partitions task-relevant output into semantic spans followed by boundary tokens; five roles cover Entity, Attribute, Relation, Detail, and Summary.Source: §3.2 · pp. 5–6

3.3 Boundary Readout

• Each boundary state summarizes the input-conditioned evidence prefix available after its completed unit; mean pooling and L2 normalization produce a cosine-scored single vector.Source: §3.3 · p. 6

3.4 Training and Inference

• Training combines supervised evidence-sequence language modeling with contrastive retrieval learning, while inference generates evidence before reading its boundaries.Source: §3.4 · pp. 6–7

4 Experiments

• MMEB evaluation covers 12 retrieval tasks, with 1,000 test queries and 1,000 candidates per query; Recall@1 is averaged across tasks and split averages cover eight in-domain and four out-of-domain tasks.Source: §4.1 · p. 7, Table 1 · p. 8
• The main model uses 500K training pairs, Qwen3-VL-8B-Instruct, a frozen vision encoder, and one training epoch, with averaged and L2-normalized readouts.Source: §4.1 · p. 7
• Baselines span direct embedding, explicit generation or reasoning, and latent-reasoning multimodal embedders.Source: §4.1 · p. 8

4.2 Main Results

• 76.9 average Recall@1 across 12 MMEB retrieval tasks is achieved by EviAlign with 500K training pairs.With Qwen2-VL-7B, EviAlign reaches 75.5 average Recall@1; under that backbone it exceeds UME-R1 and RIME.Source: §4.2 · p. 8

4.3 Controlled Study of Evidence–Readout Co-design

• 1.74 points is the co-design interaction: Semantic–Mixed advantage rises from 0.65 points with Distributed readout to 2.39 with Boundary readout.The evidence targets contain the same five labeled spans; Semantic preserves role-to-boundary correspondence, while Mixed permutes the spans.Source: §4.3 · pp. 8–9, Table 2 · p. 9

4.4 Analysis of Representation Construction

• With trailing readout, semantic evidence and free-form CoT score 73.74 vs. 73.75; five readouts without generation reach 70.30, while boundary readout reaches 76.94.The boundary-readout model outperforms the separately trained trailing-readout model by 3.20 points.Source: §4.4 · pp. 9–10, Table 3 · p. 10
• 1,502.5 ms per batch is EviAlign’s reported time at batch size 4, versus 94.6 ms for direct encoding; CoT-plus-semantic boundary readout takes 9,825.3 ms.EviAlign averages 114.3 generated tokens; 99.52% of 12,000 query generations contain all five boundary tokens exactly once and in order.Source: §4.4 · p. 10
• Scaling from 2B to 7B/8B improves overall results from 72.6 to 75.5 for Qwen2-VL and from 73.4 to 76.9 for Qwen3-VL.Out-of-domain gains are 5.8 and 8.2 points, respectively, versus in-domain gains of 1.4 and 1.2 points.Source: §4.4 · pp. 10–11
• 76.9 is the highest reported result among the tested joint configurations: performance rises from 72.8 at K = 1 to 75.4 at K = 3 and falls to 76.2 at K = 7.The five-unit schema comprises Entity, Attribute, Relation, Detail, and Summary; K = 7 adds Context and Emotion.Source: §4.4 · p. 11
• 76.94 average Recall@1 comes from aggregating all five readouts; the aggregate exceeds every individual readout on seven tasks and has the highest 12-task average.Removing any one readout reduces the average by 0.23–0.36 points; different tasks favor different individual readouts.Source: §4.4 · p. 11

5 Conclusion

• EviAlign uses the same evidence boundaries to organize generation and construct one retrieval vector, preserving standard indexing and scoring.The conclusion supports using semantic evidence structure to determine where retrieval representations are read.Source: §5 · p. 12

• The MMEB retrieval evaluation suite comprises 12 benchmarks.Source: §B.1 · p. 18
• Its challenges range from image–text alignment and composed image retrieval to entity grounding, fine-grained attribute matching, and visual document understanding.Source: §B.1 · p. 18
• The benchmarks collectively span varied modality configurations and domains.Source: §B.1 · p. 18

• VisDial retrieves the target image from a full dialogue history, testing resolution of visual references across turns.Source: §B.1.1 · p. 18
• CIRR pairs a reference photograph with relative modification text, requiring retrieval of an image reflecting the requested change.Source: §B.1.1 · p. 19
• VisualNews evaluates caption-to-image and image-to-caption retrieval for news photographs and original captions.Source: §B.1.1 · p. 19
• MSCOCO evaluates both text-to-image and image-to-text retrieval using images of everyday scenes and five human-written captions per image.Source: §B.1.1 · p. 19
• NIGHTS uses a reference image to retrieve its human-judged matching perturbed version, testing robustness to domain shifts.Source: §B.1.1 · p. 19
• WebQA isolates evidence retrieval by selecting the candidate containing an answer to a natural-language query.Source: §B.1.1 · p. 19

B.1.2 Out-of-Domain Datasets

• FashionIQ tests fine-grained attribute changes in fashion retrieval, while Wiki-SS-NQ retrieves Wikipedia screenshots that answer questions and requires processing page structure and layout.Source: §B.1.2 · p. 19
• OVEN links visual queries to knowledge entries, while EDIS matches entity-rich news queries to news images paired with headlines.Source: §B.1.2 · p. 20

B.2 Baseline Details

• The baselines include direct MLLM embedding, reasoning-driven generation, latent computation, and learned reason-token approaches, with varied training scales and representation construction.GME uses about 8 million samples; UniME-V2 reports 73.1 average Precision@1 over 12 MMEB tasks.Source: §B.2 · pp. 20–21

B.3 EviAlign-Evidence Construction

• EviAlign-Evidence uses GLM-generated semantic-evidence targets aligned to MMEB-V1 training examples; filtering retains approximately 500K query–candidate pairs for generation supervision and teacher-forced readout training.Source: §B.3 · pp. 21–22

B.4 Hyperparameter Configuration

• The hyperparameter and hardware specifications for EviAlign are reported in Table 5.Source: §B.4 · p. 22, Table 5 · p. 22

B.5 Generation, Readout-Capacity, and Evidence-Teacher Controls

• The controlled decomposition contrasts no-generation multi-readout, MLTR, free-form-CoT trailing readout, and CoT followed by semantic evidence with boundary readouts.Source: §B.5 · p. 23

B.6 Factorial Evidence–Readout Construction

• The factorial construction preserves evidence spans, unit count, boundary tokens, and length distribution while varying role-to-boundary consistency and readout placement.Distributed readout positions depend on sequence length; boundary readouts fall at evidence-unit ends, with the two conditions sharing token identities, count, and mean aggregation.Source: §B.6 · p. 23

C Additional Analyses

• Table 6 illustrates the factorial conditions with a CIRR query and abbreviated evidence spans; Mixed shows one sample permutation.Source: C Additional Analyses · p. 24, Table 6 · p. 24

C.2 Representation Analysis of Evidence-Aligned Readouts

• Removing any one readout lowers the 12-task average by 0.23–0.36 points, and no individual state matches the full five-state aggregate.Source: Table 8 · p. 25, §C.2 · p. 25

C.2.1 Readout-Level Representation Geometry

• Positive-pair alignment is 0.53 and 0.56 in the two controls and 0.63 with semantic evidence and boundary readout, while within-input dispersion remains comparable.Source: Table 11 · p. 26, §C.2.1 · p. 26

C.3 Per-Task Backbone Comparison

• The complete per-task comparison reports Qwen2-VL-7B and Qwen3-VL-8B results with in-domain, out-of-domain, and overall averages.The supplied text does not state the table’s individual values.Source: §C.3 · p. 26, Table 12 · p. 26

C.4 Structured vs. Free-Form Training Targets

• EviAlign’s structured targets score higher than free-form CoT on all three judged dimensions, yet trailing-readout retrieval remains nearly tied at 73.74 versus 73.75.The dimensions are retrieval relevance, faithfulness, and specificity; ties are excluded from win rates in Figure 5.Source: §C.4 · p. 27, Figure 5 · p. 27

C.5 Embedding Visualization

• On four representative MMEB subsets, EviAlign produces more interleaved query and target embedding distributions than VLM2Vec-V2.The subsets include in-domain CIRR and MSCOCO image-to-text, and out-of-domain EDIS and OVEN.Source: §C.5 · p. 27, Figure 6 · p. 28

C.6 Attention Visualization

• Attention visualizations show attention concentrated near semantic evidence boundaries, alongside layer-wise patterns across surrounding sequence positions.Figure 7 compares the final evidence-boundary token with the trailing token of free-form CoT and marks the five boundary positions.Source: §C.6 · p. 28, Figure 7 · p. 28

C.7 Data Scaling Analysis

• From 4K training pairs, retrieval improves progressively with more data, and scaling from 256K to 500K improves all 12 tasks.The text notes especially clear gains on CIRR, NIGHTS, Wiki-SS-NQ, and OVEN; at 1K–2K, boundary-token omissions often cause fallback to the final hidden state.Source: §C.7 · p. 29, Figure 8 · p. 29

C.8.1 Representation-Construction Efficiency

• 76.94 average Recall@1 across 12 MMEB tasks accompanies 114.3 generated tokens, 1,502.5 ms per batch, and 19.49 GB peak memory; adding CoT yields 76.59 at higher costs.Both variants produce one vector scored with cosine similarity; query-side evidence generation remains part of online encoding.Source: §C.8 · p. 29, Table 13 · p. 30, §C.8.1 · p. 30

C.8.2 Index Storage

• EviAlign’s five internal readouts are aggregated into one vector, so corpus index storage scales as |C|d rather than with the number of evidence units.The raw FP16 storage calculation for 1M items excludes compression, index metadata, and system overhead; EviAlign remains compatible with FAISS.Source: §C.8.2 · p. 30, Table 14 · p. 31

C.9 Evidence Schema Details

• The selected evidence schema uses five units—Entity, Attribute, Relation, Detail, and Summary—with stable semantic roles and explicit readout boundaries.The section says joint evidence-schema and readout configurations support this implementation among the tested settings.Source: §C.9 · pp. 30–31

D Prompt Templates

• The experiments use four prompt-template categories: inference, evidence-target construction, evidence-configuration ablations, and judgment.The inference prompt is applied identically to every query and candidate.Source: D Prompt Templates · p. 33

D.1 Inference Prompt of EviAlign

• The shared inference template represents text, images, or both with Entity, Attribute, Relation, Detail, and Summary tokens whose final-layer states provide five boundary readouts.The five boundary tokens are registered vocabulary items initialized from the [EOS] embedding.Source: §D.1 · p. 33

D.2 Training Data Construction Prompt of EviAlign-Evidence

• GLM-4.1V-9B-Thinking converts MMEB-V1 retrieval inputs into five-unit targets, which are filtered and paired with approximately 500K query–candidate examples.The prompt asks for retrieval-task-relevant information verified against the original input and exactly five output lines.Source: §D.2 · p. 34
• The construction prompt binds identifying attributes to Entity, reserves Attribute for scene-wide properties, and grounds Detail in visible discriminative cues.It also requires a specific retrieval-target Summary and permits empty Detail only when cues are absent.Source: §D.2 · p. 34

D.3 Evidence-Configuration Ablation Prompts

• The K=7 ablation inserts Context and Emotion between Detail and Summary, while K=1 and K=3 prompts remove unused units from the standard format.Data construction and inference otherwise follow the same format.Source: §D.3 · pp. 34–35
• The seven-unit ablation prompt distinguishes situational Context from affective Emotion and keeps both separate from scene-wide Attribute descriptions.It asks for empty Context or Emotion when no event context or affective tone is discernible.Source: §D.3 · p. 35

D.4 LLM-as-Judge Prompt

• The judge scores each training target for faithfulness, specificity, and retrieval relevance on a 1–5 scale, comparing paired targets and treating equal scores as ties.The output format is a JSON object with one integer score for each dimension.Source: §D.4 · p. 36