WeMM-Embedding: WeChat Multi-Modal Embedding Technical Report

1 Introduction

• CLIP-style dual encoders [33] [37] [49] align modalities but do not naturally represent mixed-modality inputs; subsequent encoder-based extensions [51] [54] remain limited in generality.Multimodal embeddings map heterogeneous inputs into a shared dense representation space, supporting classification [20], retrieval [2] [25] [48], clustering [43], recommendation [3] [13], and agentic systems [9] [14] [46].Source: §1 · p. 2
• MLLMs [1] [8] [24] support arbitrarily interleaved text, images, and videos, providing a foundation for universal multimodal embeddings.Early studies adapted MLLM hidden states into embeddings through continued contrastive training on paired multimodal data [19] [23]. Later advances in large-scale paired-data synthesis and knowledge distillation improved performance [4] [52].Source: §1 · p. 2
• 2B, 4B, and 9B WeMM-Embedding models use two-stage training, progressing from broad multimodal alignment to curated, fine-grained relevance learning.The first stage uses several hundred million heterogeneous pairs spanning modalities, tasks, and domains. The second improves semantic balance and data quality, adds harder negatives, and introduces richer relevance supervision and cross-scale knowledge transfer.Source: §1 · p. 2
• 80.6 overall on MMEB-v2 [28] places the 9B WeMM-Embedding model first on the official leaderboard, ahead of all listed open-source and proprietary models.The 2B variant surpasses previously leading 8B open-source baselines on MMEB-v2 [28]. It also compares favorably with leading proprietary models on the cross-modal retrieval suite reported in Gemini Embedding 2 [35].Source: §1 · p. 2
• 26-task in-house benchmark gains and consistent improvements across 14 online A/B tests demonstrate WeMM-Embedding's practical performance across WeChat applications.The models are deployed at scale in recommendation and search across WeChat Channels, Official Accounts, Moments, and e-commerce services.Source: §1 · p. 2

2 Data Construction

• Several hundred million examples combine public, web-scale, synthetic, and in-house data spanning text, images, videos, and interleaved inputs.A curated collection complements the large-scale corpus with greater semantic diversity, higher data quality, and more informative supervision.Source: §2 · p. 2, §2.1 · p. 3

2.1 Large-Scale Training Data Collection

• A unified source–target format accommodates optional instructions, hard negatives, and graded relevance scores, enabling task-specific batches within one multi-task pipeline.Instructions distinguish matching relations, relevance scores construct relative ordering constraints, and semantic target masking excludes repeated-target false negatives from in-batch negatives. Large-scale graded supervision uses manually assigned discrete relevance levels for ranking-oriented training; reranker-derived scores enter during curation.Source: §2.1 · pp. 3–4
• Weakly supervised image–text and video–text pairs provide broad, coarse correspondence, while explicit captions supply more direct, fine-grained visual–language supervision.Captions range from concise summaries to detailed descriptions of entities, attributes, relations, spatial context, actions, and events.Source: §2.1 · pp. 3–4
• Retrieval, classification, and multimodal question-answer pairs extend supervision across query–candidate matching, source–label recognition, perception, reasoning, and document understanding.Retrieval includes composed queries, instructions, long contexts, spatial grounding, temporal localization, and agent-related tasks; classification represents image/video categories through class names or natural-language descriptions. Question-answer pairs additionally cover relational and spatial understanding, OCR, knowledge-intensive understanding, charts, and events.Source: §2.1 · p. 4

2.2 Curated Data Construction

• Approximately one tenth the large-scale collection’s size, the curated dataset uses Semantic-ID-guided resampling to reduce repeated exposure to frequent semantic patterns.Inspired by Semantic IDs [34], an intermediate WeMM-Embedding checkpoint encodes each pair’s longer serialized side, and three-level residual k-means quantization (RQ-KMeans) [26] produces a three-element identifier. Codebook assignment densities lower sampling rates for frequent codes and retain rarer codes more often without enforcing uniformity.Source: §2.2 · p. 4
• Multimodal large language models filter source–target mismatches and refine textual fields, correcting factual inconsistencies while preserving original alt-text style and detail.Filtering evaluates whether each pair satisfies its intended matching relation in the corresponding task context.Source: §2.2 · p. 4
• Selective hard-negative enrichment generates plausible incorrect text targets and retrieves semantically similar image or video candidates using intermediate WeMM-Embedding checkpoints.Multimodal large language models generate text negatives from the source and positive target, while visual candidates come from task-specific pools. Rerankers score a smaller subset of mined candidate sets to provide finer-grained relevance supervision.Source: §2.2 · pp. 4–5

3.1 Modeling

• WeMM-Embedding encodes ordered, interleaved text, images, and videos using a final <embedding> token’s L2-normalized hidden state.Native text and visual processing pipelines produce tokens in their original segment order, optionally preceded by a task-specific instruction.Source: §3.1 · p. 5
• Multiple <embedding> tokens enable modality-specific outputs, while Matryoshka Representation Learning [21] provides multiple embedding dimensions through prefix truncation and re-normalization.Tokens placed after a video and its subsequent ASR transcript yield video-only and joint video–text representations in one causal forward pass. All supported embedding dimensions are likewise available from a single forward pass.Source: §3.1 · pp. 5–6

3.2 Training Strategy

• Two-stage training establishes a general multimodal embedding space through large-scale alignment, then refines it on curated data using supervision-dependent objectives.Both stages use a unified pair-based representation; refinement combines contrastive learning, explicit hard negatives, selective reranker supervision, and larger-model embedding distillation.Source: §3.2 · p. 6

3.2.1 Stage 1: Large-Scale Multimodal Alignment

• Several hundred million source–target pairs support Stage 1 alignment, using InfoNCE [30] with in-batch negatives, available hard negatives, and duplicate-aware masking.Each batch uses one data source, with tasks interleaved across batches; similarity is cosine-based with a learnable temperature. Masking removes candidates associated with near-duplicate sources and candidates resembling the positive target, including explicit hard negatives.Source: §3.2.1 · p. 6
• Score-gap-weighted CoSENT-style ranking preserves graded relevance, while Matryoshka Representation Learning [21] evaluates each batch’s objective independently across supported dimensions.Larger relevance-label gaps receive greater weight, while equal labels impose no ordering constraint. Dimension-specific losses use independently truncated, normalized embeddings and corresponding loss weights.Source: §3.2.1 · pp. 6–7

3.2.2 Stage 2: Curated Fine-Tuning and Distillation

• Reranker supervision is restricted to tasks with reliable gains because reranking embedding-retrieved candidates does not consistently improve multimodal performance.Dedicated multimodal rerankers score annotated positives and mined hard negatives, providing within-query ordering comparisons weighted by score gaps. Reranker-scored batches use this ranking objective instead of contrastive learning.Source: §3.2.2 · p. 8
• Bidirectional embedding distillation transfers teacher similarity distributions in both source-to-target and target-to-source directions, substantially benefiting compact WeMM-Embedding variants.Online soft targets preserve relative candidate similarities without additional offline relevance annotations and support heterogeneous batch types. Distillation supplements the selected task objective whenever a larger teacher is available.Source: §3.2.2 · pp. 8–9
• 9B WeMM-Embedding provides frozen Stage 2 supervision for the 2B and 4B variants; the final 9B model instead merges specialized variants [47].Without a larger embedding teacher, the 9B model combines variants trained with complementary data mixtures and training configurations.Source: §3.2.2 · p. 9

• Public evaluation covers the MMEB benchmark series [15] [28] and cross-modal retrieval benchmarks reported in the Gemini Embedding 2 study [35].Source: §4 · p. 9
• 26 tasks derived from real-world WeChat applications comprise the in-house benchmark.Source: §4 · p. 9
• Comparisons include representative state-of-the-art multimodal embedding models, spanning both open-source and proprietary commercial models.Source: §4 · p. 9

4.1 Performance on the MMEB Series

• MMEB evaluations use Hit@1 for image, video, audio, and agent tasks, and NDCG@5 for text and visual-document tasks.MMEB-v2 [28] spans 78 datasets covering images, videos, and visual documents, with classification, question answering, retrieval, and visual grounding tasks.Source: §4.1 · p. 9, §4.1.1 · p. 9

4.1.1 General Multimodal Representation

• 80.6 overall places WeMM-Embedding-9B first on the official MMEB-v2 leaderboard, outperforming all listed open-source and proprietary models.WeMM-Embedding-4B scores 79.2, outperforming all compared 8B–9B baselines.Source: §4.1.1 · p. 9, Table 1 · p. 10
• 77.9 overall on MMEB-v2 puts WeMM-Embedding-2B ahead of Qwen3-VL-Embedding-2B by 4.7 points and DME-2B [6] by 3.1 points.The 2B model also slightly surpasses Qwen3-VL-Embedding-8B.Source: §4.1.1 · p. 9, Table 1 · p. 10

4.1.2 Text and Agent-Centric Tasks

• 11 unsupported audio tasks receive zero scores in V3-All because the current WeMM-Embedding models do not support audio input.For VLM2Vec [19], VLM2Vec-V2 [28], GME, and Qwen3-VL-Embedding, V3-All combines originally reported MMEB-v2 results with results on newly added MMEB-v3 tasks.Source: §4.1.2 · p. 10
• 53 text tasks and 47 agent tasks expand MMEB-v3 [15] evaluation alongside audio evaluation and the MCMR image-retrieval task.Text tasks cover reasoning retrieval, instruction following, long-context retrieval, multi-condition retrieval, and general retrieval. Agent tasks cover tool, GUI, and memory retrieval.Source: §4.1.2 · p. 10
• 59.5 V3-All for WeMM-Embedding-9B and 58.2 for WeMM-Embedding-4B improve upon WeMM-Embedding-2B’s 56.0, which already outperforms all compared MMEB-v3 baselines.WeMM-Embedding-2B also exceeds all compared baselines on Text and Agent, scoring 45.3 and 45.1, respectively.Source: §4.1.2 · p. 10

Cross-Modal Retrieval Evaluation

• Evaluation measures text alignment with images, videos, and visual documents, using reported proprietary results and direct evaluations of Qwen3-VL-Embedding and WeMM-Embedding.Proprietary results come from the Gemini Embedding 2 technical report [35]. Direct evaluations use the corresponding public evaluation sets and each benchmark’s specified metric.Source: §4.2 · p. 10
• 81.7 overall for WeMM-Embedding-9B exceeds 80.8 for WeMM-Embedding-4B and 79.8 for WeMM-Embedding-2B across 12 public cross-modal retrieval benchmarks.The overall score averages results across the 12 datasets.Source: Table 3 · p. 11, §4.2 · p. 11
• 79.8 overall places WeMM-Embedding-2B above all compared open-source baselines and compares favorably with leading proprietary models.The supplied passages do not provide individual baseline scores.Source: Table 3 · p. 11, §4.2 · p. 11

4.3 In-House Evaluation

• All five benchmark categories show WeMM-Embedding-2B outperforming the representative open-source baseline.The benchmark comprises 26 real-world WeChat tasks covering classification, search, cross-domain content matching, article relevance, and video relevance. AVG denotes the average across all 26 tasks.Source: §4.3 · p. 11, Table 4 · p. 12
• 14 online A/B tests show consistent gains from WeMM-Embedding in recommendation systems, with improvements subsequently rolled out in production.Deployments span WeChat Official Accounts, WeChat Channels, and e-commerce content. Reported improvements cover content matching, recommendation quality, and user engagement, with notable gains for long-tail and newly published content.Source: §4.3 · pp. 11–12
• Complementary text, cover-image, and video-frame signals support candidate retrieval, ranking feature construction, user sequence modeling, and cross-domain content understanding.Semantic IDs derived from these representations provide compact discrete features for indexing and sequence modeling.Source: §4.3 · p. 12
• WeChat search supports unimodal and cross-modal retrieval, improving semantic relevance and retrieval quality across text, image, and video content.Search deployments cover WeChat Channels videos, WeChat Official Accounts articles, and WeChat Moments.Source: §4.3 · p. 12

4.4.1 MRL Analysis

• 98.7% of 2,048-dimensional performance is retained at 256 dimensions on both image and video tasks for WeMM-Embedding-2B on MMEB-v2.At 512 dimensions, retention reaches 99.2% for image tasks and 98.8% for video tasks. Visual-document tasks are more sensitive to dimensionality reduction, potentially because of their higher information density and dense textual content.Source: §4.4.1 · p. 12, Figure 3 · p. 13
• 256-dimensional embeddings retain more than 97% of full-dimensional performance across classification, question answering, and retrieval, supporting compact representations for efficiency-sensitive applications.Classification is least sensitive, question answering moderately sensitive, and retrieval most affected, especially at 64 and 128 dimensions. Gains diminish beyond 256 dimensions, with 256 or 512 dimensions identified as practical efficiency-sensitive choices.Source: §4.4.1 · p. 12, Figure 3 · p. 13

4.4.2 Stage-1 Design Analysis

• 3.4 points are lost on MMEB-v2 when task-consistent batching is replaced by mixed sampling, the largest tested Stage-1 design effect.Task-consistent batches provide more informative in-batch negatives; removing task-specific instructions costs 0.8 points, with the largest degradation on visual documents, indicating their value for task-specific matching objectives. Removing duplicate-aware masking costs 0.5 points; masking excludes duplicate or semantically equivalent negative targets, accommodating repeated-label tasks such as classification.Source: Table 5 · p. 13, §4.4.2 · p. 13

4.4.3 Cumulative Stage-2 Analysis

• 2.2 points of overall MMEB-v2 improvement result from cumulatively adding Stage-2 strategies to the 2B model’s Stage-1 checkpoint.Curated data improves image and video tasks through balanced resampling, quality refinement, and hard-negative enrichment; reranker supervision adds gains, largest on visual documents. Embedding-teacher distillation improves all three domains through dense similarity supervision, while higher-resolution inputs and denser video frame sampling add gains, particularly for video.Source: §4.4.3 · pp. 13–14, Table 6 · p. 14

Conclusion

• WeMM-Embedding achieves state-of-the-art public-benchmark performance and establishes a new performance–efficiency frontier.The universal multimodal embedding family spans multiple model scales.Source: §5 · p. 14
• 26-task in-house benchmark results and 14 online A/B tests show consistent gains for WeMM-Embedding.The models are deployed at scale across multiple recommendation and search systems within WeChat.Source: §5 · p. 14
• Two-stage training progresses from broad multimodal alignment to finer-grained representation learning using curated data, richer relevance supervision, and cross-scale knowledge transfer.Source: §5 · p. 14

Future Work

• Future work will extend WeMM-Embedding toward omni-modal inputs, scale the family to larger sizes, and improve data curation and fine-grained relevance supervision.Source: §5 · p. 14