PaperlayerView exact revision on arXiv

Source-linked AI summary

WeMM-Embedding: WeChat Multi-Modal Embedding Technical Report

Junjie Zhou, Ke Mei, Lei Li, Tianyi Wang, Fengyun Rao, Jing Lyu

arXiv:2608.24053v1cs.CVcs.CLcs.IR

TL;DR

Existing multimodal embedding models do not naturally represent arbitrarily interleaved multimodal inputs in a shared space. WeMM-Embedding addresses this with a scalable two-stage training strategy and achieves state-of-the-art benchmark performance, including an 80.6 overall score for its 9B variant.

  • Problem

    Existing modality-specific encoding pathways do not naturally support joint representations of interleaved multimodal inputs.

  • Method

    WeMM-Embedding trains 2B, 4B, and 9B models through broad multimodal alignment followed by curated fine-grained relevance learning and cross-scale knowledge transfer.

  • Results

    80.6 overall score: the 9B variant ranks first on the official MMEB-v2 leaderboard, while the family achieves state-of-the-art performance across public benchmarks.

  • Takeaways & Limitations

    WeMM-Embedding delivers consistent gains across a 26-task in-house benchmark and 14 online A/B tests and is deployed across WeChat recommendation and search systems.

  • Takeaways & Limitations

    Reranker supervision is restricted to settings where reranking yields reliable improvements because gains occur only on a limited subset of multimodal tasks.

Abstract

from arXiv · show

Universal multimodal embeddings are becoming a core component of modern AI systems, enabling heterogeneous content to be represented in a shared space for applications such as retrieval, recommendation, classification, and agentic systems. In this report, we present WeMM-Embedding, a family of universal multimodal embedding models supporting text, images, videos, visual documents, and arbitrarily interleaved multimodal inputs with flexible output dimensions. The family comprises 2B, 4B, and 9B variants and is trained in two stages: a large-scale multimodal alignment stage, followed by a refinement stage using curated data, fine-grained relevance supervision, and cross-scale knowledge transfer. Across extensive evaluations, WeMM-Embedding achieves leading performance on multiple public benchmarks. Notably, the 2B variant already surpasses the previously leading 8B open-source baseline on MMEB-v2, while the 9B variant further achieves a new state-of-the-art overall score of 80.6. WeMM-Embedding also demonstrates strong practical performance across WeChat applications, with substantial gains on a 26-task in-house benchmark and consistent improvements across 14 online A/B tests. It has been deployed at scale across recommendation and search applications, including WeChat Channels, Official Accounts, Moments, and e-commerce services. We have released the model weights and code to facilitate future research at https://github.com/Tencent/WeMM-Embedding.

1 Introduction

WeMM-Embedding is a universal multimodal embedding family trained through broad multimodal alignment followed by fine-grained relevance learning. It achieves leading public-benchmark performance, with parameter-efficient gains from 2B to 9B and practical applicability.

  • Motivation: Multimodal embeddings map text, images, and videos into a shared dense representation space supporting classification [20], retrieval [2] [25] [48], recommendation [3] [13], and agentic systems [9] [14] [46].They also support clustering [43] and other downstream applications.
  • Motivation: MLLMs [1] [8] [24] enable arbitrary interleaved text, image, and video inputs while providing broad pretrained capabilities for general-purpose representation learning.These advantages motivate adapting MLLM hidden states into universal multimodal embeddings.
  • Method: WeMM-Embedding spans 2B, 4B, and 9B scales and uses two-stage training that progresses from broad multimodal alignment to finer-grained relevance learning.The first stage uses several hundred million heterogeneous pairs across diverse modalities, tasks, and domains.
  • Results: 9B WeMM-Embedding reaches an overall MMEB-v2 score of 80.6, ranking first on the official leaderboard and outperforming all listed open-source and proprietary models.The result is summarized in Figure 1.
  • Results: The compact 2B variant surpasses previously leading 8B open-source baselines on MMEB-v2, demonstrating strong parameter efficiency.This comparison is summarized in Figure 1.

2 Data Construction

WeMM-Embedding is trained on several hundred million heterogeneous multimodal examples unified as source–target pairs, complemented by a curated dataset about one tenth as large. The curated stage improves semantic balance, quality, and supervision through resampling, filtering, and hard-negative enrichment.

  • Large-Scale Collection: Several hundred million examples span text, images, videos, interleaved multimodal inputs, diverse domains, tasks, and supervision sources.Data combine public datasets, web-scale weak supervision, task-oriented synthetic data, and in-house collections.
  • Unified Pair-Based Format: All tasks use a unified source–target pair format supporting instructions, hard negatives, graded relevance, and multimodal source and target structures.Task-specific batches enable a unified multi-task pipeline; semantic target masking removes false negatives caused by repeated classification targets.
  • Data Coverage and Composition: The corpus includes weakly supervised and caption pairs, retrieval and classification pairs, multimodal question-answer pairs, and graded-relevance pairs.These cover coarse and fine-grained visual-language alignment, composed and reasoning-heavy retrieval, recognition, OCR, document and chart comprehension, and ranking supervision [17].
  • Curated Dataset: The curated dataset is approximately one tenth the size of the large-scale collection and targets semantic balance, data quality, and more informative supervision.Its construction combines Semantic-ID-guided resampling, multimodal quality filtering and text refinement, and selective hard-negative enrichment.
  • Curated Dataset: Hard negatives use generated incorrect text candidates, embedding-retrieved image or video candidates, and reranker-assigned relevance scores for difficult examples.The procedure varies by target modality and provides finer-grained supervision among semantically similar candidates.

3 Modeling and Training Strategy

WeMM-Embedding comprises 2B, 4B, and 9B models built on native multimodal Qwen3.5 backbones. It encodes interleaved multimodal inputs into normalized representations and trains them through large-scale alignment followed by curated refinement.

  • Architecture and Input Encoding: The 2B, 4B, and 9B models use corresponding native multimodal Qwen3.5 backbones to encode arbitrary interleavings of text, images, and videos.Textual and visual tokens preserve their original segment order before processing by the backbone.
  • Architecture and Input Encoding: A dedicated <embedding> token yields normalized representations, while multiple tokens can extract video-only and joint video–text embeddings in one forward pass.The default token attends to all preceding multimodal content under the native causal attention mask.
  • Flexible Embedding Dimensions: Matryoshka Representation Learning [21] supports multiple embedding dimensions from one forward pass by truncating and renormalizing the final hidden state.The method is applied during training to both contrastive and graded-relevance objectives at every supported dimension.
  • Two-Stage Training: Training uses two stages: large-scale multi-task alignment, followed by curated refinement combining contrastive learning, hard negatives, selective reranker supervision, and larger-model distillation.Both stages use the unified pair-based representation, with each batch applying the objective supported by its supervision signals.
  • Stage One: Multimodal Alignment: Stage-one alignment uses InfoNCE for paired data, graded-relevance CoSENT-style learning for annotated pairs, duplicate-aware masking, and relevance-gap weighting.Higher relevance pairs are encouraged to receive higher similarities, while larger label gaps receive greater weight.
  • Stage Two: Curated Refinement: Reranker supervision is restricted to task settings with reliable gains, while teacher distributions transfer softer relative similarity structure and particularly benefit compact variants.Reranking does not consistently improve performance across multimodal tasks, whereas distillation aligns source-to-target and target-to-source similarity distributions.

4 Experiments and Analysis

WeMM-Embedding is evaluated on public and in-house benchmarks spanning image, video, visual-document, text, and agent-oriented retrieval. The evaluation compares it with representative state-of-the-art multimodal embedding models, including open-source and proprietary commercial systems.

  • Evaluation spans image, video, visual-document, text, and agent-oriented retrieval across public and in-house benchmarks.The benchmark suite includes the MMEB series, cross-modal retrieval benchmarks from the Gemini Embedding 2 study [35], and a 26-task in-house benchmark derived from real-world applications.
  • The study compares WeMM-Embedding with representative state-of-the-art multimodal embedding models from both open-source and proprietary commercial settings.
  • The in-house evaluation comprises 26 tasks derived from real-world applications.

4.1 Performance on the MMEB Series

WeMM-Embedding achieves leading performance across MMEB-v2 and MMEB-v3, with gains increasing from 2B to 9B variants despite unsupported audio tasks receiving zero scores in V3-All.

  • MMEB-v2: 78 datasets in MMEB-v2 evaluate images, videos, and visual documents across classification, question answering, retrieval, and visual grounding.The benchmark follows Hit@1 for image, video, audio, and agent tasks, and NDCG@5 for text retrieval.
  • MMEB-v2: 79.2: WeMM-Embedding-4B achieves this MMEB-v2 overall score, while the 2B variant scores 77.9 and surpasses Qwen3-VL-Embedding-8B.The 2B model exceeds Qwen3-VL-Embedding-2B and DME-2B by 4.7 and 3.1 points, respectively; the 4B model outperforms all compared 8B–9B baselines.
  • MMEB-v3: 53 text tasks and 47 agent tasks extend MMEB-v3 beyond prior modalities, while unsupported audio tasks receive zero scores in V3-All.MMEB-v3 adds complex text retrieval, agent-centric tasks, audio evaluation, and MCMR image retrieval; current models do not support audio.
  • MMEB-v3: 59.5: WeMM-Embedding-9B reaches this MMEB-v3 V3-All score, while the 2B and 4B variants score 56.0 and 58.2, respectively, widening gaps over existing models.The 2B model already outperforms all compared baselines on V3-All and leads the Text and Agent groups with scores of 45.3 and 45.1.

4.2 Cross-Modal Retrieval Evaluation

WeMM-Embedding-2B achieves 79.8 across 12 cross-modal retrieval benchmarks, outperforming all compared open-source baselines; scaling to 4B and 9B raises scores to 80.8 and 81.7.

  • Evaluation scope: The evaluation covers 12 public benchmarks measuring semantic alignment between text and visual content across images, videos, and visual documents.
  • Results: 79.8 overall, WeMM-Embedding-2B outperforms all compared open-source baselines and compares favorably with leading proprietary models; 4B and 9B reach 80.8 and 81.7.These results average performance across 12 public cross-modal retrieval benchmarks.

4.3 In-House Evaluation

WeMM-Embedding-2B substantially outperforms a representative open-source baseline across all five categories of a 26-task WeChat benchmark. Production deployments further show consistent gains across recommendation and search systems, including 14 online A/B tests.

  • 4.3 In-House Evaluation: The benchmark comprises 26 tasks derived from real-world WeChat applications across classification, search, cross-domain content matching, article relevance, and video relevance.
  • 4.3 In-House Evaluation: WeMM-Embedding-2B achieves higher scores than a representative open-source baseline across all five categories in the 26-task in-house benchmark.The benchmark evaluates classification, search, cross-domain content matching, article relevance, and video relevance; AVG denotes the average across all 26 tasks.
  • 4.3 In-House Evaluation: WeMM-Embedding delivers consistent gains in 14 online A/B tests across large-scale recommendation systems spanning WeChat Official Accounts, WeChat Channels, and e-commerce content.Its representations combine text, cover-image, and video-frame signals across retrieval, ranking features, user-sequence modeling, and cross-domain content understanding; semantic IDs support compact indexing and sequence-modeling features.
  • 4.3 In-House Evaluation: In WeChat search, the model supports unimodal and cross-modal retrieval across Channels videos, Official Accounts articles, and Moments, improving semantic relevance and retrieval quality.Together with the in-house results, these deployments extend its performance to diverse real-world tasks and large-scale recommendation and search systems.

4.4 Further Analysis

Further analysis shows that 256-dimensional Matryoshka representations retain over 97% of full-dimensional performance across task groups, while Stage-1 and Stage-2 design choices materially affect MMEB-v2 performance. Cumulatively, Stage-2 strategies improve the 2B model’s overall score by 2.2 points.

  • Matryoshka representation analysis: The 2B model is evaluated from 64 to 2,048 dimensions on MMEB-v2 modality and task subsets, with retention measured relative to 2,048 dimensions.The analysis uses Matryoshka representations, which produce nested representations at multiple embedding dimensions.
  • Matryoshka representation analysis: At 256 dimensions, image and video tasks each retain 98.7% of 2,048-dimensional performance, while all classification, question-answering, and retrieval groups retain more than 97%.At 512 dimensions, retention reaches 99.2% for image tasks and 98.8% for video tasks; visual-document tasks are more sensitive to dimensionality reduction, and retrieval is most affected among task types at very low dimensions.
  • Stage-1 design ablations: Replacing task-consistent batching with mixed sampling decreases overall performance by 3.4 points, while removing task-specific instructions causes a 0.8-point decrease.The instruction ablation has its largest degradation on visual-document tasks, indicating that explicit instructions help capture task-specific matching objectives across heterogeneous tasks.
  • Stage-2 cumulative analysis: Cumulative Stage-2 strategies improve the 2B model’s MMEB-v2 overall score by 2.2 points.Curated data improves image and video performance, reranker supervision benefits visual-document tasks, dense similarity supervision transfers across heterogeneous tasks, and expanded visual input budgets particularly improve video tasks.

5 Conclusion and Future Work

WeMM-Embedding is a multi-scale family of universal multimodal embedding models trained through broad alignment followed by fine-grained representation learning. Future work will extend input modalities and model scale while improving data curation and relevance supervision.

  • WeMM-Embedding introduces a universal multimodal embedding family spanning multiple model scales.
  • Its two-stage training progresses from broad multimodal alignment to finer-grained representation learning using curated data, richer relevance supervision, and cross-scale knowledge transfer.
  • Future work will target omni-modal inputs, larger model sizes, and improved data curation and fine-grained relevance supervision.
Loading 2608.24053v1…