PaperlayerView exact revision on arXiv

Source-linked AI summary

Douyin Multimodal Embedding Model Technical Report

Haonan Chen, Chu Li, Zhicheng Wang, Yuanwei Liu, Yuanjiang Wang, Shaohua Jiang, Zhicheng Dou

arXiv:2608.02148v1cs.IRcs.CLcs.CV

TL;DR

Industrial multimodal retrieval needs efficient billion-scale encoding without sacrificing fine-grained discrimination. DME combines contrastive pre-training with training-only semantic-sufficiency mechanisms, achieving state-of-the-art comparable-scale performance across heterogeneous MMEB-v2 tasks, including an overall score of 74.8 for DME-2B.

  • Problem

    Industrial multimodal retrieval requires efficient dense encoding and fine-grained counterpart discrimination, while explicit reasoning or reranking is difficult to deploy at billion-scale.

  • Method

    DME uses contrastive pre-training followed by Evidence-Grounded Typed Latent Reasoning and Cross-Conditional Reconstruction to improve semantic sufficiency during training.

  • Results

    DME achieves state-of-the-art comparable-scale performance across heterogeneous MMEB-v2 task groups, with DME-2B reaching an overall score of 74.8 and especially strong video and visual-document retrieval.

  • Takeaways & Limitations

    DME remains a dense bi-encoder with marginal query-side overhead and is deployed across Douyin retrieval scenarios, delivering a 2.92% relative offline improvement.

  • Takeaways & Limitations

    Explicit textual reasoning or reranking-style computation remains difficult to deploy as the main retrieval encoder in billion-scale online systems.

Abstract

from arXiv · show

Multimodal representation learning is a cornerstone of modern AI. By encoding multimodal queries and targets into vectors, it powers industrial search and recommendation and underpins modern agents. Real-world platforms with complex modalities and massive-scale content, such as Douyin, Xiaohongshu, and YouTube, demand both efficiency under billion-scale indexing and fine-grained discrimination for hard matching. Existing MLLM embedding models rarely satisfy both. Contrastive models are efficient but rely on pair-level supervision too coarse for fine-grained distinctions, while CoT-based models improve discrimination through explicit generation impractical to serve online. We present Douyin Multimodal Embedding (DME), a model trained in two stages to combine both strengths. Stage 1 performs large-scale contrastive pre-training that establishes a unified multimodal embedding space with broad modality and task coverage. Stage 2 supplements semantic sufficiency, the property that an embedding is grounded in retrieval-relevant evidence and preserves fine-grained counterpart-side semantics, via two mechanisms. Evidence-Grounded Typed Latent Reasoning organizes retrieval evidence through hidden-space latent reasoning, and Cross-Conditional Reconstruction enforces counterpart-side semantics through cross-directional autoregressive reconstruction. Both act only during training and add only marginal query-side overhead, so DME serves as efficiently as a standard contrastive encoder. On MMEB-v2, DME reaches state-of-the-art results at comparable scales for its 2B and 9B variants (74.8 and 78.4), with especially strong video and visual-document tasks. In production, DME delivers a 2.92% relative gain on Douyin's in-house offline evaluation set, is deployed across Douyin scenarios such as generative, image, and AI search, and yields a 0.1% Lifetime (LT) gain in online A/B testing on Douyin search.

1 Introduction

DME addresses the industrial tension between efficient multimodal retrieval and fine-grained discrimination by combining large-scale contrastive pre-training with training-only semantic sufficiency mechanisms. Its latent reasoning and cross-conditional reconstruction improve evidence grounding and counterpart-side semantic preservation while retaining efficient bi-encoder serving.

  • Motivation: Industrial multimodal retrieval must support heterogeneous modalities at massive scale while preserving fine-grained distinctions for hard query–document matches.Queries and contents may combine text, images, videos, visual documents, and multiple constraints, while hard negatives can differ in only localized or subtle semantics.
  • Motivation: Contrastive MLLM embedders scale efficiently but provide coarse representations, whereas explicit reasoning improves discrimination at deployment costs that hinder billion-scale online retrieval.Contrastive learning supports offline corpus encoding and vector search, while explicit textual reasoning or reranking adds substantial computation.
  • DME: DME uses two stages: heterogeneous multimodal contrastive pre-training establishes a unified embedding space, and semantic sufficiency learning adds fine-grained modeling.The model is designed to combine contrastive MLLM efficiency with reasoning-augmented retrievers’ fine-grained semantic capability.
  • Stage 2 mechanisms: Evidence-Grounded Typed Latent Reasoning organizes retrieval-relevant evidence with anchor-based latent tokens before final embedding readout, avoiding explicit long-form CoT generation.Anchors localize evidence such as text spans, image regions, OCR fragments, and video keyframes.
  • Stage 2 mechanisms: Cross-Conditional Reconstruction combines NTP and MTP to preserve counterpart-side semantics and enable token-level recovery from embeddings without additional retrieval-time decoding.The resulting representation completeness measure quantifies semantic sufficiency and guides representation optimization.
  • Results: DME-2B and DME-9B achieve state-of-the-art MMEB-v2 results among comparable-sized models, with especially strong video and visual-document retrieval and minor query-encoding latency overhead.The report also evaluates information completeness through token-level recovery analysis.

2 Related Work

Prior work established efficient multimodal embedding through large-scale contrastive alignment and increasingly unified MLLM-based embedders. DME retains the efficient bi-encoder paradigm while adding evidence-grounded and generative supervision to improve semantic sufficiency.

  • Contrastive and unified multimodal embedders: CLIP and ALIGN showed that large-scale paired supervision can induce strong cross-modal alignment, motivating later unified multimodal embedders built on stronger MLLM backbones.Examples include E5-V, GME, mmE5, VLM2Vec-V2, U-MARVEL, PDF-VLM2Vec, and Qwen3-VL-Embedding.
  • DME’s training paradigm: DME follows the efficient bi-encoder paradigm but supplements pair-level training with evidence-grounded and generative supervision to improve semantic sufficiency.This positioning combines the efficiency of contrastive embedding with additional supervision for retrieval-relevant semantics.
  • Reasoning-enhanced multimodal models: Chain-of-thought prompting and reinforcement learning strengthen multimodal reasoning through explicit rationales, structured solving, and reasoning-oriented reward optimization.These advances motivate transferring reasoning ability into representations used for matching rather than retaining it only in free-form generation.
  • Reconstruction and generative objectives: Reconstruction and generative objectives can strengthen representations by encouraging MLLMs to compress richer semantic information into embedding tokens and capture longer-range semantics.Prior examples span masked language and image modeling, contrastive-captioning, content reconstruction, joint generative-retrieval training, and multi-token prediction.

3 DME Overview

DME is an instruction-aware universal multimodal retriever that represents heterogeneous queries and documents as normalized dense vectors for efficient bi-encoder retrieval. It uses large-scale contrastive pre-training to establish broad modality alignment, then adds semantic sufficiency learning through evidence-grounded latent reasoning and cross-conditional reconstruction.

  • Retrieval setting: DME supports text-only, vision-only, video-based, and multimodal queries and documents across text, image, video, visual-document, and mixed-modality retrieval tasks.Natural-language instructions define the relevance criterion for each retrieval task.
  • Bi-encoder architecture: DME independently encodes each query and document with shared backbone and projection components, then ranks candidates by inner product between ℓ2-normalized embeddings.Side-specific retrieval-token layouts and readout functions are allowed despite the shared components.
  • Inference: At inference, DME precomputes document embeddings in an approximate nearest-neighbor index and performs one query-side encoding pass, avoiding query–candidate pairwise cross-attention.This separates training-time supervision from inference-time computation and supports large-scale deployment.
  • Two-stage training: Stage 1 trains the MLLM backbone on 25M multimodal query-target pairs with contrastive loss to establish a scalable bi-encoder embedding space.The stage uses a standard embedding token and supports heterogeneous text, image, video, visual-document, and mixed-modality pairs.
  • Two-stage training: Stage 2 refines the model on 5M higher-quality examples with teacher-generated CoT supervision, jointly optimizing contrastive and semantic sufficiency losses.Its mechanisms include Evidence-Grounded Typed Latent Reasoning and Cross-Conditional Reconstruction while retaining the normalized embedding interface.

4 Multi-Stage Learning Framework

DME uses two training stages: large-scale contrastive pre-training builds a broad multimodal embedding space, while semantic-sufficiency objectives add evidence grounding, typed latent reasoning, and counterpart-side reconstruction. These mechanisms operate during training, preserving standard bi-encoder efficiency at inference.

  • Stage 2: Stage 2 refines retrieval representations through Evidence-Grounded Typed Latent Reasoning and Cross-Conditional Reconstruction.The former grounds representations in selected multimodal evidence and typed latent states; the latter applies NTP/MTP to enforce fine-grained counterpart-side semantics.
  • Stage 1: Stage 1 uses large-scale heterogeneous contrastive query–document pairs to align text, image, and video in a unified semantic space.It trains a standard bi-encoder with the <emb> readout token, without anchor tokens, typed latent states, or reconstruction supervision.
  • Stage 2: Stage 2 contrastive data combines MMEB-v2 with diverse multimodal retrieval and question-answering datasets across image–text, video–text, visual–document, and mixed-modality tasks.Hard negatives include annotated distractors, retrieval-mined candidates, and semantically similar examples when available.
  • Stage 2-A: Stage 2-A uses teacher-generated structured supervision containing modality-specific evidence, local summaries, and typed retrieval trajectories for query, positive, and optional negative items.Evidence can reference text spans, image regions, or video frames; negative-side evidence supports rejection-oriented trajectory states rather than anchor targets.
  • Inference: Both NTP and MTP are training-only objectives, so inference remains single-pass dense-vector encoding and vector-similarity retrieval without decoding or autoregressive generation.The decoding pass and D_MTP modules are discarded after back-propagating token-level gradients into the shared backbone and readout.

5 Experiments and Analysis

DME achieves strong multimodal embedding performance across model scales and heterogeneous modalities, while improving Douyin industrial retrieval offline and online. Ablation, completeness, and efficiency analyses show that its training recipe strengthens the embedding space, preserves token-level information, and adds minimal query-encoding overhead.

  • MMEB-v2 results: 74.8 and 78.4 are DME-2B and DME-9B’s overall MMEB-v2 scores, respectively, demonstrating strong performance at both comparable model scales.DME-2B is compared with 2B-level models, while DME-9B is compared with larger models.
  • MMEB-v2 results: 75.9/79.8 on Image, 65.6/70.8 on Video, and 79.9/82.0 on VisDoc show consistent performance across modality groups for DME-2B and DME-9B.The benchmark spans heterogeneous task families including classification, question answering, retrieval, grounding, moment retrieval, visual-document retrieval, VisRAG, and out-of-distribution evaluation.
  • MMEB-v2 results: DME outperforms reasoning-enhanced baselines while using a small number of retrieval-specific latent tokens instead of explicit chain-of-thought generation or cross-encoder reranking.The latent reasoning operates within the encoder and uses lightweight latent-token inference.
  • Industrial evaluation: 2.92% relative improvement in overall retrieval quality is achieved over the previous production model, with gains ranging from +3.10% on Text2Video to +2.70% on Image2Image.The gains are consistent across all four evaluated directions.
  • Industrial evaluation: 0.1% Lifetime (LT) gain in core online business metrics is verified by Douyin search A/B testing after deploying the DME-based model.The model powers generative search and serves as a retrieval feature for ranking.
  • Ablation analysis: 3.9 points raise the overall score from 70.9 to 74.8, while Stage 1 alone raises it from 70.9 to 72.5 and particularly improves video and visual-document retrieval.The Stage 1 gains are 55.3 to 59.3 on video and 77.1 to 79.0 on visual-document retrieval, while image retrieval changes from 74.6 to 74.8.
  • Representation completeness: 87.9% q2q and 74.3% d2d Top-1 recovery on the merged set, rising above 92% at Top-10, shows that DME embeddings retain substantial token-level input information.The completeness analysis indicates that a single pooled vector preserves most of its own input’s token-level content.
  • Efficiency analysis: Below 1 ms per query of additional cost for text and video, with nearly negligible overhead for image–text queries, keeps latent reasoning close to standard dense retrieval encoding.The measurement covers only the query-encoder forward pass, excluding candidate encoding, scoring, retrieval, data loading, collation, and transfer.

6 Conclusion

DME is presented as a two-stage representation model for large-scale industrial multimodal retrieval, combining contrastive pre-training with Stage 2 mechanisms for semantic sufficiency. The report also outlines data and model-size scaling directions, with preliminary attempts yielding consistent gains.

  • 6 Conclusion: DME targets large-scale industrial multimodal retrieval through a two-stage training framework.Stage 1 performs large-scale contrastive pre-training to establish a unified multimodal embedding space.
  • 6 Conclusion: Stage 2 supplements semantic sufficiency through Evidence-Grounded Typed Latent Reasoning and Cross-Conditional Reconstruction.The former grounds embeddings in retrieval-relevant evidence; the passage truncates the description of the latter mechanism.
  • 6 Conclusion: DME is being extended through data scaling and model-size scaling to strengthen representation generality.Data scaling broadens multimodal training coverage, while larger backbones increase representational capacity; preliminary attempts along both directions yield consistent gains.

7 Contributor List

This section lists the paper’s contributors, leadership, supervision, affiliations, contribution note, and internship disclosure.

  • Contributors: Contributors are Haonan Chen, Chu Li, Zhicheng Wang, Yuanwei Liu, and Yuanjiang Wang.Haonan Chen, Chu Li, and Zhicheng Wang are marked as equal contributors.
  • Leadership and affiliations: Shaohua Jiang is the project leader, and Zhicheng Dou is the supervisor.Their affiliations span ByteDance Douyin Search Multimodal Team and Gaoling School of Artificial Intelligence, Renmin University of China.
  • Notes: Haonan Chen’s work was completed during an internship at ByteDance Douyin Search Multimodal Team.Contributors are listed alphabetically by last-name initial.

Appendix · A Detailed Formulation of Stage 2-A

The appendix gives an implementation-level formulation of Stage 2-A, covering evidence assignment, assignment balancing, summary-conditioned supervision, and typed latent objectives. It specifies how anchors, evidence pools, teacher-generated summaries, and typed trajectory states contribute to the complete Stage-2A objective.

  • A Detailed Formulation of Stage 2-A: The appendix formalizes evidence assignment, assignment balancing, summary-conditioned supervision, and typed latent objectives for Stage 2-A.These components are presented as the implementation-level formulation of the mechanisms summarized in Section 4.3.
  • A.1 Anchor Evidence Assignment: For each anchor and modality, DME aligns serialized token-level anchor distributions with evidence targets using matching costs and softmin assignment.Detached assignment weights are used in the evidence-level hit loss, allowing anchors to specialize without hard assignment.
  • A.1 Anchor Evidence Assignment: DME prevents evidence collapse by measuring average assignment mass and adding a balance regularizer to the complete anchor hit loss.Negative-side structured evidence constructs rejection-oriented trajectory states but is not an anchor hit target.
  • A.2 Summary-Conditioned Evidence Supervision: Anchor distributions produce anchor-specific evidence representations, which are aggregated across modalities into a side-level evidence pool.The pool is then supervised using the cached embedding of a teacher-generated local summary.
  • A.2 Summary-Conditioned Evidence Supervision: Summary-conditioned supervision requires the evidence pool to cover teacher-annotated positions while preserving their intended local semantics.This extends evidence supervision beyond positional coverage to semantic preservation.
  • A.3 Typed Latent Objectives: Typed latent objectives project trajectory hidden states into retrieval space and apply type-specific supervision to semantic, positive-alignment, and rejection states.Semantic states align with cached teacher-state embeddings, align_pos states retrieve corresponding positive documents, and reject_neg states use a margin-ranking objective.
  • A.3 Typed Latent Objectives: The complete Stage-2A objective combines the evidence terms with the typed latent objective.Trajectory contrastive learning uses temperature 𝜏traj, while rejection supervision uses margin 𝜇.

B Visualization of Cross-Conditional Reconstruction

Cross-conditional reconstruction preserves language-decodable generative content in both query- and document-side embeddings, including embeddings from visual and video inputs. The decoded text forms a compact, abstractive representation that emphasizes retrieval-relevant shared semantics rather than verbatim detail.

  • Reconstruction setup: Each embedding is decoded from only its pre-normalization vector, without the original text or visual tokens, using greedy decoding and one output sequence.The embedding is fed back as the sole prefix token into the shared backbone.
  • Qualitative observations: Both query- and document-side embeddings decode into coherent, on-topic text, including embeddings originating from purely visual or video inputs.Symmetric Q→D and D→Q reconstruction preserves decodable generative representations in both directions.
  • Qualitative observations: Decoded text is much shorter than the raw input, preserving salient retrieval-relevant gist while discarding surface detail as an abstractive semantic bottleneck.Examples include “Hamster eating food” for a long instruction-conditioned query and a few salient words for a full document image.
  • Qualitative observations: Query and target decodings tend to share overlapping core content, causing common relevance-bearing concepts to dominate both representations.Examples include “Hamster eating food” versus “Hamster” and “Saxophone player in a music store” versus “Saxophone player”.
  • Limitations: Greedy, lossy decoding may omit fine-grained tokens, because reconstruction enforces semantic fidelity of counterpart-side content rather than lossless textual reconstruction.For example, “October 17, 1995” may decode only as “October”.
Loading 2608.02148v1…