SAIL-Embedding Technical Report: Omni-modal Embedding Foundation Model

1 Introduction

• Existing models face limited modality support, training instability, and an industrial domain gap that matters for real-world recommendation and search.Douyin videos combine visual, textual, audio, and speech-transcription signals, and omitting or misinterpreting a modality can affect downstream tasks and user experience.Source: §1 · p. 2
• SAIL-Embedding handles arbitrary combinations of vision, text, and audio inputs, producing multidimensional representations for varied business requirements.It is adapted from vision-language-audio models and supports classification, query-to-item retrieval, and item-to-item retrieval.Source: Figure 1 · p. 2, §1 · p. 3
• Dynamic hard-negative mining and adaptive multi-source data balancing are introduced to improve training robustness and scalability across diverse datasets.Hard-negative mining focuses on challenging negatives; data balancing learns weights from distributions to reduce manual tuning while trading off data quality and diversity.Source: §1 · p. 3
• The multi-stage training combines content-aware progression with collaboration-aware recommendation enhancement to learn unified representations.Sequence-to-item distillation incorporates users’ historical behavior, and ID-to-item distillation aggregates user-specific preference signals for item recommendations.Source: §1 · p. 3
• SAIL-Embedding achieves SOTA performance against advanced baselines across diverse downstream tasks, while Douyin experiments report gains through several application pathways.For cold-start recommendation, embedding-based item-to-item recall and embeddings in the recommendation model yield a 0.05% total LT gain.Source: §1 · p. 3

2 Related Works

• Multimodal embeddings map heterogeneous inputs into a shared space for tasks including cross-modal retrieval, video understanding, and recommendation.Existing approaches are broadly grouped into dual-tower encoders and MLLM-based unified Transformer models.Source: §2 · pp. 3–4

2.1 Dual-Tower Multimodal Embedding Models

• Dual-tower models use separate modality encoders and contrastive learning, enabling cached, efficient retrieval at scale but limiting fine-grained interactions and multimodal reasoning.Extensions include audio and video branches, while BLIP and BLIP-2 bridge dual-tower and fusion-style approaches.Source: §2.1 · p. 4

2.2 MLLM-based Multimodal Embedding Models

• MLLM-based models jointly model modalities through a unified sequence framework, enabling deeper semantic understanding and contextual reasoning, but most remain limited to image and text.Examples include instruction-conditioned image-text embedding, video and visual-document support, and unified cross-modal search.Source: §2.2 · p. 4

3 Methodology

• The methodology covers dataset construction and balancing, hard-negative mining, multimodal architecture and prompting, and training objectives with techniques to improve effectiveness.The section introduces these components in sequence, beginning with datasets and ending with training techniques.Source: §3 · p. 4

3.1.1 Recommendation-aware Data Construction

• The training corpus contains over 10B samples spanning multimodal retrieval and multi-level label classification tasks.Queries and targets support item-to-item retrieval, query-to-item retrieval, and classification.Source: §3.1.1 · p. 5, Table 1 · p. 5

3.1.2 Dynamic Hard Negative Mining

• A dataset- and task-specific similarity threshold selects challenging negatives below the cutoff while filtering overly similar false negatives.The training loss uses positives, mined hard negatives, and in-batch random negatives.Source: §3.1.2 · pp. 5–6

3.1.3 Adaptive Multi-Source Data Balancing

• Adaptive balancing derives training-set weights from semantic similarity to benchmark validation sets rather than manually assigned mixing ratios.The method clusters embeddings, scores training–validation similarity, and uses those scores to determine sampling weights.Source: §3.1.3 · pp. 6–7, Figure 3 · p. 7
• Soft dataset-level weighting preserves distributional diversity while mitigating benchmark overfitting and improving robustness.The framework is presented as a way to balance data quality and diversity without hard-filtering samples.Source: §3.1.3 · pp. 6–7

3.2.1 Overall Architecture

• The architecture projects audio and visual features into a language-compatible space, then uses an LLM backbone to fuse them with text into a unified embedding.The final representation is mean-pooled from the LLM output.Source: §3.2.1 · p. 7

3.2.2 Text Tokenizer

• Text fields are cleaned, deduplicated, and concatenated, with partial fields randomly dropped to accommodate missing online inputs.Prompts specify each task and its objective, and query and target instructions differ.Source: §3.2.2 · p. 8

3.2.3 Vision Encoding Module

• A Visual Perceiver condenses video-frame tokens into 16 learnable query-token embeddings to reduce sequence length while preserving essential semantic content.Frames are patchified using patch size 14 before token reduction.Source: §3.2.3 · pp. 8–9

3.2.4 Audio Encoding Module

• CLAP encodes short audio with repeat-and-pad and long audio as non-overlapping 10-second chunks that are mean-pooled into one token.The resulting token is appended to the multimodal sequence regardless of clip length.Source: §3.2.4 · p. 9

3.2.5 Fusion Module

• The fusion module concatenates audio, visual, and text tokens, applies bidirectional self-attention, then uses tanh and mean pooling to form the embedding.The fused sequence is processed by an LLM backbone.Source: §3.2.5 · pp. 9–10

3.3.1 Content-Aware Progressive Training

• Progressive training moves from large, diverse datasets to task-aligned data and hard-negative fine-tuning, balancing general knowledge with downstream specificity while maintaining stability.The stages build fundamental multimodal capability, then refine task alignment and fine-grained distinctions.Source: §3.3.1 · p. 10

3.3.2 Loss Definition

• The objective combines NCE, COSENT, mICL, and late-fusion losses; nested 128-, 768-, and 1536-dimensional embeddings support inference-time accuracy–efficiency tradeoffs.The nested slices are each optimized with the same InfoNCE objective, preserving discriminative power without retraining for different inference dimensions.Source: §3.3.2 · pp. 10–11

3.3.3 Stochastic Specialization Training

• Stochastic specialization samples one dataset per iteration while retaining the global batch size, increasing per-domain batch size and reducing gradient variance and cross-domain processing.All datasets are visited across training; adding a dataset requires only its configuration, without changing batching or communication patterns.Source: §3.3.3 · p. 11

3.3.4 Dataset-Driven Pattern Matching

• Dataset-driven matching constructs valid query–target pairs for available modalities, generalizing contrastive objectives beyond image-to-text and excluding modalities without valid patterns.The method evaluates feasible pairs in the same forward pass and is reported to improve optimization stability through smoother training-loss convergence.Source: §3.3.4 · p. 12

3.3.5 Collaboration-aware Recommendation Enhancement Training

• Recommendation enhancement distills sequence-to-item and recommender ID-to-item representations, combining video content with user-interest signals for recommendation scenarios.Sequence modeling uses mean pooling or a three-layer transformer; ID distillation aligns multiple item-ID embeddings and adds an auxiliary item-to-item retrieval task.Source: §3.3.5 · pp. 12–13

4.1 Configuration Settings

• Training uses multiple NPUs, DeepSpeed ZeRO2, gradient checkpointing, and FusedAdam with mixed precision and gradient fusion.The visual perceiver uses latent dimension 1024, 16 layers, and sampling depth 6; CLAP audio features are mapped from 512 to a 1536-dimensional alignment state.Source: §4.1 · p. 13
• Sequence-to-item distillation defaults to mean pooling with sequence length 10, using 11M sequence samples; ID-to-item training uses 220M main-task and 20M auxiliary-task samples.The learning rate starts at 1e-5, weight decay is 1e-4, and the cosine schedule varies between 1e-5 and 6e-6.Source: §4.1 · p. 13
• Item-to-item comparisons include CLIP-based and standard VLM-based models; query-to-item comparisons additionally include Doubao-Embedding and Qwen3-Embedding.The CLIP-based and VLM-based models are fine-tuned on the Table 1 training data for fair comparison.Source: §4.1 · p. 13

4.2 Evaluation Metrics

• The framework assesses structural similarity, stable ranking, and discriminability, and is intended to provide quantitative quality measures and diagnostics for recommendation integration.The authors motivate it because standalone embedding metrics may not reflect downstream engagement, while trial-and-error deployment risks suboptimal online performance.Source: §4.2 · pp. 13, 15
• Recall@k measures retrieval on item-to-item and query-to-item benchmarks for k ∈ {1, 10, 25, 50, 100}.Positive–negative separability is measured by comparing cosine similarities to a ground-truth target and randomly sampled negatives.Source: §4.2 · p. 14
• Normalized Mutual Information measures consistency between positive- and negative-sample clusters; higher NMI indicates separation by similar feature dimensions.The described interpretation connects cluster-assignment consistency with whether multimodal features capture decision bases underlying user behavior.Source: §4.2 · pp. 14–15
• Ranking consistency uses top-k overlap and Kendall’s τ, while bijective alignment checks whether retrieving back from the nearest target returns the original query.The proportion of matching query indices in the round trip quantifies embedding fidelity and bijective consistency.Source: §4.2 · p. 15
• AUC is additionally reported for recommendation-oriented tasks.The paper describes it as an additional measurement alongside the embedding evaluation suite.Source: §4.2 · p. 15

4.3 Evaluation Dataset Introduction

• Content-understanding item-to-item benchmarks group items by shared brands, intellectual property, keywords, or summaries, including out-of-domain benchmarks for generalizability.Named sets include Brand Vehicle-i2i, Brand Phone-i2i, Film-i2i, Game Tag-i2i, Keywords-i2i, and Summary-i2i.Source: §4.3 · p. 15
• Search-based item-to-item benchmarks pair items clicked under the same query; video search-i2i contains video items, mostly delivered on Douyin.Search-i2i includes different item forms, such as videos and photos.Source: §4.3 · p. 15
• Collaborative-perception item-to-item datasets use co-occurrence pairs for video and live-streaming items, plus positive pairs selected using recommender-feature similarity scores.The recommender-system dense features are compared using cosine similarity, with pairs above a threshold retained as positives.Source: §4.3 · p. 15
• Query-to-item retrieval includes an in-domain Short Video-q2i set and two out-of-domain live-streaming benchmarks based on user search text or item summaries.The out-of-domain datasets are Live-q2i and Live-summary2i.Source: §4.3 · p. 16
• Query-to-item classification uses six downstream datasets, treating embedding cosine similarities as binary pairing probabilities and reporting AUC.All items in these classification datasets contain text features only.Source: §4.3 · p. 16

4.4.1 Item-to-Item Retrieval Task Comparison Results

• SAIL-Embedding significantly and consistently outperforms prior models in i2i search and collaborative-perception tasks, with competitive content-understanding results.The comparison spans 21 tasks across four application intents.Source: §4.4.1 · p. 16

4.4.2 Query-to-Item Retrieval Task Comparison Results

• SAIL-Embedding outperforms previous methods across every metric on all nine q2i tasks, spanning retrieval- and classification-oriented evaluations.The paper attributes this result partly to its omni-modal architecture and training strategies.Source: §4.4.2 · p. 16

4.4.3 Collaboration-aware Recommendation Enhancement Results

• CRE improves multiple recommendation metrics: ID distillation yields +5% NMI, +3.8% Kendall correlation, +1.9% Intersection, and +0.66% bijective alignment.The reported metrics cover clustering separation, ranking consistency, and bijective alignment.Source: §4.4.3 · p. 17, Table 4 · p. 17
• CRE improves Gid-i2i Recall@50 by 7.23% and Recall@100 by 7.31%, while the paper reports some degradation on Copair-i2i.The authors describe the variation as tolerable given differing content-oriented and collaborative-behavior requirements.Source: §4.4.3 · p. 17

4.4.4 Sequence Modeling Evaluation Results

• Sequence and ID distillation improve sequence-retrieval results: seq2item raises Filtered Seq2item Recall@10 by 2.67% and Recall@25 by 4.66%, with a final 4.76% improvement on Vanilla Seq2item.The evaluation uses historical viewing sequences as queries to target videos.Source: §4.4.4 · pp. 17–18, Table 5 · p. 18

4.4.5 Systematic Ablation Studies

• Ablations report gains from LLM fusion, task-specific instructions, expanded business data, COSENT loss, and progressive training across i2i and q2i tasks.Figures 9 and 10 summarize average performance across tasks; task-specific instructions raise average i2i and q2i performance by 2.62% and 1.88%, respectively.Source: Figure 9 · p. 18, §4.4.5 · pp. 18–19, Figure 10 · p. 19

4.4.6 Extensive Online Experiments

• Online, shared features benefit nearly all recommender stages and both cold-start and message-pushing settings, across Douyin and Douyin-Jingxuan.The authors report that semantic IDs bring more gain than dense embeddings, while dense embeddings retain more information.Source: §4.4.6 · pp. 19–20, Table 6 · p. 19

• SAIL-Embedding is an omni-modal embedding foundation model tailored to large-scale recommendation scenarios.It unifies vision, text, and audio and combines content-aware progressive training with collaboration-aware recommendation enhancement.Source: §5 · p. 20
• Experiments across diverse i2i and q2i benchmarks report state-of-the-art performance and generalization to real-world industrial applications.The conclusion also says systematic ablations validate the proposed components.Source: §5 · p. 20
• Future work will further integrate VLMs with recommendation objectives and mine paired recommendation signals and behavioral feedback for representation learning.The authors also plan to investigate recommendation-specific hard-negative mining.Source: §5 · p. 20

• Lin Lin, Jiefeng Long, Zhihe Wan, Yuchi Wang, Dingkang Yang, Shuang Yang, and Yueyang Yao are listed as core contributors.Source: §6 · p. 20
• Xu Chen, Zirui Guo, Shengqiang Li, Weiran Li, Hanyu Li, Yaling Mou, Yan Qiu, and Haiyang Yu are listed as contributors.Source: §6 · p. 20
• Xiao Liang is the project leader; Xiao Liang, Hongsheng Li, and Chao Feng are listed as supervisors.Source: §6 · p. 20