PaperlayerView exact revision on arXiv

Source-linked AI summary

Diffusion-Pretrained Dense and Contextual Embeddings

Sedigheh Eslami, Maksim Gaiduk, Markus Krimmel, Louis Milliken, Bo Wang, Denis Bykov

arXiv:2602.11151v2cs.LGcs.CLcs.IR

TL;DR

Public benchmarks do not fully capture production web-scale retrieval challenges. The paper introduces pplx-embed, multilingual retrieval embeddings trained through multi-stage contrastive learning on diffusion-pretrained backbones, including a document-contextualized variant. The models achieve strong performance across public and internal web-scale benchmarks.

  • Problem

    Public benchmarks do not fully capture long-tail queries, noisy documents, and distribution shifts in production web-scale retrieval.

    Source: §3.2 · p. 10

  • Method

    pplx-embed trains multilingual retrieval embeddings through multi-stage contrastive learning on diffusion-pretrained backbones, with a contextual variant encoding passages relative to document-level context.

    Source: §1 · pp. 1–2

  • Results

    Across public and internal web-scale benchmarks, pplx-embed achieves strong retrieval performance, including 69.66% average nDCG@10 on MTEB(Multilingual, v2) with INT8 pplx-embed-v1-4B.

    Source: §1 · p. 2, §6 · p. 14

  • Takeaways & Limitations

    Native quantization-aware training offers practical deployment benefits by directly producing INT8 and binary embeddings.

    Source: §6 · p. 14

Abstract

from arXiv · show

In this report, we introduce pplx-embed, a family of multilingual embedding models that employ multi-stage contrastive learning on a diffusion-pretrained language model backbone for web-scale retrieval. By leveraging bidirectional attention through diffusion-based pretraining, our models capture comprehensive bidirectional context within passages, enabling the use of mean pooling and a late chunking strategy to better preserve global context across long documents. We release two model types: pplx-embed-v1 for standard retrieval, and pplx-embed-context-v1 for contextualized embeddings that incorporate global document context into passage representations. pplx-embed-v1 achieves competitive performance on the MTEB(Multilingual, v2), MTEB(Code), MIRACL, BERGEN, and ToolRet retrieval benchmarks, while pplx-embed-context-v1 sets new records on the ConTEB benchmark. Beyond public benchmarks, pplx-embed-v1 demonstrates strong performance on our internal evaluation suite, focusing on real-world, large-scale search scenarios constructed from 1B production web pages. These results validate the models' effectiveness in production environments where retrieval quality and efficiency are critical at scale.

1. Introduction

pplx-embed uses diffusion-based bidirectional pretraining and multi-stage contrastive learning for multilingual standard and contextual retrieval. Its INT8 embeddings deliver competitive retrieval performance, with separate models for standard retrieval and document-contextualized passage embeddings.

1. Introduction

  • Diffusion continued pretraining converts a causal LLM into a bidirectional encoder, followed by multi-stage contrastive learning on large-scale question-document pairs and triplets.Contrastive training aligns embedding geometry with semantic similarity. Bidirectional attention supports more comprehensive global document context modeling than causal masking.Source: §1 · pp. 1–2
  • 0.6B- and 4B-parameter releases provide pplx-embed-v1 for standard retrieval and pplx-embed-context-v1 for document-contextualized passages, without instruction prefixes.Neither model is instruction-tuned. Native quantization-aware training produces INT8 embeddings by default, supporting competitive performance with improved efficiency.Source: §1 · p. 2
  • 69.66% average nDCG@10 on MTEB(Multilingual, v2) puts INT8 pplx-embed-v1-4B ahead of Qwen3-Embedding-4B (69.60%) and gemini-embedding-001 (67.71%).The model also performs competitively on MTEB(Code) and MIRACL.Source: §1 · p. 2
  • 81.96% average nDCG@10 on ConTEB (Conti et al., 2025) lets pplx-embed-context-v1-4B outperform voyage-context-32 (79.45%) and Anthropic Contextual3 (72.4%).This comparison evaluates contextual retrieval.Source: §1 · p. 2
  • 44.45% average nDCG@10 on ToolRet (Shi et al., 2025) places pplx-embed-v1-4B above larger 7B models.NV-Embed-v1 (Lee et al., 2025a) scores 42.71%, while GritLM-7B (Muennighoff et al., 2025) scores 41.13%. Retrieval enables pre-filtering relevant tools from large API corpora, reducing context usage.Source: §1 · p. 2
  • Three of five BERGEN (Rau et al., 2024) question-answering tasks favor pplx-embed-v1-0.6B over the larger Qwen3-Embedding-4B.Across BERGEN, pplx-embed-v1 consistently outperforms BGE-M3 (Chen et al., 2024) and Qwen3-Embedding. Internal web-scale first-stage retrieval benchmarks also show superior performance against those model families.Source: §1 · p. 2

2. PPLX Embedding

pplx-embed uses a branched curriculum combining diffusion pretraining, pair training, contextual training, and hard-negative triplet training, followed by merging and selection. Contextual training produces pplx-embed-context-v1, while merging contextual and triplet checkpoints produces pplx-embed-v1.

2. PPLX Embedding

  • pplx-embed-context-v1 comes from contextual training, while pplx-embed-v1 merges the contextual model and triplet checkpoints using Spherical Linear Interpolation.The curriculum combines four training paradigms in branches before merging and selection. Triplet training starts from checkpoints obtained after pair and contextual training.Source: §2 · pp. 2–3

2.1. Continued Diffusion Pretraining

  • Continued diffusion pretraining converts Qwen3-0.6B and 4B backbones (Yang et al., 2025) into bidirectional encoders by removing causal masking and learning masked-token reconstruction.Approximately 250 billion pretraining tokens are split equally between English educational pages and 29 other languages; masked-position cross-entropies are weighted by 1/t. Each sequence samples t uniformly from 0.001 to 1 and independently masks tokens with probability t; the autoregressive left shift is retained, while causal-mask annealing showed no substantial improvement.Source: §2.1 · p. 3

2.2. Pooling and Quantization

  • Mean pooling is combined with INT8 quantization during inference and contrastive training, while training-free post-hoc binarization incurs minimal performance loss.Straight-through gradient estimation (Bengio et al., 2013) handles rounding; integer embeddings have entries from −127 to 127 and are compared using cosine similarity. Binary quantization maps mean-pooled entries to −1 or 1.Source: §2.2 · p. 4

2.3. Pair Training

  • Pair training uses InfoNCE with in-batch document and query negatives, expanding from English-only to English-plus-cross-lingual data and finally the complete multilingual pair dataset.Negatives exceeding the positive pair’s cosine similarity by more than 0.1 are masked as likely false negatives, using a design inspired by Zhang et al. (2025b).Source: §2.3 · pp. 4–5

2.4. Contextual Training

  • Contextual training preserves whole-document information in chunk embeddings by jointly optimizing local chunk discrimination and global query-document alignment.Local losses contrast gold chunks against same-document and batch chunks; global InfoNCE masks hash-identified duplicate documents and similarity-threshold false negatives while including query-query negatives. A cosine schedule increases beta from 0.2 to 0.5, gradually increasing document-level learning to mitigate forgetting of coarse document semantics.Source: §2.4 · pp. 5–6

2.5. Triplet Training

  • Triplet training adds explicit hard negatives alongside positive documents to learn more discriminative embeddings through fine-grained relevance distinctions.Applied after pair and contextual training, it refines boundaries between similar but non-relevant documents.Source: §2 · p. 3, §2.5 · p. 6

2.6. Datasets for Contrastive Learning

  • 60 languages are represented in contrastive training data comprising 65.6% English, 6.7% cross-lingual, 1% code, and 26.7% multilingual samples.Contextual training uses ConTEB and synthesized MLDR data; triplet training uses less but higher-quality data from 12 datasets comprising 92% English, 1% code, and 7% multilingual text across 15 languages. Synthetic data use Qwen3-30B-A3B-Instruct-2507 with a two-stage, top-5-persona pipeline inspired by Zhang et al. (2025b), adapted to generate passage queries for contextual training.Source: §2.6 · p. 6

3. Evaluations

The models perform strongly across public retrieval benchmarks and web-scale internal evaluations, with pplx-embed-context-v1-4B leading ConTEB. Binary quantization causes smaller performance losses for 4B models than for 0.6B models.

3.1. Public Benchmarks

  • pplx-embed-v1-4B beats gemini-embedding-001 on MTEB Multilingual and Code, while pplx-embed-v1-0.6B beats its Qwen3 counterpart on both.On the 18 multilingual retrieval tasks, the 4B model closely rivals Qwen3-Embedding-4B while being substantially more storage-efficient. Across 12 code retrieval tasks covering 15 programming languages, it slightly trails Qwen3-Embedding-4B but also outperforms text-embedding-3-large.Source: Table 1 · p. 7, §3.1 · pp. 7–8
  • 18 MIRACL language subsets all favor pplx-embed-v1-0.6B over Qwen3-Embedding-0.6B, and its average score exceeds the pplx 4B model’s.Even the binarized 0.6B variant exceeds Qwen3-Embedding-0.6B on average. All pplx-embed-v1 models outperform text-embedding-3-large on average but trail Qwen3-Embedding-4B and gemini-embedding-001.Source: Table 2 · p. 7, §3.1 · p. 7
  • pplx-embed-context-v1-4B leads ConTEB, while the 0.6B model ranks third, above ModernBERT-Large and Anthropic Contextual but below voyage-context-3.Evaluation measures nDCG@10 across eight datasets using late chunking (Günther et al., 2024), quantization of pooled chunk embeddings, and cosine-similarity ranking.Source: Table 3 · p. 8, §3.1 · p. 8
  • 3 of 5 BERGEN tasks show the best match metric for pplx-embed-v1-4B, which outperforms Qwen3-Embedding-4B on four tasks.pplx-embed-v1-0.6B also outperforms Qwen3-Embedding-4B on three tasks. Evaluation retrieves from 24.8 million Wikipedia passages and supplies the top five passages, without reranking, to Qwen2.5-32B-Instruct for answer generation.Source: §3.1 · pp. 8–9, Table 4 · p. 9
  • 44.45% average nDCG@10 places INT8 pplx-embed-v1-4B second overall on ToolRet, while INT8 pplx-embed-v1-0.6B achieves 43.05%.The 4B model scores 42.07% nDCG@10 in the Web category. ToolRet covers 35 API-documentation retrieval tasks across Web, Code, and Custom categories, with the quantized models remaining competitive against larger full-precision baselines.Source: Table 5 · p. 9, §3.1 · p. 9

3.2. Internal Benchmarks

  • 73.46% R@10 and 86.17% R@100 put INT8 pplx-embed-v1-4B ahead of Qwen3-Embedding-4B on PPLXQ2Q’s 2.4M-target-query corpus.Qwen3-Embedding-4B scores 67.90% R@10 and 81.96% R@100; binary pplx-embed-v1-4B retains 72.41% and 85.21%, while pplx-embed-v1-0.6B achieves 71.05% and 83.86%, exceeding BGE-M3 and Qwen3-Embedding-0.6B. Constructed from five days of production logs with PII filtering, the benchmark evaluates 100K queries against progressively larger corpora, treating queries sharing a destination URL as relevant.Source: Table 6 · p. 10, §3.2 · pp. 10–11
  • 88.23% English and 91.66% Multilingual Recall@1000 put pplx-embed-v1-4B above Qwen3-Embedding-4B on PPLXQuery2Doc’s 30M-document corpus.Qwen3-Embedding-4B scores 83.13% and 88.58%, respectively; INT8 pplx-embed-v1-0.6B scores 84.43% and 89.05%, outperforming all sub-1B competitors and supporting first-stage retrieval. The benchmark contains 15,000 stratified queries, uses relevance labels derived from multi-system ranking consensus over candidates retrieved from 1 billion web pages, and evaluates corpora of 7.5M, 15M, and 30M documents.Source: Table 7 · p. 11, §3.2 · pp. 11–12, Table 8 · p. 12

3.3. Effect of Binary Quantization

  • 2–4.4 percentage points of binary-quantization loss affect pplx-embed-v1-0.6B, versus up to 1.6 percentage points for pplx-embed-v1-4B.Contextual 0.6B models lose up to 5 percentage points, compared with about 1 percentage point for contextual 4B models. The authors suggest that the 4B models’ 2560-dimensional outputs, versus 1024 for pplx-embed-v1-0.6B, may help preserve information during binarization.Source: §3.3 · p. 12

4. Diffusion vs. Autoregressive Pretraining

Diffusion pretraining with mean pooling improves English retrieval performance, while bidirectional diffusion backbones achieve lower contrastive-training loss than causally masked Qwen3. Mean pooling also enables computing multiple chunk representations from a single document for contextual embedding training.

4. Diffusion vs. Autoregressive Pretraining

  • Bidirectional diffusion backbones achieve substantially lower contrastive-training loss than causally masked Qwen3 backbones.Four configurations combine the two backbones with mean or last-token pooling, using English pair training for less than one epoch. Qwen3 retains causal masking, while diffusion retains bidirectional attention throughout training.Source: §4 · p. 12
  • ∼1 percentage point of average improvement across retrieval tasks is reported for the combination of mean pooling and diffusion pretraining.The evaluation covers MTEB(En, v2) and the English subset of MIRACLRetrievalHardNegatives.Source: §4 · pp. 12–13, Table 9 · p. 13
  • Mean pooling is crucial for contextual embedding training because it enables computing many chunk-level representations from a single document.Its role therefore extends beyond the modest benchmark improvement.Source: §4 · p. 13

5. Related Work

The work situates continued diffusion pretraining for retrieval within contrastive and contextual embedding research. It emphasizes quantization-aware embedding training for web-scale retrieval and multi-stage contextual training with a specialized dual-loss objective.

Diffusion Language Models

Contrastive Training of Text Embeddings

Contextual Embeddings

  • Earlier contextual methods use neighboring documents within batches (Morris and Rush, 2025) or training-free, single-pass document processing followed by chunk-level pooling (Günther et al., 2024).The latter approach processes document chunks together before pooling their individual representations.Source: §5 · p. 14
  • The contextual approach builds on ConTEB and in-sequence chunk-level contrast (Conti et al., 2025), using multi-stage, dual-loss training while remaining compatible with MTEB and ConTEB.ConTEB specifically evaluates whether embedding models leverage global context (Conti et al., 2025).Source: §5 · p. 14

6. Conclusion · Appendix

The pplx-embed family combines diffusion-based bidirectional attention with multi-stage training to capture global document context. Its four variants demonstrate strong retrieval performance and directly output quantized embeddings for practical deployment.

6. Conclusion

  • The pplx-embed family achieves strong retrieval performance across extensive evaluations on public and internal web-scale benchmarks.The conclusion reports no numerical benchmark results.Source: §6 · p. 14
  • Native quantization-aware training directly outputs INT8 and binary embeddings, offering practical deployment benefits.Source: §6 · p. 14
  • Diffusion-based language models provide bidirectional attention, helping the embedding models better capture global document context.Source: §6 · p. 14
  • The training pipeline develops contextualized chunk encodings relative to full documents.Source: §6 · p. 14
  • Multi-stage training progressively shapes text representations for semantic alignment and fine-grained relevance distinctions.Source: §6 · p. 14
  • 0.6B and 4B parameter scales are available for both pplx-embed-v1 and pplx-embed-context-v1, yielding four model variants.Source: §6 · p. 14

A. Details on Continued Pretraining

Continued pretraining uses a diffusion ELBO objective with independent token masking and a bidirectional reverse model. The section specifies prediction constraints, optimization settings, and the non-English data mixture.

Loss

  • Diffusion pretraining uses an ELBO objective, independently masking tokens with probability t under a linear noise schedule and learning the reverse process with a bidirectional transformer.Loss applies only at masked positions. Retaining the transformer's left shift without prepending [BOS] prevents first-token prediction, limiting the loss sum to the last L−1 positions.Source: A. Details on Continued Pretraining · p. 19

Hyperparameters

  • Pretraining uses automatic mixed bfloat16 precision, FlashAttention-2 (Dao, 2024), and AdamW with weight decay 0.01, β1 = 0.9, and β2 = 0.98.Gradients are clipped by their ℓ2 norm; the threshold is unspecified because it depends on implementation details such as loss scaling.Source: A. Details on Continued Pretraining · p. 19

Hyperparameters and Data

B. Details on Contrastive Training

Contrastive training comprises pair, contextual, and triplet training, with INT8 tanh quantization introduced during pair training and retained during contextual training and inference. Contextual training combines document-level and local objectives, while triplet training adds mined hard negatives.

Pair Training

  • Pair training applies INT8 tanh quantization from the outset, learning reduced-precision representations through InfoNCE with in-batch documents and other queries as negatives.The temperature is 0.02. To improve in-batch negatives, each batch contains one dataset, selected with probability proportional to dataset size.Source: B. Details on Contrastive Training · pp. 19, 21
  • 50,000 steps train both the 0.6B and 4B models with a global batch size of 16,384 and sequence length of 256 tokens.Learning rates are 2×10^-4 for the 0.6B model and 5×10^-5 for the 4B model, balancing training stability and convergence speed.Source: B. Details on Contrastive Training · p. 21

Contextual Training

  • Contextual training initializes from pair-trained checkpoints and uses chunk-annotated synthetic MLDR, MLDR, NarrativeQA, and SQuAD, with quantization during training and inference.Documents contain 16 chunks of 256 tokens each. Queries synthesized for MLDR chunks without queries incorporate neighborhood information to avoid relevance to multiple chunks.Source: B. Details on Contrastive Training · p. 21
  • Contextual training combines global document-level InfoNCE and local losses through hybrid Matryoshka (Kusupati et al., 2022) learning across six equally weighted embedding dimensions.The dimensions are [128, 256, 512, 1024, 2048, 2560]. The dual-objective weight β follows a cosine annealing schedule from 0.2 to 0.5.Source: B. Details on Contrastive Training · p. 21
  • Contextual training mitigates false negatives using relative similarity thresholding, duplicate document masking, and query-query false-negative masking.Thresholding masks negatives within 0.1 of positive similarity; additional query-query similarities serve as hard negatives for the global loss. Training uses 7,000 AdamW steps, learning rate 10^-5, weight decay 0.1, gradient clipping at 1.0, temperature 0.02, and batch sizes of 128 and 16 for the 0.6B and 4B models, respectively.Source: B. Details on Contrastive Training · p. 21

Triplet Training

  • 3 mined hard negatives augment in-batch negative documents per query during triplet training with InfoNCE.Both models train for 2,000 steps with sequence length 512 tokens, global batch size 512, gradient accumulation of 4, and temperature 0.03. Learning rates are 5×10^-5 for the 0.6B model and 10^-5 for the 4B model.Source: B. Details on Contrastive Training · p. 21

C. Details on MTEB Evaluation

The MTEB evaluation reports per-task nDCG@10 for multilingual and code retrieval, with a longer sequence length for LEMBPasskeyRetrieval. Task-specific adjustments address duplicate documents in SyntheticText2SQL and refutation retrieval in ArguAna.

C. Details on MTEB Evaluation

D. Details on ConTEB Evaluation

ConTEB evaluation uses surrounding-chunk context for standard-length documents and fixed overlapping partitions for exceptionally long documents.

D. Details on ConTEB Evaluation

  • Exceptionally long documents, notably ESG Reports documents exceeding 30,000 tokens, use FixedContextualEmbedder to divide documents into overlapping segments.Segment size is configurable.Source: D. Details on ConTEB Evaluation · p. 24
  • Standard-length documents use ContextualEmbedder, which incorporates surrounding chunk information during encoding to generate contextual embeddings.Source: D. Details on ConTEB Evaluation · p. 24
  • 10,000 tokens per segment with a 35-chunk overlap is the default fixed-partitioning configuration for exceptionally long documents.Source: D. Details on ConTEB Evaluation · p. 24

E. Details on BERGEN Evaluation

BERGEN evaluation uses official sentence-transformers implementations through a custom retriever. End-to-end RAG retrieves 100 passages from the KILT dump but presents only the top five to the generator.

Embedding Models

  • A custom BERGEN retriever uses official sentence-transformers implementations of pplx-embed-v1, Qwen3-Embedding, and BGE-M3 for fair evaluation.Qwen3-Embedding query encoding sets prompt_name="query".Source: E. Details on BERGEN Evaluation · p. 24

RAG Configuration

Related papers

Loading 2602.11151v2…