pplx-embed-v2-late Technical Report

1. Introduction

• Late interaction uses offline-indexed document embedding bags and MaxSim to offer richer query-document interaction than dense embeddings.Dense embeddings use cheap inner products for efficient approximate nearest-neighbor search; cross-encoders process each query-document pair together, requiring a separate forward pass per ranked candidate.Source: §1 · p. 1
• The 0.6B and 9B models are distilled at the representation level from a contrastively trained 18B teacher and share an embedding space.The report says representation distillation outperforms contrastive training alone, especially at smaller model scales.Source: §1 · p. 1
• Shared embeddings allow documents indexed by the 9B model to be queried with the 0.6B model, with query latency-critical work assigned to the smaller encoder.The report says the 0.6B model can also build an index queried by the 9B model, and indexing costs can be amortized over the index lifetime.Source: §1 · pp. 1–2
• Both models achieve state-of-the-art performance across a range of text, image, and visual document retrieval benchmarks.The report includes ViDoRe-V3 among these benchmarks.Source: §1 · p. 1

2. Training

• The 18B teacher is contrastively trained first on query-document pairs and then on query, relevant-document, and hard-negative triplets.The report reuses pair data for distillation; all late-interaction models use 128-dimensional embeddings.Source: §2 · p. 2

2.1. Contrastive Teacher Training

• The teacher uses Qwen3.5 with bidirectional full-attention layers, initialized by pruning the last 20 layers of Qwen3.5-27B; LoRA is applied across attention and feed-forward projections.Pair training uses InfoNCE with a contrastive batch size of 16,384 query-document pairs; triplet training adds a rank-32 adapter, freezes the vision tower, and uses up to four hard negatives per query.Source: §2.1 · pp. 2–3
• Pair training uses 186M rows across 981 language- and domain-partitioned sub-datasets, with a target mixture of 56.5% text-text, 30.9% text-image, and 12.6% text-visual-document pairs.A quasi-random golden-ratio Kronecker sequence is mapped through the sampling distribution to keep sub-dataset and source-dataset sampling proportional and regularly spaced; Figure 1 compares the method with independent random sampling.Source: §2.1 · p. 2, Figure 1 · p. 3

2.2. Representation Distillation

• Students match the frozen teacher’s L2-normalized token representations at unmasked query and document positions, averaging each side’s distances and weighting the two averages equally.Because the distillation loss is not contrastive, batches can mix sub-datasets to reduce gradient noise without weakening the training signal.Source: §2.2 · p. 3
• The 0.6B student has 594M parameters, with about 240M active for text encoding and 340M for image encoding, and is fully fine-tuned.The report says full fine-tuning outperforms parameter-efficient approaches, especially on visual document retrieval tasks; the 9B student fully fine-tunes its last eight text-tower layers and applies LoRA to remaining projections.Source: §2.2 · pp. 3–4

3.1. Text retrieval

• 81.3% nDCG@10 is the 9B model’s domain-specific text average, 1.6 pp above the next best; the 0.6B model averages 78.0%.The 9B model scores highest on every domain except health; the 0.6B model scores highest among models under 1B parameters on every domain.Source: §3.1 · p. 4, Table 1 · p. 5
• 74.8% and 73.6% Recall@1000 are the 9B and 0.6B scores on Q2D-Web Combined, versus 69.3% for the strongest baseline.On Markdown ViDoRe-V3, both models outperform all evaluated baselines, with the 9B model more than 4 pp above the best competitor.Source: §3.1 · p. 5, Table 2 · p. 5

3.2. Text-to-image retrieval

• 65.2% nDCG@10 on image ViDoRe-V3 gives the 9B model the highest average among vision-language models, while EVIE-8B scores 66.4%.The 0.6B model scores 62.3%; on MIRACL-Vision the models score 67.5% and 75.7%, and on PPLX-Q2I they score 60.4% and 65.0%.Source: §3.2 · p. 6, Table 3 · p. 6

3.3. Cross-model retrieval

• The 0.6B-query/9B-document pairing beats the 0.6B/0.6B pairing on every domain by 1.6 pp on average, while 9B/9B scores highest at 81.3%.The hybrid keeps query-encoding cost the same as 0.6B/0.6B; Figure 2 reports nDCG@10 across six text domains.Source: §3.3 · p. 7, Figure 2 · p. 7

3.4. Agentic retrieval

• Both models achieve higher BrowseComp+ accuracy with fewer searches than all evaluated baselines; the 9B model reaches 64.0% accuracy.That is 4.9 pp above Reason-ModernColBERT and 8.7 pp above the best dense model; Figure 3 plots accuracy and recall against average searches.Source: §3.4 · p. 7, Figure 3 · p. 7

• The 0.6B and 9B models are representation-distilled late-interaction models based on a contrastively trained 18B teacher.Source: §4 · p. 8
• On Q2D-Web and Markdown ViDoRe-V3, both models outperform all evaluated baselines compared with other models of their size.Source: §4 · p. 8
• On ViDoRe-V3 image retrieval, both models have the highest averages among vision-language models, while vision-only EVIE models score higher.Source: §4 · p. 8
• Indexing with the 9B model and querying with the 0.6B model improves over using the 0.6B model alone at the same query encoding cost.The models share an embedding space because both students are trained to match the same teacher.Source: §4 · p. 8
• In agentic retrieval, both models reach higher BrowseComp+ accuracy with fewer searches than all evaluated baselines.Source: §4 · p. 8