Arctic-Embed 2.0: Multilingual Retrieval Without Compromise

Efficiency losses

• Many state-of-the-art multilingual models require numerous parameters and large vectors, raising dense-retrieval costs for large corpora.The paper identifies both computational and economic costs.Source: §1 · p. 1

Compromised English retrieval quality

• Multilingual models commonly underperform English-only counterparts on English retrieval, a deployment pain point.Source: §1 · p. 1

• Arctic-Embed 2.0 outperforms leading open-source alternatives across English and non-English contexts, with competitive results on English and multilingual benchmarks.Source: §1 · pp. 1–2
• The paper refutes the hypothesis that pretraining on distant languages harms English retrieval and proposes alternative explanations for future investigation.Source: §1 · p. 2
• The authors report that pretrained-checkpoint evaluations often fail to predict fine-tuned performance, and that contrastive pretraining can produce negative cross-lingual transfer.Fine-tuning generally enhances cross-lingual transfer.Source: §1 · p. 2

2.1 Masked Language Modeling

• The medium and large models start from gte-multilingual-mlm-base and bge-m3-retromae, respectively, both using the XLM-R tokenizer.Source: §2.1 · p. 2

2.2 Contrastive Training Data

• Fine-tuning combines Merrick et al.'s English data mix with MIRACL multilingual training samples, using all MIRACL languages.Mr. Tydi is excluded because it overlaps with MIRACL.Source: §2.2 · p. 2

2.3 Methods of Data Filtering and Training

• Multilingual query-document pairs are filtered by retrieval consistency: pairs ranked below 20 within roughly three-million-pair shards are removed.The filtering uses embeddings from the small multilingual E5 model.Source: §2.3 · p. 2

2.4 Hard Negative Mining for Finetuning

• Stronger teacher models and higher false-positive cutoffs improved hard-negative fine-tuning data, while random ordering matched or beat hardness-based curricula.The figure summarizes these hard-negative mining ablations.Source: §2.4 · p. 2, Figure 2 · p. 3

2.5 Matryoshka Representation Learning

• MRL loss is applied during both pretraining and fine-tuning at 256 dimensions, yielding 3x and 4x compression for medium and large models.The resulting homogeneous component distribution facilitates further compression with aggressive quantization.Source: §2.5 · p. 3

3.1 Evaluation Data Sets

• CLEF's news-domain, training-free datasets assess multilingual retrieval beyond MIRACL's Wikipedia domain.Source: §3.1 · p. 3

3.2 Results

• Arctic-Embed 2.0 achieves the best results in its size categories on MTEB-R and CLEF, while remaining highly competitive on MIRACL, especially in trained languages.The paper notes that mE5, mGTE, and BGE-M3 perform well on MIRACL but are weaker on CLEF than Arctic-Embed 2.0 and closed-source offerings.Source: §3.2 · p. 3
• At 256 dimensions, the medium model scores 0.549 versus Google text-embedding-004's 0.524 on MTEB-R, retaining 99% of its original score.The large model retains 98%; Google text-embedding-004 retains 94% and OpenAI Text Embedding 3 Large 93%.Source: §3.2 · p. 3

4.1 RQ1: Cross-lingual Transfer

• The cross-lingual-transfer question asks how contrastive pretraining benefits languages absent from its data; the authors report that both research questions remain open.They focus on transfer during pretraining after observing strong MIRACL scores, including for languages not covered by contrastive pretraining.Source: §4 · p. 3, §4.1 · p. 4

4.1.1 Experiments

• The checkpoint experiments evaluate MIRACL every 2K steps through 10K, then every 10K, both directly and after finetuning.The medium model is evaluated during contrastive pretraining.Source: §4.1.1 · p. 4

4.1.2 Results

• On MIRACL’s English subset, the 130K-step checkpoint is 18.3% worse than the 8K checkpoint without finetuning, but 3.1% better after finetuning.This reversal shows why checkpoint evaluation before finetuning can mislead.Source: §4.1.2 · p. 4
• Contrastive pretraining shows little cross-lingual transfer: performance trends are negative across most language families absent from training, with finetuned gains peaking within 10K steps.Reported declines include Chinese (-13.3%), Japanese (-6.9%), and French (-2.9%).Source: §4.1.2 · p. 4

Why do many multilingual embedding models

• Adding Chinese pretraining data to an English baseline improves English retrieval, contradicting the adversarial-language hypothesis, and outperforms Spanish and German data on MTEB-R and MIRACL, and sometimes CLEF.The additions use equal amounts of English, German, Spanish, or Chinese data; Chinese-data results vary across consistency-filtering levels.Source: Why do many multilingual embedding… · p. 5, Figure 4 · p. 5
• The authors propose data quality and model capacity as alternative explanations for English retrieval gaps, including the possibility that training on many languages exceeds capacity.They also suggest lower-quality multilingual data may account for much of the gap.Source: Why do many multilingual embedding… · pp. 5–6

4.3 Reflections On The Journey

• The authors hypothesize that data quality helps explain their lack of English degradation, while noting uncertainty in quantifying quality, evaluating more languages out of domain, and understanding capacity limits.They identify incremental addition of non-English data to a proven English mix as their most effective available strategy.Source: §4.3 · p. 6
• The authors distinguish model knowledge from task calibration: extended pretraining may lower unfinetuned retrieval scores, while finetuning reverses the trend.They suggest success may combine large-scale pretraining with recalibration using denoised positives and hard negatives.Source: §4.3 · p. 6

5 Conclusion

• Prolonged contrastive pretraining does not always improve cross-lingual transfer.The authors report that high-quality pretraining data from languages distant from English can improve transfer in some settings.Source: §5 · p. 6
• High-quality pretraining data from languages distant to English can improve cross-lingual transfer in some settings.Source: §5 · p. 6
• Finetuning can reverse prolonged pretraining’s negative impact on downstream retrieval, a finding the authors identify as a direction for further inquiry.Source: §5 · p. 6

A Selection of Base Models

• The two model sizes are intended to balance efficiency and effectiveness trade-offs.Source: A Selection of Base Models · p. 9
• The medium model starts from the m-GTE checkpoint, while the large model starts from BGE-M3; both use XLM-R tokenization.m-GTE uses unpadding and xFormers acceleration, while BGE-M3 benefits from RetroMAE pretraining.Source: A Selection of Base Models · p. 9
• The tested Llama3 tokenizer brought no significant effectiveness gain and added efficiency overhead after the models were reinitialized and pretrained without MLM.Source: A Selection of Base Models · p. 9

B Pretraining Data Breakdown

• The target languages are English, French, Spanish, German, Italian, Portuguese, and Polish, using English data plus mC4, CC News, and multilingual Wikipedia.Source: B Pretraining Data Breakdown · p. 9
• CC News pages are treated as query-document pairs using titles and bodies, while Wikipedia uses titles and section headings as queries and section texts as documents.Source: B Pretraining Data Breakdown · p. 9
• Retrieval-based consistency filtering yields approximately 1.41 billion unsupervised query-document pairs.The filtering is applied to each language subset of each dataset.Source: B Pretraining Data Breakdown · p. 9

C Implementation Details

• Pretraining uses InfoNCE with temperature τ = 0.02, random in-batch negatives, and mini-batches drawn from one data source at a time.Different-language subsets of multilingual sources count as distinct sources during batching.Source: C Implementation Details · p. 9
• 1.41 billion contrastive pretraining samples are broken down by data source and language in Figure 5.The figure shows source breakdown above language breakdown.Source: Figure 5 · p. 9
• Pretraining uses a batch size of 32,768 and a three-epoch WSD schedule, with activation checkpointing and 32 H100 GPUs.The peak learning rates are 3e-5 for the large model and 1e-4 for the medium model.Source: C Implementation Details · p. 9
• Finetuning uses smaller high-quality datasets and curated negatives, with batches of 256 sets containing one query, one positive, and 10 negative documents.Query and document limits are extended to 512 tokens; the learning rates are 1e-5 for medium and 5e-6 for large models.Source: C Implementation Details · pp. 9–10
• The finetuning WSD schedule has no warmup and decays linearly for 6,000 of 9,342 total steps.Table 2 reports CLEF evaluation dataset statistics: queries, corpus documents, relevance judgments, and judgments per query.Source: C Implementation Details · p. 10, Table 2 · p. 10

D CLEF Dataset Details

• CLEF datasets are used as established benchmarks for non-English monolingual and cross-lingual retrieval.They are used to evaluate multilingual models’ out-of-domain generalizability.Source: D CLEF Dataset Details · p. 10
• Evaluation permits a maximum of 512 tokens for E5 and 8192 tokens for all other models on CLEF.The limit is enabled because CLEF includes documents longer than 512 tokens.Source: D CLEF Dataset Details · p. 10
• Table 2 reports CLEF dataset counts for queries, corpus documents, relevance judgments, and average annotations per query.Source: D CLEF Dataset Details · p. 10

E Replication of “Language Gap” with Fewer Pretraining Samples

• The 13K-step experiment varies the MLM base model, pretraining data, and finetuning data between English-only and multilingual configurations.The multilingual finetuning configuration combines English finetuning data with the non-English MIRACL training set.Source: E Replication of “Language Gap” with… · p. 10
• None of the tested multilingual treatments causes a sizable MTEB Retrieval decrease versus the all-English baseline.Table 3 reports the impact of MLM base model, pretraining data, and finetuning data configurations on downstream retrieval.Source: E Replication of “Language Gap” with… · p. 10, Table 3 · p. 10
• Adding non-English MIRACL finetuning data has a slight positive effect even with the English MLM base model.Source: E Replication of “Language Gap” with… · p. 10