Source-linked AI summary
Giga-Embeddings: Mixture-of-Experts Encoders for High-Throughput Text Embeddings
Egor Kolodin, Egor Krasnoperov, Evgeniy Kosarev, Fyodor Minkin
TL;DR
Embedding models must balance retrieval quality with the storage and serving costs of large encoders. Giga-Embeddings addresses this with a sparse MoE encoder and dimension-agnostic similarity-distribution distillation, achieving the family’s strongest aggregate benchmark performance while offering compact alternatives. The authors release all three checkpoints, but single-run scores, one serving environment, non-public training data, and aggregate metrics limit interpretation and reproduction.
Problem
Billion-parameter embedding encoders can improve representation quality but are costly to store and serve.
Method
The family combines sparse activation in a 10B MoE encoder with similarity-distribution distillation for a compact 480M encoder.
Results
The 10B-A1.8B model achieves the highest aggregate score within the family on English, Russian, Multilingual, and Code benchmarks.
Takeaways & Limitations
The released family offers high-capacity sparse, dense, and compact distilled encoders for different compute and memory budgets.
Takeaways & Limitations
Single-run scores lack uncertainty estimates, throughput reflects one serving environment, and non-public training data limits full reproduction and independent auditing.
Abstract
from arXiv · showhide
We introduce Giga-Embeddings, a family of text embedding models designed to combine strong retrieval quality with efficient serving. Its largest member is a sparse 10B-parameter Mixture-of-Experts encoder with approximately 1.8B active parameters per token. Across English, Russian, multilingual, and code MTEB benchmarks, this model achieves the strongest aggregate performance within the family on all four evaluated suites. In our vLLM benchmark with 1024-token inputs, it processes 114.5k tokens per second, providing 25 percent higher throughput than the dense 3B model and 1.56-2.65x the throughput of the evaluated external systems. The family also includes a dense 3B encoder and a distilled 480M encoder for tighter compute and memory budgets. We train the compact model using a dimension-agnostic objective that aligns teacher and student similarity distributions. The resulting 480M model scores 70.98 on Russian MTEB, surpassing FRIDA while using 42 percent fewer parameters. We release all three model checkpoints.
1. Introduction
Giga-Embeddings targets the tension between embedding quality and serving cost through sparse activation and similarity-distribution distillation. The family includes a high-capacity MoE encoder, a dense encoder, and a compact distilled encoder.
- Billion-parameter encoders can improve representation quality but are costly to store and serve.
- Giga-Embeddings combines sparse activation for high-capacity models with similarity-distribution distillation for compact encoders.
- The 10B MoE model activates approximately 1.8B parameters per token and achieves the highest measured throughput among compared systems.
- The 10B-A1.8B encoder achieves the best aggregate scores in the model family, leads the external Russian comparison, and has the highest throughput at all three sequence lengths.
- 70.98 on Russian MTEB lets the distilled 480M encoder surpass FRIDA’s 70.95 while using 42% fewer parameters.
- All three model checkpoints—480M, 3B, and 10B-A1.8B—are released as open source.
2. Related Work
Related work spans general-purpose embedding encoders, embedding distillation, and sparse-to-dense transfer. The paper situates its approach among contrastive embedding systems, listwise similarity distillation, and MoE efficiency methods.
- General-purpose text embeddings: General-purpose embedding systems evolved from efficient bi-encoders and contrastive recipes toward scaled data, negative mining, and instruction-conditioned language-model encoders.
- Embedding distillation: Embedding distillation includes softened output matching, direct compression, cross-architecture transfer, and listwise similarity distillation.
- Sparse teachers and dense students: MoE models increase total capacity while activating only a subset of experts per token.
- Sparse teachers and dense students: The paper connects sparse-to-dense transfer with final-output distillation because similarity scores can transfer knowledge without matching internal architectures.
3. Model and Training Recipe
The models adapt decoder-only language models into bidirectional embedding encoders and train them in three stages. The compact model additionally uses similarity-distribution distillation, while the larger models do not.
- 3.1. Encoder adaptation: Decoder-only backbones are converted to bidirectional encoders, mean-pooled into fixed-dimensional vectors, and normalized for cosine-similarity scoring.
- 3.1. Encoder adaptation: The 480M and 3B models are dense Qwen3 encoders, while 10B-A1.8B is a bidirectional DeepSeekMoE-style encoder with 10B total and approximately 1.8B active parameters per token.
- 3.2. Three-stage contrastive training: Training proceeds through contrastive pre-training, retrieval fine-tuning with hard negatives, and multitask fine-tuning for retrieval, classification, clustering, and semantic similarity.
- 3.2. Three-stage contrastive training: Pre-training uses in-batch InfoNCE, whereas later stages use explicit candidate groups and mined hard negatives for retrieval fine-tuning.
- 3.3. Similarity-distribution distillation: Similarity-distribution distillation is applied only to 480M during retrieval and multitask fine-tuning, aligning teacher and student similarity distributions rather than embedding coordinates.
- 3.3. Similarity-distribution distillation: The combined distillation objective transfers relative similarity structure while retaining direct contrastive supervision.
4. Evaluation
Evaluation across four MTEB suites shows the 10B-A1.8B model leading its family on every benchmark, while the MoE configuration provides the highest measured throughput. The 480M model remains compact, with consistent distillation gains and substantially lower weight storage, though external comparisons use differing settings.
- Efficiency: 0.96, 6, and 20 GB are the estimated fp16 weight-storage requirements for the 480M, 3B, and 10B-A1.8B models.These theoretical estimates exclude activations and other runtime state; the 480M model uses about 4.8% as much weight storage as the MoE model.
- Aggregate results: 69.52, 70.98, 56.97, and 72.87 are the 480M model’s scores on English, Russian, Multilingual, and Code MTEB suites.Its gaps relative to the 3B model are 2.41, 3.58, 6.92, and 4.06 points, with the largest gap on Multilingual.
- Aggregate results: The 10B-A1.8B model achieves the highest score within the family on English, Russian, Multilingual, and Code benchmarks.Its improvements over 3B are +0.30, +0.42, +1.75, and +1.48 points, respectively.
- Distillation ablation: 0.09, 0.12, and 0.22 points are the distillation gains on English, Russian, and Code scores, respectively, with the largest improvement on Code.The matched ablation isolates teacher supervision’s contribution to the 480M model.
- Inference throughput: The MoE serving configuration has the highest measured throughput at every evaluated sequence length.Relative to dense 3B, throughput is 1.28×, 1.25×, and 1.13× as high at 512, 1024, and 2048 tokens; measurements are specific to the vLLM benchmark environment.
- External baseline comparison: Public MTEB comparisons are contextual rather than controlled same-run comparisons because prompts and evaluation settings differ.Giga-Embeddings leads the Russian column, while Qwen3-Embedding-8B leads English and Multilingual and F2LLM-v2-14B leads Code.
5. Conclusion
Giga-Embeddings combines sparse activation for high-capacity embedding inference with similarity-distribution distillation for compact encoders. The 10B-A1.8B model leads the family’s aggregate benchmark scores and throughput, while the 480M model gains 0.09–0.22 points from distillation.
- Conclusion: Giga-Embeddings-10B-A1.8B adapts a sparse Mixture-of-Experts backbone into a bidirectional text-embedding encoder.It combines 10B resident parameters with approximately 1.8B active parameters per token.
- Conclusion: The 10B-A1.8B model achieves the highest throughput among five evaluated serving configurations and the best aggregate family scores across four benchmark suites.It also leads the external comparison on Russian MTEB.
6. Limitations
The evaluation has uncertainty and scope constraints: benchmark scores come from single runs, throughput findings are environment-specific and incomplete, and training data limits reproducibility. Aggregate MTEB scores may also conceal variation across tasks and languages.
- Single-run benchmark results provide no uncertainty estimates, so small score differences should be interpreted cautiously.
- Throughput measurements are specific to one serving environment, omit the 480M model, and do not isolate sparse activation from other differences.
- The non-public training mixture limits full reproduction and independent auditing for benchmark contamination.
- Aggregate MTEB scores can hide variation across individual tasks and languages, motivating broader matched evaluations and systems reporting.