Fantastic (small) Retrievers and How to Train Them: mxbai-edge-colbert-v0 Tech Report

• Prior BERT- and MiniLM-based ColBERTs have short context limits and poor efficiency, while small ColBERT models remain popular despite architectural limitations.The introduction identifies a gap between strong full-sized models and the widely used answerai-colbert-small-v1.Source: §1 · pp. 2–3
• The authors introduce 17M- and 32M-parameter models as modern, low-parameter ColBERT baselines, trained from dense embedding baselines through three stages and subsequent ablations.They intend the models to support further experiments.Source: §1 · p. 3
• 48-dimensional mxbai-edge-colbert-v0-17m outperforms ColBERTv2 despite the commonly used 128-dimensional embeddings, with low compute and memory footprint.The authors also describe strong long-context handling and very low latencies for on-device re-ranking.Source: §1 · p. 3
• The family is presented as a foundation for further ColBERT studies and a range of real-world uses.The authors describe it as the first of a hoped-for series of efficient edge models.Source: §1 · p. 3

2 Creating a Suitable Dense Base Model

• The authors leave a ColBERT-specific contrastive warm-up phase for future work while developing a backbone to study individual modifications.They state that such a warm-up may hold potential for further improvements.Source: §2 · p. 3

2.1 Contrastive Pre-Training

• Contrastive pre-training uses open datasets with approximate query-to-document matches, trained sequentially one dataset at a time, which the authors found empirically better than joint training.The section uses the contrastor framework and a common selection of datasets.Source: §2.1 · p. 4
• Higher learning rates are needed for ModernBERT and Ettin models to reach satisfying results compared with previous backbone encoders.The passage points to NanoBEIR NDCG@10 results across learning-rate runs.Source: §2.1 · p. 4

2.2 Fine-tuning

• Supervised fine-tuning uses mined hard negatives to distinguish similar documents, but negatives that are too hard can harm high-level representations and raise false-negative risk with sparse labels.The authors mix Qwen3-Embedding-8B-mined negatives with BM25-mined and randomly mined examples.Source: §2.2 · pp. 4–5

2.3 “Stella-style” Distillation

• The authors add embedding-space distillation using teacher-generated vectors and simplify its loss to L2 distance between student and teacher vectors.They sample varied documents and retrieval queries because input-length diversity is described as helpful, although retrieval relationships are not used in this stage.Source: §2.3 · p. 6
• Distillation improves NanoBEIR NDCG@10 for both dense model sizes, with larger gains for 32M than 17M.The authors hypothesize that their streamlined loss may have more difficulty bridging the larger dimensionality gap for the 17M model, but do not investigate further.Source: §2.3 · p. 7

3 ColBERT Training

• The training stage runs ablations to establish a baseline for edge use and further research, using MSMARCO data and 16-way tuples with teacher scores.The stated setup uses batch size 128 and KL-Div loss with normalized scores, except where specified otherwise.Source: §3 · p. 7

3.1 Data

• BGE-Gemma2 consistently outperforms Qwen3-Reranker as a distillation teacher, whether its scores are normalized or unnormalized.The authors associate Qwen3-Reranker's extreme score distributions with possible reranker overfitting; changing temperature did not significantly alter performance.Source: §3.1 · p. 8, Table 5 · p. 8

3.2 Ablations

• Muon appears to be a strong ColBERT training optimizer in the tested settings, despite limited experiments and the relatively small batch size commonly used for late-interaction training.The comparison varies AdamW and Muon learning rates at fixed batch size; NDCG@10 is averaged across four ablation datasets.Source: Optimizers · p. 8, Table 6 · p. 9
• Stella-style distillation improves the resulting ColBERT model even when projection heads are discarded and only the backbone is retained.The comparison uses the dense embedding model after finetuning versus the model after distillation.Source: Impact of Stella-style Distillation · p. 8, Table 7 · p. 9
• Performance declines only mildly down to a projection dimension of 48, but degrades considerably at dimensions of 32 and below.The comparison covers dimensions from 16 to 96 on the 32M model using 20% of the training data.Source: Projection Dimension · pp. 9–10, Table 8 · p. 9
• A two-layer feedforward projection with an upscaled intermediate dimension and residual connection outperforms the standard ColBERT projection on the 17M model.Across training seeds, no single-layer checkpoint came within less than 1 NDCG@10 of the two-layer checkpoints; significance testing was not performed.Source: Projection Layers · p. 10, Table 9 · p. 10
• Lower-casing significantly improves the 17M model, while the 32M model shows no significant difference or consistent trend.The comparison keeps other hyperparameters equal and trains on the full data; the proposed explanation for the 17M result is a hypothesis, not a tested mechanism.Source: Casing · p. 10, Table 10 · p. 11

4 Results

• With 17M parameters and a projection dimension of 48, mxbai-edge-colbert-v0-17m outperforms ColBERTv2, which has more than six times as many parameters and a dimension of 128.The BEIR table reports NDCG@10 for the average and sampled tasks including MSMARCO, SciFact, Touche2020, FiQA, TREC-COVID, NQ, and DBPedia; the paper says the models are overall strong but below answerai-colbert-small on short-context average.Source: Table 11 · p. 11, §4 · pp. 11–12
• On LongEmbed, both models are outperformed only by the larger GTE-ModernColBERT, while the 17M variant exceeds the current under-1B single-vector retrieval state of the art by almost 20 NDCG@10 points.The LongEmbed table includes 4k and 32k context variants where supported and otherwise uses each model's maximum sequence length.Source: §4 · pp. 12–13, Table 12 · p. 12
• The models handle 32k sequence lengths and gain performance despite retrieval training using documents truncated to 220 tokens.The models are based on an encoder with a native 8,000-token context window.Source: §4 · p. 13
• The authors identify low latency on CPU for long-document reranking as a practical advantage, noting that no existing reranker reaches similarly strong performance in that setting.Source: §4 · p. 13
• The small-ColBERT comparison reports NanoBEIR NDCG@10, long-context support, projection dimension, CPU and GPU runtime, and fp16 representation memory for 10,000 documents.Runtime averages ten runs; the reported hardware includes one RTX 4090 for GPU evaluation.Source: Table 13 · p. 13, Efficiency comparison with other small… · p. 13

5 Conclusion

• The models are the first small ColBERT models to fully benefit from a modern architecture, including long-context support and ModernBERT-generation efficiency improvements.Source: §5 · p. 14
• The family is intended as a testbed for future experiments and distillation, a performance predictor for scaling-law experiments, and support for on-device RAG and reranking.Source: §5 · p. 14
• The 17M variant outperforms ColBERTv2 with an order-of-magnitude fewer parameters and two-thirds lower vector storage and scoring-time compute requirements.Source: §5 · p. 14
• The authors intend to continue upgrading the models with future developments and anticipate real-world use.Source: §5 · p. 14

A Distillation Data

• The distillation data mix is listed in Tables 14 and 15.Source: A Distillation Data · p. 17, Table 14 · p. 17, Table 15 · p. 17
• DCLM-Pro and FineWeb contribute full documents, while dclm_sent contains DCLM-Pro documents broken into individual sentences.Source: A Distillation Data · p. 17
• ettin_tokens contains single-token inputs, and english_words pairs English words with definitions generated by Gemini 2.0 Flash.Source: A Distillation Data · p. 17

B Full BEIR Results

• Full BEIR results are presented in Tables 16 and 17.Source: B Full BEIR Results · p. 17
• Table 16 covers AVG and seven named BEIR tasks, with scores reported as NDCG@10.Source: Table 16 · p. 18
• Table 17 covers the remaining listed BEIR tasks, with scores reported as NDCG@10.Source: Table 17 · p. 18