Source-linked AI summary
From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effective
Thibault Formal, Carlos Lassance, Benjamin Piwowarski, Stéphane Clinchant
TL;DR
Sparse neural retrievers have received less attention to training improvements than dense models, despite their lexical matching and inverted-index advantages. This paper applies distillation, hard-negative mining, and retrieval-oriented initialization to SPLADE, then studies effectiveness–efficiency trade-offs across in-domain and zero-shot settings. The combined strategies achieve state-of-the-art results, including 38 MRR@10 on MS MARCO, while more expressive models are generally more effective but costlier.
Problem
Sparse retrievers inherit lexical matching and inverted-index advantages, but it is unclear whether they can benefit from the training improvements used for dense architectures.
Method
The paper extends SPLADE with distillation, hard-negative mining, retrieval-oriented PLM initialization, and analyses of effectiveness versus FLOPS efficiency.
Results
Combined strategies achieve state-of-the-art in-domain and zero-shot results; CoCondenser-EnsembleDistil reaches 38 MRR@10 on MS MARCO.
Takeaways & Limitations
Performance improvements are additive, while more expressive models are generally more effective and sparse neural models can generalize strongly to zero-shot evaluation.
Takeaways & Limitations
Efficiency is compared using FLOPS because more informative measures such as query latency can be difficult to evaluate across systems.
Abstract
from arXiv · showhide
Neural retrievers based on dense representations combined with Approximate Nearest Neighbors search have recently received a lot of attention, owing their success to distillation and/or better sampling of examples for training -- while still relying on the same backbone architecture. In the meantime, sparse representation learning fueled by traditional inverted indexing techniques has seen a growing interest, inheriting from desirable IR priors such as explicit lexical matching. While some architectural variants have been proposed, a lesser effort has been put in the training of such models. In this work, we build on SPLADE -- a sparse expansion-based retriever -- and show to which extent it is able to benefit from the same training improvements as dense models, by studying the effect of distillation, hard-negative mining as well as the Pre-trained Language Model initialization. We furthermore study the link between effectiveness and efficiency, on in-domain and zero-shot settings, leading to state-of-the-art results in both scenarios for sufficiently expressive models.
1 INTRODUCTION
Information retrieval is shifting from efficient lexical matching toward contextualized neural retrieval, while sparse models revisit lexical representations with learned weighting and expansion. This work asks whether sparse SPLADE models can benefit from training improvements developed for dense retrievers.
- Dense retrievers combine contextualized representations with Approximate Nearest Neighbors search to address vocabulary mismatch.
- Sparse representation learning couples learned term weighting or expansion with inverted indexing, preserving explicit lexical matching and efficiency-oriented IR priors.
- The paper studies whether SPLADE can benefit from distillation, hard-negative mining, and other training improvements used for dense architectures.
2 RELATED WORKS
Sparse neural retrieval extends lexical models with learned term importance and contextualized matching, but its training has received less attention than dense retrieval. The paper therefore examines whether training improvements transfer across architectures.
- Candidate-generation models must satisfy high serving-latency constraints while using either dense ANN retrieval or sparse inverted-index retrieval.
- Dense Representation Learning: Dense retrievers commonly improve representations through distillation, hard-negative mining, retrieval-oriented pre-training, or combinations of these strategies.
- Dense Representation Learning: ColBERT uses late interaction for fine-grained term-level matching, but this increases indexing cost and latency.
- Sparse Representation Learning: Sparse PLM methods learn term importance, expansion, or contextualized lexical matching while inheriting lexical-model properties.
- Motivation: The central question is whether training improvements remain additive for sparse models, allowing architecture and training innovations to be compared separately.
3 SPLADE AND METHODOLOGY
The methodology builds SPLADE sparse representations from PLM token-importance predictions, pools them with max pooling, and scores queries and documents by dot product. Training combines ranking or distillation objectives with FLOPS regularization, while hard negatives and retrieval-oriented initialization provide the main extensions.
- SPLADE: SPLADE predicts importance over the BERT WordPiece vocabulary and uses masked-language-modeling predictions to perform implicit term expansion.
- SPLADE: Max pooling aggregates token-level importance predictors after log saturation and is used as the default because it experimentally improves over sum pooling.
- SPLADE: Query and document representations are compared with a dot-product ranking score.
- Training: Training combines a contrastive or MarginMSE ranking objective with query- and document-side FLOPS regularization to encourage sparse retrieval representations.
- Pre-training: Retrieval-oriented PLM checkpoints are considered as an initialization strategy alongside changes to ranking losses and negative sources.
- Distillation: Distillation uses MarginMSE between teacher and student positive-negative margins, and serves as the default setting for subsequent scenarios.
- Hard-negative mining: Self-mining trains an initial SPLADE model, mines triplets with it, obtains teacher scores, and retrains using the mined pairs.
- Hard-negative mining: Ensemble mining uses hard negatives from BM25 and twelve dense retrievers, together with cross-encoder scores for MarginMSE distillation.
4 EXPERIMENTS AND EVALUATION
The experiments evaluate SPLADE++ training strategies across in-domain and zero-shot retrieval, measuring effectiveness alongside efficiency. Distillation, hard-negative mining, and retrieval-oriented initialization yield additive gains, while more expressive models generally improve effectiveness at higher FLOPS.
- Experimental setup: SPLADE++ models are trained on MS MARCO passage ranking with modified losses, hard-negative inputs, and/or pre-trained language-model initialization.The study trains five regularization configurations per scenario to expose effectiveness–efficiency trade-offs.
- Experimental setup: Evaluation covers MS MARCO dev, TREC DL 2019, and zero-shot BEIR retrieval, using standard retrieval metrics and FLOPS for efficiency.BEIR comparisons use a readily available subset of 13 datasets, while the figures summarize broader dataset averages.
- Results: 38 MRR@10 is reached on MS MARCO by CoCondenser-EnsembleDistil, while the study reports state-of-the-art zero-shot results for its CoCondenser scenarios.The best in-domain configuration combines distillation, ensemble mining, and a retrieval-oriented pre-trained checkpoint.
- In-domain results: Distillation provides the largest reported effectiveness boost, improving MRR@10 by +1.6 across scenarios.The result is reported for the DistilMSE scenario in the MS MARCO analysis.
- Zero-shot results: Combining SPLADE++ with BM25 produces additional zero-shot gains, indicating complementarity between pure lexical retrieval and sparse neural models.This effect is emphasized especially for zero-shot evaluation.
- Effectiveness–efficiency trade-offs: Effectiveness generally increases with model expressiveness and FLOPS, including on BEIR, whereas SelfDistil scenarios outperform EnsembleDistil counterparts in zero-shot evaluation.The authors also report that results are stable across random seeds and that improvements are less clear for some metrics, including TREC DL 2019 nDCG@10.
5 CONCLUSION
The paper extends SPLADE with distillation, hard-negative mining, and improved PLM initialization. Combined, these improvements yield state-of-the-art performance in both in-domain and zero-shot evaluation while linking effectiveness with efficiency.
- Combined distillation, hard-negative mining, and improved PLM initialization produce state-of-the-art performance in in-domain and zero-shot evaluation.
- More expressive models show better generalization as regularization controls the trade-off between effectiveness and efficiency.
A DETAILED BEIR EVALUATION
The complete BEIR evaluation reports nDCG@10 across all 18 datasets and finds that SPLADE variants achieve state-of-the-art results overall.
- SPLADE variants achieve state-of-the-art results overall on the BEIR evaluation.Table 4 reports nDCG@10 for all 18 datasets.
- The comparison includes results from corresponding papers, generally evaluated on the subset of 13 readily available BEIR datasets.