Source-linked AI summary
SPLADE v2: Sparse Lexical and Expansion Model for Information Retrieval
Thibault Formal, Carlos Lassance, Benjamin Piwowarski, Stéphane Clinchant
TL;DR
First-stage neural retrieval must improve effectiveness while preserving the efficiency and exact matching associated with sparse inverted-index methods. The paper improves SPLADE through pooling changes, document-only expansion, and distillation, and reports close to state-of-the-art results on MS MARCO and TREC DL 2019 while outperforming recent dense models on zero-shot evaluation.
Problem
Neural first-stage retrieval seeks effective representations that also support fast inference, exact term matching, and efficient inverted-index retrieval despite vocabulary mismatch.
Method
The paper improves SPLADE by modifying pooling, removing query expansion for document-only encoding, and using two-step distillation with harder negatives.
Results
Distilled SPLADE models achieve close to state-of-the-art results on MS MARCO and TREC DL 2019 and clearly outperform recent dense models on zero-shot evaluation.
Takeaways & Limitations
Max pooling substantially improves SPLADE, while the document encoder offers a model suited to faster retrieval conditions.
Takeaways & Limitations
IR computing costs remain incompletely characterized because MS MARCO results commonly use exact brute-force search and few studies examine ANN effects on metrics.
Abstract
from arXiv · showhide
In neural Information Retrieval (IR), ongoing research is directed towards improving the first retriever in ranking pipelines. Learning dense embeddings to conduct retrieval using efficient approximate nearest neighbors methods has proven to work well. Meanwhile, there has been a growing interest in learning \emph{sparse} representations for documents and queries, that could inherit from the desirable properties of bag-of-words models such as the exact matching of terms and the efficiency of inverted indexes. Introduced recently, the SPLADE model provides highly sparse representations and competitive results with respect to state-of-the-art dense and sparse approaches. In this paper, we build on SPLADE and propose several significant improvements in terms of effectiveness and/or efficiency. More specifically, we modify the pooling mechanism, benchmark a model solely based on document expansion, and introduce models trained with distillation. We also report results on the BEIR benchmark. Overall, SPLADE is considerably improved with more than $9$\% gains on NDCG@10 on TREC DL 2019, leading to state-of-the-art results on the BEIR benchmark.
1 INTRODUCTION
First-stage retrieval must balance effectiveness with strict efficiency requirements. This paper improves SPLADE to retain sparse-retrieval advantages while addressing vocabulary mismatch and efficiency.
- 1 INTRODUCTION: BOW first-stage retrieval is efficient but suffers from vocabulary mismatch when relevant documents lack query terms.Neural rankers therefore seek alternatives that support fast online inference with most computation performed offline.
- 1 INTRODUCTION: Sparse representations can inherit exact term matching and inverted-index efficiency while benefiting from neural modeling.This motivates learning sparse query and document representations for first-stage retrieval.
- 1 INTRODUCTION: SPLADE is improved through a modified pooling mechanism, a document-only expansion model, and distillation-based training.The proposed changes target effectiveness and/or efficiency.
- 1 INTRODUCTION: The distilled models achieve close to state-of-the-art results on MS MARCO passage ranking and BEIR zero-shot evaluation.The paper reports these outcomes as part of its improvements over SPLADE.
2 RELATED WORKS
Prior sparse and dense retrieval methods address efficiency, term matching, or vocabulary mismatch with different trade-offs. SPLADE-style approaches estimate vocabulary-term importance and aggregate these scores, while existing methods can remain inefficient or difficult to scale.
- 2 RELATED WORKS: Dense retrieval supports first-stage candidate generation but lacks explicit term matching and can raise scalability concerns through stored token-level embeddings.ColBERT postpones token interactions for retrieval efficiency but stores embeddings for each subterm.
- 2 RELATED WORKS: MS MARCO results commonly use exact brute-force search, leaving the computing cost of approximate nearest-neighbor retrieval insufficiently characterized.Few studies have examined how ANN search affects IR metrics, especially for large collections.
- 2 RELATED WORKS: DeepCT reweights vocabulary terms without changing document vocabulary, so it does not solve vocabulary mismatch without query expansion.Generative methods such as doc2query expand documents, adding terms and repeating existing terms to reweight them.
- 2 RELATED WORKS: SPLADE and related models estimate importance for vocabulary terms from document or query tokens, then aggregate these scores using mechanisms such as summation or maximization.This interaction-based approach differs from indirect query-prediction training used by expansion methods.
- 2 RELATED WORKS: EPIC and document-side SPARTA are not sparse enough by construction unless top-k pooling is used, whereas SparTerm supports fast retrieval.Sparse index structure and posting-list balance remain central efficiency considerations.
3 SPARSE LEXICAL REPRESENTATIONS FOR FIRST-STAGE RANKING
SPLADE learns sparse, expansion-aware representations by predicting vocabulary-term importance from input tokens, then combines ranking and sparsity objectives. The paper extends this design with max pooling, document-only scoring, and distillation to improve effectiveness or efficiency.
- 3.1 SPLADE: SPLADE predicts importance for every vocabulary token from each input token's BERT representation, producing sparse representations over a 30,522-token WordPiece vocabulary.The prediction layer is equivalent to masked-language-model prediction and can be initialized from a pretrained MLM.
- 3.1 SPLADE: The final representation sums token-level importance predictors after log saturation, while ranking uses the dot product between query and document representations.Training uses a contrastive ranking loss with in-batch negatives, including positive documents from other queries as negatives.
- 3.1 SPLADE: SPLADE jointly optimizes ranking and sparsity objectives, using distinct query and document regularization weights to apply more sparsity pressure to queries for fast retrieval.The sparsity objective is the FLOPS regularizer, designed to support efficient indexing and retrieval.
- 3.2 Pooling strategy: SPLADE-max replaces sum pooling with max pooling, and the resulting model considerably improves SPLADE performance.This change makes the model more similar to SPARTA, EPIC, and to some extent ColBERT.
- 3.3 SPLADE document encoder: SPLADE-doc removes query expansion and query-term weighting, so document term weights alone determine the ranking score and can be pre-computed offline.The design reduces inference cost while still providing competitive results in the experiments.
- 3.4 Distillation and hard negatives: DistilSPLADE-max uses two-step distillation: a SPLADE retriever and cross-encoder generate harder triplets and scores for training a new SPLADE model.The second step trains the final model from scratch with these triplets and Margin-MSE scores.
4 EXPERIMENTAL SETTING AND RESULTS
The experiments evaluate SPLADE variants on MS MARCO, TREC DL 2019, and a readily available BEIR subset, focusing on effectiveness–efficiency trade-offs. Max pooling, document-only expansion, and distillation improve or preserve retrieval performance while offering efficiency benefits.
- Experimental setting: The models are trained and evaluated on MS MARCO passage ranking, with approximately 8.8M passages and shallow relevance annotations.The evaluation also uses 6,980 development queries and 43 finely annotated TREC DL 2019 queries.
- Experimental setting: Evaluation reports Recall@1000, MRR@10 on MS MARCO dev, and NDCG@10 on TREC DL 2019, comparing first-stage sparse and dense rankers.The comparison includes BM25, doc2query-T5, SparTerm, COIL-tok, DeepImpact, ANCE, TCT-ColBERT, and TAS-B.
- Experimental setting: MS MARCO dev and TREC DL 2019 results select the best-performing model with reasonable FLOPS-based efficiency from a regularization-strength grid.Figure 1 shows MRR@10 against estimated query–document floating-point operations.
- Results: Improved SPLADE models outperform other sparse retrieval methods by a large margin on MS MARCO dev and TREC DL 2019, while remaining competitive with state-of-the-art dense methods.The reported evaluation uses Table 1 for MS MARCO and TREC results.
- Results: Max pooling adds almost 2 points in MRR@10 and NDCG@10 over the SPLADE baseline on MS MARCO and TREC, improving the effectiveness–efficiency trade-off.SPLADE-max is consistently better than SPLADE in Figure 1 and also improves BEIR performance.
- Results: The document-only max-pooling model matches the previous SPLADE performance, outperforms doc2query-T5 on MS MARCO, and requires no query encoder.At an average of 19 non-zero weights per document, it reaches MRR@10=29.6, on par with doc2query-T5; document processing requires one forward pass.
- Results: Distillation improves SPLADE substantially, reaching 0.368 MRR@10 at approximately 4 FLOPS and 0.35 MRR at approximately 0.3 FLOPS.DistilSPLADE-max outperforms all other methods on most datasets in the evaluated BEIR subset.
5 CONCLUSION
The paper improves SPLADE through a revised pooling mechanism and distillation, while identifying document-only encoding as useful for faster retrieval. The distilled model approaches state-of-the-art performance on MS MARCO and TREC DL 2019 and outperforms recent dense models in zero-shot evaluation.
- Max pooling provides a substantial improvement over the original SPLADE pooling mechanism.
- The document encoder is an interesting model for faster retrieval conditions.
- Distilled SPLADE leads to close-to-state-of-the-art results on MS MARCO and TREC DL 2019.
- Distilled SPLADE clearly outperforms recent dense models in zero-shot evaluation.