Source-linked AI summary
SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking
Thibault Formal, Benjamin Piwowarski, Stéphane Clinchant
TL;DR
First-stage retrieval needs methods that combine neural effectiveness with efficient, scalable search. SPLADE learns sparse query and document expansions using logarithmic activation and sparsity regularization, achieving competitive retrieval while allowing efficiency control. The paper presents this approach as a straightforward, end-to-end alternative that can operate with inverted indexes.
Problem
First-stage retrieval must provide strong neural effectiveness while meeting strict efficiency and scalability requirements, and dense retrieval does not explicitly model term matching.
Method
SPLADE learns sparse query and document expansions end-to-end using logarithmic activation, in-batch negatives, and explicit FLOPS regularization.
Results
SPLADE rivals state-of-the-art dense retrieval models while achieving sparsity levels similar to bag-of-words approaches without performance cost against the best first-stage rankers.
Takeaways & Limitations
SPLADE offers controllable efficiency, straightforward training, and inverted-index operation for initial retrieval.
Takeaways & Limitations
SparTerm's expansion-aware gating is intricate and prevents end-to-end training because it is learned beforehand and fixed during matching-model fine-tuning.
Abstract
from arXiv · showhide
In neural Information Retrieval, ongoing research is directed towards improving the first retriever in ranking pipelines. Learning dense embeddings to conduct retrieval using efficient approximate nearest neighbors methods has proven to work well. Meanwhile, there has been a growing interest in learning sparse representations for documents and queries, that could inherit from the desirable properties of bag-of-words models such as the exact matching of terms and the efficiency of inverted indexes. In this work, we present a new first-stage ranker based on explicit sparsity regularization and a log-saturation effect on term weights, leading to highly sparse representations and competitive results with respect to state-of-the-art dense and sparse methods. Our approach is simple, trained end-to-end in a single stage. We also explore the trade-off between effectiveness and efficiency, by controlling the contribution of the sparsity regularization.
1 INTRODUCTION
First-stage retrieval must balance neural ranking effectiveness with strict efficiency and scalability requirements. SPLADE addresses this by learning sparse, expanded representations that retain inverted-index advantages while improving lexical matching.
- 1 INTRODUCTION: First-stage retrieval is typically separated from expensive language-model reranking because online inference must remain efficient.Bag-of-words models commonly perform candidate generation in a two-stage ranking pipeline.
- 1 INTRODUCTION: Bag-of-words retrieval is efficient but suffers from vocabulary mismatch when relevant documents lack query terms.Inverted indexes support efficient retrieval, but exact lexical matching can miss semantically related documents.
- 1 INTRODUCTION: SPLADE combines logarithmic activation and sparse regularization to perform efficient document expansion with competitive results against dense methods.The paper also studies how sparsity regularization controls the efficiency–effectiveness trade-off.
2 RELATED WORKS
Related work contrasts dense neural retrieval with sparse lexical approaches. Dense methods achieve strong first-stage performance, while sparse methods aim to preserve exact matching, inverted-index efficiency, and interpretability.
- 2 RELATED WORKS: Dense BERT Siamese models became standard for candidate generation, with training strategies such as improved negative sampling and distillation affecting performance.ColBERT retains efficient first-stage retrieval while modeling postponed token-level interactions.
- 2 RELATED WORKS: Approximate-nearest-neighbor evaluation is underexamined in information retrieval because MS MARCO results are often reported with exact brute-force search.This reporting practice provides little indication of effective computing cost.
- 2 RELATED WORKS: Sparse approaches learn high-dimensional representations or contextualized term weights, but prior methods faced effectiveness or efficiency concerns.SNRM uses ℓ1 regularization, while DeepCT transfers pretrained-language-model knowledge into vocabulary-space term weights.
3 SPARSE LEXICAL REPRESENTATIONS FOR FIRST-STAGE RANKING
SparTerm builds sparse vocabulary-level representations by predicting term importance from BERT and masking dimensions for inverted-index retrieval. SPLADE replaces fixed gating with log-saturation and joint ranking–sparsity training, while sparsity regularization controls retrieval efficiency.
- SparTerm: SparTerm predicts vocabulary-term importance from BERT WordPiece representations, then sums token-level predictors with ReLU to form positive term weights.Its importance predictor is equivalent to the masked-language-model prediction and can be initialized from a pretrained MLM.
- SparTerm: SparTerm interprets the resulting representations as query/document expansion and uses binary gating to retain sparse dimensions for inverted-index retrieval.Lexical-only gating keeps terms appearing in the input, whereas expansion-aware gating learns additional dimensions while preserving input terms.
- SparTerm: SparTerm’s expansion-aware gating is learned before matching-model fine-tuning, preventing end-to-end optimization of the ranking task’s sparsification strategy.The lexical-only and expansion-aware strategies perform almost equally well, which questions the practical benefit of expansion in SparTerm.
- SPLADE: SPLADE introduces log-saturation in term-importance estimation to prevent dominant terms and naturally obtain sparse representations.Experiments report better results and sparse solutions even without additional regularization.
- SPLADE: ℓFLOPS regularization pushes down high average term weights, producing a more balanced index than ℓ1 regularization.The motivation is that minimizing ℓ1 alone does not ensure evenly distributed posting lists.
- SPLADE: SPLADE discards SparTerm’s binary gating and jointly optimizes ranking with query- and document-specific sparse regularization.Separate weights allow stronger query sparsity pressure, which is critical for fast retrieval; regularization can use ℓ1 or ℓFLOPS.
4 EXPERIMENTAL SETTING AND RESULTS
The experiments evaluate SPLADE and related sparse and dense first-stage rankers on MS MARCO and TREC DL 2019, emphasizing effectiveness, efficiency, and expansion. SPLADE achieves competitive retrieval performance while producing sparse representations and a controllable effectiveness–efficiency trade-off.
- Evaluation: SPLADE models outperform other sparse retrieval methods by a large margin, except for Recall@1000 on TREC DL, while remaining competitive with state-of-the-art dense methods.
- Evaluation: Recall@1000 close to 0.96 is achieved by ST exp-ℓ1 on MS MARCO dev, but with a much bigger average number of FLOPS.
- Effectiveness-efficiency trade-off: SPLADE reduces FLOPS to levels similar to bag-of-words approaches without performance loss compared with the best first-stage rankers.
- Method and implementation: SPLADE is trained end-to-end in a single step and uses FLOPS regularization to decrease computing cost.
- Effectiveness-efficiency trade-off: At FLOPS=0.05, strongly regularized models retain competitive performance with MRR@10=0.296.
- The role of expansion: Expansion improves recall while balancing expansion and compression; at FLOPS=0.05, documents and queries average 18 and 6 non-zero values, respectively.That configuration drops 34 terms per document on average, adds 5 expansion terms, and requires less than 1.4 GB for the index.
5 CONCLUSION
SPLADE is presented as an effective and efficient sparse first-stage retrieval model that rivals state-of-the-art dense methods. Its straightforward training, explicit sparsity control, and compatibility with inverted indexes support further sparse-retrieval improvements.
- SPLADE revisits query and document expansion to produce effective sparse representations for initial retrieval.The model uses in-batch negatives, logarithmic activation, and FLOPS regularization.
- SPLADE rivals the latest state-of-the-art dense retrieval models for first-stage retrieval.
- Its training procedure is straightforward and its sparsity or FLOPS can be explicitly controlled through regularization.
- SPLADE operates on inverted indexes, retaining an efficiency-oriented retrieval infrastructure.
- The paper positions SPLADE as a solid basis for further improvements in sparse retrieval because of its simplicity.