SparTerm: Learning Term-based Sparse Representation for Fast Text Retrieval

1 INTRODUCTION

• Traditional term-based methods such as TF-IDF and BM25 efficiently support literal matching but are considered inadequate for semantic-level matching.Dense representations show promise for semantic matching, but efficiency and accuracy tradeoffs arise for extremely large collections.Source: §1 · p. 1
• Earlier PLM-based methods DeepCT and Doc2Query reweight or expand terms through auxiliary models, while SparTerm learns the sparse representation directly.SRNM uses latent tokens, which can lose the interpretability of original terms valued in industrial systems.Source: §1 · pp. 1–2
• SparTerm uses an importance predictor and gating controller to learn sparse vocabulary-wide representations that combine term weighting with expansion.The predictor estimates importance across the vocabulary, while the controller selects which terms are activated.Source: §1 · p. 2
• SparTerm significantly outperforms previous sparse models on MSMARCO and even surpasses Doc2Query-T5 in top-ranking performance despite its 2x model size and 70x pre-training corpus size.The paper also reports state-of-the-art ranking performance among sparse models.Source: §1 · p. 2

2.1 Bag-of-words Methods

• SRNM learns latent sparse representations to address lexical mismatch, but latent tokens may not preserve exact-term retrieval or interpretability.The paper contrasts this with SparTerm, which directly learns sparse representations in the whole vocabulary.Source: §2.1 · p. 2, §2.3 · p. 2

2.2 PLMs for dense text retrieval

• Dense PLM retrieval includes bi-encoders and late interaction, but the cited analysis identifies capacity limits for long documents and exact matching, while late interaction remains costly at scale.ColBERT is reported to run 150x faster than fully interactive BERT with comparable precision, yet its late-interaction layer remains computationally infeasible for large-scale first-stage retrieval.Source: §2.2 · p. 2

2.3 PLMs for sparse text retrieval

• DeepCT uses contextualized information to estimate term weights, while Doc2Query expands document content with potential queries; both use auxiliary models unlike SparTerm.The cited passage describes Doc2Query as improving over traditional BM25.Source: §2.3 · p. 2

3.1 Overview

• SparTerm maps a passage to a vocabulary-space representation by combining importance prediction with a binary gating vector, while limiting the number of active terms.The importance predictor assigns semantic importance across the vocabulary; the controller selects which terms appear in the final representation.Source: §3.1 · pp. 2–3

3.2 The Importance Predictor

• For each contextualized passage token, the importance predictor produces a dense vocabulary-wide importance distribution, which is summed across tokens into passage-level importance.The predictor thereby covers terms beyond those literally present in the passage, supporting both term weighting and expansion.Source: §3.2 · p. 3

3.3 The Gating Controller

• SparTerm offers literal-only gating, which reduces representation learning to reweighting original passage terms, and expansion-enhanced gating, which can activate related non-literal terms.The two controller types are intended for scenarios with different lexical-matching requirements.Source: §3.3 · p. 3

3.4 Training

• The importance predictor is trained end-to-end with a ranking objective that learns term importance from passage-ranking supervision rather than directly fitting statistical term-importance distributions.The training instances contain a query, a positive passage, and a negative passage; the stated objective is the negative log likelihood of the positive passage.Source: §3.4 · p. 4
• For expansion-enhanced gating, a binary cross-entropy objective trains vocabulary-wide gate probabilities against passage-to-query or passage-to-summary targets.The framework describes four expansion types; the paper focuses on passage-to-query-like and summarization terms, and sets the positive-term loss weight much larger than the weight for terms not expected to expand.Source: §3.4 · p. 4, Table 1 · p. 4
• The authors also propose jointly training the importance predictor and gating controller using ranking supervision.The passage introduces this joint-training strategy but does not specify further details here.Source: §3.4 · p. 4

4.1 Datasets and Metrics

• The passage and document retrieval experiments use MSMARCO datasets, with a new dev set sampled for retrieval evaluation and performance measured by MRR@10 and Recall at top 10–1000.The passage collection contains 8.8M passages; the document dataset has 367,013 training queries and 5,193 dev queries.Source: §4.1 · pp. 4–5

4.2 Implementation

• SparTerm is fine-tuned on passage retrieval data in two stages: first the Gating Controller, then the jointly trained model with the controller fixed.The model is not fine-tuned on document retrieval training data; the passage-trained model is used for document ranking.Source: §4.2 · p. 5

4.3 Baselines and Experimental Settings

• The experiments compare BM25, DeepCT, Doc2query, and Doc2query-T5 with literal-only, expansion-only, and expansion-enhanced SparTerm settings.The baselines cover term reweighting and document expansion; the SparTerm settings isolate literal weighting, expansion, or both.Source: §4.3 · p. 5

5.1 Performance on Passage Full Ranking

• SparTerm (expansion-enhanced) outperforms all baselines on MRR and all except Doc2query-T5 on Recall in MSMARCO passage full ranking.The authors report stronger improvements on MRR and Recall@10-100, indicating a greater ability at top ranking than previous sparse models.Source: §5.1 · p. 5

5.2 Performance on Document Ranking

• Despite differing training data, SparTerm (expansion-enhanced) performs better than HDCT on document ranking using maximum passage scores.The authors caution that the comparison is not strictly comparable: SparTerm is fine-tuned on MSMARCO passage ranking, whereas HDCT was trained on MARCO document titles.Source: §5.2 · pp. 5–6, Table 3 · p. 6

5.3 Comparison of Different Query Representation Methods

• On the built Dev set, the symmetric neural query representation performs better than the asymmetric model on MRR and Recall.The authors suggest the asymmetric model’s doubled parameter count may make convergence harder; tf-based query representations produce sharper passage-term-weight distributions than neural queries.Source: §5.3 · p. 6, Table 4 · p. 6

5.4 Analysis of Term Weighting

• Compared with DeepCT, SparTerm’s literal-only weighting activates more terms, including important terms absent from the query.DeepCT produces sparser, sharper distributions and can miss terms such as “allergic reaction”; the smoother SparTerm distribution allows retrieval by more queries.Source: §5.4 · p. 6, Figure 3 · p. 7

5.5 Analysis of Term Expansion

• SparTerm’s gating controller assigns expansion probabilities to semantically similar terms absent from a passage, including “sign” and “temperature.”Figure 3 depicts expanded terms and their pre-binarization probabilities; darker color represents higher term weights.Source: §5.5 · p. 6, Figure 3 · p. 7
• Expanded terms include query-related passage2query terms, synonyms, and co-occurring words; the authors associate these patterns with gating-controller supervision and reused MLM pretraining.For passage2query terms such as “temperature,” nearly every passage word contributes; the authors attribute the other two patterns more likely to MLM pretraining.Source: §5.5 · p. 6, Figure 4 · p. 7

• On MSMARCO, SparTerm outperforms previous sparse models based on comparable PLM sizes and achieves state-of-the-art ranking performance among sparse models.Source: §6 · p. 7
• SparTerm maps frequency-based BoW representations to sparse term-importance distributions across the full vocabulary, combining term weighting and expansion.Source: §6 · p. 7
• Further empirical analysis examines how PLM knowledge transfers to sparse methods and reports that SAPRT significantly increases their upper limit.Source: §6 · p. 7