Source-linked AI summary
Adaptive Sparsity Optimization with Learnable Soft Top-K and Per-Term Thresholding for Efficient Retrieval
Wentai Xie, Parker Carlson, Shanxiu He, Tao Yang
TL;DR
LLM-based sparse retrieval can produce long query and document vectors, particularly with a large vocabulary, challenging retrieval-time and storage efficiency. AdaSparse combines adaptive sparsification with FLOPs regularization. With Lion-SP, it reports substantially shorter vectors and lower retrieval costs while retaining highly competitive relevance.
Problem
Lion-SP's large LLaMA-3 vocabulary and existing sparsification methods leave query and document vectors long, challenging retrieval efficiency while preserving relevance.
Source: §1 · p. 1, §3 · pp. 2–3Method
AdaSparse combines learnable soft per-vector top-K and per-term thresholds with FLOPs regularization to optimize sparsity.
Source: §1 · p. 1Results
4x-5x shorter expanded query and document lengths, 3.8x lower retrieval latency, and 4.2x less storage are reported with highly competitive relevance.
Source: §1 · p. 2Takeaways & Limitations
With Lion-SP, AdaSparse reduces retrieval latency and storage while maintaining highly competitive relevance on MS MARCO and BEIR evaluations.
Source: §1 · p. 2AdaSparse is designed and evaluated for sparsity optimization with Lion-SP; applying or extending it to other LLMs remains future work.
Source: §6 · p. 10
Abstract
from arXiv · showhide
Recent work on neural sparse retrieval has demonstrated strong relevance by leveraging Large Language Models (LLMs) for semantic term expansion. However, learned models paired with previous sparsification techniques still yield overly long document and query vectors partly due to a large LLM vocabulary, imposing a serious challenge to retrieval time and space efficiency. This paper proposes a scheme for optimizing model sparsity through a synergy of adaptive strategies, including learnable soft top-K, per-term thresholding, and FLOPs regularization to increase the sparsity of query and document vectors. Experimental results with Lion-SP model on the MS MARCO and BEIR datasets demonstrate that the proposed scheme can outperform the baselines by significantly reducing the average query and document lengths. Our scheme can achieve much shorter retrieval latency and lower storage cost while maintaining highly competitive relevance.
1 Introduction
AdaSparse targets the long vectors produced by LLM-based sparse retrieval, where large vocabularies and existing pruning strategies make sparsity difficult to improve without sacrificing relevance. It combines adaptive sparsification methods and reports shorter vectors, lower retrieval costs, and highly competitive relevance.
1 Introduction
- Lion-SP's 128,256-token LLaMA-3 vocabulary contributes to long vectors, and retrieval latency and index storage generally scale with non-zero terms.Lion-SP document vectors are reported as 8.9x longer than those of a SPLADE variant.Source: §1 · p. 1
- Earlier pruning methods offer limited learnability and fine-grained, context-aware adaptation for LLM-driven representations.These methods include hard limits, uniform top-token selection, mass-ratio pruning, and learnable hybrid thresholding.Source: §1 · p. 1
- AdaSparse combines learnable soft per-vector top-K and per-term thresholds with FLOPs regularization, avoiding hard pruning after the K-th ranked term.Soft top-K lets the model capture sparsity-penalty signals and adjust term weights during refinement.Source: §1 · p. 1
- 4x-5x shorter expanded query and document lengths, 3.8x lower retrieval latency, and 4.2x less storage are reported with highly competitive relevance.The evaluation uses Lion-SP on MS MARCO passage retrieval and BEIR zero-shot retrieval.Source: §1 · p. 2
2 Background and Related Work
The background describes how learned sparse retrieval builds query and document vectors, scores them, and uses inverted indexes for retrieval. It situates Lion-SP within SPLADE-style FLOPs-regularized models and related sparse-retrieval approaches.
2 Background and Related Work
- Query vectors are generated online, while document vectors are pre-generated offline using a learned transformer-based model.Retrieval ranks documents by similarity between their query and document vectors.Source: §2 · p. 2
- SPLADE-family models project contextualized representations into vocabulary-sized sparse vectors and apply log-smoothing, ReLU, and max-pooling across input positions.The resulting weights represent the importance of vocabulary terms for an input query or document.Source: §2 · p. 2
- Document scores are query-document vector dot products, and inverted indexes support fast, low-cost inference for sparse vectors with many zeros.The passage cites prior inverted-index work [29].Source: §2 · p. 2
- Lion-SP is the sparse Lion model, adapting LLaMA with bidirectional attention and using a SPLADE-like loss with FLOPs regularization.The loss combines a ranking objective with sparsity regularization.Source: §2 · p. 2
- Related approaches include sparse autoencoders, hard top-K sparsification for VDR, and contextual sparse representations that account for original words.The passage cites sparse autoencoders, top-K selection [2] [32], VDR [40], and CSurF.Source: §2 · p. 2
3 Design Considerations
The design considerations trace Lion-SP's sparsity challenge to LLaMA-3's large, redundant vocabulary and explain why existing pruning methods can be too coarse. The proposed design goals emphasize adaptable, fine-grained pruning and complementary regularization.
3 Design Considerations
- LLaMA-3's 128K vocabulary, word-form variants, and multilingual pretraining can increase redundant activations and lengthen English retrieval vectors.Its vocabulary is 4.2× larger than BERT's, and the passage describes separate tokens for variants such as case forms.Source: §3 · p. 2
- A randomly sampled Lion-SP-1B sparse query vector has length 272, with 9 English variants and two Chinese terms for “year”; AdaSparse condenses it to 4 English variants of “years” and 3 of “governor”.SPLADE with BERT generates a vector of length 20, including two variants of “governor”.Source: §3 · p. 2
- The paper frames the task as reducing representation size after training while balancing the relevance of Lion-SP with online efficiency constraints.Table 2 is described as comparing the advantages and disadvantages of four existing sparsification methods.Source: §3 · p. 2
- Fixed-size pruning can mismatch query complexity, FLOPs regularization may leave a tail of small weights, and global thresholds poorly fit term-specific distributions.The passage also notes that a global threshold tuned on one domain may suppress every term in some zero-shot vectors.Source: §3 · p. 3
- The design goals call for learnable soft top-K that adapts to each vector and individualized term thresholds suited to term-specific weight distributions.The proposed methods and FLOPs regularization are described as complementary, motivating a multi-objective loss.Source: §3 · p. 3
4 The Proposed Methods for AdaSparse
AdaSparse combines vector-adaptive soft top-K selection and per-term thresholding with FLOPs regularization to sparsify Lion-SP query and document vectors. The methods complement one another: soft top-K controls expansion scale, while thresholding targets weak terms.
4.1 STop: Learnable soft top-𝐾
- STop uses a context-dependent soft top-K limit to penalize terms beyond the boundary during training without explicitly pruning them.Its gate can exempt original terms and provide a gradient signal for below-boundary weights; noisy importance rankings can make it brittle, which a larger expansion factor mitigates.Source: §4.1 · pp. 3–5
4.2 PTT: Per-term thresholding
- PTT learns individualized term thresholds and sets weights below their thresholds to zero during document indexing and online query generation.It refines pruning granularity, but may prune ineffectively when a term’s document-weight distribution is extremely flat; the paper also reports sparsity resistance with a large PTT regularization weight.Source: §4.2 · p. 5
4.3 Multi-objective sparsity regularization
- AdaSparse combines rank loss, FLOPs regularization, PTT, and STop losses, with training gradients updating the model weights.During inference, it drops terms below threshold and encodes vectors without explicit top-K pruning; document vectors are indexed and query vectors retrieve from that index.Source: §4.3 · pp. 5–6
- The three sparsification techniques complement one another: FLOPs shrinks large activations, STop targets below-pivot terms, and rank loss can preserve or promote relevance-improving terms.PTT scales small weights using updated term-specific thresholds; STop can contribute a strong gradient when its gate is near 1, including where PTT or FLOPs are less effective.Source: §4.3 · p. 6
5 Evaluation Studies
The evaluation compares AdaSparse with sparse-retrieval baselines on MS MARCO and BEIR, tests sparsification settings, and examines Seismic latency as vector lengths change.
5.1 Settings
- Experiments use MS MARCO for training and in-domain evaluation and 13 BEIR datasets for zero-shot retrieval; vector lengths proxy latency, with Seismic latency measured under approximately matched effectiveness loss.The study uses LLaMA-3.2 models with 1B and 8B parameters and reports MRR@10 for MS MARCO Dev and NDCG@10 for TREC DL and BEIR.Source: §5.1 · pp. 6–7
5.2 A comparison with baselines
- ≈0% MS MARCO Dev and 0.74% BEIR performance degradation accompany AdaSparse-1B query and document lengths of 31% and 26.6% of Lion-SP-1B-repro.AdaSparse-8B reduces query length by 82% and document length by 79.6%, with performance drops below 0.3% on MS MARCO Dev and about 1.6% on BEIR. HT-1B has 1% MRR degradation on MS MARCO Dev and 1.6% on BEIR versus Lion-SP-1B-repro; its document length is 39.6% longer than AdaSparse-1B's.Source: §5.2 · p. 7, Table 6 · p. 8
- On BEIR, AdaSparse reduces average query and document lengths to 20% and 25% of Lion-SP with FLOPs-only regularization, while its expansion adapts to dataset needs.Eleven of 14 datasets have query expansion within the soft limit b=10; three BEIR datasets exceed it, with query lengths 11.7x–13.5x the original. AdaSparse's MRR@10 is 1% higher than HT-1B on MS MARCO Dev, and its BEIR NDCG@10 is 0.9% higher.Source: §5.2 · p. 8
5.3 Sensitivity of hyperparameters 𝑏and 𝜆𝑇
- Lowering query soft limit b shortens query vectors; relative to increasing λT, it produces a smaller zero-shot performance drop and similar MS MARCO relevance.The paper selects b as the primary sparsity-control parameter because it gives more direct control of vector lengths than λT; default average expansion is 8.1x for queries and 3.7x for documents.Source: §5.3 · pp. 8–9
- The smaller AdaSparse configuration retains default MS MARCO relevance with nearly identical MRR, but has a modest BEIR relevance drop from more aggressive sparsification.At comparable or better sparsity, it still outperforms other baselines in both the smaller and default configurations.Source: §5.3 · p. 9, Table 10 · p. 9
5.4 Ablation studies with different sparsification compositions
- STop reduces query and document lengths within 1% relevance degradation, and PTT further improves sparsity without compromising relevance; UTT incurs nearly 2% lower BEIR NDCG@10 than PTT.For SCIDOCS terms, weight distributions vary by term; the global UTT threshold exceeds all three PTT thresholds and brings a greater relevance penalty.Source: §5.4 · pp. 9–10, Figure 3 · p. 10
5.5 Sensitivity of Seismic’s retrieval time to query and document lengths
- Seismic latency increases monotonically with document length but stays relatively stable or rises modestly with query length, consistent with AdaSparse's tighter document expansion.The latency study varies average document length from 49 to 393 and query length from 9 to 69.Source: §5.5 · p. 10, Table 12 · p. 10
6 Concluding Remarks
AdaSparse combines learnable soft top-K and per-term thresholding with FLOPs regularization to optimize sparsity. Reported results show substantial storage and retrieval-speed gains alongside competitive relevance, while comparisons and evaluation scope leave areas for further work.
6 Concluding Remarks
- AdaSparse combines learnable soft top-K regularization and per-term thresholding with FLOPs regularization as a multi-objective sparsification scheme.Source: §6 · p. 10
- 4.2x less space and 3.8x-4x faster Seismic retrieval than Lion-SP, with relevance within 0.74% difference.Evaluations covered MS MARCO training and testing and zero-shot retrieval on BEIR datasets.Source: §6 · p. 10
- Against Lion-SP-1B with HT, AdaSparse-1B uses 1.8x less storage and is up to 2.1x faster, with 1% higher relevance.Against MRP, it delivers better relevance at similar sparsity, including 2.3% higher average NDCG@10 on BEIR.Source: §6 · p. 10
- Using LLaMA-3.2 8B significantly boosts MS MARCO Dev performance, while the BEIR gain is modest.Source: §6 · p. 10
- AdaSparse is designed and evaluated with Lion-SP; future work will extend it to other LLMs and study latency with different sparse retrievers.Like SPLADE, Lion-SP and AdaSparse do not mark titles in MS MARCO training.Source: §6 · p. 10
- 4.3x smaller index size than RepLLaMA (34GB vs. 145GB); AdaSparse relevance is higher on Dev, DL19, and DL20 but 1.5% lower on average BEIR NDCG@10.The comparisons also note that SPLADEv3 has 1.7x shorter document length, while Lion-SP with AdaSparse gains relevance on MS MARCO and BEIR.Source: Table 13 · p. 10, §6 · p. 10