Source-linked AI summary
Distilling Dense Representations for Ranking using Tightly-Coupled Teachers
Sheng-Chieh Lin, Jheng-Hong Yang, Jimmy Lin
TL;DR
Cross-encoder reranking is effective but has high query latency, motivating more efficient retrieval with precomputed representations. This paper distills ColBERT’s MaxSim scoring into dot-product representations and combines dense retrieval with sparse signals. Its advanced dense–sparse hybrids are nearly on par with cross-encoder effectiveness, and the authors describe the approach as promising for low-latency retrieval.
Problem
Cross-encoder rerankers are effective but have high query latency because they apply neural inference to query–document pairs at serving time.
Source: §1 · p. 1Method
The approach distills ColBERT’s MaxSim relevance scoring into dot-product similarity over pooled embeddings and combines dense representations with sparse signals.
Source: §1 · p. 1Results
Advanced doc2query-T5 hybrids achieve effectiveness better than ColBERT and almost on par with the cross-encoder reranker; BM25 hybrids beat CLEAR on both tasks.
Source: §5 · pp. 5–6Takeaways & Limitations
The authors identify dense–sparse hybrids as a promising solution for low-latency end-to-end text retrieval.
Source: §5 · p. 6ColBERT’s MaxSim over the full collection is infeasible, leaving it with multi-stage retrieval, and its token-embedding storage is high relative to φPoolDot.
Source: §2 · p. 3
Abstract
from arXiv · showhide
We present an approach to ranking with dense representations that applies knowledge distillation to improve the recently proposed late-interaction ColBERT model. Specifically, we distill the knowledge from ColBERT's expressive MaxSim operator for computing relevance scores into a simple dot product, thus enabling single-step ANN search. Our key insight is that during distillation, tight coupling between the teacher model and the student model enables more flexible distillation strategies and yields better learned representations. We empirically show that our approach improves query latency and greatly reduces the onerous storage requirements of ColBERT, while only making modest sacrifices in terms of effectiveness. By combining our dense representations with sparse representations derived from document expansion, we are able to approach the effectiveness of a standard cross-encoder reranker using BERT that is orders of magnitude slower.
1 Introduction
Ad hoc retrieval has long relied on sparse representations, while effective cross-encoder reranking incurs high query latency. The paper distills ColBERT into dot-product dense representations, combines them with sparse signals, and uses tightly coupled teacher–student training.
1 Introduction
- Sparse representations have long dominated ad hoc retrieval, while cross-encoder rerankers can take seconds per query despite their effectiveness.Representation-learning approaches can precompute document representations and are more efficient, but are generally less effective than cross-encoder rerankers.Source: §1 · p. 1
- The approach combines dense and sparse representations for low-latency end-to-end ad hoc passage retrieval.Sparse representations remain important for lexical signals in ad hoc retrieval.Source: §1 · p. 1
- Knowledge distillation simplifies ColBERT’s MaxSim relevance computation into dot-product similarity over pooled embeddings.This makes the student’s scoring function simpler than ColBERT’s late-interaction scoring.Source: §1 · p. 1
- Tight coupling interleaves teacher inference with distillation, enabling more flexible strategies and better learned representations.It also avoids periodic index refreshes, but requires a reasonably efficient teacher, ruling out cross-encoder teachers.Source: §1 · pp. 1–2
2 Background
Ad hoc retrieval ranks candidate texts for a query using a similarity function over query and text representations. Dot-product bi-encoders support approximate nearest-neighbor search, while ColBERT’s MaxSim offers late interaction but retains multi-stage retrieval and high storage requirements.
2 Background
- Ad hoc retrieval returns the top-k relevant texts for a query by applying a similarity function to encoded queries and candidate texts.The task is framed as text ranking, with quality assessed using metrics such as average precision or nDCG.Source: §2 · p. 2
- Dot-product similarity supports efficient approximate nearest-neighbor search for finding top-scoring passages.The passage identifies ANN and maximum inner product search as ways to approximate the search.Source: §2 · p. 2
- Transformer bi-encoders independently map queries and passages to contextualized token representations that can be transformed into lower-dimensional vectors.The vectors are formed from contextualized representations of the query or passage tokens.Source: §2 · p. 2
- Pooled embeddings can be compared by dot product, and average pooling over token embeddings is the study’s baseline.The Pool operator may also use maximum pooling or a specific token embedding such as BERT’s [CLS] embedding.Source: §2 · p. 2
- ColBERT uses MaxSim, but computing it across the collection is infeasible, so retrieval still requires a retrieve-then-rerank pipeline.ColBERT stores passage sequences of token embeddings, resulting in high storage requirements compared with pooled dot-product representations.Source: §2 · p. 3
- Hard-negative mining with ANN can reduce the training–inference gap, but periodic corpus-wide index refreshes are computationally demanding.Xiong et al. (2020) report that re-encoding the corpus takes around 10 hours every 5K training steps.Source: §2 · p. 3
- Distillation from precomputed cross-encoder relevance scores can capture reranking effectiveness, but exhaustive score computation may be intractable.The passage attributes this constraint to the computational expense of cross-encoder teachers.Source: §2 · p. 3
3 Methodology
The method distills ColBERT’s MaxSim relevance function into pooled dot-product representations for ANN retrieval, then combines dense and sparse retrieval scores. Knowledge distillation uses teacher and student probability distributions, while hybrid scoring approximates a sparse–dense combination.
3 Methodology
- Average-pooled passage embeddings are precomputed for ANN indexing, so inference encodes the query, pools its embeddings, and searches the index.The approach removes Conv1D to reduce computation and storage cost.Source: §3 · p. 3
3.1 Knowledge Distillation
- The distillation objective combines softmax cross-entropy over relevance labels with KL divergence between teacher and student distributions over sampled passages.ColBERT’s weights are frozen during student fine-tuning; τ is 0.25 and γ is 0.1.Source: §3.1 · pp. 3–4
3.2 Hybrid Dense-Sparse Ranking
- Dense and sparse retrieval each return up to 1,000 passages, and an approximate score combination substitutes a retriever’s minimum score when a passage is absent from its results.The sparse signals may come from original passages or document expansion, including docTTTTTquery.Source: §3.2 · p. 4
4 Experimental Setup
Experiments evaluate ad hoc passage retrieval on MS MARCO using development and TREC-2019 DL protocols. The training procedure has teacher fine-tuning followed by student distillation, with hybrid-ranking weights tuned separately.
4 Experimental Setup
- MS MARCO comprises 8.8 million web passages and 0.5 million relevant question–passage training pairs.Each query has, on average, one relevant passage.Source: §4 · p. 4
- Evaluation uses MS MARCO Dev with 6,980 queries and MRR@10 and R@1000, plus TREC-2019 DL with 43 queries and NDCG@10 and R@1000.TREC-2019 DL includes 9,000 annotated query–passage pairs.Source: §4 · p. 4
- Training first fine-tunes the MaxSim teacher, then freezes it while distilling into a student; both stages run for 160,000 iterations with batch size 96.The student is initialized from the trained teacher weights.Source: §4 · pp. 4–5
- The sparse–dense mixing weight α is tuned on 6,000 sampled training queries, with optimal values 0.10 for BM25 and 0.24 for doc2query-T5.Hybrid experiments use sparse signals from original passages or docTTTTTquery-expanded passages.Source: §4 · p. 5
5 Results
The dense–sparse hybrid beats CLEAR on both tasks, achieves state-of-the-art recall, and approaches cross-encoder effectiveness with substantially greater efficiency. Distillation also improves the dense bi-encoder over PoolAvg and slightly exceeds ANCE in ranking accuracy and recall.
Results
- TCT-ColBERT slightly outperforms ANCE in ranking accuracy and recall; versus PoolAvg, it gains +0.025 (+0.056) MRR@10 and +0.019 (+0.062) R@1000 on MS MARCO (TREC2019 DL).Source: §5 · p. 5
- 4× and 30× more efficient than ColBERT and the cross-encoder reranker, respectively, the advanced hybrids approach cross-encoder effectiveness, beat CLEAR, and achieve state-of-the-art recall on both tasks.The authors describe the advanced hybrids as almost on par with the cross-encoder reranker.Source: §5 · pp. 5–6
Latency
- Table 2 breaks end-to-end latency into query embedding, top-1,000 dense retrieval, and dense–sparse score combination; hybrid latency assumes parallel sparse and dense search and adds 3 ms for combination.Source: Table 2 · p. 6, §5 · p. 6
Ablation study
- Reranking exceeds retrieval in conditions 1 and 2, but in-batch subsampling (condition 3) improves retrieval accuracy, which the authors attribute to distilled knowledge from in-batch samples.Source: §5 · p. 6
- Dynamic subsampling is feasible only with a tightly coupled teacher; importance sampling beyond uniform in-batch subsampling remains future work.Source: §5 · p. 6
6 Conclusions
The authors view knowledge distillation as promising for learned dense ranking representations. They suggest that tighter teacher–student coupling could extend beyond their ColBERT-based approach to other models and contexts.
6 Conclusions
- Learned dense representations could supplement, and perhaps replace, sparse vector representations using inverted indexes.Source: §6 · p. 6
- The authors consider knowledge distillation a promising approach for learned dense representations in ranking.Source: §6 · p. 6
- The authors believe tighter teacher–student coupling could apply beyond their ColBERT-based approach to other models and contexts.Source: §6 · p. 6