Source-linked AI summary
Curriculum Learning for Dense Retrieval Distillation
Hansi Zeng, Hamed Zamani, Vishwa Vinay
TL;DR
Sparse relevance judgments limit available dense-retrieval training data, motivating distillation from teacher rankings. CL-DRD progressively supplies harder teacher-derived ordering supervision to a dense-retrieval student; experiments report effectiveness on three public passage-retrieval benchmarks, including an 11% TAS-B improvement on MS MARCO Dev MRR@10.
Problem
Dense retrieval requires substantial training data, while relevance judgments are often incomplete and cover few documents per query.
Source: §1 · p. 1Method
CL-DRD iteratively increases the difficulty of teacher-produced distillation data, moving from coarse document distinctions toward finer teacher-ranked ordering.
Source: §1 · p. 1, §2.2 · p. 2Results
Takeaways & Limitations
The experiments support applying curriculum-based distillation to enhance both single-representation TAS-B and multi-representation ColBERTv2 dense retrievers.
Source: §1 · p. 1The multi-vector setting entails higher memory requirements, indexing cost, and query-processing cost.
Source: §3 · p. 4
Abstract
from arXiv · showhide
Recent work has shown that more effective dense retrieval models can be obtained by distilling ranking knowledge from an existing base re-ranking model. In this paper, we propose a generic curriculum learning based optimization framework called CL-DRD that controls the difficulty level of training data produced by the re-ranking (teacher) model. CL-DRD iteratively optimizes the dense retrieval (student) model by increasing the difficulty of the knowledge distillation data made available to it. In more detail, we initially provide the student model coarse-grained preference pairs between documents in the teacher's ranking and progressively move towards finer-grained pairwise document ordering requirements. In our experiments, we apply a simple implementation of the CL-DRD framework to enhance two state-of-the-art dense retrieval models. Experiments on three public passage retrieval datasets demonstrate the effectiveness of our proposed framework.
1 INTRODUCTION
Dense retrieval depends on large-scale training data, but relevance judgments are often sparse. CL-DRD uses teacher-generated training data in an iterative curriculum, moving from coarse distinctions to finer document ordering; experiments apply it to two state-of-the-art models on three public passage-retrieval benchmarks.
1 INTRODUCTION
- Sparse relevance judgments constrain dense-retrieval training, while knowledge distillation transfers ranking knowledge from a more expensive re-ranking teacher to a dense-retrieval student.The teacher scores query-document pairs, allowing supervision beyond documents in the relevance judgments.Source: §1 · p. 1
- CL-DRD progressively increases teacher-generated training difficulty, first teaching coarse document distinctions and then finer ordering with a LambdaRank-adapted listwise loss.The loss focuses on teacher document order rather than exact teacher scores.Source: §1 · p. 1
- The framework was applied to TAS-B and ColBERTv2 and demonstrated effectiveness on three public passage-retrieval benchmarks.The authors also report releasing model source code and parameters for research reproducibility.Source: §1 · p. 1
2 METHODOLOGY
The methodology presents dense retrieval and knowledge distillation as the setting for CL-DRD, then describes an iterative curriculum that makes teacher-derived ranking supervision progressively finer and optimizes the student with a listwise loss.
2.1.1 Dense Retrieval.
- Query and document encoders produce dense representations whose similarity gives a relevance score, and approximate nearest-neighbor search supports efficient retrieval.The similarity function is often an inner product.Source: §2.1.1 · p. 2
2.1 Background
- Dense retrieval scores query-document pairs using learned representations, while knowledge distillation trains a student from a more complex teacher’s ranking outputs.The teacher is a cross-encoder, which jointly encodes query and document and captures more interaction information than a dual encoder.Source: §2.1.2 · p. 2
2.2 The CL-DRD Framework
- CL-DRD iteratively raises the difficulty of teacher-derived training data, and its generic framework permits different implementations and stopping criteria.The stopping criterion may use held-out ranking performance or a fixed iteration count.Source: §2.2 · p. 2
2.2.1 Generating Training Data in Each Iteration for Curriculum Learning.
- At each iteration, the teacher reranks the student’s top 200 documents into pseudo-relevant, hard-negative, and remaining groups, with K increased over iterations.The resulting cross-group pairs are easier than Type 1 pairs, which require matching the teacher’s exact within-group ordering; the experiments increase K from 5 to 10 to 30.Source: §2.2.1 · pp. 2–3
2.2.2 Optimizing the Student Model through Knowledge Distillation.
- A LambdaRank-inspired listwise loss trains the student to match teacher pairwise order, weighting comparisons by the student’s document ranks.The teacher-derived pseudo-labels determine pairwise order rather than requiring exact teacher scores.Source: §2.2.2 · p. 3, §2.2.1 · p. 3
3 EXPERIMENTS
The experiments evaluate CL-DRD with single-vector TAS-B and multi-vector ColBERTv2 on three passage-retrieval benchmarks. Both model variants improve over relevant baselines, while the curriculum ablation reports generally rising performance across iterations.
Datasets
- Training uses MS MARCO passage retrieval, with evaluation on MS MARCO-Dev, TREC-DL’19, and TREC-DL’20 using MAP@1000 and each dataset’s official metric.MS MARCO training data contains approximately 8.8 million passages and 503,000 training queries.Source: §3 · p. 3
Results and Discussion
- 11% improvement in MRR@10 on MS MARCO Dev results from applying CL-DRD to TAS-B.The comparison is against the TAS-B baseline.Source: Table 1 · p. 4, §3 · p. 4
- ColBERTv2 + CL-DRD improves all reported metrics on all collections except nDCG@10 on TREC DL’19.The improvements are larger for recall-oriented metrics such as MAP@1000 and statistically significant in a majority of cases.Source: Table 1 · p. 4, §3 · p. 4
The CL-DRD models
- The experiments apply CL-DRD to single-vector TAS-B and multi-vector ColBERTv2 models, using DistilBERT as the backbone.The ColBERTv2 implementation is the authors’ own DistilBERT-based version.Source: §3 · pp. 3–4
Results and Discussion
- Multi-vector retrieval entails higher memory requirements, indexing cost, and query-processing cost.This is a trade-off noted for models representing queries and documents with multiple vectors.Source: §3 · p. 4
Ablation Study
- Across all three datasets, performance generally improves at each curriculum iteration; reverse CL scores 0.378 MRR@10 on MS MARCO Dev, 0.715 nDCG@10 on TREC DL’19, and 0.683 nDCG@10 on TREC DL’20.The reverse-curriculum MRR@10 is significantly lower than the proposed method’s result.Source: §3 · pp. 4–5, Figure 2 · p. 4
4 CONCLUSIONS AND FUTURE WORK
The paper concludes that CL-DRD distills teacher-model knowledge through iterative supervision of increasing difficulty. Its simple implementation was effective on three public passage-retrieval benchmarks, and future work will explore more sophisticated difficulty control and informative-instance selection.
Conclusions
- CL-DRD iteratively optimizes dense retrieval models using teacher-derived supervision of increasing difficulty.The framework combines curriculum learning with knowledge distillation.Source: §4 · p. 5
- A simple implementation of CL-DRD demonstrated effectiveness on three public passage-retrieval benchmarks.Source: §4 · p. 5