Source-linked AI summary
Beyond Hard Negatives: The Importance of Score Distribution in Knowledge Distillation for Dense Retrieval
Youngjoon Jang, Seongtae Hong, Hyeonseok Moon, Heuiseok Lim
TL;DR
The paper asks whether distillation data should represent the teacher’s full score distribution rather than mainly emphasize hard negatives. It introduces Stratified Sampling, which uniformly covers that distribution, and finds robust performance across in-domain and out-of-domain settings, objectives, backbones, and candidate counts.
Problem
Existing distillation-data composition has focused primarily on difficult-example mining, leaving whether diverse regions of the teacher’s score distribution are sufficiently sampled less explored.
Method
Stratified Sampling places quantile anchors across candidate scores and selects nearby documents, using controlled candidate pools and fixed training pipelines to isolate distributional effects.
Results
Stratified Sampling consistently outperforms Top-K and Random methods across in-domain and out-of-domain environments while remaining robust across sample sizes and objective functions.
Takeaways & Limitations
Preserving the teacher-defined score distribution is more critical for generalization than intensively training only on hard negatives.
Abstract
from arXiv · showhide
Transferring knowledge from a cross-encoder teacher via Knowledge Distillation (KD) has become a standard paradigm for training retrieval models. While existing studies have largely focused on mining hard negatives to improve discrimination, the systematic composition of training data and the resulting teacher score distribution have received relatively less attention. In this work, we highlight that focusing solely on hard negatives prevents the student from learning the comprehensive preference structure of the teacher, potentially hampering generalization. To effectively emulate the teacher score distribution, we propose a Stratified Sampling strategy that uniformly covers the entire score spectrum. Experiments on in-domain and out-of-domain benchmarks confirm that Stratified Sampling, which preserves the variance and entropy of teacher scores, serves as a robust baseline, significantly outperforming top-K and random sampling in diverse settings. These findings suggest that the essence of distillation lies in preserving the diverse range of relative scores perceived by the teacher.
1 INTRODUCTION
The paper reframes distillation-data design around coverage of the teacher’s score distribution rather than relying mainly on hard-negative mining. It proposes Stratified Sampling and reports robust performance across models, objectives, and domains.
- Motivation: Heuristic top-K and random sampling may expose students to only a limited or biased segment of the teacher’s score distribution.The paper links this limitation to difficulty learning decision boundaries across varying difficulty levels.
- Approach: Stratified Sampling uniformly covers the teacher’s score distribution by selecting candidates nearest to quantile anchors.A fixed candidate pool combines top-retrieved and random documents to isolate score-distribution effects from dynamic mining.
- Experimental design: The experiments decouple model and objective-function effects through a two-stage process using Contrastive Learning followed by KL-Divergence or MarginMSE training.This design evaluates data composition within a controlled training pipeline.
- Results: Stratified Sampling consistently achieves robust performance across base models and objective functions, including varying candidate counts.The reported objectives are KL-Divergence and MarginMSE.
- Implication: Stratified Sampling improves both in-domain and out-of-domain performance without complex curriculum scheduling.The paper presents it as a practical alternative to heuristic data composition and a criterion for future distillation-data design.
2 RELATED WORKS
Prior work improves distillation through hard-negative selection, denoising, geometric constraints, adaptive examples, and curriculum learning. This paper instead emphasizes static training-data composition through a parameter-free strategy that represents the teacher’s score distribution.
- Distillation methods: Knowledge Distillation commonly transfers continuous cross-encoder scores through KL-divergence or regression objectives such as MarginMSE.These methods address the performance gap between efficient bi-encoders and cross-encoder rerankers.
- Candidate selection: Negative selection is treated as a critical performance factor, with prior approaches including hard-negative mining, denoising, in-batch, BM25, and random sampling.These strategies construct candidate sets using differing heuristics.
- Positioning: Unlike methods using geometric constraints, adaptive examples, or curriculum learning, this work focuses on static data composition.The proposed alternative avoids dynamic adjustments while targeting score-distribution representativeness.
- Positioning: Stratified Sampling is presented as a simpler, parameter-free alternative that statistically ensures representativeness of the teacher’s score distribution.The comparison is framed against advanced strategies that may require complex scheduling, auxiliary losses, or dynamic sampling pipelines.
3 EXPERIMENTAL SETUP
The experiments isolate candidate sampling as the primary variable while controlling student backbones, distillation objectives, and the candidate pool. They compare six strategies using normalized teacher scores within a two-stage training and evaluation setup.
- Experimental Design: Three student backbones are evaluated with KLDiv and MarginMSE, while the primary comparison remains candidate document composition.The backbones are bert-base-uncased, distilbert-base-uncased, and co-condenser-marco.
- Data Construction and Sampling: The controlled pool contains 200 negatives per query: 100 retrieved by Qwen3-Embedding-8B and 100 sampled uniformly from the remaining corpus.This design broadens the difficulty range while isolating score-distribution effects from a specific first-stage retriever.
- Data Construction and Sampling: Experiments compare retriever-top, reranker-top, low, mid, random, and stratified sampling strategies for selecting K candidate documents.Sampling decisions use query-level min–max normalized teacher scores over the candidate pool.
- Data Construction and Sampling: Stratified Sampling places evenly spaced quantile anchors across teacher scores and selects distinct candidates closest to each anchor.The resulting samples mimic the teacher’s score-distribution structure without random-sampling variance.
- Training: Training uses two stages: contrastive learning adapts pretrained encoders, followed by distillation on reranker-scored triplets using KL-Divergence or MarginMSE.The protocol is designed to decouple model and objective-function effects.
- Evaluation: Evaluation covers MS MARCO Dev and TREC DL 19 in-domain, plus nDCG@10 across 13 BEIR datasets out-of-domain.Reported in-domain metrics are MRR@10 and Recall@1000 for MS MARCO Dev and nDCG@10 for TREC DL 19.
4 EXPERIMENTAL RESULTS AND ANALYSIS
Experiments show that sampling strategies preserving broad, uniform coverage of the teacher’s score distribution deliver stronger and more robust distillation performance than strategies concentrated at one distributional extreme. Stratified Sampling remains robust across candidate counts, backbones, objectives, and both in-domain and out-of-domain evaluation.
- Main Results: Random and stratified sampling rank consistently near the top across three backbones and both KLDiv and MarginMSE objectives, while distribution-biased strategies degrade performance.On BEIR-13, stratified achieves near-top performance across all backbones and objectives, whereas reranker-top is unstable out of domain.
- Main Results: MarginMSE can cause training collapse under inappropriate sampling, with co-condenser reranker-top reaching MRR@10=.006 versus .307 for stratified sampling.KL-divergence preserves relative strategy rankings more stably, while regression-based objectives are more sensitive to distributional bias and negative-set noise.
- Diversity Statistics: Stratified Sampling records Cov= 0.990, Ent= 1.523, and Std= 0.359, the highest values across the score-distribution diversity metrics.Coverage is the score range, Entropy is Shannon entropy over eight equal-width bins, and Standard Deviation is the score standard deviation.
- Diversity Statistics: The ranking of diversity metrics largely aligns with model-performance ranking, suggesting that preserving the teacher’s score distribution matters more than intensive hard-negative training alone.Top or Standard strategies show low Entropy and Std, indicating fragmentary observation of the teacher’s preferences.
- Robustness of Stratified Sampling: Stratified Sampling is robust as K varies and generally outperforms other strategies, with the only exception being Random at K= 4 on BEIR.As K increases, stratified fills gaps across the score range; the best reported setting uses MarginMSE with K= 16, achieving TREC DL 19 nDCG@10=0.531 and BEIR nDCG@10=0.343.
5 CONCLUSION
The paper studies how teacher-score distribution affects dense-retrieval distillation and proposes Stratified Sampling to cover that distribution uniformly. Across in-domain and out-of-domain benchmarks, it consistently outperforms Top-K and Random methods while remaining robust across sample sizes and objectives.
- Conclusion: Stratified Sampling uniformly covers the teacher’s score spectrum and consistently outperforms Top-K and Random methods in in-domain and out-of-domain environments.It is presented as a robust, parameter-free baseline across variations in sample size K and objective function.