Source-linked AI summary
Ranking Distillation: Learning Compact Ranking Models With High Performance for Recommender System
Jiaxi Tang, Ke Wang
TL;DR
Ranking systems need strong effectiveness and fast online inference, yet powerful ranking models are parameter-heavy. The paper introduces ranking distillation, training a compact student from labeled data and a teacher’s top-K rankings. Across recommendation experiments, the student uses less than half the teacher’s parameters while achieving similar or sometimes better ranking performance.
Problem
Ranking models must combine the effectiveness of large models with the online inference efficiency of small models, while standard knowledge distillation is not directly suited to ranking problems.
Method
Ranking distillation trains a smaller student with usual training data plus top-K documents and supervision generated by a well-trained teacher.
Results
Student models trained with RD achieve similar or better ranking performance than teacher models while using less than half as many model parameters.
Takeaways & Limitations
RD generates compact ranking models with little or no effectiveness compromise, and its advantages are independent of the teacher and student model choices.
Takeaways & Limitations
Pair-wise distillation produced unstable training and sometimes failed to converge, whereas the weighted point-wise distillation loss avoided this issue.
Abstract
from arXiv · showhide
We propose a novel way to train ranking models, such as recommender systems, that are both effective and efficient. Knowledge distillation (KD) was shown to be successful in image recognition to achieve both effectiveness and efficiency. We propose a KD technique for learning to rank problems, called \emph{ranking distillation (RD)}. Specifically, we train a smaller student model to learn to rank documents/items from both the training data and the supervision of a larger teacher model. The student model achieves a similar ranking performance to that of the large teacher model, but its smaller model size makes the online inference more efficient. RD is flexible because it is orthogonal to the choices of ranking models for the teacher and student. We address the challenges of RD for ranking problems. The experiments on public data sets and state-of-the-art recommendation models showed that RD achieves its design purposes: the student model learnt with RD has a model size less than half of the teacher model while achieving a ranking performance similar to the teacher model and much better than the student model learnt without RD.
1 INTRODUCTION
Ranking models must balance ranking effectiveness with online inference efficiency, but larger neural and latent-factor models increase complexity and parameter counts. The paper proposes ranking distillation (RD), which transfers teacher supervision to compact student ranking models for recommendation and information retrieval.
- Ranking-model effectiveness depends on relevance prediction, while efficiency determines online response speed.
- Larger latent-factor and neural ranking models capture complex query-document interactions more effectively but require many parameters.
- Existing efficiency methods can reduce effectiveness through model constraints, pruning, indexing, or staged retrieval.
- Ranking distillation trains a smaller student using ground-truth data plus top-K documents and supervision generated by a larger teacher.
- RD differs from classification knowledge distillation by generating additional training data and labels from unlabeled documents.
- Experiments report similar or better ranking performance with less than half the teacher’s model parameters, while RD remains orthogonal to teacher and student model choices.
2 BACKGROUNDS
Learning to rank predicts document or item relevance and orders results near the top positions, where labeled query-document pairs are relatively scarce. The background motivates compact models by connecting higher model capacity with better MAP and higher online inference cost.
- Learning to rank retrieves documents most relevant to each query using predicted relevance scores.
- Labeled query-document pairs are much fewer than pairs with unknown labels, so ranking models must predict relevance for future queries.
- Ranking losses are categorized as point-wise, pair-wise, and list-wise; this work focuses on the more widely adopted point-wise and pair-wise losses.
- Pair-wise loss models partial-order information using document pairs whose relevance labels establish an ordering.
- Neural and latent-factor ranking models are effective but commonly require many parameters, creating online inference efficiency concerns.
- Larger models can capture complex query-document interactions and achieve higher MAP when they do not overfit the data.
3 RANKING DISTILLATION
Ranking distillation trains a smaller student ranking model using both labeled ground-truth rankings and a teacher’s top-K rankings of unlabeled documents. It combines flexible weighting schemes to focus distillation on informative teacher-ranked documents while targeting teacher-level effectiveness with lower inference cost.
- Overview: RD trains a smaller student model from ground-truth rankings and a teacher-generated top-K ranking of unlabeled documents.The student minimizes a ranking loss on labeled data together with a distillation loss on the teacher’s unlabeled-document ranking.
- Incorporating Distillation Loss: Teacher-ranked top-K unlabeled documents provide extra supervision because highly ranked items are expected to correlate strongly with the query.The method treats these documents as positive instances in the point-wise distillation loss, without adding negative instances.
- Weighting by Position Importance: The distillation loss uses weighted point-wise supervision, with weights controlling how much each teacher-ranked position contributes.The paper considers equal weighting, inverse-rank weighting, and parametrized geometric weighting; the latter emphasizes top positions when λ is small and approaches uniform weighting when λ is large.
- Weighting by Ranking Discrepancy: Dynamic discrepancy weighting concentrates training on teacher-ranked documents whose student-predicted ranks differ substantially from the teacher’s ranks.Student ranks are estimated by sequentially sampling unlabeled documents, and the discrepancy weight is near zero for well-predicted documents and near one for poorly predicted documents.
- Hybrid Weighting Scheme: Hybrid weighting combines position importance with ranking discrepancy, using position weighting alone during warm-up before switching to the hybrid scheme.The warm-up addresses inaccurate student-rank estimates early in training; the warm-up length is selected on the validation set.
- Discussion: Pair-wise distillation was unstable because its partial-order objective produced both upward and downward gradients, whereas the weighted point-wise loss contains only upward gradients.The authors report that pair-wise distillation sometimes failed to converge, while the point-wise formulation avoided this issue.
4 EXPERIMENTAL STUDIES
The experiments evaluate ranking distillation on sequential recommendation datasets using Fossil and Caser, showing that compact RD students retain teacher-level ranking performance while improving efficiency.
- 4.1 Experimental Setup: Experiments use sequential recommendation on Gowalla and Foursquare, extracting five-item histories and the immediately following item as ground truth.The study compares sequential recommenders with non-sequential baselines and uses held-out validation and test actions.
- 4.2 Overall Results: Ranking-distilled Fossil-RD and Caser-RD significantly outperform their student-only counterparts without significant degradation from the teacher models.Table 2 uses a one-tail t-test with significance level 0.05.
- 4.1 Experimental Setup: Fossil and Caser serve as sequential recommendation models, with large teacher variants trained using as many parameters as possible.Fossil uses pair-wise ranking loss, whereas Caser uses point-wise ranking loss.
- 4.2 Overall Results: Nearly half-sized RD models achieve similar ranking effectiveness while reducing online inference time similarly to their teacher models.Online efficiency is measured by parameter count and wall time for generating each user’s recommendation list.
- 4.3 Effects of Model Size and Distillation Loss: Caser-RD reaches teacher-like MAP at a medium model size of about 50% of the teacher, while smaller distilled models are more robust to overfitting.The teacher tends to overfit after about 60 iterations on sparse recommendation datasets, whereas Caser-RD and Caser-S-RD remain more robust.
- 4.3 Effects of Model Size and Distillation Loss: Balancing ranking and distillation losses is necessary: the best α is around 0.5 on Gowalla and 0.3 on Foursquare, while removing either loss performs poorly.Too much concentration on distillation loss produces bad performance on Foursquare.
- 4.4 Effects of Weighting Schemes: The hybrid weighting scheme combining position importance and ranking discrepancy achieves the best results on both datasets.Equal weighting performs worst, while position-only weighting substantially improves performance.
5 RELATED WORK
The paper distinguishes ranking distillation from related knowledge- and transfer-learning approaches by emphasizing ranking-specific supervision and same-domain teacher–student training.
- Knowledge Distillation: Applying knowledge distillation to ranking is nontrivial because ranking predicts relative document or item order rather than classification labels.Ranking also involves supervision over many documents or items for each query, unlike classification with few labels.
- Transfer Learning: Unlike cross-domain transfer learning, the teacher and student in this work are learned from the same domain while serving different model-size roles.The compact student is intended to improve online inference efficiency in addition to preserving recommendation performance.
6 CONCLUSION
The conclusion reports that ranking distillation produces compact ranking models with little or no effectiveness loss, across model choices and real-world recommendation datasets.
- 6 CONCLUSION: Ranking distillation trains a smaller student with ground-truth data plus teacher supervision to rank unlabeled documents.The paper addresses formulation, teacher-supervision representation, and the balance between training data and teacher-generated data.
- 6 CONCLUSION: The evaluation supports compact models with better online inference efficiency and ranking performance comparable to the teacher.The reported advantages are independent of the choices of teacher and student ranking models.