Source-linked AI summary
DE-RRD: A Knowledge Distillation Framework for Recommender System
SeongKu Kang, Junyoung Hwang, Wonbin Kweon, Hwanjo Yu
TL;DR
Large recommender models can be costly to run, while prediction-only distillation does not fully reveal a teacher’s knowledge. DE-RRD transfers both latent representations and prediction-based ranking knowledge; it outperforms state-of-the-art competitors and achieves teacher-comparable performance with fewer learning parameters.
Problem
Large recommender models incur high computational and memory costs and inference latency, while prediction-only distillation incompletely reveals teacher knowledge and point-wise matching limits ranking preservation.
Source: §1 · pp. 1–2Method
DE-RRD combines DE, which transfers latent teacher representations using experts and expert selection, with RRD, which distills predictions through relaxed ranking matching.
Source: §1 · p. 2Results
DE-RRD considerably outperforms state-of-the-art competitors and achieves comparable performance to the teacher with fewer learning parameters than all competitors.
Source: §1 · p. 2Takeaways & Limitations
Smaller recommender models require less computation and memory and can achieve lower latency, with larger online inference-efficiency gains for complex models such as NeuMF.
Source: §5.2 · p. 8Prediction-only methods incompletely reveal teacher knowledge, and point-wise transfer does not consider multiple items simultaneously when preserving ranking orders.
Source: §1 · pp. 1–2
Abstract
from arXiv · showhide
Recent recommender systems have started to employ knowledge distillation, which is a model compression technique distilling knowledge from a cumbersome model (teacher) to a compact model (student), to reduce inference latency while maintaining performance. The state-of-the-art methods have only focused on making the student model accurately imitate the predictions of the teacher model. They have a limitation in that the prediction results incompletely reveal the teacher's knowledge. In this paper, we propose a novel knowledge distillation framework for recommender system, called DE-RRD, which enables the student model to learn from the latent knowledge encoded in the teacher model as well as from the teacher's predictions. Concretely, DE-RRD consists of two methods: 1) Distillation Experts (DE) that directly transfers the latent knowledge from the teacher model. DE exploits "experts" and a novel expert selection strategy for effectively distilling the vast teacher's knowledge to the student with limited capacity. 2) Relaxed Ranking Distillation (RRD) that transfers the knowledge revealed from the teacher's prediction with consideration of the relaxed ranking orders among items. Our extensive experiments show that DE-RRD outperforms the state-of-the-art competitors and achieves comparable or even better performance to that of the teacher model with faster inference time.
1 INTRODUCTION
Large recommender models can perform well but incur substantial inference time and memory costs. DE-RRD addresses gaps in prediction-only distillation by transferring both teacher representations and ranking-aware prediction knowledge.
1 INTRODUCTION
- Large recommender models can improve recommendation performance, but their growing parameter counts and complex architectures increase computation, memory costs, and inference latency.These costs make large models difficult to apply on real-time platforms.Source: §1 · p. 1
- Knowledge distillation trains a compact student using teacher-generated soft labels alongside binary interaction labels, and prior recommender methods report teacher-comparable performance with lower latency.Teacher predictions can reveal hidden user-item relations and distinguish among unobserved items labeled 0.Source: §1 · p. 1
- Prediction-only methods incompletely reveal teacher knowledge, while point-wise distillation cannot account for multiple items simultaneously and can degrade ranking preservation.The teacher’s predictions may show similar preferences for two items even when the latent representations encode different aspects of them.Source: §1 · pp. 1–2
- DE-RRD combines DE, which transfers latent teacher knowledge through multiple experts and expert selection, with RRD, which relaxes ranking matching over uninteresting items.DE distinguishes knowledge according to entity correlations in the teacher representation space; RRD ignores detailed ordering among items the user would not be interested in.Source: §1 · p. 2
- DE-RRD considerably outperforms state-of-the-art competitors and achieves teacher-comparable performance with fewer learning parameters than all competitors.The introduction also reports the largest performance gain when the student has the same structure as the teacher.Source: §1 · p. 2
2 RELATED WORK
Related work covers approaches to reduce recommender inference costs and knowledge distillation, including methods that imitate teacher predictions. It identifies latent knowledge transfer and preservation of ranking order as remaining challenges.
Balancing Effectiveness and Efficiency
- Hash-based representations reduce inference costs but have constrained recommendation performance; other acceleration approaches can be model-specific or vulnerable to local optima.The reviewed approaches include tree-based structures, data compression, and approximate nearest-neighbor search.Source: §2 · p. 2
Knowledge Distillation in Recommender System
- Ranking Distillation jointly optimizes a base recommender loss and a distillation loss over teacher-ranked unobserved items, while Collaborative Distillation samples such items by rank and mimics teacher scores.RD weights items by their ranking in the student and teacher; CD weights them using teacher prediction scores.Source: §2 · p. 3
- Existing distillation methods emphasize high-ranked items, which the paper notes reveal hidden user-item patterns and have supported teacher-comparable performance with faster inference.The paper cites the high-ranked-item correlation claim to.Source: §2 · p. 3
- Existing recommender distillation methods imitate teacher predictions, but do not directly transfer its latent knowledge and may fail to preserve ranking order across items.The paper describes latent knowledge as information about users, items, and their relationships stored in the teacher.Source: §2 · p. 3
3 PROBLEM FORMULATION
The paper formulates top-N recommendation for implicit feedback as ranking unobserved items for each user. Its distillation setup trains a smaller student with teacher assistance alongside binary interaction labels.
3 PROBLEM FORMULATION
- For implicit feedback, each user's recommendation list ranks items without observed interactions, which are not necessarily negative preferences.The interaction matrix is binary, recording whether each user-item pair was observed.Source: §3 · p. 3
- Knowledge distillation trains a smaller student with help from a trained, larger teacher in addition to binary labels, aiming to maintain recommendation performance with fewer parameters.The stated goal is to improve inference efficiency without compromising effectiveness.Source: §3 · p. 3
4 DE-RRD: THE PROPOSED FRAMEWORK
DE-RRD combines transfer of the teacher’s latent representations with distillation of its predictions. Distillation Experts (DE) handles latent knowledge, while Relaxed Ranking Distillation (RRD) matches useful ranking information without requiring exact orders among uninteresting items.
4.1 Distillation Experts (DE)
- DE reconstructs the teacher’s intermediate representation from the corresponding student representation, helping the smaller student retain compressed preference information.An expert is a small feed-forward network; the student and expert are updated to reconstruct the teacher representation.Source: §4.1.1 · p. 4
- Multiple experts divide teacher knowledge into correlated representation groups, and a selection network chooses an expert to distill each entity’s knowledge.This is intended to avoid mixing weakly correlated entities, which the paper says can hinder discovery of some users’ preferences.Source: §4.1.2 · p. 4
- During training, Gumbel-Softmax relaxes expert selection toward a one-hot choice, allowing experts to specialize in information with strong correlations.The temperature is gradually decreased during training.Source: §4.1.2 · p. 5
4.2 Relaxed Ranking Distillation (RRD)
- RRD matches teacher and student recommendation lists while preserving the order of interesting items and their relative order against uninteresting items, not exact orders among the latter.It frames prediction distillation as a list-wise ranking-matching problem.Source: §4.2 · pp. 5–6
- RRD samples interesting items from near the teacher list’s top using rank-weighted probabilities, then uniformly samples uninteresting items from lower-ranked items.The number of sampled items is K interesting and L uninteresting items per user.Source: §4.2.1 · p. 6
4.3 Optimization of DE-RRD
- DE-RRD jointly optimizes the student with the base-model loss and the DE and RRD objectives in an end-to-end manner.The base model can be any existing recommender; the framework description identifies DE’s selection network and experts as its learning parameters.Source: §4.3 · p. 6, §4.2.3 · p. 6
5 EXPERIMENTS
Across 12 settings spanning two datasets, two base models, and three student sizes, DE-RRD generally outperforms existing distillation methods. The analyses also support expert selection for DE, relaxed ranking for RRD, and tuning their loss weights and sampling sizes.
5.1 Experimental Setup
- DE-RRD is evaluated in 12 settings formed by two real-world datasets, two base models, and three student sizes, using leave-one-out testing with 499 sampled unobserved items per user.Sampling of test/validation items and unobserved items is repeated five times, and results are averaged.Source: §5 · p. 6, §5.1 · p. 7
5.2 Performance Comparison
- DE-RRD generally outperforms state-of-the-art methods across student sizes and achieves teacher-comparable performance with 10–50% of the teacher’s learning parameters.The paper also reports potential online inference-efficiency gains, particularly for deep recommenders and larger real-time systems.Source: §5.2 · pp. 7–8, Figure 4 · p. 8, Table 3 · p. 9
5.3 Design Choice Analysis
- DE’s expert selection performs best among its ablations, while experts specialize in user groups with similar preferences during training.The analysis contrasts selection with attention and single-expert variants, and reports that selection improves performance in more user groups.Source: §5.3 · p. 9, Table 4 · p. 9
- RRD’s relaxed ranking addresses weaknesses of list-wise alternatives that either rank uninteresting items in detail or omit their relative order to interesting items.The paper reports adverse ranking effects for both alternatives and says RRD better focuses on interesting items while considering their relative orders with uninteresting items.Source: §5.3 · p. 9
5.4 Hyperparameter Analysis
- Best performance occurs near 10–20 experts and K=30–40, with DE and RRD loss magnitudes tuned separately for BPR and NeuMF.The reported best loss magnitudes are approximately 20% for DE with BPR and 2–5% with NeuMF; for RRD they are 7–10% with BPR and 1000% with NeuMF.Source: §5.4 · p. 10, Table 5 · p. 10
6 CONCLUSION
DE-RRD combines distillation of teacher predictions with transfer of latent knowledge through Distillation Experts and Relaxed Ranking Distillation. The paper reports that it significantly outperforms state-of-the-art competitors.
6 CONCLUSION
- DE-RRD combines teacher predictions with latent knowledge transferred from the teacher model.The conclusion presents DE and RRD as the two methods composing the framework.Source: §6 · p. 10
- DE uses experts and expert selection to distill latent collaborative-filtering knowledge into the student.DE transfers knowledge from the teacher’s representation space.Source: §6 · p. 10
- RRD transfers prediction-derived knowledge while accounting for relaxed ranking orders and focusing on interesting items.Source: §6 · p. 10