PaperlayerView exact revision on arXiv

Source-linked AI summary

Distillation from Heterogeneous Models for Top-K Recommendation

SeongKu Kang, Wonbin Kweon, Dongha Lee, Jianxun Lian, Xing Xie, Hwanjo Yu

arXiv:2303.01130v1cs.IRcs.AI

TL;DR

Heterogeneous recommender ensembles can improve accuracy but incur high inference costs, and distilling their knowledge is particularly difficult. HetComp uses teacher training trajectories to provide progressively harder ranking supervision through dynamic knowledge construction and adaptive transfer. Experiments show improved distillation efficacy and student generalization.

  • Problem

    Heterogeneous recommender ensembles can improve accuracy but make inference many times more costly than a single model, while distillation from heterogeneous teachers is difficult and understudied.

    Source: §1 · p. 1

  • Method

    HetComp uses teacher trajectories to construct progressively difficult ranking supervision for the student and adaptively transfers finer-grained ranking information.

    Source: §1 · p. 2

  • Results

    HetComp significantly improves distillation efficacy and student generalization in the reported experiments.

    Source: §6 · p. 8

  • Takeaways & Limitations

    HetComp compresses heterogeneous ensemble knowledge into a lightweight student while aiming to reduce inference costs and retain high accuracy.

    Source: §1 · p. 2, §6 · p. 8

    Knowledge selection is designed to remain model-agnostic, avoiding model-specific training-loss statistics and additional held-out performance metrics.

    Source: §4.1 · p. 4

Abstract

from arXiv · show

Recent recommender systems have shown remarkable performance by using an ensemble of heterogeneous models. However, it is exceedingly costly because it requires resources and inference latency proportional to the number of models, which remains the bottleneck for production. Our work aims to transfer the ensemble knowledge of heterogeneous teachers to a lightweight student model using knowledge distillation (KD), to reduce the huge inference costs while retaining high accuracy. Through an empirical study, we find that the efficacy of distillation severely drops when transferring knowledge from heterogeneous teachers. Nevertheless, we show that an important signal to ease the difficulty can be obtained from the teacher's training trajectory. This paper proposes a new KD framework, named HetComp, that guides the student model by transferring easy-to-hard sequences of knowledge generated from the teachers' trajectories. To provide guidance according to the student's learning state, HetComp uses dynamic knowledge construction to provide progressively difficult ranking knowledge and adaptive knowledge transfer to gradually transfer finer-grained ranking information. Our comprehensive experiments show that HetComp significantly improves the distillation quality and the generalization of the student model.

1 INTRODUCTION

Heterogeneous recommender ensembles can improve ranking accuracy but impose substantial inference costs, and transferring their knowledge to a student is difficult. HetComp uses teacher training trajectories to guide progressively harder ranking supervision, with dynamic knowledge construction and adaptive transfer.

1 INTRODUCTION

  • Heterogeneous model ensembles improve accuracy by combining distinct inductive biases, but inference can cost many times more than using a single model.Source: §1 · p. 1
  • Prior ranking distillation largely targets homogeneous teachers, while heterogeneous transfer remains understudied and can sharply reduce distillation efficacy.Source: §1 · p. 1
  • Earlier teacher states yield lower student–teacher ranking discrepancies than converged teachers, whose later predictions contain more diversified and individualized rankings.Source: §1 · pp. 1–2
  • HetComp uses teacher trajectories for easy-to-hard supervision, dynamically constructing ranking knowledge for the student’s learning state and adaptively shifting from overall relations to fine-grained item orders.Source: §1 · p. 2, Figure 1 · p. 2
  • HetComp also introduces a transfer strategy that uses knowledge from both observed and unobserved user-item interactions.Source: §1 · p. 2
  • Experiments show that HetComp improves distillation efficacy and student generalization while reducing ensemble computational burdens and retaining high accuracy.Source: §1 · p. 2

2 RELATED WORK

Prior work spans multi-teacher distillation, ranking-oriented supervision, and easy-to-hard learning, but heterogeneous teacher distillation tailored to recommendation remains underexplored.

2 RELATED WORK

  • Multi-teacher KD exploits complementary heterogeneous models [45] [51], while point-wise methods transfer per-pair importance [4] [22] [40] and ranking matching preserves teacher permutations [5] [15] [25] [41].Source: §2 · p. 2
  • Easy-to-hard learning includes curriculum learning [2, 43] and self-paced learning, and prior KD work applies teacher optimization routes or progressively finer document pairs [3] [14] [39].Source: §2 · p. 3
  • Prior recommendation KD mainly uses homogeneous teachers, while existing easy-to-hard KD targets classification or relies on domain-specific features, motivating ranking-specific heterogeneous distillation.Source: §2 · p. 3

3 PRELIMINARIES

This section formulates model-agnostic compression of heterogeneous recommender ensembles and reviews ranking-matching distillation. Its analysis finds heterogeneous-teacher knowledge harder to transfer, while teacher trajectories reveal progressively harder ranking supervision.

3.1 Problem Formulation

  • The task is to compress cumbersome teachers into a lower-cost student for real-time or resource-constrained use, with model-agnostic teacher and student choices.Source: §3.1 · p. 3

3.2 Ranking Matching Distillation

  • Ranking-matching distillation trains the student to emulate teacher permutations, matching fine-grained orders in selected top-ranked items while pushing sampled remaining items below them.The within-group ordering of the sampled remaining items need not be preserved.Source: §3.2 · p. 3

3.3 Study on Ranking Knowledge Distillation

  • The study measures student-teacher top-K permutation mismatch with discrepancy D@K, where zero means the student perfectly preserves the teacher’s top-K ranking.Source: §3.3 · p. 3
  • Compressing heterogeneous teachers produces markedly larger discrepancy than compressing a homogeneous teacher or homogeneous ensemble, indicating degraded distillation efficacy.Table 1 compares discrepancy after distillation across the supervision settings.Source: §3.3 · pp. 3–4, Table 1 · p. 4
  • Heterogeneous teachers consistently yield greater discrepancy than MF teachers, and teacher accuracy does not predict discrepancy: AE exceeds MF on CiteULike despite comparable accuracy.Source: §3.3 · p. 4
  • Across teacher types, discrepancy increases from checkpoints E1 through E4; later predictions contain more diverse, individualized rankings, motivating trajectories as a natural student curriculum.The four checkpoints correspond to 25%, 50%, 75%, and 100% of converged training epochs.Source: §3.3 · p. 4, Figure 2 · p. 4

4 METHODOLOGY

HetComp distills heterogeneous teachers through an easy-to-hard curriculum that adapts both the target ranking knowledge and the transfer objective to the student’s learning state. It separately transfers observed-item rankings and top-ranked unobserved items, then updates the constructed knowledge periodically during training.

4.1 Dynamic Knowledge Construction

  • The construction reflects differences in learning difficulty across teachers and users, while avoiding model-specific training loss and sparse per-user held-out performance measures.These choices support model-agnostic selection and avoid relying on additional held-out interactions for each user.Source: §4.1 · p. 4
  • HetComp selects progressively later permutations from each teacher’s trajectory, advancing when the student’s discrepancy to the next state falls sufficiently.The transition threshold α controls speed; a lower value moves faster. Discrepancy supports a model-agnostic curriculum.Source: §4.1 · pp. 4–5

4.2 Adaptive Knowledge Transfer

  • HetComp first transfers overall relations in the target permutation, then trains on fine-grained ranking orders after all teachers reach their final states.The overall-relation objective ranks positive items above negatives without imposing an order among the positive items.Source: §4.2 · p. 5, §4.1 · p. 5
  • It transfers observed-item rankings separately from top-ranked unobserved items, avoiding the whole-item-list strategy that can overly penalize relevant but not-yet-observed items.The loss does not force top-ranked unobserved items below observed items, allowing highly relevant unobserved items to mix near the top.Source: §4.2 · p. 5

4.3 The Overall Training Process

  • Knowledge construction is personalized and runs every 10 epochs until the student has learned from the final converged teachers; observed-item rankings are generated once before training.The update period avoids changing target permutations every epoch, which the authors describe as time-consuming and unnecessary.Source: §4.3 · p. 5

5 EXPERIMENTS

HetComp outperforms the evaluated distillation baselines across datasets and student models, while reducing discrepancy and improving generalization in the tested setting. Analyses show that teacher trajectories expose knowledge from coarse group patterns to more diverse, personalized rankings, and that dynamic knowledge construction adapts this progression to student learning.

5 EXPERIMENTS

  • Experiments use three real-world datasets, a 60/20/20 interaction split, top-K Recall and NDCG, and student models with roughly one-tenth the teacher parameters.The datasets are Amazon-music, CiteULike, and Foursquare; student and teacher dimensions are 6 and 64, respectively.Source: §5 · p. 6

5.1 Distillation Effects Comparison

  • Ranking-transfer methods outperform point-wise RD; MTD trails RRD, CL-DRD varies across settings, and DCD consistently exceeds RRD.The authors attribute MTD’s limited performance to overfitting of user-wise importance under sparse interactions; CL-DRD uses predefined difficulty rules without accounting for student state.Source: §5.1 · p. 6, §5 · p. 6
  • HetComp significantly outperforms all baselines across datasets and student models, with larger gains at K=10 than K=50 and lower discrepancy than DCD.Table 2 compares recommendation performance and Table 4 compares discrepancy.Source: §5.1 · p. 6, Table 2 · p. 7, Table 4 · p. 7
  • In the limited-teacher-knowledge setup, HetComp significantly improves over DCD in every case and the student better captures hidden preferences.Teachers train on only part of one user group’s interactions, while the student trains with all training data and incomplete teacher guidance.Source: §5.1 · p. 6, Table 3 · p. 7
  • HetComp reduces the high inference costs of a multi-model ensemble by distilling its knowledge into a compact student.Table 5 presents the accuracy-efficiency trade-off and reports average wall time for generating each user’s recommendation.Source: §5.1 · p. 6, Table 5 · p. 7

5.2 Study of HetComp

  • Across teacher training states, recommendations become more diverse and include increasing proportions of unpopular items; early states capture group-level patterns better than user-level preferences.The study reports relative ratios for unique and unpopular items in top-50 lists and evaluates group-level and user-level preference with R@50.Source: Table 6 · p. 8, §5.2 · p. 8, Figure 3 · p. 8
  • Dynamic knowledge construction moves toward harder teacher knowledge as the student learns, with user-specific selections reflecting different learning difficulties.On MF with CiteULike, DKC provides a more efficient curriculum than global-selection and fixed-interval variants; DKC-e increases discrepancy around epoch 100.Source: Figure 4 · p. 8, §5.2 · p. 8

6 CONCLUSION

The conclusion presents HetComp as a way to compress heterogeneous recommendation ensembles into a lightweight model while reducing inference costs and retaining high accuracy. It attributes its approach to easy-to-hard transfer from teacher trajectories and reports improved distillation efficacy and student generalization.

6 CONCLUSION

  • HetComp compresses heterogeneous recommendation ensembles into a lightweight model to reduce inference costs while retaining high accuracy.The conclusion presents the framework as compatible with existing models and a potential solution to the recommender-system accuracy-efficiency trade-off.Source: §6 · p. 8
  • The conclusion identifies heterogeneous-teacher distillation as challenging and teacher training trajectories as a signal that can ease the learning difficulty.HetComp uses dynamic knowledge construction for progressively difficult ranking knowledge and adaptive knowledge transfer for finer-grained ranking information.Source: §6 · p. 8
  • Extensive experiments show that HetComp improves distillation efficacy and student-model generalization.Source: §6 · p. 8

A.1 Experiment Setup

The experimental setup uses three public recommendation datasets and averages results over five independent runs. It specifies dataset preprocessing, model sizes, evaluation metrics, computing hardware, and selected implementation settings.

A.1 Experiment Setup

  • The source code for HetComp is publicly available through the authors’ GitHub repository.Source: §A.1 · p. 10
  • Experiments use Amazon-music, CiteULike, and Foursquare, with CiteULike and Foursquare preprocessing following prior work and Amazon-music receiving 10-core filtering.The datasets are publicly accessible and widely used in previous work.Source: §A.1 · p. 10, Table 7 · p. 10
  • The experiments average five independent runs and use PyTorch with CUDA on an RTX A5000 GPU and Xeon Gold 6226R CPU.Baseline public implementations are used, with sampling of top-ranked unobserved items removed for ranking-matching methods.Source: §A.1 · p. 10
  • Dynamic knowledge construction uses four teacher checkpoints at 25%, 50%, 75%, and 100% of convergence, with D@50, p=10, α=1.05, and |P−|=50.The α value is annealed by multiplying by 0.995 every p epochs; baseline-specific hyperparameters are tuned within ranges suggested by their papers.Source: §A.1 · p. 10

A.2 Study on the Ensemble of Teacher Models

The six-model heterogeneous ensemble outperforms both the best individual teacher and an ensemble of identical models. Its ranking knowledge combines teacher predictions using item rank and prediction consistency.

A.2 Study on the Ensemble of Teacher Models

  • The six teachers represent matrix factorization, metric learning, DNN, GNN, user-side autoencoder, and item-side autoencoder models.The paper identifies these as representative models with different architectures and learning objectives; the item-side autoencoder captures complementary aspects to its user-side counterpart.Source: §A.2 · p. 10
  • The six-model heterogeneous Ensemble consistently outperforms Ensemble-id and the best teacher, with all six models outperforming every five-model configuration.Ensemble-id uses six identical models with different initializations; the study reports lower prediction correlations for Ensemble than Ensemble-id on all three datasets.Source: §A.2 · p. 10, Table 8 · p. 11
  • The ensemble function selectively consolidates teacher rankings using importance that favors higher-ranked items and consistent predictions.Consistency is based on prediction variance over the latest five epochs and is precomputed before distillation without additional HetComp costs.Source: §A.2 · pp. 10–11

A.3 Offline Training Cost of HetComp

HetComp adds offline training cost, concentrated mostly in knowledge construction, while targeting lower online inference costs. Its trajectory storage is limited to top-ranked unobserved items and observed items.

A.3 Offline Training Cost of HetComp

  • Per teacher and user, HetComp stores (K × E) + |P+_u| items, using top-ranked unobserved-item permutations and observed items.K and |P+_u| are usually small in many recommendation scenarios, and satisfactory results are reported with E around 3 when the epochs are well distributed.Source: §A.3 · p. 11
  • HetComp incurs additional training computations versus RRD, but its knowledge construction occurs mostly early in student training and yields lower average epoch time than DCD.The comparison uses MF as the student; similar results are also observed with other base models, and HetComp and DCD show similar convergence behavior.Source: Table 9 · p. 11, §A.3 · p. 11

A.4 Supplementary Experiment Results

Supplementary experiments report gains across homogeneous, cross-model, and heterogeneous-ensemble teacher settings. Ablations support separating observed and unobserved item rankings, while a hyperparameter study identifies a favorable transition-speed range.

A.4 Supplementary Experiment Results

  • Ablations show that DKC and ADO improve the student, using observed-item knowledge helps, and separating observed from unobserved rankings is superior to transferring the whole permutation.Penalizing all unobserved items to rank below observed items is not effective.Source: §A.4 · p. 11, Table 10 · p. 11
  • Around 1.03–1.05 for α gives the best observed recommendation performance on both datasets.α controls transition speed in dynamic knowledge construction; Figure 5 reports R@10 comparisons across α and E.Source: §A.4 · p. 11, Figure 5 · p. 11
  • HetComp achieves the best recommendation performance in all tested teacher settings, from homogeneous and cross-model distillation to heterogeneous ensembles.The comparison uses MF as the student on CiteULike.Source: Table 11 · p. 11, §A.4 · p. 11

Related papers

Loading 2303.01130v1…