Ensemble Modeling with Contrastive Knowledge Distillation for Sequential Recommendation

1 INTRODUCTION

• Most existing work improves a single sequence encoder, while training multiple encoders as an ensemble is less explored in sequential recommendation.Sequential encoders use deep neural networks to represent users and items from interaction histories.Source: §1 · pp. 1–2
• Different network initializations can yield diverse predictions, which can improve overall accuracy when combined.The paper illustrates this with parallel networks that produce different candidate-item lists.Source: §1 · p. 2, Figure 1 · p. 2
• Independent parallel networks with averaged logits can improve performance, but without knowledge transfer their gains may be trivial.Collaborative distillation lets each network acquire information from peers while sharing its own knowledge [22] [29] [44].Source: §1 · p. 2
• EMKD uses parallel bidirectional Transformers and transfers knowledge at representation and logits levels through contrastive learning and collaborative distillation.Masked item prediction is the main task, with attribute prediction as an auxiliary task.Source: §1 · p. 2
• EMKD significantly outperforms state-of-the-art methods on public benchmark datasets.The authors also report that the ensemble approach can be adapted to other sequential recommenders and causes only a minor reduction in training efficiency.Source: §1 · p. 2

2.1 Sequential Recommendation

• Sequential recommenders use Markov models, RNNs, CNNs, and Transformers; attribute-aware methods also incorporate item context.SASRec [18] uses a unidirectional Transformer, BERT4Rec [31] a bidirectional Transformer, and attribute-aware examples include FDSA, S3-Rec, and MMInfoRec [25].Source: §2.1 · pp. 2–3

2.2 Ensemble Modeling and Knowledge Distillation

• Prior ensemble-distillation methods address computer-vision model compression, leaving efficient knowledge transfer for ensemble sequential models open.Examples include collaborative distillation in DML [44] and an online ensemble in ONE [22].Source: §2.2 · p. 3

2.3 Contrastive Learning

• Sequential recommendation uses contrastive learning for sparsity and representation challenges; EMKD instead uses it for knowledge transfer between ensemble networks.Prior examples include CL4SRec, DuoRec [26], and CML [39].Source: §2.3 · p. 3

3 PROPOSED FRAMEWORK

• EMKD averages the logits of its sequence encoders to make recommendations at inference.The framework trains an ensemble of sequence encoders.Source: §3 · p. 3
• Masked item prediction is the primary recommendation task, while attribute prediction is an auxiliary task for fusing attribute information.Source: §3 · p. 3
• Contrastive knowledge distillation transfers knowledge between parallel networks from both representations and logits.Source: §3 · p. 3

3.1 Problem Statement

• Each user has a chronological interaction sequence, and each item in that sequence has an associated set of attributes.Source: §3.1 · p. 3
• The task predicts the user’s next item by generating a probability distribution over all candidate items at time T+1.Source: §3.1 · p. 3

3.2 Ensemble Sequence Encoder

• The embedding layer combines item and positional embeddings, and the bidirectional Transformer maps them to hidden sequence representations.The input representation is E(s_u) = [v_1 + p_1, ..., v_T + p_T].Source: §3.2 · p. 3
• Each of the N parallel encoders is a bidirectional Transformer initialized with a different random seed to generate different prediction results.The Transformer combines item and positional embeddings before producing hidden representations for the sequence.Source: §3.2 · p. 3
• Figure 2 distinguishes original-sequence representations used as contrastive anchors and for attribute prediction from masked-sequence representations used as positives and for masked item prediction.The figure also depicts same-batch negative samples and logits-level KL distillation between networks.Source: Figure 2 · p. 4

3.3 Multi-Task Learning

• Masked item prediction is the main recommendation task, while attribute prediction is auxiliary and incorporates item attribute information.The framework adopts the two tasks in a multi-task learning scheme to enhance recommendation with contextual information.Source: §3.3 · p. 4, §3.3.2 · p. 4

3.3.1 Masked Item Prediction.

• For masked item prediction, the model generates M randomly masked versions of each user sequence and reconstructs only the masked items.A proportion ρ of items is replaced by the mask token, with L = ceil(ρ*T) masked positions; each network's losses across masked sequences are summed.Source: §3.3.1 · p. 4

3.3.2 Attribute Prediction.

• Attribute prediction feeds the original sequence to the parallel networks and classifies each item under its associated attributes as a multi-label task.The task uses a sigmoid attribute classifier and binary cross-entropy because an item may have multiple attributes.Source: §3.3.2 · pp. 4–5

3.4.1 Contrastive Representation Distillation.

• Representation distillation uses an original-sequence representation as an anchor, masked-sequence representations as positives, and other users' same-batch sequences as negatives.The masked sequences are treated as data-augmented versions of the original sequence.Source: §3.4.1 · p. 5
• Intra-network Contrastive Learning (ICL) contrasts representations of different masked sequences produced by the same network.ICL uses cosine similarity and a temperature hyperparameter.Source: §3.4.1 · p. 5
• Cross-network Contrastive Learning (CCL) contrasts representations of different masked sequences produced by different networks, addressing ICL's lack of cross-network comparisons.The losses across network permutations are summed for CCL.Source: §3.4.1 · p. 5
• Logits-level KD minimizes KL divergence between networks' masked-item prediction distributions, considering distributions from all M masked sequences.The teacher logits' gradient is cut off for stability.Source: §3.4.1 · p. 5
• Logits-level distillation is collaborative: every network can serve as both teacher and student, sharing knowledge with its peers.The method sums the teacher-student loss permutations across the parallel networks.Source: §3.4.1 · p. 5

3.5.1 Overall Training Objective.

• The overall training objective sums losses for masked item prediction, attribute prediction, ICL, CCL, and knowledge distillation.These objectives are jointly optimized.Source: §3.5.1 · p. 5

3.5.2 Inference.

• For next-item inference, EMKD appends a mask token to the user sequence, converts its hidden representation into item probabilities, and averages predictions across networks.The item classifier converts the representation corresponding to the appended mask token into a probability distribution.Source: §3.5.2 · p. 6

3.6 Discussion

• Diverse ensemble predictions can cover different plausible item choices when users’ preferences shift across circumstances.The paper illustrates this with a user who likes Action and Comedy movies but may choose different titles in different situations.Source: §3.6 · p. 6
• With sparse, noisy data, peer knowledge transfer can filter incorrect predictions from noisy labels and share useful knowledge across networks.The authors suggest this may improve overall accuracy and model robustness.Source: §3.6 · p. 6

4.1 Experimental Setup

• Three public benchmark datasets are evaluated with HR/NDCG@5 and @10, leave-one-out splits, whole-item ranking, and general, attribute-aware, and contrastive baselines.Beauty and Toys use product categories and brands as attributes, while ML-1M uses movie genres; users with fewer than five interactions and items with fewer than five users are filtered.Source: §4.1.2 · p. 6, §4.1.3 · p. 6, §4.1.1 · p. 6

4.3 Hyper-parameter Sensitivity (RQ2)

• More masked sequences improve performance until gains level off; temperature and distillation weights need balanced settings, while performance peaks at hidden dimensionality 256 and slightly drops at 512.The default configuration uses three parallel networks and hidden dimensionality 256.Source: §4.3.1 · p. 7, §4.3.2 · p. 7, §4.3.3 · p. 7

4.4 Ablation Study (RQ3)

• Ablations find ICL, CCL, KD, and attribute prediction beneficial; three parallel networks nearly reach the best performance, while four add cost and overfitting risk.Knowledge-transfer variants outperform independently trained ensembles, and removing any of the three distillation objectives reduces performance.Source: §4.4 · p. 7, Table 4 · p. 8

4.2 Overall Performance Comparison (RQ1)

• EMKD outperforms all baseline methods on sparse and dense datasets, with relative performance improvements ranging from 13.05% to 33.48%.Table 3 presents the overall comparison; the reported improvement range is relative to the best baseline method.Source: §4.2 · p. 7, Table 3 · p. 8

4.5 Adaptation to Other Models (RQ4)

• EMKD improves RNN, CNN, and Transformer encoders on Beauty and ML-1M after autoregressive models switch to next-item and next-position attribute prediction.The adaptation changes the masked-item prediction task used by bidirectional Transformers to next-item prediction for autoregressive encoders.Source: §4.5 · pp. 8–9, Figure 4 · p. 9

4.6 Training Efficiency (RQ5)

• Three-network EMKD is about 30% slower per epoch than MMInfoRec, yet converges similarly to DuoRec initially and continues improving after DuoRec plateaus.The comparison measures training speed per epoch and time to satisfactory performance; the experiments use a single Tesla V100 GPU.Source: §4.6 · p. 9, Figure 5 · p. 9

4.7 Parameter Scaling

• At similar parameter sizes, EMKD consistently outperforms SASRec and BERT4Rec; adding Transformer blocks alone does not necessarily improve performance and can hurt it.The comparison includes EMKD with two or three parallel networks and deeper SASRec or BERT4Rec variants.Source: Table 5 · p. 8, §4.7 · p. 9

• On three public benchmark datasets, EMKD significantly outperforms baseline methods.Source: §5 · p. 9
• EMKD uses parallel networks and contrastive knowledge distillation to transfer knowledge at both the representation and logits levels.Source: §5 · p. 9
• Masked item prediction is the main task, while attribute prediction serves as an auxiliary task in multi-task learning.Source: §5 · p. 9