PaperlayerView exact revision on arXiv

Source-linked AI summary

GOD: Enhancing Generalization via Deep Grafting for Sequential Recommendation

WooJoo Kim, JunYoung Kim, JaeHyung Lim, HwanJo Yu

arXiv:2608.16073v1cs.IRcs.LG

TL;DR

Sparse and noisy histories make it difficult for sequential recommenders to learn representations that generalize beyond observed interactions, while conventional distillation can entangle component effects. GOD uses grafted teacher-student hybrid models for component-level distillation and consistently outperforms existing baselines across three real-world datasets, by up to 13.92%.

  • Problem

    Sparse and noisy user histories make sequential recommenders vulnerable to learning spurious patterns instead of representations that generalize to future interactions.

  • Method

    GOD grafts trainable student embeddings or encoders into a frozen teacher to create hybrid source models that provide component-level supervision.

  • Results

    GOD consistently outperforms existing knowledge-distillation and self-supervised baselines across three real-world datasets, by up to 13.92%.

  • Takeaways & Limitations

    Component-level feedback from jointly distilled stable teacher knowledge and complementary grafted views improves the generalization of distilled sequential-recommendation knowledge.

  • Takeaways & Limitations

    Grafted Encoding roughly doubles training-time attention computation because it processes concatenated sequences, although it is not used during inference.

Abstract

from arXiv · show

Sequential recommenders often struggle with sparse and noisy histories, limiting generalization to unseen interactions. Knowledge distillation mitigates this by transferring dense supervision from a teacher to a student. However, most distillation methods run teacher and student independently, then match student outputs or representations to the teacher. Such supervision entangles student-component effects, blurring whether weak generalization stems from unreliable embeddings, overfitted encoding, or co-adaptation to sparse histories. In this paper, we propose Graft-Oriented Distillation (GOD), a component-level distillation framework for improved generalization through grafting. Grafting denotes replacing selected frozen-teacher components with trainable student counterparts to build hybrid source models. GOD uses these hybrid models to evaluate student embeddings with the teacher encoder and the student encoder with teacher embeddings, providing component-level feedback. At inference, GOD uses only the student, incurring no additional cost. Across three real-world datasets, GOD outperforms state-of-the-art baselines by up to 13.92%.

1 Introduction

Sparse and noisy histories entangle embedding and encoder errors in separated teacher–student paths, obscuring the source of weak generalization. GOD addresses this with grafted hybrid teacher paths that provide component-level supervision while retaining student-only inference.

  • Motivation: Sparse and noisy interaction histories make sequential recommenders prone to fitting observed co-occurrences and hinder generalization.Sequential recommendation predicts the next interacted item by modeling patterns in user histories.
  • Motivation: Separated teacher–student paths entangle unreliable embeddings, overfitted encoding, and their co-adaptation, blurring the cause of weak generalization.This limitation is especially pronounced when sparsity and noise affect sequence-representation components differently.
  • Graft-Oriented Distillation: Grafting replaces selected frozen-teacher components with trainable student counterparts, creating hybrid paths that isolate and supervise individual student components.The hybrid paths evaluate student embeddings with the teacher encoder and the student encoder with teacher embeddings.
  • Graft-Oriented Distillation: GOD uses only the non-grafted student at inference, avoiding additional inference cost.Figure 1 contrasts conventional independent KD with grafting-based KD and identifies the non-grafted student as the inference model.
  • Graft-Oriented Distillation: GOD is a component-level KD framework that grafts the student embedding table or sequence encoder into a frozen teacher while using the non-grafted student as the target.These grafted sources expose different student components to teacher-side computation and provide complementary supervision for generalization.

2 Preliminary

Sequential recommendation predicts a user’s next item from a chronologically ordered interaction sequence. Standard Transformer-based recommenders represent sequences with item and position embeddings, process them through a multi-head Transformer encoder, and compute next-item probabilities and recommendation loss.

  • Sequential Recommendation: Sequential recommendation predicts the next item from a user’s historical interaction sequence.Each user has a chronologically ordered sequence of interacted items.
  • Model Architecture: Most sequential recommenders use an embedding table and a sequence encoder, with Transformer-based models prevalent for modeling complex sequential patterns.The paper describes the standard Transformer-based formulation used in sequential recommendation.
  • Input Embedding: The model combines item and position embeddings into a fixed-length sequence representation, truncating earliest items or front-padding zeros when needed.Item and position tables provide embeddings for interacted items and their sequence positions.
  • Sequence Encoding: An L-layer multi-head Transformer encoder processes the sequence embedding and selects the last position as the sequence representation.The encoder applies all Transformer layers before producing the sequence representation.
  • Prediction Objective: Given the interaction sequence and ground-truth item index, the model computes a next-item probability distribution and recommendation loss.These quantities define the prediction and training objectives in the standard formulation.

3 Grafting Analysis for Distillation

Section 3 analyzes grafting as a source of component-level feedback for generalization and procedural alignment. It contrasts conventional independent-path distillation with hybrid grafted teachers and reports stronger layer-wise correspondence under grafting.

  • Grafting Analysis: Grafting replaces a frozen teacher component with its trainable student counterpart, forming a hybrid source model that evaluates the student component under teacher-side computation.Because the grafted component is shared with the student, distillation regularization can provide component-level feedback.
  • Grafting Analysis: Conventional knowledge distillation jointly supervises the full student path from an independent teacher, potentially obscuring whether weak generalization arises from particular student components.The analysis frames grafting as a way to disentangle student embeddings and encoder effects during distillation.
  • Empirical Evidence: DE, RRD, and BD show teacher-final-layer dominance in CKA heatmaps, with all student layers most aligned to the final teacher layer.This pattern suggests output imitation rather than layer-wise procedural alignment.
  • Empirical Evidence: Graft-Embed and Graft-Encoder show clear diagonal CKA patterns, with student layers most aligned to corresponding teacher layers.The result indicates stronger layer-wise correspondence between teacher and student encoding processes than the compared conventional distillation methods.

4 Proposed Framework: GOD

GOD performs component-level distillation by grafting student embeddings or encoders into frozen-teacher models, then contrasts their coupled representations. Grafted Encoding stabilizes training, while inference uses only the non-grafted student.

  • Hybrid Source Models: GOD constructs Non-Grafted Teacher, Embed-Grafted Teacher, and Encoder-Grafted Teacher to expose student components to teacher-side computation.The grafts separately replace the teacher embedding table or sequence encoder with its student counterpart.
  • Grafted Encoding: Grafted Encoding concatenates teacher- and student-side embeddings with mutual attention, stabilizing hybrid representations while keeping the teacher frozen.It targets Transformer-based sequential recommenders and groups the four models by their shared sequence encoder.
  • Grafted Encoding: GE roughly doubles training-time attention computation, from 𝑂(2𝑁2𝑑𝑇+2𝑁2𝑑𝑆) without GE to 𝑂(4𝑁2𝑑𝑇+ 4𝑁2𝑑𝑆) with GE.GE is not used during inference, and its overhead can be offset by faster convergence in practice.
  • Graft-aware Contrastive Learning: GOD uses contrastive learning over source-target and source-source pairs to transfer relational structure and regularize trainable student components.It down-weights highly similar pair types to reduce redundant supervision and emphasize complementary component-level signals.
  • Inference and Optimization: At inference, GOD uses only Non-Grafted Student, avoiding additional inference cost while recommendation and distillation objectives jointly optimize the student.The Non-Grafted Student supplies the sequence representation for next-item prediction.

5 Experiment · 5.1 Experimental Setup · 5.2 Overall Performance (RQ1)

Experiments evaluate GOD across three public sequential-recommendation datasets, multiple backbone architectures, and full-ranking leave-one-out evaluation. GOD achieves the best overall results, outperforming the strongest baseline by up to 13.92% and remaining effective in same-capacity self-distillation by up to 8.12%.

  • 5 Experiment: RQ1 compares GOD with existing knowledge-distillation and self-supervised methods for sequential recommendation.The study also poses questions about generalization, component effectiveness, and efficiency or hyperparameter sensitivity.
  • 5.1 Experimental Setup: Experiments use Amazon Beauty, Yelp, and MovieLens 1M, treating interactions as implicit feedback and filtering users and items with fewer than five interactions.The datasets vary in domain, scale, and sparsity.
  • 5.1 Experimental Setup: GOD and competing KD methods are applied to GRU4Rec, FMLPRec, and SASRec, spanning RNN-, MLP-, and Transformer-based recommendation architectures.Student denotes the compact model trained without KD, while Teacher denotes the larger pretrained model.
  • 5.1 Experimental Setup: Evaluation uses leave-one-out chronological splitting, full ranking without negative sampling, and HR@k and NDCG@k for k∈{10, 20}.The last, penultimate, and remaining sequence items serve as test, validation, and training interactions, respectively.
  • 5.2 Overall Performance (RQ1): 13.92%: GOD improves over the strongest baseline by up to 13.92% and achieves the best results across all datasets, backbones, and metrics.All KD methods outperform Student, indicating that teacher supervision benefits compact models under sparse sequential feedback.
  • 5.2 Overall Performance (RQ1): Sequence-aware KD methods generally outperform general recommendation KD methods, while GOD surpasses the strong BD baseline with a frozen teacher.BD often exceeds AdaRec and MSKDIK on the sparser Amazon Beauty and Yelp datasets.
  • 5.2 Overall Performance (RQ1): 8.12%: In same-capacity SASRec self-distillation, GOD achieves the best results across all datasets and metrics, improving over the strongest baseline by up to 8.12%.This setting uses an offline pretrained teacher with the same architecture and dimension as the student and compares against self-supervised contrastive baselines.

5.3 Generalization and Robustness (RQ2)

GOD generalizes robustly under sparse histories and noisy interactions, consistently outperforming competing methods across these conditions. It also remains effective across teacher–student capacity gaps and varying teacher quality, with particular benefits for smaller students and richer teachers.

  • User History Sparsity: GOD performs best across all sequence-length groups, with especially large margins for short histories where conventional KD has limited evidence.Longer histories improve all methods by providing richer preference signals.
  • Noise Robustness: GOD maintains the best performance across all noise levels, and its performance gap remains clear even under high noise ratios.Performance consistently declines as random negative items replace a larger ratio of test-time sequence items.
  • Capacity Sensitivity: GOD consistently improves over EMKD across all student–teacher dimension pairs, with larger gains for smaller students and generally richer teachers.Student dimensions are d_S∈{8, 16, 32}, while teacher dimensions are d_T∈{64, 128}.
  • Teacher Quality Sensitivity: All methods benefit from more trained teachers, while BD outperforms EMKD with weaker teachers.Teacher checkpoints are evaluated at 25%, 50%, 75%, and 100% of training progress toward the best checkpoint.

5.4 Ablation Study (RQ3)

The ablation study evaluates GOD’s source models, grafted encoding, projections, and graft-aware contrastive learning. Full GOD and its complete component designs consistently perform best, while adaptive coupling and weighting explain the gains.

  • Source Models: Full GOD achieves the best performance on all datasets, while removing either hybrid source consistently degrades performance.The larger drop without the Non-Grafted Teacher shows that it remains an important stable knowledge anchor; the two hybrid teachers provide complementary structural coupling.
  • Projection Design: Shared linear projections consistently outperform non-shared linear and 2-layer MLP projections across all datasets.The result indicates that GOD’s gains do not arise from additional projection capacity and supports simple shared projections for dimension adaptation.
  • Grafted Encoding: Full GE achieves the best performance on all datasets, while bidirectional mutual attention with full-length histories is necessary to fully exploit teacher-student coupling.Both one-direction variants outperform w/o GE, T→S only exceeds S→T only, and GE (half length) exceeds w/o GE despite matching the original concatenated sequence length.
  • Graft-aware Contrastive Learning: Full GCL achieves the best performance on all datasets, confirming the benefit of adaptive pair-type weighting.Source-Target only outperforms Unweighted, while adding source-source pairs improves performance further when their contributions are dynamically controlled.
  • Graft-aware Contrastive Learning: GCL keeps weights distributed across multiple relations, initially emphasizing grafted-source pairs before reallocating weight toward less redundant relations later in training.Pairs involving h(S,S)_u with grafted sources receive relatively high early weights, whereas h(T,T) and h(S,T) versus h(T,S) relations become more important later.

5.5 Efficiency and Parameter Sensitivity (RQ4)

GOD remains more efficient than BD and EMKD despite higher per-epoch time than Student, while delivering stable performance across a wide temperature range and mostly moderate sensitivity to distillation strength.

  • Efficiency: GOD uses more time per epoch than Student but less memory and time per epoch than BD and EMKD.The additional cost comes from hybrid source models and GE/GCL computation.
  • Efficiency: Teacher-conditioned source models provide more direct and fine-grained supervision than output-level imitation.They evaluate student embeddings and encoders through teacher-conditioned models.
  • Parameter Sensitivity: GOD is generally stable across a wide range of τ, showing limited sensitivity to temperature selection in contrastive distillation.This stability is reported in Figure 12.
  • Parameter Sensitivity: λ has a moderate effect on Amazon Beauty and Yelp, whereas MovieLens 1M degrades when λ becomes too large.The degradation may reflect excessive distillation strength over-regularizing the student on denser datasets.

6 Related Work

Prior recommendation distillation methods mainly vary in their supervision form, while sequential recommendation requires sequence-aware transfer of evolving preferences. GOD instead provides component-level feedback through grafted teacher-student computation, distinct from LLM-based semantic distillation and earlier grafting applications.

  • Knowledge distillation: Recommendation KD transfers knowledge from high-capacity teachers to compact students, with existing methods mainly differing in supervision form.Output-level methods transfer predictions or ranking preferences, including RD, RRD, and CD.
  • Sequential recommendation: Sequential recommendation KD must transfer dynamic preference patterns from interaction sequences, motivating sequence-aware supervision.AdaRec searches for scene-adaptive student architectures, while MSKDIK distills interest representation and drift knowledge across multiple stages.
  • LLM-based distillation: LLM-based KD transfers semantic understanding or reasoning ability, including content knowledge for ID-based sequential recommenders.DLLM2Rec distills LLM-derived content knowledge, while SLMRec and SLIM distill LLM knowledge or reasoning processes.
  • GOD’s distinction: Rather than injecting external semantic knowledge, GOD improves distilled knowledge through component-level feedback from grafted teacher-student computation.This distinguishes GOD from LLM-based distillation approaches.
  • Grafting: Grafting originated as the inverse of pruning in decision trees and was later adapted to neural networks to replace or augment selected components.Examples include layer grafting, NetGraft, and GrafT for layer improvement, few-shot KD, and multi-scale representation sharing.

7 Conclusion

The paper revisits knowledge distillation for sequential recommendation from a component-level perspective. GOD addresses entangled student embeddings and encoder effects by constructing teacher-conditioned source models through grafting.

  • 7 Conclusion: Conventional distillation can entangle the effects of student embeddings and encoders by supervising the complete student path with teacher-produced signals.This limitation motivates analyzing student components separately.
  • 7 Conclusion: GOD is a component-level knowledge-distillation framework for sequential recommendation.The framework is proposed to address the identified limitation of conventional distillation.
  • 7 Conclusion: GOD constructs teacher-conditioned source models through grafting by replacing selected frozen-teacher components with trainable student counterparts.This grafting design enables component-level evaluation within hybrid source models.

A GenAI Usage Disclosure

The authors disclose that ChatGPT was used only for English grammar and language refinement, while all research and technical contributions remained their original work.

  • A GenAI Usage Disclosure: ChatGPT was used solely for English grammar and language refinement, with all AI-assisted edits manually reviewed.No generative AI was used for coding, experiments, data analysis, or other parts of the research process.
Loading 2608.16073v1…