PaperlayerView exact revision on arXiv

Source-linked AI summary

GraphSAIL: Graph Structure Aware Incremental Learning for Recommender Systems

Yishi Xu, Yingxue Zhang, Wei Guo, Huifeng Guo, Ruiming Tang, Mark Coates

arXiv:2008.13517v2cs.IRcs.LGstat.ML

TL;DR

GNN-based recommender models are expensive to train and difficult to update frequently, while recent-only retraining can forget information needed for long-term preferences. GraphSAIL incrementally updates these models by distilling self-information and local and global graph structure. The authors report effectiveness on two public datasets and better performance than all baselines on an industrial dataset.

  • Problem

    GNN-based recommenders are costly to train and difficult to update frequently, while training only on recent records can cause forgetting of information needed for long-term preferences.

    Source: §1 · pp. 1–2

  • Method

    GraphSAIL incrementally trains GNN-based recommenders using distillation of node self-information and local and global graph structure.

    Source: §1 · p. 2

  • Results

    GraphSAIL was effective with two GNN recommender models on two public datasets and outperformed all baselines on a real industrial dataset.

    Source: §7 · p. 8

  • Takeaways & Limitations

    The work is an early attempt to apply incremental learning effectively to GNN-based recommender systems.

    Source: §7 · p. 8

    Local structure distillation does not capture each node’s global position relative to all other nodes.

    Source: §4.2 · p. 4

Abstract

from arXiv · show

Given the convenience of collecting information through online services, recommender systems now consume large scale data and play a more important role in improving user experience. With the recent emergence of Graph Neural Networks (GNNs), GNN-based recommender models have shown the advantage of modeling the recommender system as a user-item bipartite graph to learn representations of users and items. However, such models are expensive to train and difficult to perform frequent updates to provide the most up-to-date recommendations. In this work, we propose to update GNN-based recommender models incrementally so that the computation time can be greatly reduced and models can be updated more frequently. We develop a Graph Structure Aware Incremental Learning framework, GraphSAIL, to address the commonly experienced catastrophic forgetting problem that occurs when training a model in an incremental fashion. Our approach preserves a user's long-term preference (or an item's long-term property) during incremental model updating. GraphSAIL implements a graph structure preservation strategy which explicitly preserves each node's local structure, global structure, and self-information, respectively. We argue that our incremental training framework is the first attempt tailored for GNN based recommender systems and demonstrate its improvement compared to other incremental learning techniques on two public datasets. We further verify the effectiveness of our framework on a large-scale industrial dataset.

1 INTRODUCTION

Recommender systems need recent interaction data to track changing preferences, but frequent retraining is costly and recent-only training can forget long-term user information. GraphSAIL applies incremental learning to GNN-based recommenders, distilling self-information and local and global graph structure to preserve knowledge during updates.

1 INTRODUCTION

  • Recent interaction data helps capture changing preferences, but large training volumes make frequent retraining difficult.The introduction describes the tension between long training times and including the most recent data.Source: §1 · p. 1
  • Training only on recent records can cause catastrophic forgetting of information needed to capture users’ long-term preferences.The paper frames this as a challenge for updating recommendation models with new data.Source: §1 · p. 2
  • Incremental learning updates the current model with recent data while aiming to prevent substantial forgetting; this work focuses on regularization-based methods.The introduction distinguishes experience replay, parameter isolation, and regularization-based approaches.Source: §1 · p. 2
  • GraphSAIL distills self-information and local and global structure to preserve knowledge while incrementally updating GNN-based recommenders.Its local and global modules preserve topological information, while self-embedding distillation limits changes to individual embedding vectors.Source: §1 · p. 2
  • The paper presents GraphSAIL as the first incremental-learning approach tailored to GNN-based recommender systems, with competitive performance and computation efficiency.The authors also describe the method as generalizable and extendable to GNN-based recommenders.Source: §1 · p. 2

2 RELATED WORK

Prior work uses graph representations for recommendation and studies incremental learning, but the approaches described do not directly address incremental training for GNN-based recommenders. The paper focuses on regularization-based incremental learning for this setting.

2.1 Graph-based Recommendation

  • Graph-based recommendation work uses user-item and item-item graphs, while cited GNN models aggregate neighborhood information or learn multiple graph structures.The related work contrasts models trained on the full dataset with this paper’s focus on training incrementally on new data.Source: §2.1 · p. 2

2.2 Incremental Learning

  • Incremental-learning approaches include replaying stored historical data, isolating parameters across tasks, and regularizing to consolidate prior knowledge.The review notes that parameter isolation grows model size, which is undesirable for large-scale systems.Source: §2.2 · p. 2
  • Existing recommendation approaches often use matrix factorization or L2 regularization, while cited GNN distillation methods are not directly suited to recommendation.The paper investigates practical regularization-based incremental learning for GNN-based recommendation models.Source: §2.2 · pp. 2–3

3 PRELIMINARIES

The paper builds a data-efficient training framework on existing GCN-based recommendation models and knowledge distillation. It reviews how graph convolution produces node representations and how distillation transfers prior knowledge during incremental training.

3 PRELIMINARIES

  • The framework combines existing GCN-based recommendation systems with knowledge distillation.The paper presents these as the two components underlying its data-efficient training framework.Source: §3 · p. 3

3.1 GCN-based Recommender Models

  • GCN recommenders encode users and items by iteratively aggregating sampled multi-hop neighborhood information into their representations.Initial user and item embeddings are learned alongside the GCN parameters; the final embeddings follow aggregation at the K-th layer.Source: §3.1 · p. 3

3.2 Knowledge Distillation

  • In incremental training, a teacher trained on historical data transfers knowledge to a student trained on new data through a distillation metric in the loss.GraphSAIL uses distillation to transfer graph-topological information, rather than focusing only on predictions or middle activations.Source: §3.2 · p. 3

4 METHODOLOGY

GraphSAIL incrementally trains GNN-based recommenders by distilling local and global graph structure and node self-information. Its training objective combines these preservation losses with recommendation loss.

4.1 Local Structure Distillation

  • Local distillation matches teacher and student dot products between each node embedding and its previous-time local-neighborhood representation to preserve historical preferences.The same scheme applies to user and item nodes.Source: §4.1 · p. 3

4.2 Global Structure Distillation

  • Global distillation matches teacher and student distributions over user-preference and item-category cluster anchors, preserving each node’s position relative to other nodes.The distributions use normalized embedding similarities to K-means cluster anchors, and their matching minimizes summed KL divergences.Source: §4.2 · p. 4

4.3 Self-embedding Distillation

  • Self-embedding distillation uses mean squared error to keep each user’s and item’s embedding near its previous position, with strength weighted by the number of new records.The authors state that nodes with richer historical records receive stronger distillation.Source: §4.3 · p. 4

4.4 Model Training

  • Training uses BPR on triplets of observed positive and sampled unobserved negative interactions, combined with self-embedding, local-structure, and global-structure distillation losses.Source: §4.4 · pp. 4–5

5 EXPERIMENTS

The evaluation uses Gowalla and LastFM, splitting each dataset by timestamp into a base block and subsequent incremental blocks.

5.1 Datasets

  • The evaluation uses two public datasets, Gowalla and LastFM, which differ in size, domain, time range, and sparsity.Source: §5.1 · p. 5
  • Gowalla uses the first 70% of records as a base block and divides the remaining 30% evenly into five incremental blocks.Duplicate interactions and user and item nodes with fewer than 10 records are filtered out.Source: §5.1 · p. 5
  • LastFM uses the first 60% of records as a base block and divides the remaining 40% evenly into four incremental blocks.No interactions or nodes are filtered out because the dataset is small.Source: §5.1 · p. 5

5.2 Experiment Setting

The experiments evaluate incremental recommendation across consecutive time blocks, two GNN-based base models, several baselines, and fixed incremental-training settings.

5.2 Experiment Setting

  • Base-block training uses a random held-out subset for testing and validation; incremental block t trains the model and block t + 1 evaluates it.For the base block, the figure distinguishes training data from the shaded random test-and-validation subset.Source: Figure 3 · p. 5

5.2.1 Evaluation metrics.

  • Recall@20 measures the share of positive items recovered in each user's top 20 predictions, averaged across consecutive incremental test blocks.The base block is split 80/10/10 for training, validation, and test; hyperparameters are tuned on Inc. b1 and then fixed.Source: §5.2.1 · p. 5

5.2.2 Recommender base models.

  • The experiments adapt PinSage to separate user and item networks on the user-item graph and use MGCCF to model user-item, user-user, and item-item graphs.The two base models differ in architecture and the graph information they learn.Source: §5.2.2 · p. 5

5.2.3 Baseline Algorithms.

  • Baselines include fine-tuning on new data, unweighted embedding distillation, scalable local-structure distillation, and full-batch retraining on historical data.The scalable LSP_s variant samples 10 neighbors per node; full-batch training serves as a full-data performance and training-time reference.Source: §5.2.3 · p. 5

5.2.4 Parameter Settings.

  • All nodes use 128-dimensional embeddings with Adam, and incremental blocks use a lower learning rate and early stopping.Incremental training runs up to 15 epochs on Gowalla and 6 on LastFM; other hyperparameters are tuned on Inc. b1 and held fixed afterward.Source: §5.2.4 · pp. 5–6

5.3 Comparison with Baselines

GraphSAIL generally improves average Recall@20 over fine-tuning and performs best among the baselines, with one PinSage-on-Gowalla exception. Incremental training also substantially reduces training time on Gowalla.

5.3 Comparison with Baselines

  • GraphSAIL improves average Recall@20 over fine-tuning by 10.14% with PinSage and 7.21% with MGCCF on LastFM, and 0.64% and 3.92%, respectively, on Gowalla.These are averages over all incremental blocks.Source: §5.3 · p. 6
  • GraphSAIL has the best average Recall@20 except with PinSage on Gowalla, where baseline differences are within one standard deviation; MGCCF with GraphSAIL exceeds PinSage by 15.37% on the last block.The authors attribute the exception to the one-layer PinSage model learning slowly from incremental data, leaving little forgetting for distillation to address.Source: §5.3 · p. 6
  • On Inc. b1, GraphSAIL and LSP_s outperform fine-tuning except in the noted PinSage-Gowalla case; GraphSAIL performs best because it also captures global graph structure.The authors report that Emb_d works only on LastFM and LSP_s only on Gowalla, while GraphSAIL is consistently strong apart from the exception.Source: §5.3 · p. 6
  • Hyperparameters are tuned only on Inc. b1, and the authors cannot guarantee that preference or item-property drift remains constant in later blocks.They identify robust performance despite this uncertainty as a desired property.Source: §5.3 · p. 6
  • On Gowalla, incremental learning reduces training time by over 2 orders of magnitude versus full-batch training, supporting more frequent updates on large-scale data.Training time is compared per incremental block.Source: §5.3 · p. 6

5.4 Ablation Studies

The LastFM ablations find that global-structure preservation improves performance for both base models, and that the full component combination outperforms its subsets.

5.4 Ablation Studies

  • Self-information distillation contributes more for PinSage than MGCCF, while each component contributes to MGCCF's performance improvement.The authors conjecture that self-information matters more for the simpler one-layer PinSage model because it has less capacity to capture graph structure.Source: §5.4 · p. 6
  • On LastFM Inc. b1, all three GraphSAIL components outperform any subset for both models, and global-structure preservation significantly boosts performance.The ablation results average 10 repeated experiments.Source: §5.4 · pp. 6–7, Table 4 · p. 7
  • Local-structure distillation alone performs poorly because its hyperparameters were tuned for the combination of all components, despite its negative impact on PinSage.The full combination nevertheless outperforms every subset for both models.Source: §5.4 · pp. 6–7

5.5 Comparison between Graph-based and Non-Graph Models

The LastFM comparison finds that MGCCF uses new data more effectively than BPRMF during fine-tuning, but also shows catastrophic forgetting. The authors attribute graph models’ update behavior to information shared with neighboring nodes.

5.5 Comparison between Graph-based and Non-Graph Models

  • On LastFM, MGCCF starts with better future-interaction performance and reaches its peak quickly, but fine-tuning shows clear catastrophic forgetting.The comparison uses 10 fine-tuning epochs.Source: §5.5 · p. 7
  • BPRMF cannot learn effectively from new data during fine-tuning, unlike the rapid convergence observed for MGCCF.The comparison is between MGCCF and BPRMF on LastFM.Source: §5.5 · p. 7
  • Graph-based updates can use graph information to update a node embedding usefully even when that node is absent from the new data.The authors explain that updating a node embedding implicitly updates its neighbors.Source: §5.5 · p. 7

6 APPLICATION: CANDIDATE SELECTION FOR APP RECOMMENDATION

The App Store application uses GraphSAIL for candidate selection in an app recommendation pipeline. On the industrial dataset, it outperforms batch mode and other incremental baselines while requiring much less training time.

6 APPLICATION: CANDIDATE SELECTION FOR APP RECOMMENDATION

  • Candidate selection narrows tens or hundreds of thousands of apps to several hundred per user before matching and ranking.Matching scores represent user preference; ranking also applies diversity and business rules.Source: §6 · p. 7
  • Model-efficiency comparisons use the same machine with a single NVIDIA Tesla V100 GPU; batch-training results come from one experiment.Training time is measured until convergence, excluding test time on validation data.Source: §6 · p. 7

6.1 Dataset and Compared Models

  • The candidate-selection model uses a user-item graph built from 22 consecutive days of game-app downloads, with app and user features as node features.A simplified MGCCF GCN learns user and item embeddings, then selects candidates by embedding similarity.Source: §6.1 · p. 7
  • Batch mode trains on 14 days in a sliding window, while incremental strategies warm-start on days 0–13 and update using each new training day.The batch evaluation uses days [T,T+13] for training, T+14 for validation, and T+15 for testing; incremental evaluation uses T, T+1, and T+2 for those stages.Source: §6.1 · p. 7

6.2 Performance Comparison

  • GraphSAIL outperforms batch mode by 5.84% in Logloss and takes just 3.5% of batch mode's training time on the industrial dataset.The authors also report consistently better results than baseline models on all incremental days and better performance than the other incremental-learning baselines.Source: §6.2 · p. 7, Table 5 · p. 8

7 CONCLUSION

GraphSAIL incrementally updates GNN-based recommender models while preserving node self-information and local and global structure. The paper reports results on public and industrial datasets and presents the work as an early attempt in this area.

7 CONCLUSION

  • GraphSAIL preserves node self-information, local structure information, and global structure information during incremental updates of GNN-based recommender models.Source: §7 · p. 8
  • Experiments with two GNN recommender models on two public datasets demonstrate GraphSAIL's effectiveness, and the industrial deployment outperforms all baselines.Source: §7 · p. 8
  • The authors describe the work as an early attempt to apply incremental learning effectively to GNN-based recommender systems.Source: §7 · p. 8

Related papers

Loading 2008.13517v2…