PaperlayerView exact revision on arXiv

Source-linked AI summary

Pre-Training Graph Neural Networks for Cold-Start Users and Items Representation

Bowen Hao, Jing Zhang, Hongzhi Yin, Cuiping Li, Hong Chen

arXiv:2012.07064v1cs.IR

TL;DR

Recommendation GNNs use high-order collaborative signals, but cold-start embeddings are not explicitly optimized and cold-start neighbors are not specially handled during convolution. The paper pre-trains a GNN to reconstruct simulated cold-start embeddings, with a meta aggregator and adaptive neighbor sampler. Experiments on three datasets report effectiveness against original GNN models.

  • Problem

    Recommendation GNNs do not explicitly optimize cold-start user/item embeddings or specially handle cold-start neighbors during graph convolution.

    Source: §1 · pp. 1–2

  • Method

    The model pre-trains on simulated cold-start scenarios by reconstructing embeddings, using a self-attention meta aggregator and an adaptive neighbor sampler.

    Source: §1 · p. 2

  • Results

    Experiments on three datasets demonstrate the effectiveness of the proposed pre-training GNN against original GNN models.

    Source: §6 · p. 8

  • Takeaways & Limitations

    The proposed pretext task directly targets cold-start embedding quality, and the model is applied to downstream recommendation.

    Source: §1 · p. 2, §6 · p. 8

    The approach assumes matrix factorization can learn high-quality user/item embeddings from abundant interactions to provide reconstruction targets.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Cold-start problem is a fundamental challenge for recommendation tasks. Despite the recent advances on Graph Neural Networks (GNNs) incorporate the high-order collaborative signal to alleviate the problem, the embeddings of the cold-start users and items aren't explicitly optimized, and the cold-start neighbors are not dealt with during the graph convolution in GNNs. This paper proposes to pre-train a GNN model before applying it for recommendation. Unlike the goal of recommendation, the pre-training GNN simulates the cold-start scenarios from the users/items with sufficient interactions and takes the embedding reconstruction as the pretext task, such that it can directly improve the embedding quality and can be easily adapted to the new cold-start users/items. To further reduce the impact from the cold-start neighbors, we incorporate a self-attention-based meta aggregator to enhance the aggregation ability of each graph convolution step, and an adaptive neighbor sampler to select the effective neighbors according to the feedbacks from the pre-training GNN model. Experiments on three public recommendation datasets show the superiority of our pre-training GNN model against the original GNN models on user/item embedding inference and the recommendation task.

1 INTRODUCTION

The introduction identifies cold-start embeddings and neighbors as unresolved challenges for recommendation GNNs. It proposes embedding-reconstruction pre-training with a meta aggregator and adaptive neighbor sampler, reporting superiority over state-of-the-art GNNs on embedding inference and recommendation.

1 INTRODUCTION

  • Side information such as content features or knowledge graphs can compensate for sparse-interaction embeddings, but may be unavailable or difficult to link.Source: §1 · p. 1
  • Existing GNNs capture high-order collaborative signals but do not directly optimize cold-start embeddings or account for cold-start neighbors during convolution.Sparse-neighbor embeddings can be inaccurate and then harm the target embedding when propagated.Source: §1 · pp. 1–2
  • The proposed pre-training task simulates cold-start users/items from those with abundant interactions and reconstructs their ground-truth embeddings.Episodes sample K neighbors and optimize reconstruction loss, directly targeting embedding quality and adaptation to new cold-start users/items.Source: §1 · p. 2
  • A self-attention meta aggregator strengthens each convolution step, while an adaptive sampler selects neighbors using feedback from the pre-training GNN.The components address high-order cold-start neighbors and are jointly trained; the model is described as model-agnostic.Source: §1 · p. 2
  • Experiments on embedding inference and downstream recommendation demonstrate superiority over state-of-the-art GNN models.Source: §1 · p. 2

2 PRELIMINARIES

The preliminaries represent recommendation data as a user–item graph and define an encoding function for user and item embeddings. They describe GNN recommendation as iterative neighbor sampling and convolution, followed by relevance scoring, while noting that recommendation loss does not explicitly improve cold-start embedding quality.

2 PRELIMINARIES

  • User–item interactions are represented as a bipartite graph with user and item node sets connected by edges.Source: §2 · p. 2
  • The encoding function maps users and items to d-dimensional vectors and is pre-trained for use in downstream recommendation.Source: §2 · p. 2

2.1 GNN for Recommendation

  • A recommendation GNN samples neighbors and repeatedly convolves their embeddings to obtain user and item representations.After the final convolution step, the model calculates user–item relevance and uses BPR loss to optimize preferences.Source: §2.1 · p. 2
  • The framework applies a pre-trained GNN to recommendation and fine-tunes it for that downstream task.Source: Figure 2 · p. 3
  • The recommendation objective does not explicitly improve cold-start user/item embedding quality.Source: §2.1 · p. 3

3 THE PRE-TRAINING GNN MODEL

The model pre-trains a GNN to reconstruct cold-start embeddings, then adds a meta aggregator and adaptive neighbor sampler to address cold-start neighbors. Its modules are trained in stages and fine-tuned for downstream recommendation.

3.1 The Basic Pre-training GNN Model

  • The basic model simulates cold-start targets by sampling K neighbors from users or items with abundant interactions, then trains the GNN to reconstruct their ground-truth embeddings.Ground-truth embeddings are learned from abundant interactions using NCF; cosine similarity measures reconstruction difference.Source: §3.1 · p. 3

3.2 Meta Aggregator

  • A self-attention meta learner builds an averaged meta embedding from first-order neighbors and adds it at each graph-convolution step to enhance cold-start neighbor representations.The meta learner is trained to predict target embeddings from first-order neighbors.Source: §3.2 · p. 4

3.3 The Adaptive Neighbor Sampler

  • The adaptive sampler uses a hierarchical MDP to select high-order neighbors based on pre-training feedback, while retaining all first-order neighbors.Its state includes embedding similarities and element-wise products involving the target and previously selected neighbors; the reward reflects predicted-versus-ground-truth embedding similarity.Source: §3.3 · p. 4

3.4 Model Training

  • Training proceeds by pre-training the meta learner, then the meta aggregator, then the sampler, and finally jointly training all three modules.During joint training, parameters are updated using a linear combination of old and new values.Source: §3.4 · p. 5

3.5 Downstream Recommendation Task

  • For downstream recommendation, the pre-trained sampler selects high-order neighbors and the meta aggregator produces user and item embeddings, which are fine-tuned using BPR loss.A transformed user-item embedding product yields the relevance score.Source: §3.5 · p. 5

4 EXPERIMENT

Experiments evaluate embedding inference and downstream recommendation on three public datasets. The proposed pre-training GNN improves embedding inference and recommendation over the compared GNN models, with additional gains from its meta aggregator and adaptive neighbor sampler.

4.1 Experimental Setup

  • The experiments use MovieLens-1M, MOOCs, and Last.fm, and assess both embedding inference and downstream recommendation under distinct meta-training and meta-test settings.Intrinsic evaluation measures predicted embeddings; extrinsic evaluation fine-tunes the trained model for recommendation.Source: §4 · p. 5, §4.1 · pp. 5–6

4.2 Intrinsic Evaluations: Embedding Inference

  • GNNs improve embedding quality over NCF by 2.6%-24.9%, basic pre-training adds 1.79-15.20%, and the meta aggregator and sampler add 1.09%-18.20%.These ranges are reported in Spearman correlation.Source: §4.2 · p. 6
  • 33.9%-58.4% in Spearman correlation: the proposed pre-training GNN improves cold-start user/item embedding quality over the baselines.Table 2 reports embedding inference using 3-order neighbors.Source: §4.2 · p. 6, Table 2 · p. 7
  • With K=3, most algorithms perform best at layer depth L=3, while using only first-order neighbors performs worst.Figure 3 compares embedding inference across layer depths.Source: §4.2 · p. 6, Figure 3 · p. 7

4.3 Extrinsic Evaluation: Recommendation

  • In MOOCs cases, the adaptive sampler selects relevant second-order cold-start neighbors with few interactions that importance-based samplers miss.Examples include related finance items and computer-science users.Source: §4.3 · pp. 7–8
  • 0.40%-3.50% in NDCG: basic pre-training GNN models outperform corresponding original GNN models, with the meta aggregator and adaptive neighbor sampler each adding 0.30%-6.50%.Table 3 reports overall recommendation performance.Source: §4.3 · p. 7, Table 3 · p. 8

5 RELATED WORK

Prior cold-start approaches use side information, meta-learning, or GNNs to improve recommendation, but the cited work identifies limits in available information and embedding quality. The paper positions cold-start embedding reconstruction as a new pretext task for recommendation GNN pre-training.

5 RELATED WORK

  • Existing recommender systems incorporate spatial information, social trust paths [42], and knowledge graphs [37] to enhance cold-start user/item representations; however, side information is not always available, making it intractable to improve cold-start embedding quality.Source: §5 · p. 8
  • Meta-learning methods include metric-based and model-based [7] [20] [24] recommendation, but few capture high-order interactions.The cited meta-learning methods include [9] [26] [34].Source: §5 · p. 8
  • Recommendation GNNs such as Pinsage [45], NGCF [38], and LightGCN [13] capture high-order collaborative signals but optimize adoption likelihood rather than directly improving cold-start embeddings.The related-work passage also names FBNE and CAGR [42].Source: §5 · p. 8
  • Prior GNN pre-training uses graph-, node-, or hybrid-level pretext tasks, while this work defines cold-start user/item embedding reconstruction as a recommendation pretext task.Examples include subgraph discrimination [25], mutual-information maximization [27, 32], node feature and edge generation [17], and hybrid tasks [15] [46].Source: §5 · p. 8

6 CONCLUSION

The conclusion proposes pre-training a GNN to address cold-start recommendation by reconstructing cold-start user/item embeddings, strengthening aggregation, and adaptively sampling neighbors. Experiments on three datasets demonstrate effectiveness against original GNN models, and future work will explore multiple pretext tasks.

6 CONCLUSION

  • The proposed pretext task reconstructs cold-start user/item embeddings to explicitly improve their embedding quality.Source: §6 · p. 8
  • A self-attention-based meta aggregator improves each graph convolution step, while adaptive sampling selects neighbors according to GNN performance.Source: §6 · p. 8
  • Experiments on three datasets demonstrate the effectiveness of the pre-training GNN against original GNN models.Source: §6 · p. 8
  • Future work will explore multiple pretext tasks.Source: §6 · p. 8

Related papers

Loading 2012.07064v1…