Knowledge-aware Graph Neural Networks with Label Smoothness Regularization for Recommender Systems

Hongwei Wang; Fuzheng Zhang; Mengdi Zhang; Jure Leskovec; Miao Zhao; Wenjie Li; Zhongyuan Wang

Source-linked AI summary

Knowledge-aware Graph Neural Networks with Label Smoothness Regularization for Recommender Systems

Hongwei Wang, Fuzheng Zhang, Mengdi Zhang, Jure Leskovec, Miao Zhao, Wenjie Li, Zhongyuan Wang

arXiv:1905.04413v3May 11, 2019cs.LGcs.IRstat.ML

TL;DR

KGNN-LS addresses the limited scalability and end-to-end capability of existing knowledge-graph recommenders. It learns user-specific relation weights, propagates features with a GNN, and regularizes those weights using label smoothness; experiments show gains over state-of-the-art baselines across four scenarios, including sparse-interaction cold starts.

Problem
Existing KG-aware recommender systems rely on manual feature engineering, lack end-to-end training, and face scalability and cold-start challenges.
Method
KGNN-LS transforms a knowledge graph into a user-specific weighted graph, computes personalized item embeddings with GNN aggregation, and regularizes edge weights using label smoothness.
Results
KGNN-LS outperforms state-of-the-art baselines in four recommendation scenarios and maintains strong performance when user-item interactions are sparse.
Takeaways & Limitations
The approach provides personalized knowledge-graph representations for recommendation with desirable scalability relative to knowledge-graph size.
Takeaways & Limitations
The label-smoothness assumption constrains the model by assuming adjacent knowledge-graph entities are likely to have similar relevancy labels.

Abstract

from arXiv · show

Knowledge graphs capture structured information and relations between a set of entities or items. As such knowledge graphs represent an attractive source of information that could help improve recommender systems. However, existing approaches in this domain rely on manual feature engineering and do not allow for an end-to-end training. Here we propose Knowledge-aware Graph Neural Networks with Label Smoothness regularization (KGNN-LS) to provide better recommendations. Conceptually, our approach computes user-specific item embeddings by first applying a trainable function that identifies important knowledge graph relationships for a given user. This way we transform the knowledge graph into a user-specific weighted graph and then apply a graph neural network to compute personalized item embeddings. To provide better inductive bias, we rely on label smoothness assumption, which posits that adjacent items in the knowledge graph are likely to have similar user relevance labels/scores. Label smoothness provides regularization over the edge weights and we prove that it is equivalent to a label propagation scheme on a graph. We also develop an efficient implementation that shows strong scalability with respect to the knowledge graph size. Experiments on four datasets show that our method outperforms state of the art baselines. KGNN-LS also achieves strong performance in cold-start scenarios where user-item interactions are sparse.

1 INTRODUCTION

The paper addresses sparse and cold-start recommendation by using knowledge graphs to capture item relatedness, while seeking an end-to-end alternative to manually engineered KG-aware methods. KGNN-LS personalizes graph structure and regularizes learned edge weights to improve recommendation.

Cold-start and sparse interactions limit collaborative filtering, especially for brand-new items.
Existing KG-aware recommenders rely on manual feature engineering, lack end-to-end training, and have poor scalability.
KGNN-LS learns user-specific relation scores to transform a knowledge graph into a weighted graph reflecting personalized interests.
A graph neural network then aggregates local neighborhood information to produce user-personalized item embeddings.
Label-smoothness regularization addresses overfitting risk caused by learning flexible edge weights from sparse user-item interactions.

2 RELATED WORK

Prior work applies graph learning, embeddings, and semi-supervised label smoothness to related problems, but KGNN-LS targets heterogeneous knowledge graphs for recommendation. Its distinction is adapting graph neural networks to user- and relation-specific recommendation structure.

Spectral and non-spectral GNNs generalize convolutional representation learning from Euclidean data to graph domains.
Existing recommender GNNs mainly operate on homogeneous bipartite or user/item-similarity graphs, unlike KGNN-LS's heterogeneous KG setting.
GNNs have also been used to model knowledge graphs, but prior cited work did not target recommendation.
Graph-based semi-supervised learning commonly assumes that labels vary smoothly across graph edges, with edge weights either fixed or learnable.
Embedding-based KG recommenders pre-process graphs with knowledge graph embedding algorithms whose semantic objectives may be better suited to link prediction than recommendation.

3 PROBLEM FORMULATION

The paper formulates recommendation as predicting a user's potential interest in an unseen item from implicit interactions and a knowledge graph. Figure 1 summarizes the proposed model's transformation and propagation stages.

The interaction matrix Y records implicit feedback, with y_uv = 1 when user u engages with item v.
The knowledge graph G consists of head-relation-tail triples over entities and relation types, including item and property entities.
Figure 1 depicts conversion of the original KG into a user-specific weighted graph followed by GNN feature propagation with label-smoothness regularization.
The task is to predict whether a user will engage with an item not previously observed in the interaction data.
The prediction function produces y_hat_uv, the probability that user u will engage with item v, using model parameters, Y, and G.

4 OUR APPROACH

KGNN-LS transforms a heterogeneous knowledge graph into a user-personalized weighted graph, then applies graph neural networks and label-smoothness regularization to learn representations and edge weights. The paper connects minimum-energy label smoothing with harmonic label propagation and uses leave-one-out supervision to regularize edge weights.

Knowledge-aware Graph Neural Networks: KGNN-LS assigns user-specific importance scores to relation types, converting the heterogeneous knowledge graph into a personalized weighted graph.The score s_u(r) is computed from user and relation feature vectors through a differentiable function.
Knowledge-aware Graph Neural Networks: Each GNN layer aggregates transformed representations from an entity and its neighbors, while multiple layers incorporate information from increasingly distant knowledge-graph entities.Self-connections preserve each entity’s previous representation, and the final representation mixes features from up to L hops away.
Label Smoothness Regularization: The model learns edge weights jointly with transformation parameters, making optimization prone to overfitting when supervision comes only from sparse user-item interactions.The paper motivates label-smoothness regularization as a constraint on the learned edge-weight matrix.
Label Smoothness Regularization: Label smoothness assumes adjacent entities have similar relevancy labels and motivates an energy function whose minimum-energy label function is harmonic.For non-item entities, the harmonic value equals the average of neighboring entities’ values.
Label Smoothness Regularization: Theorem 2 shows that repeatedly propagating labels and resetting item labels to observed interactions converges to the unique fixed-point solution.The resulting solution is l*_u(E\V) = (I − P_EE)^−1 P_EV Y[u,V]^⊤.
Label Smoothness Regularization: Because propagated labels on observed items equal their true labels and unlabeled-node labels are unknown, the paper uses leave-one-out prediction to provide supervision for edge-weight regularization.A held-out item’s true label is compared with its label-propagation prediction.

5 EXPERIMENTS

The experiments evaluate KGNN-LS across four recommendation datasets and compare it with KG-free and KG-aware baselines using top-K recommendation and CTR prediction metrics.

Datasets: Experiments cover movie, book, music, and restaurant recommendation scenarios using four real-world datasets.The first three datasets are public, while the restaurant dataset is from Meituan-Dianping Group.
Datasets: MovieLens-20M, Book-Crossing, and Last.FM use sub-KGs constructed with Satori2, while Dianping-Food uses an internally constructed KG.The corresponding KGs contain different numbers of entities, edges, and relation types across datasets.
Baselines: The evaluation compares KGNN-LS with KG-free and KG-aware recommender baselines, including SVD, LibFM, LibFM + TransE, and CKE.SVD models user-item interactions with an inner product; CKE combines collaborative filtering with structural, textual, and visual knowledge.
Evaluation: Top-K recommendation is evaluated with Recall@K, while CTR prediction is evaluated with AUC.Top-K recommendation selects the K items with the highest predicted click probabilities for each test user; CTR prediction scores positive and randomly selected negative user-item pairs.

5.3 Validating the Connection between G and Y

The experiments validate KG–interaction proximity, show strong recommendation accuracy and stability, and examine cold-start behavior and hyperparameter sensitivity. Label smoothness improves performance when appropriately weighted, while excessive depth or regularization can hurt.

Connection between G and Y: Items sharing users are more likely to be close in the KG: 0.92 are within 2 hops versus 0.80 for items without common users.This supports exploiting KG proximity and the label-smoothness motivation.
Comparison with Baselines: KGNN-LS outperforms baselines in top-K recommendation and CTR prediction, with average AUC gains of 5.1%, 6.9%, 8.3%, and 4.3% across four datasets.The datasets are MovieLens-20M, Book-Crossing, Last.FM, and Dianping-Food, respectively.
Comparison with Baselines: KGNN-LS remains above baselines throughout September 2018 on Dianping-Food and exhibits low variance in daily AUC.The reported pattern suggests practical robustness and stability.
Label Smoothness: Non-zero label-smoothness regularization outperforms λ = 0, but overly large λ can mislead gradients; λ between 0.1 and 1.0 is preferable in most cases.The comparison uses R@10 on Last.FM across hidden-layer dimensions of 4, 8, and 16.
Cold-start Performance: At 20% of the MovieLens-20M training data, KGNN-LS AUC decreases by only 1.8%, versus 2.8%–8.4% for six baselines.Validation and test sets remain fixed while the training-set ratio varies from 100% to 20%.
Hyper-parameter Sensitivity: KGNN-LS performs best with 1 or 2 GNN layers and hidden dimension d = 8 ∼64; deeper or excessively wide representations can degrade performance.Four layers over-smooth representations, while overly large hidden dimensions may overfit.

5.5 Running Time Analysis

The running-time analysis evaluates how all methods scale as the MovieLens-20M knowledge graph grows, emphasizing curve trends over hardware- and configuration-dependent absolute times.

Running Time Analysis: Running time is measured while increasing the MovieLens-20M KG size up to five times its original size.Experiments use one NVIDIA Tesla M60 GPU, 12 Intel Xeon CPUs, and 128GB RAM.

6 CONCLUSION AND FUTURE WORK

The paper concludes that KGNN-LS combines user-specific KG weighting, neighborhood aggregation, and label-smoothness regularization for recommendation. Experiments show strong performance across four scenarios and scalability with KG size, while future work extends label smoothness beyond recommendation.

Conclusion: KGNN-LS applies GNNs to KGs using user-specific relation scoring and weighted neighborhood aggregation.The model is designed to capture semantic relationships and personalized preferences.
Conclusion: Label-smoothness constraints and leave-one-out loss regularize learning of KG edge weights.The conclusion also discusses how label smoothness assists edge-weight learning.
Conclusion: Experiments report that KGNN-LS outperforms state-of-the-art baselines in four recommendation scenarios and scales desirably with KG size.The four scenarios are movie, book, music, and restaurant recommendation.
Future Work: Future work will examine label smoothness on link prediction and node classification and investigate its theoretical relationship with feature propagation.These directions are presented as extensions beyond the recommendation task.

A Additional Details on Datasets

The datasets combine public movie, book, and music interactions with a Meituan-Dianping restaurant dataset, using knowledge graphs assembled from Satori or an internal company toolkit.

Interaction Data: MovieLens-20M, Book-Crossing, and Last.FM use explicit feedback converted to implicit positive interactions.MovieLens positives use a rating threshold of 4, while Book-Crossing and Last.FM use no threshold because of sparsity.
Knowledge Graph Construction: Satori supplies KGs for MovieLens-20M, Book-Crossing, and Last.FM, while Dianping-Food uses a KG constructed by Meituan-Dianping’s internal toolkit.The paper describes the KGs as sub-KGs for the first three datasets.
Knowledge Graph Construction: Satori triples contain head and tail entities or textual content connected by typed relations such as “book.book.author.”Entities are matched to dataset items through name-related triples, with ambiguous or unmatched items excluded.
Dianping-Food: Dianping-Food contains approximately 10 million user–restaurant interactions collected from May 1, 2015 to December 12, 2018.Positive interactions include clicking, buying, and adding restaurants to favorites; negatives are sampled per user.

B Additional Details on Hyper-parameter Searching

KGNN-LS searches key architectural, regularization, and optimization hyperparameters across specified grids, using fixed-size neighbor sampling for efficiency. Experiments use 6:2:2 data splits, five repetitions, averaged performance, and Adam optimization.

KGNN-LS uniformly samples S neighbors per entity to make computation more efficient despite varying knowledge-graph neighborhood sizes.The sampled-neighbor size S is selected from {2, 4, 8, 16, 32}.
The search varies hidden-layer dimension d over {4, 8, 16, 32, 64, 128} and layer count L over {1, 2, 3, 4}.The implementation uses ReLU in non-last layers and tanh in the last layer.
The label smoothness weight λ is searched over {0, 0.01, 0.1, 0.5, 1, 5}, while the L2 regularizer weight γ uses the listed 10^-9-to-10^-3 grid.Table 8 identifies λ as the label smoothness regularizer weight and γ as the L2 regularizer weight.
The learning rate η is searched from 10^-5 through 10^-1 using the specified 13-value grid.All trainable parameters are optimized with Adam.
Each dataset uses a 6:2:2 training-validation-test split, and each experiment is repeated five times with average performance reported.The implementation is described for four datasets.