Extract the Knowledge of Graph Neural Networks and Go Beyond it: An Effective Knowledge Distillation Framework

Cheng Yang; Jiawei Liu; Chuan Shi

Source-linked AI summary

Extract the Knowledge of Graph Neural Networks and Go Beyond it: An Effective Knowledge Distillation Framework

Cheng Yang, Jiawei Liu, Chuan Shi

arXiv:2103.02885v1Mar 4, 2021cs.LGcs.SI

TL;DR

GNNs provide effective graph classification but can entangle topology, features, and projections in ways that underuse structural and feature-based priors. The paper distills arbitrary GNN teachers into a student combining label propagation and feature transformation, improving teacher accuracy across benchmarks while providing a more interpretable process.

Problem
GNN prediction mechanisms can underuse structural and feature-based prior knowledge, while existing remedies remain limited in architecture compatibility and prior integration.
Method
The framework distills an arbitrary pretrained GNN into a student that combines parameterized label propagation with a feature-based 2-layer MLP and uses teacher soft predictions.
Results
1.4%–4.7% accuracy gains over corresponding teachers are reported across five benchmark datasets and seven GNN teacher models.
Takeaways & Limitations
The learned student uses both GNN and prior knowledge while offering a more interpretable prediction process than GNN teachers.
Takeaways & Limitations
The framework’s demonstrated scope is semi-supervised node classification; extending it to tasks such as unsupervised node clustering remains future work because label propagation cannot be applied without labels.

Abstract

from arXiv · show

Semi-supervised learning on graphs is an important problem in the machine learning area. In recent years, state-of-the-art classification methods based on graph neural networks (GNNs) have shown their superiority over traditional ones such as label propagation. However, the sophisticated architectures of these neural models will lead to a complex prediction mechanism, which could not make full use of valuable prior knowledge lying in the data, e.g., structurally correlated nodes tend to have the same class. In this paper, we propose a framework based on knowledge distillation to address the above issues. Our framework extracts the knowledge of an arbitrary learned GNN model (teacher model), and injects it into a well-designed student model. The student model is built with two simple prediction mechanisms, i.e., label propagation and feature transformation, which naturally preserves structure-based and feature-based prior knowledge, respectively. In specific, we design the student model as a trainable combination of parameterized label propagation and feature transformation modules. As a result, the learned student can benefit from both prior knowledge and the knowledge in GNN teachers for more effective predictions. Moreover, the learned student model has a more interpretable prediction process than GNNs. We conduct experiments on five public benchmark datasets and employ seven GNN models including GCN, GAT, APPNP, SAGE, SGC, GCNII and GLP as the teacher models. Experimental results show that the learned student model can consistently outperform its corresponding teacher model by 1.4% - 4.7% on average. Code and data are available at https://github.com/BUPT-GAMMA/CPF

1 INTRODUCTION

The paper addresses limitations of GNN prediction mechanisms by distilling arbitrary GNN teachers into a student that combines label propagation and feature transformation. Across five benchmarks, the framework improves teacher accuracy while retaining a more interpretable prediction process.

Motivation: Graph semi-supervised learning classifies unlabeled nodes using graph structure, labeled nodes, and available node features.Many applications exhibit homophily, where linked nodes tend to share labels.
Motivation: GNNs use message passing to combine neighborhood information with node features, but their entangled mechanisms can underuse prior knowledge.The paper specifically identifies homophily-based structure priors as underused in GCNs.
Motivation: Existing GNN-label-propagation approaches remain tied to GNN bodies, lack broad architectural compatibility, and omit feature-based prior knowledge.Feature-based prior knowledge treats a node’s label as determined purely by its own features.
Approach: The proposed framework distills an arbitrary learned GNN into a student combining parameterized label propagation with a feature-based 2-layer MLP.The student simulates teacher soft labels while preserving structure-based and feature-based priors.
Results: 1.4%–4.7% accuracy gains over corresponding teachers are reported across five benchmark datasets and seven GNN teacher models.The framework also yields 1.5%–2.3% relative improvements when applied to GLP.
Results: The learned student offers a more interpretable prediction process while utilizing both GNN knowledge and structure/feature-based priors.The paper reports interpretability analyses based on learned balance parameters and node confidence scores.

2 RELATED WORK

Related work covers GNNs, label-propagation hybrids, and knowledge distillation. The paper distinguishes its framework by using a flexible, simple student to surpass arbitrary GNN teachers without ensembling or iterative teacher–student distillation.

Graph Neural Networks: GNN research includes models such as GCN, GAT, and SAGE, which differ in how they propagate or aggregate neighborhood features.These models are used as teacher architectures in the paper’s experiments.
Label Propagation and GNNs: Recent methods improve GNNs by incorporating label propagation through graph filters, shared message passing, or regularization.Examples include GLP, UniMP, and GCN-LPA.
Knowledge Distillation: Knowledge distillation transfers a teacher’s soft predictions into a student, traditionally to compress models while retaining prediction quality.The approach has been widely used for compressing deep convolutional networks.
GNN Distillation: Prior GNN distillation methods compress architectures, ensemble students, or use iterative teacher–student training for improved performance.These approaches differ in motivation and architecture from the proposed framework.
Positioning: The proposed framework applies to arbitrary GNNs, combines GNN knowledge with structural and feature priors, and produces a more interpretable student without ensembling or iterative distillation.Its reported goal is to extract GNN knowledge and go beyond the teacher.

3 METHODOLOGY

The methodology formalizes knowledge distillation from GNN teachers and constructs a student that combines parameterized label propagation with feature-based transformation, followed by interpretability and complexity analysis.

Methodology: The methodology introduces the semi-supervised node-classification problem, the knowledge-distillation framework, the student architecture, and analyses of interpretability and computational complexity.The student architecture uses parameterized label propagation and a feature-based 2-layer MLP.

3.1 Semi-supervised Node Classification

Semi-supervised node classification predicts labels for unlabeled nodes in a connected graph using labeled nodes, graph structure, and node features.

Problem Formulation: Given a connected graph with labeled and unlabeled vertices, node classification predicts labels for every vertex in the unlabeled set.The graph consists of a vertex set and an edge set, while labels come from a set of possible classes.
Problem Formulation: Each node may have a d-dimensional feature vector represented as a row of the node-feature matrix X.These features can be used to improve classification accuracy.

3.2 The Knowledge Distillation Framework

The framework treats node classification as a black-box prediction problem and distills a pretrained GNN teacher’s soft predictions into a parameterized student. The student is optimized to align its predicted probability distributions with those of the teacher.

Node classification predicts labels for unlabeled nodes from graph structure, labeled nodes, and node features.
The teacher can be any pretrained GNN, while the student represents each node with a parameterized label-probability distribution.
Knowledge distillation trains the student to mimic the teacher’s soft label predictions, thereby transferring knowledge between models.
The training objective minimizes the distance between student and teacher probability distributions, using Euclidean distance in this work.

3.3 The Architecture of Student Model

The student combines label propagation and feature transformation to preserve structure-based and feature-based prior knowledge. Its trainable components extend label propagation and balance both mechanisms during prediction.

The student combines label propagation from neighboring nodes with transformation of each node’s own features.
Classical label propagation assumes linked nodes are likely to share labels and propagates labeled-node information to unlabeled nodes.
Unlabeled nodes begin with uniform label distributions, while labeled nodes retain one-hot labels during propagation.
Parameterized Label Propagation gives neighbors trainable, confidence-dependent edge weights instead of treating all neighbors equally.
The Feature Transformation module uses a 2-layer MLP followed by softmax to predict labels from raw node features.
CPF learns a node-specific α_v ∈ [0, 1] to balance PLP and FT predictions at each propagation step.

3.4 The Overall Algorithm and Details

The overall procedure trains the CPF student on a graph with labeled and unlabeled nodes, using a pretrained GNN teacher and an objective optimized over the student’s propagation, confidence, and feature-transformation parameters.

The distillation objective is expanded across the student’s K propagation layers.
The parameter set includes PLP–FT balancing parameters, PLP confidence parameters, and the FT MLP parameters.
The algorithm takes graph structure, labeled and unlabeled node sets, node features, and a pretrained GNN classifier as input.
Training initializes CPF predictions, iterates until convergence, computes edge weights, and updates parameters by optimizing the distillation objective.
The framework supports inductive and transductive settings through alternative confidence-score parameterizations.

3.5 Discussions on Interpretability and Complexity

CPF offers an interpretable prediction process by exposing the balance between label propagation and feature-based prediction. Its per-iteration time and space costs are linear in graph size and feature dimension.

CPF predicts each node’s label as a weighted average of label-propagation and feature-based MLP predictions.
The balance parameter α_v indicates whether structure-based propagation or feature-based prediction is more important for node v.
The label-propagation mechanism exposes which neighbors influence a node and by how much at each iteration.
Each training iteration and the full algorithm require O(|E| + d|V|) time and space, linear in dataset scale.

4 EXPERIMENTS

Experiments evaluate the framework across five benchmark datasets, seven GNN teachers, and multiple student variants. The full CPF students consistently improve teacher performance, remain robust across propagation depths and labeling ratios, and reveal interpretable roles for their learned parameters.

Experimental setup: Experiments use five public benchmark datasets and seven GNN models as teacher models, comparing four student variants.The variants are PLP, FT, CPF-ind, and CPF-tra.
Classification results: The full CPF-ind and CPF-tra students generally perform best, improving corresponding teachers across datasets and teacher architectures.The main exception is the APPNP teacher on Pubmed.
Classification results: 1.5% ∼2.3% relative improvements remain when the framework is applied to GLP, which already incorporates label propagation.This result supports compatibility with a teacher that already uses a label-propagation mechanism.
Classification results: 2.9/4.2/2.7/2.1/2.7% are the average relative improvements on Cora, Citeseer, Pubmed, A-Computers, and A-Photo, respectively.Citeseer benefits most among the five datasets.
Robustness analysis: 0.56% and 0.84% are the maximum accuracy gaps across propagation depths for CPF-ind and CPF-tra, respectively, while even the worst choices outperform teachers.The tested propagation-layer range is K∈{5, 6, 7, 8, 9, 10}.
Student variants: Replacing the 2-layer MLP with linear regression still improves teachers by 0.3% ∼2.3% on average, but with weaker performance.The simpler alternative may offer better interpretability at a performance cost.
Robustness analysis: 4.9/4.5/3.2/2.1% are the average relative accuracy improvements with 5/10/20/50 labeled nodes per class, respectively.The largest improvement occurs in the five-label few-shot setting.
Interpretability analysis: Learned balance and confidence parameters provide interpretable signals: larger balance values align with similar predicted neighbors, while higher confidence produces stronger label propagation and self-retention.The confidence analysis also associates high-confidence nodes with relatively small degree and same-predicted neighbors.

5 CONCLUSION

The paper presents CPF as a knowledge distillation framework that combines label propagation and feature transformation to transfer GNN knowledge into an interpretable student model. Experiments show consistent improvements across seven GNN teachers, while future work extends the framework beyond semi-supervised node classification.

CPF combines label propagation and feature transformation to preserve structure-based and feature-based prior knowledge while distilling knowledge from pretrained GNN teachers.
The learned student improves classification accuracy consistently across all seven GNN teacher models on five benchmark datasets and provides a more interpretable prediction process.
Future work will investigate applications beyond semi-supervised node classification, including unsupervised node clustering where label propagation cannot be applied without labels.

A DETAILS FOR REPRODUCIBILITY

The appendix documents teacher-model training settings for reproducibility. Settings are drawn from prior work for five classical GNNs and from original papers for GCNII and GLP.

The appendix provides additional teacher-model experimental settings to support reproducibility.
Training settings for five classical GNNs are taken from paper [23].
GCNII and GLP settings follow their respective original papers.
Specified settings include hidden-layer sizes, learning rates, dropout probabilities, attention dropout, sampling, batch sizes, and learning-rate decay.