Source-linked AI summary
Distilling Knowledge from Graph Convolutional Networks
Yiding Yang, Jiayan Qiu, Mingli Song, Dacheng Tao, Xinchao Wang
TL;DR
Existing knowledge distillation has focused on CNNs with grid-domain inputs, leaving GCNs for non-grid data comparatively overlooked. The paper introduces a local structure preserving approach that transfers teacher topology through matched local-structure distributions, and reports state-of-the-art performance across two datasets and GCN architectures.
Problem
Existing knowledge distillation methods focus on CNNs with grid-domain inputs and largely overlook GCNs that handle non-grid data.
Method
The method preserves teacher local structure by representing teacher and student local structures as distributions and matching them during distillation.
Results
The method achieves state-of-the-art knowledge distillation performance across two datasets and two GCN architectures, consistently outperforming existing methods.
Takeaways & Limitations
Topology-aware local structure preservation transfers graph information to compact student GCNs and extends to dynamic graph models.
Takeaways & Limitations
For dynamic graphs, directly computing local-structure vectors is meaningless when teacher and student neighbor nodes appear in different orders.
Abstract
from arXiv · showhide
Existing knowledge distillation methods focus on convolutional neural networks (CNNs), where the input samples like images lie in a grid domain, and have largely overlooked graph convolutional networks (GCN) that handle non-grid data. In this paper, we propose to our best knowledge the first dedicated approach to distilling knowledge from a pre-trained GCN model. To enable the knowledge transfer from the teacher GCN to the student, we propose a local structure preserving module that explicitly accounts for the topological semantics of the teacher. In this module, the local structure information from both the teacher and the student are extracted as distributions, and hence minimizing the distance between these distributions enables topology-aware knowledge transfer from the teacher, yielding a compact yet high-performance student model. Moreover, the proposed approach is readily extendable to dynamic graph models, where the input graphs for the teacher and the student may differ. We evaluate the proposed method on two different datasets using GCN models of different architectures, and demonstrate that our method achieves the state-of-the-art knowledge distillation performance for GCN models. Code is publicly available at https://github.com/ihollywhy/DistillGCN.PyTorch.
1. Introduction
The paper addresses the gap between CNN-focused knowledge distillation and GCNs operating on non-grid data by introducing topology-aware student training. Its local structure preserving method transfers teacher graph semantics and consistently outperforms compared methods across tasks and architectures.
- Motivation: Existing distillation methods focus on CNNs with grid-domain inputs, while GCNs handle non-grid data and explicitly model graph topology.Conventional methods account mainly for outputs or intermediate activations and omit topological information.
- Contribution: The paper introduces the first dedicated knowledge distillation approach tailored for GCN models.The goal is to train a smaller student with fewer layers, lower-dimensional feature maps, or fewer graph edges.
- Method: The local structure preserving method measures similarity between teacher and student local topological structures to transfer graphical semantics.It uses node features and graph connections from both models to guide the student toward a similar topological embedding.
- Evaluation: The approach is evaluated on Protein-Protein Interaction node classification and ModelNet40 3D object recognition using different GCN architectures.These experiments cover two tasks from different domains.
- Results: The method consistently outperforms all compared methods across the evaluated tasks and GCN architectures.The paper reports this result as evidence of the method’s effectiveness and generalization.
2. Related Work
The related work covers knowledge distillation and graph convolutional networks, highlighting that prior distillation strategies were designed mainly for grid-domain models. The proposed approach targets GCNs and preserves local structure by matching teacher and student distributions.
- Knowledge Distillation: Prior knowledge distillation methods compress large teacher models into smaller students by transferring output or intermediate-activation knowledge.These strategies were designed primarily for deep convolutional networks with grid-domain inputs.
- Proposed Direction: The proposed method is presented as the first knowledge-distillation attempt focused on graph convolutional networks.Its local structure preserving module computes local-structure distributions and matches teacher and student distributions.
- Knowledge Amalgamation: Knowledge amalgamation trains one student from multiple teachers, but the reviewed methods are usually limited to grid-domain models.Examples include shared backbones, layer-wise neuron sharing, and common feature spaces.
- Graph Convolutional Networks: GCNs model non-grid data as nodes with features and graphs representing relationships among nodes.They explicitly exploit graph structure, making them suitable for data such as point sets and other relational inputs.
3. Method
The method distills a teacher GCN into a smaller student by preserving local topological structure, using distributions over node neighborhoods and a divergence-based loss. It extends this transfer to dynamic graphs by aligning differing neighborhood structures through virtual edges.
- 3.1. Graph Convolutional Network: GCNs process non-grid data represented by node features and graph edges, aggregating neighboring features to produce each center node’s representation.The aggregation strategy combines pairwise feature transformations with neighbor information.
- 3.2. Motivation: The distillation objective transfers the teacher’s learned aggregation-related information because directly representing its aggregation function is difficult.The method therefore targets the topological information embedded in intermediate representations rather than copying the aggregation function itself.
- 3.3. Local Structure Preserving: LSP represents each node’s local structure as a normalized distribution over neighboring nodes, with dimensionality determined by the center node’s degree.Pairwise feature similarity is exponentiated and normalized across nodes pointing to the center node.
- 3.3. Local Structure Preserving: For teacher and student feature maps, LSP compares corresponding local-structure distributions using Kullback-Leibler divergence and averages the resulting discrepancies across nodes.A smaller per-node divergence indicates more similar local-structure distributions; the total objective also includes student cross-entropy weighted by λ.
- 3.4. Kernel Function: Kernel functions address the assumption that pairwise feature similarity is proportional to Euclidean distance by mapping vectors implicitly to a higher-dimensional space.The paper compares linear, polynomial, RBF, and L2-norm similarity choices, with specified polynomial and RBF settings.
- 3.5. Dynamic Graph: Dynamic graphs reconstruct neighbors from learned features, so the method adds virtual edges from the teacher–student graph union before computing comparable local structures.This permits distillation across different neighbor distributions and neighborhood sizes, including a teacher with larger K and a student with smaller K.
4. Experiments
Experiments evaluate the proposed GCN distillation method across node classification and 3D object recognition, using different architectures and model configurations. The method generally transfers local or dynamic graph structure effectively, producing strong student models and matching or exceeding comparison methods.
- Experimental Setup: Experiments use PPI node classification and ModelNet40 3D object recognition with different GCN architectures.GAT is used for PPI, while DGCNN with dynamic graphs is used for ModelNet40.
- Node Classification: On PPI, competing distillation methods reduce performance relative to the original loss, whereas the proposed method produces the best student performance.The authors attribute this result to transferring local structure information.
- 3D Object Recognition: On ModelNet40, the proposed method generates the best student model for both accuracy and mean class accuracy, including under dynamic graph inputs.A student with a fixed percentage of the teacher’s parameters achieves similar performance to the teacher.
- 3D Object Recognition: The proposed method guides students toward teacher-like local feature-space structure during the very early training stage.Figure 4 compares student features trained with the proposed method or cross-entropy loss across epochs against the teacher representation.
- Ablation and Performance Studies: RBF provides the best performance among the tested kernel functions, while increasing student channels can approach teacher accuracy with fewer parameters.The ablations examine kernel choices and model-complexity trade-offs.
5. Conclusion
The paper presents a GCN-specific distillation approach that preserves teacher local structure through distribution matching. Experiments across datasets and architectures report state-of-the-art distillation performance and extension to dynamic graphs.
- 5. Conclusion: The proposed method preserves teacher local structure by matching distributions over similarities between a center node and its neighbors.This converts local-structure preservation into distribution matching over intermediate feature maps.
- 5. Conclusion: Experiments on two datasets and two GCN architectures report state-of-the-art distillation performance over existing knowledge distillation methods.The approach is also described as readily extendable to dynamic graph models.