PaperlayerView exact revision on arXiv

Source-linked AI summary

Graph-less Neural Networks: Teaching Old MLPs New Tricks via Distillation

Shichang Zhang, Yozen Liu, Yizhou Sun, Neil Shah

arXiv:2110.08727v2cs.LGcs.AI

TL;DR

GNNs offer strong node-classification accuracy but their neighbor-dependent inference limits practical deployment, whereas MLPs are faster but typically less accurate. This paper distills GNN knowledge into MLPs, producing GLNNs without inference graph dependency. Across 7 datasets, GLNNs substantially improve over stand-alone MLPs, match GNNs on 6/7 datasets, and achieve large speedups.

  • Problem

    GNN graph dependency creates neighbor-fetching latency that limits deployment in latency-constrained applications, while MLPs are faster but generally less accurate for node classification.

  • Method

    GLNN distills a trained GNN teacher into a student MLP, moving graph-dependent computation to offline training and eliminating graph dependency during inference.

  • Results

    GLNNs improve MLP accuracy by 12.36% on average across 7 datasets, match GNNs on 6/7 datasets, and infer 146×-273× faster than GNNs.

  • Takeaways & Limitations

    GLNNs can be a practical choice for deploying competitive graph-learning models when inference latency and graph-independent deployment matter.

  • Takeaways & Limitations

    GLNN performance can fail when node features contain little information about labels and is not competitive for inductive performance on Arxiv in the current version.

Abstract

from arXiv · show

Graph Neural Networks (GNNs) are popular for graph machine learning and have shown great results on wide node classification tasks. Yet, they are less popular for practical deployments in the industry owing to their scalability challenges incurred by data dependency. Namely, GNN inference depends on neighbor nodes multiple hops away from the target, and fetching them burdens latency-constrained applications. Existing inference acceleration methods like pruning and quantization can speed up GNNs by reducing Multiplication-and-ACcumulation (MAC) operations, but the improvements are limited given the data dependency is not resolved. Conversely, multi-layer perceptrons (MLPs) have no graph dependency and infer much faster than GNNs, even though they are less accurate than GNNs for node classification in general. Motivated by these complementary strengths and weaknesses, we bring GNNs and MLPs together via knowledge distillation (KD). Our work shows that the performance of MLPs can be improved by large margins with GNN KD. We call the distilled MLPs Graph-less Neural Networks (GLNNs) as they have no inference graph dependency. We show that GLNNs with competitive accuracy infer faster than GNNs by 146X-273X and faster than other acceleration methods by 14X-27X. Under a production setting involving both transductive and inductive predictions across 7 datasets, GLNN accuracies improve over stand-alone MLPs by 12.36% on average and match GNNs on 6/7 datasets. Comprehensive analysis shows when and why GLNNs can achieve competitive accuracies to GNNs and suggests GLNN as a handy choice for latency-constrained applications.

1 INTRODUCTION

GNNs perform well on node classification but face deployment challenges from graph-dependent inference, while MLPs are faster yet typically less accurate. GLNN addresses this trade-off by distilling a GNN teacher into an MLP student.

  • Motivation: GNNs achieve strong node-classification results, but graph-data dependency creates scalability and deployment challenges for latency-constrained applications.Neighbor fetching contributes substantially to inference latency, while MLPs remain widely used in industry because they are easier to deploy.
  • Motivation: MLPs avoid graph dependency and neighbor-fetching latency, but their lack of graph topology typically limits node-classification performance.MLPs can also sidestep cold-start issues when neighbor information for a new node is unavailable.
  • Present work: GLNN distills a trained GNN teacher into a student MLP, shifting graph-dependent work to offline training while preserving graph-independent inference.The resulting MLP is optimized through knowledge distillation and deployed without graph dependency.
  • Present work: 146×-273× faster inference than GNNs and 14×-27× faster than other acceleration methods are reported for GLNNs with competitive performance.The reported comparison addresses inference efficiency while retaining competitive node-classification accuracy.
  • Present work: 12.36% average accuracy improvement over MLPs was observed across 7 datasets, with GLNNs matching GNNs on 6/7 datasets.The evaluation used a production setting containing both transductive and inductive predictions.

2 RELATED WORK

Prior work accelerates or distills GNNs but generally retains graph-related inference costs. GLNN instead targets neighbor-fetching latency through cross-model distillation to an MLP.

  • Graph Neural Networks: GNN research commonly uses message-passing architectures such as GCN, GAT, PPNP, GCNII, and DeeperGCN.These models differ in mechanisms including aggregation, attention, personalized PageRank, residual connections, and dense connections.
  • Inference Acceleration: Pruning and quantization reduce GNN parameters or MAC operations but do not eliminate neighbor-fetching latency.Consequently, their speed gains on GNNs are less significant than on neural networks without graph dependency.
  • GNN distillation: Existing GNN knowledge-distillation methods usually produce smaller GNN students that still require latency-inducing neighbor fetching.Other approaches, such as graph-level distillation, address different graph settings than dependent nodes within one graph.
  • Inference Acceleration: Figure 1 compares GNN and MLP fetch counts and inference time as GNN depth varies during inductive inference on OGB Products.The figure provides a direct latency comparison relevant to the paper’s motivation for graph-independent students.

3 PRELIMINARIES

The paper represents graph-learning inputs as a graph with node features and defines node-classification targets for labeled and unlabeled nodes. GNNs update node representations by aggregating neighbor messages.

  • Notations: A graph is represented as G = (V, E), with node features X and adjacency matrix A encoding edges.N denotes the total number of nodes, and each node feature is D-dimensional.
  • Notations: Node-classification targets are represented as Y ∈ R^N×K, with each row giving a K-dimensional one-hot label vector.Superscripts L and U distinguish labeled and unlabeled nodes, respectively.
  • Graph Neural Networks: Most GNNs use message passing, iteratively updating each node representation by collecting messages from its neighbors.The update proceeds through aggregation and update operations at each layer.

4 MOTIVATION

GNN inference becomes costly because deeper layers require fetching increasingly distant neighbors, whereas MLPs are faster but cannot directly exploit graph topology. GLNN combines these properties through offline distillation.

  • Motivation: A L-layer GNN on a graph with average degree R requires O(R^L) fetches to infer a node.Sequential layer fetching causes latency to grow rapidly with depth, while MLP inference grows linearly.
  • Motivation: Industrial graph scale and increasing GNN depth intensify node-fetching latency and can require graph sharding across machines.The paper cites Twitter and Facebook as examples of graphs too large for a single machine’s memory.
  • Motivation: 78.61 test accuracy for GraphSAGE versus 62.47 for an equal-sized MLP on Products illustrates the MLP accuracy gap.The paper uses this trade-off to motivate combining GNN accuracy with MLP latency.
  • GLNN framework: GLNN uses a trained GNN to generate soft targets offline, trains an MLP on node features, and deploys the distilled MLP for online predictions.The deployed model has no graph dependency during inference.

5 GRAPH-LESS NEURAL NETWORKS

GLNN distills a teacher GNN into an MLP, shifting graph-dependent work offline while retaining fast, graph-independent inference. Across transductive and production evaluations, enlarged GLNNs improve substantially over MLPs and often approach or match GNN accuracy, with graph-topology knowledge and regularization contributing to their performance.

  • Method: GLNN trains an MLP student from GNN-generated soft targets alongside true labels, then performs graph-independent inference.Knowledge distillation occurs offline during training, while the resulting model remains essentially an MLP at inference.
  • Transductive evaluation: Under transductive evaluation, GLNNs improve over MLPs by large margins and match GNNs on 5 of 7 datasets.On smaller datasets, GLNNs can outperform their teacher GNNs; on larger OGB datasets, enlarging the MLP mitigates the remaining gap.
  • Model scaling: Enlarged GLNNs match GNN performance on the OGB datasets, using GLNNw4 for Arxiv and GLNNw8 for Products.Increasing MLP width moves accuracy closer to SAGE while preserving the MLP architecture.
  • Production evaluation: In production evaluation combining inductive and transductive predictions, GLNNs are competitive with GNNs on 6/7 datasets and improve over MLPs for inductive predictions.Arxiv is the main exception, where distribution shift makes graph information harder to capture without neighbor features; GLNN still substantially improves over MLP.
  • Training behavior: Distillation can regularize training: directly trained MLPs show a larger training–validation loss gap and clearer overfitting than GLNNs.Figure 4 reports hard-label training loss for GLNN, corresponding to the first term of the distillation objective.
  • Topology and inductive bias: GLNN predictions show greater graph-topology consistency than MLP predictions, with average Lcut values of 0.8986 versus 0.7644.The teacher SAGE average is 0.9221, indicating that GLNN outputs retain much of the topology-related knowledge transferred through distillation.

6 ABLATION STUDIES

Ablations examine GLNN robustness to noisy features, inductive split rates, and teacher GNN architectures across five CPF datasets.

  • Study scope: The ablations report test accuracies averaged over five CPF datasets and also examine node-feature noise, inductive split rates, and teacher architecture.Additional experiments cover advanced teachers, a GA-MLP student, and non-homogeneous data.
  • Noisy node features: As feature noise increases, MLP and GLNN accuracy declines faster than GNN accuracy, although GLNN and GNN remain comparable at small noise levels.At α = 1, noisy features and labels become independent.
  • Inductive split rate: As the inductive portion increases, GNN and MLP performance remains roughly stable while GLNN inductive performance drops slightly.The study considers split rates up to 50-50 because larger inductive portions are described as highly atypical in practice.
  • Teacher GNN architecture: GLNNs improve over MLPs when distilled from SAGE, GCN, GAT, or APPNP teachers, with APPNP producing slightly worse performance than the other teachers.The paper suggests APPNP transfers less additional information because its first step uses the node’s own features, similar to the student MLP.

7 CONCLUSION AND FUTURE WORK

The paper explores combining GNN accuracy with MLP speed through distillation. It finds GLNNs are much faster than GNNs while generally maintaining competitive performance, but inductive performance remains noncompetitive on Arxiv.

  • Conclusion: KD from GNNs to MLPs removes inference graph dependency, producing GLNNs that are 146×-273× faster than GNNs with competitive performance.The evaluation spans seven datasets across different domains.
  • Conclusion: GLNNs are presented as a potential choice for deploying latency-constrained models.This conclusion follows the reported speed and accuracy results across seven datasets.
  • Future work: The current GLNN version does not show competitive inductive performance on the Arxiv dataset.The authors identify more advanced distillation techniques as a possible direction for improving GLNN performance.

A.1 DATASETS

The paper evaluates GLNNs across citation and other graph datasets under transductive and inductive node-classification settings. Experiments use matched teacher–student parameter budgets and examine accuracy, complexity, expressiveness, and deployment trade-offs.

  • Datasets: Four datasets are citation graphs whose node features represent paper descriptions using bag-of-word, TF-IDF, or word-embedding vectors.
  • Datasets: The experiments follow established dataset-specific splits, including CPF strategies for smaller datasets and official time- or popularity-based OGB splits.
  • Transductive and inductive settings: Transductive prediction evaluates nodes within the graph, whereas inductive prediction evaluates unseen nodes after holding out their nodes and connected edges.
  • Transductive and inductive settings: In the transductive setting, soft targets are generated using all graph nodes, including validation and test nodes, to match standard GNN evaluation and support fair comparison.
  • Complexity and expressiveness: GNNs incur graph-loading and neighbor-dependent inference costs, whereas MLPs avoid graph dependency; GLNNs therefore expose an accuracy–time trade-off under comparable parameter budgets.
  • Results and analysis: On smaller datasets, equal-sized GLNNs can outperform teacher GNNs, while larger OGB datasets require increased MLP size to mitigate remaining gaps.

F ADVANCED GNN ARCHITECTURES AS THE TEACHER

The study tests whether stronger teachers improve GLNN performance beyond the default SAGE teacher. It evaluates GLNN+ using a high-performing MLP+C&S teacher on Products.

  • Teacher selection: The main experiments use SAGE teachers to avoid confounding GLNN performance with teacher architecture.
  • Teacher selection: On Products, GLNN+ uses the leaderboard MLP+C&S model as teacher, whose reported accuracy is 84.18% and rank is #8.

G GLNN WITH FEATURE AUGMENTATION FROM ONE-HOP NEIGHBORS

The paper augments node features with one-hop neighbor information to improve GLNN performance in the challenging inductive Arxiv setting. This retains an MLP student while adding limited graph information and preserves a substantial speed advantage over GNN inference.

  • Motivation: The inductive GLNN result on Arxiv is less desirable, motivating one-hop neighbor-feature augmentation as a middle ground between pure GLNNs and GNNs.
  • Method: 1-hop GA-MLP and 1-hop GA-GLNN concatenate one-hop neighbor features with each node’s raw features before training, with the latter also using GNN distillation.
  • Results: 68.83 accuracy for 1-hop GA-GLNN exceeds 66.62 for 1-hop GA-MLP and nearly matches the 70.64 teacher.
  • Results: 29.31ms for a one-layer GNN versus 7.56ms for GLNN indicates roughly a fourfold speed difference, giving practitioners an accuracy–inference-time trade-off.
  • Evaluation: The split-rate study varies inductive fractions from 10:90 to 90:10 and reports inductive and transductive performance separately across datasets.

I GLNN UNDER NODE FEATURE HETEROGENEITY AND NON-HOMOPHILY

Additional experiments test GLNNs on heterogeneous node features and non-homophilous graphs. GLNNs improve over MLPs and can approach or exceed graph-based baselines, but struggle when labels depend mainly on graph structure.

  • Datasets: Four additional datasets broaden evaluation beyond the main benchmark, including heterogeneous-feature graphs and non-homophilous Penn94 and Pokec.
  • Heterogeneous node features: On House_class and VK_class, GLNN improves over MLP, outperforms GNN and LightGBM, and becomes competitive with the teacher BGNN.
  • Non-homophilous graphs: On Penn94, GLNN improves over MLP and becomes competitive with the teacher GCN.
  • Non-homophilous graphs: On Pokec, LINK outperforms GLNN because labels can be largely determined by graph structure without node features.

J MODEL COMPARISON WITH NOISY NODE FEATURES

With completely random node features, GNNs remain strong through graph structure, while GLNNs outperform MLPs because distillation captures label imbalance.

  • GLNN accuracy remains higher than MLP accuracy when node features are completely random.The comparison is explicitly reported for the noisy-feature ablation.
  • GNNs can perform well on random features because same-label nodes are likely connected, enabling an overfit classifier to use neighborhood information.A test node can be classified according to the labels of many connected training nodes from which it aggregates features.
  • On the A-computer dataset, inductive accuracy with random features is 0.0652 for MLP and 0.2538 for GLNN.The task has 10 classes, for which uniform labels would imply accuracy near 0.1.
  • GLNN learns the imbalanced inductive label distribution from soft labels, whereas MLP training uses uniformly sampled nodes and produces relatively uniform predictions.Class 4 comprises about 40% of the inductive test set, while the largest MLP-predicted class comprises about 17.5%.
  • Figure 9 compares true inductive labels with MLP and GLNN predicted-label distributions on the A-computer dataset.
Loading 2110.08727v2…