Source-linked AI summary
Pitfalls of Graph Neural Network Evaluation
Oleksandr Shchur, Maximilian Mumme, Aleksandar Bojchevski, Stephan Günnemann
TL;DR
The paper examines whether common evaluation practices support fair comparisons among GNN architectures for semi-supervised node classification. It standardizes training and hyperparameter selection, evaluates models across many splits and initializations, and finds that rankings are fragile while simple GCN can outperform more sophisticated architectures. The authors conclude that robust evaluation procedures are needed.
Problem
Existing GNN evaluation practices make it difficult to judge architectural progress fairly because fixed splits and differing training procedures can confound comparisons.
Method
The authors evaluate four GNN architectures with standardized training and hyperparameter selection across eight datasets, using 100 splits and 20 random initializations per split.
Results
Across multiple splits and fair tuning, simple GCN can outperform more sophisticated GNN architectures, while rankings from different data splits can differ substantially.
Takeaways & Limitations
GNN comparisons should use robust evaluation procedures that average results across multiple data splits and apply equivalent training and tuning procedures.
Takeaways & Limitations
Standard deviations provide only a rough view of variance because accuracy scores are not normally distributed; box plots provide a more accurate picture.
Abstract
from arXiv · showhide
Semi-supervised node classification in graphs is a fundamental problem in graph mining, and the recently proposed graph neural networks (GNNs) have achieved unparalleled results on this task. Due to their massive success, GNNs have attracted a lot of attention, and many novel architectures have been put forward. In this paper we show that existing evaluation strategies for GNN models have serious shortcomings. We show that using the same train/validation/test splits of the same datasets, as well as making significant changes to the training procedure (e.g. early stopping criteria) precludes a fair comparison of different architectures. We perform a thorough empirical evaluation of four prominent GNN models and show that considering different splits of the data leads to dramatically different rankings of models. Even more importantly, our findings suggest that simpler GNN architectures are able to outperform the more sophisticated ones if the hyperparameters and the training procedure are tuned fairly for all models.
1 Introduction
The paper argues that standard GNN evaluation practices make progress difficult to judge fairly. It addresses this by standardizing training and hyperparameter selection and evaluating across many data splits and initializations.
- Using the same three datasets and fixed splits can favor models that overfit those test sets rather than generalize.
- Different training procedures and hyperparameter choices make it difficult to separate architectural improvements from tuning advantages.
- The study evaluates four prominent GNN architectures within a common framework using standardized training and hyperparameter selection.
- Experiments use four established citation datasets and four newly introduced node-classification datasets.
- Each dataset is evaluated with 100 random train/validation/test splits and 20 random initializations per split.
- Thorough empirical evaluation is presented as important for understanding models' strengths and limitations, while benchmark accuracy is not treated as the only research objective.
2 Models
The paper compares four GNN architectures and four complementary baseline models for transductive semi-supervised node classification. It emphasizes that differing training procedures can obscure whether performance gains arise from architecture or tuning.
- GCN, MoNet, GraphSAGE, and GAT are the four GNN architectures compared in the study.
- GCN approximates spectral graph convolutions linearly, while MoNet learns adaptive convolution filters as a generalization of GCN.
- The original model implementations use varied early stopping, learning-rate decay, and batch-training procedures, complicating empirical attribution of performance gains.
- Logistic Regression and MLP use node attributes, whereas Label Propagation and Label Propagation NL use graph structure while ignoring node attributes.
3 Evaluation
The evaluation standardizes model training and hyperparameter selection, then assesses GNNs across datasets and many random splits and initializations. Results show that rankings are sensitive to data splits, while simpler models can perform strongly under fair tuning.
- Experimental setup: The experiments use eight datasets, including four citation networks and four newly introduced datasets, treating each graph as undirected and retaining its largest connected component.
- Experimental setup: All models retain their original architectures but use the same optimizer, initialization, training schedule, early stopping, and validation procedure.
- Experimental setup: The evaluation averages mean accuracies over 100 random splits and 20 random weight initializations for every model and dataset.
- Results: GNN approaches significantly outperform MLP, logistic regression, and label-propagation baselines across all datasets by combining structural and attribute information.
- Results: No GNN dominates across all datasets; for 5 of 8 datasets, the second- and third-best approaches are within 1% of the best average score.
- Caveat: Standard deviations provide only a rough variance summary because accuracy scores are not normally distributed; box plots give a more accurate picture.
- Results: GAT has unusually high variance on Amazon Computers and Amazon Photo because rare extremely low scores substantially reduce its average.These outliers occur in 138 of 2000 runs and fall below 40% on Amazon Photo.
- Results: Different train/validation/test splits can completely change model rankings, while GCN achieves the best overall performance under the paper’s aggregated evaluation.
4 Conclusion
The paper evaluates four GNN architectures with a reproducible framework and argues that evaluation procedures strongly affect apparent model performance. Across multiple splits and fair tuning, simpler GCN can outperform more sophisticated architectures.
- The authors empirically evaluate four state-of-the-art GNN architectures and introduce four attributed graph datasets.
- A framework for fair and reproducible comparison is open-sourced alongside the evaluation.
- Single train/validation/test splits produce fragile experimental conclusions, whereas averaging over multiple splits supports more robust comparisons.
- With identical hyperparameter selection and training procedures, a simple GCN can outperform more sophisticated GNN architectures.
A Differences in training procedures for GNN models
The compared GNN models originally use substantially different training procedures. These differences include early stopping, batch strategy, epoch budgets, and optimization schedules.
- GCN uses early stopping based on validation loss, full-batch training, a 200-epoch maximum, and fixed class-balanced train, validation, and test sizes.
- MoNet uses no early stopping, full-batch training, long dataset-specific epoch limits, alternating optimization, and learning-rate decay for CORA.
- GAT uses early stopping after 100 epochs without validation-loss or validation-accuracy improvement and permits up to 100000 epochs.
- GraphSAGE uses no early stopping, mini-batch training with batch size 512, and a maximum of 10 epochs.
B Datasets description and statistics
The datasets combine citation, co-authorship, and Amazon co-purchase graphs, with standardized graph statistics and a defined label-rate measure.
- Amazon Computers and Amazon Photo are co-purchase graphs whose nodes are goods, edges represent frequent co-purchases, and labels denote product categories.
- Coauthor CS and Coauthor Physics are co-authorship graphs connecting authors who co-authored papers, with keyword features and active-field labels.
- The experiments use standardized graphs, add self-loops, and remove three CORA-Full classes containing fewer than 50 nodes.
- Label rate is the fraction of nodes assigned to the training set, computed as (#classes · 20) / #nodes when using 20 training instances per class.
- Edge density measures the fraction of all possible graph edges that are present, computed as #edges / ( 1.
C Hyperparameter configurations and Early Stopping
The study uses a broad grid search and unified early stopping to configure and train the compared models under consistent conditions.
- Hyperparameter configurations: Grid search varies hidden size, learning rate, dropout probabilities, attention dropout for GAT, and L2 regularization strength.
- Early Stopping: Training allows up to 100k epochs but stops after 50 epochs without improved total validation loss, then restores the lowest-loss weights.
- Hyperparameter configurations: Table 4 records each model’s best grid-search configuration, including architecture-specific dropout, skip-connection, hidden-size, and multi-head settings.
D Performance of different models across datasets
The figures summarize test-accuracy distributions across repeated random splits and initializations, with a separate view exposing outliers for Amazon Photo.
- Figure 1 displays boxplots of test accuracy across all models and datasets over 100 random splits and 20 weight initializations per split.The plots show medians and 50% quantiles while excluding outliers.
- Figure 2 provides a boxplot specifically showing outliers for the Amazon Photo dataset.