PaperlayerView exact revision on arXiv

Source-linked AI summary

A Fair Comparison of Graph Neural Networks for Graph Classification

Federico Errica, Marco Podda, Davide Bacciu, Alessio Micheli

arXiv:1912.09893v3cs.LGstat.ML

TL;DR

The paper addresses poorly documented and nonstandardized experimental procedures that hinder fair GNN comparisons. It re-evaluates five GNNs in a controlled framework and finds that structure-agnostic baselines outperform GNNs on some datasets, while structural information remains underexploited there.

  • Problem

    Ambiguous procedures, improper data splits, and inconsistent features make graph-learning results difficult to reproduce and compare fairly.

  • Method

    The study runs a large, rigorous evaluation using common features and data splits, re-evaluating selected GNNs and comparing them with structure-agnostic baselines.

  • Results

    Structure-agnostic baselines outperform GNNs on some chemical datasets, suggesting that structural properties have not yet been exploited there.

  • Takeaways & Limitations

    Fair, reproducible benchmarking provides a grounding for reasoning about architectural choices and the contribution of graph structure.

  • Takeaways & Limitations

    The work does not aim to identify the best or worst-performing GNN.

Abstract

from arXiv · show

Experimental reproducibility and replicability are critical topics in machine learning. Authors have often raised concerns about their lack in scientific publications to improve the quality of the field. Recently, the graph representation learning field has attracted the attention of a wide research community, which resulted in a large stream of works. As such, several Graph Neural Network models have been developed to effectively tackle graph classification. However, experimental procedures often lack rigorousness and are hardly reproducible. Motivated by this, we provide an overview of common practices that should be avoided to fairly compare with the state of the art. To counter this troubling trend, we ran more than 47000 experiments in a controlled and uniform framework to re-evaluate five popular models across nine common benchmarks. Moreover, by comparing GNNs with structure-agnostic baselines we provide convincing evidence that, on some datasets, structural information has not been exploited yet. We believe that this work can contribute to the development of the graph learning field, by providing a much needed grounding for rigorous evaluations of graph classification models.

1 INTRODUCTION

The paper addresses reproducibility problems in graph learning by establishing a uniform framework for fairly evaluating GNNs. It also examines whether GNNs exploit graph structure and how degree features affect social-dataset performance.

  • Ambiguous procedures, irreproducible results, and improper model comparisons make GNN effectiveness difficult to assess uniformly.
  • Separating validation-based model selection from test-based model assessment is necessary to avoid over-optimistic and biased performance estimates.
  • The study compares GNN architectures in a standardized environment using the same features and data splits for all models.
  • Structure-agnostic baselines can outperform GNNs on some datasets, motivating moderation when reporting improvements over such competitors.
  • Providing node degree can improve performance on social datasets and affects how many GNN layers are needed for good results.

2 RELATED WORK

The related work presents GNNs as adaptive graph-learning models that aggregate neighborhood information, while positioning this evaluation alongside prior work on split-dependent model rankings.

  • GNNs iteratively update node states from neighboring nodes to construct representations aware of broader graph structure.
  • GraphSAGE aggregates neighborhoods with sum, mean, or max pooling before applying a linear projection, and samples neighborhoods to maintain constant complexity.
  • DGCNN, ECC, and DiffPool represent alternative convolutional or pooling designs within graph classification architectures.
  • Prior node-classification research found that changing train, validation, and test splits can dramatically change GNN performance rankings.
  • The paper follows the recommendation to use multiple test splits when seeking fair comparisons, while operating on graph classification instead of node classification.

3 RISK ASSESSMENT AND MODEL SELECTION

The paper distinguishes model selection from risk assessment so hyper-parameters are chosen without using test data. This separation supports less biased estimates of generalization performance.

  • 3.1 RISK ASSESSMENT: Risk assessment estimates the performance of a class of models, commonly using k-fold cross-validation when no explicit test set is available.
  • 3.1 RISK ASSESSMENT: In k-fold cross-validation, an internal selection procedure chooses hyper-parameters using training data only for each training/test partition.
  • Keeping model selection separate from model assessment prevents test data from influencing hyper-parameter choice.
  • 3.2 MODEL SELECTION: Model selection chooses the best candidate hyper-parameter configuration on a validation set, but validation performance is generally over-optimistic.

4 OVERVIEW OF REPRODUCIBILITY ISSUES

The review identifies widespread weaknesses in GNN evaluation and reproducibility, then motivates a standardized re-evaluation of five graph-classification models.

  • Model and criteria: The review selected DGCNN, DiffPool, ECC, GIN, and GraphSAGE based on cross-paper evaluation criteria including 10-fold CV, peer review, architectural diversity, and popularity.Quality criteria included released preprocessing, selection, and assessment code; available splits; stratification; and correctly reported test-set means and standard deviations.
  • Model findings: DGCNN provides stratified, publicly available splits, but unavailable model-selection code and unequal repeated-CV treatment weaken comparability.Learning rate and epochs were tuned on one fold and reused, while DGCNN’s CV was repeated ten times without applying the same procedure to competitors.
  • Model findings: DiffPool has unclear test-versus-validation reporting, missing standard deviations and selection artifacts, unstratified random splits, and no fixed seed.Its splits can therefore differ across executions, while validation-based early stopping cannot be independently reproduced from the available materials.
  • Model findings: ECC omits standard deviations and does not document how model selection was performed, preprocessing, stratification, splitting, or selection in its repository.The paper reports 10-fold CV, but the evaluation record is incomplete across several reproducibility criteria.
  • Model findings: GIN reports validation accuracy from 10-fold CV rather than model-evaluation performance, and its model-selection code is unavailable.The authors do list the tuned hyper-parameters, but the reported results concern model selection rather than external assessment.
  • Overall assessment: Overall, the analysis finds that GNN studies rarely satisfy good evaluation and reproducibility practices, motivating rigorous and fair re-evaluation.The review frames ambiguous procedures, irreproducible results, and improper comparisons as recurring field-wide concerns.

5 EXPERIMENTS

The experiments re-evaluate GNNs and structure-agnostic baselines on public chemical and social graph datasets under a rigorous selection-and-assessment framework. The study uses extensive hyper-parameter searches and more than 47,000 training runs.

  • Experimental design: The main experiment re-evaluates the selected GNN models on nine datasets spanning four chemical and five social benchmarks.Two additional baselines are implemented to assess how much GNN performance depends on structural information.
  • Baselines: The chemical and social baselines are structure-agnostic, enabling comparisons that help determine whether improvements reflect exploited graph topology.Significant gains over these baselines are described as a strong indicator that graph structure has been used.
  • Experimental design: Model assessment uses 10-fold cross-validation, while model selection uses an inner 90%/10% training/validation split.After selection, each model is trained three times on the whole training fold with a 10% holdout for early stopping, and test scores are averaged.
  • Hyper-parameters: Grid search selects convolutional layers, embedding dimension, learning rate, and early-stopping criterion for every model.The search also includes model-specific regularization and other hyper-parameters, while incorporating settings used in prior papers.
  • Computational considerations: More than 47,000 single training runs were required to complete the model-assessment procedures.Grid sizes ranged from 32 to 72 configurations per model, with upper bounds kept as similar as possible across models.

6 RESULTS AND DISCUSSION

Across the evaluated benchmarks, structure-agnostic baselines match or outperform GNNs on several chemical datasets, while node degrees substantially affect social-dataset performance. Published test results also differ from the study’s rigorously obtained estimates.

  • 6.1 THE IMPORTANCE OF BASELINES: None of the evaluated GNNs improves over the baseline on D&D, PROTEINS, or ENZYMES, whereas GNNs clearly outperform it on NCI1.The NCI1 result suggests that the analyzed GNNs exploit graph topology there, while the other chemical datasets show no baseline advantage for GNNs.
  • 6.1 THE IMPORTANCE OF BASELINES: Structure-agnostic baselines are essential for interpreting GNN effectiveness because small gains on some chemical datasets may reflect random initialization rather than structure exploitation.The authors recommend including such baselines until GNNs clearly outperform them.
  • 6.2 THE EFFECT OF NODE DEGREE: ≈15% baseline improvement from adding node degrees makes degree information competitive on many social datasets, including best performance on IMDB-BINARY.Degree features are almost always beneficial on social datasets; their effect is less relevant for most GNNs, except DGCNN, which explicitly needs degrees to perform well.
  • 6.3 COMPARISON WITH PUBLISHED RESULTS: Our test accuracies are in most cases different from published results, with a usually consistent gap between the two estimates.The study compares average test results with published results and also reports average validation accuracies across ten model selections.
  • 6.3 COMPARISON WITH PUBLISHED RESULTS: Validation accuracies are always higher or equal to test results, while the study’s results use rigorous, fair, and reproducible model assessment procedures.The comparison emphasizes consistent data splits and input features across competitors, unlike unclear or poorly documented published settings.

7 CONCLUSIONS

The paper proposes rigorous, reproducible GNN evaluation and re-evaluates five models across nine datasets. Its findings highlight underused graph structure on some chemical datasets and a substantial role for degree features in social datasets.

  • 7 CONCLUSIONS: The authors identify ambiguities in prior experimental settings and propose a clear, reproducible procedure for future GNN comparisons.The procedure supports more reliable reasoning about architectural effectiveness.
  • 7 CONCLUSIONS: Five GNNs were completely re-evaluated on nine datasets in a uniform environment requiring substantial computational resources.The resulting experiments provide reliable and reproducible results for practitioners comparing architectures.
  • 7 CONCLUSIONS: Structure-agnostic baselines outperform GNNs on some chemical datasets, suggesting that the analyzed models have not fully exploited structural properties there.The conclusions also report that degree features affect performance and GNN model-selection depth on social datasets.

A.1 VISUALIZATION OF THE EVALUATION FRAMEWORK

The evaluation framework uses external cross-validation to estimate generalization and hold-out validation for hyperparameter selection. A complete nested procedure is possible but increases model-selection cost.

  • A.1 VISUALIZATION OF THE EVALUATION FRAMEWORK: External k_out-fold cross-validation estimates model generalization performance, while an hold-out split selects the best hyperparameters.This separates performance assessment from hyperparameter selection within the evaluation framework.
  • A.1 VISUALIZATION OF THE EVALUATION FRAMEWORK: An inner k_in-fold cross-validation implements complete nested cross-validation but multiplies model-selection computational costs by k_in.The framework presents nested cross-validation as an alternative to the hold-out selection technique.

A.2 DATASET STATISTICS

The dataset statistics account for missing node labels by assigning each node either a constant feature of 1 or its degree, while ENZYMES uses 18 additional node attributes.

  • A.2 DATASET STATISTICS: When node labels are absent, every node receives either the same feature value of 1 or its degree.This specifies the node-feature construction used in the reported dataset statistics.
  • A.2 DATASET STATISTICS: ENZYMES uses 18 additional node attributes following the literature.

A.3 EFFECT OF NODE DEGREE ON LAYERING

The section reports selected network depth and hidden-unit counts across social and chemical datasets, including comparisons involving node-degree features.

  • Table 6 reports mean and median selected layers across 10 folds for social datasets, relating them to adding node degrees as node features.
  • Table 7 reports the median number of selected layers across 10 folds for all chemical datasets.
  • Table 8 reports mean and median selected hidden units per layer across 10 folds for social datasets with node-degree features.
  • Table 9 reports the median number of selected hidden units per layer across 10 folds for all chemical datasets.
Loading 1912.09893v3…