Source-linked AI summary
Wiki-CS: A Wikipedia-Based Benchmark for Graph Neural Networks
Péter Mernyei, Cătălina Cangea
TL;DR
Existing GNN benchmarks make comparisons difficult because of split sensitivity and similar domain structures. Wiki-CS constructs a Wikipedia-based benchmark with ten computer-science classes and evaluates node classification and link prediction; existing architectures perform similarly to their results on other benchmarks. The dataset broadens available benchmarks with a structurally different domain.
Problem
Existing GNN benchmarks use inconsistent training splits and similar domain structures, making rigorous comparisons between architectures difficult.
Method
Wiki-CS extracts and cleans a Wikipedia article graph into ten computer-science classes, then benchmarks semi-supervised node classification and single-relation link prediction.
Results
Existing GNN architectures perform similarly on Wiki-CS to their results on other benchmarks across semi-supervised node classification and link prediction.
Takeaways & Limitations
Wiki-CS provides a structurally different, publicly available benchmark that broadens the range of datasets for studying general-purpose GNN methods.
Takeaways & Limitations
Category aggregation can leave unrelated articles and make prominent-category selection arbitrary in dense subcategory graphs; benchmark performance is also affected by training-split choice.
Abstract
from arXiv · showhide
We present Wiki-CS, a novel dataset derived from Wikipedia for benchmarking Graph Neural Networks. The dataset consists of nodes corresponding to Computer Science articles, with edges based on hyperlinks and 10 classes representing different branches of the field. We use the dataset to evaluate semi-supervised node classification and single-relation link prediction models. Our experiments show that these methods perform well on a new domain, with structural properties different from earlier benchmarks. The dataset is publicly available, along with the implementation of the data pipeline and the benchmark experiments, at https://github.com/pmernyei/wiki-cs-dataset .
1. Introduction
Wiki-CS addresses weaknesses in standard GNN benchmarks by introducing a Wikipedia-based dataset with different structural properties and evaluating node classification and link prediction. It also supports more comparable evaluation through a new domain and benchmark setup.
- Existing GNN benchmarks make rigorous architectural comparisons difficult because studies use different training splits and similar domain-specific graph structures.Performance on semi-supervised node classification is especially sensitive to the training split.
- Wiki-CS evaluates semi-supervised node classification using labels for a small node subset, features for all nodes, and graph connectivity.The typical labeled subset contains 1–5% of nodes.
- Wiki-CS provides a more highly connected graph distribution than the three commonly used citation-network benchmarks.This gives GNN methods a structurally different domain for testing.
- The dataset also benchmarks relational methods for single-relation link prediction alongside a non-structural SVM baseline.
2. Related Work
Related work has highlighted benchmark saturation and inconsistent evaluation as obstacles to distinguishing GNN architectures. Wiki-CS complements newer benchmark efforts by adding experimentally evaluated data from a new domain.
- Citation-network benchmarks and larger datasets have become difficult for further improvement because state-of-the-art methods already achieve F1 scores of 0.995 and 0.97, respectively.The cited scores are reported for the standard split sizes of the larger Reddit and PPI datasets.
- Concurrent benchmark efforts seek robust comparisons through diverse domains, graph types, task types, unified evaluation, and challenging non-random data splits.The Open Graph Benchmark includes a Wikidata knowledge-graph link-prediction task, which uses source material different from Wiki-CS’s article hyperlink graph.
- Other work uses largely synthetic graphs to distinguish which aspects of GNN architectures contribute to performance on challenging benchmarks.
- Wiki-CS complements these efforts with a dataset and experimental results from a new domain, extending the range of available benchmarks.The authors frame this contribution as further evidence for the generality of GNN methods.
3. The Dataset
Wiki-CS is constructed from Wikipedia articles labeled into ten computer-science branches, with text-derived features, repeated training splits, and structural properties distinct from citation-network benchmarks. Its dense, varied, and potentially hierarchical organization provides a complementary dataset for GNN evaluation.
- 3.1. Article Selection and Label Generation: Wikipedia category labels were unsuitable directly because they were highly specific and inconsistently applied across roughly 1.5 million categories and 6 million pages.
- 3.1. Article Selection and Label Generation: A category-sanitization process aggregates subcategories to prominent ancestors selected using harmonic centrality in the subcategory graph.Figure 1 illustrates aggregation to nearest marked ancestors, with ties broken arbitrarily.
- 3.1. Article Selection and Label Generation: Manual cleanup of candidate subjects produced ten computer-science classes with very high connectivity, while articles mapped to multiple classes were removed.
- 3.2. Node Features: Node features are 300-dimensional averages of pretrained GloVe word embeddings derived from article text rather than binary bag-of-words vectors.The representation encodes vocabulary-rich text features in relatively small input vectors.
- 3.3. Training Splits: Twenty training splits were randomly selected to improve benchmark robustness because training-split choice can strongly affect semi-supervised node-classification performance.Each split uses 5% of nodes in every class for training, with separate early-stopping and validation sets.
- 3.4. Statistics and Structural Properties: Wiki-CS has significantly higher connectivity, short average node-to-node distance, and more varied neighborhood class composition than citation-network benchmarks.The authors suggest that progress may come from more involved computations within a node’s neighborhood rather than long-range connections.
- 3.4. Statistics and Structural Properties: Deep Graph Mapper visualization suggests that Wiki-CS may have a more centralized, hierarchical structure than citation networks.The unsupervised visualization uses node features and edges without labels; clusters are represented by nodes and inter-cluster connectivity by edge thickness.
4. Experiments
The experiments evaluate node classification and link prediction on WIKI-CS using multiple GNN and non-structural baselines. Node-classification performance broadly matches citation-network results, while link-prediction scores are extremely high because random false edges are easy to distinguish.
- Semi-Supervised Node Classification: 20 training splits and a shared 50% test set were used for node classification, with five evaluations per split and bootstrap 95% confidence intervals.Each split used 5% of nodes from every class for training and 22.5% for early stopping.
- Semi-Supervised Node Classification: GCN, GAT, APPNP, MLP, and SVM were evaluated, combining structural GNN models with non-structural feature-based baselines.Hyperparameter tuning used validation nodes disjoint from training and early-stopping sets.
- Semi-Supervised Node Classification: Relative model performances align well with citation-network benchmarks, providing evidence that these architectures generalize across graph domains.GAT improved very little over GCN despite WIKI-CS having large neighborhoods.
- Semi-Supervised Node Classification: Prediction mistakes cluster near class boundaries, while less-connected clusters produce consistent errors across models, possibly because nearby training samples are lacking.These patterns suggest that richer neighborhood aggregation could improve accuracy, although the stated explanation for sparse-cluster errors is tentative.
- Link Prediction: 85% of real edges were used for training, 5% for validation, and 10% for testing, with equally many uniformly sampled non-edge pairs in each group.GraphStar, VGAE, and an MLP using concatenated node-feature pairs were benchmarked for single-relation link prediction.
- Link Prediction: All link-prediction models achieved extremely high performance, including the MLP baseline, indicating that randomly selected false edges are easy to distinguish from true edges.Using only 10,000 examples of each class still produced metrics comfortably above 0.9, so the large edge count was not the main cause.
5. Conclusion
The paper presents WIKI-CS as a new GNN benchmark with structural properties unlike commonly used datasets. Experiments find that existing architectures perform similarly to their results on other benchmarks, broadening the available benchmark range.
- Conclusion: WIKI-CS is introduced as a new benchmark for GNN methods with structural properties significantly different from commonly used datasets.The dataset is made available for further study.
- Conclusion: Existing GNN architectures perform similarly on WIKI-CS and other benchmarks for semi-supervised node classification and link prediction.The authors present this as evidence that these architectures are good general-purpose methods for graph-learning tasks.
- Conclusion: The dataset broadens the range of available benchmarks for graph-learning research.WIKI-CS is publicly available for further study.
A.1. Node classification
The node-classification appendix specifies the hyperparameters used for MLP, SVM, GCN, GAT, and APPNP models, with a shared L2 loss coefficient.
- Node classification: The MLP used two layers, 35 hidden units, 0.35 dropout, and learning rate 0.003.
- Node classification: The SVM used a radial basis function kernel with C = 8.
- Node classification: The GCN used two layers, 33 hidden neurons, learning rate 0.02, 0.25 dropout, and self-loops.
- Node classification: The GAT used two layers with five hidden-layer attention heads outputting 14 units each, 0.5 dropout, learning rate 0.007, and self-loops.
- Node classification: APPNP used learning rate 0.02, 0.4 dropout, two propagation iterations, and teleport probability α = 0.11.
- Node classification: All node-classification models used an L2 loss coefficient of 5 × 10^-4.
A.2. Link prediction
The link-prediction appendix details configurations for GraphStar, VGAE, and the MLP baseline. VGAE performance improved when KL regularization on latent-space distributions was removed.
- Link prediction: GraphStar used 1,000 epochs, a 256-unit hidden layer, three layers, no dropout, and 5 × 10^-4 L2 regularization.The original training code and dynamically varying learning-rate schedule were retained.
- Link prediction: VGAE used a 16-dimensional latent space, GCN encoders, a shared 32-neuron hidden layer, no dropout, and 200 training epochs.The setup largely followed the original paper.
- Link prediction: Discarding the KL regularization loss on VGAE latent-space distributions improved performance.
- Link prediction: The MLP used three hidden layers of 128 units, 100 epochs, 0.2 dropout after each layer, learning rate 10^-4, and no weight decay.
A.3. DGM visualisations
The DGM visualisations use a fixed configuration combining a structural graph variant, an unsupervised lens, dimensionality reduction, and interval partitioning.
- All plots use the SDGM variant to visualise the original graph structure.
- The visualisations use an unsupervised DGI lens and t-SNE reduction.
- The plots divide the data into 20 intervals and apply additional parameters.
B. List of categories for each label
Table 5 lists the aggregated Wikipedia categories used to construct each class, with categories selected as aggregation targets and some unsuitable subcategories manually excluded.
- Each class consists of pages aggregated to one of its corresponding target categories.
- The sanitizer tool selected the listed categories as aggregation targets.
- Some subcategories that added unsuitable results were manually excluded.
C. Deep Graph Mapper visualisation of model predictions
Figure 4 visualises predictions from different node-classification models by coloring graph clusters according to their dominant true labels or model predictions.
- The top image colors each cluster according to its most frequent true label.
- The other plots color clusters according to the most frequent prediction of the corresponding model.
- Majority-based coloring can hide prediction differences that do not change a cluster’s dominant label.