GraphLand: Evaluating Graph Machine Learning Models on Diverse Industrial Data

1 Introduction

• Popular node-property benchmarks and GNN evaluations cover a narrow range of applications, often relying on a few academic citation networks.This limits evaluation of models intended to transfer across domains and diverse node feature sets.Source: §1 · pp. 1–2
• GraphLand provides 14 industrial node-property datasets spanning diverse domains, graph sizes, structural properties, and rich node features.The benchmark combines underused open data with newly released data from a large technology company.Source: §1 · p. 2
• Temporal splits support investigation of how distributional shifts affect GML performance in both transductive and inductive settings.The authors identify resilience to temporal shifts and dynamically evolving graphs as underexplored research directions.Source: §1 · pp. 2–3
• Current graph foundation models perform poorly on GraphLand and fail to match more classic methods; attention-based GNNs often outperform more classic GNNs.GNN performance can be strongly affected by temporal distributional shifts and dynamically evolving graph structure.Source: §1 · p. 2

2 Limitations of popular graph machine learning benchmarks

• Existing benchmark collections also have shortcomings in structural and data quality coverage, including homophily, duplicated nodes, small graph sizes, and limited class representation.Some newer non-homophilous datasets are synthetic, semi-synthetic, or provide limited node features despite richer source data.Source: §2 · p. 3
• Popular node-property datasets cover few practical applications, and do not support evaluation across a wide range of impactful industrial settings.Citation networks dominate, while common social, web, and road graphs are rarely used for GNN evaluation.Source: §2 · p. 3
• Many benchmarks provide only textual node features, while real-world graphs often include numerical and categorical attributes with varied meanings and distributions.This limits evidence about generalization to different node feature sets, a key challenge for general-purpose graph foundation models.Source: §2 · pp. 3–4

3 GraphLand: a collection of diverse industrial graph datasets

• GraphLand comprises 14 graph datasets for node classification or regression, including underused open sources and newly released data.The benchmark criteria seek impactful industrial applications, useful graph structure, structural diversity, rich numerical or categorical features, temporal information when available, and at least 10,000 nodes.Source: §3 · pp. 4–5
• Newly released web datasets use one website graph for fraud, topic, and traffic prediction, while creator-network datasets address views and explicit-content prediction.The web graph has almost 3 million nodes; the cited tasks include classification and regression.Source: §3 · p. 4
• The collection also includes city road-network speed prediction and review-service fraud detection, alongside open-data tasks on ads, products, social regions, streaming views, and crowdsourcing fraud.The open-data descriptions include device-level click-through-rate prediction, product category and price prediction, and user-region prediction.Source: §3 · pp. 4–5
• The authors note that open network-data sources beyond academic citation networks remain rarely used for GML evaluation.GraphLand includes data from applications such as web, social, and road networks.Source: §3 · p. 4
• The proposed GraphLand datasets cover a diverse range of industrial applications and exhibit very diverse graph structural properties; Table 1 provides some dataset characteristics.Source: §3 · p. 5, Table 1 · p. 5

4 Data splits and experimental settings

• Random low-label (RL) splits use 10%/10%/80% train/validation/test nodes, while random high-label (RH) splits use 50%/25%/25%.Both fixed split types accommodate different labeling-resource scenarios and support comparable experiments.Source: §4 · p. 6
• The temporal high (TH) split uses 50%/25%/25% proportions, matching RH so results can be compared when temporal distribution shifts are introduced.Such shifts can combine changes in node features, labels, and graph structural characteristics.Source: §4 · p. 6
• Temporal splits address a gap in benchmarks, where realistic temporal distribution shifts and inductive evaluation on evolving graphs are rarely represented.The inductive setting withholds validation and test nodes at training time, unlike the transductive setting where the whole graph is available.Source: §4 · p. 6
• The temporal high inductive (THI) setting reuses the TH split but provides separate graph snapshots for training, validation, and testing.This supports comparison of transductive and inductive performance under the same temporal split.Source: §4 · p. 6

5.1 Models

• The experiments use extensive hyperparameter tuning, and Table 3 reports results for random-high, temporal-high, and temporal-high inductive settings.Full results are provided in Appendix C.Source: §5.1 · p. 6, Table 3 · p. 9, §5.3 · p. 10
• The model suite includes graph-agnostic ResMLP and GBDTs, four GNN architectures, and selected GFMs; NFA adds one-hop neighbor feature statistics to graph-agnostic models.The GNNs use a shared backbone with spatial graph convolution modules, enabling comparisons with ResMLP that isolate graph access.Source: §5.1 · p. 7

5.2 Low label rate, random data split, transductive setting

• All tested GNNs significantly outperform ResMLP, and NFA improves graph-agnostic models, with LightGBM-NFA best on four datasets, including three regression tasks.GNNs still outperform ResMLP-NFA on those four datasets and outperform GBDTs on most datasets.Source: §5.2 · p. 8, Table 2 · p. 8

5.3 High label rate, random and temporal data splits, transductive and inductive settings

• Temporal splits are significantly more challenging than random splits for all models except on avazu-ctr, and inductive performance is significantly worse than transductive performance.The authors recommend frequent GNN retraining in the absence of methods more resilient to temporal shifts and evolving graph structure.Source: Table 3 · p. 9, §5.3 · p. 10

5.2 Low label rate, random data split, transductive setting

• Attention-based GNNs GAT and GT outperform classic GCN and GraphSAGE on most datasets, while all considered GFMs produce very weak results.The GFM evaluation covered classification datasets; the considered models did not support node regression, and experiments found they could not scale to large datasets.Source: §5.1 · p. 8, §5.2 · pp. 9–10

5.3 High label rate, random and temporal data splits, transductive and inductive settings

• GNNs provide substantial benefits, attention-based GNNs often perform particularly well, graph-feature-augmented GBDTs can be strong baselines, and current GFMs perform weakly on GraphLand.The authors report that GBDTs with augmented features are especially strong in regression, while GNNs outperform them on most datasets.Source: §5.3 · p. 10

• GraphLand is a set of diverse graph datasets representing realistic industrial applications of graph machine learning.Source: §6 · p. 10
• The benchmark supports investigating how temporal distributional shifts and inductive prediction influence graph machine learning performance.Source: §6 · p. 10
• The authors hope GraphLand encourages more realistic evaluation and methods resilient to temporal shifts, changing graph structure, and varied node feature sets.Source: §6 · p. 10

• GraphLand data are available through Zenodo and Kaggle, and the datasets and preprocessing code are also provided in PyTorch Geometric; the datasets contain no personal information.For newly collected data, public release was approved by a legal team.Source: §A.1 · p. 15
• The web-fraud, web-topics, and web-traffic datasets share a directed graph of nearly 3 million websites, with fraud, topic, and traffic prediction tasks.Nodes have more than 200 features, and the data was obtained from the Yandex search engine.Source: §A.1 · p. 15
• The benchmark includes social, device-interaction, and co-purchasing graphs, with prediction tasks including user views, advertisement CTR, and product categories or prices.The artnet datasets share a social graph; avazu-ctr links devices by shared website visits, and the H&M datasets form a product co-purchasing graph.Source: §A.1 · pp. 15–16
• The road datasets represent two cities’ directed road networks and predict average travel speed, while city-reviews uses a user graph for fraudulent-review detection.The road datasets were obtained from Yandex Maps; city-reviews is based on review-service logs.Source: §A.1 · p. 15
• Random stratified RL and RH splits are provided for all datasets, while temporal TH splits, with possible inductive THI use, exclude four datasets for stated data or network-evolution reasons.The exceptions are city-roads-M, city-roads-L, city-reviews, and web-traffic.Source: §A.1 · p. 16

• GraphLand datasets range from 11K to 3M nodes, with smaller graphs suited to compute-intensive models and larger graphs providing a moderate scaling challenge.Source: §A.2 · p. 16
• Structural properties vary widely: average degree, path distance, diameter, clustering, and degree assortativity differ across graphs, including both sparse and dense networks.Average distance ranges from 2.45 for hm-categories and hm-prices to 194 for city-roads-L; diameter ranges from 8 for twitch-views to 553 for city-roads-L.Source: §A.2 · pp. 16–17
• Regression datasets span positive and negative target assortativity, while unbiased homophily classifies pokec-regions and city-reviews as homophilous and the others as non-homophilous.The authors caution that unbiased homophily values should not be compared with values from other homophily measures.Source: §A.2 · p. 17
• Node features mix numerical and categorical attributes with different meanings and distributions; all but twitch-views and tolokers-2 have at least several dozen features.Some datasets have several hundred node features.Source: §A.2 · p. 17
• The authors compute graph characteristics after treating directed graphs as undirected; for the largest graphs, average distance is estimated from 100K randomly sampled node pairs.The described characteristics include clustering coefficients, degree and target assortativity, and unbiased homophily.Source: §A.2 · pp. 17–18

B.1 Models

• Graph-agnostic baselines include ResMLP and XGBoost, LightGBM, and CatBoost; NFA-augmented ResMLP and LightGBM test the value of graph information.NFA provides only aggregated features of one-hop neighbors, and the authors report that NFA versions significantly outperform their counterparts in their experiments.Source: §B.1 · p. 18
• GNNs replace ResMLP residual blocks with neighborhood-aggregation blocks and significantly outperform ResMLP across all datasets.The GNNs include GCN, mean-aggregation GraphSAGE, GAT, and a local Graph Transformer.Source: §B.1 · p. 18
• OpenGraph uses adjacency SVD factors, while AnyGraph combines adjacency and feature-matrix SVD factors; both use in-context learning for node classification.Both cast node classification as link prediction to virtual class nodes.Source: §B.1 · pp. 18–19
• GCOPE adapts through fine-tuning, whereas TS-GNN uses permutation-equivariant linear transformations and lightweight least-squares fitting before prediction.The experiments use TS-Mean, one of TS-GNN’s two proposed versions.Source: §B.1 · p. 19
• The considered GFMs support neither node regression nor large-dataset scaling, preventing evaluation on many GraphLand datasets.For TS-GNN and GCOPE, regression is an implementation limitation; for OpenGraph and AnyGraph, it is inherent to model design.Source: §B.1 · p. 19

B.2 Neighborhood feature aggregation

• NFA appends permutation-invariant aggregates of non-NaN one-hop neighbor features to each node’s original features.It uses mean, maximum, and minimum for numerical features, and one-hot encoding followed by mean aggregation for categorical features; an empty neighbor-value set yields NaN.Source: §B.2 · p. 19
• Figure 1 illustrates NFA using four neighbors, with mean, maximum, and minimum for the numerical feature and a one-hot-category mean.The example distinguishes numerical and categorical features by blue and green, respectively.Source: §B.2 · p. 20, Figure 1 · p. 20

• Directed graphs are made undirected by replacing directed edges and removing duplicates; other treatments of edge direction are left for future work.Source: §B.3 · p. 20
• Models are trained 10 times, or 5 times on four largest datasets; GNNs and ResMLP use full-batch training, with GBDTs trained on CPUs.Experiments used an NVIDIA Tesla A100 80GB GPU except for GBDTs, which ran on AMD EPYC CPUs; each GNN model was trained for 1,000 steps with the best validation step selected.Source: §B.3 · pp. 20–21
• GBDT hyperparameters receive 100 Optuna Bayesian-optimization iterations, while GNN learning rate and dropout are grid-searched and architecture settings are mostly fixed.Table 4 gives the Optuna search distributions for GBDT models; GNNs use three aggregation blocks, with hidden dimensions reduced on the largest datasets to avoid out-of-memory errors.Source: §B.3 · pp. 20–21, Table 4 · p. 21
• Neural models tune standard scaling or quantile transformation for numerical features, while LightGBM and CatBoost handle categorical features directly.For regression, neural models also search between original and standard-scaled targets.Source: §B.3 · p. 21

• The main text reports complete results for the RL setting.Source: C Complete experimental results · p. 21
• For RH, TH, and THI, the main text omits some baselines and datasets without temporal splits because of space limitations.Omitted baselines include ResMLP, XGBoost, CatBoost, and ResMLP-NFA; omitted datasets include city-roads-M, city-roads-L, city-reviews, and web-traffic.Source: C Complete experimental results · p. 21
• Complete RH, TH, and THI results are in Tables 5, 6, and 7, respectively; per dataset, tables highlight the best result and those whose mean differs from it by no more than the sum of both results’ standard deviations.Source: C Complete experimental results · p. 21

• GraphLand’s 14 datasets cover diverse domains and graph structures but still represent only a small part of situations where graph modeling can be useful.Source: D Limitations and broader impact · p. 21
• Errors in realistic tasks such as fraud detection can harm service users, including innocent people wrongly labeled fraudulent.The authors argue that high-quality, anonymized datasets can provide a testbed for investigating methods that reduce undesirable model errors.Source: D Limitations and broader impact · p. 21
• The RH and TH tables cover transductive settings and the THI table covers an inductive setting; captions mark best and statistically indistinguishable results.For classification, multiclass results use accuracy and binary results use Average Precision.Source: Table 5 · p. 22, D Limitations and broader impact · p. 22, Table 6 · p. 23