PaperlayerView exact revision on arXiv

Source-linked AI summary

GOOD: A Graph Out-of-Distribution Benchmark

Shurui Gui, Xiner Li, Limei Wang, Shuiwang Ji

arXiv:2206.08452v2cs.LGcs.AI

TL;DR

Graph OOD lacks a systematic benchmark that distinguishes covariate from concept shifts and reflects graph-specific distribution changes. GOOD constructs controlled splits across graph and node prediction datasets and evaluates OOD baselines. The benchmark reveals substantial ID–OOD performance gaps and different method trends across shift types, while remaining limited in coverage and scope.

  • Problem

    Graph OOD is an emerging area lacking a systematic benchmark that distinguishes covariate and concept shifts for method evaluation.

  • Method

    GOOD constructs covariate, concept, and no-shift splits across graph and node prediction datasets using carefully designed environments and data splits.

  • Results

    GOOD shows substantial and consistent performance gaps between in-distribution and out-of-distribution settings, with different algorithm trends across covariate and concept shifts.

  • Takeaways & Limitations

    OOD algorithms improve generalization abilities but not significantly, and methods may improve one shift type without improving both.

  • Takeaways & Limitations

    The benchmark does not consider link prediction tasks and does not explicitly consider unobserved confounders.

Abstract

from arXiv · show

Out-of-distribution (OOD) learning deals with scenarios in which training and test data follow different distributions. Although general OOD problems have been intensively studied in machine learning, graph OOD is only an emerging area of research. Currently, there lacks a systematic benchmark tailored to graph OOD method evaluation. In this work, we aim at developing an OOD benchmark, known as GOOD, for graphs specifically. We explicitly make distinctions between covariate and concept shifts and design data splits that accurately reflect different shifts. We consider both graph and node prediction tasks as there are key differences in designing shifts. Overall, GOOD contains 11 datasets with 17 domain selections. When combined with covariate, concept, and no shifts, we obtain 51 different splits. We provide performance results on 10 commonly used baseline methods with 10 random runs. This results in 510 dataset-model combinations in total. Our results show significant performance gaps between in-distribution and OOD settings. Our results also shed light on different performance trends between covariate and concept shifts by different methods. Our GOOD benchmark is a growing project and expects to expand in both quantity and variety of resources as the area develops. The GOOD benchmark can be accessed via https://github.com/divelab/GOOD/.

1 Introduction

GOOD is a systematic graph OOD benchmark designed to distinguish covariate and concept shifts while addressing graph-specific feature and topology distribution changes. It covers graph and node prediction settings with carefully designed splits for evaluating OOD methods.

  • Graph OOD remains an emerging research area without a systematic benchmark tailored to evaluating graph OOD methods.
  • Covariate shift changes P_train(X) and P_test(X) while preserving P_train(Y |X) = P_test(Y |X); concept shift changes P(Y |X) while preserving P(X).
  • Graph OOD must account for both feature and structure distribution shifts because graph inputs combine node or edge features F with adjacency structure A.
  • GOOD includes graph-level and node-level datasets and generates no-shift, covariate-shift, and concept-shift splits for comparison across baselines.
  • GOOD compares covariate and concept shifts under controlled domain selections, including an example where DIR performs favorably under concept shift but fails under the corresponding covariate shift.

2 Related Work

Prior OOD benchmarks span computer vision, real-world data, and graph-related settings, but existing benchmarks do not consistently distinguish or compare covariate and concept shifts. The paper also evaluates graph-specific OOD methods to support further graph OOD research.

  • OOD benchmarks have been curated to evaluate algorithms across distribution-shift settings, including DomainBed, OoD-Bench, and WILDS.DomainBed targets computer vision, OoD-Bench categorizes diversity and correlation shifts, and WILDS collects real-world data.
  • Existing benchmarks often ignore one shift type or fail to compare covariate and concept shifts for the same feature on the same dataset.The paper identifies this as a limitation of both general and graph OOD benchmarks.
  • Graph benchmark inputs use X as shorthand for graph inputs, so a feature may denote either a node feature or a specific graph structure.
  • The study evaluates 4 graph-specific OOD methods, whereas GDS and DrugOOD do not benchmark graph-specific OOD methods.

3 The GOOD Benchmark Design

GOOD designs graph OOD splits that distinguish covariate shifts from concept shifts using environment-aware constructions. Its framework selects relevant feature or domain differences while representing training and test distributions as weighted combinations of environments.

  • Shift definitions: GOOD separates covariate and concept shifts by changing P(X) while preserving P(Y |X), or changing P(Y |X) while preserving P(X).
  • Covariate shifts: Covariate splits vary Xind across domains while keeping Pdi(Xinv, Xass) fixed and Pdi(Y |X) invariant.Each domain is defined by a unique discrete Xind value.
  • Dataset-specific construction: GOOD applies covariate and concept split designs to synthetic, semi-artificial, and real-world datasets, adapting graph generation, feature modification, or data partitioning to each setting.
  • Concept shifts: Concept splits vary domain-output correlations across concepts, creating spurious associations between Xind and Y while preserving invariant information in Xinv.Training concepts use similar domain-output correlations, whereas training, validation, and test concepts are designed to differ substantially.
  • Environments: GOOD treats domains or concepts as environments and represents train/test distributions as weighted combinations of environment distributions.Training and test environments are selected as subsets of the full environment set.

4 The GOOD Datasets

GOOD contains 11 graph datasets spanning molecular, language, citation, social, webpage, and synthetic data. The benchmark uses selected domain features to construct varied shifts for both graph-level and node-level prediction, with performance evaluated across 51 splits.

  • Benchmark overview: 11 datasets cover graph-level and node-level tasks, with selected domain features used to create no-shift, covariate-shift, and concept-shift splits.
  • Graph prediction tasks: Molecular datasets include GOOD-HIV, GOOD-PCBA, and GOOD-ZINC, using scaffold and size domains for classification, multitask classification, and solubility regression.GOOD-PCBA evaluates average precision averaged over 128 highly imbalanced bioassay tasks.
  • Graph prediction tasks: GOOD-SST2 represents sentences as grammar-tree graphs and uses sentence length as a domain for sentiment classification.
  • Synthetic and semi-artificial datasets: GOOD-CMNIST and GOOD-Motif provide feature-shift and structure-shift checks, with labels based on digit concepts or motifs and shifts applied separately.
  • Node prediction tasks: Node-level datasets include citation, social, webpage, and synthetic networks, with domains such as words, degree, publication time, language, university, and node color.GOOD-CBAS predicts four node roles on a graph formed from 80 house-like motifs attached to a 300-node Barabási–Albert base graph.
  • Evaluation: Table 1 reports ERM ID and OOD performance across 51 splits, averaging results over 10 runs and distinguishing IDID, OODID, and OODOOD settings.

5 Experimental Studies

GOOD evaluates ERM and nine OOD algorithms across graph and node prediction settings, using shared GNN backbones for fair comparisons. Results reveal method- and shift-specific performance patterns, with several graph-specific methods improving selectively.

  • Baseline methods: 10 baseline methods are evaluated across GOOD’s datasets and splits, using shared GNN backbones for fair comparisons.GIN-Virtual and GCN serve as backbones for graph and node prediction tasks, respectively; GOOD-Motif uses GIN.
  • In-distribution versus out-of-distribution performance gap: ID and OOD results exhibit substantial and consistent performance gaps across domain selections and distribution shifts.ERM experiments also show that OOD validation generally selects models with better generalization than ID validation.
  • Quantitative comparison and analysis: VREx outperforms other methods on 7 of 34 OOD splits, while GroupDRO outperforms on 8 of 34 OOD splits.VREx is especially favorable for covariate shifts in graph prediction, whereas GroupDRO’s results indicate an advantage from fair optimization.
  • Quantitative comparison and analysis: DANN outperforms on 4 OOD splits and is especially suitable for graph concept-shift splits, while Deep Coral wins on 1 OOD split but often performs better on ID tests.Both methods target feature-distribution discrepancies, but their advantages differ between shift settings and evaluation regimes.
  • Quantitative comparison and analysis: IRM performs similarly to ERM and outperforms on 3 OOD results, highlighting the difficulty of achieving invariant prediction in nonlinear settings.EERM and SRGNN generally achieve average performance and outperform on only a few splits.
  • Quantitative comparison and analysis: Mixup-For-Graph outperforms on 6 of 14 node-task OOD splits but fails on graph prediction tasks, whereas DIR outperforms on 3 graph-classification splits.DIR’s benefit is specific to concept shifts and does not extend to covariate shifts or regression tasks.

6 Discussions

GOOD’s comparisons indicate that current OOD algorithms improve generalization only modestly and that gains depend on the shift type. The benchmark is intended to expand as graph OOD research develops, including currently omitted task coverage.

  • Discussion: Current OOD algorithms improve generalization abilities, but not significantly, and an algorithm may improve one shift type without improving both.The discussion distinguishes covariate- and concept-shift performance rather than treating OOD generalization as uniform.
  • Discussion: Using environment information to convey the shift type expected during testing is identified as a promising direction.The proposed direction follows the observation that models cannot be expected to solve unknown distribution shifts.
  • Limitations and future expansion: The benchmark currently excludes link prediction and is expected to grow in methods, datasets, and domain-selection variety.Future expansions particularly target additional graph-specific algorithms and broader dataset coverage.

A GOOD Dataset Details

GOOD provides 11 datasets and 51 splits spanning no shift, covariate shift, and concept shift settings. Its dataset construction combines real-world, semi-artificial, and synthetic procedures with explicit environment-variable designs.

  • Dataset and split overview: 11 datasets with 17 domain selections produce 51 splits: two shifted splits and one no-shift split per domain selection.Shifted splits contain training, ID validation, ID test, OOD validation, and OOD test sets; no-shift splits omit the OOD sets.
  • Real-world datasets: Real-world covariate splits sort graphs or nodes by domain and assign domains to training, validation, and test sets.The independence between Y and X_ind theoretically ensures that these designs do not contain concept shift.
  • Semi-artificial datasets: Semi-artificial datasets modify or append node features to represent assigned domains or concepts, using GOOD-CMNIST as the current example.The approach avoids modifying graph structures, which could break the semantics of original graphs.
  • Synthetic datasets: GOOD-Motif generates graphs from five label-independent base graphs and three label-dependent motifs to construct covariate and concept splits.Concept splits vary motif–base correlations from high probabilities in training to weak and nonexistent correlations in validation and test sets.
  • Semi-artificial datasets: GOOD-CBAS uses four-channel RGBA coloring, differing from GOOD-CMNIST’s three-channel coloring strategy.The coloring distinction is part of the semi-artificial dataset construction.
  • Environment-variable design: Environment-variable relationships differ by shift and dataset type, including association with X_ind alone for covariate shift and with both X_ind and Y for concept shift.The stated causal structures vary across synthetic, semi-artificial, and real-world datasets.

B Experimental Details

The experiments standardize backbones and training procedures across datasets and baseline methods while tuning selected algorithm-specific hyperparameters. Checkpoint selection uses ID and OOD validation sets, and implementation resources are released for reproduction.

  • Model backbones: The experiments use GIN-Virtual Node for graph prediction and GCN for node prediction, with the same backbone across methods on each dataset.GOOD-Motif is an exception, using GIN; Mixup uses a modified GCN backbone.
  • General model and hyperparameters: GOOD-SST2 uses 100 maximum epochs for DIR and 200 for other methods, while GOOD-Twitch and GOOD-WebKB use learning rates of 5e-3 for EERM and 1e-3 otherwise.These are dataset- and method-specific exceptions to the general hyperparameter settings.
  • Training procedure: Training uses Adam with zero weight decay, dropout 0.5, mean global pooling, ReLU activation, and hidden dimension 300.Dataset-specific convolutional layers, batch sizes, epoch limits, iterations, and learning rates are listed in Table 4.
  • OOD algorithm hyperparameters: One or two algorithm-specific hyperparameters are tuned for each OOD method, including penalty weights, GroupDRO step size, Mixup alpha, and related parameters.The tuned quantities correspond to each method’s regularization, optimization, or mixing mechanism.
  • Reproducibility: Best checkpoints are selected separately using ID and OOD validation sets, and datasets, code, and checkpoints are provided for reproduction.The implementation is based on PyTorch Geometric, with additional usage guidance in the package documentation.

C Empirical Results and Analysis

The benchmark reveals substantial ID/OOD performance gaps and distinct method behaviors across shift types and prediction tasks. OOD-specific methods often improve selected OOD splits, but gains are limited and shift-dependent.

  • OOD performance is significantly worse than corresponding ID performance across the evaluated splits.The benchmark therefore produces meaningful distribution shifts rather than merely relabeling in-distribution data.
  • OOD validation usually selects models with better generalization than ID validation because it contains shifts similar to the OOD test set.This advantage is not universal when models lack sufficient shift generalization.
  • VREx outperforms other methods on 7 of 34 OOD splits, while GroupDRO leads on 8 of 34 OOD splits.VREx is especially favorable for covariate shifts in graph prediction, whereas GroupDRO benefits fair optimization.
  • Mixup outperforms 6 of 14 node-task OOD splits but fails on graph prediction tasks, while DIR leads on 3 graph-classification splits.DIR specifically benefits concept shifts but does not transfer its advantage to covariate shifts or regression tasks.
  • OOD algorithms that perform well on selected OOD splits usually do not perform equally well in corresponding ID settings, whereas Mixup performs well in both.The authors associate Mixup’s broader gains with its data-augmentation nature.
  • Current OOD algorithms improve generalization abilities but not significantly, with distinct performance differences between covariate and concept shifts.The authors therefore call for more robust methods, shift-specific designs, and shift-specific evaluation metrics.

D.1 Complete numerical results

The appendix reports complete ID/OOD test performances from ID/OOD validation across GOOD datasets and baseline methods. Tables use averages and standard deviations over 10 random runs and identify the best results.

  • Complete ID/OOD test performances from ID/OOD validation are reported for 10 baselines across 11 datasets.The results appear in Tables 5–21.
  • Table 5 evaluates GOOD-HIV with the scaffold domain and reports averages ± standard deviations across 10 random runs.Bold numbers denote the best results, and the no-shift random split contains only the ID setting.
  • The appendix includes performance tables for GOOD-HIV with size domain, GOOD-PCBA with scaffold domain, and GOOD-ZINC with size domain.These tables provide dataset- and domain-specific numerical comparisons among the listed methods.

D.2 Metric score curves

Metric score curves are reported for ERM across all splits of the 11 GOOD datasets. Curves for other algorithms are provided through the project’s GitHub repository.

  • Metric score curves are reported for ERM with all splits across 11 datasets.The corresponding figures are numbered Fig. 3–13.
  • Curve figures for algorithms other than ERM are available in the GOOD GitHub repository.
  • The displayed curves provide a direct visual complement to the appendix’s numerical performance tables.

D.3 Comparison between training, validation and test scores

The appendix compares training, validation, and test scores to make ID/OOD performance gaps directly visible. These comparisons are intended to reveal distribution shifts by definition across the reported datasets and methods.

  • Training, validation, and test scores are compared directly to expose performance gaps between data splits.The comparisons are used to reveal distribution shift by definition.
  • The tables report numerical results for multiple GOOD datasets, domains, and baseline methods.Reported examples include GOOD-ZINC, GOOD-SST2, GOOD-CMNIST, GOOD-Motif, GOOD-Cora, and GOOD-Arxiv.
  • GOOD-CMNIST, GOOD-SST2, GOOD-Motif, GOOD-Cora, and GOOD-Arxiv receive dedicated comparisons in the appendix tables.The cited tables associate these datasets with color, length, base or size, word or degree, and time domains.
  • Results are presented as method-specific score rows containing repeated mean-and-variation values across the compared splits.The appendix includes rows for methods such as ERM, IRM, VREx, GroupDRO, DANN, Mixup, and DIR.

F.1 GOOD dataset loading

GOOD provides APIs and a pipeline for loading datasets, configuring models, selecting OOD algorithms, and running training or evaluation.

  • GOOD dataset loading: GOOD is maintained as an integrated project with continuous-integration tests and ongoing expansion of datasets, splits, and methods.The repository welcomes public use and technical discussions.
  • GOOD dataset loading: GOOD offers APIs for loading OOD algorithms, graph neural networks, and datasets with only several lines of code.The project also provides split-generalization code and an extensible base class for creating new OOD methods.
  • GOOD dataset loading: Dataset loaders support direct imports or registry-based loading with specified domains and shift types.Examples load GOODHIV and GOODCMNIST datasets using domains such as scaffold or color and covariate shifts.
  • GOOD dataset loading: The training and evaluation pipeline loads a configuration, initializes a model and dataset, and loads an OOD algorithm before execution.Training uses the configured model, loader, and OOD algorithm; evaluation can run on the test split.
Loading 2206.08452v2…