Source-linked AI summary
DeeperGCN: All You Need to Train Deeper GCNs
Guohao Li, Chenxin Xiong, Ali Thabet, Bernard Ghanem
TL;DR
GCNs face unclear aggregation choices and uncertain behavior of deep-network modules on large-scale graphs. DeeperGCN introduces generalized aggregation, modified skip connections, and graph normalization, achieving new state-of-the-art results across four OGB datasets.
Problem
Aggregation choices remain unclear, and the behavior of skip connections and common aggregations on large graphs is uncertain.
Method
DeeperGCN combines differentiable permutation-invariant generalized aggregation with modified residual connections and a graph normalization layer.
Results
DeeperGCN improves state-of-the-art performance by 7.8%, 0.2%, 6.7% and 0.9% on ogbn-proteins, ogbn-arxiv, ogbg-ppa and ogbg-molhiv, respectively.
Takeaways & Limitations
The proposed tools set new state-of-the-art results on four challenging OGB datasets and provide a new view of aggregation-function design.
Takeaways & Limitations
Training very deep DeeperGCN models requires more GPU memory and time and may increase energy consumption.
Abstract
from arXiv · showhide
Graph Convolutional Networks (GCNs) have been drawing significant attention with the power of representation learning on graphs. Unlike Convolutional Neural Networks (CNNs), which are able to take advantage of stacking very deep layers, GCNs suffer from vanishing gradient, over-smoothing and over-fitting issues when going deeper. These challenges limit the representation power of GCNs on large-scale graphs. This paper proposes DeeperGCN that is capable of successfully and reliably training very deep GCNs. We define differentiable generalized aggregation functions to unify different message aggregation operations (e.g. mean, max). We also propose a novel normalization layer namely MsgNorm and a pre-activation version of residual connections for GCNs. Extensive experiments on Open Graph Benchmark (OGB) show DeeperGCN significantly boosts performance over the state-of-the-art on the large scale graph learning tasks of node property prediction and graph property prediction. Please visit https://www.deepgcns.org for more information.
1 Introduction
GCNs face unclear aggregation choices and training challenges on large graphs, so DeeperGCN combines generalized aggregation, modified skip connections, and graph normalization. On four OGB datasets, the resulting techniques improve state-of-the-art performance.
- Motivation: GCNs update node features by aggregating information from connected neighbors, but suitable aggregation mechanisms remain unclear.Aggregation functions must be permutation invariant; mean and sum suit some node-classification tasks, while max benefits 3D point-cloud processing.
- Motivation: Prior deep-GCN modules address vanishing gradients and expand receptive fields, but their behavior on large-scale graphs remains uncertain.These modules adapt residual, dense, and dilated-convolution designs from CNNs, enabling GCNs deeper than 100 layers.
- Contributions: DeeperGCN introduces a differentiable Generalized Aggregation Function that covers common aggregations and supports task-specific tuning or end-to-end learning.The function is permutation invariant and includes operations such as mean and max.
- Contributions: Modified skip connections and a novel graph normalization layer enhance GCN performance on large-scale graphs.The paper evaluates each addition experimentally as part of a suite of techniques for training GCNs.
- Results: 7.8%, 0.2%, 6.7% and 0.9% improve current state-of-the-art performance on ogbn-proteins, ogbn-arxiv, ogbg-ppa and ogbg-molhiv, respectively.These results come from extensive experiments on four Open Graph Benchmark datasets.
2 Related Work
Related work develops permutation-invariant neighborhood aggregation and increasingly deep GCN architectures. Earlier methods generally remain shallow, while DeepGCNs extend depth using CNN-inspired connectivity modules.
- Aggregation Functions for GCNs: GCN aggregation functions must be permutation invariant, and prior work studies mean, max, LSTM, attention-weighted mean, and learned aggregators.These approaches differ in how they combine unordered neighbor information, with some aggregators performing better empirically.
- Aggregation Functions for GCNs: Mean, max, and sum aggregators preserve graph invariance or equivariance, but mean and max are less powerful than the Weisfeiler-Lehman test.They nevertheless perform effectively on node-classification and 3D point-cloud tasks.
- Training Deep GCNs: Most earlier deep-GCN approaches degrade beyond 10 layers, limiting their usable depth.Several works attempted deeper training through different architectural strategies.
- Training Deep GCNs: DeepGCNs train 56-layer GCNs by adapting residual and dense connections from deep CNN architectures.These designs represent a prior response to the difficulty of training deeper graph-convolutional networks.
3 Representation Learning on Graphs
The paper formulates graph representation learning as transforming graphs through message passing. Each layer constructs neighbor messages, aggregates them with a permutation-invariant function, and updates vertex features.
- Graph Representation: A graph is represented as vertices and edges, with optional vertex features in R^D and edge features in R^C.For directed graphs, an edge specifies direction; for undirected graphs, it connects two vertices.
- GCNs for Learning Graph Representation: GCN representation learning uses an operator F that maps an input graph G to a transformed graph G′ with learned features or topology.The paper focuses on the GCN family and its message-passing framework.
- Message Passing: At each layer, message construction combines a vertex, neighbor, and corresponding edge features into an individual neighbor message.The message-construction function ρ(l) is applied to the relevant vertex and edge features.
- Message Passing: Message aggregation applies a permutation-invariant set function to the unordered messages from a vertex’s neighbors.This invariance guarantees invariance or equivariance to isomorphic graphs; common choices include mean, max, and sum.
- Message Passing: The vertex update combines the original vertex features with the aggregated message to produce the next-layer representation.The update is implemented through the vertex-update function φ(l).
4 Aggregation functions for GCNs
The section develops differentiable, permutation-invariant aggregation families that can interpolate between mean and max, then integrates them with positive-message construction, MsgNorm, and pre-activation residual connections for deep GCNs.
- Generalized aggregation functions: Generalized mean-max aggregators form a continuous, differentiable family of permutation-invariant functions that can interpolate between mean and max for task-specific aggregation.SoftMax and PowerMean are proposed as two such families, with learnable or tunable parameters.
- Generalized aggregation functions: SoftMax aggregation approaches mean at low inverse temperatures and max at high inverse temperatures, with β controlling the transition.Formally, β→0 yields Mean(·), while β→∞ yields Max(·).
- Generalized aggregation functions: PowerMean aggregation unifies harmonic, geometric, arithmetic, and quadratic means, while p=1 gives arithmetic mean and p→∞ approaches max.Its stated guarantees require positive message features.
- Generalized Aggregation Networks: The generalized message aggregator keeps message features nonnegative, enabling SoftMax or PowerMean aggregation within the message-passing GCN.ReLU produces nonnegative values, and ε is set to 10^-7 to maintain positivity.
- Generalized Aggregation Networks: Pre-activation residual connections reorder the block as Normalization → ReLU → GraphConv → Addition, and empirically perform better than the post-activation ordering.The design keeps the residual-function output range unrestricted before addition.
- Generalized Aggregation Networks: MsgNorm normalizes the aggregated message and combines it with vertex features using a learnable scaling factor before the vertex update.The scaling factor s is initialized to 1, and the normalized message is scaled relative to the vertex-feature norm.
5 Experiments
The experiments evaluate generalized aggregators, residual connections, and message normalization across large-scale OGB datasets. Ablations on ogbn-proteins and comparisons with state-of-the-art methods assess their effects on deep GCN performance.
- Experimental setup: Experiments use OGB datasets for ablations, cross-dataset evaluation, and comparisons with state-of-the-art methods.The study includes node-property and graph-property prediction datasets, with ogbn-proteins used for component ablations.
- Generalized message aggregators: 0.860 ROC-AUC is achieved by 112-layer ResGEN with SoftMax_Agg when β = 10^4 on ogbn-proteins.SoftMax_Agg interpolates between Mean-like and Max-like behavior as β changes; PowerMean_Agg performs best at p = 5 among tested values.
- Dynamic aggregation: Learning β or β&s raises average SoftMax_Agg performance from 0.838 to 0.850, while DyResGEN achieves 0.860 when β is learned.DyResGEN learns aggregation parameters dynamically rather than exhaustively searching fixed hyperparameters.
6 Conclusion
DeeperGCN combines generalized aggregation, residual-connection, and message-normalization ideas to train deeper GCNs. The resulting models achieve new state-of-the-art performance on four challenging OGB datasets, while deeper training requires more memory, time, and potentially energy.
- DeeperGCN introduces differentiable generalized message aggregation, a new residual-connection variant, and message normalization layers for deeper GCN training.The aggregation function defines a family of permutation-invariant functions.
- New state-of-the-art performance is reported on four challenging Open Graph Benchmark datasets.
- DeeperGCN models significantly outperform existing state-of-the-art methods on OGB biological and chemical graph datasets.The authors connect these results to possible benefits for drug discovery and protein research.
- Training very deep DeeperGCN models requires more GPU memory and time and may increase energy consumption.The paper identifies efficiency optimization as future work.
A Discussion on Generalized Message Aggregation Functions
The generalized aggregation framework provides differentiable, permutation-invariant functions that include mean and max behavior and can be tuned for different tasks. The proposed functions also approach min aggregation at extreme parameter values, while sum aggregation is left for future work.
- Generalized aggregation functions form a differentiable family of permutation-invariant aggregators for GCN message passing.
- SoftMax_Aggβ(·) and PowerMean_Aggp(·) cover mean and max aggregation within the generalized function space.
- Both proposed variants can be instantiated as a min aggregator as β or p approaches −∞.
- The generalized aggregation parameters can be tuned for diverse tasks or learned end-to-end because the functions are fully differentiable.
- Sum aggregation is not generalized in this work, although the authors describe a possible degree-control extension for future work.
B Proof for Proposition 4
The proof establishes that SoftMax_Aggβ is permutation invariant and converges to mean aggregation as β approaches zero and max aggregation as β approaches infinity.
- SoftMax_Aggβ remains unchanged under any permutation of the input message set, satisfying permutation invariance.
- As β approaches zero, SoftMax_Aggβ converges to mean aggregation.
- As β approaches infinity, SoftMax_Aggβ converges to max aggregation.
- Therefore, SoftMax_Aggβ is a generalized mean-max aggregation function.
C Proof for Proposition 5
The proof shows that PowerMean_Aggp is permutation invariant and interpolates between mean and max aggregation through its parameter p.
- PowerMean_Aggp is invariant to permutations of the input message set.
- At p = 1, PowerMean_Aggp equals mean aggregation.
- As p approaches infinity, PowerMean_Aggp converges to max aggregation.
- Thus, PowerMean_Aggp is a generalized mean-max aggregation function across message-feature dimensions.
D Analysis of DyResGEN
This section analyzes DyResGEN on ogbn-proteins using learning curves for its learnable aggregation and message-normalization parameters. The experiments use 7-layer models with dropout to reduce over-fitting.
- D Analysis of DyResGEN: DyResGEN is evaluated on the ogbn-proteins dataset using 7-layer models with SoftMax_Aggβ(·) and PowerMean_Aggp(·) aggregators.The learning curves track the learnable parameters β, p, and s for the two aggregator variants.
- D Analysis of DyResGEN: The experiments include MsgNorm layers with a learnable norm scaling factor s.All learnable parameters are initialized as 1.
- D Analysis of DyResGEN: Dropout with a rate of 0.1 is used in each layer to prevent over-fitting.
E More Details on the Experiments
The experiments cover four OGB datasets with dataset-specific model configurations and report results averaged over 10 independent runs. Additional figures show learning curves for DyResGEN variants using SoftMax and PowerMean aggregation with MsgNorm.
- E More Details on the Experiments: Experiments cover ogbn-proteins, ogbn-arxiv, ogbg-ppa, and ogbg-molhiv using dataset-specific model configurations.
- E More Details on the Experiments: Results are averaged over 10 independent runs, with the corresponding mean and standard deviation reported in Table 6.
- ogbn-proteins and ogbg-ppa: For ogbn-proteins and ogbg-ppa, node features are initialized by aggregating connected-edge features with Sum aggregation.This applies because these datasets provide no node features.
- ogbn-proteins: The ogbn-proteins configuration trains a 112-layer DyResGEN with hidden channel size 64, layer normalization, and 0.1 dropout.Training uses Adam at learning rate 0.01 for 1000 epochs.
- ogbn-arxiv and ogbg-ppa: The ogbn-arxiv and ogbg-ppa configurations use 28-layer ResGEN models with SoftMax_Aggβ(·) and fixed β values of 0.1 and 0.01, respectively.Both configurations use dropout rate 0.5 and Adam with learning rate 0.01.
- ogbg-molhiv: The ogbg-molhiv configuration uses a 7-layer DyResGEN with learnable β, hidden channel size 256, batch normalization, and dropout rate 0.5.Training uses Adam at learning rate 0.01 for 300 epochs.