PaperlayerView exact revision on arXiv

Source-linked AI summary

Dirichlet Energy Constrained Learning for Deep Graph Neural Networks

Kaixiong Zhou, Xiao Huang, Daochen Zha, Rui Chen, Li Li, Soo-Hyun Choi, Xia Hu

arXiv:2107.02392v1cs.LG

TL;DR

Deep GNNs degrade with depth because recursive aggregation causes over-smoothing, while existing remedies lack a general theoretical design principle. The paper constrains Dirichlet energy at each layer and builds EGNN around those limits. EGNN reaches 64 layers and achieves superior or competitive node-classification performance on benchmarks.

  • Problem

    Deep GNNs suffer over-smoothing as recursive aggregation makes node embeddings similar, and existing methods lack a generalizable theoretical principle for designing and training deep architectures.

  • Method

    The paper proposes Dirichlet energy constrained learning and implements it in EGNN using energy-aware weights, residual connections, and shifted ReLU activation.

  • Results

    EGNN can be trained to reach 64 layers and achieves superior node-classification performance with deep layer stacking on benchmarks.

  • Takeaways & Limitations

    Constraining Dirichlet energy within layer-wise lower and upper limits provides a theoretical guide for designing and training deep GNNs.

  • Takeaways & Limitations

    The theoretical derivation simplifies GCN by removing the nonlinear activation and considers its impact separately in model design.

Abstract

from arXiv · show

Graph neural networks (GNNs) integrate deep architectures and topological structure modeling in an effective way. However, the performance of existing GNNs would decrease significantly when they stack many layers, because of the over-smoothing issue. Node embeddings tend to converge to similar vectors when GNNs keep recursively aggregating the representations of neighbors. To enable deep GNNs, several methods have been explored recently. But they are developed from either techniques in convolutional neural networks or heuristic strategies. There is no generalizable and theoretical principle to guide the design of deep GNNs. To this end, we analyze the bottleneck of deep GNNs by leveraging the Dirichlet energy of node embeddings, and propose a generalizable principle to guide the training of deep GNNs. Based on it, a novel deep GNN framework -- EGNN is designed. It could provide lower and upper constraints in terms of Dirichlet energy at each layer to avoid over-smoothing. Experimental results demonstrate that EGNN achieves state-of-the-art performance by using deep layers.

1 Introduction

Deep GNNs suffer from over-smoothing as recursive aggregation makes node representations indistinguishable, while existing remedies lack a general theoretical design principle. The paper proposes Dirichlet energy constrained learning and EGNN to train deep models within appropriate energy ranges.

  • Motivation: Stacking more than a few GNN layers can significantly reduce performance because node representations converge to indistinguishable vectors.Experiments often find the best performance with fewer than 3 layers, limiting access to high-order neighbors.
  • Motivation: Existing remedies draw from CNN techniques or heuristics, and many achieve performance comparable to or worse than shallow models.These approaches include embedding normalization, residual connections, random embedding propagation, and edge dropping.
  • Proposed Principle: The paper introduces Dirichlet energy constrained learning, which regularizes each layer's energy within an appropriate range to avoid over-smoothing and over-separating.Training jointly optimizes the task loss and energy value.
  • EGNN: EGNN implements the principle through orthogonal weight controlling, lower-bounded residual connections, and shifted ReLU activation.These components respectively address the upper energy limit, lower energy limit, and the trade-off between nonlinear and linear mappings.
  • Results: EGNN can be trained to reach 64 layers and achieves competitive benchmark performance, while the principle explains most existing deep-GNN techniques.The reported result supports using energy constraints as a general design and training guide.

2 Problem Statement

A GCN layer transforms node embeddings through normalized neighborhood aggregation, feature weights, and an activation function. Repeating this operation makes embeddings increasingly similar, causing over-smoothing and limiting high-order neighborhood modeling.

  • Problem Setup: A graph is represented as G = (A, X), where A is the adjacency matrix and X is the node feature matrix.The node classification task predicts labels for test nodes using a graph and labels from training nodes.
  • Problem Setup: The augmented normalized adjacency matrix ˜P is constructed from self-looped adjacency and degree matrices for neighborhood aggregation.The corresponding augmented normalized Laplacian is ˜∆ := I_n − ˜P.
  • GCN Operation: At layer k, GCN updates node embeddings by aggregating neighbors, applying trainable feature weights, and then using an activation function.The initial representation is X(0) = X, and final embeddings are optimized with a node-classification loss.
  • Over-smoothing: Recursive aggregation makes node embeddings similar as depth increases, producing over-smoothing that prevents traditional GNNs from exploring many-hop neighbors.Because high-order dependencies matter for node classification, shallow GNNs can have sub-optimal downstream performance.

3 Dirichlet Energy Constrained Learning

The paper uses Dirichlet energy to measure embedding smoothness and constrains it at every layer between lower and upper limits. This prevents both over-smoothing from excessively small energy and over-separating from excessively large energy, while accommodating graph- and layer-dependent behavior.

  • Energy Measure: Dirichlet energy measures embedding smoothness through weighted node-pair distances: smaller values indicate over-smoothing, whereas larger values indicate over-separating.For classification, the desired energy separates different classes while keeping same-class nodes close.
  • Analysis Assumption: The theoretical derivation simplifies GCN by removing nonlinear activation, whose effect is handled separately in model design.The analysis is therefore based on a linearized propagation form involving repeated graph propagation and weight products.
  • Failure Modes: Without proper weight design and training, Dirichlet energy can become too small or too large, causing over-smoothing or energy explosion and over-separating.Common initialization and L2 regularization can make intermediate weights nearly zero, while large singular values can produce excessive energy.
  • Constrained Learning: Dirichlet energy constrained learning defines layer-wise lower and upper limits and jointly optimizes node-classification loss with energy during training.The appropriate range varies across layers and applications rather than being fixed in advance.
  • Energy Bounds: The lower limit keeps each layer's energy above a threshold to overcome over-smoothing, while the upper limit prevents over-separating relative to the transformed initial feature.The lower and upper controls are parameterized by c_min and c_max.
  • Energy Bounds: The constrained formulation requires c_min E(X^(k−1)) ≤ E(X^(k)) ≤ c_max E(X^(0)) for k = 1, …, K.This expresses the desired energy interval across the network's layers.

4 Energetic Graph Neural Networks - EGNN

EGNN implements Dirichlet energy constrained learning through weight control, residual connections, and activation design. These components target both lower- and upper-energy limits while explaining how existing deep-GNN techniques affect energy.

  • Constrained learning: EGNN replaces direct constrained optimization with weight control, residual connections, and activation design to efficiently satisfy Dirichlet-energy constraints.The direct problem is difficult because computing energy is expensive and many constraints create a complex optimization landscape.
  • Orthogonal weight controlling: Orthogonal initialization sets first-layer weights to √cmax · Id and higher-layer weights to Id, ensuring E(X(k)) ≤ cmaxE(X(0)) at initialization.The corresponding square singular values are cmax for the first layer and 1 for higher layers.
  • Orthogonal weight controlling: Weight penalization keeps trainable weights near their initialized values, balancing upper-energy control against freedom to optimize the node-classification loss.A larger γ enforces stronger regularization, whereas a smaller γ permits more adaptation to task data.
  • Lower-bounded residual connection: Residual strengths satisfy α + β = cmin, and under stated conditions EGNN maintains cminE(X(k−1)) ≤ E(X(k)) ≤ cmaxE(X(0)).The lower and upper bounds require the conditions stated in Lemmas 4 and 5.
  • SReLU activation: Because ReLU can further reduce Dirichlet energy, EGNN uses SReLU to interpolate between nonlinear and approximately linear behavior through a trainable shift b.The shift is initialized negatively and adapted through back-propagation.
  • Connections to existing methods: The energy perspective interprets normalization, DropEdge, and residual methods as strategies affecting embedding distances, information propagation, or energy bounds.DropEdge can slow energy decrease by reducing the relevant eigenvalue, but its dropping rate must be chosen carefully for different tasks.

5 Experiments

Experiments evaluate EGNN against deep GNN baselines on four node-classification benchmarks, examining accuracy, energy constraints, component ablations, and hyperparameter sensitivity. EGNN generally performs best in deep settings while maintaining Dirichlet energy within prescribed limits and remaining effective across broad hyperparameter ranges.

  • Experimental setup: EGNN is evaluated for node classification on Cora, Pubmed, Coauthor-Physics, and Ogbn-arxiv against seven state-of-the-art baselines.Results use 2/16/64 layers for Cora and Pubmed, and 2/16/32 layers for Coauthor-Physics and Ogbn-arxiv.
  • Node classification results: EGNN generally outperforms all baselines across the four datasets, especially at depths K ≥16.Its node-classification accuracy consistently improves with stacking until K = 32 or 64, whereas many competing deep models decline with depth.
  • Dirichlet energy visualization: EGNN constrains Dirichlet energy at every layer between learned lower and upper limits in Cora and Pubmed.In contrast, other methods exhibit energy close to zero or overly large because of over-smoothing or over-separating node embeddings.
  • Ablation studies: Orthogonal weight initialization and regularization are crucial for training deep EGNN models.Replacing them with Glorot initialization and Frobenius regularization can produce overly large or small Dirichlet energy and reduce classification performance on three datasets.
  • Ablation studies: An appropriate cmin range of [0.1, 0.75] supports deep EGNN performance, while cmin = 0.95 preserves too much residual information for higher layers to learn new neighborhoods.Without the residual connection, Dirichlet energy may approach zero.
  • Ablation studies: SReLU performs slightly better than linear identity and ReLU by balancing linear and nonlinear activation effects.The paper attributes this to preventing a significant drop in Dirichlet energy while preserving model learning ability.
  • Hyperparameter analysis: EGNN is not sensitive to b, γ, cmin, and cmax across wide ranges, including b ≤0, γ ≥1, cmin in [0.1, 0.75], and cmax in [0.2, 1].Figure 2 reports the Cora study for 64-layer EGNN; the paper states similar tendencies for other datasets.

6 Conclusions

The paper proposes Dirichlet energy constrained learning and designs EGNN to support deep GNNs. Experiments show that energy constraints help EGNN achieve strong node-classification performance with deep layer stacking.

  • Dirichlet energy constrained learning regularizes each layer within lower and upper limits to avoid over-smoothing and over-separating.The principle jointly optimizes task loss and energy value.
  • EGNN is designed from the theoretical energy-constraint results to guide deep graph-neural architecture design and training.
  • Experiments on benchmark datasets show that EGNN can achieve superior node-classification performance with deep layer stacking.
  • The constrained-learning principle connects and combines previous deep-GNN methods and may guide the discovery of deeper GNNs.

A.1 Dataset Statistics

The experiments use four benchmark graph datasets: Cora, Pubmed, Coauthor-Physics, and Ogbn-arxiv. Dataset statistics are reported in Table 3.

  • Four benchmark datasets are used: Cora, Pubmed, Coauthor-Physics, and Ogbn-arxiv.Cora and Pubmed use public train/validation/test splits, while Coauthor-Physics is randomly split following previous practice.
  • Table 3 summarizes the data statistics for the benchmark datasets.

A.2 Baselines

The evaluation compares EGNN with GCN and several deep GNN baselines based on GCN for node classification. The baselines use normalization, edge dropping, simplified propagation, layer combination, or personalized propagation.

  • The baseline set includes GCN and state-of-the-art deep GNNs based on GCN for node-classification evaluation.
  • PairNorm normalizes node embeddings between graph-convolution layers to alleviate over-smoothing.
  • DropEdge randomly removes input-graph edges during training to reduce the convergence speed of over-smoothing.
  • SGC removes hidden weights and activation functions from GCN, while JKNet combines hidden embeddings at the final layer.
  • APPNP improves vanilla GCN message propagation using personalized PageRank while simplifying the model.

A.3 Implementation Details

Experiments train for up to 1500 epochs with Adam and early stopping, using benchmark-specific hyperparameters. Results are averaged over 10 independent runs.

  • Each experiment uses a maximum of 1500 epochs, the Adam optimizer, and early stopping.
  • Key training hyperparameters for each benchmark are listed in Table 4.
  • Reported experiment results are averages over 10 independent runs.

A.4 Lower Limit Setting

EGNN selects lower-limit settings from validation performance and Dirichlet energy, with residual strengths constrained by α + β = cmin. The chosen values vary by dataset and depth.

  • Lower-limit selection: EGNN selects cmin from [0.1, 0.75] using validation classification performance and Dirichlet energy.The residual connection strengths satisfy α + β = cmin.
  • Dataset-specific settings: For Cora, cmin is 0.2 below 32 layers and 0.15 at 32 or more layers, with α = β = 0.1.
  • Dataset-specific settings: For Pubmed, cmin is 0.12 below 32 layers and 0.11 at 32 or more layers, with β = cmin and α = 0.
  • Dataset-specific settings: For Coauthor-Physics, cmin is 0.12 with β = 0.1 and α = 0.02.
  • Dataset-specific settings: For the remaining Coauthor-Physics settings, cmin and β are 0.6 and 0.1 below 32 layers, and 0.75 and 0.25 at 32 or more layers, with α = 0.5.

A.5 Proof for Lemma 1

The proof bounds each layer’s Dirichlet energy by controlling the trainable weights and residual graph convolution. Orthogonal initialization, residual connections, and activation properties establish upper and lower energy constraints across layers.

  • Layer-wise energy bounds: Lemma 1 bounds the k-th layer’s Dirichlet energy using the minimum and maximum square singular values of the weight transformation.The upper bound uses σmax(W^(k)(W^(k))^T), while the lower bound uses σmin(W^(k)(W^(k))^T).
  • Upper-limit control: Orthogonal initialization sets W^(1) to √cmax·I_d and later weights to I_d, yielding the initial upper-energy control E(X^(k)) ≤ cmaxE(X^(0)).The first-layer square singular values are cmax, while higher-layer values are 1.
  • Lower-limit control: If cmax ≥ cmin/(2cmin − 1)^2, orthogonal control and residual connections ensure E(X^(k)) ≥ cminE(X^(k−1)).The proof derives this condition from the minimum square eigenvalue at the first layer and shows the lower bound persists at higher layers.
  • Residual formulation: EGNN’s residual graph convolution combines propagated features, the previous-layer embedding, and the initial embedding before applying W^(k).The residual strengths satisfy α + β = cmin, and Q is defined as (1 − cmin)P̃ + αI_n.
  • Practical conditions: With cmax = 1, the upper bound is readily satisfied, while appropriate parameter ranges make EGNN trainable across deep layers.The reported broad ranges are b ∈ (−∞, 0], γ ∈ [1, ∞], cmin ∈ [0.1, 0.75], and cmax ∈ [0.2, 1].
  • Activation functions: ReLU and Leaky-ReLU do not increase Dirichlet energy, so activation functions preserve the established upper bound.The proof uses their Lipschitz and positive-homogeneity properties.
Loading 2107.02392v1…