Source-linked AI summary
PDE-GCN: Novel Architectures for Graph Neural Networks Motivated by Partial Differential Equations
Moshe Eliasof, Eldad Haber, Eran Treister
TL;DR
Deep GCNs can suffer from over-smoothing, while existing architectures are often tailored to particular applications. PDE-GCN introduces PDE-motivated graph architectures with theoretical analysis and reports comparable or better performance across diverse problems.
Problem
GCNs can become over-smoothed when deep, and are typically customized to specific domains and applications.
Method
PDE-GCN models graph networks as discretized PDEs on graphs or manifolds, using general graph operators and diffusion or hyperbolic dynamics.
Results
PDE-GCN achieves similar or better performance than state-of-the-art models across multiple datasets and applications.
Takeaways & Limitations
The architectures can be made deep without over-smoothing and can generalize across geometric and non-geometric graph problems.
Takeaways & Limitations
The theoretical behavior depends on assumptions including monotonicity, sign preservation, and bounded activation magnitude.
Abstract
from arXiv · showhide
Graph neural networks are increasingly becoming the go-to approach in various fields such as computer vision, computational biology and chemistry, where data are naturally explained by graphs. However, unlike traditional convolutional neural networks, deep graph networks do not necessarily yield better performance than shallow graph networks. This behavior usually stems from the over-smoothing phenomenon. In this work, we propose a family of architectures to control this behavior by design. Our networks are motivated by numerical methods for solving Partial Differential Equations (PDEs) on manifolds, and as such, their behavior can be explained by similar analysis. Moreover, as we demonstrate using an extensive set of experiments, our PDE-motivated networks can generalize and be effective for various types of problems from different fields. Our architectures obtain better or on par with the current state-of-the-art results for problems that are typically approached using different architectures.
1 Introduction
GCNs face over-smoothing and are often customized to specific applications. PDE-GCN proposes a universal PDE-based design intended to support deep, flexible, and broadly applicable graph architectures.
- Over-smoothing can make node feature vectors nearly identical, limiting the performance of deep GCNs.
- PDE-GCN designs graph architectures by interpreting GCNs as PDEs on graphs or manifolds.
- General graph convolution operators based on graph gradient and divergence provide a more flexible architecture-design abstraction.
- The proposed family formulates PDE dynamics for multiple graph problems, including node classification and dense shape correspondence.
- PDE-GCN supports deep GCNs without over-smoothing and is evaluated across datasets and applications against state-of-the-art models.
2 Related work
Related work addresses over-smoothing through regularization, augmentation, or architecture design. PDE-GCN instead uses discretized PDE dynamics to support both geometric and non-geometric tasks with diffusion, hyperbolic, or mixed dynamics.
- Many GCNs use message passing, graph-Laplacian polynomials, dynamic graphs, or learned edge-weighting schemes for graph convolution.
- Existing approaches mitigate over-smoothing through normalization, edge removal, intermediate-layer aggregation, or predefined propagation kernels.
- PDE-GCN uses discretized PDEs to construct architectures that inherently avoid over-smoothing rather than relying only on added regularization.
- Unlike DiffGCN and GRAND, PDE-GCN targets geometric and non-geometric tasks and supports diffusion, hyperbolic, and mixed layer dynamics.
- Figure 1 contrasts diffusion dynamics that smooth manifold information with hyperbolic dynamics that produce a non-uniform field.
3 Methods
PDE-GCN models graph computations as discretized nonlinear PDEs on manifolds, using graph differential operators and dynamics chosen for smoothing or conservation. Its discrete architectures include diffusion, hyperbolic, and learnable mixtures with theoretical stability and behavior guarantees.
- PDE formulation: The method models graph features on a manifold and propagates them through nonlinear diffusion or hyperbolic PDEs from initial features f_0 to final features f_T.The PDEs use gradient, divergence, Laplacian, coefficient matrices, activation functions, and appropriate boundary conditions.
- PDE formulation: Diffusive dynamics are smoothing, whereas hyperbolic dynamics are bounded by a conserving mapping under the stated activation assumptions.For the hyperbolic PDE, the network energy satisfies E_net ≤ c_K, where c_K depends on K but not time.
- Discrete operators: Graph differential operators discretize manifold operators: the gradient maps vertex features to edges, divergence maps edge features to nodes, and the graph Laplacian is approximated by G^⊤G.The gradient uses learnable edge weight matrices, with the implementation scaling weights by the geometric mean of endpoint degrees.
- PDE-GCN architectures: The network discretizes diffusion with forward Euler and the second-order hyperbolic equation with leapfrog integration, using trainable 1 × 1 convolutions and a stability-preserving step size h.The layer convolutions are initialized as identity so the network begins from diffusion or hyperbolic dynamics.
- PDE-GCN architectures: PDE-GCNM mixes diffusion and hyperbolic dynamics through a learned coefficient α, updating each layer from the current and previous feature states.The mixture coefficient is parameterized as α = sigmoid(β), with 0 ≤ α ≤ 1; a separate coefficient per layer was possible but showed no benefit.
- Choice of dynamics: Learning α produces dynamics consistent with task requirements: node classification favors no second-order derivatives, while conservation-demanding applications favor hyperbolic equations.The paper relates diffusion to tasks such as segmentation and denoising, and hyperbolic dynamics to volume- or distance-preserving correspondence and protein-folding applications.
4 Experiments
Across classification, protein-interaction, shape-correspondence, and cross-domain tests, PDE-GCN models generalize across tasks while avoiding the degradation associated with over-smoothing. Experiments also show that diffusion, hyperbolic, or mixed dynamics suit different problem types.
- Experimental scope: PDE-GCN is evaluated on diverse tasks, including 3D shape classification, node classification, protein-protein interaction, and dense shape correspondence.The experiments vary in application domain and output type, using task-specific PDE-GCN blocks and final convolution steps.
- Overall results: PDE-GCN obtains performance that is either remarkably better or on par with state-of-the-art models across the considered tasks and datasets.This paper-level result is supported by the reported experiments across multiple domains and benchmark settings.
- Learning PDE dynamics: Learning the mixture coefficient favors diffusion for node classification and hyperbolic dynamics for dense correspondence, matching the differing demands of these problems.The mixed PDE-GCN combines diffusion and hyperbolic dynamics through a learned parameter α.
- Semi-supervised node classification: PDE-GCND reaches 75.6% accuracy with 32 layers on CiteSeer, compared with 74.6% using two layers, without performance degradation from increased depth.The result is also reported as a new state-of-the-art accuracy.
- Fully-supervised node classification: PDE-GCNM reaches 93.24% accuracy on Texas, compared with 77.84% for GCNII*, while similar improvements are reported for Cornell and Wisconsin.The strongest gains occur on the smaller datasets in the fully supervised evaluation.
- Inductive learning: PDE-GCND achieves 99.07 Micro-averaged F1 score on PPI, close to GCNII* at 99.58 and above methods including GAT, JKNet, and GeniePath.The evaluation uses an eight-layer PDE-GCND network for inductive learning.
- Ablation study: Removing the positive-semi-definite operator can reduce accuracy, while enforcing positive semi-definiteness improves accuracy by up to 3%.The ablation also compares activation functions and examines whether PDE-GCNM reproduces PDE-GCND results for semi-supervised learning.
5 Summary
PDE-GCN uses PDE-inspired dynamics to build graph networks that can be diffusive, conservative, or mixed, with the data selecting the appropriate behavior. The proposed networks can be made deep without over-smoothing and achieved state-of-the-art or improved performance across the studied problems.
- 5 Summary: PDE-GCN defines graph networks through PDE dynamics that are diffusive, conservative, or combinations of both.The architectures are motivated by time-dependent PDEs discretized on manifolds and graphs.
- 5 Summary: The data can select diffusive or hyperbolic dynamics according to whether anisotropic diffusion or conservation is favorable.Experiments found hyperbolic behavior for conservation-requiring problems and diffusive behavior when anisotropic diffusion is favorable.
- 5 Summary: The ablation study evaluates PDE-GCN accuracy on semi-supervised node classification.The supplied table caption identifies the task and reports accuracy as the evaluation metric.
- 5 Summary: PDE-GCN can be deep without over-smoothing and delivered state-of-the-art performance or improved it for virtually every studied problem.The authors also report dramatic improvements on data-poor problems, which they attribute to implicit regularization from the imposed dynamics and operators.
A Theorems and proofs
The appendix establishes energy bounds for the diffusive and hyperbolic PDE-GCN dynamics under stated activation-function assumptions. These bounds provide theoretical control over forward propagation, including time-independent bounds for the hyperbolic case.
- A Theorems and proofs: Under Neumann boundary conditions, a monotonically non-decreasing and sign-preserving activation yields non-increasing feature norm for the diffusive PDE.The implementation realizes these boundary conditions through the construction of the differential operators.
- A Theorems and proofs: Integration by parts gives 1/2 ∂/∂t∥f∥2 = −(K∇f, σ(K∇f)), which is used to analyze diffusive propagation.The inner-product identity connects the feature norm derivative to the graph differential operator and activation.
- A Theorems and proofs: For the hyperbolic PDE, a monotonically non-decreasing, sign-preserving activation satisfying |σ(x)| ≤|x| gives Enet ≤cK.Here cK depends on K but is independent of time.
- A Theorems and proofs: The hyperbolic energy is associated with the corresponding linear wave-like equation.The proof begins by defining this energy before establishing the network bound.
- A Theorems and proofs: For constant K, the linear energy Elin is constant in time, while the activation assumption gives Enet ≤Elin and therefore Enet ≤cK.The assumptions include sign preservation and |σ(x)| ≤|x|.
B Architectures in details
The experimental architectures share an opening layer, PDE-GCN block sequence, and closing layer, with task-specific classifier components. The three variants support node classification and inductive learning, global shape classification, and dense shape correspondence.
- B Architectures in details: All architectures comprise an opening 1 × 1 convolution, a sequence of PDE-GCN layers, and a closing 1 × 1 convolution.Additional final convolution steps may serve as classifiers.
- B Architectures in details: The first architecture uses only a closing layer for semi-supervised and fully supervised node classification, ablations, and inductive PPI learning.Its high-level structure matches GCNII, while the employed GCN block is replaced by PDE-GCN.
- B Architectures in details: The second architecture applies globalmax pooling and two MLP layers for ModelNet-10 shape classification.Each MLP layer contains a 1 × 1 convolution, ReLU, batch normalization, and dropout with probability 0.5.
- B Architectures in details: The third architecture adds ELU and final 1 × 1 convolution layers for dense shape correspondence on FAUST.Each FAUST mesh has n = 6890 vertices.
C Hyper-parameters details
The experiments use task-dependent PDE-GCN blocks, depths, widths, and final convolution steps, with hyper-parameters selected by grid search. Optimization uses Adam and task-appropriate cross-entropy losses.
- C Hyper-parameters details: Experiments select suitable PDE-GCN D, H, or M blocks with task-dependent depths, widths, and final convolution steps.The appendix reports the selected hyper-parameters for the network components.
- C Hyper-parameters details: The learning rates for PDE-GCN layers and opening or closing layers are denoted LRGCN and LRoc, respectively.Weight decay for opening and closing layers is denoted WDoc, while PDE-GCN layers use no weight decay throughout.
C.1 GCN generalization
The experiments used paper-consistent or reported hyper-parameter settings across semi-supervised Cora node classification and ModelNet-10 classification. ModelNet-10 additionally used a shared learning rate, no weight decay, and fixed hidden feature size.
- GCNII on Cora used the same settings as reported in the original GCNII paper.
- DGCNN and PDE-GCNH on Cora used the hyper-parameters reported in Table 12.
- ModelNet-10 used learning rate 0.01, no weight decay, and hidden feature size c = 64 for all networks.
C.2 Learning PDE dynamics
The learning-dynamics experiment used an eight-layer mixed PDE-GCNM initialized midway between PDE-GCND and PDE-GCNH. Its hyper-parameters were reported in the experiment tables.
- Learning PDE dynamics: The experiment used an 8-layer mixed PDE-GCNM with α = 0.5.
- Learning PDE dynamics: With α = 0.5, the mixed PDE-GCNM was balanced between PDE-GCND and PDE-GCNH.
- Learning PDE dynamics: The learning-dynamics hyper-parameters were summarized in Table 11.
- Learning PDE dynamics: The experiment hyper-parameters were summarized in Table 12.
- Learning PDE dynamics: The experiment hyper-parameters were summarized in Table 13.
C.5 Ablation study
The ablation study reused the hyper-parameters reported in Table 12. This establishes consistency of the ablation configuration with that reported setup.
- C.5 Ablation study: The ablation experiment used the same hyper-parameters as those reported in Table 12.
- C.5 Ablation study: Table 12 served as the reference configuration for the ablation experiment.
- C.5 Ablation study: The ablation setup was aligned with the hyper-parameters reported in Table 12.