PaperlayerView exact revision on arXiv

Source-linked AI summary

DeepGCNs: Making GCNs Go as Deep as CNNs

Guohao Li, Matthias Müller, Guocheng Qian, Itzel C. Delgadillo, Abdulellah Abualshour, Ali Thabet, Bernard Ghanem

arXiv:1910.06849v3cs.CVcs.LGeess.IV

TL;DR

GCNs process non-Euclidean data but have been constrained by shallow architectures and training problems such as vanishing gradients. This work transfers residual, dense, and dilated-convolution concepts from CNNs to GCNs, enabling models up to 112 layers. Across point-cloud segmentation and biological-graph classification, the resulting DeepGCNs achieve strong reported performance.

  • Problem

    GCNs support non-Euclidean data, but deep architectures are limited by vanishing gradients and related over-smoothing, restricting most models to shallow depths.

  • Method

    The paper adapts residual connections, dense connections, and dilated convolutions to GCNs, and evaluates variants including MRGCN across graph operators and tasks.

  • Results

    DeepGCNs train successfully at 112 layers and improve S3DIS by 3.9% mIOU, outperform prior methods on many PartNet classes, and reach an F1 score of 99.43 on PPI.

  • Takeaways & Limitations

    CNN concepts can be transferred to GCNs to support deeper architectures and strong performance across point-cloud and biological-graph tasks.

  • Takeaways & Limitations

    The paper leaves alternative operators, architectures, distance measures, graph-construction choices, and dilation schedules for future work.

Abstract

from arXiv · show

Convolutional Neural Networks (CNNs) have been very successful at solving a variety of computer vision tasks such as object classification and detection, semantic segmentation, activity understanding, to name just a few. One key enabling factor for their great performance has been the ability to train very deep networks. Despite their huge success in many tasks, CNNs do not work well with non-Euclidean data, which is prevalent in many real-world applications. Graph Convolutional Networks (GCNs) offer an alternative that allows for non-Eucledian data input to a neural network. While GCNs already achieve encouraging results, they are currently limited to architectures with a relatively small number of layers, primarily due to vanishing gradients during training. This work transfers concepts such as residual/dense connections and dilated convolutions from CNNs to GCNs in order to successfully train very deep GCNs. We show the benefit of using deep GCNs (with as many as 112 layers) experimentally across various datasets and tasks. Specifically, we achieve very promising performance in part segmentation and semantic segmentation on point clouds and in node classification of protein functions across biological protein-protein interaction (PPI) graphs. We believe that the insights in this work will open avenues for future research on GCNs and their application to further tasks not explored in this paper. The source code for this work is available at https://github.com/lightaime/deep_gcns_torch and https://github.com/lightaime/deep_gcns for PyTorch and TensorFlow implementation respectively.

1 INTRODUCTION

GCNs extend graph learning to non-Euclidean data but have typically remained shallow because deep architectures suffer training instability and over-smoothing. DeepGCNs transfer residual, dense, and dilated-convolution ideas from CNNs, enabling very deep models and strong results across point-cloud and biological-graph tasks.

  • GCNs address irregular data such as point clouds and graphs, but existing applications largely rely on simple, shallow architectures.
  • Vanishing gradients and over-smoothing make deep GCNs difficult to train, leaving most architectures no deeper than 4 layers.
  • Residual connections provide additional gradient paths, while dense connections add further cross-layer connectivity for deep networks.
  • Dilated convolutions increase receptive fields with depth while reducing the loss of spatial resolution caused by pooling.
  • 112 layers diverge without residual connections but train stably with them under the same S3DIS semantic-segmentation hyperparameters.
  • 3.9% mIOU improvement on S3DIS, strong PartNet class performance, and an F1 score of 99.43 on PPI demonstrate DeepGCN benefits across tasks.
  • The journal extension adds more-than-100-layer architectures, PartNet and PPI experiments, MRGCN studies, and reproducibility resources.

2 RELATED WORK

GCNs support applications involving irregular relational and geometric data, including social networks, biological graphs, scene graphs, and point clouds. However, prior efforts to deepen GCNs generally reported performance degradation beyond modest depths, motivating CNN-inspired remedies.

  • GCNs model non-Euclidean relationships in social networks, protein applications, recommendation systems, and point-cloud processing.
  • GCNs are used in computer vision for scene graphs, object segmentation, semantic-relation prediction, image reconstruction, and human-joint modeling.
  • Point clouds are well suited to graph processing because their unordered structure poses a representational challenge for systematic methods.
  • Prior deeper-GCN studies observed degradation beyond 3 layers for semi-supervised node classification and beyond 10 layers for collective classification.
  • Residual and dense connections, together with dilated convolutions, transfer CNN strategies for addressing vanishing gradients and limited receptive fields in GCNs.

3 METHODOLOGY

The methodology adapts graph convolution operations and CNN-inspired connectivity to construct deeper GCNs. It combines residual or dense layer connections with dilated, dynamically recomputed neighborhoods while keeping the prediction pipeline consistent across architectures.

  • Graph Representation Learning: GCNs represent vertices with feature vectors and update each vertex by aggregating features from its neighborhood before applying a nonlinear transformation.The framework uses neighborhood aggregation and vertex-feature update functions; its point-cloud experiments use max-pooling over feature differences.
  • Dynamic Edges: Dynamic graph convolution recomputes neighbors between vertices at each layer, producing an effectively larger receptive field for deeper GCNs.The framework constructs k-NN graphs in each layer's feature space and uses dilated k-NN to select neighbors within a larger neighborhood region.
  • Point-Cloud Architecture: The point-cloud architecture contains a GCN backbone, a fusion block for global feature generation and fusion, and an MLP block for point-wise label prediction.PlainGCN, ResGCN, and DenseGCN differ in the backbone, while the fusion and prediction blocks are kept the same for fair comparison.
  • Deep GCN Design: DeepGCNs transfer residual connections, dense connections, and dilated aggregation from CNNs to GCNs to improve convergence and support deeper architectures.The study analyzes each component's effect on the stability and accuracy of deep GCN training.
  • Layer Connections: Residual GCNs add transformed graph representations vertex-wise, whereas DenseGCNs concatenate the input graph with intermediate layer outputs.The residual model is called ResGCN; DenseGCN densely fuses representations from previous layers through vertex-wise concatenation.
  • Dilated Aggregation: Dilated aggregation selects k nearest neighbors from a k×d neighborhood by skipping every d neighbors, using feature-space distance and stochastic aggregation during training.The dilation rate increases with network depth in the point-cloud experiments, and random neighbor sampling is used with a small probability for generalization.

4 EXPERIMENTS ON 3D POINT CLOUDS

The framework addresses vanishing gradients in GCNs with residual and dense connections, while dilated graph convolutions enlarge receptive fields for 3D point-cloud segmentation.

  • ResGCN and DenseGCN address the vanishing gradient problem in GCNs.
  • Dilated graph convolution enlarges the receptive field of GCNs.
  • Point-cloud segmentation represents each point as a graph vertex and predicts every vertex’s category at the output layer.
  • Dynamic dilated k-NN constructs edges in coordinate space at the first layer and feature space at subsequent layers.

4.2 Experimental Setup

The experiments evaluate point-cloud segmentation with OA and mIoU, use an ablation study on S3DIS, and compare a 28-layer reference model across datasets and methods.

  • OA and mIoU across all classes evaluate semantic and part segmentation performance.
  • The S3DIS experiments use area 5 for ablation analysis of each component.
  • ResGCN-28 uses a 28-layer backbone with residual graph connections and stochastic dilated graph convolutions.
  • ResGCN-28 is evaluated across all six S3DIS areas against the shallow DGCNN baseline and other state-of-the-art methods.
  • PartNet evaluates whether the proposed method generalizes to point-cloud part segmentation beyond S3DIS.

4.3 Network Architectures

All architectures share GCN backbone, fusion, and MLP prediction blocks; experiments vary the backbone through residual, dense, and dilated graph connections.

  • All architectures contain GCN backbone, fusion, and MLP prediction blocks, with differences confined to the backbone.
  • The fusion block concatenates features from every GCN layer, aggregates global graph information, and combines it with local vertex features.
  • PlainGCN stacks dynamic k-NN EdgeConv layers without skip connections.
  • ResGCN adds dynamic dilated k-NN and residual graph connections without increasing the number of parameters.
  • DenseGCN adds dynamic dilated k-NN and dense graph connections by concatenating intermediate representations from previous layers.

4.4 Implementation

Implementations use TensorFlow for S3DIS and PyTorch for PartNet, with shared optimization settings and regularization choices across experiments.

  • S3DIS models are implemented in TensorFlow, while PartNet models are implemented in PyTorch.
  • All experiments use Adam with an initial learning rate of 0.001 and a schedule decaying 50% every 3 × 10^5 gradient descent steps.
  • Batch normalization is applied to every layer, and dropout with rate 0.3 is used at the second MLP layer.

4.5 Results

Across S3DIS and PartNet, deep GCN performance depends on residual connections, dilation, normalization, and sufficient capacity, while the resulting architectures outperform shallower or ablated baselines.

  • Dilation: 2.85% mean IoU improvement comes from dilated graph convolutions, whose expanded receptive field especially benefits deep networks with residual connections.Without residual connections, dilation can instead degrade performance because varying neighbors may further hinder convergence.
  • Nearest neighbors: 2.5% and 3.3% performance drops result when the number of neighbors is reduced by factors of 2 and 4, respectively.More neighbors help only when network capacity is sufficiently large.
  • Depth and width: Increasing network depth improves performance only when residual graph connections and dilated graph convolutions are used, while increasing width provides a similar performance gain.Higher capacity helps the network learn nuances needed for difficult cases.
  • Normalization: 6.1% and 4% performance drops occur for ResGCN-28 and ResGCN-7 respectively when batch normalization is removed.ResGCN-28 without batch normalization is outperformed by ResGCN-7 with batch normalization, indicating the importance of normalization for deep models.

5 EXPERIMENTS ON BIOLOGICAL NETWORKS

Experiments on PPI biological networks evaluate deep GCNs across graph operators, widths, depths, connections, performance, and memory. Residual and dense graph connections support deeper models, while MRGCN combines strong performance with lower memory use.

  • Experimental setup: The PPI dataset contains 24 graphs—20 for training, 2 for validation, and 2 for testing—for multi-label node classification.Each graph corresponds to a different human tissue.
  • Experimental setup: The experiments vary graph connections, filters from 32 to 256, depths from 3 to 112 layers, and multiple GCN operators.The study also applies residual connections across GCN variants.
  • Effect of graph connections: At depth 112 with 32 filters, ResMRGCN-112 achieves a 37.66% higher m-F1 score than PlainMRGCN-112.DenseMRGCN performs slightly better than ResMRGCN at the same depth and width.
  • Effect of network depth: Increasing depth improves performance with residual or dense connections, whereas PlainMRGCN performance drops significantly beyond 14 layers.DenseMRGCN improves reliably with depth but consumes more memory because feature maps are concatenated.
  • Effect of network width: Increasing filters consistently improves performance and can aid convergence in deep networks, but the experiments limit width to 256 filters because larger widths consume substantial memory.Width and depth therefore involve a performance–memory trade-off.
  • Memory and operator comparisons: ResMRGCN achieves the best performance among the compared GCN variants while using around 15% of the GPU memory used by ResEdgeConv.The comparison uses 56 layers and 256 filters per layer with residual graph connections.
  • Comparison to state-of-the-art: DenseMRGCN-14 and ResMRGCN-28 outperform Cluster-GCN by 0.07% and 0.05%, respectively, while ten Table 6 models surpass Cluster-GCN.The comparison uses m-F1 on PPI node classification.

6 CONCLUSION

The conclusion presents residual and dense connections plus dilated convolutions as transferable CNN concepts for training deep GCNs. Experiments across point clouds and biological graphs report strong performance and generalization across GCN operators.

  • 6 CONCLUSION: Residual and dense connections and dilated convolutions are adapted from CNNs to enable DeepGCN architectures.The approach is intended to alleviate training difficulty in deeper GCNs.
  • 6 CONCLUSION: Experiments on point-cloud segmentation and biological-graph node classification show benefits from deeper architectures, including state-of-the-art performance.The reported tasks include semantic segmentation and part segmentation of 3D point clouds.
  • 6 CONCLUSION: The approach generalizes across several GCN operators, while dilated graph convolutions enlarge receptive fields without loss of resolution.On point-cloud semantic segmentation, strong results are reported with relatively few nearest neighbors.
  • 6 CONCLUSION: For biological graph node classification, MRGCN is the best-performing operator and is proposed as a memory-efficient alternative to EdgeConv.ResMRGCN-112 and DenseMRGCN-56 both converged successfully on PPI.

7 FUTURE WORK

Future work proposes extending CNN-to-GCN transfers and refining graph construction, dilation schedules, and connection designs. The authors also identify task-specific representation and dataset-size boundaries.

  • 7 FUTURE WORK: Future studies could transfer deformable convolutions, feature pyramid architectures, and other CNN operators to GCNs.The paper also suggests combining residual and dense connections.
  • 7 FUTURE WORK: Graph-construction research could examine distance measures, layer-specific k values, and improved dilation-rate schedules for dilated k-NN graphs.These changes target how receptive fields are formed across layers.
  • 7 FUTURE WORK: For point-cloud semantic segmentation, processing data in 1m×1m columns is described as sub-optimal for graph representation, motivating improved sampling.The authors expect more suitable sampling to improve performance.
  • 7 FUTURE WORK: Existing node-classification datasets are described as relatively small, motivating further study on broader biological graph data.The passage identifies dataset scale as a boundary for the current task setting.

8 QUALITATIVE RESULTS FOR DEEPGCNS

The paper reports qualitative DeepGCN results on S3DIS and PartNet. The cited passage identifies the datasets and figures but does not state a qualitative comparison outcome.

  • 8 QUALITATIVE RESULTS FOR DEEPGCNS: Figures 10–14 present qualitative DeepGCN results on S3DIS, while Figure 15 presents qualitative results on PartNet.The passage identifies the figure ranges and datasets without describing the visual outcomes.

9 RUN-TIME OVERHEAD OF DYNAMIC K-NN

Dynamic k-NN adds substantial inference time relative to fixed k-NN, while less frequent updates can reduce computation. Additional experiments compare ResGCN-28 with DGCNN and examine architectural factors in segmentation.

  • 45.63ms inference time with fixed k-NN increased by 150.88ms when computing dynamic k-NN.
  • Updating dynamic k-NN every 3 layers is proposed as a way to reduce computation.
  • ResGCN-28 outperforms the DGCNN baseline across all classes using average per-class IoU across areas.
  • Qualitative S3DIS experiments examine stochastic dilation, nearest-neighbor count, network depth, network width, and the benefit of wider and deeper networks with half as many neighbors.
  • Qualitative PartNet results compare model performance with ground truth for part segmentation.
Loading 1910.06849v3…