Source-linked AI summary
DeepGCNs: Can GCNs Go as Deep as CNNs?
Guohao Li, Matthias Müller, Ali Thabet, Bernard Ghanem
TL;DR
GCNs suit non-Euclidean data but are usually shallow because stacking layers causes vanishing gradients and over-smoothing. The paper adapts residual and dense connections and dilated convolutions from CNNs, enabling a 56-layer GCN that improves S3DIS state-of-the-art performance by 3.7%.
Problem
GCNs are generally limited to shallow architectures because deeper stacking causes vanishing gradients and over-smoothing.
Method
The paper adapts residual connections, dense connections, and dilated convolutions from CNNs to GCN architectures.
Results
3.7%: a 56-layer GCN improves state-of-the-art performance on the S3DIS point-cloud semantic-segmentation dataset.
Takeaways & Limitations
Deep GCNs can achieve high point-cloud semantic-segmentation performance, including with only 8 nearest neighbors in ResGCN-56.
Takeaways & Limitations
The authors could not investigate very deep architectures in detail because of computational constraints, and identify 1m × 1m column processing as sub-optimal for graph representation.
Abstract
from arXiv · showhide
Convolutional Neural Networks (CNNs) achieve impressive performance in a wide variety of fields. Their success benefited from a massive boost when very deep CNN models were able to be reliably trained. Despite their merits, CNNs fail to properly address problems with non-Euclidean data. To overcome this challenge, Graph Convolutional Networks (GCNs) build graphs to represent non-Euclidean data, borrow concepts from CNNs, and apply them in training. GCNs show promising results, but they are usually limited to very shallow models due to the vanishing gradient problem. As a result, most state-of-the-art GCN models are no deeper than 3 or 4 layers. In this work, we present new ways to successfully train very deep GCNs. We do this by borrowing concepts from CNNs, specifically residual/dense connections and dilated convolutions, and adapting them to GCN architectures. Extensive experiments show the positive effect of these deep GCN frameworks. Finally, we use these new concepts to build a very deep 56-layer GCN, and show how it significantly boosts performance (+3.7% mIoU over state-of-the-art) in the task of point cloud semantic segmentation. We believe that the community can greatly benefit from this work, as it opens up many opportunities for advancing GCN-based research.
1. Introduction
GCNs address non-Euclidean data but have generally remained shallow because deeper architectures suffer vanishing gradients and over-smoothing. This work adapts residual, dense, and dilated CNN concepts to train GCNs up to 56 layers and improve point-cloud segmentation performance.
- GCNs operate directly on non-Euclidean data, where CNNs have limited performance.
- Stacking GCN layers causes vanishing gradients and over-smoothing, with vertex features eventually converging to the same value.
- Most state-of-the-art GCNs are no deeper than 4 layers because of these training limitations.
- Residual and dense connections plus dilated convolutions are adapted from CNNs to improve deep-GCN training.
- 56 layers: the adapted framework enables successful GCN training to this depth and improves S3DIS point-cloud segmentation state of the art by 3.7%.
2. Related Work
Related work applies GCNs to diverse non-Euclidean domains, including social networks, vision, and point clouds, but prior deep-GCN studies generally report degradation or shallow optimal depths. DeepGCNs connect this limitation to early CNN challenges and adapt residual, dense, and dilated connections.
- GCNs model non-Euclidean relationships in social networks, scene graphs, human-joint action recognition, and point clouds.
- Point-cloud research has used 2D views, voxelization, and direct processing of unordered point sets to address their unstructured representation.
- Prior GCN studies report performance degradation beyond 3 or 10 layers, or use gates to support deeper models.
- Residual connections alleviate vanishing gradients, dense connections add cross-layer links, and dilated convolutions enlarge receptive fields without resolution loss.
- DeepGCNs adapt residual/dense connections and dilated convolutions to train very deep graph convolutional networks.
3. Methodology
The methodology adapts residual and dense connections plus dilated aggregation to GCNs, targeting reliable training of deeper architectures for point-cloud processing. The framework combines graph feature transformation, global feature fusion, and point-wise prediction while varying the GCN backbone design.
- Representation Learning on Graphs: GCNs aggregate neighboring vertex features and apply nonlinear updates to produce new vertex representations.The graph is represented through vertex features and edges connecting neighboring vertices.
- Motivation: The experiments address the difficulty of training deep GCNs, where stacking layers causes vanishing gradients, over-smoothing, and shallow state-of-the-art models.The proposed framework transfers ideas from deep CNNs to GCNs to enable deeper models that reliably converge and achieve stronger inference performance.
- Architecture: The point-cloud segmentation architecture uses GCN backbone, fusion, and MLP prediction blocks, while PlainGCN, ResGCN, and DenseGCN vary the backbone connections.ResGCN uses vertex-wise addition, whereas DenseGCN uses vertex-wise concatenation; the fusion and prediction blocks are kept consistent for comparison.
- Residual Learning for GCNs: Residual GCNs add transformed graph representations to the input graph through vertex-wise skip connections.This residual mapping is used to form the next-layer graph representation and is referred to as ResGCN.
- Dense Connections for GCNs: DenseGCNs concatenate the input graph with intermediate layer outputs, densely fusing representations from previous GCN layers.If each GCN output has D dimensions and the input has D0 dimensions, the next-layer feature dimension is D0 + D × (l + 1).
- Dilated Aggregation in GCNs: Dilated aggregation selects k neighbors from a k × d nearest-neighbor region by skipping every d neighbors, enlarging the receptive field without pooling.The resulting dilated edges are used by GCN aggregation and update functions; stochastic dilation is also used during training.
4. Experiments
The experiments adapt residual and dense graph connections plus dilated graph convolutions to deep GCNs, evaluating their stability and performance on large-scale point cloud segmentation. Results show that these components enable deeper models and improve segmentation over shallow and ablated baselines.
- Experimental setup: ResGCN and DenseGCN adapt residual or dense graph connections and dilated graph convolutions to address vanishing gradients and enlarge receptive fields.The framework is evaluated through extensive experiments and ablations on large-scale point cloud segmentation.
- Ablation studies: Residual graph connections are essential for training deeper networks: removing them from PlainGCN-28 causes a -12% mIoU degradation.The experiments associate residual connections with more stable gradients, analogous to findings in CNNs.
- Ablation studies: 2.85% improvement in mean IoU comes from dilated graph convolutions, which expand the network’s receptive field.Dilation helps deep networks especially when combined with residual graph connections; without them, performance can degrade.
- Ablation studies: Dense graph connections yield similar performance gains to residual connections, but naive implementations have prohibitive memory costs.The largest DenseGCN fits only 32 filters and 8 neighbors, versus 64 filters and 16 neighbors for ResGCN-28, making residual connections more practical.
- Comparison to state-of-the-art: 3.9% absolute mean-IoU improvement over DGCNN is achieved by ResGCN-28, which also outperforms all baselines in 9 of 13 classes.ResGCN-28 reaches 51.1% on board and improves state of the art by about 10% on sofa.
5. Conclusion and Future Work
The work shows that residual and dense connections plus dilated graph convolutions support deeper GCNs and strong point-cloud segmentation performance, while identifying computational and sampling limitations for future work.
- Conclusion: ResGCN-56 performs well on point-cloud semantic segmentation with only 8 nearest neighbors, compared with 16 for ResGCN-28.The authors also trained ResGCN-151 for 80 epochs with 3 nearest neighbors and obtained similar results to ResGCN-28 and ResGCN-56.
- Conclusion: Residual and dense graph connections substantially improve difficult classes such as board, bookcase, and sofa compared with PlainGCN-28.The qualitative comparison concerns otherwise identical 28-layer models.
- Conclusion: Almost 4% improvement over state-of-the-art and success on 9 out of 13 classes are reported for ResGCN-28 on S3DIS.The table reports overall point accuracy and mean IoU across all areas.
- Future Work: Computational constraints prevented detailed investigation of the deepest architectures.The authors leave this investigation for future work.
- Future Work: For point-cloud semantic segmentation, processing data in 1m × 1m columns is considered sub-optimal for graph representation.The authors suggest more suitable sampling could improve performance on this task.
A. Deep GCN Variants
The paper extends several GCN operations with residual or dense graph connections and dilated graph convolutions, using EdgeConv as the primary deep-GCN foundation and defining related variants.
- Deep GCN Variants: The deep-GCN variants are based on EdgeConv, GraphSAGE, GIN, and the proposed Max-Relative GCN operation.The variants are named ResEdgeConv, ResGraphSAGE, ResGIN, and ResMRGCN.
- Deep GCN Variants: Residual, dense, and dilated graph mechanisms are added to GCN operations to support training deeper architectures.EdgeConv is explicitly extended with all three concepts.
- Deep GCN Variants: ResGraphSAGE uses a max-pooling aggregator, with separate variants that omit or apply post-aggregation normalization.The paper states that the pooling aggregator outperforms the other GraphSAGE aggregators in its cited experiments.
- Deep GCN Variants: ResGIN learns ε at each layer to assign different weights to central-vertex and aggregated-neighborhood features.This distinguishes GIN from the other described GCN operations.
- Deep GCN Variants: MRGCN aggregates relative neighborhood features with a max operator, and its residual form uses hidden states from successive layers.The relative features are formed as h_u^l − h_v^l.
- Deep GCN Variants: Dilated k-NN defines each vertex neighborhood using a dilation-based graph construction, while vertex-wise max and sum operators aggregate features.The prose identifies N^(d)(v_l) as the neighborhood obtained from Dilated k-NN.
B. Results for Deep GCN Variants
Across 28-layer GCN variants, residual graph connections and dilated graph convolutions improve convergence relative to PlainGCN and generalize across multiple GCN operations.
- Results: 28-layer residual GCN variants with dilated graph convolutions converge better than PlainGCN on area 5 of S3DIS.The comparison includes ResEdgeConv, ResGraphSAGE, ResGIN, and ResMRGCN under shared training conditions.
- Results: Residual and dilated graph concepts generalize to different GCN types and enable training very deep GCNs.The conclusion is based on the reported behavior of the tested variants.
- Results: ResGIN-ε achieves high training accuracy but fails to generalize to the test set.The same phenomenon was observed in the original GIN paper, where setting ε to 0 gave the best performance.
C. Qualitative Results for the Ablation Study
The ablation study compares ResGCN-28 with variants that remove or alter its residual-connection and dilated-convolution components under matched training conditions.
- Qualitative Results for the Ablation Study: Figure 5 compares ResGCN-28 against several ablated variants on area 5 of S3DIS.The reference model has 28 layers, residual graph connections, and dilated graph convolutions.
- Qualitative Results for the Ablation Study: All ablation models are trained for 100 epochs on all areas except area 5 using the same hyper-parameters.Area 5 is used for evaluation.
D. Run-time Overhead of Dynamic k-NN
Dynamic k-NN adds substantial inference overhead compared with fixed k-NN, though updating neighborhoods less frequently can reduce computation.
- 45.63ms is the inference time for fixed k-NN in the 28-layer, k=16 reference model.
- 150.88ms is added to inference time when dynamic k-NN is computed.
- Computing dynamic k-NN every 3 layers is suggested as a way to reduce computation.
E. Comparison with DGCNN over All Classes
ResGCN-28 consistently outperforms the DGCNN baseline across all classes on average per-class IoU results. Qualitative results also examine the effects of dilation, neighborhood size, depth, width, and their combination.
- ResGCN-28 outperforms DGCNN across all classes on average per-class IoU results.The comparison uses a 28-layer reference network with residual graph connections and dilated graph convolutions.
- The DGCNN comparison reproduces results because the original paper did not provide results over all classes.
- Qualitative experiments examine the importance of stochastic dilated convolutions, nearest-neighbor count, network depth, and network width.
- A qualitative experiment shows the benefit of a wider and deeper network even with half the number of nearest neighbors.