PaperlayerView exact revision on arXiv

Source-linked AI summary

Understanding and Resolving Performance Degradation in Graph Convolutional Networks

Kuangqi Zhou, Yanfei Dong, Kaixin Wang, Wee Sun Lee, Bryan Hooi, Huan Xu, Jiashi Feng

arXiv:2006.07107v3cs.LGstat.ML

TL;DR

Deep GCNs tend to lose node-classification performance as they get deeper, while prior work has focused on propagation and given less attention to transformation. This paper experimentally studies both operations and proposes NodeNorm to control node-wise variance. NodeNorm enables deep GCNs to outperform shallow ones in some cases and match them on benchmark datasets.

  • Problem

    Deep GCNs tend to perform worse as depth increases, and prior studies have focused on PROPs while the role of TRANs remains less understood.

    Source: §1 · p. 1

  • Method

    The authors experimentally disentangle PROP and TRAN effects and propose NodeNorm, which scales each node’s hidden features using its standard deviation.

    Source: §1 · pp. 1–2

  • Results

    With NodeNorm, 64-layer GCNs outperform 2-layer ones in 3 cases and achieve comparable results on 6 benchmark datasets.

    Source: §1 · p. 2

  • Takeaways & Limitations

    NodeNorm is reported to resolve degradation for GAT and GraphSage and improve performance for GCNII and GEN.

    Source: §1 · p. 2

Abstract

from arXiv · show

A Graph Convolutional Network (GCN) stacks several layers and in each layer performs a PROPagation operation (PROP) and a TRANsformation operation (TRAN) for learning node representations over graph-structured data. Though powerful, GCNs tend to suffer performance drop when the model gets deep. Previous works focus on PROPs to study and mitigate this issue, but the role of TRANs is barely investigated. In this work, we study performance degradation of GCNs by experimentally examining how stacking only TRANs or PROPs works. We find that TRANs contribute significantly, or even more than PROPs, to declining performance, and moreover that they tend to amplify node-wise feature variance in GCNs, causing variance inflammation that we identify as a key factor for causing performance drop. Motivated by such observations, we propose a variance-controlling technique termed Node Normalization (NodeNorm), which scales each node's features using its own standard deviation. Experimental results validate the effectiveness of NodeNorm on addressing performance degradation of GCNs. Specifically, it enables deep GCNs to outperform shallow ones in cases where deep models are needed, and to achieve comparable results with shallow ones on 6 benchmark datasets. NodeNorm is a generic plug-in and can well generalize to other GNN architectures. Code is publicly available at https://github.com/miafei/NodeNorm.

1 INTRODUCTION

The introduction argues that TRANs have been under-studied in deep-GCN degradation and experimentally links them to variance inflation. It presents NodeNorm as a variance-control method and reports improved deep-GCN results and transfer to other GNNs.

1 INTRODUCTION

  • Ablations find that hidden-layer TRAN-only GCNs generally drop more in performance than PROP-only GCNs.This challenges the prior focus on PROP-related oversmoothing and gradient vanishing as explanations for degradation.Source: §1 · p. 1
  • TRANs tend to amplify node-wise feature variance as GCNs deepen, and high-variance nodes are harder to classify with deep than shallow models.The paper calls this increasing variance “variance inflammation”; it notes that oversmoothing can obscure the factor.Source: §1 · pp. 1–2
  • NodeNorm scales each node’s hidden features using its standard deviation, with a p-th-root factor that controls normalization strength.The introduction also reports that LayerNorm mitigates variance inflammation and improves deep-model performance through its variance-scaling step.Source: §1 · p. 2
  • NodeNorm lets 64-layer GCNs outperform 2-layer models in 3 cases and achieve comparable results on 6 benchmark datasets.It also outperforms two existing methods for addressing deep-GCN degradation in most cases.Source: §1 · p. 2
  • NodeNorm resolves degradation for GAT and GraphSage and improves performance for GCNII and GEN.The latter architectures are described as not suffering performance degradation.Source: §1 · p. 2

2 RELATED WORKS

The related work introduces common GNN operations and reviews research on depth-related performance degradation. It describes prior emphasis on PROP effects and identifies limited investigation of TRANs’ contribution.

2.1 Graph Neural Networks (GNNs)

  • GNN layers combine propagation and transformation; GCN, GraphSage, and GAT are presented as popular architectures used for node classification.GraphSage aggregates information from sampled neighbors, while GAT uses attention for propagation.Source: §2.1 · p. 2

2.2 Performance degradation problem of deep GNNs

  • Performance degradation with increasing depth has been observed in GCNs, GraphSage, and GAT.The passage notes that the problem was first observed in GCNs.Source: §2.2 · p. 2
  • Prior GCN work emphasizes PROP-related oversmoothing and gradient vanishing, with methods including DropEdge and PairNorm targeting oversmoothing.Oversmoothing is described as PROPs making node embeddings indistinguishable.Source: §2.2 · p. 2
  • Prior TRAN studies either do not investigate or justify TRANs’ role, or analyze them jointly with PROPs under assumptions considered inapplicable to practical settings.The cited theoretical analysis assumes a sufficiently dense input graph and depth approaching infinity.Source: §2.2 · pp. 2–3
  • JKNet can still degrade at 32 or 64 layers, while GCNII and GEN are reported to perform well when very deep; the paper describes its approach as complementary.The cited passage says applying the proposed variance-control technique can also improve these architectures.Source: §2.2 · p. 3

3 UNDERSTANDING AND RESOLVING PERFORMANCE DEGRADATION OF DEEP GCNS

The section examines why deeper GCNs degrade, finding that TRANs contribute substantially and amplify node-wise feature variance. It proposes NodeNorm to control that variance.

3.1 Preliminaries

  • Deep GCNs often perform worse as layers are added, while prior work mainly attributes this degradation to PROP-driven oversmoothing and gives less attention to TRANs.GCN layers combine propagation and transformation, which both influence learned node representations.Source: §3.1 · p. 3

3.2 Transformation operations contribute significantly to performance degradation

  • On Cora, 64-layer T-GCN accuracy is 0.4, versus 0.6 for vanilla GCN and 0.72 for P-GCN, showing stronger degradation from TRANs than PROPs.Figure 1 compares GCN, T-GCN, and P-GCN test accuracy across depths on Cora, Citeseer, and Pubmed.Source: §3.2 · pp. 3–4, Figure 1 · p. 4

3.3 Transformation operations cause variance inflammation

  • TRANs tend to amplify node-wise feature variance, which generally rises with depth in T-GCNs but shows no increasing trend in P-GCNs.The paper calls this increase in node-wise feature variance variance inflammation.Source: §3.3 · p. 4
  • In a 64-layer model, nodes with larger feature variance are harder to classify, and the model performs significantly worse than a 2-layer model.The section also reports that variance rises sharply from 2 to 64 layers.Source: §3.3 · p. 4

3.4 Techniques to mitigate variance inflammation

  • NodeNorm scales each node’s feature vector by the p-th root of its standard deviation, with p ≥ 1; smaller p controls variance more strictly.NodeNorm1 normalizes variance to 1 for all nodes.Source: §3.4 · pp. 4–5
  • LayerNorm also performs node-wise variance scaling, and the authors report that it can be trained to reduce variance inflammation in deep GCNs.LayerNorm additionally includes mean subtraction and feature-wise linear transformation.Source: §3.4 · p. 5

4 EXPERIMENTS

The experiments evaluate NodeNorm in settings where deeper GCNs are needed, on benchmark datasets, and across other GNN architectures. They also compare NodeNorm with LayerNorm and ablate LayerNorm’s operations.

4.1 Evaluating effects of proposed NodeNorm in cases requiring deep models

  • On cases requiring depth, NodeNorm-augmented GCNs achieve their best performance with deep models, exceeding vanilla GCNs’ best results, generally achieved by shallow models.The table reports classification accuracy and the depth associated with each result.Source: §4.1.2 · p. 6, Table 1 · p. 6
  • NodeNorm2 performs best in five of eight scenarios, while NodeNorm1 is among the top three in most scenarios; the optimal depth generally increases as p decreases.The authors suggest shallower models may need less strict variance control because they experience less variance inflammation.Source: §4.1.2 · pp. 6–7

4.2 Evaluating effects of proposed NodeNorm on benchmark datasets

  • NodeNorm1 makes 64-layer GCNs comparable to shallow models on six benchmark datasets, and stricter variance control improves results for very deep models.As p decreases from 3 to 1, performance increases for deep models such as 64-layer GCNs.Source: §4.2 · p. 7
  • NodeNorm1 enables 32- or 64-layer GCNs to match 2-layer GCNs on benchmark datasets, while DropEdge and PairNorm’s deep models still underperform their shallow counterparts.The comparison is reported for Cora, Citeseer, and Pubmed.Source: Table 2 · p. 7, §4.2 · p. 8

4.3 Applying variance-controlling to other GNN architectures

  • NodeNorm generalizes to GAT and GraphSage, mitigating their depth-related performance degradation and letting deep models compete with shallow counterparts.The comparison is shown for classification accuracy across model depths.Source: §4.3 · p. 8, Figure 4 · p. 8
  • NodeNorm1 improves GEN and GCNII performance; for GCNII, the reported improvement is 10% on Texas and Wisconsin.The experiments use Ogbn-proteins for GEN and Cornell, Texas, and Wisconsin for GCNII.Source: §4.3 · p. 8, Table 4 · p. 8

4.4 Investigating the effectiveness of LayerNorm

  • LayerNorm helps deeper GCNs outperform shallow ones in the three exemplar cases, but NodeNorm1, NodeNorm2, and NodeNorm3 outperform LayerNorm in 5, 7, and 5 of eight settings, respectively.The authors suggest softer variance control or LayerNorm’s learnable parameters as possible explanations.Source: §4.4 · p. 9
  • In 64-layer experiments, LayerNorm*, LayerNorm, and NodeNorm1 perform comparably, whereas mean-subtraction-only LayerNorm-MS performs significantly worse, even than baseline GCNs on four datasets.NodeNorm1 is equivalent to LayerNorm’s variance-scaling step; the authors identify variance-scaling, rather than mean-subtraction or linear transformation, as the effective operation.Source: §4.4 · p. 9, Figure 5 · p. 9

5 CONCLUSION

The conclusion attributes GCN performance degradation partly to TRANs and increasing node-wise feature variance, and presents NodeNorm as a way to control that variance.

5 CONCLUSION

  • NodeNorm enables 64-layer GCNs to compete with and even outperform 2-layer ones, and outperforms existing best methods for addressing GCN performance degradation.The paper also reports that NodeNorm generalizes to other GNN architectures.Source: §5 · p. 9
  • TRANs amplify node-wise feature variance as GCNs deepen, and deep GCNs perform significantly worse on nodes with relatively large feature variance.The paper hypothesizes and experimentally justifies that mitigating this variance inflammation addresses GCN performance degradation.Source: §5 · p. 9
  • TRANs contribute significantly to GCN performance decline, adding a degradation-related factor beyond the paper's focus on TRAN effects.The conclusion frames this finding as a new understanding of GCN performance degradation.Source: §5 · p. 9
  • NodeNorm normalizes each node's hidden features using that node's own standard deviation and can generalize to other GNN architectures.Source: §5 · p. 9

Related papers

Loading 2006.07107v3…