PaperlayerView exact revision on arXiv

Source-linked AI summary

Graph Adversarial Training: Dynamically Regularizing Based on Graph Structure

Fuli Feng, Xiangnan He, Jie Tang, Tat-Seng Chua

arXiv:1902.08226v2cs.LGcs.SIstat.ML

TL;DR

Graph neural networks can be especially vulnerable to feature perturbations because graph smoothing aggregates impacts across connected nodes, while standard adversarial training does not model these connections. The paper proposes GraphAT, which attacks and regularizes graph smoothness; experiments with GCN report a 4.51% result and performance exceeding GCN trained with VAT.

  • Problem

    Graph neural networks are vulnerable to intentional feature perturbations, and standard adversarial training does not account for perturbation impacts from connected examples.

  • Method

    GraphAT constructs perturbations that attack graph smoothness and trains models with a graph adversarial regularizer that reduces prediction divergence across connected examples.

  • Results

    4.51% is reported for GraphAT, and it also beats GCN trained with VAT.

  • Takeaways & Limitations

    GraphAT is presented as a generic dynamic regularization technique for training graph neural networks while encouraging smooth predictions over the graph.

  • Takeaways & Limitations

    The paper focuses on graph-based learning with only one graph and identifies handling multiple graphs as future work.

Abstract

from arXiv · show

Recent efforts show that neural networks are vulnerable to small but intentional perturbations on input features in visual classification tasks. Due to the additional consideration of connections between examples (\eg articles with citation link tend to be in the same class), graph neural networks could be more sensitive to the perturbations, since the perturbations from connected examples exacerbate the impact on a target example. Adversarial Training (AT), a dynamic regularization technique, can resist the worst-case perturbations on input features and is a promising choice to improve model robustness and generalization. However, existing AT methods focus on standard classification, being less effective when training models on graph since it does not model the impact from connected examples. In this work, we explore adversarial training on graph, aiming to improve the robustness and generalization of models learned on graph. We propose Graph Adversarial Training (GraphAT), which takes the impact from connected examples into account when learning to construct and resist perturbations. We give a general formulation of GraphAT, which can be seen as a dynamic regularization scheme based on the graph structure. To demonstrate the utility of GraphAT, we employ it on a state-of-the-art graph neural network model --- Graph Convolutional Network (GCN). We conduct experiments on two citation graphs (Citeseer and Cora) and a knowledge graph (NELL), verifying the effectiveness of GraphAT which outperforms normal training on GCN by 4.51% in node classification accuracy. Codes are available via: https://github.com/fulifeng/GraphAT.

1 INTRODUCTION

Graph neural networks are vulnerable to feature perturbations because graph smoothing aggregates disturbances from connected nodes. GraphAT addresses this by constructing and resisting perturbations with graph structure incorporated, improving GCN node-classification performance on three benchmarks.

  • Motivation: Graph neural networks can be more vulnerable than standard neural networks because smoothing aggregates perturbation effects from connected nodes.Perturbations on connected nodes can affect a target node’s propagated embedding and prediction.
  • Motivation: Adversarial Training proactively simulates perturbations during training to stabilize models and improve robustness.Existing AT methods treat examples independently and therefore miss impacts from connected examples.
  • Method: GraphAT constructs and resists perturbations by accounting for the graph structure and connected examples.It attacks graph smoothness during perturbation generation and reduces prediction divergence between perturbed targets and their connected examples.
  • Experiments: 4.51% accuracy improvement over normal GCN training was achieved across experiments on two citation graphs and one knowledge graph.The method was evaluated on node classification using GCN.
  • Contributions: GraphAT is presented as a dynamic regularization method that can enhance graph neural networks’ robustness against node-feature perturbations.The paper’s contributions include a general formulation and an efficient perturbation-construction algorithm.
  • Method: The graph adversarial regularizer dynamically encourages similar predictions for perturbed target examples and their connected examples.A linear approximation based on back-propagation efficiently constructs adversarial perturbations.

2 RELATED WORK

Related work establishes graph-based learning for relational data, adversarial training for robustness, and the gap addressed by incorporating graph structure into adversarial training.

  • Graph-based Learning: Graph-based learning predicts node labels using node features and graph structure, with graph embedding and convolution-based methods as major approaches.Convolution-based methods iteratively aggregate neighboring representations to learn node embeddings.
  • Graph-based Learning: Neural graph-based learning models are effective for node classification but remain vulnerable to intentionally designed perturbations, motivating robustness research.The paper identifies limited prior attention to strengthening robustness in these models.
  • Adversarial Training: Adversarial training dynamically generates perturbations that attack the training objective and then minimizes an additional regularization term over the resulting examples.Existing approaches smooth predictions around individual inputs in supervised and semi-supervised settings.
  • Adversarial Training: This work incorporates relations between examples into adversarial training so classifiers learn robust predictions that remain smooth over graph structure.The authors describe this as, to their knowledge, the first attempt to incorporate graph structure in adversarial training.
  • Adversarial Attacks on Graph Models: Earlier graph attack methods generate a separate adversarial graph for each node, requiring N graphs and causing unaffordable memory overhead.The paper therefore devises an efficient method for generating adversarial examples for graph adversarial training.

3 PRELIMINARIES

The preliminaries define attributed graphs and transductive node classification, then describe graph-based learning as jointly optimizing supervised prediction loss and graph smoothness.

  • Graph Notation: An attributed graph is represented as G = (A, D, X), combining adjacency, node-degree, and node-feature information.A is typically binary for the unweighted graphs studied, while X contains feature vectors for all N nodes.
  • Node Classification: Transductive node classification predicts labels for nodes whose features and associated edges are observed during training, with only a portion of labels given.The prediction function is written as ˆyi = f(xi, G|Θ), where Θ contains learnable model parameters.
  • Graph-based Learning Objective: Graph-based node-classification models jointly optimize supervised loss on labeled nodes and a graph smoothness constraint.The smoothness term encourages connected nodes to have similar predictions, balanced against classification loss by λ.

4 METHODOLOGY

GraphAT constructs perturbations that attack prediction smoothness across connected nodes and trains the model to resist them; GraphVAT additionally regularizes local prediction smoothness around clean examples.

  • Graph Adversarial Training: GraphAT is formulated as the original graph-learning objective plus a graph adversarial regularizer that aligns perturbed-node predictions with connected-node predictions.The divergence function d measures the difference between predictions, while rg_i perturbs the input feature of node i.
  • Graph Adversarial Training: GraphAT maximizes perturbations that break smoothness between a target node and its neighbors, then minimizes an objective with an additional regularizer over adversarial examples.This minimax process is intended to make the model robust against perturbations propagated through the graph.
  • Graph Adversarial Training: GraphAT regularization is dynamic because adversarial examples adapt to current parameters and predictions, and it also augments training with generated examples.This distinguishes it from static graph-based regularizations such as graph Laplacian regularization.
  • Perturbation Approximation: Graph adversarial perturbations are approximated linearly from the input gradient, which can be calculated efficiently with one backpropagation.The current model parameters are treated as a constant during this perturbation calculation.
  • Virtual Graph Adversarial Training: GraphVAT extends GraphAT with virtual adversarial regularization to smooth prediction distributions around each clean example, including unlabeled nodes.It jointly attacks local smoothness around individual examples and smoothness between connected examples.
  • Virtual Perturbation Approximation: For unlabeled nodes, virtual adversarial perturbations require a second-order approximation because the first-order gradient of prediction divergence is zero.Power iteration is used to approximate the required perturbation direction for efficiency.
  • GCN Implementation: GCN projects node representations, propagates them through a normalized adjacency matrix with self-connections, and applies a nonlinear activation at each layer.The representation of a node aggregates transformed representations from connected nodes, including itself.
  • GCN Implementation: GraphAT and GraphVAT train GCN by using cross-entropy as the standard objective term, alongside their respective adversarial regularizers.The GCN objective also includes an L2-norm term to prevent overfitting.

5.1 Experimental Settings

The experiments evaluate graph adversarial training for node classification on two citation graphs and one knowledge graph, using GCN and several established baselines. Dataset-specific settings, baseline definitions, and GraphAT hyperparameter selection are described for comparison.

  • Datasets: Experiments use Citeseer and Cora citation networks plus the NELL knowledge graph for node classification.Citation-network nodes represent documents with normalized bag-of-words features; NELL is represented as a bipartite graph of entity and relation nodes.
  • Datasets: Citation-network training uses features from all nodes but only 20 labels per class, with 500 validation and 1,000 test nodes.Each document has a normalized bag-of-words feature vector and a class label.
  • Baselines: The comparison includes LP, DeepWalk, SemiEmb, Planetoid, GCN, and GraphSGAN.The baselines span label propagation, graph embedding, Laplacian-regularized embeddings, graph convolution, and adversarial generation.
  • Parameter Settings: GraphAT trains GCN with graph adversarial training and uses six hyperparameters covering GCN architecture, perturbations, regularization, and neighbor sampling.The parameters include hidden-layer size, L2 weight, dropout ratio, perturbation scale, adversarial-regularizer weight, and sampled-neighbor count.
  • Parameter Settings: For fair comparison, standard GCN supplies the hidden-layer size and L2 weight, while GraphAT uses zero dropout and grid-searches perturbation scale, regularizer weight, and neighbor count.The searched ranges are [0.01, 0.05, 0.1, 0.5, 1], [0.01, 0.05, 0.1, 0.5, 1, 5], and, respectively.

5.2 Performance Comparison

GraphVAT improves GCN node-classification performance across three graph benchmarks and generally benefits nodes across connection densities, especially sparsely connected nodes. Ablations indicate that combining graph- and virtual-adversarial regularization performs best, while graph-structure perturbations vary by dataset.

  • Model Comparison: 6.35%, 1.47%, and 5.72% relative improvements over standard GCN on Citeseer, Cora, and NELL, respectively, are achieved by GraphVAT.GraphVAT also achieves comparable performance to GraphSGAN and outperforms other baselines in all cases.
  • Performance by Node Degree: Both GCN and GraphVAT perform best on nodes with degrees in, while nodes receive fewer propagated neighbor signals and [6, N] nodes are harder to classify.The explanation for the [6, N] group is presented as a postulate concerning general entities with heterogeneous relations.
  • Performance by Node Degree: GraphVAT generally outperforms GCN across node-degree groups, except for the [6, N] groups of Cora and NELL.The authors attribute the NELL exception partly to possible GCN underfitting, which additional regularization may worsen.
  • Performance by Node Degree: 5.45% average improvement is obtained by GraphVAT over GCN for nodes with degrees in.This group contains sparsely connected nodes, for which graph adversarial training is reported to be particularly effective.
  • Method Ablation: GraphVAT achieves the best performance in all ablation cases, outperforming both GCN-VAT and GraphAT.The result supports jointly considering virtual perturbations on individual nodes and graph perturbations involving neighbors.
  • Method Ablation: 1.38% and 4.04% improvements over GCN-VAT occur on Citeseer and Cora, whereas GraphAT is 1.58% worse on NELL.The authors speculate that NELL’s bipartite relation nodes lack bag-of-words descriptions and labels, making graph perturbations less effective.

5.3 Effect of Hyperparameters

GraphAT is evaluated across the graph adversarial regularizer weight β, perturbation scale ǫ, and sampled-neighbor count k. Its performance is generally smooth near optimal settings, with occasional degradation linked to early stopping and underfitting.

  • GraphAT’s performance is generally smooth near the optimal value of each hyperparameter, indicating limited sensitivity to hyperparameter choices.
  • Early stopping causes significantly worse performance at k = 3 and k = 5, although disabling early stopping would allow expected convergence.
  • GraphAT performs best with β around 0.1, which approximately balances supervised loss and graph adversarial regularization.
  • GraphAT performs well for ǫ in [1e-4, 1e-2], but performance decreases significantly as ǫ increases.
  • Tuning ǫ alone is investigated with β = 1 and k = 1 as fixed empirical values, reducing the hyperparameter combinations considered.

5.4 Impact of Graph Adversarial Training

Graph adversarial training affects both training behavior and robustness of GCN. It stabilizes predictions over graph connections and substantially reduces accuracy loss under graph adversarial perturbations.

  • GCN and GraphAT become stable after 100 epochs on Citeseer and Cora, indicating that graph adversarial training does not affect GCN convergence speed.
  • Figure 5 varies β, ǫ, and k on validation and testing across three datasets, holding the other hyperparameters at optimal values during each investigation.
  • Figure 6 plots GCN and GraphAT training curves on validation and testing for Citeseer and Cora.
  • 13.9% average accuracy loss for GCN falls to 2.9% for GraphAT under graph adversarial perturbations with ǫ = 0.01.
  • Graph adversarial training reduces divergence between connected-node predictions, producing smoother predictions over the graph structure.

6 CONCLUSION

The paper proposes GraphAT as a graph-structure-aware adversarial training method and demonstrates its effectiveness when training GCN for node classification. Experiments show a 4.51% average improvement, while future work broadens evaluation beyond the current setting.

  • GraphAT accounts for relations between examples by iteratively generating adversarial examples that attack graph smoothness and training on those examples.
  • 4.51% average improvement is achieved when training GCN with GraphAT on three benchmark datasets.
  • GraphAT beats GCN trained with VAT, indicating the necessity of considering graph structure in adversarial training.
  • Future work includes testing GraphAT on more graph neural network models, additional graph-learning tasks, multiple graphs, and specialized graph structures.
Loading 1902.08226v2…