Batch Virtual Adversarial Training for Graph Convolutional Networks

1. Introduction

• Unlike neighborhood aggregation, which smooths representations between connected nodes, these graph methods do not account for classifier-output smoothness under local perturbations.Prior work reports that smoothing classifier outputs against local perturbations can improve generalization, especially in semi-supervised learning.Source: §1 · p. 1
• In graph-based VAT, overlapping receptive fields can make perturbations accumulate, so a node’s resulting perturbation may not be its worst-case virtual adversarial perturbation.The paper links this overlap to reduced effectiveness in smoothing model outputs and pushing decision boundaries away from real data instances.Source: §1 · p. 1
• S-BVAT generates perturbations for a subset of separated nodes, while O-BVAT generates perturbations for all nodes.For S-BVAT, selected nodes have non-overlapping receptive fields; O-BVAT optimizes all-node perturbations together.Source: §1 · pp. 1–2, Figure 1 · p. 2
• BVAT establishes state-of-the-art results on Cora, Citeseer, Pubmed, and Nell with tolerable additional computation complexity.The evaluation covers three citation datasets and one knowledge graph dataset.Source: §1 · p. 2

2. Related Work

• Graph representation learning for semi-supervised and unsupervised learning has developed along spectral and non-spectral directions.The passage lists examples including graph convolutional and neighborhood-aggregation methods.Source: §2 · p. 2
• VAT has succeeded in varied domains, but its straightforward graph-model application is less effective because node predictions are interrelated.BVAT is proposed to address this limitation.Source: §2 · p. 2
• Graph adversarial attacks consider connectivity to generate perturbations, whereas BVAT targets semi-supervised node classification rather than attacks.Graph partition networks also split graphs into disjoint subgraphs, but to improve information propagation efficiency rather than output-distribution smoothness.Source: §2 · p. 2

3.1. Virtual Adversarial Training for Graph Convolutional Networks

• VAT trains a classifier to remain smooth against local worst-case virtual adversarial perturbations, using local distributional smoothness (LDS) as a regularizer.The prediction distributions are compared using KL divergence.Source: §3.1 · p. 2
• A direct GCN extension averages LDS across nodes, but overlapping receptive fields accumulate perturbations and can violate the perturbation norm bound.The overall loss also includes labeled-node cross-entropy and conditional entropy, whose coefficient encourages one-hot predictions.Source: §3.1 · pp. 2–3

3.2. Batch Virtual Adversarial Training

• BVAT uses graph connectivity to reduce perturbation interaction, either by sampling distant nodes or optimizing perturbations across all nodes.Both approaches fit batch gradient descent and add tolerable computation complexity.Source: §3.2 · p. 3
• S-BVAT samples B nodes with pairwise shortest-path distance at least 2K, keeping their K-layer receptive fields disjoint.It averages LDS over the sampled nodes as an approximate all-node regularizer; one-step power iteration is reported sufficient for high performance.Source: §3.2 · p. 3
• O-BVAT optimizes a perturbation matrix for all nodes to maximize average LDS while penalizing its Frobenius norm.The norm penalty keeps the perturbation small, and Adam optimizes the matrix for T iterations.Source: §3.2 · p. 3

4.1. Effectiveness of BVAT

• S-BVAT and O-BVAT achieve higher Rvadv than VAT, while O-BVAT’s Rvadv converges at about 10 steps.For S-BVAT and VAT, Rvadv jumps from T = 0 to T = 1, with little further increase at larger T.Source: §4.1 · p. 4, Figure 2 · p. 4
• Models trained with S-BVAT or O-BVAT have lower Rvadv than vanilla GCNs, indicating greater robustness to adversarial perturbations and smoother input-space behavior.Source: §4.1 · p. 4, Figure 2 · p. 4

4. Experiments

• The experiments compare VAT and BVAT variants with state-of-the-art methods on four benchmarks, reporting averages over 10 runs with different random seeds.Source: §4.2 · p. 4

4.2. Semi-supervised Node Classification

• All BVAT variants outperform vanilla GCNs and random-perturbation GCNs across four datasets; O-BVAT establishes state-of-the-art results.The evaluation covers Cora, Citeseer, Pubmed, and Nell.Source: §4.2 · p. 4, Table 1 · p. 4

• BVAT algorithms smooth the output distribution of graph-based classifiers.Source: §5 · p. 4
• The conclusion identifies sample-based and optimization-based virtual adversarial training as BVAT’s two algorithms.Source: §5 · p. 5
• Experiments demonstrate BVAT’s effectiveness in semi-supervised node classification, and the paper reports results exceeding current state-of-the-art methods by a large margin.Source: §5 · p. 5

• S-BVAT repeatedly selects random nodes and removes their neighborhoods through 2K hops from the candidate pool, yielding a subset of separable nodes.Source: A. Algorithms for BVAT · p. 7
• For each selected node, S-BVAT initializes and normalizes a perturbation, then calculates it using the gradient of LDS with respect to the perturbation.Source: A. Algorithms for BVAT · p. 7
• O-BVAT initializes perturbations for all nodes with Gaussian noise, performs T Adam gradient-ascent iterations, and returns the model-weight gradient.Source: A. Algorithms for BVAT · p. 7

• The evaluation uses Cora, Citeseer, Pubmed, and Nell with the same train, validation, and test splits as prior work; dataset statistics are summarized in Table 2.Source: B. Experimental Setup · p. 7, Table 2 · p. 7
• Nell’s 65,755 × 61,278 sparse feature matrix exceeds the stated GPU memory when densified, so BVAT constructs sparse perturbations for it.Source: B. Experimental Setup · p. 7
• The setup matches GCN architecture and training settings; S-BVAT uses B = 100 and T = 1, while O-BVAT uses Adam at learning rate 0.001 for T = 10.Perturbation sizes and γ vary by dataset, and α and β are tuned because dataset label rates, feature sizes, and edge counts differ.Source: B. Experimental Setup · p. 7

C. Ablation study of ϵ and α

• Pubmed and Nell use ϵ = 0.003 because their input features have smaller norms.This contrasts with ϵ = 0.03 for Cora and Citeseer.Source: C. Ablation study of ϵ and α · p. 7
• S-BVAT validation performance on Cora and Citeseer is insensitive to ϵ from 0.01 to 0.1.The study selects ϵ = 0.03 for both datasets.Source: C. Ablation study of ϵ and α · p. 7, Figure 3 · p. 8
• S-BVAT and O-BVAT maintain high performance on Cora as α varies from 0 to 1.5.The authors conclude virtual adversarial perturbations play a crucial role in smoothing the output distribution and improving performance, while conditional entropy is extra regularization for semi-supervised classification.Source: C. Ablation study of ϵ and α · p. 7, Figure 3 · p. 8

D. Computation Complexity Analysis

• 0.0355154, 0.03946125, and 0.1064431 seconds per epoch are reported for GCN, GCN with S-BVAT, and GCN with O-BVAT, respectively.Measurements were averaged on Cora using a GTX 1080Ti.Source: D. Computation Complexity Analysis · p. 8
• S-BVAT adds two forward propagations and one back propagation, while O-BVAT adds T + 1 forward and T back propagations, with T = 10.The added gradients are for the LDS loss with respect to input features, without updating classifier parameters.Source: D. Computation Complexity Analysis · p. 8
• The authors consider BVAT's additional computation cost acceptable because its classification performance is noticeably better than GCN's.O-BVAT takes less than 3× the time of GCN in the reported comparison.Source: D. Computation Complexity Analysis · p. 8