Source-linked AI summary
Attention-based Graph Neural Network for Semi-supervised Learning
Kiran K. Thekumparampil, Chong Wang, Sewoong Oh, Li-Jia Li
TL;DR
Graph-based semi-supervised learning must classify nodes with few labels while exploiting graph structure and node features. The paper removes intermediate perceptrons, replaces propagation with adaptive neighbor attention, and reports consistent gains on benchmark citation networks, while more complex attention was unstable.
Problem
Graph-based semi-supervised learning seeks accurate node classification from a small labeled subset, unlabeled data, node features, and graph structure.
Method
AGNN removes intermediate fully connected layers and uses graph-respecting attention to weight neighborhood contributions dynamically.
Results
The method consistently outperforms competing methods on standard benchmark citation network datasets.
Takeaways & Limitations
Learned attention weights provide insights into how neighbors influence one another, while reduced complexity supports deeper models in the semi-supervised setting.
Takeaways & Limitations
More complex attention models made training unstable and did not yield higher accuracy, especially when labeled nodes were scarce.
Abstract
from arXiv · showhide
Recently popularized graph neural networks achieve the state-of-the-art accuracy on a number of standard benchmark datasets for graph-based semi-supervised learning, improving significantly over existing approaches. These architectures alternate between a propagation layer that aggregates the hidden states of the local neighborhood and a fully-connected layer. Perhaps surprisingly, we show that a linear model, that removes all the intermediate fully-connected layers, is still able to achieve a performance comparable to the state-of-the-art models. This significantly reduces the number of parameters, which is critical for semi-supervised learning where number of labeled examples are small. This in turn allows a room for designing more innovative propagation layers. Based on this insight, we propose a novel graph neural network that removes all the intermediate fully-connected layers, and replaces the propagation layers with attention mechanisms that respect the structure of the graph. The attention mechanism allows us to learn a dynamic and adaptive local summary of the neighborhood to achieve more accurate predictions. In a number of experiments on benchmark citation networks datasets, we demonstrate that our approach outperforms competing methods. By examining the attention weights among neighbors, we show that our model provides some interesting insights on how neighbors influence each other.
1 Introduction
Graph-based semi-supervised learning uses a small labeled subset, all node features, and an explicitly provided graph containing relations beyond those features. The paper uses this setting to develop a neural architecture that improves node classification.
- Graphs can provide pairwise relations among labeled and unlabeled data points when labeled data are limited.
- Citation networks represent papers as feature-bearing nodes and citations as edges that capture relations not necessarily inferred from bag-of-words features.
- Semi-supervised graph learning classifies nodes using a small subset of labeled nodes, all node features, and graph structure.
- Graph neural networks recently improved results on standard graph-based semi-supervised learning benchmarks.
- The paper introduces an attention-based graph neural network that adaptively weights neighborhood information and reduces model complexity.
2 Related Work
Earlier graph-based semi-supervised methods include graph regularization, latent-variable approaches, node embeddings, and graph neural networks. The related work frames GCN as a powerful benchmark architecture for the paper’s analysis.
- Graph Laplacian methods encourage nearby nodes to have similar labels, while expectation-maximization methods treat unlabeled labels as latent variables.
- Label Propagation enforces agreement with labeled instances using a weighted graph Laplacian and admits a closed-form solution.
- Parametric approaches use deep neural networks, including SemiEmb and Planetoid, which replaces graph regularization with a skip-gram-based loss.
- Graph neural networks extend neural networks to graph-structured data through repeated message aggregation and shared node-wise parameters.
- GCN is presented as a simple architecture achieving state-of-the-art accuracy on the benchmark citation networks considered in this paper.
3 Dissection of Graph Neural Network
Typical GNNs alternate graph propagation with shared perceptrons, while GCN uses normalized propagation and nonlinear transformations. The paper’s analysis finds that linear propagation can retain comparable performance, motivating attention-based propagation.
- A typical GNN alternates propagation layers, which aggregate neighboring hidden states, with shared single-layer perceptrons applied independently at each node.
- Propagation uses a matrix P to perform graph-structured local averaging while respecting the adjacency pattern.
- GCN stacks two normalized propagation and perceptron layers, using ReLU and softmax in its nonlinear architecture.
- Graph Linear Network removes intermediate nonlinear activations, leaving linear degree-weighted propagation followed by a multinomial logistic-regression classifier.
- GLN achieves accuracy comparable to, and sometimes better than, the best GNN, suggesting propagation contributes more than the perceptron layer on citation networks.
4 Attention-based Graph Neural Network (AGNN).
AGNN removes intermediate perceptron layers and concentrates model capacity in graph propagation, using attention to weight neighbors dynamically according to their hidden states. Its single-parameter-per-layer design targets stable training with few labels while preserving graph structure.
- Architecture: AGNN removes all intermediate perceptron layers and focuses on improving propagation layers.The design retains an input word-embedding layer and output layer while simplifying intermediate processing.
- Architecture: The attention mechanism learns which neighboring nodes are more relevant and weights their contributions accordingly.Attention is computed over each node’s graph neighborhood rather than over all entities.
- Architecture: AGNN uses one scalar parameter β(t) at each intermediate layer, supporting training when labeled examples per class are small.The paper notes that the standard benchmark setting has only 20 labeled samples per class.
- Propagation: The propagation matrix is zero for absent edges, so attention respects the graph’s existing neighborhood structure.The matrix P(t) depends on the input hidden states H(t).
- Propagation: Propagation is dynamic because it changes across layers with β(t) and hidden states, and adaptive because it learns to weight relevant neighbors more highly.Self-loops preserve each node’s own features and hidden states during propagation.
- Attention mechanism: Attention relevance is measured by cosine similarity between neighboring hidden states, and the softmax makes each propagation row sum to one.The paper reports that attention selects same-class neighbors as more relevant and that AGNN achieves the best performance in Table 2.
5 Experiments on Benchmark Citation Networks
Across fixed and random benchmark splits, AGNN achieves the strongest reported accuracy, while attention analysis shows how its propagation weights relate to node classes and neighborhoods.
- Fixed data splits: AGNN achieves the best accuracy on all three fixed-split citation-network datasets, with gaps larger than the standard error.The fixed split uses 20 labeled nodes per class, 500 validation nodes, and 1,000 test nodes.
- Random splits: AGNN consistently achieves the best performance across random splits, although accuracy decreases for nearly all methods when class-balanced labeling is not enforced.Randomly selected labels may be unevenly distributed across topics.
- Larger training set: AGNN consistently outperforms the current state-of-the-art architecture when the Cora training set contains more labeled nodes.The experiment uses 3-fold and 10-fold cross-validation with uniformly random partitions.
- Inter-class relevance score: Relevance scores are dominated by same-topic relations, while selected off-diagonal relations identify cross-topic citation influences in CiteSeer and Cora.Examples include HCI→Agents in CiteSeer and CB→PM in Cora.
- Inter-class relevance score: AGNN’s attention assigns greater weight to same-class edges, as shown by comparing the top 100 and bottom 100 edges ranked by relevance.Table 6 reports the fraction of edges connecting nodes from the same class.
- Neighborhood attention: In illustrative test cases misclassified by GCN, AGNN assigns stronger attention to nodes sharing the target’s hidden class within the 2-hop neighborhood.The examples include targets from PubMed and CiteSeer, none of whose colors were revealed during training.
6 Conclusions
The paper presents AGNN for semi-supervised graph classification and reports consistent gains over competing methods on citation-network benchmarks. Learned attention offers insight into neighbor influence, while reduced complexity supports deeper models and avoids instability seen with more complex alternatives.
- AGNN consistently outperforms competing methods on standard benchmark citation-network datasets.
- Learned attention provides insights into how neighbors influence each other.
- More complex attention models were unstable during training and did not improve accuracy.
- Removing nonlinear layers reduces model complexity and enables training deeper four-layer models than the shallower two-layer GCN.
A Additional experiments on interpretability of attention
Additional experiments visualize attention in citation networks and examine how the model weights neighboring topics. The figures provide qualitative examples of attention strength around nodes that GCN misclassifies but AGNN correctly classifies.
- PubMed attention analysis reports relevance scores across its three topic classes and includes examples of nodes corrected by AGNN.
- Figure 3 displays average attention from a column class to a row class.
B Performance of GCN on other dataset splits.
The appendix reports GCN performance under random splits and larger training-set splits, extending the evaluation beyond the fixed benchmark split. It also provides accuracy tables and standard errors across repeated runs.
- GCN performance is reported for random splits and larger-training-set dataset splits from Section 5.1.
- Figure 5 gives Cora examples where AGNN correctly classifies test nodes mistaken by GCN, with attention strength encoded by node size.
- Table 7 reports classification accuracy with random data splits.
- Random-split results report average testing accuracy and standard error over 20 runs, while larger-training-set results use 10 runs.
C Experiment and Architect Details
The appendix documents training choices, attention examples, larger-label-set accuracy, and AGNN hyper-parameters. These materials specify optimization and regularization settings alongside illustrative PubMed attention patterns.
- Training uses Adam with full batches, weight decay on learnable weights, and dropout in the first and last layers.
- Figure 6 shows PubMed neighborhoods where node sizes represent attention strength and none of the nodes are labeled.
- Table 8 reports classification accuracy with larger sets of labelled nodes.
- Table 9 lists hyper-parameters for the AGNN model.