PaperlayerView exact revision on arXiv

Source-linked AI summary

GaAN: Gated Attention Networks for Learning on Large and Spatiotemporal Graphs

Jiani Zhang, Xingjian Shi, Junyuan Xie, Hao Ma, Irwin King, Dit-Yan Yeung

arXiv:1803.07294v1cs.LGcs.SI

TL;DR

Graph learning needs architectures that can exploit structural information while avoiding equal treatment of attention heads. The paper proposes GaAN, which gates individual heads, and extends it into GGRU for traffic-speed forecasting. Experiments on three real-world datasets report state-of-the-art results for inductive node classification and traffic forecasting.

  • Problem

    Graph learning tasks require effective ways to express and exploit graph structure, while traditional multi-head attention treats all heads equally despite differing importance.

  • Method

    GaAN uses a lightweight convolutional subnetwork to compute soft gates controlling attention-head importance, and GGRU applies the aggregator to spatiotemporal forecasting.

  • Results

    GaAN beats previous state-of-the-art algorithms on inductive node classification and traffic speed forecasting across three real-world datasets.

  • Takeaways & Limitations

    The gated aggregator supports both graph node classification and graph-based recurrent modeling for traffic forecasting within one framework.

  • Takeaways & Limitations

    The reported comparison includes a model-training and sampling-strategy fairness caveat for one baseline.

Abstract

from arXiv · show

We propose a new network architecture, Gated Attention Networks (GaAN), for learning on graphs. Unlike the traditional multi-head attention mechanism, which equally consumes all attention heads, GaAN uses a convolutional sub-network to control each attention head's importance. We demonstrate the effectiveness of GaAN on the inductive node classification problem. Moreover, with GaAN as a building block, we construct the Graph Gated Recurrent Unit (GGRU) to address the traffic speed forecasting problem. Extensive experiments on three real-world datasets show that our GaAN framework achieves state-of-the-art results on both tasks.

1 INTRODUCTION

Graph learning tasks require effective ways to express and exploit structural information. GaAN addresses a limitation of equal treatment of attention heads by gating their importance and supports both node classification and traffic forecasting.

  • Graph-structured tasks include social-network classification, protein-interface prediction, and road-network traffic forecasting.
  • Graph convolution aggregates localized neighboring features and stacks aggregators to learn local and global graph representations end-to-end.
  • Equal treatment of attention heads can waste the contribution of heads that are inherently more important.
  • GaAN uses a lightweight convolutional subnetwork to compute a soft gate for each attention head and control its importance.The authors state that the added subnetwork has negligible computational overhead and is easy to train.
  • The framework applies GaAN to inductive node classification and extends it into GGRU for spatiotemporal traffic-speed forecasting.
  • The paper reports a new gated attention aggregator, a unified graph-recurrent framework, and state-of-the-art prediction performance on three real-world datasets.

2 NOTATIONS

The notation defines conventions for layers, activations, transformations, concatenation, and vector products used throughout the paper.

  • Vectors use bold lowercase letters, matrices use bold uppercase letters, and sets use calligraphic letters.
  • A fully connected layer is written as θ(x) = α(Wx + b), with θ = {W, b} denoting its parameters.
  • The notation distinguishes LeakyReLU h(·), sigmoid σ(·), activation-free linear transforms FCθ(x), and parameter sets with different subscripts.The LeakyReLU negative slope is 0.1.
  • The operator ⊕ denotes concatenation, while ◦ denotes the Hadamard product and ⟨·, ·⟩ denotes the vector dot product.

3 RELATED WORK

The related work covers neural attention, scalable graph convolution, and recurrent graph models for spatiotemporal forecasting. GaAN builds on multi-head attention while adding gates to control head outputs.

  • Neural attention mechanism: Neural attention computes weighted combinations of value vectors from query-key similarities, while multi-head attention concatenates outputs from multiple heads.
  • Neural attention mechanism: GaAN differs from standard multi-head attention by computing additional gates that control the importance of each head’s output.
  • Graph convolutional networks on large graph: Large-graph convolution is difficult because memory complexity scales with the total number of nodes, motivating GraphSAGE neighborhood sampling.
  • Graph convolution networks for spatiotemporal forecasting: GCRNN and DCRNN replace recurrent-network fully connected layers with graph convolution operators for spatiotemporal forecasting.
  • Graph convolution networks for spatiotemporal forecasting: DCRNN targets traffic-speed prediction from historical speeds and a road graph, additionally accounting for directed graph edges.

4 GATED ATTENTION NETWORKS

This section defines graph aggregators and contrasts pooling, pairwise-sum, multi-head attention, and gated attention designs. GaAN adds node-specific soft gates to attention heads, using a lightweight convolutional subnetwork to modulate their importance.

  • Generic formulation of graph aggregators: A graph aggregator maps a center node and its neighboring reference vectors to the center node’s output representation.Its learnable function is written as y_i = γ_Θ(x_i, {z_Ni}).
  • Multi-head attention aggregator: Multi-head attention projects center features into queries and neighbor features into keys and values, then concatenates head outputs for a final output layer.Each head uses dot-product attention weights over neighboring nodes.
  • Gated attention aggregator: GaAN assigns each attention head a soft gate between 0 and 1, allowing node-specific control over head importance.A convolutional network ψ_g uses center and neighbor features to generate gates, combining average and max pooling with negligible overhead when its intermediate dimension is small.
  • Graph pooling aggregators: Pooling aggregators directly aggregate neighboring features without modeling correlation between a neighbor and the center node.The center feature is concatenated or added to the pooled vector before the output function; pooling may be average, max, or sum pooling.
  • Graph pairwise sum aggregators: Pairwise-sum aggregators compute K weighted neighborhood sums, with each weight depending on the center-neighbor pair rather than other neighbors.The paper evaluates Pairwise + sigmoid and Pairwise + tanh as representative baselines.

5 INDUCTIVE NODE CLASSIFICATION

The inductive node-classification experiments evaluate graph aggregators on PPI and Reddit under sampled-neighborhood training, comparing GaAN with prior methods and controlled ablations. GaAN achieves the best reported F1 scores while gates, larger sampling sizes, and output dimensions affect performance.

  • Task and setup: Inductive node classification predicts labels for unseen testing nodes using stacked graph aggregators and sampled mini-batch neighborhoods.The sampling hierarchy expands recursively, then aggregates representations in reverse order to produce predictions.
  • Sampling: Neighborhood sampling reduces large-graph costs by limiting each node to min(|N_i|, S_ℓ) sampled neighbors and merging repeated nodes within each mini-batch.Variable-length GPU kernels avoid padding sampled neighborhoods to a common size.
  • Main results: GaAN achieves the best F1 score on both inductive node-classification benchmarks, outperforming GraphSAGE, GAT, FastGCN, and the evaluated aggregator baselines.The experiments compare prior state-of-the-art models, five aggregator-based models, and a fully connected network.
  • Ablation analysis: GaAN consistently outperforms multi-head attention with the same number of heads, while attention-based models outperform pooling and pairwise-sum models with fewer parameters.The comparison attributes the GaAN advantage to adding gates that control the importance of attention heads.
  • Ablation analysis: Larger sampling sizes steadily improve performance, and larger output dimensions improve PPI results while GaAN remains the strongest model.On Reddit, increasing the number of heads does not always improve results; on PPI, larger K improves prediction.
  • Ablation analysis: Gate visualizations show diverse gate combinations across nodes, with most nodes receiving different importance values across attention heads.This indicates that the gate-generation network learns node-dependent head weighting.

6 TRAFFIC SPEED FORECASTING

The paper formulates traffic speed forecasting as sequence prediction on a fixed spatiotemporal graph and builds Graph GRU encoder-decoder models from graph aggregators. Experiments on METR-LA compare these models across forecasting horizons, with GaAN-based GGRU outperforming the cited baselines.

  • 6.1 GRAPH GRU: Traffic speed forecasting is formulated as predicting future sensor-network speeds from previous observations on a fixed spatiotemporal graph.The graph represents road-network relationships between sensors, and the input and target are sequences.
  • 6.1 GRAPH GRU: GGRU replaces recurrent aggregation with graph-based update and reset gates, using node inputs, hidden states, and graph connectivity.The graph defines connections between nodes, while Ut and Rt control how Ht is calculated.
  • 6.1 GRAPH GRU: GGRU encoder-decoder models predict K future traffic-speed steps from J observed steps, using scheduled sampling in the decoder.Figure 4 illustrates two Graph GRU layers predicting a length-3 output from a length-2 input sequence.
  • 6.2 EXPERIMENTAL SETUP: The study compares six GGRU variations against fully-connected LSTM, GCRNN, and DCRNN using two-layer encoder and decoder architectures.Attention-based models use K = 4, da = 16, and dv = 16; GaAN additionally uses dm = 64 and max pooling in gate generation.
  • 6.3 MAIN RESULTS: GGRU models consistently outperform GCRNN on average forecasting scores, while GaAN-based GGRU exceeds DCRNN despite not using edge information.The comparison covers 15-minute, 30-minute, and 1-hour forecasting horizons.

7 CONCLUSION AND FUTURE WORK

The paper applies GaAN to inductive node classification and traffic speed forecasting through GGRU, reporting state-of-the-art performance on both tasks. Future work targets edge features, massive graphs, and natural language processing applications.

  • 7 CONCLUSION AND FUTURE WORK: GaAN is applied to inductive node classification and traffic speed forecasting, where it beats previous state-of-the-art algorithms.The conclusion identifies both tasks as challenging applications of the model.
  • 7 CONCLUSION AND FUTURE WORK: Future work will integrate edge features and extend GaAN to graphs with millions or billions of nodes.The authors also propose applying GaAN to natural language processing tasks such as machine translation.
  • 7 CONCLUSION AND FUTURE WORK: The authors identify natural language processing, including machine translation, as a future application area for GaAN.
Loading 1803.07294v1…