PaperlayerView exact revision on arXiv

Source-linked AI summary

Graph Attention Multi-Layer Perceptron

Wentao Zhang, Ziqi Yin, Zeang Sheng, Yang Li, Wen Ouyang, Xiaosen Li, Yangyu Tao, Zhi Yang, Bin Cui

arXiv:2206.04355v1cs.LGcs.AI

TL;DR

Large, sparse graphs make GNNs difficult to scale, and fixed-hop neighborhoods can over-smooth nodes whose receptive fields expand at different rates. GAMLP addresses this with node-adaptive attention over multi-scale feature and label propagation. Across 14 datasets, it achieves state-of-the-art performance with high scalability and efficiency, including leading results on large OGB graphs and substantial industrial speedups.

  • Problem

    Large sparse graphs are costly for GNNs, while fixed-hop propagation can mismatch nodes’ receptive-field needs and cause over-smoothing.

  • Method

    GAMLP uses recursive and Jumping Knowledge attention to learn node-adaptive correlations across multi-scale propagated features and labels.

  • Results

    GAMLP achieves state-of-the-art performance on 14 graph datasets while maintaining high scalability and efficiency.

  • Takeaways & Limitations

    GAMLP supports large-scale graph applications through node-adaptive receptive-field modeling and has been deployed in Tencent applications.

Abstract

from arXiv · show

Graph neural networks (GNNs) have achieved great success in many graph-based applications. However, the enormous size and high sparsity level of graphs hinder their applications under industrial scenarios. Although some scalable GNNs are proposed for large-scale graphs, they adopt a fixed $K$-hop neighborhood for each node, thus facing the over-smoothing issue when adopting large propagation depths for nodes within sparse regions. To tackle the above issue, we propose a new GNN architecture -- Graph Attention Multi-Layer Perceptron (GAMLP), which can capture the underlying correlations between different scales of graph knowledge. We have deployed GAMLP in Tencent with the Angel platform, and we further evaluate GAMLP on both real-world datasets and large-scale industrial datasets. Extensive experiments on these 14 graph datasets demonstrate that GAMLP achieves state-of-the-art performance while enjoying high scalability and efficiency. Specifically, it outperforms GAT by 1.3\% regarding predictive accuracy on our large-scale Tencent Video dataset while achieving up to $50\times$ training speedup. Besides, it ranks top-1 on both the leaderboards of the largest homogeneous and heterogeneous graph (i.e., ogbn-papers100M and ogbn-mag) of Open Graph Benchmark.

1 INTRODUCTION

GNN receptive fields expand rapidly, making large sparse graphs costly to process, while fixed-hop propagation and shared scale weights can mismatch nodes’ needs. GAMLP addresses this by learning node-adaptive correlations across multi-scale graph information.

  • Motivation: GNN receptive fields grow exponentially with layer depth, increasing computation, memory, and distributed feature-pulling costs.The receptive field contains nodes within a node’s K-hop neighborhood.
  • Motivation: Sampling reduces recursive neighborhood expansion but remains limited by communication costs and sampling quality.
  • Motivation: Fixed propagation depths and shared scale weights can be unsuitable because nodes have different optimal propagation steps and receptive-field expansion speeds.Figure 1 contrasts node-level accuracy across propagation steps and receptive-field growth in dense versus sparse regions.
  • GAMLP: GAMLP learns the importance and correlations of multi-scale knowledge in a node-adaptive manner.
  • GAMLP: GAMLP uses recursive and Jumping Knowledge attention to capture correlations between propagated information at different depths.The mechanisms operate at node granularity and can use both node features and labels from neighborhoods of different sizes.
  • Results: GAMLP achieves state-of-the-art performance on 14 graph datasets while maintaining scalability and efficiency.It exceeds the second-best method by 1.03% on ogbn-products and 1.32% on ogbn-papers100M.

2 PRELIMINARIES

Scalable GNNs simplify propagation or use sampling to reduce large-graph costs, but fixed propagation remains insensitive to node-specific receptive-field growth. The paper’s setup also incorporates graph structure, node features, and optionally propagated training labels.

  • 2.1 Problem Formulation: The problem is to predict unlabeled node labels using graph structure, node features, and supervision from labeled nodes.The graph has n nodes, m edges, c classes, and feature matrix X.
  • 2.2 Scalable GNNs: Sampling-based GNNs reduce computation by selecting neighbors, layers, or clusters, but use sampling procedures to construct mini-batch computation.
  • 2.2 Scalable GNNs: SGC precomputes K-step propagated features, enabling scalable computation but imposing the same propagation steps and fixed receptive field on every node.
  • 2.2 Scalable GNNs: Layer-wise methods combine features from different propagation depths, but fixed graph-wise propagation can over-smooth or under-smooth nodes as receptive fields expand inconsistently.
  • 2.2 Scalable GNNs: GAMLP assigns each node a personalized combination of propagated features from different steps.This node-wise perspective distinguishes GAMLP from methods using a shared combination across nodes.
  • 2.3 Label Utilization in GNNs: Label propagation methods treat partially observed training labels as graph inputs, while other approaches propagate errors or concatenate labels with node features.

3 GRAPH ATTENTION MULTI-LAYER PERCEPTRON

GAMLP decomposes graph learning into propagation, node-adaptive receptive-field attention, and MLP training. It combines multi-scale features and labels with node-specific weights to improve scalability and accommodate different neighborhood requirements.

  • Architecture: GAMLP separates feature and label propagation, receptive-field attention-based combination, and MLP training, with propagation preprocessed once.This decomposition avoids repeated intermediate-neighbor communication during MLP training.
  • Node-wise Propagation: Node-wise feature propagation produces representations from the node itself and progressively larger k-hop neighborhoods, then combines them with learned node-specific weights.Each weight measures the importance of a propagation depth for a particular node.
  • Node-wise Propagation: Node-wise label propagation generates multi-step label representations and applies a last residual connection that smooths each step toward the final propagated label.The coefficient α_l controls the proportion of the final propagated label in the l-step representation.
  • Node-adaptive Attention: Recursive and JK attention learn node-adaptive weights for propagated information, capturing correlations across propagation depths and receptive fields.JK attention uses a concatenated multi-scale branch to guide the attention-based combination branch.
  • Model Training: GAMLP combines feature and label representations through separate MLPs, with β controlling the importance of the combined label before producing the final embedding.Lower β is recommended when labels are noisy or available at a low rate.

4 EXPERIMENTS

Experiments on 14 graph datasets evaluate GAMLP across transductive and inductive settings, graph types, sparsity levels, and ablations. GAMLP generally outperforms the evaluated baselines, with node-adaptive propagation supporting performance on sparse graphs.

  • Experimental Setup: GAMLP is evaluated on 14 datasets spanning transductive and inductive settings, including homogeneous and heterogeneous graphs.The transductive suite includes citation, user-item, co-author, OGB, and Tencent Video graphs; inductive evaluation uses PPI, Flickr, and Reddit.
  • End-to-end Comparison: GAMLP variants outperform baseline methods on almost all transductive datasets, including 1.6% on Citeseer and 1.0% on Amazon Computers.GAMLP leads by 1.03% on ogbn-products and 1.32% on ogbn-papers100M.
  • End-to-end Comparison: GAMLP consistently outperforms baselines under inductive evaluation, with GAMLP(JK) exceeding GraphSAINT by more than 3.0% on Flickr.The results indicate strong performance for predicting properties of unseen nodes.
  • Ablation Study: Label utilization raises GAMLP(R) accuracy from 81.43% to 83.59%, while raw-label residuals and uniform fusion reduce performance.The ablation supports the proposed last residual connections rather than direct raw-label adoption or intuitive uniform fusion.
  • Performance on Sparse Graphs: GAMLP outperforms baselines in most tested edge- and label-sparsity settings by using node-wise propagation to capture long-range dependencies.Edges are randomly removed and training nodes per class vary from 1 to 20.
  • Heterogeneous Graphs: For heterogeneous graphs, GAMLP processes subgraphs formed from edge-type combinations and aggregates propagated features and labels before final prediction.The adaptation follows the NARS design and uses 1-d convolution for cross-subgraph aggregation.

5 DEPLOYMENT IN TENCENT

GAMLP is deployed in Tencent through distributed preprocessing and model training. The deployment supports large-scale applications, including WeSee short-video classification, where efficiency and predictive accuracy are compared against other methods.

  • Implementation: GAMLP training separates distributed graph feature preprocessing from distributed model training.Propagated features and labels are pre-computed, followed by parameter-server training of the model.
  • Implementation: GAMLP is implemented with Angel and distributed SGD, with Spark executors retrieving model weights, propagated features, and labels from a parameter server.The implementation is designed for large-scale graph training.
  • Applications: Tencent uses GAMLP in applications including WeSee short-video classification and WeChat payment prediction.The WeSee task classifies short videos into 253 predefined classes using a user-video graph.
  • Results: The WeSee evaluation compares relative training time and predictive accuracy for short-video classification.SGC training time is used as the reference point in the comparison.

6 CONCLUSION

GAMLP is a scalable, efficient, and deep graph model that uses receptive field attention through recursive and JK attention. It was deployed in Tencent and achieved strong predictive performance across 14 graph datasets and a large-scale short-video dataset.

  • GAMLP uses recursive attention and JK attention to learn representations over receptive fields with different sizes in a node-adaptive manner.
  • GAMLP was deployed in Tencent and served many real-world applications.
  • GAMLP achieved high predictive performance on 14 graph datasets and exceeded compared baselines on a large-scale short-video dataset.Its training time on the short-video dataset was comparable to SGC.

B.1 Deep propagation is possible

GAMLP remains accurate at very large propagation depths, unlike SGC and JK-Net, whose accuracy declines rapidly as depth increases. Its attention weights also adapt across propagation steps and node-degree ranges.

  • Deep propagation is possible: Over 50 propagation steps, GAMLP maintains high predictive accuracy, exceeding 80.0% at depth 100 on PubMed.
  • Deep propagation is possible: SGC and JK-Net accuracy drops rapidly as propagation depth increases from 10 to 100.
  • Deep propagation is possible: GAMLP(JK) combines multi-scale propagated features using attention weights that vary with propagation steps and input-node degree ranges.The analysis uses a maximum propagation step of 6 and groups nodes into degree ranges 1–4, 5–8, and 9–12.

B.3 Choices for 𝛼𝑙in Last Residual Connection

GAMLP uses a cosine weighting scheme in its last residual connection, motivated by over-fitting at large label-propagation steps. On ogbn-products, this scheme outperforms linear-decreasing and fixed-weight alternatives.

  • Choices for α_l in Last Residual Connection: GAMLP adopts α_l = cos(πl/2L) to penalize labels at large propagation steps after observing over-fitting with another weighting choice.
  • Choices for α_l in Last Residual Connection: The cosine weighting scheme outperforms linear-decreasing and fixed-weight alternatives on ogbn-products.
  • Choices for α_l in Last Residual Connection: GAMLP's efficiency comparison on ogbn-products includes sampling-based GraphSAINT and Cluster-GCN, graph-wise SGC, and layer-wise SIGN.

C.1 Experiment Environment

The experiments use repeated runs on specified hardware and compare GAMLP with documented baselines and configurations. Reproduction details include dataset-specific hyperparameters and released PyTorch code.

  • Experiment Environment: Most methods are repeated ten times with means and standard deviations reported, while ogbn-papers100M methods are run five times.
  • Experiment Environment: Experiments run on an Intel Xeon Platinum 8255C CPU and a single Tesla V100 GPU with 32GB memory under Ubuntu and CUDA 10.1.
  • Experiment Environment: The efficiency comparison on ogbn-products covers GraphSAINT, Cluster-GCN, SGC, SIGN, and two GAMLP variants.
  • Experiment Environment: A PyTorch implementation of GAMLP is available on GitHub for reproducing the experiments.
  • Experiment Environment: GAMLP hyperparameters are provided in detailed tables and tuned with OpenBox or taken from original-paper settings.
Loading 2206.04355v1…