PaperlayerView exact revision on arXiv

Source-linked AI summary

GPT-GNN: Generative Pre-Training of Graph Neural Networks

Ziniu Hu, Yuxiao Dong, Kuansan Wang, Kai-Wei Chang, Yizhou Sun

arXiv:2006.15437v1cs.LGcs.SIstat.ML

TL;DR

GNNs often need expensive task-specific labels, motivating methods that learn transferable representations from unlabeled graphs. GPT-GNN generatively pre-trains a GNN by reconstructing attributed graph structure and attributes, then transfers it to downstream tasks. Across large-scale OAG and Amazon data, the framework improves downstream performance, including a reported 9.1% average gain on OAG node classification and link prediction over GNNs without pre-training.

  • Problem

    Training GNNs for different tasks requires abundant task-specific labels that are expensive or infeasible to obtain, particularly on large-scale graphs.

  • Method

    GPT-GNN pre-trains a generic GNN on an unlabeled attributed graph by autoregressively reconstructing node attributes and graph structure through separate Attribute Generation and Edge Generation components.

  • Results

    GPT-GNN significantly improves various downstream tasks across large-scale OAG and Amazon data, with OAG node classification and link prediction averaging 9.1% higher performance than state-of-the-art GNNs without pre-training.

  • Takeaways & Limitations

    The pre-trained GNN can initialize models for different downstream tasks on the same graph or graphs within the same domain using few labeled examples.

  • Takeaways & Limitations

    A naive generation factorization that separately models attributes and connections neglects their dependency, preventing informative guidance for GNN pre-training.

Abstract

from arXiv · show

Graph neural networks (GNNs) have been demonstrated to be powerful in modeling graph-structured data. However, training GNNs usually requires abundant task-specific labeled data, which is often arduously expensive to obtain. One effective way to reduce the labeling effort is to pre-train an expressive GNN model on unlabeled data with self-supervision and then transfer the learned model to downstream tasks with only a few labels. In this paper, we present the GPT-GNN framework to initialize GNNs by generative pre-training. GPT-GNN introduces a self-supervised attributed graph generation task to pre-train a GNN so that it can capture the structural and semantic properties of the graph. We factorize the likelihood of the graph generation into two components: 1) Attribute Generation and 2) Edge Generation. By modeling both components, GPT-GNN captures the inherent dependency between node attributes and graph structure during the generative process. Comprehensive experiments on the billion-scale Open Academic Graph and Amazon recommendation data demonstrate that GPT-GNN significantly outperforms state-of-the-art GNN models without pre-training by up to 9.1% across various downstream tasks.

1 INTRODUCTION

GPT-GNN addresses the high cost and scarcity of task-specific labels by generatively pre-training GNNs on unlabeled attributed graphs. It models both node attributes and graph structure, then transfers the pre-trained model to downstream tasks, improving performance across large-scale graph benchmarks.

  • Task-specific GNN training often requires expensive, sometimes infeasible labeled datasets, especially for large-scale graphs.Author disambiguation in academic graphs is given as an example of a task lacking sufficient ground truth.
  • GPT-GNN pre-trains GNNs by reconstructing input attributed graphs to capture their structural and semantic properties.The pre-trained model is intended to generalize to downstream tasks with few fine-tuning steps on graphs within the same domain.
  • Existing neural graph generation methods are limited for GNN pre-training because they often omit node attributes and target small graphs.Omitting attributes fails to model dependencies between attributes and graph structure, which are central to convolutional aggregation.
  • GPT-GNN uses a self-supervised attributed graph generation task and initializes downstream models with the resulting pre-trained GNN.The transfer setting covers the input graph or graphs of the same type or domain.
  • The graph-generation objective jointly models Attribute Generation and Edge Generation, capturing dependencies between node attributes and graph structure.Joint optimization is equivalent to maximizing the likelihood of the whole attributed graph.
  • 9.1% average performance improvement was reported for node classification and link prediction on OAG over state-of-the-art GNN models without pre-training.Experiments used the 179-million-node, 2-billion-edge Open Academic Graph and 113-million-node Amazon recommendation data; GPT-GNN also improved different base GNNs under varied settings.

2 PRELIMINARIES AND RELATED WORK

This section reviews GNN message passing and prior graph pre-training approaches. It distinguishes transferable generic GNN pre-training from node embeddings and motivates generative objectives as a way to model richer graph semantics and structure.

  • 2.1 Preliminaries of Graph Neural Networks: GNNs use graph structure as a computation graph, aggregating local neighborhood information through message passing to produce contextual node representations.The section introduces extraction and aggregation as basic GNN operators, with mean, sum, max, pooling, normalization, and attention-based variants.
  • 2.2 Pre-Training for Graphs: Network or graph embedding methods parameterize node vectors to preserve similarity measures, but those embeddings cannot initialize other models for fine-tuning.GPT-GNN instead targets a generic GNN usable across different tasks in a transfer-learning setting.
  • 2.2 Pre-Training for Graphs: Prior unsupervised GNN methods reconstruct structure, use random-walk similarity, or maximize mutual information between node and graph representations.The reviewed methods improve over purely supervised settings but can ignore rich semantics and higher-order graph structure.
  • 2.2 Pre-Training for Graphs: Other approaches pre-train graph-level representations by maximizing mutual information between whole-graph representations and substructures or combining node- and graph-level strategies.These methods are discussed in relation to graph classification tasks.
  • 2.2 Pre-Training for Graphs: Computer vision and NLP commonly pre-train models before fine-tuning them on downstream tasks, including self-supervised approaches in both domains.The section presents these fields as broader contexts for graph pre-training.

3 GENERATIVE PRE-TRAINING OF GNNS

GPT-GNN pre-trains a general GNN from an unlabeled attributed graph by reconstructing its node attributes and edges, then uses the model to initialize downstream tasks. Its dependency-aware factorization preserves interactions between graph structure and attributes while enabling efficient generation.

  • GNN Pre-Training: GNN pre-training learns a general model from a single large-scale graph without labels for downstream tasks on the same graph or domain.
  • Generative Pre-Training: GPT-GNN pre-trains GNNs by reconstructing or generating the input graph’s structure and attributes and maximizing its graph likelihood.
  • Attributed Graph Generation: The graph likelihood is factorized autoregressively over a node permutation, generating each node’s attributes and connections from previously generated nodes.
  • Attributed Graph Generation: The naive independent factorization neglects the dependency between node attributes and connections, which the proposed mechanism is designed to capture.
  • Attributed Graph Generation: A dependency-aware factorization first generates node attributes from observed edges, then generates masked edges from observed edges and generated attributes.
  • Efficient Generation: Attribute generation uses decoders suited to the attribute type, while edge generation predicts masked connections with a pairwise score and negative contrastive estimation.

4 EVALUATION

GPT-GNN is evaluated across large heterogeneous and homogeneous graphs, transfer settings, pre-training variants, and base GNN architectures. It consistently improves downstream performance, with gains depending on dataset, transfer setting, and pre-training component.

  • Overall results: GPT-GNN significantly enhances all downstream tasks on OAG and Amazon, achieving average relative gains of 13.3% and 5.7% over the non-pre-trained base model.It also consistently outperforms other pre-training frameworks across tasks, transfer settings, and datasets.
  • Different transfer settings: 11.7% and 4.6% performance gains remain under the most challenging combined field-and-time transfer setting on OAG and Amazon, respectively.Field transfer yields larger gains than time transfer, while combined transfer yields the least gain.
  • Ablation studies on pre-training tasks: Combining Attribute Generation and Edge Generation produces the best performance on both datasets, although the stronger individual component differs by dataset.On OAG, Edge Generation gains 10.3% versus 7.4% for Attribute Generation; on Amazon, the corresponding gains are 4.1% and 5.2%.
  • Ablation studies on pre-training tasks: 10.3% versus 7.4% and 5.2% versus 3.1% versus 1.3% show GPT-GNN’s Edge Generation outperforming GAE and GraphSage on OAG and Amazon.The paper attributes this advantage to autoregressive modeling of dependencies between masked edges, which GAE discards.
  • Ablation studies on the base GNN: GPT-GNN improves downstream performance for HGT, GCN, GAT, RGCN, and HAN under combined transfer with 10% training data.The framework’s gains therefore extend across the evaluated GNN architectures, with GPT-GNN using HGT producing the most promising result in that experiment.
  • Ablation studies on design choices: Removing node separation can make Attribute Generation perform worse than no pre-training, while removing the adaptive queue causes consistent performance drops across tasks.Node separation addresses attribute information leakage; the adaptive queue adds negative samples and reduces the gap between sampled and full graphs.
  • Training data size: 10–20% of labeled data with pre-training achieves performance comparable to supervised learning with all 100% of training data in the paper–field evaluation.GPT-GNN performs best among the compared pre-training frameworks as labeled data increases from 10% to 100%.

5 CONCLUSION

GPT-GNN is a generative framework for pre-training GNNs by autoregressively reconstructing graph attributes and structure. Fine-tuning with few labeled examples yields gains across tasks, datasets, and transfer settings.

  • GPT-GNN autoregressively reconstructs both input-graph attributes and structure through a factorized graph-generation objective.The framework separates attribute and edge generation while also separating generation nodes to avoid information leakage.
  • GPT-GNN introduces an adaptive node representation queue to reduce the gap between sampled-graph and full-graph likelihoods.
  • Pre-trained GNNs achieve significant gains across downstream tasks and datasets when fine-tuned with few labeled data.
  • 10–20% of labeled data with generative pre-training offers comparative downstream performance to supervised GNNs trained with 100% of the data.

A DATASET DETAILS

The evaluation uses heterogeneous OAG and Amazon graphs with distinct schemas, attributes, and node-level classification tasks. OAG supports three downstream tasks, while Amazon focuses on five-class review-rating prediction.

  • Open Academic Graph: OAG contains five node types and 14 edge types, including six hierarchical Field levels and differentiated author-order relations.
  • Open Academic Graph: OAG downstream tasks predict Paper–Field, Paper–Venue, and Author Disambiguation labels using contextual node representations and softmax classification.
  • Attributes: Paper and author attributes are extracted from paper titles because their node counts make traditional node-embedding algorithms unsuitable.
  • Amazon Review Recommendation Dataset: Amazon contains reviews, users, products, and product metadata such as color, size, style, and quantity, with reviews carrying text and ratings.
  • Amazon Review Recommendation Dataset: Amazon downstream evaluation performs five-class rating classification for each new review using contextual review representations.
  • Attributes: Amazon review embeddings come from pre-trained XLNet, while other node attributes average embeddings of associated reviews.

B OVERALL PIPELINE OF GPT-GNN

GPT-GNN samples attributed subgraphs, chooses a node permutation and masked edges, then computes contextual embeddings for generative losses. The design enables parallel loss calculation from one GNN forward pass.

  • Sampling and ordering: GPT-GNN samples an attributed subgraph from graph G as each generative pre-training instance.
  • Sampling and ordering: The pipeline determines a node permutation order so one forward pass can produce whole-graph representations and parallelize per-node loss computation.
  • Edge masking and encoding: For each node, GPT-GNN randomly selects outward edges to mask before computing contextualized embeddings for the whole graph.
  • Edge masking and encoding: The resulting contextualized node embeddings are used to calculate the generative losses for attribute and edge reconstruction.

C IMPLEMENTATION DETAILS AND CONVERGENCE CURVES

Implementation uses sampled subgraphs and convergence monitoring for large-scale pre-training and downstream fine-tuning. The reported curves indicate sustained pre-training optimization and improved generalization from pre-training.

  • Implementation details: OAG sampling averages 3561-node subgraphs after six sampling repetitions, while Amazon averages 1478-node subgraphs after eight repetitions.Each batch contains 32 sampled graphs for generative pre-training.
  • Convergence curves: Figure 5 compares fine-tuning convergence for GPT-GNN and no-pretrain under different training-data percentages.
  • Convergence curves: Figure 6 tracks pre-training validation error against epoch and time, with GPT-GNN taking about 12 hours to converge.
  • Convergence curves: The steadily decreasing validation loss suggests the generative task does not converge immediately to a trivial solution.
  • Convergence curves: GPT-GNN consistently produces a more generalized model than no-pretrain and is more robust to over-fitting across training-data percentages.

D PAPER TITLE GENERATION EXAMPLES

GPT-GNN generates paper titles from partial neighborhoods, capturing each paper’s main meaning. These results indicate that graphs contain rich semantic information that supports downstream generalization.

  • GPT-GNN can generate paper titles from partial neighborhoods using an Attribute Generation Node with the input attribute replaced by a share vector.The OAG title-generation results are reported in table 4.
  • The generated titles capture the main meaning of each paper despite relying only on partial neighborhoods.
  • The results show that the graph itself contains rich semantic information, helping explain why a pre-trained model can generalize to downstream tasks.
Loading 2006.15437v1…