PaperlayerView exact revision on arXiv

Source-linked AI summary

GraphPrompt: Unifying Pre-Training and Downstream Tasks for Graph Neural Networks

Zemin Liu, Xingtong Yu, Yuan Fang, Xinming Zhang

arXiv:2302.08043v3cs.LGcs.CL

TL;DR

GraphPrompt targets the mismatch between graph pre-training and downstream objectives, especially when prompting must support different graph tasks. It unifies these tasks through subgraph similarity and adds task-specific learnable prompts for downstream ReadOut aggregation. Across five public datasets, GraphPrompt significantly outperforms various state-of-the-art baselines.

  • Problem

    Graph prompting remains limited and lacks a universal treatment for different downstream tasks, while pre-training and downstream objectives can differ substantially.

    Source: §1 · p. 2

  • Method

    GraphPrompt maps graph pre-training and downstream tasks to a common subgraph-similarity template and uses task-specific learnable prompts to guide downstream ReadOut.

    Source: §1 · pp. 2–3, §4.3 · p. 6

  • Results

    GraphPrompt significantly outperforms various state-of-the-art baselines across experiments on five public datasets.

    Source: §6 · p. 9

  • Takeaways & Limitations

    The framework supports unified use of a pre-trained GNN across node and graph classification, including limited-supervision few-shot settings.

    Source: §1 · p. 2, §5.2 · p. 7

    The evaluation considers node and graph classification in few-shot settings, with only label-free graphs used for pre-training.

    Source: §3.1 · p. 3, §5.1 · p. 7

Abstract

from arXiv · show

Graphs can model complex relationships between objects, enabling a myriad of Web applications such as online page/article classification and social recommendation. While graph neural networks(GNNs) have emerged as a powerful tool for graph representation learning, in an end-to-end supervised setting, their performance heavily rely on a large amount of task-specific supervision. To reduce labeling requirement, the "pre-train, fine-tune" and "pre-train, prompt" paradigms have become increasingly common. In particular, prompting is a popular alternative to fine-tuning in natural language processing, which is designed to narrow the gap between pre-training and downstream objectives in a task-specific manner. However, existing study of prompting on graphs is still limited, lacking a universal treatment to appeal to different downstream tasks. In this paper, we propose GraphPrompt, a novel pre-training and prompting framework on graphs. GraphPrompt not only unifies pre-training and downstream tasks into a common task template, but also employs a learnable prompt to assist a downstream task in locating the most relevant knowledge from the pre-train model in a task-specific manner. Finally, we conduct extensive experiments on five public datasets to evaluate and analyze GraphPrompt.

1 INTRODUCTION

GraphPrompt addresses the mismatch between graph pre-training and downstream objectives by unifying tasks through subgraph similarity and using task-specific learnable prompts for downstream aggregation.

  • GNN-based graph representation learning supports diverse Web tasks, including page classification and social recommendation.Source: §1 · p. 1
  • Pre-training preserves intrinsic graph properties, whereas fine-tuning minimizes downstream task loss, creating potentially large objective discrepancies.The discrepancy can involve pre-training link connectivity while downstream tasks predict node or graph properties.Source: §1 · pp. 1–2
  • GraphPrompt seeks a universal graph-task template so one pre-trained model can support different downstream tasks.The framework is motivated by the masked-language-modeling template used across language-model pre-training and downstream tasks.Source: §1 · p. 2
  • GraphPrompt maps pre-training and downstream tasks to subgraph similarity, using link prediction for pre-training and node or graph classification downstream.Subgraphs provide a common structure for node-level and graph-level tasks.Source: §1 · p. 2
  • A task-specific learnable prompt guides ReadOut aggregation so different downstream tasks can use different aggregation schemes.Node classification can emphasize target-node features, while graph classification can emphasize features associated with graph classes.Source: §1 · pp. 2–3

2 RELATED WORK

Prior graph pre-training methods often overlook the gap between pre-training and downstream objectives. Existing graph prompting is limited, motivating a unified approach that accommodates multiple downstream task types.

  • Existing graph pre-training approaches do not adequately address differing pre-training and downstream objectives, limiting generalization across tasks.Source: §2 · p. 3
  • L2P-GNN simulates fine-tuning during meta-training, but downstream tasks may still differ from the simulated task.Source: §2 · p. 3
  • GPPT uses learnable graph prompts for node classification but does not provide a unification mechanism for graph classification.Source: §2 · p. 3

3 PRELIMINARIES

The paper defines graph, node-classification, and graph-classification settings, then describes GNN message passing and the few-shot supervision regime.

3.1 Problem Definition

  • A graph is defined as G=(V,E) with node features X∈R^|V|×d, while downstream tasks assign labels to nodes or graphs.Source: §3.1 · p. 3
  • The downstream node-classification and graph-classification tasks use limited supervision, with k labeled samples per class in the few-shot setting.Source: §3.1 · p. 3

3.2 Graph Neural Networks

  • GNNs generate node representations by recursively aggregating messages from neighboring nodes across layers.The aggregation function may use mean pooling, neural attention, or multilayer perceptrons.Source: §3.2 · pp. 3–4

4 PROPOSED APPROACH

GraphPrompt unifies link prediction, node classification, and graph classification through subgraph similarity learning, then uses task-specific learnable prompts to adapt ReadOut for downstream objectives.

Unification Framework

  • Subgraphs provide a common representation for node-level instances through contextual neighborhoods and graph-level instances through the graph itself.A contextual subgraph contains nodes within a predetermined hop threshold, while a graph’s maximum subgraph is the graph itself.Source: §4.1 · p. 4
  • Link prediction, node classification, and graph classification are reformulated as subgraph similarity learning tasks.The framework compares representations of linked versus unlinked nodes or instances against class prototypical subgraphs.Source: §4.1 · p. 4

Prompting for Downstream Tasks

  • The unified subgraph-similarity template lays the foundation for transferring pre-trained knowledge across node and graph classification tasks.The framework preserves a shared task structure while allowing downstream prompts to distinguish task characteristics.Source: §4.1 · p. 5, §4.2 · p. 5

Pre-Training Phase

  • GraphPrompt pre-trains on label-free graphs with self-supervised link prediction by increasing similarity between linked nodes’ contextual subgraphs and decreasing similarity to unlinked nodes.Training triplets contain an anchor node, one neighbor, and one non-linked node.Source: §4.2 · p. 5

Prompting for Downstream Tasks

  • A task-specific learnable prompt adapts the ReadOut operation so downstream tasks can extract relevant prior knowledge from node representations.The implemented prompt performs feature-weighted summation, while sum pooling is used as the basic ReadOut scheme.Source: §4.3 · pp. 5–6, §4.1 · p. 5

5 EXPERIMENTS

Experiments evaluate GraphPrompt on few-shot node and graph classification across five benchmark datasets, comparing it with end-to-end GNNs and graph pre-training methods. GraphPrompt generally performs best, particularly with fewer shots, while its prompt-based design also reduces downstream parameter and FLOP requirements.

Few-shot node classification

  • GraphPrompt outperforms all baselines across Flickr, PROTEINS, and ENZYMES for few-shot node classification.The authors attribute this to unification and task-specific prompt-assisted aggregation in ReadOut.Source: §5.2 · p. 7

Few-shot graph classification

  • GraphPrompt significantly outperforms baselines on PROTEINS, COX2, ENZYMES, and BZR for few-shot graph classification.On PROTEINS and ENZYMES, the same pre-trained model supports both node- and graph-level tasks.Source: §5.2 · p. 7

Performance with different shots

  • GraphPrompt consistently outperforms baselines especially with lower shot counts.For graph classification, some baselines surpass it with 20 or more shots, especially on ENZYMES, where 30 shots use 30% of the graphs for training.Source: §5.2 · p. 8

Ablation study

  • Without the prompt vector, performance is usually worst, while replacing it with a linear transformation matrix also hurts performance.The ablation compares no-prompt and linear-prompt variants across node and graph classification datasets.Source: §5.3 · p. 8

Parameter efficiency

  • GraphPrompt requires the least downstream parameters and FLOPs among the compared models.Fine-tuning the pre-trained weights in GraphPrompt+ft substantially increases the number of parameters to update.Source: §5.3 · p. 8

6 CONCLUSIONS

The conclusion presents GraphPrompt as a graph prompting framework that unifies pre-training and downstream objectives while using task-specific prompts. Experiments on five public datasets show that it significantly outperforms state-of-the-art baselines.

6 CONCLUSIONS

  • GraphPrompt maps different graph tasks to a common task template to narrow the gap between pre-training and downstream objectives.The framework targets limitations of supervised and pre-train, fine-tune paradigms.Source: §6 · p. 9
  • A learnable task-specific prompt vector guides each downstream task to use the pre-trained model.The prompt is designed to distinguish task individuality and achieve task-specific optima.Source: §6 · p. 9
  • GraphPrompt significantly outperforms various state-of-the-art baselines on five public datasets.Source: §6 · p. 9

A Algorithm and Complexity Analysis

The algorithm initializes and tunes a task-specific prompt using embeddings from a frozen pre-trained GNN, subgraph ReadOut, class prototypes, and a contrastive-style loss. Its stated embedding and extraction costs depend on graph degree, GNN depth, hop radius, and hidden dimension.

Algorithm

  • Algorithm 1 outputs a task-specific prompt vector after tuning it with a pre-trained GNN and labeled samples.The inputs include graphs, task-specific subgraphs, a labeled set, class set, and pre-trained model.Source: A Algorithm and Complexity Analysis · p. 11
  • The prompt modulates node embeddings before ReadOut computes each task-specific subgraph representation.The algorithm applies element-wise multiplication between the prompt vector and node embeddings.Source: A Algorithm and Complexity Analysis · p. 11
  • Class prototypes are formed by taking the mean of node or graph embedding vectors.These prototypes are then used in the labeled-sample loss calculation.Source: A Algorithm and Complexity Analysis · p. 11

Complexity analysis

  • For a node, GNN embedding costs O(D·d̄^k), while subgraph extraction costs O(d̄^δ).The embedding calculation for the node’s subgraph with ReadOut is O(D·d̄^k·d̄^δ).Source: A Algorithm and Complexity Analysis · p. 11

B Further Descriptions of Datasets

The appendix describes five benchmark datasets spanning image-sharing, protein, molecular, enzyme, and ligand graphs, and specifies how they are assigned to node- or graph-classification tasks. It also lists the compared end-to-end GNN and graph-prompting baselines.

Datasets

Task assignment

  • Node classification uses Flickr, PROTEINS, and ENZYMES, whereas graph classification uses PROTEINS, COX2, ENZYMES, and BZR.For node classification, only graphs with more than 50 nodes are selected.Source: B Further Descriptions of Datasets · p. 11

Baselines

D Further Implementation Details

The paper specifies compact GNN baselines with architectures and hidden dimensions tailored to each model.

E Further Experimental Results

Further experiments examine GraphPrompt's scalability and hyperparameter sensitivity across node and graph classification. Prompt tuning scales linearly with graph size, while hop and hidden-dimension effects differ by task.

Scalability study

Parameter sensitivity

F Data Ethics Statement

The study uses publicly available datasets under their usage terms and conditions, without personally identifiable information or human or animal subjects.

Related papers

Loading 2302.08043v3…