Source-linked AI summary
GraphPrompt: Unifying Pre-Training and Downstream Tasks for Graph Neural Networks
Zemin Liu, Xingtong Yu, Yuan Fang, Xinming Zhang
TL;DR
GraphPrompt targets the mismatch between graph pre-training and downstream objectives, especially when prompting must support different graph tasks. It unifies these tasks through subgraph similarity and adds task-specific learnable prompts for downstream ReadOut aggregation. Across five public datasets, GraphPrompt significantly outperforms various state-of-the-art baselines.
Problem
Graph prompting remains limited and lacks a universal treatment for different downstream tasks, while pre-training and downstream objectives can differ substantially.
Method
GraphPrompt maps graph pre-training and downstream tasks to a common subgraph-similarity template and uses task-specific learnable prompts to guide downstream ReadOut.
Results
GraphPrompt significantly outperforms various state-of-the-art baselines across experiments on five public datasets.
Takeaways & Limitations
The framework supports unified use of a pre-trained GNN across node and graph classification, including limited-supervision few-shot settings.
Takeaways & Limitations
The evaluation considers node and graph classification in few-shot settings, with only label-free graphs used for pre-training.
Abstract
from arXiv · showhide
Graphs can model complex relationships between objects, enabling a myriad of Web applications such as online page/article classification and social recommendation. While graph neural networks(GNNs) have emerged as a powerful tool for graph representation learning, in an end-to-end supervised setting, their performance heavily rely on a large amount of task-specific supervision. To reduce labeling requirement, the "pre-train, fine-tune" and "pre-train, prompt" paradigms have become increasingly common. In particular, prompting is a popular alternative to fine-tuning in natural language processing, which is designed to narrow the gap between pre-training and downstream objectives in a task-specific manner. However, existing study of prompting on graphs is still limited, lacking a universal treatment to appeal to different downstream tasks. In this paper, we propose GraphPrompt, a novel pre-training and prompting framework on graphs. GraphPrompt not only unifies pre-training and downstream tasks into a common task template, but also employs a learnable prompt to assist a downstream task in locating the most relevant knowledge from the pre-train model in a task-specific manner. Finally, we conduct extensive experiments on five public datasets to evaluate and analyze GraphPrompt.
1 INTRODUCTION
GraphPrompt addresses the mismatch between graph pre-training and downstream objectives by unifying tasks through subgraph similarity and using task-specific learnable prompts for downstream aggregation.
- GNN-based graph representation learning supports diverse Web tasks, including page classification and social recommendation.
- Pre-training preserves intrinsic graph properties, whereas fine-tuning minimizes downstream task loss, creating potentially large objective discrepancies.The discrepancy can involve pre-training link connectivity while downstream tasks predict node or graph properties.
- GraphPrompt seeks a universal graph-task template so one pre-trained model can support different downstream tasks.The framework is motivated by the masked-language-modeling template used across language-model pre-training and downstream tasks.
- GraphPrompt maps pre-training and downstream tasks to subgraph similarity, using link prediction for pre-training and node or graph classification downstream.Subgraphs provide a common structure for node-level and graph-level tasks.
- A task-specific learnable prompt guides ReadOut aggregation so different downstream tasks can use different aggregation schemes.Node classification can emphasize target-node features, while graph classification can emphasize features associated with graph classes.
2 RELATED WORK
Prior graph pre-training methods often overlook the gap between pre-training and downstream objectives. Existing graph prompting is limited, motivating a unified approach that accommodates multiple downstream task types.
- Existing graph pre-training approaches do not adequately address differing pre-training and downstream objectives, limiting generalization across tasks.
- L2P-GNN simulates fine-tuning during meta-training, but downstream tasks may still differ from the simulated task.
- GPPT uses learnable graph prompts for node classification but does not provide a unification mechanism for graph classification.
3 PRELIMINARIES
The paper defines graph, node-classification, and graph-classification settings, then describes GNN message passing and the few-shot supervision regime.
- 3.1 Problem Definition: A graph is defined as G=(V,E) with node features X∈R^|V|×d, while downstream tasks assign labels to nodes or graphs.
- 3.1 Problem Definition: The downstream node-classification and graph-classification tasks use limited supervision, with k labeled samples per class in the few-shot setting.
- 3.2 Graph Neural Networks: GNNs generate node representations by recursively aggregating messages from neighboring nodes across layers.The aggregation function may use mean pooling, neural attention, or multilayer perceptrons.
4 PROPOSED APPROACH
GraphPrompt unifies link prediction, node classification, and graph classification through subgraph similarity learning, then uses task-specific learnable prompts to adapt ReadOut for downstream objectives.
- Unification Framework: Subgraphs provide a common representation for node-level instances through contextual neighborhoods and graph-level instances through the graph itself.A contextual subgraph contains nodes within a predetermined hop threshold, while a graph’s maximum subgraph is the graph itself.
- Unification Framework: Link prediction, node classification, and graph classification are reformulated as subgraph similarity learning tasks.The framework compares representations of linked versus unlinked nodes or instances against class prototypical subgraphs.
- Prompting for Downstream Tasks: The unified subgraph-similarity template lays the foundation for transferring pre-trained knowledge across node and graph classification tasks.The framework preserves a shared task structure while allowing downstream prompts to distinguish task characteristics.
- Pre-Training Phase: GraphPrompt pre-trains on label-free graphs with self-supervised link prediction by increasing similarity between linked nodes’ contextual subgraphs and decreasing similarity to unlinked nodes.Training triplets contain an anchor node, one neighbor, and one non-linked node.
- Prompting for Downstream Tasks: A task-specific learnable prompt adapts the ReadOut operation so downstream tasks can extract relevant prior knowledge from node representations.The implemented prompt performs feature-weighted summation, while sum pooling is used as the basic ReadOut scheme.
5 EXPERIMENTS
Experiments evaluate GraphPrompt on few-shot node and graph classification across five benchmark datasets, comparing it with end-to-end GNNs and graph pre-training methods. GraphPrompt generally performs best, particularly with fewer shots, while its prompt-based design also reduces downstream parameter and FLOP requirements.
- Few-shot node classification: GraphPrompt outperforms all baselines across Flickr, PROTEINS, and ENZYMES for few-shot node classification.The authors attribute this to unification and task-specific prompt-assisted aggregation in ReadOut.
- Few-shot graph classification: GraphPrompt significantly outperforms baselines on PROTEINS, COX2, ENZYMES, and BZR for few-shot graph classification.On PROTEINS and ENZYMES, the same pre-trained model supports both node- and graph-level tasks.
- Performance with different shots: GraphPrompt consistently outperforms baselines especially with lower shot counts.For graph classification, some baselines surpass it with 20 or more shots, especially on ENZYMES, where 30 shots use 30% of the graphs for training.
- Ablation study: Without the prompt vector, performance is usually worst, while replacing it with a linear transformation matrix also hurts performance.The ablation compares no-prompt and linear-prompt variants across node and graph classification datasets.
- Parameter efficiency: GraphPrompt requires the least downstream parameters and FLOPs among the compared models.Fine-tuning the pre-trained weights in GraphPrompt+ft substantially increases the number of parameters to update.
6 CONCLUSIONS
The conclusion presents GraphPrompt as a graph prompting framework that unifies pre-training and downstream objectives while using task-specific prompts. Experiments on five public datasets show that it significantly outperforms state-of-the-art baselines.
- 6 CONCLUSIONS: GraphPrompt maps different graph tasks to a common task template to narrow the gap between pre-training and downstream objectives.The framework targets limitations of supervised and pre-train, fine-tune paradigms.
- 6 CONCLUSIONS: A learnable task-specific prompt vector guides each downstream task to use the pre-trained model.The prompt is designed to distinguish task individuality and achieve task-specific optima.
- 6 CONCLUSIONS: GraphPrompt significantly outperforms various state-of-the-art baselines on five public datasets.
A Algorithm and Complexity Analysis
The algorithm initializes and tunes a task-specific prompt using embeddings from a frozen pre-trained GNN, subgraph ReadOut, class prototypes, and a contrastive-style loss. Its stated embedding and extraction costs depend on graph degree, GNN depth, hop radius, and hidden dimension.
- Algorithm: Algorithm 1 outputs a task-specific prompt vector after tuning it with a pre-trained GNN and labeled samples.The inputs include graphs, task-specific subgraphs, a labeled set, class set, and pre-trained model.
- Algorithm: The prompt modulates node embeddings before ReadOut computes each task-specific subgraph representation.The algorithm applies element-wise multiplication between the prompt vector and node embeddings.
- Algorithm: Class prototypes are formed by taking the mean of node or graph embedding vectors.These prototypes are then used in the labeled-sample loss calculation.
- Complexity analysis: For a node, GNN embedding costs O(D·d̄^k), while subgraph extraction costs O(d̄^δ).The embedding calculation for the node’s subgraph with ReadOut is O(D·d̄^k·d̄^δ).
B Further Descriptions of Datasets
The appendix describes five benchmark datasets spanning image-sharing, protein, molecular, enzyme, and ligand graphs, and specifies how they are assigned to node- or graph-classification tasks. It also lists the compared end-to-end GNN and graph-prompting baselines.
- Datasets: Flickr is an image-sharing network whose nodes are images assigned to seven categories.Edges connect images sharing properties such as a commenter or location.
- Datasets: PROTEINS contains protein graphs with three node categories and two graph classes.Nodes represent secondary structures, while edges encode neighboring relations in amino-acid sequence or 3D space.
- Datasets: COX2 contains 467 molecular structures, ENZYMES contains 600 enzymes across six categories, and BZR contains 405 two-category ligands.
- Task assignment: Node classification uses Flickr, PROTEINS, and ENZYMES, whereas graph classification uses PROTEINS, COX2, ENZYMES, and BZR.For node classification, only graphs with more than 50 nodes are selected.
- Baselines: The baselines include end-to-end GNNs such as GCN, GraphSAGE, and GAT, plus GPPT, which reformulates node classification as link prediction with a learnable prompt.
D Further Implementation Details
The paper specifies compact GNN baselines with architectures and hidden dimensions tailored to each model.
- GCN, GraphSAGE, and GIN use 3-layer architectures with hidden dimension 32.
- GAT uses a 2-layer architecture with hidden dimension 32 and four attention heads in its first layer.
- GraphSAGE uses the mean aggregator.
E Further Experimental Results
Further experiments examine GraphPrompt's scalability and hyperparameter sensitivity across node and graph classification. Prompt tuning scales linearly with graph size, while hop and hidden-dimension effects differ by task.
- Scalability study: Prompt tuning time increases linearly with graph size on PROTEINS graph classification.Graphs were grouped into six size ranges of approximately 50–100 nodes, with 10 sampled graphs per group.
- Scalability study: GraphPrompt-ft requires more tuning time than GraphPrompt, indicating greater inefficiency for the fine-tuning variant.
- Parameter sensitivity: Increasing the subgraph hop count gradually decreases node-classification performance.Larger subgraphs can introduce irrelevant information and over-smoothing.
- Parameter sensitivity: The hop count shows no clear performance trend for graph classification because the whole graph is used downstream.Both small and large subgraphs can help capture substructure information at different scales.
- Parameter sensitivity: Hidden dimensions 32 or 64 appear robust across node and graph classification.Smaller dimensions such as 32 and 64 favor node classification, whereas 64 and 128 may favor graph classification.
F Data Ethics Statement
The study uses publicly available datasets under their usage terms and conditions, without personally identifiable information or human or animal subjects.
- The experiments use only publicly available datasets.
- The datasets are used according to their usage terms and conditions, if applicable.
- The research uses no personally identifiable information and involves no human or animal subjects.