PaperlayerView exact revision on arXiv

Source-linked AI summary

All in One: Multi-task Prompting for Graph Neural Networks

Xiangguo Sun, Hong Cheng, Jia Li, Bo Liu, Jihong Guan

arXiv:2307.01504v2cs.SIcs.AIcs.LG

TL;DR

The paper addresses the mismatch between graph pre-training and diverse node-, edge-, and graph-level tasks. It introduces a multi-task prompting framework that unifies prompt design, reformulates tasks as graph-level problems, and uses meta-learning, with experiments reporting improvements across task levels.

  • Problem

    Graph pre-training pretexts can be incompatible with diverse node-, edge-, and graph-level downstream tasks, limiting transfer and risking negative transfer.

  • Method

    The framework unifies prompt tokens, token structures, and insertion patterns, reformulates node- and edge-level tasks as graph-level tasks, and uses meta-learning for prompt initialization.

  • Results

    The method improves over compared approaches by 1.10%–8.81% on node-level tasks, 1.28%–12.26% on edge-level tasks, and 0.14%–10.77% on graph-level tasks.

  • Takeaways & Limitations

    Experiments demonstrate the effectiveness of the proposed framework for multi-task graph prompting under few-shot settings.

Abstract

from arXiv · show

Recently, ''pre-training and fine-tuning'' has been adopted as a standard workflow for many graph tasks since it can take general graph knowledge to relieve the lack of graph annotations from each application. However, graph tasks with node level, edge level, and graph level are far diversified, making the pre-training pretext often incompatible with these multiple tasks. This gap may even cause a ''negative transfer'' to the specific application, leading to poor results. Inspired by the prompt learning in natural language processing (NLP), which has presented significant effectiveness in leveraging prior knowledge for various NLP tasks, we study the prompting topic for graphs with the motivation of filling the gap between pre-trained models and various graph tasks. In this paper, we propose a novel multi-task prompting method for graph models. Specifically, we first unify the format of graph prompts and language prompts with the prompt token, token structure, and inserting pattern. In this way, the prompting idea from NLP can be seamlessly introduced to the graph area. Then, to further narrow the gap between various graph tasks and state-of-the-art pre-training strategies, we further study the task space of various graph applications and reformulate downstream problems to the graph-level task. Afterward, we introduce meta-learning to efficiently learn a better initialization for the multi-task prompt of graphs so that our prompting framework can be more reliable and general for different tasks. We conduct extensive experiments, results from which demonstrate the superiority of our method.

1 INTRODUCTION

The paper identifies a mismatch between graph pre-training and diverse downstream tasks, then proposes multi-task graph prompting inspired by NLP to bridge it.

  • Graph supervised learning depends heavily on scarce labels and can overfit when test data is out of distribution.
  • Binary edge-prediction pre-training may transfer poorly to node- and graph-level tasks because graph tasks operate at different levels.
  • Graph prompts require jointly designing prompt content, token organization, and insertion into the original graph.
  • Unlike NLP tasks, graph downstream tasks may lack a shared task subspace, complicating pre-training-task alignment and generalization.
  • The proposed framework unifies language and graph prompts, reformulates node- and edge-level tasks as graph-level tasks, and uses meta-learning for prompt initialization.

2 BACKGROUND

The background positions GNNs as expressive graph representation models built around message passing and adaptive transfer under limited annotation.

  • GNNs capture underlying message-passing patterns to produce representations for graph-based applications.
  • Representative graph architectures include graph attention networks, graph convolutional networks, and Graph Transformers.

3.1 Overview of Our Framework

The framework converts diverse graph tasks into unified graph-level instances, applies a structured prompt graph, and learns adaptive prompts through meta-learning.

  • The framework aims to insert a learned prompt graph into the original graph to bridge pre-training strategies and multiple downstream tasks.
  • It reformulates various downstream tasks as graph-level tasks to provide a common format.
  • The prompt graph uses learnable tokens, inner structures, and adaptive inserting patterns.
  • A meta-learning process learns more adaptive graph prompts for multi-task settings.

3.2 Reformulating Downstream Tasks

The paper treats graph-level tasks as a shared target space and converts node- and edge-level problems into graph classification using induced subgraphs.

  • Graph tasks differ substantially because node-level and edge-level operations are not directly equivalent.
  • Node and edge operations can be represented as graph-level operations, establishing a hierarchical relation among task types.
  • Graph-level tasks provide the largest overlapping task subspace for knowledge transfer, motivating downstream-task reformulation.
  • Node-level tasks become graph-level tasks by assigning target-node labels to induced local graphs.
  • Edge-level tasks become graph-level tasks by assigning edge labels to induced graphs around node pairs.
  • For weighted graphs, induced neighborhoods use shortest-path distance; for unweighted graphs, distance corresponds to hop length.

3.3 Prompt Graph Design

The paper represents graph prompts through prompt tokens, token structures, and inserting patterns, unifying their format with language prompts while addressing graph-specific organization challenges.

  • 3.3 Prompt Graph Design: Graph and language prompts are unified around prompt tokens, token structures, and inserting patterns.Graph prompting requires defining how tokens are organized and inserted because graphs lack sentence-like positions.
  • 3.3.1 Prompting NLP and Graph in One Way.: A graph instance G = (V, E) contains node features and edges, while the prompt graph G_p = (P, S) contains prompt tokens and their relations.Each prompt token is represented in the feature space used by the input graph.
  • 3.3.2 Prompt Tokens.: Prompt tokens have the same dimensionality as input node features and can modify node features before the graph enters the pre-trained model.A prompted feature may be formed as x̂_i = x_i + p_j.
  • 3.3.2 Prompt Tokens.: The prompt-token structure S records pairwise relations among tokens, which are implicit rather than automatically specified as in language prompts.The framework therefore treats token structure as a separate graph-prompt design component.
  • 3.3.3 Token Structures.: Token structures can use learnable connection parameters, dot-product pruning with a sigmoid threshold, or independent tokens with S = ∅.These alternatives provide different ways to determine which prompt tokens connect.
  • 3.3.3 Token Structures.: An inserting function ψ combines the original graph G and prompt graph G_p into a manipulated graph G_m = ψ(G, G_p).The insertion pattern determines how prompt information is connected to the original graph.
  • 3.3.4 Inserting Patterns.: One inserting pattern uses weighted connections between prompt tokens and input nodes, such as x̂_i = x_i + Σ_k w_ik p_k.The weights w_ik prune unnecessary connections.
  • 3.3.4 Inserting Patterns.: A simplified insertion variant also adds a prompt contribution to the original node feature, using the same general prompted-feature form.The paper presents this as an alternative special case of the inserting pattern.

3.4 Multi-task Prompting via Meta Learning

The framework converts node, edge, and graph tasks into unified meta-prompting tasks, then uses inner adaptation and outer meta-learning to learn prompt and task-head initializations.

  • 3.4 Multi-task Prompting via Meta Learning: Each task τ_i is represented with supporting data D^s_τi and query data D^q_τi for meta-learning.Graph classification uses labeled graphs as both support and query examples.
  • 3.4.1 Constructing Meta Prompting Tasks.: Node classification becomes graph classification by generating an induced graph per node and assigning it the target node label.Each induced graph is placed in the support or query set.
  • 3.4.1 Constructing Meta Prompting Tasks.: Edge classification is converted similarly by generating edge-induced graphs whose labels are determined by their two endpoints.These graphs are generated for both training and testing.
  • 3.4.1 Constructing Meta Prompting Tasks.: The pipeline combines prompt parameters θ, a frozen pre-trained backbone π*, and tasker parameters ϕ.The task loss is evaluated on data passed through this prompted pipeline.
  • 3.4.2 Applying Meta-learning to Graph Prompting.: Meta-learning seeks effective initializations (θ, ϕ) by minimizing meta-loss across multiple tasks.The method updates these initializations using query-data performance after task-specific adaptation.
  • 3.4.2 Applying Meta-learning to Graph Prompting.: The task head, or answering function, connects prompts to downstream answers and may be tunable or hand-crafted.The paper also gives a hand-crafted answering template that avoids a tunable task head.
  • 3.4.3 Overall Learning Process.: Multi-task episodes batch node, edge, and graph classification tasks, while the overall method treats each class as a binary task sharing one task head.The framework can also accommodate other tasks with a few adaptations.

3.5 Why It Works?

The paper explains prompting as input manipulation that can imitate graph transformations, extend compatibility across tasks and pretexts, and retain lower prompt-specific complexity than full model tuning.

  • 3.5.1 Compatibility: GPPT is limited to node classification with binary edge prediction, whereas this framework supports node-, edge-, and graph-level tasks with varied graph-level pretexts.The comparison highlights broader task and pretext compatibility.
  • 3.5.2 Flexibility: Prompting manipulates input data to match a pretext, making flexible graph operations a bottleneck for prompting performance.The paper considers transformations such as changing node features or adding and removing edges or subgraphs.
  • 3.5.2 Flexibility: A learned prompt token can imitate arbitrary graph manipulation in the representation space of a frozen pre-trained graph model.The approximation error depends on immutable nonlinear model layers and the learned prompt quality.
  • 3.5.2 Flexibility: The method extends a standalone prompt token to a prompt graph with multiple tokens and learnable inner structure.The prompt graph is inserted into the original graph through a learned inserting pattern.
  • 3.5.2 Flexibility: Efficient tuning can reduce the new representation error bound, empirically making it significantly smaller than the standalone-token bound.The paper links this reduction to more flexible transformations for matching different pre-training strategies.
  • 3.5.3 Efficiency: Prompt-graph parameter complexity is O(nd), compared with typical GAT parameter terms involving L, K, and d.Here n is the number of prompt tokens and d is the maximum layer dimension.
  • 3.5.3 Efficiency: Compared with the original GCN computation, prompt insertion adds only O(Lnd^2 + Lmd + nd) time when n ≪ d, n ≪ N, and m ≪ M.N and M denote original nodes and edges, while n and m denote prompt tokens and edges.
  • 3.5.4 Compatibility: The framework can transfer across tasks while adapting only the prompt, and it can use a hand-crafted answering template without tuning a task head.A GraphCL-based prompt example targets node, edge, and graph classification.

4 EVALUATION

The evaluation compares prompting with supervised and pre-training baselines across node-, edge-, and graph-level few-shot tasks, then examines transferability across tasks and domains. Results report improvements across all three task levels and stronger transfer behavior for the proposed prompt method.

  • Evaluation setup: The evaluation covers node-level, edge-level, and graph-level tasks across five public datasets: Cora, CiteSeer, Reddit, Amazon, and Pubmed.Edge labels are determined from endpoint classes, while subgraph labels follow majority node labels.
  • Evaluation setup: Compared approaches include supervised GAT, GCN, and Graph Transformer models, pre-training with fine-tuning, and frozen-backbone prompt methods.The prompt methods modify the input graph with a learnable prompt graph and reformulate downstream tasks to fit pre-training strategies.
  • Few-shot performance: In few-shot settings, supervised methods generally underperform pre-training and prompt methods because their required annotations are limited.Pre-training methods leverage prior knowledge, but selecting and tuning a suitable pre-training strategy remains task-dependent.
  • Few-shot performance: 1.10% to 8.81% improvements are reported on node-level tasks, 1.28% to 12.26% on edge-level tasks, and 0.14% to 10.77% on graph-level tasks.The node-level comparison uses 100 labeled samples per class, producing substantially lower labeled ratios than the cited GPPT setting.
  • Transferability: On Amazon, the prompt method significantly outperforms hard transfer and fine-tuning when transferring from graph- or node-level sources to an edge-level target.Graph-level source tasks show better adaptability than node-level source tasks for the edge-level target.
  • Transferability: Good prompt transferability also appears across source domains Amazon and PubMed when evaluating on the Cora target domain.SVD reduces differing input feature dimensions to a common 100-dimensional representation.

4.4 Ablation Study (RQ3)

The ablation and efficiency analyses identify meta-learning, token structure, and prompt insertion as important components, while token-efficient prompts reduce parameter and optimization costs. Graph transformations further show that the designed prompt can reduce representation error.

  • Ablation components: The ablation compares the complete framework with variants removing meta-learning, task-head tuning, token structure, or prompt-to-input insertion links.These variants isolate the contributions of the main prompt components.
  • Component effects: Meta-learning and token structure contribute significantly to final performance, while the prompt-to-input inserting pattern is especially crucial.The inserting ablation removes links between prompt tokens and the input graph.
  • Component effects: Removing the tunable task head still yields competitive performance, indicating that the prompt can bridge upstream pre-training and downstream tasks.This result is reported for the “w/o h” variant in Figure 5.
  • Efficiency: The prompt-based method converges faster than traditional pre-training and supervised methods, supporting an efficiency advantage during tuning.The paper attributes this advantage to the smaller prompt parameter space.
  • Graph transformation: Designed prompt graphs reduce the error between original and manipulated graphs under node dropping, edge dropping, and feature masking.The comparison includes naive prompts and designed prompts with 3, 5, and 10 tokens.

5 CONCLUSION

The paper studies multi-task graph prompting in few-shot settings and proposes a unified reformulation with meta-learning. Extensive experiments demonstrate the framework’s effectiveness.

  • The paper addresses multi-task graph prompting under few-shot settings.
  • The method reformulates different-level graph tasks into unified tasks and designs an effective prompt graph.
  • Meta-learning is used to improve prompt initialization for multi-task performance.
  • Extensive experiments demonstrate the effectiveness of the proposed framework.

A APPENDIX

The appendix extends evaluation across additional datasets, task types, and classification settings. Across the reported experiments, the prompt-based method outperforms competing approaches.

  • Additional Datasets: Additional datasets evaluate graph-level classification, edge-level regression, graph-level regression, and link prediction.ENZYMES and ProteinsFull support graph-level classification; Movielens and QM9 support regression; PersonalityCafe and Facebook support link prediction.
  • Multi-label v.s. Multi-class Classification: The multi-class experiments show that the prompt-based method still outperforms the other methods.
  • Additional Graph-level Classification: Graph-level classification remains especially effective in the multi-class and few-shot setting.
  • Edge/Graph-level Regression: Regression experiments use MAE and MSE on 100-shot edge-induced graphs, with prompt-based methods outperforming traditional approaches.
  • Link Prediction: The link-prediction evaluation partitions edges into message-passing, supervision-training, and testing sets, with non-adjacent nodes sampled as negatives.
Loading 2307.01504v2…