PaperlayerView exact revision on arXiv

Source-linked AI summary

Task-Specific Prompt with Global Context for Multi-Task Graph Pre-Training

Zhiyang Qiu, Yangtao Wang, Xiaocui Li, Yanzhao Xie, Siyuan Chen, Wensheng Zhang

arXiv:2609.00047v1cs.LGcs.AI

TL;DR

Existing multi-task graph pre-training methods commonly use randomly initialized prompts, leaving task-relevant and graph-structural information insufficiently encoded at initialization. TPGC initializes prompts with task and structural priors through complementary modules, and experiments report consistently stronger few-shot performance with fewer downstream tunable parameters and lower runtime.

  • Problem

    Existing methods commonly initialize pre-training prompts randomly, without explicitly encoding pre-training-task information or graph structural information at initialization.

  • Method

    TPGC uses TPIM for short homologous multi-task pre-training on a related auxiliary graph and SPIM for injecting transferable global structural context into prompt initialization.

  • Results

    TPGC consistently outperforms strong graph prompt learning baselines across few-shot settings while using fewer downstream tunable parameters and lower runtime.

  • Takeaways & Limitations

    Initializing prompts with task-aware prior knowledge and transferable structural context improves prompt quality, few-shot performance, and optimization stability across node and graph classification tasks.

  • Takeaways & Limitations

    The current design may be limited when handling heterophilous graphs, where connected nodes often have different labels.

Abstract

from arXiv · show

Graph prompt learning is an effective paradigm to adapt pre-trained graph models to downstream tasks in low-resource scenarios. However, existing multi-task graph pre-training frameworks generally use randomly initialized prompts, leading to poor alignment between the prompt space, pretext objectives and graph structural characteristics. This greatly weakens the task relevance, structural awareness and transferability of prompt representations. To address this challenge, we propose TPGC, a dual-prior prompt initialization solution that explicitly models the synergy between task prior and structural prior. Specifically, the Task-Prior Injection Module first conducts a short homologous multi-task pre-training on an auxiliary graph, enabling prompt initialization to inherit optimization preferences associated with multiple pretext tasks. Built on the task-aware representations, the Structure-Prior Injection Module further extracts transferable global structural context from the auxiliary graph, converting it into layer-wise prompt vectors by aggregating structurally informative node embeddings. Extensive experiments on 6 mainstream benchmarks covering node and graph classification show that TPGC achieves consistently better performance under few-shot settings than state-of-the-art baselines, with fewer downstream tunable parameters and lower runtime. The code is available at https://github.com/Virgilqiu/TPGC

1. Introduction

Existing graph prompt learning methods commonly use random prompt initialization, leaving task relevance and graph structural information insufficiently encoded. TPGC addresses this with task-prior and structural-prior injection, and reports stronger few-shot performance with lower downstream cost.

  • Research gap: Random prompt initialization leaves prompts misaligned with downstream tasks and graph structural information.Existing methods lack an explicit mechanism to encode pre-training objectives and graph topology at initialization.
  • Proposed approach: TPGC introduces task-specific prompts with global context for multi-task graph pre-training.The method explicitly preserves task-specific prior and transferable structural knowledge during initialization.
  • Proposed approach: TPIM extracts task-specific prior through one epoch of homologous pre-training on a related auxiliary graph.This process injects information associated with multiple pre-training objectives before target-domain pre-training.
  • Proposed approach: SPIM complements TPIM by injecting transferable structural prior from the auxiliary graph’s global context into the prompt space.Together, the modules produce prompts that are task-specific and global-context-aware.
  • Results: TPGC consistently outperforms strong graph prompt learning baselines across few-shot settings while using fewer downstream tunable parameters and lower runtime.Experiments cover mainstream node-level and graph-level benchmark datasets and report robustness across few-shot settings.

2. Related Works

Graph pre-training seeks transferable knowledge, but multi-task objectives can interfere and transferred representations may misalign with downstream objectives. Related graph prompt learning work motivates parameter-efficient adaptation, while TPGC targets multi-task prompt initialization with task and structural guidance.

  • Graph pre-training: Graph pre-training reduces reliance on task-specific annotations by distilling transferable knowledge from unlabeled graphs.
  • Graph pre-training: Jointly optimized pretext tasks can emphasize different graph properties and produce inconsistent optimization directions.This creates a coordination challenge for multi-task graph pre-training and transfer across heterogeneous graph domains.
  • Graph prompt learning: Graph prompt learning introduces learnable context vectors to adapt pre-trained models without updating all model parameters.This parameter-efficient paradigm bridges graph pre-training and downstream adaptation.
  • Prompt initialization: Prior graph prompt methods leave initialization insufficiently explored for multi-task graph pre-training, where shared prompts must coordinate multiple objectives.Existing initialization studies mainly address single pretext-task settings or other modalities.
  • Prompt initialization: TPGC addresses this gap by combining task-specific prompts with global contextual guidance for stronger transfer of structural and semantic knowledge.

3. Proposed Methodology

TPGC initializes multi-task graph pre-training prompts by combining task-aware optimization preferences with transferable global structural context. Its three-stage pipeline transfers these initialized prompts into target pre-training and lightweight downstream adaptation.

  • Overall Workflow: TPGC has three stages: dual-prior prompt initialization, target-domain multi-task pre-training, and downstream prompt-based adaptation.Pre-trained prompts are frozen for downstream transfer, while task-specific prompts add downstream flexibility through branch fusion.
  • Task-Prior Injection Module: TPIM performs short homologous multi-task pre-training on a semantically and structurally related auxiliary graph to obtain task-aware initialization.The auxiliary process uses the same pretext-task setting as the target stage and updates the encoder to a task-aware auxiliary encoder.
  • Structure-Prior Injection Module: SPIM extracts layer-wise auxiliary node representations and converts task-related global structural context into initialized prompt vectors.It uses task-aware aggregation for local-global or augmentation-based tasks and global mean pooling for relation-oriented tasks.
  • Structure-Prior Injection Module: For each task and layer, SPIM selects top-k nodes by cosine similarity to a task-related global embedding and aggregates them with similarity-aware weights.The resulting vector is normalized and repeated across prompting layers to form the task-specific initialized prompt group.
  • Prompt Transfer: Input-layer prompts require learnable projection when auxiliary and target graphs have different feature dimensions, whereas hidden-layer prompts transfer directly.The final target prompt group therefore preserves transferred structural information while accommodating input-space dimensional differences.
  • Prompt Transfer: The dual-prior process replaces random prompt initialization with prompts encoding task-aware optimization preferences and transferable structural context.These prompts become the starting point for target-graph multi-task pre-training and are subsequently reused for downstream adaptation.

4.1. Experimental Settings

The evaluation covers six benchmarks spanning node and graph classification, compares TPGC with supervised, graph pre-training, and graph prompt-learning baselines, and uses few-shot settings. Experiments implement the baselines and TPGC with specified architectures, tasks, and training configurations.

  • Datasets: Six benchmarks cover citation, protein, enzyme, and molecular graphs across node-level and graph-level prediction tasks.The datasets include Cora, Citeseer, PROTEINS, ENZYMES, COX2, and BZR.
  • Evaluation protocol: Node and graph classification are evaluated under 1-shot and 5-shot settings.Table 1 reports node classification results, while Table 2 reports graph classification results.
  • Baselines: The evaluation compares TPGC with end-to-end GNN, graph pre-training, and graph prompt-learning baselines.The baseline families include GCN, GAT, DGI, GraphCL, GPPT, GraphPrompt, MultiGPrompt, and ProNoG.
  • Implementation Details: TPGC uses a 1-layer GCN with hidden dimension 256 and one prompt-initialization pre-training epoch.Its pre-training tasks are DGI, GraphCL, LP, and DSSL, with DSSL enabled only on ENZYMES.

4.2. Performance Comparisons

TPGC performs strongly across node and graph classification in few-shot settings, generally matching or surpassing prompt-learning baselines while maintaining favorable downstream efficiency. Its robustness persists as the shot count varies, and component studies evaluate the dual-prior initialization.

  • Node Classification: TPGC is best on three of four node-classification datasets in each shot setting and second-best on the remaining dataset.Against MultiGPrompt, it improves Cora by 0.84% in 1-shot and 3.20% in 5-shot settings.
  • Graph Classification: TPGC is best on BZR, PROTEINS, and ENZYMES and second on COX2 across 1-shot and 5-shot graph classification.On PROTEINS, it improves over MultiGPrompt by 0.72% and 0.53% in the 1-shot and 5-shot settings.
  • Robustness: For shot counts from 1 to 10, TPGC remains competitive, especially against prompt-learning baselines when m≤5.Performance generally improves for all methods as more labeled samples are provided, while TPGC retains advantages or highly competitive results.
  • Efficiency: TPGC and MultiGPrompt each use 522 downstream tunable parameters, whereas ProNoG uses 2564 and incurs heavier computation.The comparison is reported for ENZYMES under 1-shot evaluation with 50 downstream training epochs.
  • Ablation Study: Ablation studies compare full TPGC with variants removing components of its prompt-initialization strategy.Tables 4 and 5 assess key components on node and graph classification datasets.

4.3. Ablation Studies

Ablation studies show that TPGC’s task-prior and structure-prior modules are complementary: retaining both yields the most reliable prompt initialization across few-shot node and graph classification.

  • TPGC achieves the strongest or tied-strongest performance on almost all datasets under both 1-shot and 5-shot settings.
  • Removing either TPIM or SPIM generally degrades performance in both few-shot settings, confirming that both components contribute.
  • Removing TPIM shows that auxiliary-graph pre-training is important for injecting task-aware prior before target-domain optimization.
  • Removing SPIM weakens performance on several datasets, indicating that global averaging preserves informative structural patterns less effectively than top-k sampling.
  • Compared with Random Init., full TPGC produces more consistent gains across node and graph classification tasks.
  • TPIM and SPIM together provide the most effective prompt initialization strategy for multitask graph pre-training under different few-shot settings.

4.4. Visualization

Visualization and convergence analyses indicate that TPGC places prompts in more informative regions of graph representation space and enables faster, more stable pre-training than random initialization.

  • TPGC prompt vectors align better with graph representation structure, whereas random prompts occupy less informative regions.
  • Under random initialization, prompt vectors scatter through noisy regions and often remain far from meaningful node clusters.
  • TPGC initialization makes pre-training loss decrease more rapidly and reach a lower level earlier than random initialization.
  • TPGC helps pre-training converge faster and more stably.

5. Conclusion and Future Works

The paper concludes that TPGC improves prompt quality, few-shot performance, and optimization stability by combining task-aware prior knowledge with transferable structural context, while identifying heterophilous graphs as a remaining limitation.

  • TPGC initializes graph prompts with both task-aware prior knowledge and transferable structural context.
  • Experiments show that SPIM and TPIM enhance prompt quality, improve few-shot performance, and stabilize optimization across node and graph classification tasks.
  • The current design may remain limited on heterophilous graphs, motivating future work on prompt initialization for such scenarios.

CRediT Authorship Contribution Statement

The authors contributed across conceptualization, investigation, methodology, data curation, validation, writing, resources, supervision, and review.

  • Zhiyang Qiu contributed conceptualization, data curation, investigation, validation, and original-draft writing.
  • Yangtao Wang contributed conceptualization, investigation, methodology, resources, review and editing, and supervision.
  • Xiaocui Li, Yanzhao Xie, and Siyuan Chen contributed conceptualization and review and editing.
  • Wensheng Zhang contributed conceptualization and supervision.
Loading 2609.00047v1…