PaperlayerView exact revision on arXiv

Source-linked AI summary

SAMGPT: Text-free Graph Foundation Model for Multi-domain Pre-training and Cross-domain Adaptation

Xingtong Yu, Zechuan Gong, Chang Zhou, Yuan Fang, Hui Zhang

arXiv:2502.05424v2cs.CLcs.AI

TL;DR

Existing graph methods struggle to pre-train across structurally different domains and adapt text-free models to unseen targets. SAMGPT aligns multi-domain structure with structure tokens and dual prompts, and experiments on seven benchmark datasets show it significantly outperforms state-of-the-art baselines. The study focuses on few-shot node and graph classification, especially when m ≤ 5.

  • Problem

    Graph foundation models need to learn from multiple source domains and adapt to unseen targets, but existing text-free methods generally align features while neglecting structural differences.

  • Method

    SAMGPT uses structure tokens to harmonize source-domain structural distributions during pre-training and holistic and specific prompts to adapt unified and domain-specific knowledge.

  • Results

    SAMGPT significantly outperforms various state-of-the-art baseline methods across experiments on seven benchmark datasets.

  • Takeaways & Limitations

    SAMGPT supports text-free multi-domain graph pre-training and cross-domain adaptation by tailoring holistic and domain-specific structural knowledge to target domains.

  • Takeaways & Limitations

    The evaluation focuses on low-shot settings, typically m ≤ 5; with more labeled data, conventional fine-tuning or supervised methods may become sufficient.

Abstract

from arXiv · show

Graphs are able to model interconnected entities in many online services, supporting a wide range of applications on the Web. This raises an important question: How can we train a graph foundational model on multiple source domains and adapt to an unseen target domain? A major obstacle is that graphs from different domains often exhibit divergent characteristics. Some studies leverage large language models to align multiple domains based on textual descriptions associated with the graphs, limiting their applicability to text-attributed graphs. For text-free graphs, a few recent works attempt to align different feature distributions across domains, while generally neglecting structural differences. In this work, we propose a novel Structure Alignment framework for text-free Multi-domain Graph Pre-Training and cross-domain adaptation (SAMGPT). It is designed to learn multi-domain knowledge from graphs originating in multiple source domains, which can then be adapted to address applications in an unseen target domain. Specifically, we introduce a set of structure tokens to harmonize structure-based aggregation across source domains during the pre-training phase. Next, for cross-domain adaptation, we design dual prompts, namely, holistic prompts and specific prompts, which adapt unified multi-domain structural knowledge and fine-grained, domain-specific information, respectively, to a target domain. Finally, we conduct comprehensive experiments on seven public datasets to evaluate and analyze the effectiveness of SAMGPT.

1 Introduction

The paper asks how to build a universal graph model from multiple graph domains and adapt it to unseen targets despite substantial structural variation. SAMGPT addresses this with structure alignment during pre-training and dual prompts during cross-domain adaptation.

  • Motivation: Web graphs support diverse applications, motivating a universal graph model trained across multiple domains.The paper identifies social network analysis, Web mining, and recommendation systems as examples.
  • Research gap: Single-domain graph pre-training usually reuses graphs from the same dataset, limiting universal multi-domain modeling.Existing approaches learn from unlabeled graphs and adapt through fine-tuning or prompt learning, but typically assume matching pre-training and downstream domains.
  • Research gap: Graphs from different datasets have distinct structural characteristics, making multi-domain integration and cross-domain adaptation challenging.The paper contrasts social, citation, and e-commerce graph patterns and notes that prior single-source adaptation methods do not exploit multiple source domains.
  • Approach: SAMGPT uses learnable structure tokens to align structural distributions across source domains during pre-training.The tokens capture domain-specific structural patterns while harmonizing structure-based aggregation.
  • Approach: SAMGPT uses holistic and specific prompts to adapt unified multi-domain knowledge and fine-grained source-domain information to a target domain.Holistic prompts align the target with unified knowledge, while specific prompts learn mixtures of source-domain structure tokens.

2 Related Work

Related graph pre-training methods use self-supervised learning and adaptation techniques, but multi-domain approaches generally align features while neglecting structural variance.

  • Graph pre-training: Graph pre-training extracts inherent graph properties through generative or contrastive self-supervised learning.Pre-trained models are later used for downstream tasks through fine-tuning or parameter-efficient prompt-based adaptation.
  • Multi-domain alignment: Existing multi-domain methods align feature dimensions, semantics, or distributions across graph domains.GCOPE uses domain-specific virtual nodes, while MDGPT uses domain-specific tokens to align feature semantics.
  • Research gap: These approaches do not account for structural variance across different domains.The related-work discussion motivates SAMGPT’s focus on structure alignment for text-free graphs.

3 Preliminaries

The paper formalizes graph encoding, multi-domain feature alignment, cross-domain adaptation, and few-shot classification. Its scope emphasizes low-shot downstream tasks where labeled data are limited.

  • Graph representation: A graph is represented as G = (V, E, X), with nodes, edges, and a node feature matrix.A collection of graphs is denoted as G, and each row of X represents a node feature vector.
  • Graph encoder: Message-passing GNNs update node embeddings by aggregating features or embeddings from neighboring nodes across layers.The resulting embeddings reflect both input features and graph structure.
  • Multi-domain pre-training: Multi-domain pre-training uses unlabeled graph-domain pairs and aligns feature dimensions and semantic spaces across source domains.Dimension alignment maps domain-specific feature dimensions to a common dimension, followed by feature alignment before graph-encoder pre-training.
  • Cross-domain adaptation: Cross-domain adaptation targets graphs from an unseen domain whose feature characteristics may differ from those of the source domains.The target domain is distinct from every source domain, and feature adaptation uses the pre-trained graph encoder.
  • Few-shot classification: The downstream scope is few-shot node and graph classification with only m labeled examples per class.The paper focuses on low-shot settings such as m ≤ 5, reflecting cases where labels are expensive or difficult to obtain.

4 Proposed Approach: SAMGPT

SAMGPT combines multi-domain graph pre-training with cross-domain adaptation by aligning both feature and structural information. Structure tokens harmonize source-domain aggregation, while dual prompts transfer holistic and domain-specific structural knowledge to target tasks.

  • SAMGPT has two phases: multi-domain pre-training and cross-domain adaptation.The framework is presented as an overview followed by these two stages.
  • Multi-domain graph pre-training: During pre-training, feature distributions are aligned and domain-specific structure tokens modify structure-based aggregation at every encoder layer.The structure-token-enhanced encoder is then trained with a self-supervised loss.
  • Multi-domain graph pre-training: Layer-wise structural patterns can differ across domains, so each source domain receives its own structure-token set to produce structure-aligned embeddings.The structure-aligned outputs are stacked across domains and fused with feature-aligned embeddings.
  • Cross-domain structure adaptation: Cross-domain adaptation uses holistic prompts for shared source knowledge and specific prompts that mix source-domain structure tokens for fine-grained target-domain information.Specific prompts are injected into encoder layers, with learnable coefficients determining the mixture while the pre-trained encoder remains frozen.
  • Cross-domain structure adaptation: Prompt tuning combines holistic and specific structural representations with feature adaptation for downstream node and graph classification.The downstream objective uses the same subgraph-similarity task template as pre-training and optimizes the cross-domain adaptation loss.

5 Experiments

The experiments evaluate SAMGPT on seven benchmark graph domains, few-shot node and graph classification, source-domain ablations, component ablations, and varying homophily patterns. Across these settings, SAMGPT consistently outperforms the compared methods, while its structure alignment and dual prompts support gains from additional source domains and diverse graph structures.

  • Experimental Setup: Experiments use seven benchmark datasets as distinct domains, with each domain alternately serving as the target while the remaining six provide source domains.The setup evaluates cross-domain adaptation across citation, e-commerce, Web, and social-network graphs.
  • Performance Evaluation: SAMGPT achieves outstanding one-shot performance for both node and graph classification across various target domains.The comparison includes graph pre-training, cross-domain, and end-to-end supervised baselines.
  • Data Ablation: SAMGPT remains superior across different numbers of source domains, whereas GraphPrompt and GCOPE often worsen as datasets are added.SAMGPT instead improves consistently when more source domains are incorporated, supporting the value of structure alignment and adaptation.
  • Model Ablation: Removing structure tokens, specific prompts, or holistic prompts reduces performance, showing that all three components contribute to SAMGPT’s results.Structure tokens align multi-domain structural knowledge, while specific and holistic prompts incorporate domain-specific and holistic information.
  • Homophily Sensitivity: SAMGPT consistently surpasses GraphPrompt and GCOPE on both homophilic and heterophilic graphs.The analysis covers one-shot node classification on Cora, Photo, Facebook, Chameleon, Cornell, and Squirrel.

6 Conclusions

The paper concludes that SAMGPT is a text-free graph foundation model for multi-domain pre-training and cross-domain adaptation. It uses structure tokens and dual prompts, and experiments on seven benchmark datasets show significant gains over state-of-the-art baselines.

  • Conclusion: SAMGPT supports multi-domain graph pre-training and cross-domain adaptation for text-free graphs.The framework is designed to learn from multiple source domains and adapt to an unseen target domain.
  • Conclusion: Structure tokens harmonize structural distributions across source domains, while dual prompts tailor holistic and domain-specific structural knowledge to the target domain.These are the framework’s two principal stages of structural alignment and adaptation.
  • Conclusion: Experiments on seven benchmark datasets show that SAMGPT significantly outperforms various state-of-the-art baseline methods.The conclusion summarizes the paper’s reported empirical evaluation.

Appendices

SAMGPT consists of multi-domain graph pretraining followed by downstream adaptation. Pretraining aligns feature dimensions, feature semantics, and structural information, while adaptation applies feature adaptation and dual prompts to the target domain.

  • Pipeline: SAMGPT has two stages: multi-domain graph pretraining and downstream adaptation.The stages separate source-domain knowledge learning from target-domain task adaptation.
  • Multi-domain Pretraining: During pretraining, DAL aligns feature dimensions, feature alignment unifies feature semantic spaces, and source-specific structure tokens align structural information.The aligned feature and structure embeddings are then fused for pretraining optimization.
  • Downstream Adaptation: During cross-domain adaptation, target features are aligned and adapted before dual prompts modify the pretrained encoder’s structural aggregation.Holistic prompts adapt holistic knowledge, while specific prompts adapt source-domain-specific structural knowledge.

B Complexity Analysis

The adaptation procedure adds prompt-related computation to the pretrained GNN, but the encoder’s neighborhood aggregation remains the dominant complexity. The algorithm combines feature adaptation, holistic and specific prompts, embedding fusion, prototype updates, and downstream optimization.

  • Adaptation Procedure: For a target graph, structural adaptation injects holistic and specific prompts into the pretrained GNN to modify node encoding.This defines the role of dual prompts in the target-domain pipeline.
  • Complexity: The pretrained GNN has complexity O(n^L · |V_T|), while holistic prompts add O(L · |V_T|) and specific prompts add O(L · K).Here n is the maximum neighborhood size, L is the number of layers, |V_T| is the target-node count, and K is the number of source domains.
  • Complexity: Because n^L · |V_T| typically exceeds L · |V_T| + L · K, encoding by the pretrained GNN dominates the overall complexity.The structural adaptation therefore adds only marginal overhead under the stated condition.
  • Algorithm 1: Algorithm 1 aligns target features, generates feature-adapted embeddings, applies dual prompts layer by layer, fuses embeddings, updates class prototypes, and optimizes downstream parameters.The procedure outputs optimized holistic prompts, coefficients, and feature-adaptation parameters.

C Further Descriptions of Datasets

This section describes benchmark datasets and the baseline models used for comparison. The baselines include end-to-end graph neural networks and graph pre-training methods.

  • Datasets: The benchmark descriptions include Cora and Citeseer, with publication networks, class labels, edges, and node-feature representations.Cora contains 2,708 publications, seven classes, 5,429 edges, and 1,433-word binary features; the supplied passage begins the Citeseer description.
  • Baseline models: End-to-end baselines include GCN, which uses mean neighborhood aggregation, and GAT, which assigns varying attention weights to neighbors.These mechanisms determine how adjacent-node information contributes to learned node representations.
  • Baseline models: Graph pre-training baselines use mutual-information maximization, node–graph embedding alignment, graph augmentations, or link prediction for self-supervision.The listed methods are DGI, InfoGraph, GraphCL, and GPPT, while GPF provides universal prompt-based tuning for pre-trained graph models.

E Implementation Details

The experiments use official baseline implementations with settings recommended by their original works, while SAMGPT uses a fixed GCN backbone and unified feature dimension. The section also specifies differing architectures and adaptation choices across baselines.

  • Baseline settings: Official codes were used for open-source baselines, and each model was tuned according to settings recommended in its original work.This procedure was intended to achieve optimal baseline performance under the reported configurations.
  • Baseline settings: GCN uses three layers with hidden dimension 256, while GAT uses two layers with dimension 64 and eight first-layer attention heads.These are the specified end-to-end GNN configurations.
  • Pre-training baselines: DGI, InfoGraph, and GraphCL use GCN backbones with hidden dimension 256, while GraphCL applies edge dropping with augmentation ratio 0.2.DGI uses one layer and PReLU; InfoGraph uses three layers; GraphCL uses one layer.
  • Prompt-based baselines: GPPT uses a two-layer GraphSAGE with mean aggregation, while GraphPrompt and GPF use three- and five-layer GCN backbones, respectively.The supplied settings assign hidden dimension 256 to these models.
  • Adaptation and SAMGPT: GCOPE uses a two-layer GCN with hidden dimension 100 and fine-tuning for downstream adaptation, whereas other baselines use unified feature dimension 50.SAMGPT uses a three-layer GCN with hidden dimension 256 and unified feature dimension 50.

F Details about Heterophilic Datasets

The section introduces heterophilic datasets used to test robustness across varying homophily ratios, including Chameleon and Cornell. It also reports sensitivity analyses showing that balanced alignment and prompt aggregation perform robustly.

  • Dataset scope: SAMGPT is evaluated on homophilic and heterophilic datasets to assess robustness across graphs with varying homophily ratios.The heterophilic experiments are reported in Section 5.4.
  • Chameleon: Chameleon is a Wikipedia-based network with 2,277 pages, five traffic-based groups, 36,101 edges, and noun-derived node features.Its pages are categorized according to average monthly traffic.
  • Cornell: Cornell is a webpage network with 183 nodes and 295 hyperlink edges, using webpage bag-of-words features and five manually assigned categories.The supplied description identifies student, project, course, and staff among the categories.
  • Sensitivity analysis: Accuracy peaks at α=1 as α increases, while β follows a similar trend, supporting balanced feature–structure and holistic–specific adaptation.The experiments vary α and β for 1-shot node and graph classification on Cora, Photo, and Facebook; error bars show standard deviation.

H Data Ethics Statement

The study uses publicly available datasets under their usage terms and conditions. It reports no personally identifiable information and no human or animal subjects.

  • Data sources: The experiments use only publicly available datasets.The listed datasets include Cora, Citeseer, Pubmed, Photo, Computers, Facebook, LastFM, Chameleon, Cornell, and Squirrel.
  • Data governance: The datasets were used in accordance with applicable usage terms and conditions.The statement qualifies this compliance with “if any.”
  • Ethics: No personally identifiable information was used, and the research involved no human or animal subjects.
Loading 2502.05424v2…