PaperlayerView exact revision on arXiv

Source-linked AI summary

Text-Free Multi-domain Graph Pre-training: Toward Graph Foundation Models

Xingtong Yu, Chang Zhou, Yuan Fang, Xinming Zhang

arXiv:2405.13934v4cs.LG

TL;DR

Graph foundation models trained across domains could support graph tasks in varied fields, but domain differences complicate integration and existing language-mediated approaches are limited to text-attributed graphs. MDGPT uses domain tokens and dual prompts to align source and target-domain knowledge without relying on text. Experiments on six public datasets show that MDGPT significantly outperforms state-of-the-art baselines.

  • Problem

    Graph pre-training largely uses same-domain data, while integrating diverse domains is challenging and language-mediated multi-domain methods are limited to text-attributed graphs.

    Source: §1 · p. 1

  • Method

    MDGPT uses learnable domain tokens to align source-domain features and unifying and mixing prompts to adapt target domains with unified and domain-specific knowledge.

    Source: §1 · p. 2

  • Results

    Across six public datasets, MDGPT significantly outperforms various state-of-the-art baselines.

    Source: §6 · p. 9

  • Takeaways & Limitations

    MDGPT provides a text-free multi-domain pre-training and prompting framework as a plausible path toward graph foundation models.

    Source: §1 · p. 2

    The study focuses on text-free graphs; integrating text-free and text-attributed graphs across domains remains a future direction.

    Source: §6 · p. 9

Abstract

from arXiv · show

Given the ubiquity of graph data, it is intriguing to ask: Is it possible to train a graph foundation model on a broad range of graph data across diverse domains? A major hurdle toward this goal lies in the fact that graphs from different domains often exhibit profoundly divergent characteristics. Although there have been some initial efforts in integrating multi-domain graphs for pre-training, they primarily rely on textual descriptions to align the graphs, limiting their application to text-attributed graphs. Moreover, different source domains may conflict or interfere with each other, and their relevance to the target domain can vary significantly. To address these issues, we propose MDGPT, a text free Multi-Domain Graph Pre-Training and adaptation framework designed to exploit multi-domain knowledge for graph learning. First, we propose a set of domain tokens to to align features across source domains for synergistic pre-training. Second, we propose a dual prompts, consisting of a unifying prompt and a mixing prompt, to further adapt the target domain with unified multi-domain knowledge and a tailored mixture of domain-specific knowledge. Finally, we conduct extensive experiments involving six public datasets to evaluate and analyze MDGPT, which outperforms prior art by up to 37.9%.

1 Introduction

The introduction motivates graph pre-training across domains as a route toward a universal graph model, but notes that domain differences complicate knowledge integration and transfer. MDGPT addresses this with domain tokens for source-domain alignment and dual prompts for downstream adaptation.

1 Introduction

  • Graph pre-training on data from many domains could support a universal model adapted to graph tasks across downstream fields.Graph data span e-commerce, citation, and social applications.Source: §1 · p. 1
  • Existing graph pre-training generally uses graphs from one domain, while multi-domain graphs pose alignment challenges and text-mediated approaches are limited to text-attributed graphs.Different domains can have divergent feature dimensions and semantics, creating conflicts when their graphs are integrated.Source: §1 · pp. 1–2
  • MDGPT uses learnable domain-specific tokens to bridge feature semantics across source domains for synergistic multi-domain pre-training.The introduction describes each token as associated with a source domain.Source: §1 · p. 2
  • For downstream adaptation, MDGPT combines a unifying prompt for global pre-trained knowledge with a mixing prompt for tailored source-domain knowledge.The introduction also lists text-free multi-domain pre-training and prompting as the framework's contributions.Source: §1 · p. 2

2 Related Work

Prior graph pre-training commonly assumes similar source and target domains, while cross-domain approaches often use a single source domain. Multi-domain methods include language-mediated approaches limited to text-attributed graphs and virtual-node approaches that do not explicitly align features.

2 Related Work

  • Conventional graph pre-training assumes that pre-training and downstream graphs come from the same or similar domains, limiting cross-domain generalization.Examples include subgraphs of a larger graph and collections of graphs from the same field.Source: §2 · pp. 2–3
  • Cross-domain methods transfer knowledge from a single source domain, and some are tailored to particular tasks or domains or require domain expertise [8] [38].These approaches therefore do not use broad multi-domain knowledge.Source: §2 · p. 3
  • OFA and HiGPT use natural language to process multi-domain node features, while GCOPE connects domains with virtual nodes without explicitly aligning features [16] [33].GCOPE also transfers unified pre-trained knowledge while overlooking source-domain-specific knowledge.Source: §2 · p. 3

3 Preliminaries

The preliminaries define graphs and multi-domain pre-training data, then specify downstream few-shot adaptation to target domains that may or may not appear during pre-training.

3 Preliminaries

  • A graph is represented by nodes, edges, and a node feature matrix, with each node assigned a feature vector.A set of graphs is denoted by G.Source: §3 · p. 3
  • Multi-domain pre-training uses source graphs associated with domains whose node and edge distributions, feature dimensions, and feature semantics can differ.More than one graph may belong to the same source domain.Source: §3 · p. 3
  • Downstream few-shot adaptation considers target domains that may be seen or unseen during pre-training, with each task requiring only a small number of task-specific labels.The section identifies node and graph classification as downstream task types.Source: §3 · p. 3

4 Proposed Approach

MDGPT aligns and pre-trains graph features across source domains, then adapts target tasks with prompts that combine unified and domain-specific knowledge. Its framework uses lightweight prompting around a frozen pre-trained graph encoder.

4.1 Overall framework

  • Pre-training aligns source dimensions, adds domain tokens, and trains an encoder with self-supervision; adaptation prompts target features before the pre-trained encoder.The framework uses a unifying prompt and a mixing prompt for target adaptation.Source: §4.1 · p. 3

4.2 Multi-domain pre-training

  • Domain-specific SVD alignment gives source graphs a shared feature dimension, but their semantic spaces can still differ and interfere.The paper uses SVD for dimension alignment; equal dimensions alone do not unify feature semantics.Source: §4.2 · pp. 3–4
  • Learnable domain tokens adjust aligned features, creating unified feature spaces that support synergistic integration and preserve domain-specific knowledge.The tokens are vectors with the aligned feature dimension, applied to features from their corresponding domains.Source: §4.2 · p. 4
  • Pre-training uses sampled positive-edge and negative-node triplets with a universal task template, optimizing a loss to produce encoder weights and domain tokens.The loss uses cosine similarity with a temperature hyperparameter.Source: §4.2 · p. 4

4.3 Downstream adaptation

  • A unifying vector carries global prior knowledge, while a learnable mixture of pre-trained domain tokens weights source-specific knowledge for target adaptation.The mixing coefficients are tuned to adapt the target to source-domain knowledge.Source: §4.3 · p. 5
  • Prompts modify target features in separate encoder passes; only the unifying prompt and mixing coefficients are tuned, keeping pre-trained weights frozen and reducing updated parameters.The paper presents this parameter-efficient setup as especially amenable to few-shot settings.Source: §4.3 · p. 5

5 Experiments

Experiments evaluate MDGPT on five benchmark graphs, using few-shot node and graph classification to test transfer to unseen and seen domains. Results cover comparative performance, source-domain and component ablations, feature-dimension sensitivity, homophily, and adaptation parameter efficiency.

5.2 Few-shot performance evaluation

  • MDGPT beats the best competitor by 3.5–37.9% on unseen-domain one-shot node classification and 4.3–13.9% on graph classification.The node- and graph-classification results are reported in Tables 1 and 2, respectively.Source: Table 1 · p. 6, Table 2 · p. 6, §5.2 · p. 6
  • In low-shot settings with m≤5, MDGPT significantly surpasses all baselines; as shots increase, all methods improve and MDGPT remains competitive or better.Figure 3 examines the effect of shot count on node classification across three target domains.Source: §5.2 · p. 7, Figure 3 · p. 7

5.3 Ablation Study

  • As source domains grow from one to four, several baselines worsen on particular additions, while MDGPT consistently improves.DGI worsens when Photo and Computers are added, GraphPrompt when Computers is added, and GCOPE when PubMed is added.Source: §5.3 · p. 7, Figure 4 · p. 8
  • Ablations indicate that domain tokens and mixing prompts each contribute to performance, the unifying prompt also helps, and using both prompts is beneficial.Variant 3 with domain tokens outperforms Variants 1 and 2; Variant 4 exceeds Variant 3, and Variant 2 has higher accuracy than Variant 1.Source: §5.3 · p. 7, Table 4 · p. 8

5.4 Hyperparameter Analysis

  • Accuracy initially rises as aligned feature dimension increases, peaks around ˜d=50, then gradually declines; ˜d=50 is robust across most datasets and tasks.The reported pattern applies to both node and graph classification; the authors attribute the decline to an amplified semantic gap across domains.Source: §5.4 · p. 7, Figure 5 · p. 8

5.5 Homophily Sensitivity

  • Across homophilic and heterophilic source and target graphs, MDGPT leverages multi-domain knowledge and consistently outperforms GCOPE.The evaluated datasets include Cora, Citeseer, Pubmed, Texas, Wisconsin, Cornell, Squirrel, and Chameleon.Source: §5.5 · pp. 7–8, Table 3 · p. 7

5.6 Node Classification on Seen Domains

  • On seen target domains, one-shot node-classification results follow the same trend as those on unseen domains.Cora and Citeseer are evaluated as seen targets after pre-training on five source domains.Source: §5.6 · p. 8, Table 5 · p. 8

5.7 Parameters Efficiency

  • MDGPT has the fewest tunable parameters during adaptation and significantly outperforms all baselines.Only the dual prompts are updated; the unifying prompt dimension matches unified-feature dimension, while the mixing prompt dimension matches the number of source domains.Source: §5.7 · p. 8, Table 6 · p. 8

6 Conclusions and Future Work

MDGPT uses domain tokens and dual prompts to combine unified and domain-specific knowledge for graph learning, and experiments on six public datasets show it outperforms state-of-the-art baselines. The paper focuses on text-free graphs and identifies combining them with text-attributed graphs as future work.

6 Conclusions and Future Work

  • MDGPT uses domain tokens to unify source-domain semantic spaces and dual prompts to align target domains with unified and tailored domain-specific knowledge.The authors report experiments on six public datasets and state that MDGPT significantly outperforms state-of-the-art baselines.Source: §6 · p. 9
  • The experiments cover six public datasets and show MDGPT significantly outperforms various state-of-the-art baselines.Source: §6 · p. 9
  • The study focuses on text-free graphs; integrating text-free and text-attributed graphs across domains is proposed as future work.The authors note that text-attributed graphs provide auxiliary knowledge and a gateway to connect with large language models.Source: §6 · p. 9

A Further Descriptions of Datasets

The appendix describes six datasets spanning citation, product co-purchasing, and social-interaction graphs. Their node or item features and labels vary by dataset.

A Further Descriptions of Datasets

B Further Descriptions of Baselines

The appendix groups baselines into end-to-end GNNs, graph pre-training models, cross-domain methods, prompting approaches, and multi-domain pre-training. It briefly describes the methods represented in each group.

B Further Descriptions of Baselines

C Implementation Details

The appendix specifies experimental settings, sampling procedures, and model configurations. It evaluates adaptation on held-out domains, including Reddit as a cross-domain target, and reports results averaged over repeated few-shot tasks.

C Implementation Details

  • Experiments use Adam and ran on Ubuntu 18.04.5 with a GeForce RTX 4090 GPU and Intel Xeon Platinum 8352V CPU.The listed Linux version is 5.15.0-78-generic.Source: C Implementation Details · p. 12
  • For transfer experiments, four of Cora, Citeseer, PubMed, Amazon Photo, and Amazon Computers are used for pre-training and the remainder for adaptation; Reddit is also tested as a held-out target.The passage states that all pre-training methods use the specified multi-domain datasets.Source: C Implementation Details · p. 12
  • Downstream node and graph classification use 1-shot tasks and vary the shot count from 1 to 10; each setting averages 500 outcomes from 100 sampled tasks and five random seeds.Source: C Implementation Details · p. 12
  • Baseline comparisons use official code where available, reproduce GCOPE, and tune models using literature-recommended settings.Source: C Implementation Details · p. 12
  • GCN uses three layers and 256 hidden dimensions; GAT uses two layers, 64 hidden dimensions, and eight attention heads in its first layer.Source: C Implementation Details · p. 12
  • DGI and GraphCL use one-layer GCNs, while InfoGraph uses a three-layer GCN; GraphCL applies edge dropping at a default ratio of 0.2.Source: C Implementation Details · p. 12
  • Baseline configurations include 3-layer GCNs for Hassani and GraphPrompt, a 2-layer GraphSAGE for GPPT, and a 5-layer GCN for GPF, all with 256 hidden dimensions.Source: C Implementation Details · p. 12
  • MDGPT uses a 3-layer GCN with 256 hidden dimensions and 50 unified feature dimensions; most baselines also use 50, while GCOPE uses 100.Source: C Implementation Details · p. 12
Loading 2405.13934v4…