PaperlayerView exact revision on arXiv

Source-linked AI summary

All in One and One for All: A Simple yet Effective Method towards Cross-domain Graph Pretraining

Haihong Zhao, Aochuan Chen, Xiangguo Sun, Hong Cheng, Jia Li

arXiv:2402.09834v2cs.LG

TL;DR

Cross-domain graph pretraining often produces negative transfer because graph structures and feature semantics differ, a problem that is especially important in few-shot learning. GCOPE uses coordinators and feature-alignment mechanisms to unify multiple graph datasets during pretraining and transfer their knowledge. Experiments across homophilic and heterophilic datasets demonstrate the method’s efficacy, while its non-parametric SVD component limits generalizability.

  • Problem

    Cross-domain graph pretraining faces negative transfer because graph structures and feature semantics differ, limiting knowledge transfer in few-shot learning.

    Source: §1 · p. 2, §2 · p. 3

  • Method

    GCOPE uses virtual coordinators, feature projection, and a unified pretraining framework to connect disparate graphs and align their representations.

    Source: §1 · p. 2, §3.2.1 · p. 3

  • Results

    Experiments across multiple homophilic and heterophilic graph datasets demonstrate GCOPE’s efficacy in mitigating negative transfer.

    Source: §4.2 · p. 7, §6 · p. 9

  • Takeaways & Limitations

    GCOPE provides a unified graph pretraining approach that preserves multi-domain knowledge and supports transfer across graph datasets.

    Source: §1 · p. 2, §6 · p. 9

    The non-parametric nature of SVD may constrain GCOPE’s generalizability across diverse datasets.

    Source: §6 · p. 9

Abstract

from arXiv · show

Large Language Models (LLMs) have revolutionized the fields of computer vision (CV) and natural language processing (NLP). One of the most notable advancements of LLMs is that a single model is trained on vast and diverse datasets spanning multiple domains -- a paradigm we term `All in One'. This methodology empowers LLMs with super generalization capabilities, facilitating an encompassing comprehension of varied data distributions. Leveraging these capabilities, a single LLM demonstrates remarkable versatility across a variety of domains -- a paradigm we term `One for All'. However, applying this idea to the graph field remains a formidable challenge, with cross-domain pretraining often resulting in negative transfer. This issue is particularly important in few-shot learning scenarios, where the paucity of training data necessitates the incorporation of external knowledge sources. In response to this challenge, we propose a novel approach called Graph COordinators for PrEtraining (GCOPE), that harnesses the underlying commonalities across diverse graph datasets to enhance few-shot learning. Our novel methodology involves a unification framework that amalgamates disparate graph datasets during the pretraining phase to distill and transfer meaningful knowledge to target tasks. Extensive experiments across multiple graph datasets demonstrate the superior efficacy of our approach. By successfully leveraging the synergistic potential of multiple graph datasets for pretraining, our work stands as a pioneering contribution to the realm of graph foundational model.

1 INTRODUCTION

Graph pretraining seeks an “all in one” and “one for all” capability, but cross-domain transfer is hindered by structural and semantic disparities. GCOPE addresses this challenge with coordinators and a unified pretraining framework designed to preserve and transfer knowledge across graph datasets.

1 INTRODUCTION

  • Graph pretraining aims to extend the cross-domain generalization demonstrated by foundation models in computer vision and natural language processing.The paper frames this as combining diverse-domain knowledge during pretraining and applying it across downstream domains.Source: §1 · pp. 1–2
  • Structural diversity between homophilic and heterophilic graphs makes it difficult to identify common patterns while preserving dataset-specific information.The challenge is to leverage shared structure without losing the characteristics of individual graphs.Source: §1 · p. 2
  • Graph features also lack a universal semantic framework, limiting alignment methods that depend on textual descriptions or known feature meanings.Many graphs contain latent feature vectors whose individual dimensions are not interpretable enough to describe textually.Source: §1 · p. 2
  • GCOPE introduces virtual coordinators that dynamically bridge graph datasets, learn latent data-manipulation strategies, and promote unified representations.The coordinators are motivated by graph prompts and are used during pretraining to balance commonalities and differences across datasets.Source: §1 · p. 2
  • The framework unifies diverse graph structures, preserves multi-domain knowledge, supports multiple pretraining and transfer methods, and shows superior performance across graph datasets.Its transfer stage can use both fine-tuning and prompting, including in few-shot settings.Source: §1 · p. 2

2 MOTIVATION

The motivation section identifies negative transfer as a consequence of graph-domain differences and explains why simply combining source datasets is insufficient. It therefore targets structural isolation and feature misalignment through a dataset-independent multi-graph pretraining scheme.

2 MOTIVATION

  • The motivating transfer setting uses homophilic source graphs, Pubmed and Photos, and heterophilic target graphs, Wisconsin, Texas, Cornell, Chameleon, and Squirrel.Figure 1 presents this single-source cross-domain transfer arrangement.Source: Figure 1 · p. 2
  • Single-source cross-domain transfer negatively affects target tasks, while brute-force source-dataset combination still fails to improve performance.The experiments examine both single-source and multiple-source transfer scenarios.Source: §2 · p. 3
  • Different graph topologies create isolated dataset ecosystems, blocking information flow when graphs are merged for joint training.Structural disparity is especially pronounced between homophilic and heterophilic datasets.Source: §2 · p. 3
  • Heterogeneous, context-dependent graph attributes lack a straightforward common representation, making cross-dataset feature alignment difficult.Unlike text and images, graph attributes do not share a common reference framework.Source: §2 · p. 3
  • The paper formulates pretraining over M graph datasets and seeks a scheme independent of datasets, network architectures, and downstream tasks.The objective is to encode knowledge transferable to downstream graph classification tasks.Source: §2 · p. 3

3 METHOD

GCOPE unifies multi-domain graph pretraining by projecting features and connecting datasets through learnable coordinators. The framework supports cross-domain knowledge transfer, downstream adaptation, and efficient training while retaining graph-specific information.

3.1 Overview of Our Framework

  • GCOPE jointly pretrains graph models across multiple datasets using GraphCL or SimGRACE objectives and techniques designed for cross-domain alignment.The framework targets unified representations while remaining compatible with multiple graph pretraining approaches.Source: §3.1 · p. 3, §3.3 · p. 4

3.2.1 Feature Projection.

  • Feature projection maps graph features into a shared dimension, but projection alone does not resolve semantic alignment across datasets.SVD and attention mechanisms are given as representative projection operations.Source: §3.2.1 · p. 3

3.2.2 Graph Coordinators.

  • Coordinators are learnable virtual nodes that connect every node within their assigned graph and bridge coordinators across graphs.These connections create a joint adjacency structure and permit information flow between otherwise separate datasets.Source: §3.2.2 · p. 4
  • Jointly sampling nodes from diverse graphs encourages unified representations and supports generalization across domains.Aggregated batches expose the model to features from multiple datasets within one learning iteration.Source: §3.2.2 · p. 4

3.2.3 Why It Works?

  • The coordinator features can be interpreted as graph prompts that induce equivalent graph transformations, harmonizing structurally different graphs such as homophilic and heterophilic graphs.The paper connects this interpretation to the established prompt-as-transformation view [7] [36].Source: §3.2.3 · p. 4

3.3 Pretraining on Multi-domain Graphs

  • An auxiliary feature-reconstruction loss preserves salient graph information after feature projection while complementing the base pretraining objective.The reconstruction compares projected original node features with features reconstructed from embedded node representations.Source: §3.3 · pp. 4–5

3.4 Applying Knowledge to Downstream Data

  • Downstream tasks are converted to graph-level problems through induced subgraphs so their task space aligns with graph-level pretraining.The framework is evaluated with both conventional fine-tuning and graph prompting adaptation.Source: §3.4 · p. 5

3.5 Complexity Analysis

  • The additional time cost is O(LMd^2 + L(N + M)d + Md), and with M ≪ N it scales nearly linearly with the number of original nodes.Additional parameter complexity is O(Md_A), scaling linearly with the number of pretraining datasets.Source: §3.5 · p. 5

4 EXPERIMENTS

Experiments evaluate GCOPE across homophilic and heterophilic graphs, inter-coordinator connectivity, reconstruction weighting, graph-prompt transfer, and projection operations. GCOPE generally enables positive cross-domain transfer, with inter-coordinator edges, a small reconstruction weight, ProG prompting, and SVD projection supported by the reported findings.

4.1 Experimental Setup

  • The evaluation uses ten datasets spanning five homophilic and five heterophilic graphs, with nine source datasets and one target dataset in each 10-fold split.Features are projected to 100 dimensions with SVD, and results are reported for C-way-K-shot node classification transfer.Source: §4.1 · pp. 5–6

4.2 Cross-domain Transfer Performance with Few-shot Learning Settings (RQ1)

  • GCOPE with finetuning achieves positive transfer, significantly outperforming almost all baselines, with node-classification accuracy improvements ranging from 3.25% to 37.66%.Coordinators connect all nine source datasets into one cohesive joint distribution, facilitating shared information and graph representation learning.Source: §4.2 · p. 7

4.3 Interconnectivity Analysis (RQ2)

  • Inter-coordinator edges are essential and effective: GCOPE with such edges outperforms the otherwise matched variant without them.The comparison uses GCOPE/w and GCOPE/wo with FAGCN and GraphCL under the same partitioning strategy.Source: §4.3 · p. 7, Table 4 · p. 7

4.4 Reconstruction Analysis (RQ3)

  • GCOPE without reconstruction already outperforms supervised pretraining, while reconstruction performs best at 𝜆=0.2.Larger 𝜆 values degrade performance because reconstruction receives excessive emphasis; the reconstruction module aligns graph features across datasets.Source: §4.4 · p. 7

4.5 Transferring by Graph Prompt (RQ4)

  • GCOPE with ProG and with finetuning both outperform supervised and isolated-pretraining methods in downstream node classification.ProG achieves positive transfer with minimal tunable downstream parameters, although it is slightly below GCOPE with finetuning.Source: §4.5 · p. 7, Table 5 · p. 8

4.6 Compatibility Analysis of Projection Operation (RQ5)

  • SVD significantly outperforms attention-based projection across ten cross-domain datasets under C-way-1-shot transfer.Attention modules require 32,792,288 tuned parameters, whereas SVD is non-parametric; SVD is therefore used as GCOPE’s default projection.Source: §4.6 · pp. 7–8, Table 6 · p. 8

5 RELATED WORK

Graph pretraining methods use generation-based, auxiliary property-based, or contrast-based objectives, while graph transfer commonly relies on finetuning or graph prompts. Existing methods have not achieved pretraining across multiple graph domains, leaving source-data diversity and efficient transfer open challenges.

Graph Pretraining

  • Graph pretraining is categorized into generation-based, auxiliary property-based, and contrast-based methods.Generation-based methods reconstruct features or structure, auxiliary methods use additional supervision such as clustering labels, and contrastive methods optimize positive and negative embedding pairs.Source: §5 · p. 8
  • Existing graph pretraining methods have not achieved effective pretraining across multiple graph datasets from different domains.Consequently, pretraining efficacy remains constrained by the size and diversity of source data.Source: §5 · p. 8

Graph Transfer Learning

  • Finetuning adapts pretrained GNNs by updating the final layer or the entire model for a downstream task.The related-work passage describes finetuning as a longstanding transfer strategy with state-of-the-art performance.Source: §5 · p. 8
  • Graph prompts transfer pretrained knowledge through prompt tokens, token structures, and inserting patterns that fuse prompts with target data.Their minimal parameter footprint makes them efficient, although they are not guaranteed to outperform finetuning in every scenario.Source: §5 · pp. 8–9

6 CONCLUSION

The study analyzes negative transfer in graph learning and introduces GCOPE to mitigate it by connecting and aligning disparate graphs. Experiments across homophilic and heterophilic datasets demonstrate the method’s efficacy, while its non-parametric SVD component may constrain generalizability.

  • GCOPE mitigates negative transfer by amalgamating disparate graphs, establishing interconnections, and aligning their features.Source: §6 · p. 9
  • Experiments across homophilic and heterophilic datasets demonstrate the efficacy of the proposed method.Source: §6 · p. 9
  • The non-parametric nature of SVD may constrain GCOPE’s generalizability across diverse datasets.The authors propose a future learning-based feature projection pipeline to align graph features more robustly.Source: §6 · p. 9

A.1 Codes and Datasets

The appendix documents the released code and describes the citation, Amazon, WebKB, and Wikipedia graph datasets used in the study. These datasets pair network structures with domain-specific node features and labels.

  • Code for GCOPE is available through the project’s GitHub repository.Source: §A.1 · p. 10
  • Citation, Amazon, WebKB, and Wikipedia networks provide varied graph domains with dataset-specific nodes, edges, features, and labels.Source: §A.1 · p. 10
  • The datasets span publications, products, university webpages, and Wikipedia pages, supporting evaluation across heterogeneous application settings.The passage describes citation networks, co-purchase networks, hyperlink networks, and page-page networks.Source: §A.1 · p. 10

A.2 Parameter Analysis

The appendix compares parameter counts for GCOPE and baseline methods, as well as finetuning and prompting in downstream tasks. Table A8 reports this analysis using a specified backbone and pretraining setup.

  • Parameter counts are compared for two baselines and GCOPE, along with finetuning and prompting for downstream tasks.Source: §A.2 · p. 10
  • Table A8 reports the parameter analysis with FAGCN as backbone and Citeseer as the downstream dataset.Source: Table A8 · p. 10
  • IP and GCOPE are pretrained by GraphCL in the parameter-analysis setup.Source: Table A8 · p. 10

A.3 Cross-domain Transfer Performance with other C-way-K-shot settings

Additional 3/5-shot experiments show that GCOPE remains superior to competing baseline methods in cross-domain transfer, although its improvements decrease modestly as the shot count increases.

  • GCOPE maintains superiority over other baseline methods in the reported 3/5-shot cross-domain transfer scenarios.Source: §A.3 · p. 10
  • The appendix reports cross-domain transfer performance for 3/5-shot learning scenarios in Tables A1–A4.Source: §A.3 · p. 10
  • GCOPE’s improvements modestly decrease from 1-shot to 5-shot learning scenarios.Source: §A.3 · p. 10

A.4 Quantitative Analysis of Graph Coordinators

The appendix evaluates how coordinator count and inter-coordinator connectivity affect cross-domain few-shot transfer. Across the reported settings, GCOPE generally outperforms supervised and IP baselines, while GCOPE/f shows positive transfer across all datasets.

A.4 Quantitative Analysis of Graph Coordinators

  • GCOPE with varying numbers of graph coordinators generally outperforms both Supervised and IP methods across datasets.The analysis reports slight performance fluctuations across datasets, with results presented in Tables A5 and A6.Source: §A.4 · p. 10, Table A5 · p. 12, Table A6 · p. 12
  • The reported transfer results include C-way-3-shot and C-way-5-shot evaluations on both homophilic and heterophilic datasets, with IMP denoting average improvement over other methods.The tables report mean±std Acc/AUC/F1 and identify GraphCL and SimGRACE as GCL and Sim baselines.Source: Table A1 · p. 11, Table A2 · p. 11, Table A3 · p. 11

A.5 Dynamical Study of inter-coordinator edges

  • GCOPE/f demonstrates positive transfer across all datasets, whereas Texas outperforms supervision on Cora but is inferior to GCOPE/f and exhibits negative transfer on Citeseer.This comparison is reported for the study of inter-coordinator edges.Source: §A.5 · p. 11

A.4 Quantitative Analysis of Graph Coordinators

  • The coordinator-count experiments cover homophilic and heterophilic datasets under C-way-1-shot transfer, using one, three, or five coordinators per dataset.Tables A5 and A6 report mean±std Acc/AUC/F1 results for these settings.Source: §A.4 · p. 10, Table A5 · p. 12, Table A6 · p. 12

A.5 Dynamical Study of inter-coordinator edges

  • The connectivity study compares fully connected and dynamically connected inter-coordinator edges on two homophilic and two heterophilic datasets.Dynamic connections are defined by computing similarity between inter-dataset coordinators.Source: §A.5 · p. 11, Table A7 · p. 12
Loading 2402.09834v2…