A Survey on Self-Supervised Graph Foundation Models: Knowledge-Based Perspective

Motivation

• Self-supervised pre-training and downstream tuning are presented as promising routes toward graph foundation models that generalize across application tasks.Self-supervision targets graph knowledge rather than labels tied to a specific task space.Source: §1 · p. 1

Limitations of Existing Surveys

• Existing surveys lack recent coverage, including masked graph autoencoders and graph manifold learning, and often focus on applications rather than graph foundation models.Source: §1 · p. 2
• Architecture-based surveys use broad generative–contrastive categories that do not distinguish whether methods exploit links, clusters, or other graph knowledge.The same knowledge type can appear in different learning frameworks.Source: §1 · p. 2
• Existing surveys are also limited to particular backbones, while language-model-centric views can overlook structural patterns and leave the ideal GFM architecture unresolved.Source: §1 · p. 2

Our Contributions

• The proposed taxonomy organizes pre-training by microscopic node and link knowledge, mesoscopic context and cluster knowledge, and macroscopic whole-graph structure and manifolds.It is intended to analyze GNNs, graph Transformers, and graph language models within one perspective.Source: §1 · p. 2
• The survey analyzes 9 knowledge categories and 300 references, with more than 500 related papers collated for comparison and future GFM development.Source: §1 · p. 2

Basic Concepts

• A graph is G = (V, E), with adjacency matrix A encoding links, node features X for attributed graphs, and neighborhood N_i defined by adjacent nodes.Source: §2 · p. 2
• A graph model encodes graph data as Z = f(G; Θ), while a graph foundation model is pre-trained on varied unsupervised graph data for diverse graph-related tasks.Source: §2 · p. 2

Pre-training Task

• A pretext is a self-supervised task used during graph-model pre-training, and it must use no manual labels while improving one or more downstream tasks.Source: §2 · p. 2

Downstream Adaptation

• During downstream adaptation, optional branches, pre-trained parameter tuning, and task-specific labels determine whether the process includes supervised fine-tuning.Source: §2 · p. 2

3.1 Node Features

• Node feature tasks reconstruct or denoise attributes, discriminate node instances or representation dimensions, and target semantic information from attributed graphs.Feature prediction includes masked autoencoding, autoregressive masking, and noise recovery, while discrimination promotes perturbation-invariant or augmentation-invariant representations.Source: §3.1 · pp. 3–4
• Node instance discrimination maximizes similarity between corresponding nodes across perturbed graph views and minimizes similarity to sampled negatives, learning general representations.It is also used as a self-supervised tuning strategy spanning link prediction, node classification, and graph classification.Source: §3.1 · pp. 3–4
• Dimension discrimination contrasts representation columns by maximizing correspondence between matching dimensions and minimizing correlation between different dimensions.Its benefits may diminish for smaller representation dimensions, and regularization seeks to avoid global or local representation collapse.Source: §3.1 · p. 4

3 MICROSCOPIC PRE-TRAINING TASKS

• Microscopic task efficiency varies substantially: feature prediction and bootstrapping scale as O(nd), whereas InfoNCE node discrimination can reach O(n^2d) and dimension discrimination costs O(nd^2).Link prediction requires O(d|E|) similarity computation and adjacency memory of O(n^2) when dense or O(|E|) when sparse.Source: §3.1 · p. 4, §3.3 · p. 5

3.2 Node Properties

• Node-property tasks predict centrality, clustering coefficients, or node order to encode structural roles and local relationships.These properties can be task-specific and may fail to distinguish graphs sharing the same degree distribution.Source: §3.2 · p. 5

3.3 Links

• Link tasks predict, denoise, or reconstruct edges and edge features, using decoders, variational latent spaces, masking, or contrastive losses.Link prediction captures relationships omitted by node-feature methods but can overemphasize local structure and therefore benefits from compatibility with feature semantics and higher-order structure.Source: §3.3 · p. 5

4 MESOSCOPIC PRE-TRAINING TASKS

• Mesoscopic pretexts capture local-range properties while encoding higher-order information and long-range dependencies, but clustering/partitioning costs can become prohibitive for large networks.All-pairs shortest-path computation can be expensive: Floyd-Warshall takes O(n3) time and O(n2) memory, while vanilla spectral clustering takes O(n3) time.Source: §4 · p. 5, §4.2 · p. 7, §4.4 · p. 8

4.1 Context

• Context tasks learn local higher-order structure by contrasting nodes with neighboring or sampled contextual subgraphs, often using homophily-based positives and negative sampling.GraphSAGE uses neighborhood subgraphs, while later methods use k-hop neighborhoods, personalized PageRank, structural and feature similarity, or multi-scale message-passing views.Source: §4.1 · pp. 5–6
• Context learning can add limited value to message-passing GNNs because those models already extract local structure, whereas it may better benefit models emphasizing global interactions.Cluster methods commonly assume non-overlapping communities, and clustering or partitioning can become costly on large networks.Source: §4.1 · p. 6, §4.4 · p. 8

4.2 Long-range Similarities

• Long-range similarity tasks predict topological or feature-based relationships between nonneighboring nodes and align similarity graphs with the original graph.They can bridge disconnected components and combine feature semantics with graph structure, but feature and structural similarities may conflict.Source: §4.2 · pp. 6–7

4.3 Motifs

• Motif tasks learn representations of recurring subgraphs through pseudo-label prediction, node-motif matching, fragment-graph lookup, or motif-aware contrastive views.Most methods target molecular graphs; diverse motifs can enlarge motif dictionaries and increase memory overhead.Source: §4.3 · p. 7

4.4 Clusters

• Cluster tasks capture higher-level organization through feature-based node clustering or connectivity-based graph partitioning, using pseudo-labels, prototypes, regression, reconstruction, or contrastive objectives.Node clustering relies mainly on feature similarity, whereas graph partitioning uses structural community patterns and also applies to unattributed graphs.Source: §4.4 · p. 7, Figure 6 · p. 8

5.1 Global Structure

• Global structure tasks learn graph-level representations through graph instance discrimination, node-graph discrimination, and graph similarity prediction.Methods use readout functions, mutual-information estimators, graph kernels, graph edit distance, or isomorphic similarity.Source: §5.1 · p. 8
• Node-graph discrimination contrasts global graph representations with node representations, linking local and global information and applying to both small and large graphs.Group discrimination simplifies this objective by predicting whether an instance belongs to an original or perturbed view, avoiding graph-instance similarity calculations.Source: §5.1 · p. 8
• Global structure tasks offer holistic information useful for small graphs and graph-level prediction, but simple readouts may lose structural details and perturbations may alter small-graph semantics.Graph kernel prediction is additionally constrained by the cost of calculating kernels on large networks.Source: §5.1 · p. 8

5.2 Manifolds

• Manifold-based tasks embed graphs in hyperbolic, spherical, Euclidean, or product spaces to represent hierarchical and other geometric structures that Euclidean spaces handle poorly.Cross-manifold discrimination compares views across spaces, while some methods adaptively select a suitable manifold for each graph.Source: §5.2 · pp. 8–9
• Manifold learning remains open to more flexible geometric approaches, while switching between spaces increases wall-clock computation and storing multiple representations increases memory use.Hyperbolic exponential and logarithmic maps require additional norm and inverse-trigonometric calculations without changing the algorithmic upper bound.Source: §5.2 · p. 9

6.1 Graph Fine-tuning

• Graph fine-tuning adapts pretrained models through task-specific branches, ranging from full parameter updates to modular, meta-learning, cross-domain, and parameter-efficient designs.Methods also quantify pretraining–downstream task similarity using optimal transport, gradient similarity, or representation consistency.Source: §6.1 · p. 9
• Fine-tuning can harm pretrained generalization and remains constrained by scenario-specific designs, although parameter-efficient methods reduce resource requirements.Such methods are less common for pure GNNs because those models are relatively small.Source: §6.1 · p. 9

6.2 Graph Prompting

• Graph prompting updates learnable prompts that encode downstream information while keeping the pretrained model frozen, functioning as data-level parameter-efficient adaptation.Prompt integration may use addition, multiplication, concatenation, aggregation, or transformation, depending on prompt form and task requirements.Source: §6.2 · pp. 9–11
• Prompt designs encode node features, class or type prototypes, edges, contextual neighborhoods, prompt graphs, spectral properties, or graphlets for downstream prediction.These designs support tasks including node and graph classification, link prediction, subgraph similarity, few-shot learning, and zero-shot learning.Source: §6.2 · pp. 10–11
• Graph prompting offers unified downstream templates but remains difficult to explain and is concentrated on discriminative tasks rather than generative or open-ended applications.The survey distinguishes a graph prompt added to downstream data from a prompt graph composed of prompt and class nodes.Source: §6.2 · p. 11

7.1 Pre-training GLMs with Self-supervision

• GLM pretraining comprises language modeling and graph pretraining, applicable to GNNs, graph Transformers, and language models such as T5 and BERT.Language modeling includes autoregressive, masked, and hybrid objectives.Source: §7 · p. 11, §7.1 · p. 11, §7.1.1 · p. 11

7.1.1 Language Modeling Pre-training

• Graph-aware language pretraining combines LMs with GNNs and adapts masked or autoregressive objectives to node attributes, neighborhoods, and graph-derived sequences.Examples include contextualized MLM using neighboring nodes and autoregressive training on shortest-path text sequences.Source: §7.1.1 · p. 12, §7.1.2 · p. 12

7.1.2 Graph Pre-training

• Graph pretraining uses link prediction, node instance discrimination, node–text discrimination, and graph-specific reconstruction to align language and structural representations.Higher-order context and long-range similarity are incorporated through neighborhood summaries, contextual discrimination, common-neighbor measures, SimRank, and random-walk sequences.Source: §7.1.2 · p. 12

7.2 Tuning GLMs

• Downstream tuning is dominant for GLMs because large-scale LLM pretraining is computationally demanding, with LLMs serving as core models under the survey’s classification.This perspective treats later graph-specific training as tuning rather than viewing LLMs merely as auxiliary enhancers, predictors, encoders, or aligners.Source: §7.2 · pp. 12–13

7.2.1 Prompting for GLMs

• GLM prompting textualizes graphs and adds graph knowledge through similar neighbors, pseudo-code reasoning, or GNN-selected in-context examples.Prompting is used for graph question answering, reasoning, and other open-ended tasks.Source: §7.2.1 · p. 13

7.2.2 Fine-tuning for GLMs

• GLM fine-tuning primarily uses PEFT modules to preserve language semantics while aligning text and graph spaces, with GNN and non-GNN modules, backbone tuning, and data augmentation as alternatives.Augmentations include LLM-generated text, pseudo-labels, and graph structure; structural knowledge includes links, paths, cycles, flows, and Hamiltonian paths.Source: §7.2.2 · pp. 13–14
• GLMs still face underexplored billion-scale capabilities, hallucinations, and spurious-factor interference, motivating stronger graph-specific architectures and knowledge-aware training.The survey notes that GLM performance can decline considerably under structural attacks.Source: §7.2.2 · p. 14

8.1 Combining Different Graph Knowledge Patterns

• Future GFMs should combine graph knowledge through multi-task pretraining, adaptive objective weighting, distillation, or multi-task prompting because pretext effectiveness depends on downstream scenarios.AutoSSL, ParetoGNN, GraphTCM, AGSSL, WAS, ULTRA-DP, and MultiGPrompt provide representative strategies.Source: §8.1 · pp. 14–15

8.2 Knowledge Adaptation across Graph Types

• Knowledge adaptation across heterogeneous, knowledge, dynamic, and hypergraphs can reuse node, link, context, temporal, and higher-order pretexts while modeling type- and time-specific structure.Examples include meta-paths, network schemas, relation triples, temporal prompts, and hyperedge-based contrastive or membership prediction.Source: §8.2 · p. 15
• Dynamic graph methods incorporate temporal features, time-aware similarity, dual prompts, sequential LLM reasoning, and weighted combinations of graph snapshots.Hypergraph methods treat hyperedges as higher-order context and learn through propagation, augmentation, contrast, or node–hyperedge membership prediction.Source: §8.2 · p. 15

8.3 Tackling Potential Biases in Future GFMs

• Bias mitigation addresses imbalanced classes, degrees, and sensitive representations through structure-aware augmentation, balanced sampling, adversarial training, focal objectives, and fair prompting or PEFT.These approaches modify data, parameters, or representations to emphasize minority samples or reduce sensitive-attribute disparities.Source: §8.3 · pp. 15–16
• Structural attacks can disrupt graph knowledge through malicious features, nodes, or links, and structural attacks often outperform feature-based attacks.Defenses include mutual-information-based robustness training, random edge dropping, and LLM-assisted link removal or restoration.Source: §8.3 · p. 16

8.4 Building Powerful and Explainable GFMs

• Explainable GFMs can use natural-language reasoning and retrieval-augmented generation to reduce hallucinations, enrich graph knowledge, and support reasoning over paths and clusters.The survey identifies Chain-of-Thoughts, Tree-of-Thoughts, knowledge graphs, long-range paths, and clusters as relevant directions.Source: §8.4 · p. 16

Reference organization

• Tables 1–3 summarize self-supervised graph pre-training, graph downstream tuning, and graph language models in strict chronological order by earliest publication or preprinting time.The “Time” column determines ordering, while paper and open-source code links are provided through the survey’s GitHub list.Source: Appendix A · p. 21, Table 1 · pp. 21–23

Self-supervised graph pre-training

• The cited pre-training inventory covers node, link, feature, cluster, motif, manifold, and global-structure knowledge through discrimination, prediction, masking, and partitioning tasks.Representative entries include CG3, GraphCL, CommDGI, SelfMGNN, and GraphMAE.Source: Appendix A · pp. 22–23

Downstream tuning methods

• Table 2 summarizes graph downstream tuning methods, with cited entries covering fine-tuning, supervised fine-tuning, parameter-efficient fine-tuning, and prompting.Examples include L2P-GNN, GPPT, GPF, AdapterGNN, and PRODIGY.Source: Table 2 · p. 25, Appendix A · p. 25

• Downstream applications include node classification, link prediction, graph classification, and graph question answering, with LLM4NG and LLM4DyG among the cited examples.Source: Appendix A · pp. 25–26
• Recent entries extend the catalog through 2024 and early 2025, including IGAP, P2TAG, GraphCLIP, and EdgePrompt.These newer models use prompting or supervised fine-tuning for node, link, and graph-level applications.Source: Appendix A · p. 25
• The listed methods draw on diverse graph knowledge types, including node features, links, context, clusters, node properties, and long-range similarities, for node classification and neighborhood prediction.Source: Appendix A · p. 26
• The table catalogs graph language models using prompting, MLM, AR, SFT, and miscellaneous training strategies, while distinguishing frozen LLM backbones and adapter modules.Adapters include non-GNN/GT modules such as LoRA and linear projectors; AR and MLM denote autoregressive and masked language modeling, respectively, while SFT denotes supervised fine-tuning.Source: Table 3 · pp. 26–27