Graph Foundation Models: Concepts, Opportunities and Challenges

• Task-specific graph learning has advanced through GNNs, but expressive power and generalizability remain challenges as datasets and task ranges expand.Earlier shallow methods were typically limited to transductive learning.Source: §1 · p. 1
• GFMs are envisioned as pre-trained on extensive graph data and adaptable across downstream tasks, with emergence and homogenization as anticipated characteristics.Deep graph learning instead trains end-to-end for specific tasks on specific datasets.Source: §1 · p. 1, Figure 1 · p. 2
• GFM design raises questions about backbone capacity, graph-structure modeling in LLMs, and pre-training and adaptation across diverse graph tasks.The article notes that GNN parameter counts are significantly smaller than language backbones and that LLMs’ ability to handle graph data remains uncertain.Source: §1 · p. 2
• The article categorizes GFM research as GNN-based, LLM-based, or GNN+LLM-based, according to how each approach relies on these model families.The categories cover innovations to graph-learning paradigms, graph-to-text or token conversion, and forms of GNN–LLM synergy.Source: §1 · p. 2
• The article presents itself as the first systematic GFM survey, distinguishing its taxonomy from earlier surveys focused on other modalities or knowledge graphs.It also identifies future research directions for GFMs.Source: §1 · p. 2

2.1.1 Graph Data

• Graph data is non-Euclidean and varies across domains and graph types, complicating the design of universal models.Examples include social networks, biology, transportation, and homogeneous, heterogeneous, hyper-, and dynamic graphs.Source: §2.1.1 · pp. 2–3

2.1.2 Backbone Architectures

• GNNs commonly use message passing, while deeper networks face over-smoothing and over-squashing challenges.Graph transformers with fully connected attention and long-range relationship modeling help alleviate these issues.Source: §2.1.2 · p. 3

2.1.3 Learning Paradigms

• Deep graph learning uses supervised, semi-supervised, and unsupervised paradigms, including self-supervision that generates labels from data.Examples include graph classification, node classification, clustering, and link prediction.Source: §2.1.3 · p. 3

2.2 Language Foundation Models

• Language foundation models use pre-training and adaptation, with scaling and self-supervised learning helping produce models for many downstream tasks.The background describes foundation models as trained on extensive, diverse datasets and adapted across tasks.Source: §2.2 · p. 3

2.2.1 Language Data

• Language data is described as Euclidean and easier to model than graph data, with semantic information enhancing language-model knowledge transferability.Source: §2.2.1 · p. 3

2.2.2 Backbone Architectures

• LLMs primarily use Transformer architectures, with tokens as inputs; larger pre-trained language models are described as exhibiting emergent abilities.The passage compares GPT-2 at 1.5B parameters with GPT-3 at 175B parameters.Source: §2.2.2 · p. 3

2.2.3 Learning Paradigms

• Language-model adaptation includes fine-tuning and prompting: fine-tuning adjusts a pre-trained model, while prompting reshapes tasks to align with its training.Fine-tuning can transfer knowledge between source and target tasks; prompt-tuning adapts tasks without adjusting model parameters.Source: §2.2.3 · pp. 3–4

3.1 Definition and Key Characteristics

• GFMs are expected to use pre-training on broad graph data and adapt to a wide range of downstream graph tasks.Source: §3.1 · p. 4
• Emergence and homogenization are proposed GFM characteristics, but few studies explore emergent abilities and no homogenization approach has consensus.Emergence includes in-context learning, graph reasoning, and zero-shot graph generation; homogenization spans node classification, link prediction, and graph classification.Source: §3.1 · p. 4

3.2 Key Technologies

• GFM pre-training uses self-supervised learning on large graph datasets to capture structure, relationships, and patterns.Strategies include contrasting positive and negative samples and reconstructing graph structure or predicting original features.Source: §3.2 · p. 4
• Adaptation tailors pre-trained GFMs through full fine-tuning, parameter-efficient fine-tuning, or prompt-tuning.Full fine-tuning can require substantial data and compute, while parameter-efficient fine-tuning adjusts only some parameters; graph prompt-tuning methods remain relatively few.Source: §3.2 · p. 4

3.3.1 Similarities

• GFMs and language foundation models share the goals of expressive power and broad generalization, and both use pre-training followed by task-specific adaptation.Source: §3.3.1 · p. 5

3.3.2 Intrinsic Differences

• GFMs handle sparse, non-Euclidean graphs with domain-specific structures and diverse task formats, unlike language models’ sequential text and shared task format.Graph tasks include node classification, link prediction, and graph classification; their varied formats increase downstream adaptation challenges.Source: §3.3.2 · p. 5

3.3.3 Extrinsic Differences

• GFMs lack a unified backbone, face harder homogenization and weaker cross-dataset generalization, and have only a few studies reporting emergent abilities.Source: §3.3.3 · pp. 5–6

3.4 Summary

• The survey groups implementation approaches as GNN-based, LLM-based, or GNN+LLM-based, with corresponding differences in backbone, pre-training, and adaptation strategies.Source: §3.4 · p. 6

4.1.1 Message Passing-Based Methods

• Message-passing GNNs iteratively aggregate information from neighboring nodes, with GCN, GAT, GraphSAGE, and GIN among common backbones.GCN uses localized spectral convolution; GAT uses attention-weighted aggregation, while GraphSAGE samples neighbors and GIN has expressive power equivalent to a 1-WL test.Source: §4.1.1 · p. 6

4.1.2 Graph Transformer-Based Methods

• Graph transformers model every node pair through fully connected attention, unlike adjacency-constrained message passing, and show promise on molecular prediction.Their long-range modeling and expressive power address limitations including over-smoothing and over-squashing in traditional GNNs.Source: §4.1.2 · pp. 6–7, Figure 2 · p. 6

4.2.1 Contrastive Methods

• Contrastive pre-training maximizes agreement across graph views, either at the same scale or across scales, to capture invariant semantics.DGI’s cross-scale approach captures node- and graph-level information but ignores differences between nodes.Source: §4.2.1 · p. 7

4.2.2 Generative Methods

• Generative pre-training learns graph structure and attribute semantics through graph reconstruction or property prediction.Methods reconstruct adjacency, edges, or features, or predict local context and graph motifs; the quality and interpretability of generated content remain difficult to guarantee.Source: §4.2.2 · p. 7

4.3.1 Fine-Tuning

• Fine-tuning adapts pre-trained models when pre-training and downstream objectives differ, including settings involving unseen graphs.It may require sufficient labeled data and repeated task-specific tuning, which incurs computational costs.Source: §4.3 · p. 7, §4.3.1 · p. 8

4.3.2 Prompt Tuning

• Prompt tuning avoids full-parameter tuning and supports multi-task or zero-shot adaptation, with prompts applied before or after the GNN backbone.Pre-prompts modify graph inputs; post-prompts modify representations after message passing, but prompt semantics and interpretability need further study.Source: §4.3.2 · p. 8

4.4 Discussion

• GNN-based models offer permutation invariance, efficient training, and potential to generalize from sparse labels; work also extends to heterogeneous, hyper-, and temporal graphs.Examples include pre-training and prompt-based methods tailored to high-order heterogeneous graphs, hyperedge prediction, and spatio-temporal representations.Source: §4.4 · p. 8
• GNN-based models underuse textual attributes and general knowledge, limiting generalization across domains and performance on tasks needing broad context or common-sense reasoning.Integrating LLMs is identified as a promising direction for incorporating textual knowledge and semantics.Source: §4.4 · p. 9

5 LLM-BASED MODELS

• LLM-based graph learning treats LLMs or text-processing transformers as graph-learning backbones, with natural-language task descriptions offering a route to unify graph tasks.The surveyed works also explore graph reasoning and combining graph properties with textual information.Source: §5 · p. 9

5.1 Backbone Architectures

• Graph-to-token adds an encoder to turn nodes into embeddings, while graph-to-text directly describes graphs in natural language for LLM input.Graph-to-token typically requires a trainable transformer or open-source LLM; graph-to-text can use even closed LLMs.Source: §5.1 · p. 9, §5.1.1 · p. 9, §5.1.2 · p. 9

5.1.1 Graph-to-token

• Graph-to-token approaches include GIMLET and InstructGLM, but conveying underlying graph structure to LLMs remains a challenge.InstructGLM incorporates node feature vectors as distinct vocabulary tokens; multimodal transformer approaches without pre-training and adaptation are not classified as GFMs here.Source: §5.1.1 · p. 9

5.1.2 Graph-to-text

• Graph-to-text methods describe graph information with prompts, but edge-list descriptions have shown limitations on complex graph problems.GPT4Graph reports self-prompting as more effective for informing LLMs about graph structures, and Graph-LLM identifies neighbor summarization as the most effective existing prompt technique.Source: §5.1.2 · pp. 9–10

5.2 Pre-Training

• LLM-based graph models commonly use language modeling, which trains next-token probability prediction from preceding textual context.The section also describes masked language modeling, which predicts randomly masked tokens from surrounding context; the mask token may be absent during fine-tuning.Source: §5.2.1 · p. 10, §5.2.2 · pp. 10–11

5.3 Adaptation

• Because manual prompts take time to create and can be suboptimal, automatic prompting has been explored for adaptation.GPT4Graph uses generated graph summaries, exploration, and completion prompts; Graph-LLM uses neighbor summaries.Source: §5.3 · p. 11, §5.3.1 · p. 11, §5.3.2 · p. 11

5.3.2 Automatic Prompting

• Fine-tuning approaches used elsewhere, including parameter-efficient methods, had not yet been applied to graph tasks in the surveyed work.The authors anticipate future integration of these adaptation approaches into graph tasks.Source: §5.3.2 · p. 11

5.4 Discussion

• Current LLM graph methods struggle with long graph descriptions, multi-hop reasoning, dense topology, and dynamic graphs; graph-to-text is input-length limited, while graph-to-token costs more computation.Future directions include more efficient structured graph modeling and exploring hypergraphs and heterogeneous graphs beyond temporal graphs.Source: §5.4 · p. 11

6.1 Backbone Architectures

• GNN-centric, symmetric, and LLM-centric methods distinguish which component leads prediction or how GNNs and LLMs work together.These approaches aim to use both graph and text information across varied tasks.Source: §6.1 · p. 12, §6 · p. 12

6.1.1 GNN-centric Methods

• GNN-centric methods use LLM-derived text features for GNN prediction, while graph-aware encoding can address graph-agnostic text representations.TAPE generates predictions and explanations from text before constructing node features; GIANT fine-tunes language models with graph-structure-aware self-supervision.Source: §6.1.1 · p. 12

6.1.2 Symmetric Methods

• Symmetric methods align graph and text representations; GraphFormer exchanges neighborhood information iteratively, while GLEM uses variational EM to incorporate local text and global structure.GraphFormer’s memory complexity grows with graph size because neighborhood texts are encoded; GLEM is described as easing this scalability issue.Source: §6.1.2 · p. 13

6.1.3 LLM-centric Methods

• LLM-centric methods use graph models to address LLM difficulties with precise calculations, multi-step reasoning, spatial and topological perception, and temporal progression.Examples include graph instruction tuning, soft prompts, and retrieval-augmented generation for textual-graph question answering.Source: §6.1.3 · p. 13

6.2.1 GNN or LLM-based

• Pretraining strategies include language-model objectives, text-text contrastive learning, and graph reconstruction for incorporating graph information.The survey also distinguishes approaches that pretrain GNNs or LLMs from alignment-based methods.Source: §6.2 · p. 13, §6.2.1 · pp. 13–14

6.2.2 Alignment-based

• Alignment-based pretraining uses graph-text contrastive learning to align graph-encoder and text-encoder embeddings for downstream tasks.For molecules, the described contrastive objective brings active molecules closer to the bioassay representation and inactive molecules farther away.Source: §6.2.2 · p. 14

6.3 Adaptation

• Adaptation includes fine-tuning and prompt-tuning, with parameter-efficient fine-tuning and tuning-free prompting among the alternatives to broad parameter updates.Some works use prompt-tuning with limited labeled data, while tuning-free prompting relies on pretrained language-model knowledge without additional parameter adjustment.Source: §6.3 · p. 14, §6.3.1 · p. 14, §6.3.2 · p. 14

6.4 Discussion

• The combined models can produce embeddings with text, structure, and external knowledge for downstream and cross-modal tasks, but aligning GNN and LLM representations remains challenging.The survey calls for robust alignment measures and methods, and notes work extending these approaches to heterogeneous graphs and hypergraphs.Source: §6.4 · p. 14

7.1.1 Data Quantity and Quality

• Limited open-source large-scale graph data, concentrated mainly in single domains, constrains development of GFMs for diverse domains.Existing augmentation methods are typically tailored to individual GNN models, leaving effective augmentation for LLM-based and GNN+LLM-based models open for exploration.Source: §7.1.1 · pp. 14–15

7.1.2 Evaluation

• Evaluating GFMs on open-ended tasks is challenging because those tasks lack labels, and applicability of existing LLM evaluation methods remains unresolved.The survey also identifies robustness, trustworthiness, and holistic performance as evaluation targets.Source: §7.1.2 · p. 15

7.2.1 Model Architectures

• It remains unknown whether architectures beyond Transformers can handle graph data, and research on GFM emergence and scaling laws is limited.The survey also identifies alignment in GNN+LLM models and handling multiple graph types as open architectural directions.Source: §7.2.1 · p. 15

7.2.2 Model Training

• The applicable scope of different GFM pretraining tasks and the possibility of a unified task remain open questions.Cross-domain support and applying techniques such as distillation, RLHF, or model editing to GFMs are also future research directions.Source: §7.2.2 · p. 15

7.3.1 Killer Applications

• Whether GFMs can catalyze groundbreaking graph-task applications is unclear, though the survey discusses possible roles in drug discovery and urban computing.Proposed directions include graph-based models for open-ended tasks and using graph learning to enhance LLM reasoning.Source: §7.3.1 · p. 15

7.3.2 Trustworthiness

• GFM trustworthiness must address risks including hallucination, privacy, fairness, and robustness to attacks.The survey discusses confidence calibration, counterfactual reasoning, federated learning, RLHF, and red teaming as possible approaches, whose applicability to GFMs remains unknown.Source: §7.3.2 · pp. 15–16

8 CONCLUSIONS

• The article proposes the concept of graph foundation models (GFMs) for models trained on broad graph data and applied to diverse downstream graph tasks.It introduces relevant concepts and representative methods.Source: §8 · p. 16
• Existing GFM work is grouped into GNN-based, LLM-based, and GNN+LLM-based models.The review discusses backbone architectures, pre-training, and adaptation strategies for each category.Source: §8 · p. 16
• The article reviews the current GFM landscape and identifies future directions for the evolving field.Source: §8 · p. 16

A. Impact of Graph Data on GFMs

• Heterogeneous graphs require specific backbone architectures and optimization objectives, while meta-paths can map them into homogeneous graphs for separate embedding and fusion.Dynamic graphs and hypergraphs pose additional challenges.Source: A. Impact of Graph Data on GFMs · p. 21
• Large graphs increase storage and computation demands and can intensify over-smoothing and over-squashing in GNN-based models.Their long-range dependencies may require more neural-network layers and parameters.Source: A. Impact of Graph Data on GFMs · p. 21
• Cross-domain graphs lack unified semantics, making transfer challenging and potentially weakening performance or causing negative transfer on new datasets.Applying the same GFM across domains remains a significant challenge.Source: A. Impact of Graph Data on GFMs · p. 21

B. Impact of Graph Tasks on GFMs

• Graph tasks are divided into node-level, edge-level, and graph-level tasks, each covering distinct prediction or analysis targets.Source: B. Impact of Graph Tasks on GFMs · p. 21
• Node-level tasks predict or analyze individual nodes, including classification, regression, and clustering-coefficient prediction.In social networks, node classification can identify users from different social circles.Source: B. Impact of Graph Tasks on GFMs · p. 21
• Edge-level tasks concern individual edges and include link prediction, shortest-path prediction, connectivity prediction, and maximum-flow prediction.In e-commerce, link prediction can estimate products users may be interested in.Source: B. Impact of Graph Tasks on GFMs · p. 21
• Graph-level tasks address whole-graph properties or outputs, including classification, regression, generation, clustering, and graph condensation.In bioinformatics, graph property prediction can target molecular activity or toxicity.Source: B. Impact of Graph Tasks on GFMs · p. 21
• Task heterogeneity complicates GFM homogenization: structural or feature information may matter more depending on the task, and similar tasks have lower transfer difficulty.The passage also notes that increased expressive power may help many tasks but may not be necessary for graph generation.Source: B. Impact of Graph Tasks on GFMs · p. 21

C. Details of approaches involved as GNN-based models

• Table 1 categorizes the GNN-based methods.Source: C. Details of approaches involved as… · p. 21

D. Details of approaches involved as LLM-based models

• Table 2 categorizes the surveyed LLM-based methods.The supplied passage identifies the category and table but gives no further method-level details.Source: D. Details of approaches involved as… · p. 21

E. Details of approaches involved as GNN+LLM-based models

• The survey categorizes GNN+LLM-based methods in Table 3.The supplied passages identify the category and table but provide no further method-level details.Source: E. Details of approaches involved as… · p. 21, Table 3 · p. 23