PaperlayerView exact revision on arXiv

Source-linked AI summary

Boosting Graph Foundation Model from Structural Perspective

Yao Cheng, Yige Zhao, Jianxiang Yu, Xiang Li

arXiv:2407.19941v2cs.LG

TL;DR

Language-model graph foundation approaches unify semantic representations but leave domain-specific graph structures insufficiently addressed. BooG combines language-model attributes with subgraphs, super nodes, virtual edges, and contrastive pre-training. Across seven datasets, it outperforms state-of-the-art competitors in most cases.

  • Problem

    Language-model-based graph approaches learn unified semantic representations but remain constrained by the distinct structural characteristics of graphs from different domains.

    Source: §1 · p. 2

  • Method

    BooG encodes graph attributes and class descriptions in a shared language-model space and uses subgraphs, super nodes, virtual edges, and contrastive pre-training to learn graph representations.

    Source: §1 · pp. 2–3

  • Results

    Across seven datasets covering domains and tasks, BooG outperforms state-of-the-art competitors in most cases.

    Source: §1 · p. 3

  • Takeaways & Limitations

    BooG’s supernode-centered structure combines neighborhood information with class-label preferences across graph domains and task levels.

    Source: §6 · p. 17

    During pre-training, BooG relies solely on textual descriptions of dataset classes rather than node or graph class labels.

    Source: §4.1 · p. 6

Abstract

from arXiv · show

Graph foundation models have recently attracted significant attention due to its strong generalizability. Although existing methods resort to language models to learn unified semantic representations across domains, they disregard the unique structural characteristics of graphs from different domains. To address the problem, in this paper, we boost graph foundation model from structural perspective and propose BooG. The model constructs virtual super nodes to unify structural characteristics of graph data from different domains. Specifically, the super nodes fuse the information of anchor nodes and class labels, where each anchor node captures the information of a node or a graph instance to be classified. Instead of using the raw graph structure, we connect super nodes to all nodes within their neighborhood by virtual edges. This new structure allows for effective information aggregation while unifying cross-domain structural characteristics. Additionally, we propose a novel pre-training objective based on contrastive learning, which learns more expressive representations for graph data and generalizes effectively to different domains and downstream tasks. Experimental results on various datasets and tasks demonstrate the superior performance of BooG. We provide our code and data here: https://anonymous.4open.science/r/BooG-EE42/.

1 Introduction

Graph models face label costs and task-specific designs, while existing language-model-based graph foundation approaches remain constrained by domain-specific structure. BooG combines unified text representations, a standardized structure, and contrastive pre-training; experiments across seven datasets show it outperforms state-of-the-art competitors in most cases.

1 Introduction

  • Graph neural networks often need many training labels and are commonly designed for a single task, while graph self-supervised learning uses data structure or characteristics without explicit instance labels.Graph contrastive learning compares similarity and dissimilarity between instances.Source: §1 · p. 2
  • Existing language-model graph approaches learn unified semantic representations but remain constrained by domain-specific structural characteristics.Citation networks may have high clustering, whereas molecular graphs often have fixed topological structures and physicochemical properties.Source: §1 · p. 2
  • BooG encodes graph attributes and class descriptions with a pre-trained language model, then unifies graph structure using subgraphs, super nodes, and virtual edges.The super nodes aggregate neighborhood information together with associated class labels.Source: §1 · pp. 2–3
  • Across seven datasets spanning domains and tasks, BooG outperforms state-of-the-art competitors in most cases.The paper also reports ablations validating the necessity of each BooG component.Source: §1 · p. 3

2 Related Works

The related work describes graph neural networks, graph self-supervised learning, and language-model-based graph foundation models. These approaches address representation learning and task adaptation, but the reviewed methods retain limitations in label demand, pre-training/task alignment, or cross-domain structural generalization.

2.1 Graph Neural Network.

  • Graph neural networks capture patterns from node features and structure, but often require many labels and task-specific designs that hinder transfer across tasks.Node classification designs may emphasize local information, while graph classification needs more global information.Source: §2.1 · p. 3

2.2 Graph Self-Supervised Learning.

  • Graph self-supervised learning uses pre-training and fine-tuning, including contrastive or generative objectives, but pre-training/downstream task disparity can yield suboptimal performance.GraphCL applies self-supervised contrastive learning to node embeddings; DGI maximizes mutual information between local and global graph views.Source: §2.2 · p. 3

2.3 Graph Foundation Model based on LLM.

  • LLM-integrated graph methods use language models as either the task backbone or a source of unified instance representations, yet remain constrained by domain-specific graph structures.UniGraph uses masked graph modeling and graph instruction fine-tuning; OFA aligns graph feature spaces for supervised training across graphs.Source: §2.3 · p. 4

3 Preliminaries

The preliminaries define text-attributed graphs and describe message passing as neighbor aggregation followed by node-embedding updates. They then specify BooG’s evaluation across supervised, few-shot, and zero-shot scenarios on node classification, graph classification, and link prediction.

3.1 Text-Attributed Graphs

  • A text-attributed graph has nodes and edges with sequential-text features, and a dataset associates text attributes with its classes.The graph may have node text features without edge text features.Source: §3.1 · p. 4

3.2 Message Propagation

  • Message passing consists of aggregating information from a node’s neighbors and then updating its embedding; BooG applies aggregation on its newly constructed structure.The aggregation step precedes the embedding update.Source: §3.2 · p. 4

3.3 Learning Scenarios

  • BooG evaluates generalization across unseen datasets on node classification, graph classification, and link prediction, using supervised, few-shot, and zero-shot settings.Few-shot tasks provide K labeled examples for each of N classes; zero-shot sets K to 0.Source: §3.3 · pp. 4–5
  • In zero-shot evaluation, similarity matching predicts unseen instances with the pre-trained model frozen; supervised and few-shot scenarios adjust an MLP while keeping that model frozen.The framework standardizes node- and graph-level inputs as subgraphs and uses super nodes to aggregate neighborhood information and class labels.Source: Figure 1 · p. 5

4 Methodology

BooG encodes graph text and class descriptions with a shared language model, then uses anchor-centered subgraphs and class-conditioned super nodes for pretraining and downstream prediction.

4.1 Unifying Graph Attributes

  • Node text and class descriptions are encoded by a pretrained language model, while pretraining uses class descriptions rather than instance labels.Source: §4.1 · pp. 5–6

4.2 Unifying Graph Structures

  • BooG converts tasks to anchor-centered subgraphs and adds class-fused super nodes that aggregate each anchor’s neighborhood through virtual edges, standardizing structure across domains and tasks.Source: §4.2 · pp. 6–8

4.3 Self-Supervised Learning Paradigm

  • Contrastive pretraining treats each original super-node embedding as a positive for its aggregated embedding and other class hypotheses as negatives, without requiring sample-label pairs.Source: §4.3 · p. 8

4.4 Tuning for Downstream Tasks

  • The pretrained model remains frozen at evaluation: zero-shot prediction uses similarity matching, while supervised and few-shot tasks train MLP classifiers with cross-entropy.Source: §4.4 · p. 9

4.5 Time Complexity

  • BooG has linear time complexity with respect to the number of instances n.The analysis gives O(n) scaling in n, with other terms depending on embedding dimensions, sequence length, neighbors, and classes.Source: §4.5 · p. 9

4.6 Why BooG Works?

  • BooG’s generalization design combines shared language-model encoding of graph text with a super-node-centered structure that joins neighborhood information to class labels across domains and task levels.Source: §4.6 · pp. 9–10

5 Experiments

Across seven datasets and multiple learning scenarios, BooG generally performs strongly; the experiments also examine its components, pre-training data, representation quality, hyper-parameters, and efficiency.

5.1 Experimental Settings

  • The evaluation covers seven datasets across domains and tasks, with average accuracy reported using dataset-specific splits or random seeds.BooG and self-supervised baselines are pre-trained on Cora for the main comparisons, then evaluated on target-dataset test sets.Source: §5.1 · pp. 10, 11, 13

5.2 Performance Results

5.3 Visualization

  • After pre-training, Cora node representations for different classes become more distinctly separated in the t-SNE visualization.The authors interpret this separation as evidence of richer, more expressive representations.Source: Figure 4 · p. 14, §5.3 · pp. 14–15

5.4 Ablation Study

  • BooG outperforms variants without class labels, anchor nodes, or similarity matching on Cora and HIV in supervised and 5-shot settings.The study reports performance declines for variants without class labels or anchor nodes, highlighting the importance of super nodes, especially for graph-domain transfer.Source: Figure 5 · p. 14, §5.4 · p. 15

5.5 Performance on different pre-training datasets

  • Joint pre-training on multiple datasets performs better on unseen-domain datasets than pre-training on a single dataset.BooG also achieves good performance across different pre-training datasets and learning scenarios.Source: §5.5 · p. 15, Table 7 · p. 16, Table 8 · p. 16

5.6 Sensitivity Analysis

  • Moderate α values typically work better by balancing anchor-node semantics and class priors, while β performs well across a wide range.Too-small α reduces class-conditioned modeling; too-large α may suppress instance-specific signals.Source: §5.6 · p. 16, Figure 6 · p. 17

5.7 Efficiency Analysis

  • 43 minutes for BooG pre-training on Cora compares with 52 minutes for GraphCL, 63 minutes for GraphMAE, and 2.8 hours for UniGraph.The paper attributes BooG’s efficiency to its lightweight design, which avoids costly full-graph augmentation or reconstruction.Source: §5.7 · pp. 16–17, Table 9 · p. 17
  • BooG’s MLP classifier takes 5 minutes for supervised tuning and 2 minutes in 5-shot settings; inference uses similarity matching.The authors describe downstream tuning and inference as fast.Source: §5.7 · p. 17

6 Conclusion

BooG combines language-model-based attribute unification with a supernode-centered structure for cross-domain and cross-task graph learning. Experiments across seven datasets show that it outperforms state-of-the-art competitors in most cases.

6 Conclusion

  • BooG encodes node text and class-label descriptions into a shared space using a pre-trained language model.This attribute unification is intended to handle graph data from diverse domains.Source: §6 · p. 17
  • Subgraphs pair anchor nodes with neighbors, while super nodes and virtual edges aggregate neighborhood information with class labels.This provides a standardized structure across task levels and graph domains.Source: §6 · p. 17
  • Across seven datasets and varied domains and tasks, BooG outperforms state-of-the-art competitors in most cases.Source: §6 · p. 17
Loading 2407.19941v2…