PaperlayerView exact revision on arXiv

Source-linked AI summary

GraphGPT: Graph Instruction Tuning for Large Language Models

Jiabin Tang, Yuhao Yang, Wei Wei, Lei Shi, Lixin Su, Suqi Cheng, Dawei Yin, Chao Huang

arXiv:2310.13023v3cs.CLcs.AI

TL;DR

Graph learning methods often depend on task-specific labels, limiting generalization when labels are scarce, especially in zero-shot settings. GraphGPT aligns graph structure with LLMs through text-graph grounding and dual-stage graph instruction tuning, and it reports strong generalization across supervised and zero-shot graph learning tasks.

  • Problem

    Existing graph representation methods often require downstream task-specific labels, while integrating graph structure into LLMs presents alignment, comprehension, and reasoning challenges.

  • Method

    GraphGPT combines text-graph grounding with dual-stage instruction tuning, using self-supervised graph matching and task-specific graph instructions to align LLMs with graph structure.

  • Results

    GraphGPT demonstrates strong generalization across diverse supervised and zero-shot graph learning datasets and tasks, outperforming reported state-of-the-art baselines.

  • Takeaways & Limitations

    The framework provides a graph-oriented LLM that can comprehend graph structures and adapt to diverse downstream graph learning tasks.

Abstract

from arXiv · show

Graph Neural Networks (GNNs) have evolved to understand graph structures through recursive exchanges and aggregations among nodes. To enhance robustness, self-supervised learning (SSL) has become a vital tool for data augmentation. Traditional methods often depend on fine-tuning with task-specific labels, limiting their effectiveness when labeled data is scarce. Our research tackles this by advancing graph model generalization in zero-shot learning environments. Inspired by the success of large language models (LLMs), we aim to create a graph-oriented LLM capable of exceptional generalization across various datasets and tasks without relying on downstream graph data. We introduce the GraphGPT framework, which integrates LLMs with graph structural knowledge through graph instruction tuning. This framework includes a text-graph grounding component to link textual and graph structures and a dual-stage instruction tuning approach with a lightweight graph-text alignment projector. These innovations allow LLMs to comprehend complex graph structures and enhance adaptability across diverse datasets and tasks. Our framework demonstrates superior generalization in both supervised and zero-shot graph learning tasks, surpassing existing benchmarks. The open-sourced model implementation of our GraphGPT is available at https://github.com/HKUDS/GraphGPT.

1 INTRODUCTION

GraphGPT addresses the limited generalization of graph models that depend on downstream labels by aligning graph structural knowledge with LLM reasoning for zero-shot learning. It introduces text-graph grounding and graph instruction tuning, then evaluates generalization across supervised and zero-shot graph tasks.

  • GNNs capture graph relationships through message passing and aggregation, supporting applications including social, recommendation, and biological network analysis.
  • SSL uses unlabeled graph data to pre-train robust representations, but common approaches often still require downstream task-specific labels.
  • Limited access to high-quality labels constrains practical generalization, especially in cold-start recommendation and traffic prediction for new cities.
  • GraphGPT targets zero-shot generalization across diverse graph datasets and tasks by combining LLM capabilities with graph structural knowledge.
  • The framework addresses graph-LLM integration through text-graph grounding and a graph instruction tuning paradigm designed to improve structural comprehension and task adaptability.
  • GraphGPT is evaluated on supervised and zero-shot graph learning tasks and is reported to show superior generalization against state-of-the-art baselines.

2 PRELIMINARIES

Graph data represents entities and relationships through nodes, edges, topology, and node features. GNNs learn node representations by repeatedly propagating and aggregating information over neighboring nodes.

  • A graph G(V, E, A, X) contains nodes V, edges E, adjacency matrix A, and node feature matrix X.
  • The adjacency matrix A encodes graph topology, while X stores node attributes with feature dimensionality F.
  • At layer l, a node representation h_v^(l) is updated by combining its previous representation with aggregated information from neighboring nodes.
  • These structure-aware representations support graph tasks such as node classification and link prediction.

3 METHODOLOGY

GraphGPT aligns graph structure with language representations using a flexible graph encoder, text-graph grounding, and a dual-stage instruction tuning process. Self-supervised graph matching supplies structural knowledge, while task-specific instructions adapt reasoning to downstream graph tasks.

  • Structural Information Encoding with Text-Graph Grounding: GraphGPT uses a flexible message-passing graph encoder, including graph transformers or graph convolutional networks, to represent graph structure.
  • Structural Information Encoding with Text-Graph Grounding: Text-graph grounding encodes graph structure and node text with separate graph and text encoders, then normalizes their representations for cross-modal alignment.
  • Structural Information Encoding with Text-Graph Grounding: The alignment objective compares graph representations with corresponding and neighborhood-derived text representations using contrastive signals.
  • Self-Supervised Instruction Tuning: Stage one performs self-supervised instruction tuning on unlabeled graph structures through a structure-aware graph matching task.
  • Self-Supervised Instruction Tuning: Graph matching presents graph information, a human question, and a response objective that reorders shuffled node text to match graph tokens.
  • Task-Specific Instruction Tuning: Stage two applies task-specific graph instructions to customize reasoning for node classification and link prediction.
  • Task-Specific Instruction Tuning: After both stages, instruction tuning and selective parameter freezing refine graph understanding for downstream tasks.
  • Task-Specific Instruction Tuning: Chain-of-Thought distillation from ChatGPT supplies reasoning instructions without increasing the model’s parameter count.

4 EVALUATION

The evaluation examines GraphGPT across supervised and zero-shot graph learning, multiple datasets, baselines, and research questions concerning generalization, components, and efficiency.

  • The experiments address supervised and zero-shot performance, multi-task generalization without catastrophic forgetting, component contributions, and scalability.
  • GraphGPT is evaluated on OGB-arxiv, PubMed, and Cora datasets.OGB-arxiv is a 40-class citation network; PubMed contains 19,717 diabetes publications across three categories.
  • Raw textual node features are encoded with pre-trained BERT into a unified vector space before evaluation.Cora and PubMed use 3:1:1 train-validation-test splits, while OGB-arxiv follows its public 6:2:3 split.
  • Performance comparisons include MLP, GraphSAGE, GCN, GAT, RevGNN, DGI, and other state-of-the-art methods.
  • Implementation uses PyTorch, Transformers, Vicuna-7B base models, a batch size of 2 per GPU, learning rate 2e-3, and maximum input length 2048.Training runs for 3 epochs initially and 2 epochs during task-specific instruction tuning.
  • Node classification results are reported under supervised and zero-shot settings using Table 1.The experiments use different dataset transfer settings, with classifier handling for varying class counts in GNN baselines.

4.2 Overall Performance Comparison (RQ1)

GraphGPT outperforms state-of-the-art baselines in supervised and zero-shot node classification, with especially large gains after transfer to new datasets. Its dual-stage tuning and chain-of-thought distillation support structural understanding and complex-task performance.

  • GraphGPT consistently outperforms state-of-the-art baselines in both supervised and zero-shot graph learning scenarios.Strong GNN baselines perform well supervised but decline substantially when transferred to new datasets without further training.
  • 2-10 times increase in accuracy is achieved by GraphGPT in the zero-shot graph learning scenario.
  • GraphGPT preserves graph structure, whereas Baichuan-7B and Vicuna-7B rely solely on text information for predictions.The framework aligns graph-encoder structural information with language tokens and supports reasoning about graph structure.
  • Structure-aware graph matching in stage one improves zero-shot transferability by aligning structurally informative graph tokens with language tokens.Using only stage-two task-specific tuning makes the model more prone to dataset-specific overfitting and weaker transfer.
  • 0.7011 accuracy is obtained for Arxiv-PubMed with standard instructions on the simpler three-class PubMed task.COT distillation substantially benefits more complex tasks such as Cora with 70 classes.
  • COT distillation incorporates GPT-3.5 reasoning knowledge to improve performance on complex graph learning tasks.

4.3 Generalization Ability Investigation (RQ2)

GraphGPT improves transfer across datasets and tasks by combining more instruction data without the performance degradation observed in traditional GNN-based approaches.

  • Adding a relatively small PubMed instruction set of 20,000+ items to Arxiv significantly improves GraphGPT’s zero-shot transfer performance on Cora.
  • Traditional GNN-based models trained separately on Arxiv and PubMed instead experience deteriorated transfer performance on Cora.
  • Combining Arxiv and PubMed instruction data improves GraphGPT’s performance on the original Arxiv dataset after iterative training.
  • The authors attribute traditional GNN degradation to catastrophic forgetting, whereas unified graph instruction tuning retains generalized graph structure patterns.
  • Mixing standard, chain-of-thought, blended, and link-prediction instructions is used to study GraphGPT’s multitask generalization.

4.4 Module Ablation Study (RQ3)

The ablation study examines graph instruction tuning, LLM-based reasoning, and computational efficiency to identify the contributions of GraphGPT’s components.

  • The ablation study evaluates individual framework sub-modules under supervised and zero-shot settings.
  • Graph instruction tuning significantly outperforms the base Vicuna-7B-v1.5 model without graph structural information on node classification across three datasets.
  • The improvement is achieved without changing the LLM’s original parameters, using only a lightweight projector that aligns graph and natural-language tokens.
  • Integrating the LLM significantly enhances the graph encoder’s performance, particularly for zero-shot predictions, compared with using default graph encoders alone.

4.5 Model Efficiency Study (RQ4)

GraphGPT’s efficiency study evaluates training and inference costs, emphasizing parameter-efficient tuning and comparative inference speed and accuracy.

  • The efficiency evaluation covers both model training and inference stages.
  • Tuning LLM parameters causes out-of-memory errors even at batch size 1, while GraphGPT’s frozen-parameter strategy remains stable at batch size 2.
  • The graph-text projector tuning strategy reduces the number of tuned parameters by more than 50 times compared with tuning the LLM parameters.
  • GraphGPT’s inference speed and accuracy are compared with baichuan-7B, vicuna-7B-v1.1, and vicuna-7B-v1.5 on Arxiv and Cora chain-of-thought instruction datasets.

4.6 Model Case Study (RQ5)

The case study compares GraphGPT with traditional LLM prompting for downstream graph learning and examines the token efficiency of graph representations.

  • GraphGPT is evaluated against ChatGPT using node-content, text-based graph-structure, and graph-instruction prompts on Arxiv data.
  • For a 103-node subgraph, GraphGPT requires 750 LLM input tokens, compared with 4649 tokens for the text-based method.
  • The graph-token representation substantially reduces token consumption and thereby lowers training and inference resource requirements.

5 RELATED WORK

Prior graph-learning methods improve representations through self-supervision and prompting, but generally still depend on downstream labels. GraphGPT addresses this gap by aligning LLMs with graph structures through dual-stage graph instruction tuning.

  • Self-supervised Learning and Pre-training on Graphs: SSL-enhanced graph methods pre-train representations from graph-intrinsic properties but commonly require downstream fine-tuning with supervision labels.This limits their applicability when high-quality labels are scarce or unavailable.
  • Prompt-Tuning for Graph Neural Networks: Prompt-based GNN methods unify or reformulate graph tasks, yet still rely on downstream supervision labels for accurate learning.Examples include masked edge pre-training followed by prompted node classification and unified graph-level task formats.
  • Large Language Models: Table 6 compares prediction results between GraphGPT and ChatGPT.The table is presented as a direct comparison of the two models' prediction results.
  • Large Language Models: LLMs have demonstrated broad NLP capabilities and motivated tuning-free prompting techniques such as in-context learning and Chain-of-Thought.The related work positions open-source LLMs as a foundation for extending generalization beyond task-specific graph fine-tuning.
  • Large Language Models: Alignment between LLMs and graph structures remains largely unexplored, unlike alignment with visual information.This identifies graph-structure integration as an open multimodal alignment direction.
  • Large Language Models: GraphGPT introduces dual-stage graph instruction tuning with a graph-text alignment projector to connect LLM language capacity with graph learning.The approach responds to limitations of text-only graph prompts in representing complex structures.

6 CONCLUSION

GraphGPT is an effective and scalable graph large language model designed to improve graph-model generalization. Its graph-text alignment projector and dual-stage instruction tuning support supervised and zero-shot learning across diverse datasets and tasks.

  • 6 CONCLUSION: GraphGPT injects graph-specific structural knowledge into an LLM through dual-stage graph instruction tuning and a graph-text alignment projector.The projector enables the LLM to comprehend and interpret graph structural components.
  • 6 CONCLUSION: Extensive evaluations demonstrate GraphGPT's effectiveness in supervised and zero-shot graph learning scenarios.The framework is evaluated across different settings rather than a single learning regime.
  • 6 CONCLUSION: GraphGPT handles diverse downstream datasets and tasks without suffering from catastrophic forgetting.The conclusion characterizes this as a strong generalization ability.
  • 6 CONCLUSION: Future work could explore pruning redundant or less important LLM parameters to reduce model size while preserving performance.This is presented as a potential avenue rather than an evaluated result.
Loading 2310.13023v3…