Source-linked AI summary
Language is All a Graph Needs
Ruosong Ye, Caiqi Zhang, Runhui Wang, Shuyuan Xu, Yongfeng Zhang
TL;DR
Graph learning has only limited integration with generative LLMs, motivating whether LLMs can replace GNNs as a graph foundation model. InstructGLM describes multiscale graph structure in natural language and instruction-tunes an LLM for graph tasks, outperforming GNNs and Graph Transformers across three datasets while remaining limited by LLM input length.
Problem
Existing LLM-based graph learning remains limited, with prior approaches requiring GNN combinations or specialized token and attention designs rather than direct natural-language structural representation.
Method
InstructGLM uses scalable natural-language instruction prompts to describe multiscale graph structure and instruction-tunes a generative LLM for graph tasks.
Results
InstructGLM achieves single-model state-of-the-art performance, surpassing single-model GNNs and Graph Transformers on ogbn-arxiv, Cora, and PubMed.
Takeaways & Limitations
The results demonstrate the potential of generative LLMs as a new foundation model for graph machine learning.
Takeaways & Limitations
LLM input-token limits can prevent one prompt from covering all high-order neighbors in large graphs, while multiple descriptions rapidly increase training-data volume.
Abstract
from arXiv · showhide
The emergence of large-scale pre-trained language models has revolutionized various AI research domains. Transformers-based Large Language Models (LLMs) have gradually replaced CNNs and RNNs to unify fields of computer vision and natural language processing. Compared with independent data samples such as images, videos or texts, graphs usually contain rich structural and relational information. Meanwhile, language, especially natural language, being one of the most expressive mediums, excels in describing complex structures. However, existing work on incorporating graph problems into the generative language modeling framework remains very limited. Considering the rising prominence of LLMs, it becomes essential to explore whether LLMs can also replace GNNs as the foundation model for graphs. In this paper, we propose InstructGLM (Instruction-finetuned Graph Language Model) with highly scalable prompts based on natural language instructions. We use natural language to describe multi-scale geometric structure of the graph and then instruction finetune an LLM to perform graph tasks, which enables Generative Graph Learning. Our method surpasses all GNN baselines on ogbn-arxiv, Cora and PubMed datasets, underscoring its effectiveness and sheds light on generative LLMs as new foundation model for graph machine learning. Our code is open-sourced at https://github.com/agiresearch/InstructGLM.
1 Introduction
The paper positions natural language as a flexible, scalable, and compatible representation for graph structure, and introduces InstructGLM to solve graph tasks through instruction-tuned generative LLMs. Experiments report top-ranked performance across three datasets against competitive graph baselines.
- Research gap: Existing LLM-based graph methods either combine LLMs with GNNs or require novel token embeddings and complex graph-attention modules.These approaches incur computational overhead or restrict direct structural learning.
- Motivation and advantages: Natural language can describe connectivity at arbitrary hop levels, intermediate paths, and multimodal node or edge features.This avoids iterative message passing and directly conveys both graph structure and content.
- Motivation and advantages: Injecting graph structure into multiple natural-language sentences supports mini-batch training, independent gradient propagation, and scalable distributed training with low communication overhead.
- Motivation and advantages: Structure descriptions let InstructGLM reformulate graph-learning pipelines as language-modeling tasks and integrate graph learning with vision, language, and recommendation.
- Approach: InstructGLM uses rule-based, highly scalable instruction prompts to describe graph topology in natural language for node classification and link prediction.The framework also supports zero-shot use of its prompts on LLMs.
- Results: InstructGLM outperforms competitive GNN and Transformer-based methods across ogbn-arxiv, Cora, and PubMed, achieving top-ranked performance.The study also examines self-supervised link prediction as an auxiliary task for node classification.
2 Related Work
Related work centers on GNNs, graph Transformers, hybrid GNN–language-model systems, and emerging direct uses of LLMs for graph problems. These approaches face trade-offs involving topology modeling, multimodal compatibility, computational cost, and pipeline complexity.
- GNN-based methods: GNNs learn node features and graph topology through message passing and aggregation but can suffer from over-smoothing.
- GNN-based methods: GNNs cannot directly process non-numeric text or image data, so they require feature-engineering preprocessing such as BoW, TF-IDF, or Skip-gram.
- Graph Transformers: Graph Transformers represent nodes and edges as tokens, but large-scale processing is computationally intensive and global attention may not capture topology effectively.
- Fuse GNN and Transformers: Hybrid methods combine GNN structural learning with Transformer multimodal features through feature generation, distillation, iterative training, or modified attention.
- Fuse GNN and Transformers: Hybrid GNN–Transformer pipelines train multiple models, increasing computational overhead and retaining susceptibility to GNN limitations such as over-smoothing.
- LLMs for graphs: Emerging LLM graph methods include processor selection, explanation-based augmentation, and prompts or benchmarks for graph construction, evaluation, biology, and structural reasoning.
3 InstructGLM
InstructGLM represents graph structure and features as natural-language prompts, then instruction-tunes a generative LLM to solve graph tasks through a unified language-modeling pipeline.
- Framework: InstructGLM uses natural language to describe graph structure and node or edge meta features, then applies instruction tuning to graph learning.The framework is designed as a generative LLM pipeline for graph-related tasks.
- Prompt Design: Prompts encode a central node’s multi-hop connectivity, optional meta features, and intermediate paths to convey graph structure at multiple scales.Prompt design varies by maximum hop level, feature inclusion, and whether connecting paths are represented.
- Design Advantages: Natural-language graph descriptions provide flexibility for arbitrary hop levels and multimodal node or edge features, while aligning graph learning with other AI domains.The paper identifies flexibility and compatibility as advantages of the approach over graph-specific pipelines.
- Instruction Tuning: InstructGLM fuses instruction prompts spanning different hop levels and structural information, enabling mutual enhancement among instructions in a unified generative pipeline.The model can use an encoder-decoder or decoder-only generative LLM and is trained with language-modeling NLL loss.
- Auxiliary Link Prediction: Self-supervised link prediction samples neighbors or non-neighbors at arbitrary hop levels and uses discriminative or generative prompts as an auxiliary task.The auxiliary task allows arbitrary nodes to act as central nodes and supports the multitask instruction-tuning framework.
4 Experiments
Experiments evaluate InstructGLM for node classification on ogbn-arxiv, Cora, and PubMed, comparing it with GNN and Graph Transformer baselines. The results show strong performance across standard and extremely low-label settings, while ablations identify multi-hop prompts and auxiliary link prediction as beneficial.
- Experimental Setup: InstructGLM is evaluated primarily for node classification on ogbn-arxiv, Cora, and PubMed, using accuracy and citation-graph text features.The experiments also use self-supervised link prediction as an auxiliary task.
- Main Results: InstructGLM achieves single-model state-of-the-art performance across all three datasets, surpassing representative GNNs and graph Transformer learners.The comparisons include top-ranked methods from the relevant benchmarks.
- Main Results: 1.02% and 2.08% improvements over the best GNN and Transformer methods, respectively, are reported on Cora.These are the reported improvements for the Cora dataset.
- Main Results: 3.18% and 4.87% improvements over the best GNN and Transformer methods, respectively, are reported on PubMed.These are the reported improvements for the PubMed dataset.
- Ablation Study: Multi-hop structural information and self-supervised link prediction both enhance node-classification performance in ablation experiments.Multi-hop prompts incorporate information from neighbors at higher hop levels.
5 Conclusions and Future Work
The paper presents InstructGLM as an initial demonstration of representing graph structure with natural language and instruction-tuning generative LLMs for graph learning. Its results support further exploration of LLMs as foundation models for graph machine learning and extensions to additional tasks and modalities.
- Conclusion: InstructGLM is presented as the first reported attempt to describe graph structure in natural language and instruction-tune generative LLMs for graph learning.The authors identify extensions to additional graph tasks and non-natural-language token modalities as future directions.
Limitations
InstructGLM is constrained by the input-token limits of its underlying LLM, which restrict how many high-order neighbors can appear in one graph-description sentence.
- Limitation: LLM input-token limits can exclude high-order neighbors from a single prompt for large graphs, while enumerating them in multiple sentences rapidly increases training-data volume.The implementation mitigates this constraint by repeatedly sampling multi-hop neighbors until reaching the token limit.
Ethics Statement
The paper states that InstructGLM is intended as a natural-language interface for graph machine-learning tasks and reports no obvious evidence of significant negative societal impacts under appropriate use.
- Ethics Statement: Under normal and appropriate usage circumstances, the authors report no obvious evidence or tendency toward significant negative societal impacts.The stated scope concerns InstructGLM as a natural-language interface for graph machine-learning tasks.
A Implementation Details
The experiments use multi-prompt instruction tuning, aligned node embeddings, specified datasets and splits, grid-searched hyperparameters, and reported compute times.
- The experiments use a multi-prompt instruction-tuning framework and report test accuracy as the evaluation metric.
- A simple MLP aligns default node feature embeddings with natural-language word-token embeddings.
- ogbn-arxiv uses 54%/18%/28% train/validation/test splits, while Cora and PubMed use 60%/20%/20% splits with raw text information.
- PubMed low-label-ratio experiments use 20 labeled training nodes per class, corresponding to a 0.3% label ratio.
- Inference takes about 35ms on Flan-T5-Large and 450ms on Llama-7b per graph prompt sentence.
- Learning rate and batch size are selected by grid search, using AdamW with zero weight decay for 4 epochs.
B Dataset Statistics
The detailed dataset statistics are provided in Table 5.
- Table 5 presents the detailed statistics of the datasets.
C Detailed Discussions on Future Work
The paper identifies future work in richer graph features, stronger prompt generation, model integration, and broader graph and modality coverage.
- Future TAG experiments could use advanced embedding features such as TAPE and SimTeG instead of only default OGB-feature embeddings.
- LLM-based Chain-of-Thought, structure summaries, and data augmentation could generate more powerful instruction prompts.
- InstructGLM could be integrated with GAN or GLEM for multi-model iterative training and with GNNs for knowledge distillation.
- Classic techniques including label reuse, Self-KD, and Correct & Smooth could further enhance model performance.
- The prompt design can extend to diverse graphs and modalities within a unified generative language-modeling framework.
- With slight prompt modifications, the framework supports inductive node classification, graph classification, path prediction, and relation-based question answering.
D Instruction Prompts
The appendix defines a four-digit prompt scheme spanning node classification and link prediction, feature inclusion, hop order, intermediate paths, and target formats.
- D Instruction Prompts: Prompt numbers contain four digits, with the first identifying node classification or link prediction.
- D Instruction Prompts: The second digit indicates whether node or edge features beyond numerical embeddings are included.
- D Instruction Prompts: The third digit specifies whether prompts encode 1-hop, 2-hop, or 3-hop structural information.
- D Instruction Prompts: The fourth digit indicates whether intermediate nodes or paths in high-order connections are included.
- D.1 Node Classification Task-specific prefix:: Node-classification prompts ask for a category using central nodes with neighbors or text features across one to three hops.
- D.1 Node Classification Task-specific prefix:: Some node-classification prompts additionally expose intermediate nodes or paths connecting the central node to higher-order neighbors.
- D.2 Link Prediction Task-specific prefix:: Link-prediction prompts use yes/no or node-id targets to test or identify one- to three-hop connections.
- D.2 Link Prediction Task-specific prefix:: Feature-aware link-prediction prompts attach text features to central, neighboring, candidate, and intermediate nodes.