PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Language Models on Graphs: A Comprehensive Survey

Bowen Jin, Gang Liu, Chi Han, Meng Jiang, Heng Ji, Jiawei Han

arXiv:2312.02783v4cs.CLcs.LG

TL;DR

LLMs are strong at text processing and reasoning, but many applications combine text with graph structure and it remains unclear how well graph reasoning generalizes. This survey categorizes these settings, reviews methods and resources, and identifies future research directions.

  • Problem

    Graph-associated text and graph-based reasoning are not adequately covered by LLMs designed mainly for pure text processing.

  • Method

    The paper systematically reviews graph scenarios, LLM-on-graph techniques, applications, datasets, open-source codebases, and future directions.

  • Results

    The survey organizes LLM-on-graph scenarios into pure graphs, text-attributed graphs, and text-paired graphs, and techniques into Predictor, Encoder, and Aligner categories.

  • Takeaways & Limitations

    The paper provides a structured reference for comparing LLM-on-graph methods and locating available resources and research directions.

  • Takeaways & Limitations

    Current benchmarks lack diverse heterogeneous, spatial-temporal, legal, health, and comprehensive chemistry coverage.

Abstract

from arXiv · show

Large language models (LLMs), such as GPT4 and LLaMA, are creating significant advancements in natural language processing, due to their strong text encoding/decoding ability and newly found emergent capability (e.g., reasoning). While LLMs are mainly designed to process pure texts, there are many real-world scenarios where text data is associated with rich structure information in the form of graphs (e.g., academic networks, and e-commerce networks) or scenarios where graph data is paired with rich textual information (e.g., molecules with descriptions). Besides, although LLMs have shown their pure text-based reasoning ability, it is underexplored whether such ability can be generalized to graphs (i.e., graph-based reasoning). In this paper, we provide a systematic review of scenarios and techniques related to large language models on graphs. We first summarize potential scenarios of adopting LLMs on graphs into three categories, namely pure graphs, text-attributed graphs, and text-paired graphs. We then discuss detailed techniques for utilizing LLMs on graphs, including LLM as Predictor, LLM as Encoder, and LLM as Aligner, and compare the advantages and disadvantages of different schools of models. Furthermore, we discuss the real-world applications of such methods and summarize open-source codes and benchmark datasets. Finally, we conclude with potential future research directions in this fast-growing field. The related source can be found at https://github.com/PeterGriffinJin/Awesome-Language-Model-on-Graphs.

1 INTRODUCTION

LLMs bring strong text modeling and reasoning capabilities to graph-related settings, where structure and text frequently coexist. This survey organizes the emerging field, reviews its techniques and resources, and identifies future directions.

  • Graph-related applications combine textual data with graph structure in academic, e-commerce, and molecular settings.Understanding both modalities supports author, paper, collaboration, recommendation, and molecule modeling.
  • Graph reasoning remains an open extension of LLM text-based reasoning, including connectivity, shortest paths, subgraph matching, and logical rule induction.
  • The survey categorizes graph scenarios as pure graphs, text-attributed graphs, and text-paired graphs, and techniques as LLM Predictor, Encoder, and Aligner.
  • It systematically reviews representative techniques, provides comparisons, and compiles benchmark datasets, open-source codebases, and practical applications.
  • The paper proposes six prospective avenues for future exploration of language models on graphs.

2 DEFINITIONS & BACKGROUND

The survey defines graph and text-related concepts, distinguishes language models from graph Transformers, and summarizes their modeling objectives and architectures. It also describes the lifecycle and scope of LLMs alongside core GNN mechanisms.

  • Graph definitions: A graph is represented as G = (V, E), with nodes V, edges E, and adjacency set N(v) for each node.
  • Graph definitions: Text-attributed, edge-textual, and graph-textual structures attach text respectively to nodes, edges, or entire graphs.
  • Models: LLMs commonly undergo pretraining, optional domain-specific finetuning, and prompting to develop language understanding and completion capabilities.
  • Models: The survey uses “LLM” broadly, encompassing medium-scale pretrained language models such as BERT and large-scale models such as GPT-4.
  • Graph neural networks: GNNs use propagation and aggregation to obtain node representations, while graph-level models apply READOUT functions such as mean or max pooling.
  • Models: Language models process word-token sequences for text encoding or generation, whereas graph Transformers use node tokens and graph-specific positional information.

3 CATEGORIZATION AND FRAMEWORK

The framework organizes graph settings by their relationship to text and organizes methods by the LLM’s role in solving graph-related problems. It distinguishes prediction, encoding, and alignment pathways.

  • Graph scenarios: Pure graphs lack semantically rich text, text-attributed graphs attach text to nodes or edges, and text-paired graphs pair descriptions with entire graph structures.
  • Graph scenarios: Pure graphs test graph reasoning or provide knowledge, while text-attributed graphs support representation learning from textual and structural information.
  • LLM techniques: LLM as Predictor makes the LLM the final component producing representations or predictions, with graph information injected through sequences, architecture changes, or finetuning.
  • LLM techniques: LLM as Encoder uses LLMs as initial text encoders, leaving GNNs as final components for node- or edge-level tasks.

4 PURE GRAPHS

Pure-graph studies examine how LLMs can reason over serialized graph structures, spanning simple connectivity questions to harder algorithmic problems. Results show preliminary graph understanding, but performance degrades on complex or large graphs, while feature encoding can substantially improve task-specific performance.

  • Scope: Pure graphs provide a setting for testing LLM graph reasoning on problems ranging from shortest paths and subgraphs to flow networks.These problems connect graph theory with real-world applications and include both simple and computationally difficult tasks.
  • Graph Representation: Researchers commonly serialize input graphs by verbalizing edge or adjacency lists, or by encoding graph structure into implicit feature sequences.Plain verbalization includes natural-language descriptions of edges and node neighborhoods.
  • Direct Answering: Direct answer generation is simple but fundamentally limited by finite sequence length and computational operations on complex problems such as NP-complete tasks.Most studies report preliminary graph understanding but unsatisfactory performance on more complex problems or larger graphs.
  • Direct Answering: Zero-shot and few-shot prompting can solve easier questions such as connectivity, neighbor identification, and graph-size counting, but struggles with cycle detection and Hamiltonian pathfinding.Additional examples improve performance especially on easier problems, while paraphrasing helps inconsistently.
  • Heuristic Reasoning: Chain-of-thought reasoning improves performance on simpler tasks such as cycle and shortest-path detection, but gains are inconsistent or diminish on more complex problems.Heuristic and step-by-step reasoning approaches attempt to exploit LLM reasoning beyond direct answer mapping.
  • Structured Reasoning: Search-based reasoning can provide interpretable evidence, while algorithmic prompting asks LLMs to recall relevant algorithms and execute reasoning step by step.These approaches reflect complementary attempts to make graph reasoning more structured.
  • Discussion: No consensus exists on graph representation because future solutions must balance computational efficiency against information completeness.The paper suggests that heuristic and algorithmic reasoning may be combined, including better heuristics for efficient search.

5 TEXT-ATTRIBUTED GRAPHS

Text-attributed graph methods combine textual node or edge information with graph structure through three LLM roles: Predictor, Encoder, and Aligner. Predictor approaches inject structure through graph sequences, graph tokens, or architecture changes, while graph-aware fine-tuning incorporates structural signals into language-model representations.

  • Text-attributed graphs associate nodes or edges with text and require models to encode both textual information and graph structure.
  • Graph as Sequence: Graph as Sequence methods inject structure at the input by serializing graphs into sequences, using rule-based or GNN-based Graph2Seq functions.
  • Graph as Sequence: Rule-based methods describe local graph structure with natural-language templates, requiring no extra model parameters and supporting tasks such as node classification and link prediction.
  • Graph as Sequence: GNN-based methods encode ego-graphs into special tokens concatenated with text, capturing hidden structural representations while requiring a bridge between graph and text modalities.
  • Graph-Empowered LLMs: Graph-Empowered LLMs jointly encode text and graph information inside modified architectures, including layer-wise language–graph interaction and asymmetric attention mechanisms.
  • Graph-Aware LLM Finetuning: Graph-aware fine-tuning injects structure through graph-based training signals, enabling text encoders to learn representations that capture graph homophily.
  • Discussion: Open questions include structure-aware graph code sequences, generative graph-empowered architectures, and stronger pretraining for broader task generalization.

5.2 LLM as Encoder

LLM-as-Encoder methods use language models to extract textual features that GNNs combine with graph structure for downstream predictions. The survey organizes these methods around joint or staged training, augmentation, and distillation, while noting representation, efficiency, and task-scope limitations.

  • Core architecture: LLMs encode node or edge text into initial feature vectors, which GNNs propagate and aggregate to produce representations containing textual and structural information.The resulting representations support downstream tasks.
  • 5.2.1 Optimization: One-step training jointly optimizes the LLM-GNN cascaded architecture for downstream tasks.This approach is convenient but may suffer from memory-related efficiency issues and LLM underfitting.
  • 5.2.1 Optimization: Two-step training first adapts LLMs to graph data and then fine-tunes the complete LLM-GNN pipeline.GIANT uses neighborhood prediction to improve node-classification features, while LM-GNN graph-aware pre-fine-tunes the LLM.
  • 5.2.2 Data Augmentation: Data augmentation uses LLM-generated pseudo-labels, prediction text, or explanation text to provide additional training signals for LLM-GNN models.These generated data complement original text or support GNN fine-tuning.
  • 5.2.3 Knowledge Distillation: Knowledge distillation transfers a text-and-structure-aware LLM-GNN teacher into a pure-LLM student to reduce neighbor sampling during inference.AdsGNN preserves topology with an L2 loss, while GraD combines distillation and task objectives.
  • 5.2.4 Discussion: Current LLM-as-Encoder pipelines mainly support representation learning, while generation tasks and efficient inference remain open challenges.Suggested directions include token-level graph encoding and distillation into smaller language models or MLPs.

5.3 LLM as Aligner

LLM-as-Aligner methods train language and graph encoders as interacting components, allowing textual and structural signals to inform one another. The survey distinguishes prediction alignment from latent-space alignment and identifies multimodal and multiplex semantic alignment as unresolved challenges.

  • Overview: LLM-as-Aligner methods use separate LLM and GNN components that mutually exchange textual and structural signals during iterative or parallel training.They are categorized by whether alignment occurs through predictions or latent representations.
  • 5.3.1 LLM-GNN Prediction Alignment: Prediction alignment trains the LLM and GNN iteratively, using labels generated from one modality as pseudo-labels for training the other.LTRN merges pseudo-labels across iterations, while GLEM frames the process as an E-step and M-step.
  • 5.3.2 LLM-GNN Latent Space Alignment: Latent-space alignment connects LLM and GNN representations through cross-modality contrastive learning.ConGrat extends InfoNCE with graph-specific alternatives, and related methods align neighborhood similarity distributions.
  • 5.3.3 Discussion: Existing approaches generally assume a single homogeneous text-graph semantic relation, which does not capture multimodal attributes or multiplex relationships.The survey calls for alignment methods that handle additional modalities and multiple semantic distributions.

6 TEXT-PAIRED GRAPHS

Text-paired graph methods combine graph structures with textual information for molecular and other graph-level tasks. The survey covers sequence-based inputs, graph-empowered architectures, cross-modal alignment, and their limitations.

  • Text-paired graph methods address graph-level tasks by combining graph structures with associated textual information.Examples include molecular graphs paired with toxicity, solubility, and permeability descriptions.
  • Graph as Sequence: Graph-as-sequence methods linearize graphs, tokenize the resulting sequences, and train or fine-tune encoder-only, encoder-decoder, or decoder-only LLMs.Rule-based linearization converts molecular graphs into text sequences that LLMs can process.
  • Graph as Sequence: MolT5 formulates molecule-text translation as a multilingual problem using C4 text data and one million SMILES for pretraining.Text+Chem T5 extends the input and output domains to SMILES and texts, while ChatMol supports multi-turn molecule design.
  • Graph-Empowered LLMs: Graph-empowered LLMs jointly encode graph structures and text by modifying Transformer components such as positional encodings and attention.GIMLET represents nodes and text tokens in one sequence and uses graph-aware relative distances based on shortest paths and edge features.
  • Graph-Empowered LLMs: Cross-attention methods interact graph and text representations within encoder layers to integrate information across modalities.Text2Mol and Prot2Text implement graph-text interaction in hidden layers using trainable transformations.
  • Discussion: Rule-based linearization can limit LLM performance because newer encodings may be absent from pretraining and sequence order can break graph permutation invariance.The resulting sequence prior can reduce task difficulty by imposing an order, but it may also limit the expressiveness of the learned representation.
  • Alignment: Contrastive alignment methods map GNN and LLM representations into a unified space and distinguish paired graph-text inputs from negative pairs.MoleculeSTM, MolFM, and GIT-Mol use contrastive objectives with negative sampling, while CLAMP incorporates task labels.
  • Discussion: Future graph decoders could use GNNs or diffusion models instead of relying predominantly on text-based generation of linearized graph structures.The survey notes that sequence-based graph generation may be sensitive to linearization order.

7 RESOURCES AND APPLICATIONS

The survey compiles datasets, software resources, evaluation practices, and applications for LLMs on pure, text-attributed, and text-paired graphs. These resources span graph reasoning, prediction, recommendation, molecular modeling, and scientific discovery.

  • The survey organizes datasets across pure graphs, text-attributed graphs, and text-paired graphs.The collections are presented in Tables 5, 2, and 3, respectively.
  • Datasets: Pure-graph benchmarks evaluate reasoning problems such as connectivity, shortest path, and graph diameter, while GraphQA adds real-world graph descriptions.NL-Graph, LLMtoGraph, and GUC study standard graph reasoning problems.
  • Datasets: Text-attributed graph datasets cover academic, e-commerce, book, social-media, and Wikipedia domains with classification, prediction, regression, and recommendation tasks.Common metrics include Accuracy, Macro-F1, Micro-F1, MRR, NDCG, and Hit Ratio.
  • Datasets: Text-paired graph evaluations use AUC for graph classification, MAE, RMSE, and R2 for regression, BLEU for text generation, and validity, novelty, and uniqueness for molecule generation.The dataset collection includes graph-text pairs, graphs, and assays, with task abbreviations defined in Table 3.
  • Evaluation: Heuristic molecule-generation evaluations can be problematic because unintended modes may produce artifacts such as superfluous carbon atoms.The passage contrasts these methods with efficient but less accurate alternatives.
  • Resources: Open-source resources include HuggingFace Transformers and Datasets, Fairseq, and PyTorch Geometric for language-model and graph-learning workflows.These libraries support model use, dataset access, evaluation, and GNN implementations.
  • Applications: Applications include virtual screening, synthesis planning, academic recommendation and classification, user-behavior modeling, and education tasks such as knowledge tracing.Synthesis planning combines molecular graphs with textual reaction conditions, additives, and solvents.

8 FUTURE DIRECTIONS

The survey identifies benchmark coverage and multimodal foundation modeling as major future directions. It calls for broader graph settings and models that can jointly handle graphs, text, and images while balancing unification with domain customization.

  • Better Benchmark Datasets: Pure-graph benchmarks should expand beyond homogeneous graphs to heterogeneous and spatial-temporal graphs, while text-attributed benchmarks should cover legal, health, and other domains.The survey also notes a lack of comprehensive text-paired graph benchmarks.
  • Multi-Modal Foundation Models: Multimodal foundation models could jointly encode texts, graphs, and images for objects such as molecules and products.The survey frames this as useful but challenging and questions whether unified architectures can suit different data types.
  • Multi-Modal Foundation Models: Future work must weigh unified foundation models against domain-customized architectures to avoid unnecessary economic and intellectual costs.The survey presents this as an open design question rather than a settled architectural choice.

9 CONCLUSION

The paper provides a comprehensive review of large language models on graphs. It categorizes scenarios and techniques, surveys resources and applications, and proposes future research directions.

  • The survey categorizes graph scenarios, reviews methods within each scenario, summarizes datasets and open-source codebases, and discusses applications and future directions.Its scope covers pure graphs, text-attributed graphs, and text-paired graphs.

Training & Inference Framework with LLMs

The survey describes two main ways to apply language models on graphs: pretraining followed by finetuning, or pretraining followed by prompting. It also distinguishes full, efficient, and instruction tuning, while organizing graph methods and problems across several task settings.

  • Training and inference paradigms: Pretraining-then-finetuning is typically used for medium-scale language models, whereas pretraining-then-prompting is typically used for large-scale models.Pretraining initializes language understanding and inference ability for downstream tasks.
  • Finetuning strategies: Full finetuning updates all language-model parameters but can incur heavy computational overload and overfitting.
  • Finetuning strategies: Efficient finetuning updates only a parameter subset, including prompt tuning, prefix tuning, adapters, LoRA, graph neural prompts, and graph-enhanced prefixes.
  • Finetuning strategies: Instruction tuning uses downstream task instructions to encourage generalization to unseen inference tasks and can combine with either full or efficient finetuning.It has been applied to node classification, link prediction, and graph-level tasks.
  • Training and inference paradigms: Prompting solves downstream tasks without updating model parameters by formatting test samples as natural-language sequences with in-context demonstrations.The survey also notes chain-of-thought and tree-of-thought prompting as follow-up techniques.
  • Graph-task organization: The survey catalogs LLM reasoning methods and problems for pure graphs, alongside model collections for text-attributed and text-paired graphs.The pure-graph problem collection includes standard algorithmic complexity information, while text-paired methods include linearized, vectorized, predictor, and aligner approaches.
Loading 2312.02783v4…