PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey of Graph Meets Large Language Model: Progress and Future Directions

Yuhan Li, Zhixun Li, Peisong Wang, Jia Li, Xiangguo Sun, Hong Cheng, Jeffrey Xu Yu

arXiv:2311.12399v4cs.LGcs.CLcs.SI

TL;DR

Graph–LLM research lacks a systematic review organized around how LLMs enhance graph learning. This survey proposes a role-based taxonomy, reviews representative methods, and identifies limitations and future directions for graph–text integration.

  • Problem

    Existing graph–LLM research lacks comprehensive coverage and a taxonomy focused on how LLMs enhance graphs when graph and text modalities coexist.

    Source: §1 · p. 2

  • Method

    The survey categorizes graph–text techniques as enhancer, predictor, or alignment component and systematically reviews representative studies within those categories.

    Source: §1 · p. 2, §7 · p. 10

  • Results

    The survey synthesizes existing strategies for adapting LLMs to graphs and identifies limitations alongside future research directions.

    Source: §1 · p. 2, §7 · p. 10

  • Takeaways & Limitations

    The taxonomy and review clarify advancements and challenges in graph learning with LLMs.

    Source: §7 · p. 10

    Current studies face unresolved issues including applicability across graph domains, data leakage, transferability, efficiency, and theoretical expressive ability.

    Source: §6 · p. 9

Abstract

from arXiv · show

Graph plays a significant role in representing and analyzing complex relationships in real-world applications such as citation networks, social networks, and biological data. Recently, Large Language Models (LLMs), which have achieved tremendous success in various domains, have also been leveraged in graph-related tasks to surpass traditional Graph Neural Networks (GNNs) based methods and yield state-of-the-art performance. In this survey, we first present a comprehensive review and analysis of existing methods that integrate LLMs with graphs. First of all, we propose a new taxonomy, which organizes existing methods into three categories based on the role (i.e., enhancer, predictor, and alignment component) played by LLMs in graph-related tasks. Then we systematically survey the representative methods along the three categories of the taxonomy. Finally, we discuss the remaining limitations of existing studies and highlight promising avenues for future research. The relevant papers are summarized and will be consistently updated at: https://github.com/yhLeeee/Awesome-LLMs-in-Graph-tasks.

1 Introduction

Graphs represent relationships across many real-world domains, while LLMs contribute strong textual understanding; integrating them with GNNs combines structural and contextual information for graph tasks.

1 Introduction

  • Graph data underlies citation, social, and molecular networks, and GNNs learn node-, edge-, or graph-level representations through recursive message passing.These representations support different downstream graph-related tasks.Source: §1 · p. 1
  • Figure 1 reports successful graph–LLM integration across diverse graph domains and downstream tasks.The supplied figure passage establishes broad task and domain coverage but does not provide numerical comparisons.Source: Figure 1 · p. 1
  • LLMs have achieved broad downstream-task performance and are increasingly being extended beyond text to multimodal data, including graphs.Examples include Transformers, BERT, and GPT and their variants.Source: §1 · p. 1
  • LLMs and GNNs complement one another by combining textual understanding with graph-structural modeling for more comprehensive graph learning.GNNs capture structural relationships but may use semantically constrained node features, whereas LLMs encode text but often struggle with graph structure.Source: §1 · pp. 1–2
  • The survey organizes representative approaches by whether LLMs enhance GNNs, act as predictors, or align with GNNs, illustrated by TAPE, InstructGLM, and MoleculeSTM.TAPE improves initial node embeddings with LLM-generated semantic knowledge; InstructGLM replaces the GNN predictor; MoleculeSTM aligns graph and language representations.Source: §1 · p. 2

2 Preliminary

The preliminary section introduces GNN message passing, defines the survey’s broad LLM usage, and presents a taxonomy for graph–text methods based on LLM roles.

2.1 Graph Neural Networks

  • GNNs iteratively aggregate neighbor information and update node features, while graph pre-training and prompting reduce annotation demands and reformulate downstream tasks.Contrastive and generative pre-training are described, alongside prompting schemes such as GPPT and All in One.Source: §2.1 · p. 2

2.2 Large Language Models

  • The survey defines LLMs broadly to include billion-parameter LLMs and million-parameter PLMs because GNN–LLM integration often does not require very large language models.LLMs are described as models pretrained on substantial data, while PLMs are earlier, moderately sized pretrained models.Source: §2.2 · p. 2

2.3 Proposed Taxonomy

  • The survey proceeds by systematically reviewing methods under the three principal taxonomy categories.Figure 2 presents the taxonomy and representative examples.Source: Figure 2 · p. 3, §2.3 · p. 3

2.2 Large Language Models

  • LLMs are divided into non-autoregressive and autoregressive families, associated respectively with masked-language modeling and next-token prediction.BERT, SciBERT, and RoBERTa exemplify the former, while Flan-T5, ChatGLM, GPT-3, PaLM, Galactica, and LLaMA exemplify the latter.Source: §2.2 · p. 3

2.3 Proposed Taxonomy

  • The proposed taxonomy classifies graph–text techniques as LLM as Enhancer, LLM as Predictor, and GNN-LLM Alignment, with an additional Others category for rare LLM involvement.Enhancers improve GNN classification, predictors use graph structure for predictions, and alignment methods semantically strengthen GNNs.Source: §2.3 · p. 3
  • Representative enhancer methods use LLMs as annotators, architecture-search controllers, or sample generators to augment GNN training.LLM-GNN selects nodes for annotation, GPT4GNAS explores GNN architectures, and ENG generates labeled training samples.Source: §2.3 · p. 3

3 LLM as Enhancer

LLM-as-enhancer methods improve graph learning by enriching node text or directly producing embeddings before GNN processing. The survey distinguishes explanation-based and embedding-based enhancement, while noting flexibility alongside scalability and access constraints.

3 LLM as Enhancer

  • LLM-as-enhancer methods improve node representations by enriching text attributes or producing embeddings that GNNs use for graph learning.The approaches target text-attributed graphs and attach enhanced embeddings to graph structure or downstream classifiers.Source: §3 · p. 3

3.1 Explanation-based Enhancement

  • Explanation-based enhancement prompts LLMs to generate explanations, entities, or pseudo labels, which are encoded with the original text before GNN processing.TAPE, KEA, LLM4Mol, and LLMRec instantiate this pipeline for paper, molecular, and recommendation graphs.Source: §3.1 · p. 4

3.2 Embedding-based Enhancement

  • Embedding-based enhancement directly uses LLM-produced text embeddings as initial node embeddings and incorporates graph structure during language-model training or fine-tuning.These methods include structural objectives such as link prediction, graph adapters, relation tokens, and textualized random walks.Source: §3.2 · p. 4
  • Embedding-based methods require embedding-visible or open-source LLMs because closed-source systems restrict access to parameters and output embeddings.GALM pre-trains language models and a GNN aggregator on a large graph corpus before downstream fine-tuning.Source: §3.2 · p. 4

3.3 Discussions

  • LLM-as-enhancer approaches capture textual and structural information flexibly, but explanation-based methods may require N API queries for a graph with N nodes.Their plug-and-play separation of GNNs and LLMs can also enable assistance from closed-source LLMs.Source: §3.3 · p. 5

4 LLM as Predictor

LLM-as-predictor methods use language models to make graph-task predictions by converting structure into text or injecting GNN-derived structural representations. Flattening offers broad textual processing but limited context, whereas GNN-based integration is more efficient yet harder to train jointly.

4 LLM as Predictor

  • LLM-as-predictor methods formulate graph classification and reasoning within a generative paradigm, using flatten-based or GNN-based structural integration.The central challenge is representing graph structures in a form that LLMs can process.Source: §4 · p. 5

4.1 Flatten-based Prediction

  • Flatten-based prediction transforms graph structures into node or token sequences and parses LLM outputs into predicted labels.The pipeline applies Flat(·) to obtain Gseq, then Parse(·) to extract the prediction.Source: §4.1 · p. 5

LLM Tuning

  • Parsing commonly uses regular expressions or constrained multiple-choice outputs, while methods such as GIMLET and InstructGLM can fine-tune LLMs to emit labels directly.Some works set decoding temperature to 0 to reduce prediction variance.Source: §4.1 · p. 5

LLM Frozen

  • Flattening strategies include graph languages, syntax trees, SMILES, node and edge lists, adjacency representations, natural narration, and multimodal descriptions.Methods vary according to whether LLM parameters remain frozen or are tuned, with some using distance-based or hop-scaled prompts.Source: LLM Frozen · pp. 5–6, LLM Tuning · p. 6

4.2 GNN-based Prediction

  • GNN-based prediction feeds structure-aware embeddings from GNNs into LLMs, using projectors, prefixes, graph tokens, or layer-wise injection to combine graph and text information.This integration can standardize prediction outputs through training and includes Q-Former, linear, and direct-injection designs.Source: §4.2 · p. 6

4.3 Discussions

  • Flatten-based prediction may be more effective, whereas GNN-based prediction tends to be more efficient; flattening limits long-range dependencies and GNN integration faces vanishing gradients in deep transformers.Direct LLM predictors show strong textual processing and remarkable zero-shot performance compared with traditional GNNs.Source: §4.3 · p. 6

5 GNN-LLM Alignment

GNN-LLM alignment coordinates graph and text embedding spaces through symmetric or asymmetric designs, preserving modality-specific functions while enabling joint graph-text learning. Symmetric methods treat both modalities equally, whereas asymmetric methods use graph encoders to enhance LLMs; data scarcity limits alignment applicability.

5 GNN-LLM Alignment

  • Symmetric alignment coordinates graph and text encoders equally, while asymmetric alignment prioritizes one modality to assist or enhance the other.The survey further divides asymmetric methods into graph-nested transformers and graph-aware distillation.Source: §5 · p. 6, §5.1 · p. 6, §5.2 · p. 7

5.1 Symmetric

  • Two-tower symmetric models separately encode graphs and text, then commonly use contrastive learning because simple concatenation produces loosely coupled multimodal representations.Feature extraction precedes a modified InfoNCE objective, and both encoders are updated through backpropagation.Source: §5.1 · pp. 6–7
  • Contrastive alignment spans molecular, social, knowledge, citation, and recommendation graphs, with methods adding cross-modal attention, multilevel objectives, prompts, or node- and neighborhood-level alignment.Text2Mol uses a transformer decoder for early fusion, while MoMu, MoleculeSTM, ConGraT, RLMRec, G2P2, and GRENADE extend contrastive alignment across domains and granularities.Source: §5.1 · p. 7
  • Iterative symmetric alignment allows modalities to interact repeatedly, as GLEM uses an Expectation-Maximization framework in which one encoder generates pseudo-labels for the other.Source: §5.1 · p. 7

5.2 Asymmetric

  • Graph-nested transformers insert GNN processing into transformer layers, passing node-aware representations through the LLM and supporting classification, retrieval, reranking, and link prediction.Graph-former gathers node embeddings for graph transformation, while Patton adds network-contextualized masked language modeling and masked node prediction for text-rich graphs.Source: §5.2 · pp. 7–8
  • Graph-aware distillation uses GNNs as teachers to transfer structural information into LLMs, after which the auxiliary GNN can be discarded for more scalable inference.GRAD iteratively updates both models, while THLM combines heterogeneous topology learning with masked language modeling before downstream fine-tuning.Source: §5.2 · p. 8

5.3 Discussions

  • Alignment methods improve modality-specific representations, but only a few graph datasets—particularly molecular datasets—contain native graph-text pairs, constraining applicability.Source: §5.3 · p. 8

6 Future Directions

The survey identifies future directions spanning broader graph coverage, trustworthy evaluation, transferability, explainability, efficiency, expressive power, and agentic use of LLMs. These directions address settings where graphs lack text, benchmarks may leak training data, and LLM computation or theoretical guarantees remain limited.

Dealing with non-TAG

  • Many real-world graphs lack node text, and describing structures such as traffic or superpixel graphs in human-readable language may be unsuitable or yield suboptimal performance.The survey highlights constructing graph foundation models without relying on rich textual attributes as a research direction.Source: §6 · pp. 8–9

Dealing with data leakage

  • Pretrained LLMs may have memorized benchmark test data, especially in citation networks, undermining evaluation reliability and motivating fair, systematic, comprehensive benchmarks.Newer post-cutoff citation datasets reduce leakage concerns but remain limited to citation domains with weak graph-structural impact.Source: §6 · p. 9

Improving transferability

  • Graph transfer remains difficult across datasets and domains because graphs differ in size, connectivity, node and edge types, and topology, despite LLMs’ zero- and few-shot language abilities.OFA provides a unified text-based cross-domain approach, but transferability remains an open research topic.Source: §6 · p. 9

Improving explainability

  • Further work should improve explainability through prompting methods such as in-context learning and chain-of-thought reasoning.Source: §6 · p. 9

Improving efficiency

  • LLM-based graph learning can be slower, more expensive, and more hardware-intensive than dedicated GNNs, motivating parameter-efficient and resource-conscious adaptation methods.The survey cites API billing costs, local training and inference demands, and methods including LoRA and prefix tuning.Source: §6 · p. 9

Analysis and improvement of expressive ability

  • Theoretical work should test whether LLMs surpass GNN or 1-WL expressive power and determine how to provide permutation equivariance for graph learning.Source: §6 · p. 9

LLMs as agent

  • LLM agents could decompose complex graph tasks into subtasks and select tools such as GNNs for each subtask, potentially improving performance and generalizability.Source: §6 · p. 9

7 Conclusion

This survey reviews strategies for adapting LLMs to graphs through a taxonomy based on enhancer, predictor, and alignment roles. It synthesizes representative studies, discusses limitations, and identifies future directions for graph learning with LLMs.

7 Conclusion

  • The survey categorizes graph-LLM techniques by whether LLMs act as enhancers, predictors, or alignment components.It systematically reviews representative studies within this taxonomy and discusses limitations and future research directions.Source: §7 · p. 10
Loading 2311.12399v4…