PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Language Models on Graphs: A Comprehensive Survey

Bowen Jin, Gang Liu, Chi Han, Meng Jiang, Heng Ji, Jiawei Han

arXiv:2312.02783v4cs.CLcs.LG

TL;DR

LLMs are primarily designed for text, while many applications combine graph structure with text and graph-based reasoning remains underexplored. This survey organizes graph settings and LLM techniques, and reports that LLMs handle easier graph questions but struggle with more complex ones.

  • Problem

    LLMs are primarily designed for text, while many real-world applications combine graph structure with text, making their use across these settings an important research question.

    Source: §1 · p. 1

  • Method

    The survey categorizes graph settings as pure graphs, text-attributed graphs, and text-paired graphs, and organizes techniques as LLM Predictor, Encoder, or Aligner.

    Source: §1 · p. 2, §9 · p. 16

  • Results

    LLMs answer easier graph questions such as connectivity and neighbor identification but fail on cycle detection and Hamiltonian pathfinding; few-shot examples improve easier problems, though performance remains unsatisfactory.

    Source: §4.1 · p. 6

  • Takeaways & Limitations

    The review identifies graph-based reasoning as important for real-world applications and as an area with open questions.

    Source: §4 · p. 5, §5.1.4 · p. 8

    Existing graph-as-sequence methods are mainly rule-based or GNN-based; the former rely on natural-language graph descriptions, while the latter require a trained GNN component.

    Source: §5.1.4 · p. 8

Abstract

from arXiv · show

Large language models (LLMs), such as GPT4 and LLaMA, are creating significant advancements in natural language processing, due to their strong text encoding/decoding ability and newly found emergent capability (e.g., reasoning). While LLMs are mainly designed to process pure texts, there are many real-world scenarios where text data is associated with rich structure information in the form of graphs (e.g., academic networks, and e-commerce networks) or scenarios where graph data is paired with rich textual information (e.g., molecules with descriptions). Besides, although LLMs have shown their pure text-based reasoning ability, it is underexplored whether such ability can be generalized to graphs (i.e., graph-based reasoning). In this paper, we provide a systematic review of scenarios and techniques related to large language models on graphs. We first summarize potential scenarios of adopting LLMs on graphs into three categories, namely pure graphs, text-attributed graphs, and text-paired graphs. We then discuss detailed techniques for utilizing LLMs on graphs, including LLM as Predictor, LLM as Encoder, and LLM as Aligner, and compare the advantages and disadvantages of different schools of models. Furthermore, we discuss the real-world applications of such methods and summarize open-source codes and benchmark datasets. Finally, we conclude with potential future research directions in this fast-growing field. The related source can be found at https://github.com/PeterGriffinJin/Awesome-Language-Model-on-Graphs.

1 INTRODUCTION

The survey examines how LLMs can address graph-related scenarios and challenges beyond pure-text processing. It organizes this emerging field by graph–text relationship and LLM role, then reviews techniques, resources, applications, and future directions.

Introduction

  • Text-associated graphs require LLMs to handle structure, while graph reasoning on pure graphs remains an open challenge spanning connectivity, shortest paths, subgraph matching, and rule induction.The introduction motivates these challenges with text-attributed academic networks and text-paired molecules.Source: §1 · pp. 1–2
  • The survey classifies scenarios as pure graphs, text-attributed graphs, and text-paired graphs, and techniques as LLM as Predictor, Encoder, or Aligner.Predictors produce final answers, Encoders extract features for GNNs, and Aligners align LLM and GNN latent spaces.Source: §1 · p. 2
  • The survey reviews representative models across graph scenarios, illustrates their techniques, and compares them; earlier surveys cover graph foundation models [20], LLMs and knowledge graphs [21], or LLM-on-graph techniques [202] [203].The authors position their survey as covering applications across diverse graph scenarios, an area they say prior surveys do not treat systematically.Source: §1 · p. 2
  • The paper collects benchmark datasets, open-source codebases, and practical applications of LLMs on graphs.It presents these resources as support for researchers entering the field.Source: §1 · p. 2
  • The survey examines foundational principles and proposes six prospective avenues for future research on LLMs on graphs.Source: §1 · p. 2

2 DEFINITIONS & BACKGROUND

The section defines graph structures and how text may be attached to nodes, edges, or whole graphs, then establishes the language- and graph-model background used throughout the survey.

2.1 Definitions

  • A graph is G = (V, E), with nodes V, edges E, and N(v) denoting the nodes adjacent to v.A directed edge from vj to vi is represented as eij = (vi, vj).Source: §2.1 · p. 2
  • Text may be associated with each node, each edge, or an entire graph, yielding text-attributed, textual-edge, and text-paired graphs, respectively.Node-level text can represent article content, edge-level text can represent messages, and graph-level text can describe a molecule.Source: §2.1 · pp. 2–3

2.2 Background

  • The survey treats “large” as an evolving direction rather than a fixed size threshold, including medium-scale PLMs such as BERT and large-scale LMs such as GPT-4 [21].LLMs may be pretrained, finetuned on domain-specific data, aligned to human preferences, and prompted for downstream tasks.Source: §2.2 · p. 3
  • GNNs such as GCN, GraphSAGE, and GAT use propagation and aggregation for node representations, while graph-level tasks use READOUT pooling and graph Transformers add global attention.Graph Transformers incorporate graph inductive biases through positional or structural encodings, message-passing layers, or attention-efficiency improvements, and are described as state of the art for many pure graph problems.Source: §2.2 · p. 3
  • Language models process word tokens and text, whereas graph Transformers process node tokens and graph structure, using graph distances or Laplacian eigenvalues for positional encoding.When a Transformer pretrained on text uses both word and node tokens, the survey calls it a language model.Source: §2.2 · pp. 3–4

3 CATEGORIZATION AND FRAMEWORK

The survey organizes graph settings for language models into three scenario types and classifies graph techniques by whether LLMs predict, encode text for GNNs, or align with GNNs.

3.1 Categorization of Graph Scenarios with LLMs

  • Pure graphs lack semantically rich text and are used to test LLM graph reasoning or provide knowledge that can alleviate hallucination.Traffic and power transmission graphs are examples.Source: §3.1 · p. 4
  • Text-attributed graphs associate semantically rich text with nodes or edges, while text-paired graphs provide descriptions for an entire graph, such as a molecule.Text-attributed settings include academic, e-commerce, social, and legal case networks; text complements graph structure in understanding molecules.Source: §3.1 · p. 4

3.2 Categorization of LLMs on Graph Techniques

  • LLM-as-Predictor methods make the LLM the final prediction or representation component, using graph-token sequences, graph-empowered architectures, or graph-supervised finetuning.Graph-token sequences may be natural-language descriptions or graph-encoder representations; graph-empowered methods modify the Transformer architecture for joint text and graph encoding.Source: §3.2 · pp. 4–5
  • LLM-as-Encoder methods feed LLM-produced node or edge text embeddings to GNNs, which encode graph structure and produce downstream representations.These methods face convergence, sparse-data, and efficiency issues, addressed through optimization, data augmentation, and knowledge distillation.Source: §3.2 · p. 5
  • LLM-as-Aligner methods combine LLM text encoding and GNN graph encoding through prediction alignment or latent-space alignment.Prediction alignment iteratively trains one modality using pseudo-labels generated by the other; latent-space alignment uses contrastive learning on text and graph embeddings.Source: §3.2 · p. 5

4 PURE GRAPHS

The section surveys how LLMs are applied to pure-graph reasoning, from graph serialization and direct answering to heuristic and algorithmic methods. It reports preliminary graph understanding but persistent limits on complex reasoning, alongside open questions about representation and method design.

4 PURE GRAPHS

  • Pure-graph reasoning addresses algorithmic problems with real-world connections, while LLMs’ autoregressive processing can produce indefinite sequences of intermediate steps, [48], –, [191].The surveyed problems range from connectivity, shortest paths, and cycle detection to maximum flow and Hamiltonian pathfinding; graph structure is commonly serialized into input sequences.Source: §4 · p. 5
  • Graph inputs are represented through verbalized structures or implicit feature sequences, while paraphrasing aims to make verbalized inputs more natural or concise [41], [122]–[130].Format-Explanation and Role Prompting improve performance on some problems but not systematically [124]; grounding graphs in real-world scenarios can also help inconsistently [129]. Feature-sequence approaches typically train a graph encoder and fine-tune the LLM [41].Source: §4 · p. 5

4.1 Direct Answering

  • LLMs answer simpler graph questions but struggle with complex reasoning, especially on larger graphs [41], [122]–[124], [126], [129].They handle connectivity, neighbor identification, and graph-size counting, but fail on cycle detection and Hamiltonian pathfinding; few-shot examples help more on easier problems. Feature-sequence encoding with fine-tuning improved results on substructure counting, maximum triplet sum, shortest path, and bipartite matching [41].Source: §4.1 · pp. 5–6

4.2 Heuristic Reasoning

  • Heuristic reasoning can help on simpler graph problems, but chain-of-thought gains are inconsistent or diminish on harder tasks [41], [122]–[124], [126], [129], [130].Build-a-Graph and Context-Summarization retrieve relevant subgraphs as evidence, while RoG plans relation paths and iterative methods retrieve neighbors during search [122], [124], [127], [128], [130]. Search-based approaches offer interpretable evidence and address scalability when knowledge from multiple graphs is available.Source: §4.2 · p. 6

4.3 Algorithmic Reasoning

  • Algorithmic Prompting does not consistently outperform heuristic reasoning, while Graph-ToolFormer executes LLM-generated API calls on external graphs [122], [125].Graph-ToolFormer is suited to converting real-world tasks into pure-graph reasoning and has demonstrated efficacy in knowledge graphs, social networks, and recommendation systems [125].Source: §4.3 · p. 6

4.4 Discussion

  • No best practice for graph representation has emerged; future methods must balance computational efficiency with information completeness [208],.The reasoning methods are not mutually exclusive, and the section suggests using LLM-generated heuristics to improve the efficiency of algorithmic search.Source: §4.4 · p. 6

5 TEXT-ATTRIBUTED GRAPHS · 5.1 LLM as Predictor

For text-attributed graphs, LLM methods must account for both node or edge text and graph structure. The survey organizes LLM-as-Predictor methods by how they inject structure and identifies open directions for graph-aware sequences and broader graph-empowered models.

5 TEXT-ATTRIBUTED GRAPHS

  • Text-attributed graph learning must encode both node or edge text and graph structure, and the survey groups LLM methods as Predictor, Encoder, or Aligner.Academic and legal case networks are examples of text-attributed graphs.Source: §5 · p. 6

5.1 LLM as Predictor

  • LLM-as-Predictor methods divide into Graph as Sequence, Graph-Empowered LLMs, and Graph-Aware LLM finetuning, according to whether structure enters through input, architecture, or loss.Sequence methods serialize graphs with text; empowered models modify Transformer architecture; finetuning methods use graph supervision to learn graph-contextualized representations.Source: §5.1 · p. 7

5.1.1 Graph as Sequence

  • Rule-based Graph as Sequence methods describe local graph structure in natural-language prompts; the structural information helps node classification when node text is scarce.InstructGLM [45] uses ego-graph templates of up to three hops for node classification and link prediction, while GraphText [64] uses syntax trees to transfer structure into text.Source: §5.1.1 · p. 7
  • GNN-based Graph as Sequence methods encode ego-graphs as special tokens concatenated with text, but must bridge graph and text modalities.GNP [40] projects graph tokens into text-token space; GraphGPT [44] trains a text-grounded GNN using a text encoder and contrastive learning, while DGTL [75] and METERN [74] use other graph-token mechanisms.Source: §5.1.1 · p. 7

5.1.2 Graph-Empowered LLMs

  • Graph-Empowered LLMs add virtual structure tokens within Transformer layers and modify attention to jointly encode text and graph structure.GreaseLM uses interacting language and graph components; DRAGON pretrains it with unsupervised signals, and GraphFormers and Heterformer aggregate or add virtual neighbor tokens. Edgeformers applies joint encoding to textual edges.Source: §5.1.2 · pp. 7–8

5.1.3 Graph-Aware LLM Finetuning

  • Graph-Aware LLM Finetuning trains text-only language models with graph structure signals to learn representations that capture graph homophily, although encoding itself considers only text.Two-tower methods include SPECTER, which uses citation-based positive pairs and triplet loss, and other methods use alternative pair sampling or objectives, including SciNCL, Touchup-G [53], TwHIN-BERT, MICoL, E2EG [59], and WalkLM.Source: §5.1.3 · p. 8

5.1.4 Discussion

  • Future work could use code LLMs to produce structure-aware graph sequences for zero-shot inference and develop graph-empowered models for generation and more diverse network settings.The survey notes that current graph-empowered methods mainly target representation learning with encoder-only LLMs, and that pretraining has focused chiefly on homogeneous text-attributed networks rather than heterogeneous, dynamic, or textual-edge networks.Source: §5.1.4 · p. 8

5.2 LLM as Encoder

LLM-as-encoder methods use LLMs to extract textual features that GNNs combine with graph structure for downstream tasks. The section describes training, data augmentation, and distillation approaches, as well as limits in task scope and efficiency.

5.2 LLM as Encoder

  • LLM-GNN cascades combine textual node features from LLMs with graph structure in GNN representations for downstream tasks.The LLM output serves as an initial node feature vector; the resulting representation contains both textual and structural information.Source: §5.2 · pp. 8–9

5.2.1 Optimization

  • One-step training jointly optimizes the LLM-GNN pipeline, while two-step training adapts the LLM first and then fine-tunes the cascade.One-step approaches include TextGNN, AdsGNN, and GNN-LM, but may face efficiency and local-minimum issues. Two-step methods such as GIANT, LM-GNN, and SimTeG [34] can improve text representations and alleviate insufficient LLM training, but cost more time and computation; LoRA can alleviate overfitting.Source: §5.2.1 · p. 9

5.2.2 Data Augmentation

  • Data augmentation uses LLM-generated pseudo-labels, prediction and explanation text, or labeled nodes and edges to support GNN learning.LLM-GNN [63] uses pseudo-labels, TAPE [69] adds generated text, and ENG [70] generates labeled nodes and connects them to other nodes for semi-supervised node classification.Source: §5.2.2 · p. 9

5.2.3 Knowledge Distillation

  • Knowledge distillation transfers a trained LLM-GNN cascade to an LLM student that avoids neighbor sampling at inference.AdsGNN uses an L2 loss to preserve topology, while GraD uses distillation and task objectives to optimize the teacher and transfer its capability.Source: §5.2.3 · p. 9

5.2.4 Discussion

  • Encoder cascades mainly support representation learning, while neighbor sampling and encoding make inference costly even after distillation to an LLM.For generation tasks, the discussion proposes encoding LLM-generated token-level representations with a GNN and designing suitable decoders. To improve efficiency, it suggests distillation to a smaller LM or MLP; GNN-to-MLP distillation methods are cited as precedent.Source: §5.2.4 · pp. 9–10

5.3 LLM as Aligner

LLM-as-aligner methods pair text-encoding LLMs with structure-encoding GNNs, which can exchange signals and are trained iteratively or in parallel. The survey distinguishes prediction alignment from latent-space alignment and notes limitations of homogeneous text-graph alignment.

5.3.1 LLM-GNN Prediction Alignment

  • LLM-GNN prediction alignment uses mutually generated pseudo-labels to train text and structure encoders iteratively toward joint encoding.LTRN merges LLM- and GNN-generated pseudo-labels across iterations; GLEM frames the process as a pseudo-likelihood variational framework with an LLM E-step and GNN M-step.Source: §5.3.1 · p. 10

5.3.2 LLM-GNN Latent Space Alignment

  • Latent-space alignment connects LLM and GNN encodings through cross-modality contrastive learning and related knowledge-alignment objectives.ConGrat [52] extends InfoNCE with graph-specific likely text-node choices, while GRENADE aligns neighborhood-similarity distributions using KL divergence. G2P2 adds text-summary and node-summary interactions, using label texts for zero-shot and soft prompts for few-shot classification; THLM [32] pretrains with a heterogeneous GNN and supports downstream fine-tuning.Source: §5.3.2 · p. 10

5.3.3 Discussion

  • Homogeneous text-graph alignment may miss multimodal attributes and multiplex semantic relations, which a single alignment cannot comprehensively capture [74].The discussion identifies images as a possible co-occurring modality and calls for research on aligning multimodal attributes in graph scenarios.Source: §5.3.3 · p. 10

6 TEXT-PAIRED GRAPHS

This section surveys LLM methods for graphs paired with graph-level text, especially molecular graphs, covering prediction, graph–text alignment, and limitations and opportunities in current approaches.

6.1 LLM as Predictor

  • Text-paired graph methods either linearize graphs for existing LLMs or design graph-empowered architectures that jointly encode graph structure and text.The survey also organizes the methods as LLMs used as predictors and methods that align GNNs with LLMs.Source: §6 · p. 10, §6.1 · p. 10, §6.1.1 · p. 11

6.1.1 Graph as Sequence

  • Molecular graph sequences use notations such as SMILES and InChI, but sequence-based models can produce invalid structures; SELFIES yields valid molecular graphs.The pipeline linearizes, tokenizes, then trains or fine-tunes encoder-only, encoder-decoder, or decoder-only LLMs. Models support classification, regression, molecule generation, captioning, and molecule-text translation; in-context approaches include LLM-ICL and MolReGPT.Source: §6.1.1 · p. 11

6.1.2 Graph-Empowered LLMs

  • Graph-empowered LLMs jointly encode graph and text, using graph-aware positional encoding in GIMLET or cross-attention between modalities in Text2Mol and Prot2Text [159].GIMLET’s node-token encoding uses graph distances and edge features, while its bidirectional attention lets text selectively attend to nodes.Source: §6.1.2 · pp. 11–12

6.1.3 Discussion

  • Sequence linearization can impose order-dependent bias and rely on notations absent from pretraining, while graph features, permutation-invariant GNNs, and graph priors offer alternatives.Prediction heads use classification or regression losses, whereas generation-based prediction depends on carefully designed context; reasoning methods instead use GNN-proposed candidates or chemical tools [144],,.Source: §6.1.3 · p. 12

6.2.1 Latent Space Alignment

  • GNN and LLM representations are projected into a shared space for contrastive alignment; CLAMP incorporates classification labels, and MolCA uses a Q-Former.Methods including MoMu [172], MoMu-v2, MoleculeSTM, MolFM [160], and GIT-Mol [156] use augmented graph views, negative sampling, or mutual-information objectives to align graph–text pairs.Source: §6.2.1 · pp. 12–13

6.2.2 Discussion

  • Most reviewed alignment methods use five-layer, 300-dimensional GINs, which may constrain semantic representation; graph generation typically relies on order-sensitive text decoders rather than GNN decoders.The survey suggests deeper, larger GNNs and Transformer encoders, and identifies graph diffusion models as a possible route to GNN-based generation.Source: §6.2.2 · p. 13

7 RESOURCES AND APPLICATIONS

The survey organizes datasets and evaluation practices across pure graphs, text-attributed graphs, and text-paired graphs, and identifies open-source tools and applications spanning scientific discovery, social science, and domain-specific tasks.

7.1 Datasets, Splitting and Evaluation

  • Datasets for pure graphs, text-attributed graphs, and text-paired graphs are summarized in Tables 5, 2, and 3, respectively.The survey presents these collections as resources for the three graph scenarios.Source: §7.1 · p. 13

7.1.1 Pure Graphs

  • Pure-graph datasets cover standard reasoning tasks, real-world graph grounding, and temporal graph reasoning, with accuracy commonly used for evaluation.NL-Graph [122], LLMtoGraph [123], and GUC [124] cover connectivity, shortest path, and graph diameter; GraphQA [129] adds real-world scenarios, while LLM4DyG [126] focuses on evolving graphs.Source: §7.1.1 · p. 13

7.1.2 Text-Attributed Graphs

  • Text-attributed graph datasets span academic, e-commerce, book, social-media, and Wikipedia domains, supporting classification, prediction, regression, and recommendation tasks.Classification uses Accuracy, Macro-F1, or Micro-F1; link prediction and recommendation use MRR, NDCG, or Hit Ratio; regression uses MAE or RMSE.Source: §7.1.2 · p. 13, Table 2 · p. 14

7.1.3 Text-Paired Graphs

  • Text-paired graph evaluation includes varied splitting strategies and task-specific metrics, but BLEU can be less accurate and heuristic molecule-generation metrics can be gamed.Splits include random, source-based, activity-cliff and scaffold, and data-balancing [141] approaches; graph classification uses AUC, regression uses MAE, RMSE, and R2, and molecule-generation heuristics consider validity, novelty, and uniqueness, but may reward unintended modes such as adding excess carbon atoms.Source: §7.1.3 · pp. 13–14, Table 3 · p. 14

7.2 Open-Source Implementations

  • The open-source toolkit landscape combines Transformer libraries, graph-learning libraries, and cheminformatics software for molecular graph operations.HF Transformers, Datasets, and Evaluate support language models and data handling; Fairseq supports Transformers; PyG and DGL support graph machine learning; RDKit supports cheminformatics and molecule/SMILES conversion.Source: §7.2 · p. 14

7.3.1 Scientific Discovery

  • Graph-and-text methods can support virtual screening, constrained molecular generation, and synthesis planning for scientific discovery.Textual knowledge from scientific papers may help address limited and imbalanced labels in virtual screening [141], while generation can align molecular hypotheses with textual requirements and structural constraints,,; LLMs can suggest synthesis paths or operate existing planning tools [144].Source: §7.3.1 · pp. 14–15

7.3.2 Computational Social Science

  • In computational social science, text-attributed user and interaction graphs support recommendation, user analysis, community detection, and personalized response generation.E-commerce graphs connect users and products through interactions such as views and purchases, with text on nodes and edges; social-media graphs connect users through interactions and associated messages or profiles.Source: §7.3.2 · p. 15

7.3.3 Specific Domains

  • Domain-specific text-and-graph applications include paper and legal-opinion analysis, as well as educational knowledge tracing and student-performance prediction.Academic graphs connect papers through citation and authorship relations for recommendation, classification, and author identification,,,; legal citation graphs support clause classification and opinion recommendation,, [100]; coursework graphs support educational prediction,.Source: §7.3.3 · p. 15

8 FUTURE DIRECTIONS

The survey identifies future work in broadening graph benchmarks and task coverage, improving efficiency and reliability, and exploring unified multimodal models and dynamic graph agents.

Better Benchmark Datasets

  • Graph benchmarks need broader coverage of heterogeneous and spatial-temporal graphs, legal and health domains, diverse chemistry tasks, and 3D molecule-text pairs [136].Preparing scientific papers as ready-to-use data paired with relevant molecular graphs remains cumbersome.Source: §8 · p. 15

Broader Task Space with LLMs

  • Future work should study LLM encoders for text-captioned molecules, use augmentation and distillation to build domain-specific GNNs, and tackle joint text-structure generation for text-attributed graphs.Graph generation has been explored for text-paired graphs, but remains open for text-attributed graphs.Source: §8 · p. 15

Efficient LLMs on Graphs

  • Graph linearization creates longer sequences as graphs grow, increasing time and memory costs while confronting LLM input limits; graph-aware efficient tuning also remains understudied.LLM optimization is computationally expensive, and existing general methods such as LoRA do not address graph-aware tuning specifically.Source: §8 · p. 15

Generalizable and Robust LLMs on Graphs

  • Research should improve cross-domain transfer and consistent predictions under obfuscations and attacks, while addressing robustness and hallucination issues in graph modeling.The survey defines generalizability as transferring knowledge between domain graphs and robustness as maintaining consistent predictions under obfuscations and attacks.Source: §8 · p. 15

Multi-Modal Foundation Models

  • A key open question is whether one foundation model should jointly encode text, graphs, and images or whether architectures should be tailored to different domains.The survey notes that resolving this tension could avoid unnecessary efforts and clarify graph-related tasks.Source: §8 · p. 16

LLMs as Dynamic Agents on Graphs

  • Dynamic graph agents could use multi-hop retrieval to find relevant information more accurately and correct answers, mitigating hallucinations caused by one-pass generation and retriever limits.The survey points to dynamically consulted graphs such as academic networks and Wikipedia as settings for knowledge-guided reasoning.Source: §8 · p. 16

9 CONCLUSION

The paper comprehensively reviews large language models on graphs, covering graph scenarios, techniques, methods, resources, applications, and future directions.

9 CONCLUSION

  • The review categorizes graph scenarios for language models and summarizes techniques for applying them to graphs.It also analyzes and compares methods within each scenario.Source: §9 · p. 16
  • The paper summarizes available datasets, open-source codebases, and applications of large language models on graphs.Source: §9 · p. 16
  • The conclusion suggests future directions for large language models on graphs.Source: §9 · p. 16

APPENDIX · Training & Inference Framework with LLMs

The framework distinguishes pretraining-then-finetuning, typically used for medium-scale LLMs, from pretraining-then-prompting, typically used for large-scale LLMs. It presents fine-tuning variants and prompting as ways to adapt LLMs for graph tasks, with prompting leaving model parameters unchanged.

Training & Inference Framework with LLMs

  • Pretraining-then-finetuning is typically used for medium-scale LLMs, while pretraining-then-prompting is typically used for large-scale LLMs.Pretraining uses unsupervised objectives to initialize language understanding and inference ability for downstream tasks.Source: Training & Inference Framework with LLMs · p. 21
  • Graph-domain pretraining objectives include document relation prediction, network-contextualized masked language modeling, contrastive social prediction [?], and context graph prediction [32].Pure-text objectives include masked language modeling, auto-regressive causal language modeling, corruption-reconstruction language modeling, and text-to-text transfer modeling.Source: Training & Inference Framework with LLMs · p. 21
  • Full finetuning updates all model parameters, can fully stimulate downstream-task potential, but may incur heavy computational overload and overfitting [34].Source: Training & Inference Framework with LLMs · p. 21
  • Efficient finetuning updates only a parameter subset, using methods such as prompt tuning, prefix tuning, adapters, LoRA, graph neural prompts [40], and graph-enhanced prefixes [41].Source: Training & Inference Framework with LLMs · p. 21
  • Instruction tuning uses downstream-task instructions to encourage generalization to unseen inference tasks and applies to graph node classification [44], link prediction [45], and graph-level tasks.It is orthogonal to full and efficient finetuning, so either approach can be used for instruction tuning.Source: Training & Inference Framework with LLMs · p. 21
  • Prompting formulates test samples as natural-language sequences and uses in-context demonstrations for inference without parameter updates, with chain-of-thought, tree-of-thought [48], and graph-of-thought [49] variants.This approach is particularly popular for large-scale autoregressive language models.Source: Training & Inference Framework with LLMs · p. 21

Related papers

Loading 2312.02783v4…