PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploring the Potential of Large Language Models (LLMs) in Learning on Graphs

Zhikai Chen, Haitao Mao, Hang Li, Wei Jin, Hongzhi Wen, Xiaochi Wei, Shuaiqiang Wang, Dawei Yin, Wenqi Fan, Hui Liu, Jiliang Tang

arXiv:2307.03393v4cs.LGcs.AI

TL;DR

For text-attributed graph node classification, conventional GNN pipelines use shallow text embeddings, while designing an LLM-compatible pipeline remains a challenge. The paper studies LLMs as text enhancers followed by GNNs and as standalone predictors; in TAPE, LLM-generated explanations are the main source of effectiveness and are more stable than pseudo-labels across datasets.

  • Problem

    Conventional graph pipelines encode textual node attributes with shallow embeddings, and designing an LLM-compatible pipeline for graph-learning tasks remains a challenge.

    Source: §1 · p. 1

  • Method

    The paper studies LLMs as enhancers that refine node text for GNN prediction and as standalone predictors using structural and attribute information expressed in natural language.

    Source: §1 · p. 2

  • Results

    In TAPE, LLM-generated explanations are the main source of effectiveness, are more stable than pseudo-labels across datasets, and perform better in low-label settings.

    Source: §4.2.1 · p. 8

  • Takeaways & Limitations

    LLMs can provide zero-shot predictions on several real-world graphs, offering the possibility of inference on datasets without labels.

    Source: §5.3 · p. 13

    The study focuses on node classification for text-attributed graphs; extension to other graph-learning tasks and graph types remains unexplored.

    Source: §7.3 · p. 16

Abstract

from arXiv · show

Learning on Graphs has attracted immense attention due to its wide real-world applications. The most popular pipeline for learning on graphs with textual node attributes primarily relies on Graph Neural Networks (GNNs), and utilizes shallow text embedding as initial node representations, which has limitations in general knowledge and profound semantic understanding. In recent years, Large Language Models (LLMs) have been proven to possess extensive common knowledge and powerful semantic comprehension abilities that have revolutionized existing workflows to handle text data. In this paper, we aim to explore the potential of LLMs in graph machine learning, especially the node classification task, and investigate two possible pipelines: LLMs-as-Enhancers and LLMs-as-Predictors. The former leverages LLMs to enhance nodes' text attributes with their massive knowledge and then generate predictions through GNNs. The latter attempts to directly employ LLMs as standalone predictors. We conduct comprehensive and systematical studies on these two pipelines under various settings. From comprehensive empirical results, we make original observations and find new insights that open new possibilities and suggest promising directions to leverage LLMs for learning on graphs. Our codes and datasets are available at https://github.com/CurryTang/Graph-LLM.

1. INTRODUCTION

The introduction asks whether LLMs can address the semantic limitations of shallow text embeddings in text-attributed graph node classification and whether they can predict directly from explicit graph structure. It investigates LLMs-as-Enhancers, which improve text attributes before GNN prediction, and LLMs-as-Predictors, which generate predictions directly.

1. INTRODUCTION

  • Text-attributed graphs combine node attributes with graph structure, but shallow embeddings may miss word senses and semantic information needed for downstream tasks.The introduction motivates exploring LLMs’ context-aware knowledge and semantic comprehension as a way to address these limitations.Source: §1 · p. 1
  • LLMs-as-Enhancers and LLMs-as-Predictors are the paper’s two pipelines for incorporating LLMs into text-attributed graph node classification.Enhancers use LLMs to refine node text before GNN prediction; Predictors use natural language to present structural and attribute information to LLMs for direct prediction.Source: §1 · p. 2
  • Deep sentence embedding models are effective and efficient for encoding node attributes, while LLM-based text augmentation also improves downstream performance.The paper proposes two text-attribute enhancement strategies and compares their effectiveness.Source: §1 · p. 2
  • LLMs show preliminary effectiveness as graph predictors, but their inaccurate predictions and potential test-data leakage require caution.The study also explores which structural and attribute information can help LLMs understand and use graph structure.Source: §1 · p. 2
  • LLMs may serve as node annotators because a decent portion of their annotations is accurate.Source: §1 · p. 2

2. PRELIMINARIES

The preliminaries define text-attributed graphs and node classification, then establish the GNN and LLM concepts and model distinctions used in the study.

Text-Attributed Graphs and Node Classification

  • A text-attributed graph pairs nodes and an adjacency matrix with a text attribute per node; node classification predicts labels for unlabeled nodes from labeled examples.The paper illustrates the task with Ogbn-arxiv papers, whose titles and abstracts are node attributes and whose citation links are edges [23].Source: §2 · p. 2

Graph Neural Networks

  • GNNs classify text-attributed graph nodes by repeatedly aggregating neighboring-node information, then passing final representations through a fully connected prediction layer.Aggregation may use summation or maximum, while message and update functions may be differentiable functions such as MLPs.Source: §2 · p. 2

Large Language Models

  • LLMs are language models pretrained on extensive text corpora, and the paper distinguishes embedding-visible models from embedding-invisible text-only services.BERT [9], Sentence-BERT [54], and Deberta [21] are examples of embedding-visible models; ChatGPT is an embedding-invisible service.Source: §2 · pp. 2–3

Types of LLMs

  • The taxonomy separates fine-tunable PLMs from deep sentence-embedding models, which generally need no further task-specific fine-tuning and may be local or online.Sentence-BERT [54] exemplifies local open-source models, while text-ada-embedding-002 [44] exemplifies online closed-source models.Source: §2 · p. 3
  • Compared with PLMs, larger LLMs have orders of magnitude more parameters; open-source models such as LLaMA [63] expose parameters and embeddings but are cumbersome to fine-tune, while service models restrict access.PLMs, deep sentence-embedding models, and open-source LLMs are often embedding-visible; closed-source LLM services such as ChatGPT and GPT4 [46] are embedding-invisible.Source: §2 · p. 3

3. PIPELINES FOR LLMS IN GRAPHS

For node classification in textual graphs, the section presents two pipelines: LLMs enhance text attributes before GNN prediction, or LLMs predict directly from prompts containing graph and label information.

LLMs-as-Enhancers

  • Feature-level enhancement encodes each text attribute as node features, using either sequential LLM-to-GNN cascading or iterative PLM–GNN co-training with pseudo-labels.The iterative structure is suitable only for PLMs because it involves fine-tuning.Source: §3 · p. 3
  • Text-level enhancement augments text attributes, encodes them with embedding-visible LLMs, and ensembles the resulting node features with the original features for GNN prediction.Source: §3 · p. 3

LLMs-as-Predictors

  • LLMs-as-Predictors use prompts to represent graph structure, text attributes, and label information, then use embedding-invisible LLMs to predict node classes.Source: §3 · p. 3

4. LLMS AS THE ENHANCERS

The section studies two ways to use LLMs as graph-learning enhancers: enriching node features with embeddings and augmenting node text with generated information. It evaluates their predictive performance and training scalability across feature- and text-level approaches.

4.1 Feature-level Enhancement

  • Feature-level enhancement feeds embedding-visible LLM outputs to GNNs, using benchmark datasets and low- and high-label splits to evaluate encoder and integration choices.The study uses Cora, Pubmed, Ogbn-arxiv, and Ogbn-products; experiments run over 10 seeds and report average accuracy and variance.Source: §4.1 · pp. 3–5

4.1.1 Node Classification Performance Comparison

  • Feature-level results show that embedding effectiveness depends on the predictor: fine-tuned LLMs can fail with few labels, while deep sentence embeddings paired with GNNs are competitive across splits.TF-IDF can approach or surpass fine-tuned PLM embeddings with GNNs despite doing worse with MLPs; larger LLMs alone do not ensure better node-classification embeddings.Source: §4.1.1 · p. 5, Table 1 · p. 6, Table 2 · p. 6

4.1.2 Scalability Investigation

  • Iterative GLEM adds substantial training computation compared with cascading Deberta-base, while deep sentence embedding models are more time- and memory-efficient than fine-tuned PLMs.Feature dimension also largely determines memory use and time cost; this investigation concerns training-stage scalability, not inference efficiency.Source: §4.1.2 · pp. 6–7, Table 4 · p. 7

4.2 Text-level Enhancement

  • Text-level enhancement uses embedding-invisible LLMs to add information through TAPE explanations or KEA entity descriptions, then encodes and combines augmented and original text features.TAPE generates pseudo labels and explanations; KEA adds descriptions of knowledge entities without relying on explicit LLM predictions.Source: §4.2 · pp. 7–8, Figure 3 · p. 8

4.2.1 Experimental Setups

  • TAPE explanations are more stable than pseudo labels and help in low-label settings; e5-encoded TA + E achieves top-three performance in almost all settings.The authors attribute TAPE's effectiveness mainly to explanations and use TA + E without pseudo labels in subsequent analysis.Source: §4.2.1 · p. 8, Table 5 · p. 9, Table 6 · p. 9
  • KEA improves original text attributes, with KEA variants outperforming TA on both datasets; KEA beats TA + E on Cora, while TA + E does better on Pubmed.The authors suggest TA + E depends more on LLM prediction ability, whereas KEA uses commonsense knowledge and may be more stable across datasets.Source: §4.2.1 · p. 9

5. LLMS AS THE PREDICTORS

The section evaluates LLMs as standalone node predictors, first using text alone and then testing prompts that incorporate graph structure, alongside pseudo-annotation and OOD case studies.

5.1.1 Observations

  • LLMs show preliminary effectiveness without structural information, but their performance varies by dataset and prompts, with annotation bias and multiple plausible labels limiting label matching.They achieve remarkable zero-shot performance on Pubmed and performance comparable to fine-tuned PLMs on Ogbn-products, but lag GNNs on Cora and Pubmed. Chain-of-thought provides no gain; on Ogbn-arxiv, the specialized “arxiv cs subcategory” label design significantly outperforms the other tested designs, including on papers after 2023.Source: §5.1.1 · p. 11, Table 10 · p. 12, Table 13 · p. 12

5.2 Incorporating Structural Information in the Prompts

  • Summarizing two-hop neighborhoods generally improves LLM predictions over a structure-free zero-shot prompt, except on Pubmed, where heterophilous neighbors can mislead predictions.The method represents an ego-graph through neighboring nodes’ attributes and training labels, then summarizes sampled neighbors; context limits lead the study to summarize five neighbors at a time. On Pubmed, 15 of 19 two-hop neighbors in an analyzed example had labels different from the center node.Source: §5.2 · pp. 11–12, Table 14 · p. 13

5.3 Case Study: LLMs as the Pseudo Annotators

  • Training a GCN on randomly selected LLM pseudo-labels can match a GCN trained with 10 shots per class, while pseudo-label quality strongly affects downstream performance.The study annotates 140 Cora nodes and 60 Pubmed nodes; about 67% of Cora pseudo-labels and 93% of Pubmed pseudo-labels match ground truth. Simply asking LLMs for confidence yields mostly 1s, and output probabilities from text-davinci-003 are also consistently close to 1.Source: §5.3 · pp. 13–14, Table 16 · p. 13

5.4 Case Study: Applying LLMs to handle out-of-distribution data

  • LLMs-as-Predictors consistently outperform the best GNN-based OOD baselines, with a small gap between IID validation and OOD test performance.On GOOD-Arxiv, the study evaluates four shifts using a simple structure-ignorant prompt.Source: Table 18 · p. 14, §5.4 · p. 14

6. RELATED WORK

The related work reviews graph-learning approaches that use LLMs to enhance node attributes or make predictions directly. It distinguishes feature-level and text-level enhancement from predictor pipelines that represent graph information as text or embeddings.

6.1 LLMs as the Enhancers

  • PLM-based feature enhancement uses cascading or iterative integration: GIANT [6] adds structure-aware pretraining, while Graphformers and GLEM [83] co-train PLMs and GNNs.Cascading embeddings do not account for graph structure and remain fixed during GNN training; iterative methods exchange generated embeddings, and GLEM also uses pseudo labels. SimTEG [10] addresses fine-tuning overfitting, OneForAll [33] unifies feature spaces, and DRAGON [77] extends iterative integration to knowledge graphs.Source: §6.1 · p. 14
  • Text-level enhancement encodes LLM-generated prediction explanations as augmented features, while Graph-Toolformer [80] uses LLMs to connect natural-language commands with GNNs.The explanation-feature approach [22] reports state-of-the-art performance on the Ogbn-arxiv leaderboard [23], but offers limited analysis of why it works; Graph-Toolformer leaves GNN features and training unchanged and is outside this paper’s scope.Source: §6.1 · pp. 14–15

6.2 LLMs as the Predictors

  • Closed-source LLM predictors [18] [65] are studied for graph reasoning and classification, but reports find weak node-classification analysis and difficulty with long-range reasoning.GPT4Graph [18] reports competitive short-range knowledge-graph reasoning but struggles with long-range reasoning and node classification; NLGraph [65] focuses on synthetic reasoning tasks such as shortest path, maximum flow, and bipartite matching. Text-only inputs require graph-to-language conversion, which can lose information due to input-length limits.Source: §6.2 · p. 15
  • Embedding-based predictor methods tune LLMs to understand graph information: InstructGLM [79] combines instructions with embedded node features, and GraphGPT [62] aligns graph and text spaces.GraphGPT [62] uses self-supervised instruction tuning followed by task-specific fine-tuning; GraphLLM [3] and DGTL [50] apply this pipeline to graph reasoning and representation learning.Source: §6.2 · p. 15

7. CONCLUSIONS,LIMITATIONS,ANDFU-TURE DIRECTIONS

The paper summarizes findings on LLMs-as-Enhancers and LLMs-as-Predictors, then identifies limitations and future directions for applying LLMs to graph learning.

7.1 Key Findings

  • Deep sentence embeddings work well as feature-level enhancers, ensembling LLM-augmented and original attributes improves performance, and LLM predictors show preliminary zero-shot promise alongside evaluation concerns.The authors flag ambiguous labels in citation datasets and possible test-data leakage on Ogbn-arxiv.Source: §7.1 · p. 15

7.2 Limitations

  • Why deep sentence embeddings outperform PLMs remains unclear; the Ogbn-products gap versus GLEM may reflect domain, and GNN effectiveness varies across embeddings, possibly due to anisotropy [12].The study calls for analysis of original and aggregated feature spaces.Source: §7.2 · pp. 15–16
  • Text-level augmentation methods TAPE and KEA require at least N LLM API queries for N nodes, making large graphs costly and leaving Ogbn-arxiv and Ogbn-products results unreported.Source: §7.2 · p. 16
  • Graph-representation prompts were hand-crafted through trial and error; exploring formats such as XML, YAML, or GML [55] and automatic prompt generation remains open.Source: §7.2 · p. 16

7.3 Future Directions

  • Future work should extend the pipelines beyond text-attributed graph node classification, addressing long-range context [11] and graphs with non-natural-language information such as molecular graphs [13].Source: §7.3 · p. 16
  • More efficient LLM use is needed because API processing of large graphs is costly, while local inference and parameter-updated training require substantial hardware resources.Source: §7.3 · p. 16
  • Evaluation should address possible training-corpus contamination and ambiguous labels; newer datasets and multi-label ground truths may help, but reasonable ground truths remain challenging.Source: §7.3 · p. 16
  • Aligning graph and LLM feature spaces remains an open challenge: natural-language translation loses information and is context-limited, while embedding inputs with tuning add computational overhead.A multimodal study shows potential for a linear transformation with fixed LLM parameters to bridge visual and language representations [47].Source: §7.3 · p. 16

APPENDIX · A. DATASETS

The study uses five real-world graph datasets, whose statistics are presented in Table 19.

A. DATASETS

  • The study uses five real-world graph datasets, with their statistics reported in Table 19.The supplied passages do not include the datasets’ names or their statistical values.Source: A. DATASETS · p. 20, Table 19 · p. 20

A.1 Dataset Description

The appendix introduces citation and OGB benchmark datasets, noting dataset-specific text-attribute availability and the release of preprocessed versions to support future studies.

A.1 Dataset Description

  • Cora has seven paper-topic categories, while PubMed contains scientific-journal papers grouped into three diabetes categories.The appendix identifies their raw text attributes as obtainable from external sources.Source: §A.1 · p. 20
  • The Citeseer graph uses 3,186 nodes and 4,277 edges because the TAG version provides text attributes for only 3,186 nodes.Citeseer is described as a paper-citation dataset.Source: §A.1 · p. 20
  • Ogbn-arxiv and Ogbn-products are selected from the OGB benchmark.The appendix directs readers to OGB documentation for dataset descriptions.Source: §A.1 · p. 20

B. EXPERIMENT SETUPS · B.1 Computing Environment

The experiments used PyG, DGL, and transformers modules for baseline implementations and ran on a server with eight NVIDIA RTX A5000 GPUs.

B.1 Computing Environment

  • Baselines used PyG [13], DGL, and transformers [71]; experiments ran on eight NVIDIA RTX A5000 GPUs, each with 24GB VRAM.The GPUs were housed in a GPU server.Source: §B.1 · p. 20

B.2 Hyperparameters

The study adopts established hyperparameter settings for several models and searches specified ranges for GCN, GAT, and MLP.

B.2 Hyperparameters

  • RevGAT, GraphSage, and SAGN use the best OGB leaderboard hyperparameters, while Deberta-base on Cora and Pubmed follows TAPE [22].GLEM's LM part follows the hyperparameter settings in its repository.Source: §B.2 · p. 20
  • For GCN, GAT, and MLP, the search covers hidden dimensions {8, 16, 32, 64, 128, 256} and layers {1, 2, 3}.Source: §B.2 · p. 20
  • Their search ranges also include normalization {None, BatchNorm}, learning rates {1e-2, 5e-2, 5e-3, 1e-3}, and weight decay {1e-5, 5e-5, 5e-4, 0}.Source: §B.2 · p. 20
  • The search further tests dropout {0., 0.1, 0.5, 0.8} and GAT head counts {1, 4, 8}.Source: §B.2 · p. 20

C. DEMONSTRATIONS OF TAPE

In Pubmed, paper labels sometimes appear in raw text attributes, which may help explain LLMs’ superior zero-shot performance and why MLP outperforms GCN and GAT at high labeling ratios.

C. DEMONSTRATIONS OF TAPE

  • Pubmed paper labels sometimes appear in raw text attributes, as illustrated in Table 20.The paper suggests this property may contribute to LLMs’ superior zero-shot performance on Pubmed.Source: C. DEMONSTRATIONS OF TAPE · p. 20, Table 20 · p. 20
  • When attributes alone determine categories, adding neighbors from other categories can introduce noise, potentially explaining why GCN and GAT trail MLP at high labeling ratios.Source: C. DEMONSTRATIONS OF TAPE · p. 20

Related papers

Loading 2307.03393v4…