PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploring the Potential of Large Language Models (LLMs) in Learning on Graphs

Zhikai Chen, Haitao Mao, Hang Li, Wei Jin, Hongzhi Wen, Xiaochi Wei, Shuaiqiang Wang, Dawei Yin, Wenqi Fan, Hui Liu, Jiliang Tang

arXiv:2307.03393v4cs.LGcs.AI

TL;DR

The paper examines how LLMs can improve node classification on text-attributed graphs, where conventional GNN pipelines rely on shallow text embeddings. It studies LLMs as enhancers that support GNNs and as direct predictors, reporting systematic empirical findings while noting limitations involving embedding understanding, augmentation costs, task and graph scope, contamination, and label ambiguity.

  • Problem

    Conventional graph pipelines for textual node attributes rely on shallow embeddings with limited general knowledge and profound semantic understanding, motivating study of LLM-compatible node-classification pipelines.

  • Method

    The paper studies LLMs-as-Enhancers, which augment text before GNN prediction, and LLMs-as-Predictors, which directly generate node-classification predictions from textualized graph information.

  • Results

    The study reports comprehensive empirical observations on both pipelines, including LLM zero-shot prediction on several real-world graphs and performance gaps relative to GNNs on some datasets.

  • Takeaways & Limitations

    LLMs provide promising possibilities for graph machine learning through enhanced textual attributes, direct prediction, and use as pseudo annotators for training smaller models.

  • Takeaways & Limitations

    The study primarily focuses on node classification for text-attributed graphs, while long-range information, other tasks and graph types, contamination, and ambiguous labels remain challenges.

Abstract

from arXiv · show

Learning on Graphs has attracted immense attention due to its wide real-world applications. The most popular pipeline for learning on graphs with textual node attributes primarily relies on Graph Neural Networks (GNNs), and utilizes shallow text embedding as initial node representations, which has limitations in general knowledge and profound semantic understanding. In recent years, Large Language Models (LLMs) have been proven to possess extensive common knowledge and powerful semantic comprehension abilities that have revolutionized existing workflows to handle text data. In this paper, we aim to explore the potential of LLMs in graph machine learning, especially the node classification task, and investigate two possible pipelines: LLMs-as-Enhancers and LLMs-as-Predictors. The former leverages LLMs to enhance nodes' text attributes with their massive knowledge and then generate predictions through GNNs. The latter attempts to directly employ LLMs as standalone predictors. We conduct comprehensive and systematical studies on these two pipelines under various settings. From comprehensive empirical results, we make original observations and find new insights that open new possibilities and suggest promising directions to leverage LLMs for learning on graphs. Our codes and datasets are available at https://github.com/CurryTang/Graph-LLM.

1. INTRODUCTION

Text-attributed graphs combine node text with graph structure, creating a need to model both for node classification. The paper explores two LLM-compatible pipelines that either enhance textual attributes for GNNs or directly predict labels.

  • Text-attributed graphs associate nodes with text attributes, including product descriptions and citation-paper titles or abstracts.These graphs support applications such as social network analysis and information retrieval.
  • Node classification requires capturing both node attributes and graph structure, but conventional pipelines commonly use shallow embeddings before GNN processing.Examples include Bag-of-Words and Word2Vec embeddings.
  • The paper investigates LLMs-as-Enhancers, which refine textual information before GNN prediction, and LLMs-as-Predictors, which generate final predictions directly.The predictor pipeline represents structural, attribute, and label information through natural language.
  • The study tailors strategies to different LLM capabilities and usage restrictions while examining their potential for node classification.The work studies how LLM knowledge and semantic comprehension can enhance GNNs and how LLMs can adapt to explicit graph-structured tasks.
  • The paper organizes its evaluation around the enhancer and predictor pipelines, followed by related work and discussion of limitations and future directions.

2. PRELIMINARIES

The preliminaries define text-attributed graphs and node classification, then introduce GNN message passing and the LLM categories used in the study. These foundations distinguish graph structure, textual attributes, and model interfaces.

  • A text-attributed graph consists of nodes, an adjacency matrix, and a text attribute associated with each node.The adjacency matrix is represented as A ∈R^|V|×|V|, and node v_i has text attribute s_i.
  • Node classification predicts labels for unlabeled nodes using labeled nodes, as illustrated by classifying citation-network papers into categories.In Ogbn-arxiv, nodes are papers, text attributes contain titles and abstracts, and edges represent citations.
  • GNNs update node representations by aggregating neighboring information through message passing and then use final representations for classification.The aggregation and update operations are differentiable functions in the described formulation.
  • LLMs are language models pretrained on extensive text corpora to harness knowledge acquired during pretraining.
  • Embedding-visible LLMs expose embeddings for manipulation, whereas embedding-invisible LLMs provide restricted text-based interfaces.BERT, Sentence-BERT, and DeBERTa are examples of embedding-visible models; ChatGPT is an embedding-invisible example.

3. PIPELINES FOR LLMS IN GRAPHS

The paper proposes enhancer and predictor pipelines for node classification on text-attributed graphs. Enhancers augment text before GNN inference, while predictors encode graph information in prompts for direct LLM prediction.

  • LLM-enhanced text attributes are encoded into augmented node features, which GNNs combine with original features to make predictions.This text-level enhancement uses embedding-visible LLMs after LLM transformation of each text attribute.
  • The two pipelines differ in whether LLMs augment inputs for GNNs or directly produce predictions.
  • The LLMs-as-Predictors pipeline prompts embedding-invisible LLMs with textual representations of graph structure, node attributes, and labels.The LLM then directly makes node-classification predictions.

4. LLMS AS THE ENHANCERS

The enhancer pipeline enriches text attributes with embedding-visible or embedding-invisible LLMs before GNN-based prediction. Experiments compare feature-level and text-level enhancement across datasets, labeling regimes, model types, and efficiency settings.

  • Enhancement strategies: Feature-level enhancement encodes text attributes into initial node features, while text-level enhancement generates augmented attributes that are encoded with the originals.The study examines cascading and iterative feature-level structures alongside a text-level enhancement structure.
  • Experimental setup: The experiments evaluate Cora, Pubmed, Ogbn-arxiv, and Ogbn-products under dataset-specific labeling splits and compare multiple GNN, MLP, PLM, and sentence-embedding configurations.Cora and Pubmed use low- and high-labeling-rate settings, whereas Ogbn-arxiv and Ogbn-products use official splits.
  • Feature-level results: Fine-tuned PLM embeddings usually outperform TF-IDF with MLP predictors, but TF-IDF is close to or surpasses PLM embeddings when a GNN predictor aggregates structure.The results indicate that GNN effectiveness varies with the type of text embedding, without identifying a simple determining metric.
  • Feature-level results: Fine-tuned LLM embeddings perform poorly in low-labeling-rate settings, while simple cascading with deep sentence embeddings and GNNs forms a strong baseline.The paper attributes the low-labeling result to limited training samples and reports the cascading baseline across multiple tables.
  • Scalability investigation: Iterative structures can achieve strong performance with adequate labels but introduce massive training computation overhead, limiting scalability under constrained resources.On larger datasets such as Ogbn-arxiv, fine-tuning-based methods may take several orders of magnitude longer than methods without fine-tuning.
  • Text-level enhancement: TAPE benefits mainly from LLM-generated explanations, whereas KEA consistently improves original attributes with e5, and the stronger enhancement varies by dataset.Explanations are more stable than pseudo labels and perform better in low-labeling settings; KEA outperforms original attributes, while TAPE performs better on Pubmed.

5. LLMS AS THE PREDICTORS

The paper evaluates LLMs as standalone predictors for node classification, with and without graph structure, and studies their effectiveness, prompting sensitivity, pseudo-labeling potential, and OOD robustness.

  • 5.1 Structure-Ignorant Prediction: LLMs-as-Predictors treat node classification as text classification, using ChatGPT on five textual graph datasets without structural information.The datasets are Cora, Citeseer, Pubmed, Ogbn-arxiv, and Ogbn-products.
  • 5.1 Observations: LLMs show strong zero-shot performance on some datasets, but lag behind GNNs on Cora, Citeseer, and Ogbn-arxiv.They perform remarkably on Pubmed and comparably to fine-tuned PLMs on Ogbn-products.
  • 5.1 Observations: Chain-of-thought prompting does not improve node classification because multiple reasonable labels or reasoning paths can conflict with the single ground-truth label.The paper also finds that small semantic changes in prompt label design can substantially affect performance.
  • 5.2 Structural Information: Whole-graph prompting is infeasible under LLM context limits, motivating ego-graph prompts that summarize selected neighbors, typically within two hops.The authors summarize the attributes of five neighbors at a time and note that repeated sampling could provide finer-grained neighborhood information.
  • 5.2 Structural Information: Neighborhood summarization generally improves zero-shot or few-shot performance relative to structure-ignorant prompts, except on Pubmed.The authors connect this potential gain to the homophily assumption that neighboring nodes tend to share labels.
  • 5.3 Pseudo Annotation: LLMs can serve as pseudo-annotators, but downstream GNN performance depends strongly on pseudo-label quality, while confidence estimates from prompting or logits are often uninformative.Directly prompted confidence is frequently 1, and output probabilities from text-davinci-003 are consistently close to 1.
  • 5.4 OOD Generalization: LLMs-as-Predictors consistently outperform the best GNN-based OOD baselines, with only a small gap between IID validation and OOD test performance.The study uses a simple structure-ignorant prompt and notes that better in-context samples or structural information may further improve OOD results.

6. RELATED WORK

Related work studies how LLMs and GNNs can be combined for text-attributed graphs. Existing approaches enhance node features with language-model knowledge or use LLMs directly for graph prediction and reasoning.

  • LLMs-as-Enhancers: Compared with shallow embeddings, LLMs offer richer commonsense knowledge that may improve downstream graph-learning tasks.Several studies use pretrained language models as text encoders to transform node attributes into features.
  • LLMs-as-Enhancers: Some enhancement methods generate prediction explanations with LLMs, encode them through PLMs, and ensemble the resulting features with original node features.The described approach reports state-of-the-art performance on the Ogbn-arxiv leaderboard but provides limited analysis of why it works.
  • LLMs-as-Predictors: LLMs-as-Predictors remove the requirement for GNNs at prediction time by relying solely on LLMs for final graph-task predictions.Early work includes closed-source LLMs used without parameter tuning, while later methods align graph and text spaces or apply instruction tuning.
  • LLMs-as-Predictors: GraphGPT uses contrastive learning and dual-stage instruction tuning to improve graph understanding and task-specific prediction.Other related systems apply the predictor pipeline to graph reasoning and graph representation learning.

7. CONCLUSIONS,LIMITATIONS,ANDFU-TURE DIRECTIONS

The paper identifies findings for its two LLM-based graph-learning pipelines and outlines limitations involving embedding understanding, augmentation cost, task and graph scope, evaluation, and feature-space alignment.

  • Key Findings: The study proposes LLMs-as-Enhancers and LLMs-as-Predictors for learning on text-attributed graphs.The former incorporates LLMs into text attributes, while the latter uses LLMs to generate final predictions.
  • Key Findings: Deep sentence embedding models provide effective and scalable feature-level enhancement across dataset split settings.The paper identifies them as good candidates for enhancing text attributes at the feature level.
  • Key Findings: Ensembling LLM-augmented attributes with original attributes improves performance across datasets and data splits.This finding concerns text-level enhancement using LLM augmentations.
  • Limitations: The study has limited understanding of why deep sentence embeddings outperform PLM embeddings on node classification.It also observes a performance gap between deep sentence embedding models and GLEM on Ogbn-products and provides limited explanations for differing GNN effectiveness across embeddings.
  • Limitations: LLM text-level augmentations require at least N API queries for a graph with N nodes, creating substantial expense on large-scale datasets.The paper therefore does not present results for Ogbn-arxiv and Ogbn-products in this setting.
  • Future Directions: Future work should extend the pipelines beyond node classification and natural-language graphs while addressing long-range information, efficiency, and operational cost.The paper also identifies evaluation design and alignment between graph-model and LLM feature spaces as open directions.

A. DATASETS

This section introduces the real-world graph datasets used in the study and points to their dataset statistics in Table 19.

  • Datasets: The study uses five real-world graph datasets in its experiments.Their statistics are reported in Table 19.
  • Datasets: The dataset appendix connects the five-dataset selection with a consolidated statistics table.
  • Datasets: Table 19 presents statistics for the graph datasets used in the work.

A.1 Dataset Description

The dataset descriptions cover graph sources, text-attribute availability, and access to structural and label information, with preprocessed releases planned to support future studies.

  • Dataset Description: The dataset descriptions explain each graph dataset and note that raw text attributes can be cumbersome to obtain for some datasets.
  • Dataset Description: Structural and label information for the datasets can be obtained from PyG.The paper states that preprocessed dataset versions will also be released.
  • Dataset Description: Ogbn-arxiv and Ogbn-products are selected from the OGB benchmark, whose dataset descriptions are available through the OGB documentation.

B. EXPERIMENT SETUPS

The experiments implement baseline models with established graph-learning and transformer libraries and run them on a multi-GPU server.

  • Experiment Setups: All baseline models are implemented with PyG, DGL, and transformers modules.
  • Experiment Setups: The experiments use a GPU server equipped with eight NVIDIA RTX A5000 GPUs.
  • Experiment Setups: Each NVIDIA RTX A5000 GPU in the experimental server has 24GB of VRAM.

B.2 Hyperparameters

The study specifies hyperparameter settings for baseline graph models, including search ranges for architecture, optimization, regularization, and attention choices.

  • RevGAT, GraphSage, and SAGN use their best OGB leaderboard hyperparameters.
  • Deberta-base on Cora and Pubmed follows TAPE’s hyperparameter setting, while GLEM follows its repository configuration.
  • GCN, GAT, and MLP search hidden dimensions from 8 to 256 and layer counts from 1 to 3.
  • The search varies normalization, learning rate, weight decay, dropout, and GAT head count.Normalization includes None and BatchNorm; dropout ranges from 0.0 to 0.8, and GAT uses 1, 4, or 8 heads.

C. DEMONSTRATIONS OF TAPE

The Pubmed examples show that labels can sometimes appear directly in raw text attributes. This property may contribute to LLMs’ superior zero-shot performance and make neighborhood information noisy at high labeling ratios.

  • Pubmed papers sometimes contain their labels directly in the raw text attributes.
  • This textual label visibility may be related to LLMs’ superior zero-shot performance on Pubmed.
  • When attributes already determine categories, adding neighbors from other categories can introduce noise and allow MLP to outperform GCN and GAT.The passage specifically links this behavior to the high labeling ratio setting.
  • Table 20 presents an illustrative Pubmed example of this phenomenon.
Loading 2307.03393v4…