PaperlayerView exact revision on arXiv

Source-linked AI summary

Disentangled Representation Learning with Large Language Models for Text-Attributed Graphs

Yijian Qin, Xin Wang, Ziwei Zhang, Wenwu Zhu

arXiv:2310.18152v4cs.CLcs.LG

TL;DR

Prompt-based LLM methods for text-attributed graphs can leave complex structural relationships insufficiently understood. DGTL combines pretrained LLM text encoding with disentangled GNN structure features while keeping the LLM frozen. Across TAG benchmarks, the paper reports competitive performance and human-understandable explanations.

  • Problem

    Prompt-only methods for TAGs convey graph structure to LLMs but provide insufficient understanding of complex structural relationships and dependencies.

    Source: §1 · pp. 1–2

  • Method

    DGTL encodes TAG text with a pretrained LLM and uses tailored disentangled GNN layers to capture neighborhood structure from multiple factors for the LLM predictor, which remains frozen.

    Source: §1 · p. 2

  • Results

    Across various TAG benchmarks, DGTL achieves competitive performance compared with state-of-the-art baselines and offers human-understandable explanations for predictions.

    Source: §1 · pp. 2–3, §6 · p. 10

  • Takeaways & Limitations

    DGTL pairs TAG predictions with natural-language explanations, supporting interpretability for these tasks.

    Source: §6 · p. 10

    The study focuses on semi-supervised node classification, with text for all nodes and labels for only part of the training nodes available during training.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

Text-attributed graphs (TAGs) are prevalent on the web and research over TAGs such as citation networks, e-commerce networks and social networks has attracted considerable attention in the web community. Recently, large language models (LLMs) have demonstrated exceptional capabilities across a wide range of tasks. However, the existing works focus on harnessing the potential of LLMs solely relying on prompts to convey graph structure information to LLMs, thus suffering from insufficient understanding of the complex structural relationships within TAGs. To address this problem, in this paper we present the Disentangled Graph-Text Learner (DGTL) model, which is able to enhance the reasoning and predicting capabilities of LLMs for TAGs. Our proposed DGTL model incorporates graph structure information through tailored disentangled graph neural network (GNN) layers, enabling LLMs to capture the intricate relationships hidden in text-attributed graphs from multiple structural factors. Furthermore, DGTL operates with frozen pre-trained LLMs, reducing computational costs and allowing much more flexibility in combining with different LLM models. Experimental evaluations demonstrate the effectiveness of the proposed DGTL model on achieving superior or comparable performance over state-of-the-art baselines. Additionally, we also demonstrate that our DGTL model can offer natural language explanations for predictions, thereby significantly enhancing model interpretability.

1. Introduction

DGTL addresses the limited structural understanding of prompt-based LLM methods for text-attributed graph tasks by combining disentangled graph learning with pretrained LLMs. The paper reports competitive benchmark performance and human-understandable explanations while keeping the LLM frozen.

1. Introduction

  • TAGs connect textual entities through graph relations and occur in settings such as citation networks, e-commerce, and social media.The paper frames TAG representation learning as useful for capturing semantic relationships and dependencies for downstream tasks.Source: §1 · p. 1
  • Prompt-based methods convey graph structure to LLMs but can leave complex TAG relationships and dependencies insufficiently understood or exploited.Examples include summarizing neighbors or describing all neighbors in prompts.Source: §1 · pp. 1–2
  • DGTL is motivated by two challenges: TAG structure contains entangled relevant and irrelevant information, and task adaptation must balance pretrained and task-specific knowledge.The model is designed to extract pertinent structural information while leaving the LLM frozen.Source: §1 · p. 2
  • DGTL encodes TAG text with a pretrained LLM, then uses tailored disentangled GNN layers to capture neighborhood information from multiple structural factors for LLM prediction.The learned factorized features are injected into the LLM predictor to convey complex graph structure.Source: §1 · p. 2
  • The pretrained LLM remains frozen while the disentangled GNN is tuned, reducing computation costs and mitigating catastrophic forgetting.The approach is described as flexible for use with different LLM models.Source: §1 · pp. 2–3
  • Across various TAG benchmarks, DGTL achieves superior or comparable performance to state-of-the-art baselines and provides human-understandable natural-language explanations.The reported evaluations cover multiple TAG benchmarks.Source: §1 · pp. 2–3

2. Related Works

The related-work section introduces text-attributed graphs and reviews graph-learning approaches that combine text attributes with neighborhood structure. It then surveys LLM graph-task methods, including prompt-based TAG approaches and their structural-context challenges.

2.1. Text-attributed Graphs

  • TAG nodes carry text attributes and graph relations; GNN pipelines aggregate neighborhood representations, while text features have ranged from hand-crafted representations to language-model features.Common TAG benchmarks include Cora, CiteSeer, PubMed, and OGBN-arXiv, whose nodes are papers and edges are reference relationships.Source: §2.1 · p. 3

2.2. LLMs for Graph Tasks

  • Synthetic-graph studies represent graphs as natural-language descriptions or code to evaluate LLM graph reasoning across varied tasks (Wang et al., 2023; Guo et al., 2023; Liu & Wu, 2023).The cited evaluations span connectivity, shortest path, maximum flow, and other graph tasks.Source: §2.2 · p. 3
  • For TAGs, LLMs have been used to enhance node-text representations or predict from prompts; prompt-based neighborhood context can produce lengthy inputs, while DGTL compresses neighbors into tokens and freezes the LLM.Graph-LLM explores ego-graph descriptions and neighbor summaries, and InstructGLM creates node-specific vocabulary tokens.Source: §2.2 · p. 3

3. Problem Formulation and Preliminaries

This section defines text-attributed graphs and the semi-supervised node-classification task, then reviews message-passing GNNs and transformer-based language models. It outlines how self-attention and language-model pretraining support contextual text representations.

3.1. Text-attributed Graphs

  • A TAG is represented as G = (V, A, y), with each node associated with a text description that is converted into real-valued embeddings before GNN message passing.Source: §3.1 · p. 4
  • In semi-supervised TAG node classification, text and graph structure are given during training alongside labels for training nodes, and labels are predicted for test nodes.A TAG is represented as G = (V, A, y), with text descriptions associated with its nodes and text embeddings prepared before GNN message passing.Source: §3.1 · p. 4

3.2. Graph Neural Network

  • GNN message passing forms node representations by aggregating neighbor representations and applying an update function at each layer.The neighbors are derived from the adjacency matrix.Source: §3.2 · p. 4

3.3. Large Language Models

  • Transformer self-attention computes scores between input positions so models can focus on relevant text and capture long-range contextual relationships.Source: §3.3 · p. 4
  • Decoder-based LLMs are introduced through a self-attention formulation whose hidden features are projected to queries, keys, and values.Source: §3.3 · p. 4
  • Multiple self-attention layers and heads learn different representations and capture diverse local and global dependencies in text.Source: §3.3 · p. 4
  • LLMs are commonly pretrained with unsupervised language modeling on text sequences, acquiring language patterns and commonsense knowledge before downstream fine-tuning.The paper describes fine-tuning with task-specific supervised learning so models can adapt to tasks and domains.Source: §3.3 · p. 4

4. Method

DGTL combines upstream LLM text embeddings with disentangled graph learning, then injects neighborhood features into a downstream LLM. Its method also uses frozen LLMs and trains the graph components through category prediction.

4.1. Overall Framework

  • DGTL generates text embeddings, learns disentangled neighborhood features, and injects them into a downstream LLM while keeping pre-trained LLMs frozen.The approach tunes the disentangled GNN, reducing fine-tuning computation and allowing adaptation to updated LLMs with minimal tuning.Source: §4.1 · p. 5, Figure 2 · p. 5

4.2. Text Embedding Generation

  • Averaging last-layer hidden states represents the entire input text rather than relying only on the end-of-sentence token.The authors argue that averaging captures information and context across all input positions.Source: §4.2 · p. 5

4.3. Disentangled Graph Learning

  • Multiple parallel two-layer GNNs learn neighborhood features over differentiated, continuously weighted graph structures to capture varied topological and semantic information.The structures let the downstream LLM use specific types of neighborhood information for prediction.Source: §4.3 · p. 6

4.4. Neighborhood Information Injection

  • DGTL places disentangled embeddings at reserved prompt positions and injects them into query, key, and value calculations across LLM layers.Varying position encodings are intended to encourage different GNN components to learn diverse knowledge, while all-layer injection provides direct gradient flow to the GNNs.Source: §4.4 · p. 6

4.5. Fine-tuning Procedure

  • Fine-tuning samples mini-batches, predicts each node’s category at the first generated token, and updates GNN parameters using the category-prediction loss.The loss is calculated for the label token; parameters are updated after the nodes in each mini-batch are predicted.Source: §4.5 · pp. 6–7

5. Experiments

Experiments cover citation and e-commerce node-classification benchmarks, using Llama-2-13B-chat and tailored prompts. DGTL performs strongly against GNN and LLM predictors, produces structurally relevant explanations, and benefits from disentanglement.

5.1.1. DATASETS

  • Cora and PubMed represent citation networks of computer-science and biomedical papers, while Book-History represents books connected by frequent co-purchases; all tasks classify nodes.The paper text is titles and abstracts for citation nodes and descriptions for books; training uses 20 nodes per category and testing uses 1,000 remaining nodes.Source: §5.1.1 · p. 7

5.1.2. IMPLEMENTATIONS

  • Experiments use Llama-2-13B-chat, dataset-specific prompts, 16 disentangled channels for citation datasets and 8 for e-commerce, and 32 hidden dimensions per channel.The disentangled-structure hyperparameter δ is 0.8, and Adam uses a 0.001 learning rate.Source: §5.1.2 · pp. 7–8

5.1.3. BASELINES

  • Baselines include text-feature-enhanced GNN predictors and LLM predictors using 0-hop or neighborhood-summary prompts, evaluated with accuracy or exact match, respectively.The GNN category also includes shallow TF-IDF and Word2vec features, GLEM, and GCN, GAT, or MLP backbones.Source: §5.1.3 · p. 8

5.2. Experimental Results

  • DGTL is comparable to SOTA GNN baselines on Cora, outperforms all baselines on PubMed, and exceeds LLM-predictor baselines by a large margin.The comparisons are reported in Tables 3 and 4.Source: §5.2 · p. 8, Table 3 · p. 8, Table 4 · p. 8

5.3. Interpretability with Structural Information

  • DGTL’s shown explanations support correct paper and book classifications with neighborhood evidence, while the compared baselines make incorrect predictions without structural interpretation.Examples include rule-learning evidence from cited papers on Cora and related military-history books on Book-History.Source: §5.3 · p. 9, Table 5 · p. 9

5.4. Ablation Study

  • The ablation study finds disentanglement greatly beneficial and crucial for learning better structural information and enabling more accurate downstream LLM predictions.The ablation results are reported in Table 4.Source: Table 4 · p. 8, §5.4 · p. 9

6. Conclusion

The paper addresses integrating graph structure into LLMs for TAG tasks with DGTL, which uses tailored disentangled GNN layers. Experiments report competitive performance and human-understandable natural-language explanations.

6. Conclusion

  • DGTL uses tailored disentangled GNN layers to capture complex TAG structure, enhancing LLM reasoning and prediction while providing natural-language explanations.Experiments on TAG benchmarks report competitive performance against state-of-the-art baselines.Source: §6 · p. 10

Related papers

Loading 2310.18152v4…