PaperlayerView exact revision on arXiv

Source-linked AI summary

Harnessing Explanations: LLM-to-LM Interpreter for Enhanced Text-Attributed Graph Representation Learning

Xiaoxin He, Xavier Bresson, Thomas Laurent, Adam Perold, Yann LeCun, Bryan Hooi

arXiv:2305.19523v5cs.LG

TL;DR

TAG pipelines often rely on shallow text features, while combining LLM text reasoning with GNN structure remains challenging because LLMs require substantial resources. This paper uses LLM-generated explanations translated by an LM interpreter into node features for GNNs, achieving strong TAG results and faster training.

  • Problem

    Shallow text embeddings capture less semantic complexity than multi-layer language models, while LLMs require substantial computational resources and scalable integration with GNNs remains needed.

  • Method

    The method prompts an LLM for zero-shot predictions and explanations, translates those explanations into enriched node representations with an LM interpreter, and supplies them to a downstream GNN.

  • Results

    The method achieves top-1 performance on ogbn-arxiv and is 2.88× faster than GLEM, while also performing strongly on PubMed, Cora, and tape-arxiv23.

  • Takeaways & Limitations

    The approach provides a modular and efficient way to use LLM knowledge and explanations for TAG representation learning, with open-source code, models, features, and datasets.

  • Takeaways & Limitations

    The approach requires manually customized prompts for each dataset, whose effectiveness may vary with the dataset and task.

Abstract

from arXiv · show

Representation learning on text-attributed graphs (TAGs) has become a critical research problem in recent years. A typical example of a TAG is a paper citation graph, where the text of each paper serves as node attributes. Initial graph neural network (GNN) pipelines handled these text attributes by transforming them into shallow or hand-crafted features, such as skip-gram or bag-of-words features. Recent efforts have focused on enhancing these pipelines with language models (LMs), which typically demand intricate designs and substantial computational resources. With the advent of powerful large language models (LLMs) such as GPT or Llama2, which demonstrate an ability to reason and to utilize general knowledge, there is a growing need for techniques which combine the textual modelling abilities of LLMs with the structural learning capabilities of GNNs. Hence, in this work, we focus on leveraging LLMs to capture textual information as features, which can be used to boost GNN performance on downstream tasks. A key innovation is our use of explanations as features: we prompt an LLM to perform zero-shot classification, request textual explanations for its decision-making process, and design an LLM-to-LM interpreter to translate these explanations into informative features for downstream GNNs. Our experiments demonstrate that our method achieves state-of-the-art results on well-established TAG datasets, including Cora, PubMed, ogbn-arxiv, as well as our newly introduced dataset, tape-arxiv23. Furthermore, our method significantly speeds up training, achieving a 2.88 times improvement over the closest baseline on ogbn-arxiv. Lastly, we believe the versatility of the proposed method extends beyond TAGs and holds the potential to enhance other tasks involving graph-text data. Our codes and datasets are available at: https://github.com/XiaoxinHe/TAPE.

1 INTRODUCTION

TAG representation learning combines textual node attributes with graph topology, but shallow features capture limited semantics. This work uses LLM-generated predictions and explanations, interpreted by a smaller LM, to enrich GNN features and improve accuracy and efficiency.

  • TAGs represent entities such as papers as nodes, with textual attributes and edges encoding relationships.
  • Standard TAG pipelines use skip-gram or bag-of-words features, whose semantic complexity is limited relative to multilayer language models.
  • The framework prompts an LLM for ranked predictions and explanations, then uses an LM interpreter to convert the original text, predictions, and explanations into node features for GNNs.
  • The approach is designed for LMaaS compatibility, extracting LLM information without requiring fine-tuning or extensive local computational resources.
  • 2.88× faster than GLEM, the method achieves top-1 performance on ogbn-arxiv while also performing strongly on PubMed and Cora.
  • The paper releases codes, pretrained networks, enriched features, TAG versions of Cora and PubMed, and the new tape-arxiv23 citation graph dataset.

2 RELATED WORK

TAG representation methods have progressed from shallow text features to LM-based embeddings, while LLM-based approaches offer a newer route for combining language understanding with graph learning.

  • Shallow TAG pipelines transform text attributes into skip-gram or bag-of-words features before graph-based learning.
  • LM-based pipelines fine-tune pretrained models such as BERT to generate embeddings adapted to TAG domains and contexts.
  • LLM-based TAG research explores models such as ChatGPT, PaLM, and LLaMA for improving graph prediction and understanding text attributes.

3 FORMALIZATION

The paper formalizes TAG node classification, language-model text encoding, prompted LLM generation, and GNN neighbor aggregation as components of graph-text prediction.

  • A TAG consists of nodes, an adjacency matrix, and sequential text attributes, with labels predicted for unlabeled nodes from labeled nodes.
  • A language model encodes each node’s text attributes into a vector representation used for classification.
  • Prompting adapts a pretrained LLM by specifying task context in natural language rather than relying solely on task-specific fine-tuning.
  • The LLM models the conditional probability of each output sequence token given the prompt and previously generated tokens.
  • GNN layers update node representations by aggregating neighbor information, then feed the final representation to a classifier.

4 PROPOSED METHOD

TAPE uses LLM-generated predictions and explanations as supplementary text attributes, translating them into frozen features for downstream GNNs. It combines original-text, explanation, and ranked-prediction features through independently trained models whose outputs are averaged.

  • LLM prediction and explanation: TAPE queries an LLM for ranked predictions and textual explanations, then uses these outputs as supplementary attributes for downstream models.The pipeline comprises LLM prediction and explanation generation, LM interpretation, and GNN training.
  • LM interpreter and embeddings: A smaller LM interpreter converts original text and LLM explanations into fixed-length node embeddings for downstream GNN applications.Separate LMs process the original text and explanations, and their outputs are further mapped through MLPs.
  • Feature construction: TAPE represents each node with original-text, explanation, and ranked-prediction features, denoted hTAPE = {horig, hexpl, hpred}.Ranked predictions are one-hot encoded, concatenated, and linearly transformed into fixed-size features.
  • Efficiency: Frozen features and decoupled LM-GNN training improve modularity and efficiency compared with iterative LM-GNN training.The method reports a 2.88× speedup over GLEM on ogbn-arxiv using the same backbone LM and GNN.
  • GNN fusion: Separate GNNs process the three feature sources and their predictions are averaged to capture complementary information without changing GNN architecture or memory requirements.The framework trains models independently on horig, hexpl, and hpred before fusing their outputs.
  • Theoretical analysis: Under fidelity and non-redundancy conditions, explanations reduce the conditional entropy of the target given the smaller LM representation.The theorem states H(y|Z, E) < H(y|Z) when explanation fidelity error is smaller than the additional information supplied by the LLM representation.

5 EXPERIMENTS

Experiments evaluate TAPE across five TAG datasets and compare it with GNN-, LM-, and LM-GNN-based methods. TAPE consistently performs best across datasets and models, while also showing strong generalization and improved training efficiency.

  • Experimental setup: TAPE is evaluated on Cora, PubMed, ogbn-arxiv, ogbn-products, and tape-arxiv23 using comparisons with GNN- and LM-based methods.The evaluation includes shallow, GIANT, and TAPE node features alongside multiple GNN architectures.
  • Main results: TAPE consistently outperforms other methods across all datasets and models, while shallow features perform poorly and LM-based features improve results.Fine-tuned LMs also perform well, but TAPE features provide further gains.
  • Generalization: 84.23% accuracy on tape-arxiv23 exceeds the LLM’s 73.56% accuracy, supporting generalization beyond GPT-3.5’s knowledge cutoff.The dataset contains papers published in 2023 or later, and the experiments also demonstrate a cost-effective open-source Llama2 alternative.
  • Scalability: 2.88× less computation time than GLEM is achieved on ogbn-arxiv while using the same LM and GNN models.The comparison reports validation and test accuracy, parameters, batch size, and total training time for the two training paradigms.
  • Ablation study: Ablations on ogbn-arxiv show that the full TAPE method outperforms alternative feature settings across validation and test sets.The study assesses original text, explanations, predicted features, and their proposed combination.

6 CONCLUSION

The paper positions TAPE as an early contribution combining LLMs and GNNs for TAG tasks and anticipates broader research and application interest. It identifies manual dataset-specific prompting as a limitation and future-work target.

  • Conclusion: The authors present TAPE as a pioneering contribution intended to provide a strong baseline for future TAG research.They connect the convergence of LLMs and GNNs with opportunities in research and industrial applications.
  • Limitation and future work: TAPE requires manually crafted prompts customized to each dataset, whose effectiveness may vary by dataset and task.Future work includes automated prompt generation, alternative prompt designs, and handling dynamic TAGs.

REPRODUCIBILITY STATEMENT

The paper provides replication-oriented references, dataset documentation, and open access to code, datasets, trained models, and enriched features. It also points readers to supporting theorem and dataset materials.

  • Theorem support: Appendix A contains the detailed proof for the theorem presented in Section 4.4.The statement directs readers to the appendix for comprehensive theorem understanding.
  • Dataset documentation: Dataset sources and preprocessing are summarized in Appendix G, with detailed tape-arxiv23 collection procedures in Appendix C.The paper also states that TAG versions of missing-text datasets were collected and formatted.
  • Open resources: The source repository provides step-by-step replication instructions, datasets, trained models, and enriched features.It includes TAG versions of Cora and PubMed and the tape-arxiv23 dataset.

A THEORETICAL ANALYSIS

The theoretical analysis argues that LLM-generated explanations can improve predictions when they faithfully represent LLM knowledge and add information absent from another model’s representation.

  • Conditions: Fidelity requires explanations E to serve as a good proxy for the LLM’s raw-text representation ZL.The paper defines ZL as the vectorial representation of raw text modeled by the LLM.
  • Conditions: Non-redundancy requires ZL to contain information that is not present in the other model’s representation Z.This condition captures the LLM’s unique knowledge relative to the other model.
  • Theorem 1: Theorem 1 states that conditional entropy decreases when an LM uses both its representation Z and LLM explanations E, under fidelity and non-redundancy.The result is expressed as H(y|Z, E) < H(y|Z).
  • Proof: The proof derives the result using mutual-information and conditional-entropy properties, including an upper bound on conditional mutual information.The derivation applies entropy decomposition, conditional-entropy monotonicity, and the two stated conditions.

B TIME ANALYSIS AND MONEY ESTIMATION

The method queries an LLM once per node, stores predictions and explanations for reuse, and estimates both API cost and deployment time for ogbn-arxiv.

  • Time analysis: Approximately 9 hours is the estimated ogbn-arxiv deployment time under the 90,000-token-per-minute rate limit.The estimate is based on 169,343 nodes and the stated average sequence lengths.
  • Money estimation: $128 is the estimated ChatGPT-3.5 API cost for processing ogbn-arxiv under the stated token and pricing assumptions.The estimate uses approximately 285 input tokens and 164 output tokens per node across 169,343 nodes.
  • Cost-effective alternatives: Llama2-13b-chat provides a free alternative requiring approximately 16 hours on four A5000 GPUs.The paper presents this as a cost-effective alternative to paid API querying.
  • Query reuse: The method reduces repeated querying by making one LLM query and storing predictions and explanations for subsequent use.The released GPT responses can also be reused publicly.

C ADDRESSING LABEL LEAKAGE CONCERNS WITH A NEW DATASET

The paper addresses possible label leakage by introducing a post-cutoff arXiv dataset, while also evaluating prompt robustness and explanation-based classification across TAG settings.

  • New dataset: tape-arxiv23 contains 46,198 papers and 78,548 citation connections from papers published between January and September 2023.Its publication dates are beyond GPT-3.5’s November 2022 launch and stated knowledge cutoff.
  • Case study: A PubMed case study shows explanations correcting a node-12390 prediction from experimentally induced diabetes to type 2 diabetes.The authors attribute the improvement to explanations’ concision, focus, reasoning, and external knowledge.
  • Prompt robustness: Placing the title after the abstract produces a minor performance improvement, while most prompts perform similarly.The observation is reported in the prompt-variation study.
  • Prompt robustness: With RevGAT on ogbn-arxiv, TAPE accuracy remains between 0.7660 and 0.7750 across prompt designs.The authors describe this consistency as robustness to prompt configuration.

G.1 DATASET DESCRIPTION

The experiments cover five text-attributed graph datasets, use standard or explicitly defined splits, and contrast shallow text features with graph learning inputs.

  • Dataset descriptions: Cora contains 2,708 publications, seven classes, and 5,429 citation links.Each paper cites or is cited by at least one other paper in the final corpus.
  • Dataset descriptions: PubMed contains 19,717 diabetes publications classified into three diabetes-related classes and connected by 44,338 links.The classes are experimentally induced, type 1, and type 2 diabetes.
  • Dataset descriptions: ogbn-arxiv is a directed citation graph whose task predicts 40 manually assigned computer-science subject areas.Nodes are arXiv papers and directed edges represent citations.
  • Dataset descriptions: tape-arxiv23 is a directed citation graph of computer-science arXiv papers published in 2023 or later with the same 40-area prediction task.Its post-2022 publication range distinguishes it from ogbn-arxiv for leakage analysis.
  • Feature extraction: Traditional preprocessing methods are computationally efficient but may not capture full semantic meaning, limiting node-feature expressiveness.The paper frames this as a limitation of shallow text preprocessing and feature extraction.

H.3 DETAILED ABLATION STUDY

The ablation studies examine feature sources, LM fine-tuning, LM selection, memory use, and comparisons with GLEM. Results indicate that combining feature sources and fine-tuning improves performance while maintaining an efficient memory-to-accuracy trade-off.

  • Effect of node features: Using the full set of original text, explanation, and prediction features yields the best performance on ogbn-arxiv.Removing any feature reduces performance, although the drop varies across GNN models.
  • Effect of LM fine-tuning: Fine-tuning the LM is important, as omitting fine-tuning causes a marked performance decline on ogbn-arxiv.The comparison evaluates no fine-tuning against fine-tuning one or two LMs.
  • Effect of LM fine-tuning: Fine-tuning the same LM and different LMs produces similar outcomes, with two distinct LMs providing only a slight advantage.The small difference suggests that using a single LM can simplify and expedite the approach.
  • Effect of different LMs: The method is insensitive to the choice among the evaluated LMs, indicating robustness to LM selection.The study compares DeBERTa-base with all-roberta-large-v1 and e5-large.
  • Memory utilization: The model achieves the best accuracy without requiring more memory than pure LM or shallow-feature GNN models.The reported comparison describes the strongest memory-to-accuracy ratio among the evaluated training paradigms.
Loading 2305.19523v5…