PaperlayerView exact revision on arXiv

Source-linked AI summary

LLaGA: Large Language and Graph Assistant

Runjin Chen, Tong Zhao, Ajay Jaiswal, Neil Shah, Zhangyang Wang

arXiv:2402.08170v3cs.LGcs.AI

TL;DR

Graph neural networks often struggle with multiple tasks, while language models require specialized adaptation to process graph structure. LLaGA converts graphs into structure-aware node sequences and maps them into token embedding space through a versatile projector. Across four datasets and three tasks, one model performs strongly in supervised and zero-shot settings while supporting generalization and explanations.

  • Problem

    GNNs have weak multi-task handling, while directly applying LLMs to graphs is difficult because graph structure is not readily represented in language.

  • Method

    LLaGA reorganizes graph nodes into template-based, structure-aware sequences and maps their representations into LLM token embedding space using a versatile projector.

  • Results

    One LLaGA model performs consistently across various graph datasets and tasks, with strong supervised and zero-shot performance and generalization to unseen datasets or tasks.

  • Takeaways & Limitations

    LLaGA combines graph-task versatility, transfer to unseen datasets or tasks, and textual interpretation of node embeddings within a single framework.

Abstract

from arXiv · show

Graph Neural Networks (GNNs) have empowered the advance in graph-structured data analysis. Recently, the rise of Large Language Models (LLMs) like GPT-4 has heralded a new era in deep learning. However, their application to graph data poses distinct challenges due to the inherent difficulty of translating graph structures to language. To this end, we introduce the Large Language and Graph Assistant (LLaGA), an innovative model that effectively integrates LLM capabilities to handle the complexities of graph-structured data. LLaGA retains the general-purpose nature of LLMs while adapting graph data into a format compatible with LLM input. LLaGA achieves this by reorganizing graph nodes to structure-aware sequences and then mapping these into the token embedding space through a versatile projector. LLaGA excels in versatility, generalizability and interpretability, allowing it to perform consistently well across different datasets and tasks, extend its ability to unseen datasets or tasks, and provide explanations for graphs. Our extensive experiments across popular graph benchmarks show that LLaGA delivers outstanding performance across four datasets and three tasks using one single model, surpassing state-of-the-art graph models in both supervised and zero-shot scenarios. Our code is available at \url{https://github.com/VITA-Group/LLaGA}.

1. Introduction

GNNs handle complex graph structures but remain weak at multi-task learning, while LLMs offer general-purpose task solving yet struggle to represent graphs directly. LLaGA addresses this gap by translating graph structure into structure-aware sequences and aligning them with token embeddings.

  • GNNs typically target single tasks and struggle to maintain performance across multiple tasks, even with self-supervised learning.Downstream use still commonly requires task-specific heads or tuning.
  • LLMs can solve varied tasks with one model and explain answers, but graph structures are difficult to represent directly in language.Plain-text graph descriptions can be verbose and fail to preserve intrinsic graph characteristics.
  • LLaGA integrates LLMs with graph data by using templates to encode structural details and a versatile projector to align graph and token spaces.The approach avoids complex natural-language descriptions while preserving structural information in node sequences.
  • LLaGA supports multiple graph tasks across datasets without task-specific adjustments and can generalize to unseen datasets or tasks.The framework is also designed to provide interpretations of node embeddings.
  • LLaGA is presented as a single model that performs consistently across various graph datasets and tasks while retaining general-purpose LLM capabilities.The paper contrasts this scope with specialized graph models and task-focused adaptations.

2. Methodology

LLaGA translates graphs into structure-aware node sequences and then into LLM-compatible token embeddings, using templates, structural encodings, and a trainable projector. It tunes this translation across multiple graph tasks so one model can support varied analyses and generate explanations.

  • Structure-Aware Graph Translation: LLaGA reorganizes graph inputs into node embedding sequences using two node-level templates that encode structural information around a node.The templates support node-level tasks and tasks such as link prediction.
  • Structure-Aware Graph Translation: The Neighborhood Detail Template samples fixed-size multi-hop computational trees, pads undersized neighborhoods, and traverses them level by level into fixed-length sequences.Placeholder nodes fill missing neighbors and preserve the template’s fixed shape.
  • Structure-Aware Graph Translation: LLaGA encodes node text with off-the-shelf text models, represents placeholders with zero vectors, and appends fixed-template Laplacian embeddings to capture structural position.For a fixed sample size, the computational-tree shape is unchanged, so the Laplacian embedding is computed once per template.
  • Structure-Aware Graph Translation: The Hop-Field Overview Template summarizes each neighborhood hop with one embedding, trading individual-neighbor detail for a broader receptive field.Its hop embeddings are obtained through parameter-free message passing on encoded text features.
  • Alignment Tuning: A simple MLP projector maps node embeddings h1, h2, ..., hn to token embeddings e1, e2, ..., en, with only projector parameters tuned during training.The projected sequence replaces the node-sequence portion of the tokenized chat prompt while preserving its positions.
  • Alignment Tuning: LLaGA tunes the projector with node classification, link prediction, and node description, enabling a versatile graph-to-token translator that can generate explanations.The node description task aligns node embeddings with descriptive texts and supports interpretability.

3. Experimental Results

LLaGA is evaluated across graph tasks, datasets, model settings, interpretation, zero-shot transfer, and template ablations. The experiments assess performance, interpretability, generalization, and the contribution of structural encodings.

  • Experimental Setup: The experiments cover node classification, link prediction, and node description across ogbn-Arxiv, ogbn-Products, Pubmed, and Cora.The datasets span citation and e-commerce domains and range from small to large scales.
  • Overall Performance: LLaGA is compared with GNN, transformer-based, and general-LLM baselines under four training settings.The settings are Single Focus, Task Expert, Classification Expert, and General Model; LLaGA uses one projector across tasks.
  • Overall Performance: LLaGA demonstrates superior performance across settings, especially in multi-task scenarios, with minimal degradation or improvements where many baselines decline.The authors attribute this pattern to extracting common patterns across datasets and tasks.
  • Interpretation Ability: LLaGA generates meaningful node descriptions: its Sbert score exceeds the random-sample base, and label extraction accuracy supports the generated content.Examples show reasonable interpretations even when predicted labels differ from ground truth.
  • Zero-Shot Ability: In zero-shot evaluation, LLaGA transfers knowledge across both in-domain and out-of-domain datasets, including citation graphs and markedly different domains.The reported result indicates that LLaGA can discern and leverage similar patterns across datasets.
  • Template Ablation: Both Neighborhood Detail and Hop-Field Overview templates improve performance over using only the center-node embedding, particularly for structurally dependent link prediction.The ablation supports their role in encoding surrounding structural information.

4. Related Work

Prior graph-learning methods often struggle to maintain consistent performance across datasets and tasks. Related work uses self-supervised GNN strategies or combines LLMs with GNNs and language-based graph representations, but these approaches retain task or model-scope limitations.

  • GNN Limitations: GNNs commonly transform nodes into compact vectors for downstream classification, but a single model may not perform consistently across datasets and tasks.This limitation motivates methods beyond single-task or single-setting graph models.
  • Self-Supervised GNNs: Self-supervised GNN methods improve generalization through graph-specific pretext tasks such as mutual-information maximization, whitening decorrelation, and generative reconstruction.Research has also examined combining multi-task and self-supervised learning.
  • LLM-Graph Integration: LLM-GNN approaches enrich graph textual attributes with LLM knowledge, but predictions largely remain dependent on GNNs, limiting their scope.Other work represents graphs linguistically rather than directly using graph structure.

5. Conclusion

LLaGA integrates LLMs with graph data by encoding graph structure into sequences and aligning node embeddings with token embeddings. The resulting single model supports multiple graph tasks, unseen datasets or tasks, and textual explanations.

  • Framework: LLaGA transforms graph structure into sequences with templates and maps node embeddings into token embedding space through a tuned projector.This alignment enables LLMs to perform graph tasks without replacing their general-purpose language capabilities.
  • Interpretability: LLaGA also generates textual explanations for node embeddings, extending graph prediction with interpretable descriptions.This capability is part of the framework’s reported contribution alongside versatility and generalizability.
  • Capabilities: The framework supports node classification and link prediction across datasets and can generalize to unseen datasets or tasks without adaptation.The paper reports effectiveness in both supervised and zero-shot graph learning scenarios.

6. Impact Statements

LLaGA’s broader impact is framed around applying versatile LLM-based graph learning to domains where graph data is central. The paper specifically identifies bioinformatics, social network analysis, and knowledge graphs.

  • Potential Applications: LLaGA’s broader impact extends to fields including bioinformatics, social network analysis, and knowledge graphs.These examples are presented as domains where graph data is pivotal.

A. Dataset Statistics

The paper evaluates citation and e-commerce graphs whose nodes, attributes, and edges represent different real-world entities and relationships. Experiments use standard dataset-specific splits for node-level tasks and constructed splits for link prediction.

  • Citation-graph nodes represent papers with titles and abstracts as features, while edges denote co-citations.
  • In ogbn-Products, nodes represent Amazon products with item descriptions, and edges indicate co-purchases.
  • Node-level train/validation/test splits are 6:2:3 for Arxiv, 8:2:90 for Products, and 6:2:2 for Pubmed and Cora.
  • For link prediction, node pairs are sampled from node-level training and test sets, with edge-level training sets matching node-level training-set sizes.

B. Zero-Shot Ability on Node Classification

The zero-shot node-classification evaluation tests whether graph-to-token alignment transfers across datasets and domains without task-specific classification heads. LLaGA is compared with GraphGPT under prompts with and without central-node textual attributes.

  • Zero-shot training establishes graph-to-token alignment from node-description tasks before testing node classification on different datasets.
  • The in-domain transfer trains on Arxiv and Pubmed and tests on Cora, whereas out-of-domain transfer trains on citation graphs and tests on Products.
  • LLaGA consistently outperforms GraphGPT across zero-shot prompt settings.
  • Adding central-node textual attributes offers some zero-shot advantages, while node-sequence-only prompts may support graphs with difficult-to-describe attributes.

C. Flexibility with Text Encoding Methods

The study tests whether LLaGA remains effective when node-attribute text embeddings are produced by different encoders. Across the evaluated encoders, LLaGA consistently surpasses leading GNN baselines.

  • LLaGA is evaluated with SimTeG, SBERT, and RoBERTa as alternative node-attribute encoding models.
  • All models, including baselines, are trained in a classification-expert setting, while LLaGA uses the Hop-Field Overview Template for structure encoding.
  • LLaGA consistently surpasses other leading GNNs regardless of the selected text encoding model.

D. Integration with Various LLMs

The paper examines whether LLaGA can use different base LLMs while retaining its graph-structure encoding setup. The section reports experiments replacing Vicuna-7B with LLaMA2-7B and OPT-2.7B.

  • Table 10 reports LLaGA’s performance after substituting its base LLM.
  • Vicuna-7B is the primary base LLM, with LLaMA2-7B and OPT-2.7B evaluated as alternatives.
  • The alternative-LLM experiments use the Hop-Field Overview Template for structural encoding and train models in a classification setting.

E. Experiment Variance

The experiments were repeated five times on relatively small Cora and Pubmed datasets, with variance information reported in Table 11.

  • Experiment Variance: Table 11 reports variance information for the Cora and Pubmed datasets.
  • Experiment Variance: Five training and inference runs were performed on relatively small datasets.The passage identifies repeated evaluation but does not provide the resulting variance values.
  • Experiment Variance: The reported variance analysis concerns both training and inference.
Loading 2402.08170v3…