Can GNN be Good Adapter for LLMs?

1 INTRODUCTION

• Prior cascading GNN-LMs incur message-passing costs and do not scale to billion-parameter models, while self-supervised approaches train GNNs and language models separately.The authors identify that separation as a potential source of sub-optimal graph-aware tuning.Source: §1 · p. 2
• GraphAdapter freezes the language model and uses trainable GNN adapters to alter its final output, enriching textual features with graph structure rather than using graph information only as supervision.The adapter is described as lightweight, with few trainable parameters and low computational costs.Source: §1 · p. 2
• After graph-specific pretraining, GraphAdapter can be fine-tuned for multiple downstream tasks.The introduction gives node classification with a head atop the last-token embeddings as an example.Source: §1 · p. 2
• Pretraining uses next-word prediction on node texts, applies the GNN adapter only at the transformer’s final layer, and mean-pools adapter and language-model logits.Final-layer use lets transformer computations run once and be cached during adapter training.Source: §1 · p. 2
• 4.7% over state-of-the-art cascaded GNN-LM methods and 5.4% over self-supervised GNN-LMs on average, with 30X fewer training parameters and storage.Experiments span multiple real-world TAGs, including social and citation networks.Source: §1 · p. 2

2 RELATED WORK

• Language models capture textual semantics, while GNNs model graph structure; parameter-efficient adapters and prompts reduce language-model fine-tuning demands.Examples include LoRA [14] and prompt tuning [16] [23].Source: §2 · pp. 2–3
• Cascading GNN-LMs face high computational overhead; freezing language models reduces computation, while neighbor sampling reduces captured graph information.Other work explores joint training through knowledge distillation [26] or Expectation Maximization [42].Source: §2 · p. 3
• Self-supervised GNN-LMs use graph-related tasks to tune language models, but the graph and language model remain separate and the language model cannot directly perceive graph information.The passage also notes that these methods do not use language-graph connections to help GNNs learn structural features.Source: §2 · p. 3
• Other LLM-for-graph approaches convert graphs to text [10] [40] or graph representations in prompts [32], whereas this paper focuses on using TAG text semantics to model TAGs.The related-work discussion also mentions large models enhancing TAG textual features [7] [13].Source: §2 · p. 3

3.1 Pretrained Language Model

• Autoregressive pretraining models a token sequence’s joint probability by predicting each next token from preceding tokens and minimizing negative log-likelihood.Text is represented as a token sequence, with the first token often marked [CLS].Source: §3.1 · p. 3
• Sentence representations can use the first token, mean-pooled hidden states, or the last token; prompts add task-specific tokens, with the last-token state serving as the prompted representation.The passage says prompts can reduce the gap between pretraining and downstream tasks [19] [23].Source: §3.1 · p. 3

3.2 Graph Neural Network

• GNN message passing iteratively aggregates neighboring node features and applies trainable transformations to produce structural representations.The aggregation can use mean-pooling, and the input includes node features and an adjacency matrix.Source: §3.2 · pp. 3–4

3.3 Text-Attributed Graph

• A text-attributed graph has nodes with tokenized text, and the task is to model both textual and structural data to predict node labels.Source: §3.3 · p. 4

4.1 Overview

• GraphAdapter combines GNN-derived node structure with language-model representations to model structure-related semantics in text-attributed graphs.The method uses GNNs as adapters for frozen pretrained language models.Source: §4.1 · p. 4, §4 · p. 4

4.2 Pre-training on TAGs

• During language-structure pre-training, the GNN representation is fused with language-model context states to predict the next token in node text.The paper calls this language-structure pre-training because text context supervises structure learning.Source: §4.1 · p. 4, §4.2 · p. 4
• The pre-training prediction averages language-model-only and graph-aware probabilities, directing optimization toward tokens the language model predicts poorly.The language-model prediction is reused so tokens predictable from context need not rely on graph structure.Source: §4.2 · p. 5
• Only the GNN and fusion block are trainable during pre-training; frozen transformer hidden states can be stored in advance, reducing training computation.The fusion block concatenates context hidden states and the structural representation before passing them through an MLP.Source: §4.2 · p. 5

4.3 Fine-tuning with Prompts

• For downstream tasks, prompts are appended to node text, and the prompt-conditioned language-model state is fused with node structure to form a task representation.For node classification, a new linear transformation produces the output.Source: §4.3 · pp. 5–6

5.1 Experiment setup

• Evaluation uses three public real-world datasets and node classification, with accuracy for Arxiv and Reddit and ROC-AUC for Instagram.Source: §5.1 · p. 6

5.2 Performance

• GraphAdapter improves over directly combining GNNs with frozen PLMs by 1.67% on RoBERTa, 1.89% on GPT-2, and 2.77% on Llama 2.Source: Table 2 · p. 7, §5.2 · p. 7
• 1.98% average pre-training gain; GraphAdapter exceeds cascaded GNN-LMs by 4.72% and self-supervised GNN-LMs by 5.40% on average.Source: §5.2 · p. 7
• Under the same PLM, GraphAdapter is comparable to fine-tuned SOTA; on small datasets it beats GLEM by 1.57%, while fine-tuned GIANT is 0.51% lower on average.Source: Table 3 · p. 7, §5.2 · p. 7

5.3 In-depth Analysis.

• With equal-sized backbones, GNN-based GraphAdapter gains 0.91% after pre-training on average, versus 0.19% for MLP; the authors conclude graph structure is prerequisite.Source: Figure 3 · p. 8, §5.3 · p. 8
• Removing pre-training plus graph structure drops performance by 1.95%; removing residual learning drops 1.02%, attributed to noise from training GNNs on all text.Source: §5.3 · pp. 7–8, Table 4 · p. 8
• 0.78% average downstream improvement follows GNN-block pre-training, which the authors interpret as stronger graph-structure modeling.Source: Table 5 · p. 8, §5.3 · p. 8
• Reinitializing the fusion layer reduces performance by 1.03% on average, versus 0.82% after reinitializing the GNN; the authors say fusion learns to combine their knowledge.Source: Table 6 · p. 8, §5.3 · p. 8

• GraphAdapter adapts LLMs to text-attributed graphs without LLM fine-tuning; across node-classification tasks, it improves accuracy by 5% over baselines.Source: §6 · p. 8

• Table 7 reports statistics for the experiment datasets.Source: Table 7 · p. 9

• Arxiv is a citation network of papers and abstracts where the task is predicting paper subjects.Source: §A.1 · p. 9
• Instagram classifies commercial versus normal users; Reddit classifies users above versus below the median historical-post score, using up to their last three posts as text features.Source: §A.1 · pp. 9–10

A.2 Baselines

• The comparison includes frozen-PLM GNNs, GIANT and GLEM, and LLM-based alternatives including TAPE, MLPs, Graph2Text, and LoRA.TAPE is reported only on Arxiv because only interpretation data for that dataset is published.Source: §A.2 · p. 10

A.3 Prompts

• Prompts are designed for each dataset according to its downstream task and graph information.Table 8 lists the detailed prompts for the three datasets.Source: §A.3 · p. 10, Table 8 · p. 10
• The study does not search for suitable prompts, although different prompts may significantly affect PLM performance.Prompt selection is outside the paper's stated focus.Source: §A.3 · p. 10
• Baseline sentence representations use the same prompts as the original text for BERT or RoBERTa, and the last token for Llama 2.Source: §A.3 · p. 10

A.4 Implementation Details

• Pretraining uses a two-layer GraphSAGE and a two-layer MLP fusion layer for 50 rounds.Source: §A.4 · p. 10
• The GNN hidden size is 128 on Arxiv, 64 on Instagram, and 128 on Reddit.The implementation also uses language-model techniques including layer normalization and warm-up.Source: §A.4 · p. 10
• For BERT and RoBERTa, GraphAdapter uses masked-token prediction, masking 20% of tokens rather than training on all tokens.Source: §A.4 · p. 10

B.1 Ablation Studies

• The original GAT attention mechanism is ineffective for pretraining, while a dot-product variant yields positive results and supports attention-based GNN pretraining.The modified variant is denoted GAT*.Source: §B.1 · pp. 10–11, Table 9 · p. 11
• Fusion of GNNs and LLMs performs better in most cases, and next-token pretraining enhances their performance.Table 10 reports additional ablations of the fusion component and individual components.Source: §B.1 · p. 10, Table 10 · p. 11

B.3 Efficient

• GraphAdapter's estimated training complexity is O(N × T_infer + |S_all| × T_GNN), avoiding dependence on PLM training time and PLM parameter loading during training.The authors note that complexity comparisons are challenging because methods use different base language models; they report running-time comparisons on Arxiv in Table 11.Source: Table 11 · p. 11, §B.3 · p. 11

B.2 Analysis of Prompt

• Task-related prompts improve LLM performance on TAGs, and GraphAdapter adds gains over prompts across the reported prompt settings.The analysis covers Arxiv, Instagram, and Reddit.Source: §B.2 · p. 11, Table 12 · p. 12, Table 13 · p. 12

B.4 Case Studies

• Across three Arxiv cases, pretraining lets GraphAdapter make correct predictions where the unpretrained model fails on distracting graph features or weak signals.The cases include distracting text, distracting graph features, and cases where neither modality provides a strong signal.Source: §B.4 · p. 11, Table 15 · p. 12