PaperlayerView exact revision on arXiv

Source-linked AI summary

SimTeG: A Frustratingly Simple Approach Improves Textual Graph Learning

Keyu Duan, Qian Liu, Tat-Seng Chua, Shuicheng Yan, Wei Tsang Ooi, Qizhe Xie, Junxian He

arXiv:2308.02565v1cs.CLcs.AI

TL;DR

Textual-graph benchmarks often rely on traditional text-feature engineering, while more recent language-model approaches may require complex tasks or joint training. SimTeG fine-tunes a language model for the downstream task, then trains a GNN on its node representations and graph structure. The paper reports improved GNN performance across node classification and link prediction benchmarks, including new state-of-the-art results on OGBN-Arxiv.

  • Problem

    Many graph benchmarks still use traditional text-feature engineering, while language-model approaches have used complex feature-learning tasks or joint-training frameworks.

    Source: §1 · p. 2

  • Method

    SimTeG parameter-efficiently fine-tunes a language model on the downstream task, generates node representations, and trains a GNN on them with graph structure.

    Source: §1 · p. 2, §4 · p. 4

  • Results

    SimTeG improves GNN performance across node-classification and link-prediction benchmarks, reaching 78.04% for a new state-of-the-art result on OGBN-Arxiv.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The approach lets a simple two-layer GraphSAGE reach on-par state-of-the-art performance on OGBN-Arxiv using suitable language-model backbones.

    Source: §6 · p. 9

    The OGBL-Citation2-2.7M link-prediction graph is an intersection with OGBN-Papers100M because text resources were unavailable for all OGBL-Citation2 papers; it contains 93% of the original nodes.

    Source: A2.2 DETAILS OF TG VERSION FOR THE… · p. 15

Abstract

from arXiv · show

Textual graphs (TGs) are graphs whose nodes correspond to text (sentences or documents), which are widely prevalent. The representation learning of TGs involves two stages: (i) unsupervised feature extraction and (ii) supervised graph representation learning. In recent years, extensive efforts have been devoted to the latter stage, where Graph Neural Networks (GNNs) have dominated. However, the former stage for most existing graph benchmarks still relies on traditional feature engineering techniques. More recently, with the rapid development of language models (LMs), researchers have focused on leveraging LMs to facilitate the learning of TGs, either by jointly training them in a computationally intensive framework (merging the two stages), or designing complex self-supervised training tasks for feature extraction (enhancing the first stage). In this work, we present SimTeG, a frustratingly Simple approach for Textual Graph learning that does not innovate in frameworks, models, and tasks. Instead, we first perform supervised parameter-efficient fine-tuning (PEFT) on a pre-trained LM on the downstream task, such as node classification. We then generate node embeddings using the last hidden states of finetuned LM. These derived features can be further utilized by any GNN for training on the same task. We evaluate our approach on two fundamental graph representation learning tasks: node classification and link prediction. Through extensive experiments, we show that our approach significantly improves the performance of various GNNs on multiple graph benchmarks.

1 INTRODUCTION

Textual graph learning relies on both text features and graph structure, but feature extraction in many benchmarks still uses traditional methods. SimTeG instead fine-tunes a language model for the downstream task and uses its representations to train GNNs, reporting improvements across tasks and benchmarks.

1 INTRODUCTION

  • Textual graphs represent relationships among text-bearing nodes; node classification and link prediction are fundamental tasks in which text attributes offer critical information.Existing benchmarks often use bag-of-words or skip-gram features, while recent alternatives use specialized tasks or joint and end-to-end training frameworks.Source: §1 · pp. 1–2
  • SimTeG fine-tunes a language model on task labels, then uses its representations to train a GNN on the same downstream task.The approach avoids introducing a new training task, model architecture, or framework.Source: §1 · p. 2
  • SimTeG consistently outperforms official features and pretrained-LM features on three graph benchmarks across node classification and link prediction, reaching 78.02% on OGBN-Arxiv with an SOTA GNN.The paper reports the 78.02% result as new state of the art.Source: §1 · p. 2
  • A two-layer GraphSAGE using SimTeG and suitable language-model backbones reaches 77.48% on OGBN-Arxiv, on par with state-of-the-art performance.The authors say SimTeG complements the performance gap between GNN backbones by improving simple GNNs.Source: §1 · p. 2
  • Full language-model fine-tuning can overfit and collapse the feature space, hindering downstream GNN training; the authors report that PEFT alleviates this issue.They identify PEFT as crucial for producing representative embeddings.Source: §1 · p. 2
  • SimTeG is moderately sensitive to language-model choice, and its performance is generally positively correlated with the model’s text-embedding performance on classification and retrieval.The authors expect further improvement with more powerful text-embedding language models.Source: §1 · p. 2

2 RELATED WORKS

Related work on textual graphs uses specialized embedding strategies, joint language-model and GNN training, or fused architectures. The paper distinguishes SimTeG as a simple alternative to these approaches and separates textual graphs from other graph-and-text research lines.

2 RELATED WORKS

  • Prior textual-graph methods either design strategies for language-model node embeddings or jointly train language models and GNNs, often requiring specific communication between them.Examples include self-supervised graph learning, additional supporting text generation, variational expectation maximization, and graph-to-language-model distillation.Source: §2 · p. 3
  • SimTeG is distinguished from He et al. (2023), which generates additional node text for citation-network node classification; SimTeG investigates broader datasets and tasks.The paper also reports using that additional text with its method to achieve new state-of-the-art performance on OGBN-Arxiv.Source: §2 · p. 3
  • Graph Transformers generally embed topology and node features for graph-level problems and specific domains, whereas textual graphs use text-aligned nodes and ground-truth graph structure.The paper also distinguishes work applying GNNs to text over self-constructed or synthetic graphs.Source: §2 · p. 3

3 PRELIMINARIES

The preliminaries define textual graphs and the node-classification and link-prediction tasks, then describe how node-level GNNs use graph structure. They also explain why combining language models and GNNs in end-to-end training can be difficult at scale.

3 PRELIMINARIES

  • A textual graph is G = (V, E, T), with text items aligned to nodes; its edges can be represented by a sparse adjacency matrix A.Each adjacency entry indicates whether two nodes are linked.Source: §3 · p. 3
  • Node classification predicts unlabeled-node categories, while link prediction estimates whether a link exists between a pair of nodes.Without graph structure, these tasks reduce respectively to text classification and text similarity.Source: §3 · p. 3
  • Node-level GNNs generate representations by recursively aggregating features from multi-hop neighbors, with a classifier or similarity function applied for the respective task.The passage formulates a graph-convolution layer as Xl+1 = Ψl(CXl).Source: §3 · p. 3
  • End-to-end GNN–language-model training is difficult because GNNs typically need thousands of examples per batch, while texts consume more GPU memory than embeddings.The passage also notes that full-batch GNN training generally outperforms mini-batch training on multiple benchmarks.Source: §3 · p. 4
  • Common sentence-embedding methods include mean pooling over token embeddings and taking the [CLS] token embedding.The passage also notes that models specifically developed for sentence embeddings have shown promising benchmark results.Source: §3 · p. 4

4 SIMTEG: METHODOLOGY

SimTeG decouples language-model and GNN training: it fine-tunes the language model for the downstream task, generates node representations, then trains a GNN with those representations and graph structure. The method uses parameter-efficient fine-tuning to help regularize features and examines how model choice relates to feature quality.

4 SIMTEG: METHODOLOGY

  • SimTeG first fine-tunes a language model using the downstream-task loss, then generates node representations and trains a GNN on those representations and the graph structure.Both stages use the same task, and the training is fully decoupled so existing language models and GNNs can be used.Source: §4 · p. 4
  • The method uses mean pooling over the language model’s final-layer outputs; the authors find it more stable and faster to converge than using only the [CLS] embedding.The GNN stage replaces the language model in the task loss while using the generated node representations and graph structure.Source: §4 · p. 4
  • Parameter-efficient fine-tuning can alleviate overfitting in the language-model stage and provide better-regularized node features for downstream GNN training.The paper uses LoRA as its PEFT implementation.Source: §4 · pp. 4–5
  • The authors select information-retrieval-pretrained language models, using MTEB performance on retrieval and classification alongside model size to choose specific backbones.The selection rationale is that the graph tasks resemble text classification and text retrieval when graph structure is omitted.Source: §4 · p. 5
  • In t-SNE plots, X-SimTeG is more distinguishable than X-Fix and X-OGB; X-Fix is more distinguishable than X-OGB, and OGBN-Products features appear less distinguishable than OGBN-Arxiv features.The authors associate the weaker visual distinction on OGBN-Products with less improvement from SimTeG there.Source: §4 · p. 5, Figure 2 · p. 5

5 EXPERIMENTS

Across node classification and link prediction benchmarks, SimTeG features improve GNN performance, including results with simple GraphSAGE. The experiments also examine convergence, PEFT’s role in limiting overfitting, and sensitivity to language-model choice.

5 EXPERIMENTS

  • Experiments cover OGBN-Arxiv and OGBN-Products for node classification and OGBL-Citation2 for link prediction, comparing official features and other embedding methods across GNN backbones.Because OGBL-Citation2 has no official text resources, the study uses an intersection subset of about 2.7M nodes, versus about 2.9M in the original dataset.Source: §5 · pp. 5–6

5.1 Q1: HOW MUCH COULD SIMTEG generally IMPROVE THE LEARNING OF GNNS ON NODE CLASSIFICATION AND LINK PREDICTION?

  • Across all datasets and backbones, SimTeG improves over original features; SimTeG with GraphSAGE also exceeds X-OGB with SOTA GNNs on all three datasets.The study compares SOTA GNNs, GraphSAGE, and MLP across OGBN-Arxiv, OGBN-Products, and OGBL-Citation2.Source: §5.1 · p. 6, Table 1 · p. 7
  • 78.03% test accuracy with RevGAT on OGBN-Arxiv exceeded the previous SOTA result of 77.50% using additional text attributes and ensembling.The ensemble combines embeddings from e5-large and all-roberta-large-v1 with original and additional text attributes.Source: §5.1 · p. 7, Table 2 · p. 7
  • ∆MLP on OGBN-Arxiv is 2.98, indicating more important text attributes there; Products’ bag-of-words-like text is cited as accounting for smaller LM-feature gains.The authors use ∆MLP—the performance gap between MLP and the SOTA GNN—as an approximate indicator of text-attribute importance.Source: §5.1 · pp. 7–8

5.2 Q2: DOES X-SIMTEG FACILITATE BETTER CONVERGENCE FOR GNNS?

  • SimTeG generally speeds and stabilizes GNN training; GraphSAGE converges within 2 epochs on OGBN-Arxiv and OGBN-Products, while X-FIX converges more slowly than X-OGB.X-FIX denotes embeddings from pretrained LMs without finetuning; convergence is represented by training accuracy for the two node-classification datasets.Source: §5.2 · p. 8, Figure 3 · p. 8

5.3 Q3: IS PEFT A NECESSITY FOR LM FINETUNING STAGE?

  • PEFT mitigates LM-finetuning overfitting and yields features that help downstream GNNs gain more than features from full-finetuning, despite full-finetuning’s stronger LM-stage results.Table 3 compares PEFT (SimTeG) with full-finetuning using e5-large and GraphSAGE; ∆overfit is train accuracy minus test accuracy.Source: §5.3 · p. 8, Table 3 · p. 9

5.4 Q4: HOW SENSITIVE IS GNN TRAINING SENSITIVE TO THE SELECTION OF LMS?

  • GNN performance is moderately sensitive to LM choice: overall results track MTEB ranking, and retrieval-pretrained all-roberta-large-v1 generally beats same-architecture MLM-pretrained roberta-large.The LM-backbone comparison reports GraphSAGE and MLP results across 10 runs.Source: §5.4 · p. 9, Table 4 · p. 9

6 CONCLUSION

SimTeG uses embeddings from a parameter-efficiently fine-tuned language model as input to GNNs, with reported OGBN-Arxiv results of 77.48% for two-layer GraphSAGE and 78.04% for a state-of-the-art GNN.

6 CONCLUSION

  • 77.48% on OGBN-Arxiv is reported for two-layer GraphSAGE trained on SimTeG features, while a state-of-the-art GNN achieves 78.04%.The paper describes the GraphSAGE result as on-par state-of-the-art and the latter as a new state of the art.Source: §6 · p. 9
  • SimTeG first parameter-efficiently fine-tunes a language model, then uses its generated node embeddings to train a GNN.The paper presents this as a simple approach for textual graph representation learning.Source: §6 · p. 9

A1 MORE EXPERIMENT RESULTS

The appendix captions describe additional node-classification and link-prediction results, plus a comparison of two language models with the same architecture but different pretraining strategies.

A1 MORE EXPERIMENT RESULTS

  • Table A5 reports node-classification accuracy on ogbn-arxiv and ogbn-products, averaged over 10 runs, with improvements over X-OGB and ∆MLP and ∆GNN comparisons.The caption defines ↑(%) as improvement over X-OGB and ∆MLP and ∆GNN as extreme-value differences across the stated model groups.Source: Table A5 · p. 13
  • Table A6 reports link-prediction results on OGBL-Citation2-2.7M, averaged over five runs, including improvements over X-OGB and ∆ comparisons involving MLP, GraphSAGE, and SEAL.Its caption says positive ∆ values indicate MLP or GraphSAGE performs better than SEAL.Source: Table A6 · p. 13
  • Table A7 compares Graph and MLP performance using all-roberta-large-v1 and roberta-large, which share an architecture but differ in pretraining strategy.Results are based on 10 runs; the caption says the best results are bolded for each comparison in X-Fix and X-SimTeG.Source: Table A7 · p. 14

A2 REPRODUCIBILITY STATEMENT

The appendix provides code and dataset details for reproducing SimTeG experiments, along with dataset statistics and hyperparameter-search information.

A2 REPRODUCIBILITY STATEMENT

  • The authors provide source code and SimTeG node features for reproducibility and further research.The passage gives links to the code repository and feature dataset.Source: A2 REPRODUCIBILITY STATEMENT · p. 14

A2.1 PSEUDO CODE OF SIMTEG

  • The pseudocode separates supervised LM fine-tuning from GNN training for node classification and link prediction, using task-specific losses and optimizers.Node classification uses cross-entropy; link prediction uses binary cross-entropy.Source: A2.1 PSEUDO CODE OF SIMTEG · p. 14

A2.2 DETAILS OF TG VERSION FOR THE THREE OGB DATASETS

  • The appendix describes text construction for OGBN-Arxiv and OGBN-Products and defines OGBL-Citation2-2.7M as an intersection subgraph covering 93% of OGBL-Citation2 nodes.Arxiv uses paper titles and abstracts; Products uses item descriptions. Citation2-2.7M is reported to offer roughly on-par baseline performance.Source: A2.2 DETAILS OF TG VERSION FOR THE… · p. 15

A2.3 PROPERTIES OF LANGUAGE MODELS

  • Table A10 provides statistics for OGBN-Arxiv, OGBN-Products, and OGBL-Citation2-2.7M.Source: Table A10 · p. 15

A2.4 HYPERPARAMETER SEARCH SPACE

Loading 2308.02565v1…