Source-linked AI summary
Learning on Large-scale Text-attributed Graphs via Variational Inference
Jianan Zhao, Meng Qu, Chaozhuo Li, Hao Yan, Qian Liu, Rui Li, Xing Xie, Jian Tang
TL;DR
TAG learning requires combining textual semantics with graph structure without the prohibitive cost of jointly training large language models and GNNs. GLEM uses variational EM to update the modules separately while exchanging pseudo-labels, and experiments report improved effectiveness, efficiency, and scalability across multiple datasets and settings.
Problem
Learning effective representations on large text-attributed graphs requires integrating text and graph structure, but jointly training large language models and GNNs has severe scalability costs.
Method
GLEM uses a pseudo-likelihood variational EM framework that alternates E-step language-model updates and M-step GNN updates, with each module learning from the other's pseudo-labels.
Results
Across multiple datasets and two settings, experiments demonstrate GLEM's effectiveness and efficiency, including state-of-the-art results on three TAG datasets in the OGB benchmark.
Takeaways & Limitations
Separate yet mutually enhancing LM and GNN training enables GLEM to scale to large TAGs while using both textual and structural semantics.
Takeaways & Limitations
Language models have high memory costs and ignore node interactions, while GNNs underuse textual information and struggle on nodes with few neighbors.
Abstract
from arXiv · showhide
This paper studies learning on text-attributed graphs (TAGs), where each node is associated with a text description. An ideal solution for such a problem would be integrating both the text and graph structure information with large language models and graph neural networks (GNNs). However, the problem becomes very challenging when graphs are large due to the high computational complexity brought by training large language models and GNNs together. In this paper, we propose an efficient and effective solution to learning on large text-attributed graphs by fusing graph structure and language learning with a variational Expectation-Maximization (EM) framework, called GLEM. Instead of simultaneously training large language models and GNNs on big graphs, GLEM proposes to alternatively update the two modules in the E-step and M-step. Such a procedure allows training the two modules separately while simultaneously allowing the two modules to interact and mutually enhance each other. Extensive experiments on multiple data sets demonstrate the efficiency and effectiveness of the proposed approach.
1 INTRODUCTION
Text-attributed graphs combine rich textual semantics with structural information, but jointly training language models and GNNs becomes difficult to scale. GLEM addresses this by alternatively updating the two modules in a variational EM framework while exchanging pseudo-labels.
- Text-attributed graphs associate nodes with text, making them relevant to graph learning, information retrieval, and natural language processing.
- Effective node representations can support applications such as node classification and link prediction by combining textual and structural information.Text provides node semantics, while graph structure preserves proximity between nodes.
- Jointly cascading a language-model text encoder with a GNN creates memory costs proportional to graph size because neighborhood texts must also be encoded.On densely connected real-world graphs, this cost can become unaffordable.
- Existing scalable approaches reduce language-model capacity or graph size, but these compromises can degrade representation-learning performance.Examples include freezing language-model parameters and sampling graph edges for message passing.
- GLEM alternates language-model and GNN updates through variational EM, allowing separate training while the modules exchange pseudo-labels and mutually enhance one another.The LM models text-conditioned label distributions, whereas the GNN models global conditional label distributions from neighboring nodes.
2 RELATED WORK
Prior TAG methods combine language models and GNNs but often leave embeddings fixed, reduce model or graph capacity, or address different graph settings. GLEM instead jointly leverages textual and structural semantics through separate, collaborative training.
- TAG representation learning has increasingly focused on node classification, progressing from convolutional or recurrent text models to transformers and pretrained language models.
- A common fusion strategy feeds fixed language-model embeddings into a GNN, while newer methods improve embeddings through domain-adaptive pretraining or graph-structure prediction.These embeddings remain unlearnable during GNN training.
- GLEM uses a pseudo-likelihood variational framework to train an LM and GNN separately, supporting scalability while encouraging collaboration between textual and structural semantics.The paper characterizes existing fusion methods as suffering from either unsatisfactory results or poor scalability.
- Some GNN-based text-classification methods operate on synthetic or co-occurrence graphs because their graph structures are unobserved, unlike GLEM's observed node relationships.
- Compared with GMNN, GLEM targets text-attributed graphs and fuses a GNN with a language model rather than combining two GNNs.The paper reports state-of-the-art results on several benchmarks.
3 BACKGROUND
The background formulates TAG node classification as predicting unlabeled node labels from text and graph structure. Language models capture textual information, while GNNs propagate structural information; GLEM combines them through separate EM updates.
- A text-attributed graph consists of nodes, an adjacency matrix, and a sequential text feature associated with each node.The node-classification task predicts labels for unlabeled nodes from a few labeled nodes.
- Node labels can be predicted from textual information with language models or structural information with graph neural networks.
- Language models for node classification: A language model encodes each node sentence into a vector and applies an MLP with softmax to predict its label distribution.
- Language models for node classification: Language models can incur high memory costs and ignore interactions between nodes, limiting performance for nodes with insufficient text features.
- Graph neural networks for node classification: GNNs use message passing and aggregation over neighboring nodes to update representations that capture structural interactions.The mechanism uses message and aggregation functions over each node's neighbors.
- Graph neural networks for node classification: GNNs leverage structural information effectively but often struggle to use textual information and perform poorly on nodes with few neighbors.Figure 1 presents GLEM's separate LM and GNN training within a variational EM framework.
4 METHODOLOGY
GLEM combines a language model and GNN through a pseudo-likelihood variational EM framework, alternating their optimization to improve scalability for node classification on TAGs. The LM models labels from local text, while the GNN uses text, graph structure, and neighboring labels to model structural interactions.
- 4 METHODOLOGY: GLEM addresses scalability by training the LM and GNN in separate alternating E-steps and M-steps rather than end-to-end.The separate updates retain interaction between the modules while avoiding simultaneous training.
- 4 METHODOLOGY: The LM qθ predicts each node’s label distribution from its own text, whereas the GNN pφ uses node texts, graph structure, and other node labels through message passing.The mean-field q assumes node labels are independent and each label depends only on its node’s text.
- 4 METHODOLOGY: The variational framework optimizes an evidence lower bound over a variational distribution q and model distribution p, alternating updates to tighten the bound and maximize pseudo-likelihood.The framework handles unobserved node labels through variational inference and pseudo-likelihood optimization.
- 4.3 E-STEP: LM OPTIMIZATION: In the E-step, the fixed GNN supplies inferred labels that train the LM, while labeled-node supervision and pseudo-label distillation contribute to the LM objective.The reverse-KL objective avoids directly handling the entropy of qθ; unlabeled-node labels are approximated using LM-generated pseudo-labels.
- 4.4 M-STEP: GNN OPTIMIZATION: In the M-step, the LM supplies text embeddings and pseudo-labels to train the GNN, whose objective balances pseudo-label distillation against supervised learning with observed labels.The balance between the two terms is controlled by the hyperparameter β.
5 EXPERIMENTS
Experiments evaluate GLEM in transductive and structure-free inductive node classification across three OGB benchmarks, comparing language, graph, and fusion methods. Results show improved accuracy, scalability to larger language models, and competitive efficiency through alternating updates.
- Experimental setup: Experiments cover transductive and structure-free inductive node classification on ogbn-arxiv, ogbn-products, and ogbn-papers100M.The inductive setting evaluates transfer to unseen nodes using only text attributes.
- Transductive node classification: GLEM-LM significantly improves over fine-tuned language models by incorporating structural information for message passing.Fine-tuned language models remain competitive, highlighting the value of text attributes.
- Transductive node classification: GLEM-GNN significantly outperforms methods with fixed node embeddings in most cases and achieves state-of-the-art results on all three OGB datasets.Its language module dynamically generates embeddings and pseudo-labels for the graph module.
- Scalability: GLEM generalizes to DeBERTa-large with about 0.4B parameters while producing consistent improvements for every language model tested.This supports the scalability of the EM-based optimization paradigm.
- Structure-free inductive node classification: Structure-free inductive prediction is especially challenging for GNNs, while GLEM-LM and GLEM-generated embeddings improve inference for language models, MLPs, and GNNs.GLEM-LM combines local semantics with neighboring structural information during training, and generated embeddings support structure-free inference.
- Comparison of different training paradigms: GLEM achieves optimal classification results while remaining close to static training in time per epoch, unlike joint training’s poorer efficiency and effectiveness.Static training is most efficient but has lower accuracy, whereas joint training reduces graph structure and performs worst on both dimensions.
- Convergence: Both GLEM modules consistently improve across alternating E-steps and M-steps, converging in a few iterations and requiring only one iteration on ogbn-arxiv.The convergence curves track validation accuracy for GLEM-GNN and GLEM-LM on ogbn-arxiv and OGB-Products.
6 CONCLUSION
GLEM fuses language models and GNNs for node representation learning in text-attributed graphs through alternating variational updates. Experiments across datasets and settings demonstrate its effectiveness and efficiency.
- GLEM integrates an LM and a GNN through a pseudo-likelihood variational framework for node representation learning on TAGs.
- Alternating E-step and M-step updates train the LM and GNN separately, improving scalability.
- Each module learns from pseudo-labels predicted by the other, mutually enhancing graph and language learning.
- Experiments on multiple datasets and in two settings demonstrate GLEM's effectiveness and efficiency.
A SENSITIVITY ANALYSIS
The sensitivity analysis examines how pseudo-label weights affect GLEM-GCN performance. LM pseudo-label weighting is important, whereas GNN pseudo-label weighting is less sensitive and optimal settings vary across models.
- The LM-PL-weight α is important because GLEM-LM supplies node features and pseudo-labels to GLEM-GNN.
- Guiding the LM with GNN pseudo-labels consistently improves both LM and GNN performance over optimizing the LM with gold labels only.The comparison baseline is α = 0.
- The GNN-PL-weight β is not very sensitive when balancing LM pseudo-labels during GLEM-GNN training.
- Optimal α and β values vary across GNN and LM choices, so the parameters require careful selection.