Source-linked AI summary
BertGCN: Transductive Text Classification by Combining GCN and BERT
Yuxiao Lin, Yuxian Meng, Xiaofei Sun, Qinghong Han, Kun Kuang, Jiwei Li, Fei Wu
TL;DR
Existing transductive text classifiers do not incorporate large-scale pretraining, leaving its effectiveness unclear. BertGCN combines BERT with corpus-level graph convolution and achieves the best performance across all evaluated datasets.
Problem
Existing transductive text classification models do not incorporate large-scale pretraining, whose effectiveness for this setting remains unclear.
Method
BertGCN jointly trains BERT and GCN modules on a heterogeneous word-document graph to propagate label information across labeled and unlabeled documents.
Results
BertGCN and RoBERTaGCN perform best across all datasets, with especially significant gains over their pretrained-only counterparts on 20NG and Ohsumed.
Takeaways & Limitations
The results demonstrate that graph neural networks can benefit significantly from large-scale pretraining in transductive text classification.
Takeaways & Limitations
The memory bank makes stored BERT embeddings inconsistent across training steps, requiring a low learning rate that increases training time.
Abstract
from arXiv · showhide
In this work, we propose BertGCN, a model that combines large scale pretraining and transductive learning for text classification. BertGCN constructs a heterogeneous graph over the dataset and represents documents as nodes using BERT representations. By jointly training the BERT and GCN modules within BertGCN, the proposed model is able to leverage the advantages of both worlds: large-scale pretraining which takes the advantage of the massive amount of raw data and transductive learning which jointly learns representations for both training data and unlabeled test data by propagating label influence through graph convolution. Experiments show that BertGCN achieves SOTA performances on a wide range of text classification datasets. Code is available at https://github.com/ZeroRin/BertGCN.
1 Introduction
BertGCN combines large-scale pretraining with transductive learning for text classification by jointly training BERT and GCN modules on a heterogeneous corpus graph. This design uses pretrained representations alongside graph-based propagation across labeled and unlabeled instances.
- Motivation: Text classification is a core NLP task, while transductive learning uses both labeled and unlabeled examples during training.Graph neural networks are presented as an effective approach for transductive learning.
- BertGCN: Jointly training BERT and GCN enables label influence to propagate across training and test instances, allowing unlabeled data to contribute to representation learning.Graph-based representations also incorporate neighboring nodes, making the model more immune to data outliers.
- Motivation: Large-scale pretrained models learn rich text semantics from massive unlabeled corpora, motivating their integration with transductive text classification.Existing transductive text-classification models are identified as lacking this combination.
- BertGCN: BertGCN constructs a heterogeneous graph with word and document nodes, initializes node embeddings with pretrained BERT representations, and uses GCNs for classification.The model combines pretrained language representations with graph convolution over the corpus.
2 Related Work
Related work spans neural architectures for text classification and graph neural networks, including GCNs, GATs, and graph auto-encoders. BertGCN differs from prior BERT–GNN approaches by combining pretrained models with corpus-level graphs over different samples rather than token-level graphs within one document.
- Graph Neural Networks: GNNs capture dependencies and relations between graph nodes through message passing along connecting edges.Practical categories include graph convolutional networks, graph attention networks, and graph auto-encoders.
- Neural Text Classification: Neural architectures have demonstrated effectiveness for text classification compared with traditional statistical feature-based methods.Related approaches also jointly train label embeddings with input texts.
- BERT and GNNs: BertGCN combines large-scale pretrained models and GNNs, whereas existing BERT–GNN methods model token relationships within individual documents for inductive learning.BertGCN instead models relationships between different samples across the whole corpus to use similarity between labeled and unlabeled data.
3 Method
BertGCN combines BERT document embeddings with a heterogeneous TextGCN-style graph, propagating information across labeled and unlabeled documents for classification. Joint optimization uses GCN predictions, an auxiliary BERT classifier, and a memory bank to make training feasible with mini-batches.
- BERT-GCN integration: BERT-style document embeddings initialize document nodes, after which GCN message passing produces final document representations for softmax classification.The GCN iteratively propagates messages across training and test examples.
- Graph construction: BertGCN constructs a heterogeneous graph of word and document nodes, using TF-IDF for word-document edges and PPMI for word-word edges.The graph includes training and test documents.
- Joint training: Cross-entropy loss over labeled document nodes jointly optimizes the BERT and GCN parameters.GCN outputs are fed to the softmax classifier for prediction.
- Auxiliary classifier: An auxiliary classifier applies a dense softmax layer directly to BERT embeddings, and its prediction is linearly interpolated with the BertGCN prediction.This auxiliary classifier yields faster convergence and better performance; λ = 1 uses full BertGCN, while λ = 0 uses only BERT.
- Memory-bank training: A memory bank stores document embeddings for all nodes, enabling mini-batch training by updating sampled labeled and unlabeled records while treating the remaining memory-bank entries as constant.At each epoch, embeddings are recomputed with the current BERT module before mini-batch updates.
- Memory-bank training: Because stored embeddings are inconsistent across an epoch, BertGCN uses a small BERT learning rate and fine-tunes BERT on the target dataset before training to accelerate optimization.The small learning rate improves consistency but makes training take more time.
4 Experiments
Experiments evaluate BertGCN and related variants on five text-classification benchmarks against pretrained and graph-based baselines. BertGCN and RoBERTaGCN perform best overall, with graph propagation particularly beneficial for longer-document datasets such as 20NG and Ohsumed.
- Datasets and setup: Experiments cover five benchmarks: 20NG, R8, R52, Ohsumed, and Movie Review.Data preprocessing follows TextGCN protocols.
- Datasets and setup: The comparison includes TextGCN, SGC, BERT, and RoBERTa, while BertGCN uses BERTbase with a two-layer GCN.The GCN and fine-tuned BERT learning rates are initialized to 1e-3 and 1e-5, respectively.
- Main results: BertGCN and RoBERTaGCN perform best across all datasets, while pretrained BERT and RoBERTa generally outperform GCN variants except on 20NG.The table reports mean test accuracy over 10 runs.
- Main results: Graph-based enhancements provide significant gains over BERT and RoBERTa on 20NG and Ohsumed, where longer documents create more document connections for message passing.For shorter-document datasets such as R52 and MR, the graph structure is more limited and the performance boost is smaller.
- Ablation and analysis: On 20NG, RoBERTaGCN reaches its best accuracy at λ = 0.7, slightly exceeding the λ = 1 configuration using only the GCN prediction.λ controls the trade-off between training BertGCN and BERT, and its optimum can vary by task.
- Ablation and analysis: Using the same learning rate for RoBERTa and GCN prevents training because of memory-bank embedding inconsistency, motivating a smaller BERT learning rate and fine-tuned initialization.The two strategies are evaluated on RoBERTaGCN using the 20NG development set.
5 Conclusion and Future Work
BertGCN combines large-scale pretraining with transductive learning for text classification, using an efficient memory-bank training strategy. Its framework is compatible with any document encoder and graph model, and experiments demonstrate its effectiveness.
- Conclusion: BertGCN combines large-scale pretraining and transductive learning to improve text classification.The model is designed to leverage the advantages of both approaches.
- Conclusion: A memory bank stores all document embeddings and updates a sampled mini-batch subset during training.This enables efficient BertGCN training.
- Conclusion: The BertGCN framework can be built on top of any document encoder and any graph model.
- Conclusion: Experiments demonstrate the power of the proposed BertGCN model.
A Dataset Details
The experiments evaluate text classification across five datasets spanning newsgroup, Reuters, medical-reference, and movie-review domains, with dataset-specific categories and train/test splits.
- 20NG: 20NG contains 18,846 posts across 20 topics, split chronologically into 11,314 training and 7,532 test samples.The bydate version uses posting dates for the split.
- Reuters: R8 and R52 are Reuters subsets with 8 and 52 categories, containing 5,485/2,189 and 6,532/2,568 training/test documents, respectively.The datasets differ in both category count and split sizes.
- OHSUMED: OHSUMED uses 7,400 MEDLINE references from 23 disease categories, with 3,357 documents for training and 4,043 for testing.This classification dataset follows the setup used in previous work.
- MR: MR is a 10,662-review movie dataset for binary sentiment classification, using the train/test split from Tang et al. (2015).MR refers to Pang and Lee (2005).
B Baselines
The paper compares BertGCN with graph-based baselines TextGCN and SGC, as well as pretrained language models BERT and RoBERTa.
- Graph-based baselines: TextGCN applies graph convolution to a word-document heterogeneous graph with identity-matrix node features.Its graph includes both word and document nodes.
- Graph-based baselines: SGC simplifies GCN by removing nonlinearities and collapsing weight matrices between consecutive layers.The simplifications reduce GCN complexity.
- Pretrained language-model baselines: BERT serves as a large-scale pretrained NLP baseline.BERT is identified as a pretrained language model rather than a graph-based method.
- Pretrained language-model baselines: RoBERTa is included as a robustly optimized BERT baseline.The supplied passage identifies RoBERTa as a robustly optimized version of BERT.