Source-linked AI summary
Text Level Graph Neural Network for Text Classification
Lianzhe Huang, Dehong Ma, Sujian Li, Xiaodong Zhang, Houfeng WANG
TL;DR
Previous corpus-level GNNs for text classification face high memory consumption and difficulty supporting online testing. The paper instead builds text-level graphs with globally shared parameters and small contextual windows, achieving state-of-the-art performance with a significant memory advantage.
Problem
Previous GNN-based text classification methods use fixed corpus-level graphs, creating high memory consumption and making online testing difficult.
Method
The model builds a graph for each input text using globally shared node representations and edge weights, small windows, and message passing.
Results
The model achieves state-of-the-art performance on several text classification datasets while consuming significantly fewer memory resources than previous methods.
Takeaways & Limitations
Text-level graphs support online testing and substantially reduce graph scale and GPU memory while retaining globally shared parameters.
Abstract
from arXiv · showhide
Recently, researches have explored the graph neural network (GNN) techniques on text classification, since GNN does well in handling complex structures and preserving global information. However, previous methods based on GNN are mainly faced with the practical problems of fixed corpus level graph structure which do not support online testing and high memory consumption. To tackle the problems, we propose a new GNN based model that builds graphs for each input text with global parameters sharing instead of a single graph for the whole corpus. This method removes the burden of dependence between an individual text and entire corpus which support online testing, but still preserve global information. Besides, we build graphs by much smaller windows in the text, which not only extract more local features but also significantly reduce the edge numbers as well as memory consumption. Experiments show that our model outperforms existing models on several text classification datasets even with consuming less memory.
1 Introduction
The paper addresses practical limitations of corpus-level GNNs for text classification by constructing smaller, text-specific graphs with globally shared parameters. This design supports online testing, reduces memory use, and achieves strong classification results.
- Challenges: Corpus-level GNNs require high memory because numerous fixed-weight edges encourage large connection windows for global representations.They also depend on corpus-specific graph structure and parameters, preventing graph modification after training.
- Approach: The proposed method builds a separate graph for each input text while globally sharing node representations and edge weights.Message passing updates node representations using information from neighboring nodes, and all node representations are summarized for prediction.
- Approach: Small contextual windows reduce edges and memory consumption while allowing nodes to incorporate information from nearby words for context-specific meaning.The approach excludes many distant words considered to have little relation to the current word.
- Results: The method achieves state-of-the-art results on several text classification datasets while consuming significantly fewer memory resources than previous methods.
2 Method
The method builds a separate graph for each input text, using globally shared parameters and local word connections. Message passing incorporates contextual information before the model predicts the text label.
- Text-level graph construction: Each input text receives its own graph, whose parameters come from global shared matrices rather than a corpus-level graph.The architecture builds the graph, applies message passing, and predicts the label from learned node representations.
- Text-level graph construction: Words connect to adjacent words within a small window, with p specifying how many neighboring words each node connects.Edges connect each word to nearby words; infrequent edges can be mapped to a shared “public” edge.
- Text-level graph construction: The text-level design reduces graph nodes and edges, lowering GPU memory use and allowing graphs for new texts because they depend only on text content.Unlike previous graph construction, the approach is not dependent on the entire corpus.
- Message passing mechanism: Message passing aggregates information from neighboring nodes and updates each node representation using its original representation and received messages.The max reduction combines maximum values across dimensions, while η_n controls how much of the former representation is retained.
- Message passing mechanism: Neighborhood influence supplies contextual information, while globally shared graph parameters preserve global information for word representations.This contextual updating is intended to help determine precise meanings for polysemous words.
- Prediction: The model uses all node representations to predict the text label through an output mapping and trains by minimizing cross-entropy loss.The output mapping uses W and bias b to produce predictions from the text’s node set.
3 Experiments
The experiments evaluate the proposed model on several text classification datasets against neural and graph-based baselines, including analyses of accuracy, memory consumption, connection windows, and ablations.
- Experimental Setup: The experiments use R8, R52, and Ohsumed, with 10% of each training set reserved for validation.R8 and R52 are subsets of Reuters 21578, while Ohsumed is extracted from MEDLINE.
- Experimental Setup: The comparison includes CNN, LSTM, fastText, Graph-CNN, and Text-GCN baselines, with some baseline results taken from prior work.Models using pre-trained embeddings use 300-dimensional GloVe vectors.
- Experimental Results: The proposed model achieves state-of-the-art results on several text classification datasets and outperforms traditional and graph-based baselines.The authors attribute improvements over traditional models to graph-based neighbor structure and globally shared edge relationships, and improvements over Graph-CNN and Text-GCN to trainable edges and contextual representations.
- Analysis of Memory Consumption: The proposed model has a significant memory-consumption advantage over Text-GCN because its smaller contextual windows produce fewer edges.Text-GCN uses larger windows to obtain more accurate co-occurrence weights, whereas the proposed model yields a sparser edge-weight matrix.
- Analysis of Edges: Accuracy is best at about 3 connected neighborhoods, while larger windows eventually make the graphs increasingly similar to fully connected graphs and reduce accuracy.The analysis compares R8 and Ohsumed across p values from 1 to 19 and fully connected graphs.
- Ablation Study: Ablations show worse performance with fixed PMI-initialized edges, mean reduction, or random node initialization without pre-trained embeddings.The authors associate the results with trainable edge modeling, max reduction’s emphasis on discriminative features, and the contribution of pre-trained word embeddings.
4 Related Work
GNNs extend deep learning to graph-structured data and have been applied to text classification alongside CNNs and RNNs. Text-GCN achieves strong results but retains corpus-level graph limitations, motivating the model presented here.
- Graph Neural Networks: GNNs model non-Euclidean data such as trees and graphs, unlike traditional neural networks designed for regular grid data.
- Text Classification: CNNs and RNNs have been used for text representation and achieved better results than traditional text-classification models.
- Graph-Based Text Classification: Text-GCN achieved state-of-the-art results on several mainstream datasets but suffers from high memory consumption and lacks online-training support.
5 Conclusion
The paper proposes a graph-based text-classification model using text-level graphs instead of one corpus-level graph. Experiments report state-of-the-art performance with a significant memory-consumption advantage.
- The proposed model uses text-level graphs instead of a single graph for the whole corpus.
- Experimental results show state-of-the-art performance and a significant advantage in memory consumption.