Source-linked AI summary
Graph Convolutional Networks for Text Classification
Liang Yao, Chengsheng Mao, Yuan Luo
TL;DR
Text classification needs representations that capture information beyond the local sequences emphasized by many CNN and RNN methods. Text GCN instead models a whole corpus as a heterogeneous word-document graph and jointly learns word and document embeddings for node-based classification. It outperforms numerous state-of-the-art methods on multiple benchmark datasets, while its transductive design limits rapid prediction for unseen documents.
Problem
CNN- and RNN-based text classifiers mainly model local consecutive word sequences, leaving global word co-occurrence information insufficiently represented.
Method
Text GCN builds a heterogeneous graph for an entire corpus, connects words and documents using corpus relations, and jointly learns their embeddings with a graph convolutional network.
Results
Text GCN outperforms numerous state-of-the-art text classification methods on multiple benchmark datasets and learns predictive word and document embeddings.
Takeaways & Limitations
Text GCN can capture global word co-occurrence information and utilize limited labeled documents well.
Takeaways & Limitations
Because Text GCN is inherently transductive, it cannot quickly generate embeddings and predictions for unseen test documents.
Abstract
from arXiv · showhide
Text classification is an important and classical problem in natural language processing. There have been a number of studies that applied convolutional neural networks (convolution on regular grid, e.g., sequence) to classification. However, only a limited number of studies have explored the more flexible graph convolutional neural networks (convolution on non-grid, e.g., arbitrary graph) for the task. In this work, we propose to use graph convolutional networks for text classification. We build a single text graph for a corpus based on word co-occurrence and document word relations, then learn a Text Graph Convolutional Network (Text GCN) for the corpus. Our Text GCN is initialized with one-hot representation for word and document, it then jointly learns the embeddings for both words and documents, as supervised by the known class labels for documents. Our experimental results on multiple benchmark datasets demonstrate that a vanilla Text GCN without any external word embeddings or knowledge outperforms state-of-the-art methods for text classification. On the other hand, Text GCN also learns predictive word and document embeddings. In addition, experimental results show that the improvement of Text GCN over state-of-the-art comparison methods become more prominent as we lower the percentage of training data, suggesting the robustness of Text GCN to less training data in text classification.
Introduction
Text classification research has advanced from hand-crafted features to CNNs, RNNs, and graph neural networks, but local sequential models do not explicitly capture global corpus-level word co-occurrence. Text GCN addresses this gap by representing an entire corpus as a heterogeneous graph and jointly learning word and document embeddings for classification.
- Prior approaches: Text classification uses hand-crafted lexical features and deep models such as CNNs and RNNs to learn text representations.Traditional examples include bag-of-words and n-grams, while deep models include CNNs and LSTMs.
- Text GCN: Text GCN constructs one heterogeneous corpus graph containing word and document nodes, with word-word and document-word edges encoding corpus relations.The method uses word co-occurrence for word-word edges and word frequency and document frequency for word-document edges.
- Contributions: Text GCN jointly learns word and document embeddings for text classification without relying on pre-trained word embeddings or external knowledge.The contribution passage reports strong benchmark performance and predictive embeddings learned automatically.
- Research gap: CNN- and RNN-based methods mainly focus on local consecutive word sequences and do not explicitly use global corpus word co-occurrence.Global co-occurrence can carry non-consecutive and long-distance semantics.
- Graph neural networks: Graph neural networks operate on arbitrarily structured graphs and can preserve global graph structure in learned embeddings.GCNs generalize convolution beyond regular grids and have achieved strong results on benchmark graph datasets.
Method
Text GCN represents a corpus as a heterogeneous graph of word and document nodes, then applies graph convolutions to learn node embeddings. Edges encode document-word TF-IDF relations and positive word co-occurrence, while a two-layer model propagates information across neighborhoods and trains on labeled documents.
- Graph Convolutional Networks: GCNs operate directly on graphs, producing node embeddings from neighborhood properties and integrating larger neighborhoods when layers are stacked.A two-layer GCN allows message passing between nodes up to two steps apart.
- Text Graph Construction: Text GCN builds one heterogeneous corpus graph containing document and word nodes, with one-hot vectors as initial node features.The graph has one node for every document and unique vocabulary word.
- Text Graph Construction: Document-word edges use TF-IDF weights, while word-word edges use positive PMI from word co-occurrence in corpus sliding windows.Negative PMI values do not produce word-word edges.
- Model and Training: After graph construction, a two-layer GCN produces word and document embeddings whose output dimension equals the number of classes, followed by softmax classification.The model is trained with cross-entropy over labeled documents.
- Model and Training: Two GCN layers enable information exchange between documents despite the absence of direct document-document edges.Documents can communicate through shared word nodes within two propagation steps.
Experiment
The experiments evaluate Text GCN on benchmark text-classification datasets against established classifiers and embedding methods, then examine sensitivity to graph and training-data settings. Text GCN achieves strong classification, learns discriminative word and document embeddings, and remains effective with limited labeled data, while its transductive design limits prediction for unseen documents.
- Test Performance: Text GCN does not outperform CNN and LSTM-based models on MR, where word order and short documents provide fewer graph edges.CNN and LSTM explicitly model consecutive word sequences, whereas GCN ignores word order.
- Parameter Sensitivity: Test accuracy increases with sliding-window size before leveling off beyond 15, indicating a trade-off between insufficient and overly broad word co-occurrence edges.Small windows provide insufficient global co-occurrence information, while large windows may connect weakly related nodes.
- Parameter Sensitivity: Higher-dimensional first-layer embeddings do not improve classification, while low dimensions may inadequately propagate label information and increase training costs at high dimensions.The reported trend is similar to the sliding-window analysis.
- Effects of the Size of Labeled Data: 0.8063 ± 0.0025 accuracy on 20NG with 20% training data and 0.8830 ± 0.0027 on R8 with 1% show strong performance with limited labels.These scores exceed some baseline models trained with the full training sets.
- Document Visualization: Text GCN learns more discriminative document embeddings than PV-DBOW and PTE, with second-layer embeddings more distinguishable than first-layer embeddings.The comparison uses t-SNE visualizations of 20NG test-document embeddings.
- Word Visualization: Second-layer word embeddings group words by document-class label, and their highest-valued words are interpretable for several 20NG classes.The analysis reports top-ten words for each class, including “jpeg,” “graphics,” and “image” for one class.
- Discussion: Text GCN is inherently transductive, so it cannot quickly generate embeddings or predictions for unseen test documents.The model includes unlabeled test-document nodes during GCN training.
Conclusion and Future Work
Text GCN turns document classification into node classification on a heterogeneous word-document graph, capturing global word co-occurrence and using limited labeled documents effectively. Future work includes inductive generalization, attention mechanisms, and unsupervised learning for large unlabeled corpora.
- Conclusion: Text GCN builds a heterogeneous word-document graph for the whole corpus and converts document classification into node classification.
- Conclusion: Text GCN captures global word co-occurrence information and utilizes limited labeled documents effectively.
- Conclusion: A simple two-layer Text GCN outperforms numerous state-of-the-art methods across multiple benchmark datasets.
- Future Work: Future directions include inductive Text GCN models, attention mechanisms for classification, and unsupervised representation learning on large-scale unlabeled text.