Source-linked AI summary
Tensor Graph Convolutional Networks for Text Classification
Xien Liu, Xinxin You, Xiao Zhang, Ji Wu, Ping Lv
TL;DR
Text classification needs graph-based learning that can represent heterogeneous semantic, syntactic, and sequential information from free text. TensorGCN constructs a text graph tensor and jointly applies intra-graph and inter-graph propagation; experiments report effective performance, while the method does not support forcing strongly heterogeneous graphs to share one trainable weight.
Problem
Graph-based text learning must construct graphs from free text, and jointly learning heterogeneous graphs requires a mechanism that preserves their differences while harmonizing their information.
Method
TensorGCN constructs semantic, syntactic, and sequential text graphs and jointly performs neighborhood aggregation within graphs and information harmonization between graphs.
Results
TensorGCN significantly outperforms all baselines in the reported benchmark comparison, and the graph tensor performs best among single-graph, two-graph, and three-graph configurations.
Takeaways & Limitations
The three text graphs provide complementary contextual constraints, while inter-graph propagation is important for jointly learning heterogeneous graph information.
Takeaways & Limitations
Forcing all graphs to share the same trainable weights does not work when the graphs are strongly heterogeneous.
Abstract
from arXiv · showhide
Compared to sequential learning models, graph-based neural networks exhibit some excellent properties, such as ability capturing global information. In this paper, we investigate graph-based neural networks for text classification problem. A new framework TensorGCN (tensor graph convolutional networks), is presented for this task. A text graph tensor is firstly constructed to describe semantic, syntactic, and sequential contextual information. Then, two kinds of propagation learning perform on the text graph tensor. The first is intra-graph propagation used for aggregating information from neighborhood nodes in a single graph. The second is inter-graph propagation used for harmonizing heterogeneous information between graphs. Extensive experiments are conducted on benchmark datasets, and the results illustrate the effectiveness of our proposed framework. Our proposed TensorGCN presents an effective way to harmonize and integrate heterogeneous information from different kinds of graphs.
Introduction
TensorGCN addresses text classification by constructing a text graph tensor that captures semantic, syntactic, and sequential context, then learning across these heterogeneous graphs.
- Introduction: Graph-based text learning must construct graphs from free text before applying a graph-based learning framework.TextGCN previously used a text graph for classification and achieved state-of-the-art performance on some benchmark datasets.
- Introduction: TensorGCN constructs semantic-based, syntactic-based, and sequential-based text graphs to form a text graph tensor.The framework is introduced for text classification and is illustrated in Figure 1.
- Introduction: The graph tensor captures semantic, syntactic, and sequential contextual information from the text.These contexts correspond to the different graphs in the tensor.
- Introduction: TensorGCN performs intra-graph propagation to aggregate neighborhood information and inter-graph propagation to harmonize heterogeneous information between graphs.The two propagation processes are performed for each layer.
Related work
Prior graph-based text models use limited graph information or impose parameter sharing across heterogeneous graphs, motivating TensorGCN’s multi-graph learning design.
- Related work: TextGCN uses one text graph to describe local co-occurrence constraints, leaving non-sequential text information incompletely considered.The cited prior work applies standard graph convolutional networks to text classification.
- Related work: Other work uses semantic and syntactic graphs independently rather than jointly learning their information.A separate multi-channel framework aligns entities but uses shared trainable parameters across two graphs.
- Related work: When graphs are very different and heterogeneous, forcing them to share trainable parameters does not work.This limitation is identified for the multi-channel graph learning framework discussed in the related work.
Methods
The paper uses graph convolutional networks as the base component for learning over a tensor of text graphs, after defining graph tensors and their construction from a text corpus.
- Methods: TensorGCN uses graph convolutional networks as the base component for text graph tensor learning.The paper motivates this choice by GCN’s simplicity and effectiveness in practice.
- Methods: The methods section covers GCN, a simple definition of graph tensors, graph-tensor construction from a text corpus, and the TensorGCN learning model.These components are presented in sequence as the basis for the proposed approach.
Graph convolutional networks (GCN)
A GCN updates node representations by propagating information through a graph adjacency matrix, applying trainable transformations and nonlinear activation across layers.
- Graph convolutional networks (GCN): A GCN operates directly on graph data by encoding graph structure and node features with a propagation rule.The graph is represented as G = (V, E, A), where A is the adjacency matrix.
- Graph convolutional networks (GCN): Each GCN layer maps H^(l) and A to the next hidden representation H^(l+1).The layers are indexed from l = 0 through L.
- Graph convolutional networks (GCN): The common propagation rule uses a symmetrically normalized adjacency matrix with added self-connections, a trainable weight matrix, and a nonlinear activation.The normalized matrix is based on A + I, while the final layer uses softmax and outputs one feature per label.
- Graph convolutional networks (GCN): H^(0) is the initial feature matrix, with each row representing a node’s initial input feature.Hidden-layer feature dimensions determine the number of features for each node at each layer.
Graph tensor definition
A graph tensor packages multiple graphs that share nodes but differ in edges and adjacency matrices, along with graph-specific node features.
- A graph tensor consists of multiple graphs sharing the same node set while differing in adjacency structure.
- Each graph G_i contains nodes V_i, edges E_i, and adjacency matrix A_i.
- The adjacency matrices are packed into an adjacency tensor A ∈ R^r×n×n, with one matrix for each graph.
- The graph feature tensor stacks one feature matrix per graph, with H^(l) ∈ R^r×n×d_l at layer l.
- At layer zero, H^(0) represents the initialized input features for the graph tensor.
Text graph tensor construction
The text graph tensor represents documents through word-document and word-word relations, combining semantic, syntactic, and sequential graphs constructed from corpus statistics and learned representations.
- Text graphs use words and documents as nodes, with word-document edges weighted by TF-IDF and word-word edges representing relations.
- The semantic graph uses LSTM-derived word embeddings and cosine similarity to identify semantic relationships within documents.
- A word pair receives a semantic edge weight based on its corpus-wide semantic relationship count normalized by its co-occurrence count.
- The syntactic graph extracts word dependencies with Stanford CoreNLP, treats directed dependencies as undirected, and weights pairs by corpus counts.
- The sequential graph models local word co-occurrence using point-wise mutual information computed over sliding windows.
- Figure 2 illustrates semantic graph construction by collecting LSTM-based semantic word pairs across the text corpus.
Graph tensor learning
TensorGCN learns directly on a graph tensor rather than collapsing heterogeneous graphs into one representation. Each layer first propagates information within graphs, then exchanges information across graphs to harmonize their representations.
- Preliminary model: merge edges + GCN: The preliminary “Merge edges + GCN” model reduces a graph tensor to one graph by pooling the adjacency matrices before standard GCN learning.Pooling can use max or mean pooling.
- Preliminary model: merge edges + GCN: This edge-merging approach can destroy graph-specific structure because different graphs encode different properties.
- Intra-graph propagation: TensorGCN performs intra-graph propagation first, aggregating each node’s neighborhood information separately within every graph.The graph tensor uses normalized symmetric adjacency matrices, with self-connections added during normalization.
- Model output: The model uses graph- and layer-specific trainable weights, and mean-pools graph representations in the final layer for document classification.Experiments reported in the paper found that forcing all graphs to share trainable weights did not work.
- Inter-graph propagation: It then performs inter-graph propagation, exchanging information between graphs through virtual graphs constructed from copied nodes across the tensor.The resulting virtual-graph adjacency tensor contains n virtual graphs, with edge weights set to 1.
- Inter-graph propagation: Inter-graph propagation is intended to gradually fuse heterogeneous information from different graphs into an accordant representation.Unlike standard GCN propagation, the virtual-graph adjacency matrices are not symmetrically normalized and do not receive self-connections.
Experiments and results analysis
Experiments on five benchmark datasets evaluate TensorGCN against established baselines and analyze its graph construction and multi-graph learning strategies. TensorGCN achieves the strongest reported performance, while complementary graph types and inter-graph propagation contribute to the results.
- Experimental setup: Experiments use five benchmark datasets spanning sentiment, news, and medical-literature classification tasks.The datasets are 20-Newsgroups, Ohsumed, R52 Reuters, R8 Reuters, and Movie Review.
- Experimental setup: Baselines include traditional, word-embedding, sequence-learning, and graph-based models, evaluated with the same datasets and settings as prior work.Reported baseline results from Yao, Mao, and Luo (2019) are used for a fair comparison.
- Overall performance: TensorGCN significantly outperforms all baselines, including state-of-the-art embedding-learning and graph-based models.Text GCN is equivalent to TensorGCN when only the sequential-based graph is used.
- Text graph tensor analysis: The graph tensor has the best performance, while every pair of graphs outperforms any single graph, indicating complementary context constraints.The analyzed graphs are SeqGraph, SynGraph, and SemGraph.
- Text graph tensor analysis: Removing SynGraph produces worse accuracy on Ohsumed and MR, showing that syntactic dependency information is important for those datasets.Figure 4 illustrates how syntactic dependency information enhances classification performance on an MR example.
- TensorGCN learning analysis: TensorGCN performs better than TensorGCN(intra) on almost all test datasets, making inter-graph propagation important for jointly learning heterogeneous graphs.TensorGCN(intra) omits inter-graph propagation and uses only intra-graph propagation.
Conclusions
The paper proposes a text graph tensor that captures semantic, syntactic, and sequential context, together with TensorGCN for jointly learning these heterogeneous graphs. Experiments indicate that the contexts are complementary and that intra-graph and inter-graph propagation effectively integrate multi-graph information.
- The text graph tensor captures semantic, syntactic, and sequential contextual information for text representation learning.
- Experimental results show that the three context constraints are complementary and important for text representation learning.
- TensorGCN harmonizes and integrates heterogeneous information from multiple graphs through simultaneous intra-graph and inter-graph propagation.