Source-linked AI summary
Be More with Less: Hypergraph Attention Networks for Inductive Text Classification
Kaize Ding, Jianling Wang, Jundong Li, Dingcheng Li, Huan Liu
TL;DR
Existing GNN-based text classification methods are limited in capturing high-order word interactions and in efficiently handling large datasets and new documents. HyperGAT models documents as hypergraphs with dual attention, and experiments report superiority over state-of-the-art methods. The paper concludes that this approach provides more expressive power with less computational consumption for text representation learning.
Problem
Existing GNN-based text classification methods cannot capture high-order word interactions and are computationally inefficient for large datasets and new documents.
Method
HyperGAT models each text document as a document-level hypergraph and uses a dual attention mechanism to learn expressive text representations inductively.
Results
HyperGAT outperforms state-of-the-art methods on text classification across the reported benchmark evaluations.
Takeaways & Limitations
The proposed approach acquires more expressive power with less computational consumption for text representation learning.
Takeaways & Limitations
The study discusses sequential and semantic hyperedges, while leaving other meaningful hyperedges, such as syntactic-related ones, for future work.
Abstract
from arXiv · showhide
Text classification is a critical research topic with broad applications in natural language processing. Recently, graph neural networks (GNNs) have received increasing attention in the research community and demonstrated their promising results on this canonical task. Despite the success, their performance could be largely jeopardized in practice since they are: (1) unable to capture high-order interaction between words; (2) inefficient to handle large datasets and new documents. To address those issues, in this paper, we propose a principled model -- hypergraph attention networks (HyperGAT), which can obtain more expressive power with less computational consumption for text representation learning. Extensive experiments on various benchmark datasets demonstrate the efficacy of the proposed approach on the text classification task.
1 Introduction
Text classification supports broad NLP applications, while existing GNN approaches face limits in modeling high-order word interactions and efficiently handling new documents. HyperGAT addresses these issues with document-level hypergraphs and dual attention, and benchmark experiments report superiority over other state-of-the-art methods.
- Text classification is a fundamental NLP task with applications including sentiment analysis, topic labeling, and disease diagnosis.
- Existing GNN-based methods use corpus-level graphs and treat text classification as semi-supervised node classification, leveraging training and test documents.
- Pairwise word relations can miss high-order interactions, as illustrated by the idiom “eat humble pie.”A simple graph may misinterpret the meaning of “pie” from its pairwise connections.
- Accessing test documents during training makes existing methods transductive, requiring retraining from scratch when new documents arrive.
- HyperGAT models each text document with a document-level hypergraph, whose hyperedges can connect arbitrary numbers of nodes.The approach is designed to increase expressive power while reducing computational consumption.
- HyperGAT uses a dual attention mechanism for representation learning on text hypergraphs and is reported superior to other state-of-the-art methods across multiple benchmark datasets.
2 Related Work
Text classification has been approached with neural representation learning, attention mechanisms, and graph-based models. HyperGAT extends this line of work by applying hypergraph modeling to capture higher-order dependencies while addressing limitations of transductive document-level graphs.
- Graph Neural Networks: GNNs learn latent node representations through neighborhood aggregation and message passing among local graph neighbors.Graph attention networks add trainable attention weights to assign fine-grained neighbor importance.
- Hypergraph Methods: Hypergraph neural networks extend graph-based learning to capture high-order dependencies between nodes.HyperGAT applies this capability to text classification.
- Neural Text Classification: CNNs and RNNs are representative neural models that learn text embeddings for classification.Attentional models were later developed to improve representation expressiveness.
- Research Gap: Existing document-level graph methods are computationally inefficient, transductive, and unable to capture high-order word interactions.These limitations motivate HyperGAT’s inductive hypergraph-based approach.
3 Methodology
HyperGAT models each document as a text hypergraph and learns representations through dual node- and edge-level attention. The resulting document representation is classified with a softmax layer, while unseen documents can be processed inductively without retraining.
- 3.2 Documents as Text Hypergraphs: HyperGAT represents each document as a hypergraph whose nodes are words and whose hyperedges connect multiple words with heterogeneous context.Hyperedges model sequential and semantic context within individual documents.
- 3.2 Documents as Text Hypergraphs: Semantic hyperedges connect the top K words associated with each latent LDA topic, enriching high-order semantic context.Each topic is treated as a hyperedge over its highest-probability words.
- 3.3 Hypergraph Attention Networks: HyperGAT uses two aggregation functions with dual attention to capture heterogeneous high-order word interactions and emphasize informative information at different granularities.Node-level attention weights words within hyperedges, while edge-level attention weights hyperedges connected to a node.
- 3.3 Hypergraph Attention Networks: The dual attention mechanism highlights important nodes when constructing hyperedge representations and informative hyperedges when updating node representations.The two stages aggregate node features to hyperedges and hyperedge features to nodes.
- 3.4 Inductive Text Classification: After L HyperGAT layers, mean pooling produces a document representation that is mapped to predicted label scores by a softmax classifier.Training minimizes cross-entropy loss over labeled documents.
- 3.4 Inductive Text Classification: HyperGAT processes unseen documents by feeding their text hypergraphs into the learned model, eliminating mandatory test-document access and retraining.This supports inductive handling of newly added data.
4 Experiments
Experiments evaluate HyperGAT across five benchmark datasets, baselines, computational cost, sensitivity, ablations, and embedding visualizations. HyperGAT outperforms the compared methods while using document-level hypergraphs for inductive, high-order text representation learning.
- Experimental Setting: Experiments use five benchmark datasets spanning news, medical, and sentiment classification, with word-embedding, sequence-based, and graph-based baselines.The datasets are 20NG, R8, R52, Ohsumed, and MR.
- Classification Performance: HyperGAT outperforms all baselines on the five evaluation datasets, while graph-based methods generally perform strongly on the first four datasets.Sequence-based methods are stronger than most graph-based baselines on MR, where sequential context is important for sentiment classification.
- Computational Efficiency: HyperGAT reduces memory consumption by training on batches of document-level hypergraphs and generalizes to unseen documents without retraining the whole model.TextGCN instead constructs a large document-word graph using both training and test documents.
- Model Sensitivity: The best HyperGAT performance occurs with a first-layer embedding size of 300, while smaller sizes may reduce expressiveness and larger sizes may overfit.Figure 2 reports this sensitivity on 20NG and Ohsumed.
- Model Sensitivity: With more labeled training data, all evaluated methods improve, but HyperGAT significantly outperforms other baselines when labeled data are limited.Figure 3 reports results for Ohsumed and MR across training proportions from 2.5% to 75%.
- Ablation Analysis and Case Studies: Ablations and visualizations support the roles of layered high-order interactions, sequential and semantic hyperedges, and dual attention in HyperGAT.Sequential hyperedges are more important in the ablation comparison, semantic hyperedges enhance expressiveness, and HyperGAT learns more expressive document representations in the Ohsumed t-SNE visualization.
5 Conclusion
The study proposes HyperGAT, a graph-based method that models documents with document-level hypergraphs for inductive text classification. It aims to learn discriminative text representations with greater expressive power and lower computational consumption, and experiments report superiority over state-of-the-art methods.
- HyperGAT models text documents with document-level hypergraphs for inductive text classification.
- The model forms a new family of GNNs named HyperGAT to learn discriminative text representations.
- HyperGAT is designed to acquire more expressive power with less computational consumption for text representation learning.
- Extensive experiments demonstrate the proposed model’s superiority over state-of-the-art methods.
A.1 Implementation Details
The implementation trains LDA on training documents to construct semantic hyperedges, then trains a two-layer HyperGAT model with specified dimensions and settings. Validation accuracy is reported across five datasets.
- LDA is trained only on each dataset’s training documents, with topic number equal to the number of classes.The top 10 keywords from each topic are used to construct semantic hyperedges.
- Semantic hyperedges are constructed from the top 10 keywords of each LDA topic.
- HyperGAT uses two layers with 300 and 100 embedding dimensions, one-hot node attributes, and batch size 8.The model is implemented in PyTorch, optimized with Adam, and trained and tested on a 12 GB Titan Xp GPU.
- 0.9355 ± 0.0011, 0.9755 ± 0.0019, 0.9375 ± 0.0023, 0.6964 ± 0.0024 and 0.7779 ± 0.0015 validation accuracy are reported for 20NG, R8, R52, Ohsumed and MR, respectively.
A.2 Space Complexity Analysis
HyperGAT reduces the scale of the adjacency representation by using document-level hypergraphs rather than a corpus-level document-word graph. Its mini-batch formulation therefore targets lower memory consumption.
- TextGCN’s adjacency matrix has size (N + M)2, where N is vocabulary size and M is document size.
- For each HyperGAT document hypergraph, the adjacency matrix has size n×m, where n is the number of words and m is the number of hyperedges.
- Mini-batch training with document-level hypergraphs provides the basis for lower memory consumption than the corpus-level formulation.