Source-linked AI summary
VGCN-BERT: Augmenting BERT with Graph Embedding for Text Classification
Zhibin Lu, Pan Du, Jian-Yun Nie
TL;DR
Text classification models such as BERT capture contextual information but are limited in representing vocabulary-level global information, while GCNs provide that global perspective. The paper proposes VGCN-BERT, which integrates vocabulary-graph and word embeddings through BERT's attention layers. Across five datasets, the model outperforms BERT and GCN alone, although its graph setup requires entities from test data to be present in the graph.
Problem
BERT captures local and contextual information but remains limited in explicitly encoding vocabulary-level global information and long-range dependencies.
Method
VGCN-BERT constructs a vocabulary graph from word relations, combines graph and word embeddings in BERT, and lets local and global information interact through attention layers.
Results
Across five text-classification datasets, VGCN-BERT outperforms BERT and GCN alone.
Takeaways & Limitations
The results support combining BERT's local information with vocabulary-graph global information through interacting network layers.
Takeaways & Limitations
The vocabulary-graph setup requires all entities, including unseen test data, to be present in the graph, limiting use when test data are unavailable during training.
Abstract
from arXiv · showhide
Much progress has been made recently on text classification with methods based on neural networks. In particular, models using attention mechanism such as BERT have shown to have the capability of capturing the contextual information within a sentence or document. However, their ability of capturing the global information about the vocabulary of a language is more limited. This latter is the strength of Graph Convolutional Networks (GCN). In this paper, we propose VGCN-BERT model which combines the capability of BERT with a Vocabulary Graph Convolutional Network (VGCN). Local information and global information interact through different layers of BERT, allowing them to influence mutually and to build together a final representation for classification. In our experiments on several text classification datasets, our approach outperforms BERT and GCN alone, and achieve higher effectiveness than that reported in previous studies.
1 Introduction
BERT captures contextual and local information but remains limited in explicitly representing vocabulary-level global information and long-range dependencies. VGCN-BERT combines vocabulary-graph information with BERT through attention, and outperforms BERT and GCN alone across five datasets.
- BERT and related neural models capture semantic, syntactic, and contextual information, but deep learning models can remain limited in encoding long-range dependencies.
- Self-attention derives representations from the sentence or document without explicitly incorporating knowledge about the language vocabulary.
- Implicit positive sentiment can be underweighted when a movie-review expression is not connected explicitly to the meaning of innovation.
- VGCN-BERT constructs a vocabulary graph from word co-occurrence information and feeds graph and word embeddings together into BERT's self-attention encoder.
- On five datasets spanning sentiment analysis, grammaticality detection, and hate speech detection, VGCN-BERT outperforms BERT and GCN alone.
- The model integrates local and global information through interactions across network layers, rather than treating them as separate representations.
2 Related Work
Prior work uses self-attention to model sentence context and graph networks to encode global word relations, but these mechanisms have complementary weaknesses. Existing GCN-BERT combinations often concatenate outputs or target specialized domains, motivating tighter interaction between local and global information.
- BERT uses multilayer, multi-head self-attention to enhance input representations with contextual information from different parts of a text.
- Graph neural networks represent global word relations in graphs and incorporate neighboring-word information through graph convolution.
- Text GCN uses a heterogeneous graph of words and documents with co-occurrence, tf-idf, and document-similarity relations.
- GCN-based models capture global graph information but omit local features such as word order, whereas BERT captures local textual context.
- Existing GCN-BERT combinations include graph embeddings followed by BERT for medical-code prediction and simple concatenation of GCN and BERT outputs for citation recommendation.
- The proposed interaction aims to let graph information influence the input text and textual representations spread over the graph during prediction.
- Application-specific vocabulary graphs can capture dependencies beyond general connections represented by pretrained word embeddings.
3 Proposed Method
VGCN-BERT complements BERT’s sentence-level representation with a vocabulary graph that encodes global language information. The relevant graph is embedded, combined with word embeddings, and integrated through BERT’s self-attention layers for classification.
- Vocabulary Graph: The model selects the vocabulary-graph portion relevant to the input sentence and transforms it into a graph embedding.The graph is constructed from word co-occurrences, and its relevant subgraph is selected before embedding.
- Vocabulary Graph: The vocabulary graph uses NPMI-based word connections, with edges created when the association exceeds a threshold.The reported effective threshold range is 0.0 to 0.3, and the graph uses whole-sentence windows to obtain long-range dependency information.
- Vocabulary GCN: Unlike document-node GCNs, the proposed graph is built on vocabulary words, so it does not require every classified document to appear as a graph node.The paper identifies the requirement to include training, validation, and test entities as a limitation of conventional GCN applications.
- Vocabulary GCN: A vocabulary GCN applies graph convolutions to combine input words with related vocabulary terms and produce graph embeddings.The convolution first extracts the graph portion relevant to the input, then performs two convolutional layers with a ReLU function.
- Integrating VGCN into BERT: VGCN-BERT feeds both sequence word embeddings and vocabulary graph embeddings into BERT’s transformer, where self-attention integrates local and global information layer by layer.The final representation is passed to a fully connected layer for classification.
4 Experiment
The experiments evaluate VGCN-BERT against neural and graph-based baselines on five text-classification datasets. VGCN-BERT generally performs best, with results attributed to combining and allowing interaction between local and global information.
- Experimental setup: VGCN-BERT is evaluated against BERT, VGCN, Text GCN, MLP, Bi-LSTM, and Vanilla-VGCN-BERT on five text-classification datasets.The datasets cover sentiment analysis, grammaticality detection, and hate speech detection.
- Evaluation: The experiments use weighted average F1-score and macro F1-score on test sets, with five runs under the same preprocessing and random seed.FountaHate uses mean squared error to leverage annotator voting information; the other models use cross-entropy.
- Results: VGCN-BERT outperforms all baseline models except Vanilla-VGCN-BERT on MR, and exceeds both VGCN and BERT alone.The comparison supports combining vocabulary-graph information with BERT rather than using either mechanism alone.
- Results: BERT outperforms MLP and Bi-LSTM among local-information models, while VGCN and Text GCN show similar performance among vocabulary-graph models.These comparisons distinguish the performance of local-information and graph-based baselines.
- Results: Vanilla-VGCN-BERT and VGCN-BERT generally outperform other baselines, with VGCN-BERT usually better because it permits local-global interactions.Vanilla-VGCN-BERT concatenates separate representations without interaction, whereas VGCN-BERT lets the two information types interact through attention.
- Visualization: In the visualization example, VGCN-BERT attends substantially to graph-embedding dimensions associated with “innovation,” while BERT and Vanilla-VGCN-BERT do not effectively use graph embedding.The graph embedding integrates sentence-local and vocabulary-global information, supporting the correct positive classification for the ambiguous review.
5 Conclusion and Future Work
The paper proposes VGCN-BERT to combine BERT’s local information with global vocabulary information through interacting attention layers. Experiments on five datasets show better results than BERT and VGCN alone, while future work considers richer vocabulary graphs.
- Conclusion: VGCN-BERT integrates a vocabulary graph embedding module with BERT so local and global information interact through attention layers.The vocabulary graph is intended to complement the local information captured by BERT.
- Conclusion: Experiments on five classification datasets show that graph embeddings provide useful global information to BERT and improve performance over BERT and VGCN alone.The reported results indicate that the model benefits from both mechanisms.
- Future Work: Future work will consider WordNet and other lexical resources alongside co-occurrence graphs because they may provide word connections that NPMI does not cover.The proposed direction is to combine several lexical resources in the vocabulary graph.