PaperlayerView exact revision on arXiv

Source-linked AI summary

Recursive Graphical Neural Networks for Text Classification

Wei Li, Shuheng Li, Shuming Ma, Yancheng He, Deli Chen, Xu Sun

arXiv:1909.08166v1cs.CL

TL;DR

Sequence-based models can struggle to represent long-range dependencies, while graph convolutional networks can suffer from over-smoothing. ReGNN uses LSTM-filtered neighbor information and a graph-level node, and its text-classification experiments report strong baseline comparisons and reduced over-smoothing.

  • Problem

    Sequence-based models can have weaker ability to capture long-range dependencies, while stacking GNN layers can make node values converge to similar representations.

    Source: Introduction · p. 1

  • Method

    ReGNN builds text graphs using word co-occurrence information and uses LSTM to filter attention-aggregated neighbor information, with a graph-level node representing and interacting with the whole graph.

    Source: Introduction · p. 1, Text Graph Construction · p. 2, Updation · p. 3

  • Results

    ReGNN surpasses strong baselines on most widely used benchmarks and largely alleviates over-smoothing across single- and multiple-label text classification.

    Source: Conclusion · p. 7

  • Takeaways & Limitations

    The paper’s analysis indicates that using LSTM between GNN layers can alleviate over-smoothing faced by many graph-based models.

    Source: Conclusion · p. 7

Abstract

from arXiv · show

The complicated syntax structure of natural language is hard to be explicitly modeled by sequence-based models. Graph is a natural structure to describe the complicated relation between tokens. The recent advance in Graph Neural Networks (GNN) provides a powerful tool to model graph structure data, but simple graph models such as Graph Convolutional Networks (GCN) suffer from over-smoothing problem, that is, when stacking multiple layers, all nodes will converge to the same value. In this paper, we propose a novel Recursive Graphical Neural Networks model (ReGNN) to represent text organized in the form of graph. In our proposed model, LSTM is used to dynamically decide which part of the aggregated neighbor information should be transmitted to upper layers thus alleviating the over-smoothing problem. Furthermore, to encourage the exchange between the local and global information, a global graph-level node is designed. We conduct experiments on both single and multiple label text classification tasks. Experiment results show that our ReGNN model surpasses the strong baselines significantly in most of the datasets and greatly alleviates the over-smoothing problem.

Introduction

Sequence models can struggle with long-range word dependencies, while deeper graph networks face over-smoothing. ReGNN uses LSTM to filter graph-neighbor information and adds a graph-level node; experiments report strong results across text-classification tasks.

Introduction

  • Sequence-based models can have weaker long-range dependency modeling, while stacking GNN layers can make node representations converge to similar values.The passage also notes that shallow GNN structures limit expressive power.Source: Introduction · p. 1
  • ReGNN encodes text graphs built from word co-occurrence information, using LSTM to filter aggregated neighbor information when updating node states.The model dynamically decides which part of the aggregated information to use.Source: Introduction · p. 1

A graph-level node

  • A graph-level node is added to represent the whole graph and interact with ordinary nodes.Source: A graph-level node · p. 1
  • ReGNN surpasses strong baselines on most widely used benchmark datasets and largely overcomes over-smoothing in single- and multiple-label text classification.The model is evaluated on both task types.Source: A graph-level node · p. 1

Related Work

The related work contrasts sequence models’ local or sequential focus with graph models’ ability to represent longer dependencies. It also motivates combining neighbor attention, gating, and global-node exchange.

Related Work

  • Linear LSTMs may not explicitly model long word dependencies, while global-node exchange gives word nodes global information and lets the global node select important nodes.Source: Related Work · p. 2
  • CNN filters capture local word- and phrase-level features, while the proposed graph connections also model longer dependencies and global information.Source: Related Work · p. 2
  • The proposed attention aggregates neighbor information, and an LSTM gate filters what is updated, accommodating different numbers of neighbors.The passage contrasts neighbor-focused aggregation with self-attention over all words, which can introduce noise.Source: Related Work · p. 2
  • The related work connects graph-based text modeling and global nodes with ReGNN’s use of LSTM updating to avoid GCN over-smoothing.Source: Related Work · p. 2

Approach

ReGNN turns text into a word graph, updates node states layer by layer using attention-aggregated neighbors, and includes a graph-level node. The approach also specifies task-specific classifiers and contrasts its computation with sequence-based models.

Approach

  • The approach constructs a text graph from word co-occurrence, aggregates neighbors with attention, updates node states with LSTM, and adds a graph-level node.These are the main stages of the described graph-representation process.Source: Approach · p. 2, Figure 1 · p. 2

Text Graph Construction

  • Each word becomes a graph node; positive PMI scores select up to n−2 neighbor candidates, alongside the two adjacent words.Co-occurrence counts are collected across the corpus, and PMI is calculated from word pairs appearing in the same sentence.Source: Text Graph Construction · p. 2

Recursive Graphical Neural Networks

Updation

  • Node states are updated recurrently by layer rather than sequentially across the text, using the prior node state, original embedding, graph-level state, and aggregated neighbors.The original embedding serves a role similar to a residual connection; the graph-level state supports interaction between each node and the whole graph.Source: Updation · p. 3

Aggregation

  • Attention aggregates information from a flexible number of graph neighbors, while optional positional embeddings let the model incorporate word positions.The LSTM-based update then determines which aggregated information is useful to pass to higher layers as layer depth increases.Source: Aggregation · p. 3

Comparison with Linear LSTM

  • ReGNN processes all graph nodes concurrently by layer, with theoretical O(1) time given enough memory, versus O(N) for linear LSTM; its graph-level node interacts with ordinary nodes at each layer.The exact time depends on the layer-count hyperparameter, treated as constant; the paper contrasts this interaction with pooling methods that it says suffer from bias.Source: Comparison with Linear LSTM · pp. 3–4

Comparison with Transformer and GAT

  • ReGNN applies attention to graph-structured data, unlike the linear structure of Transformer; the paper says fully connecting a graph for Transformer may introduce noise.The paper also says GAT resembles Transformer’s attention but lacks explicit over-smoothing prevention and a graph-level node.Source: Comparison with Transformer and GAT · p. 4

Text Classifier

  • For single-label classification, a linear layer predicts from the graph-level vector; for multiple labels, an attention-based RNN decoder generates variable-length label sequences.The authors choose the multiple-label decoder for its flexibility without threshold-hyperparameter tuning.Source: Text Classifier · p. 4

Experiment

ReGNN is evaluated on single- and multi-label text classification, with comparisons spanning text and graph models. It outperforms baselines across most reported results, while analyses examine neighbor counts, ablations, and over-smoothing.

Datasets

  • Evaluations use R8, R52, Ohsumed, SST, and MR for single-label classification, plus Reuters21578 and Ohsumed for multi-label classification.The single-label Ohsumed set includes only documents with one category; the multi-label Ohsumed set uses all abstracts and labels.Source: Datasets · p. 4

Baselines

  • Comparisons include GraphSAGE, S-LSTM, Text-GCN, HAN, and Transformer alongside other text-classification baselines.Source: Baselines · p. 5

Settings

  • The default depth is six layers for ReGNN, S-LSTM, and Transformer, versus two for other graph-based models; hidden size is 300.Single-label tasks use pretrained 300-dimensional GloVe embeddings; multi-label tasks use randomly initialized word and label embeddings.Source: Settings · p. 5

Experiment Results

Analysis

  • Accuracy rises as the maximum neighbor count increases through five, then steadies as positive-PMI candidate words are limited.Source: Figure 2 · p. 6, Analysis · p. 6

Abalation Study

  • Removing LSTM, attention, or the graph-level node lowers accuracy; removing LSTM causes the largest decline, especially for multi-label classification.The paper attributes the greater multi-label sensitivity to the need for different text parts to remain distinguishable for different labels.Source: Abalation Study · p. 7, Table 5 · p. 7

Effectiveness on Over-smoothing Problem

Conclusion

ReGNN represents word-co-occurrence text graphs with dense vectors, using LSTM to select attention-aggregated neighbor information for node updates. Experiments on single- and multi-label classification support its effectiveness and show that LSTM between GNN layers can alleviate over-smoothing.

Conclusion

  • ReGNN represents text graphs built from word co-occurrence and uses LSTM to select which attention-aggregated neighbor information updates node hidden states.The text graph is represented with dense vectors.Source: Conclusion · p. 7
  • Experiments on single- and multi-label text classification support the effectiveness of ReGNN.The conclusion also reports that LSTM between GNN layers can effectively alleviate over-smoothing.Source: Conclusion · p. 7
  • The model is not limited to classification; applying it to tasks such as generation is proposed as future work.Source: Conclusion · p. 7

Related papers

Loading 1909.08166v1…