Source-linked AI summary
Neural Discourse Structure for Text Categorization
Yangfeng Ji, Noah Smith
TL;DR
Text categorization can benefit from discourse structure, but the value of that structure depends on the genre and the quality of the discourse parse. The paper combines an automatically derived RST-style dependency tree with a recursive neural network and unnormalized attention, finding improvements on four of five tasks while identifying parsing-quality and genre limitations.
Problem
The paper asks whether discourse structure can broadly improve text categorization beyond existing approaches that model salient passages.
Method
A recursive neural network uses an automatically derived discourse dependency tree and unnormalized attention to represent documents and focus on salient content.
Results
Variants of the model outperform prior work on four of five tasks, while discourse quality affects accuracy and the method does not benefit legislative-bill categorization.
Takeaways & Limitations
Automatically derived discourse structure can help text categorization, and better discourse parsing may produce greater gains, especially for new genres.
Takeaways & Limitations
The method does not benefit legislative-bill categorization, whose discourse conventions diverge from those of news.
Abstract
from arXiv · showhide
We show that discourse structure, as defined by Rhetorical Structure Theory and provided by an existing discourse parser, benefits text categorization. Our approach uses a recursive neural network and a newly proposed attention mechanism to compute a representation of the text that focuses on salient content, from the perspective of both RST and the task. Experiments consider variants of the approach and illustrate its strengths and weaknesses.
1 Introduction
The paper investigates whether discourse structure can improve text categorization by guiding attention toward salient content. It uses an automatically parsed RST structure with a recursive neural network and introduces an unnormalized attention mechanism.
- Discourse-based methods are motivated by prior findings that weighting salient passages can improve categorization, including sentiment classification.
- The study evaluates discourse structure for text categorization across five tasks using an automatically derived DPLP discourse parse.
- The model learns sentence importance from positions and relations in the discourse tree through a recursive neural network.
- The proposed attention mechanism is unnormalized, so each sentence’s weight need not depend on the importance of sibling sentences.
- Variants of the model outperform prior work on four of five tasks, but underperform on legislative-bill prediction because that genre differs from the parser’s training data.
2 Background: Rhetorical Structure Theory
RST represents a document as a tree of discourse units linked by relations, with nuclei generally more central than satellites. The paper uses an automatically parsed RST structure transformed into dependencies as an inductive bias for categorization.
- RST represents documents as trees whose leaves are elementary discourse units and whose internal nodes encode relations such as CONTRAST and ELABORATION.
- Within most RST relations, the nucleus is more essential to the writer’s purpose than the satellite.
- A restaurant-review example illustrates hierarchical spans and nucleus–satellite structure, including a CONTRAST relation spanning the whole review.
- RST supports applications including summarization, argumentation mining, and sentiment analysis, but automatic parsing is used because manual annotation is expensive.
- The paper transforms the RST tree into a dependency structure in which the root can summarize the review, providing an inductive bias for identifying salient content.
3 Model
The model recursively composes EDU representations over a discourse dependency tree, using relation-sensitive and task-learned attention to emphasize salient content. Simpler variants isolate the effects of discourse structure, relation labels, and composition.
- 3.2 Full Recursive Model: The FULL model recursively composes EDU representations over a discourse dependency tree and classifies the document from the root representation.Each EDU is represented separately, and composition combines parent and child representations into larger subtrees.
- 3.2 Full Recursive Model: The composition function uses relation-specific matrices and unnormalized attention to weight children according to their content and discourse relations.Unlike normalized attention, each child’s weight does not depend on the parent’s other children, matching RST’s treatment of parallel relations.
- 3.2 Full Recursive Model: The model learns to weight EDUs for categorization, replacing manually designed discourse-based weighting schemes with end-to-end learned parameters.This weighting is implemented through the attention mechanism and relation-specific composition matrices.
- 3.3 Unlabeled Model: The UNLABELED variant retains the discourse tree and learned attention but removes discourse relation labels, reducing the number of parameters.It tests whether tree structure alone contributes beyond relation-specific information.
- 3.4 Simpler Variants: ROOT represents a document with its root EDU, whereas ADDITIVE averages all EDU representations without using discourse structure.ADDITIVE controls for other design choices by providing a discourse-independent baseline.
4 Implementation Details
The implementation preprocesses all datasets consistently, obtains discourse structures with the DPLP RST parser, and varies embeddings according to dataset size. Training uses end-to-end optimization with gradient clipping, dropout, and grid-searched hyperparameters.
- Preprocessing: All datasets are lowercased, punctuation-only tokens are removed, numbers are replaced by a special token, and rare words are mapped to UNK.The vocabulary is reduced until approximately 5% of tokens are mapped to UNK.
- Discourse parsing: Discourse structures are produced by DPLP, an open-source RST parser that segments documents into elementary discourse units.DPLP uses greedy decoding and averages 2,000 parses per minute on a single CPU.
- Word embeddings: Pretrained GloVe embeddings are used for datasets with 10,000 or fewer training examples, while larger datasets use randomly initialized trainable embeddings.The embedding choice therefore depends on the number of training examples.
- Learning and hyperparameters: Training uses online optimization, gradient-norm clipping at τ = 5.0, dropout of 0.3, and grid search over representation dimensions and learning settings.Gradient clipping addresses exploding gradients, while dropout is used to avoid overfitting.
5 Datasets
The experiments use five datasets spanning sentiment, framing, legislative debate, and bill-survival categorization, with varied corpus sizes and evaluation settings. Each dataset is compared against task-specific prior work or baselines.
- The five datasets cover Yelp sentiment, news framing, congressional floor-debate votes, movie-review sentiment, and legislative bill survival.The datasets vary substantially in size, labels, and categorization goals.
- Sentiment analysis on Yelp reviews: Yelp uses 650,000 training and 50,000 test examples to predict ordinal ratings from 1–5.The experiments compare against hierarchical attention networks.
- Framing dimensions in news articles: The Media Frames Corpus contains around 4,200 immigration news articles labeled with 15 framing dimensions, with experiments predicting each article’s primary frame.Results are averaged across 10-fold evaluation, using Card et al. (2016) as the state-of-the-art comparison.
- Congressional floor debates: The congressional floor-debate task predicts whether a speaker votes “yea” or “nay” from each speech segment.The principal comparison is Yogatama and Smith (2014), which uses structured regularization based on linguistic components.
- Movie reviews: The movie-review corpus contains 1,000 positive and 1,000 negative reviews, evaluated with standard ten-fold cross validation against discourse-based sentiment methods.Compared systems include Bhatia et al. (2015) and Hogenboom et al. (2015).
- Congressional bill corpus: The congressional bill corpus contains 51,762 bills, and the task predicts whether each bill survives based on its content.The experiments use Yogatama and Smith (2014) as the best published baseline and reserve 10% of training examples for development.
6 Experiments
Across five datasets, discourse-aware models generally improve text categorization, but their gains depend on parser quality, genre, dataset size, and model capacity.
- Results: On four of five datasets, UNLABELED outperforms prior methods, while FULL is stronger only on Yelp and can be overparameterized on smaller datasets.On MFC and Movies, discourse-ignorant ADDITIVE outperforms FULL; FULL has nearly 20 times as many parameters as UNLABELED, which has twice as many as ADDITIVE.
- Results: Discourse structure benefits categorization in some genres, but legislative bills are an exception where discourse works against accuracy despite abundant data.Bills are more than ten times larger than three datasets where UNLABELED outperformed prior methods, so the drop is not attributed to insufficient data.
- Results: The ROOT model performs poorly across cases, indicating that discourse information is not helping merely by selecting one EDU for classification.
- Qualitative analysis: Qualitative examples show discourse structures helping FULL emphasize important sentences, while parser errors can misidentify salient content and confuse the model.In the negative example, a manually constructed structure is expected to improve prediction through a more appropriate dependency and relation-specific composition matrices.
- Effect of parsing performance: Lower discourse-parser F1 leads to lower Yelp classification accuracy, and at 25% of the RST training data the method underperforms Yang et al. (2016).The authors suggest that larger annotated datasets or improved parsing models could produce greater categorization gains.
- Attention mechanism: The normalized-attention FULL variant reaches 70.3% Yelp accuracy, 1.5% below FULL, supporting the decision not to normalize attention.The normalized variant assigns child attention weights that sum to one, whereas the proposed mechanism is inspired by RST’s lack of competition among satellites.
7 Related Work
Prior text-categorization work learned sentence or feature importance from task supervision, while related discourse-based sentiment models used hand-crafted weighting. This paper instead combines discourse-tree structure with an attention mechanism that can adjust its linguistic bias.
- Prior representation learning: Earlier approaches modeled text as characters, words, latent topics, or learned representations without explicitly weighting passages by discourse structure.These approaches include bag-of-words, matrix decomposition, latent variables, and neural architectures.
- Discourse-guided categorization: The paper builds a neural text-categorization model on a discourse dependency tree, favoring central EDUs while allowing task supervision to overcome that bias.Unlike prior attention and latent-variable approaches, the model incorporates linguistic structure directly.
- Learned importance weighting: Yessenalina et al. (2010) and Yang et al. (2016) learned sentence importance from task supervision, but neither incorporated a linguistic bias.The former used latent sentence informativeness, whereas the latter learned a neural attention function.
- Linguistic regularization: Yogatama and Smith (2014) regularized sentence importance without considering discourse structure, leaving discourse-based weighting as a distinct contribution here.Their sentence regularizer encouraged ignoring task-irrelevant training-set sentences.
- Discourse for sentiment analysis: Bhatia et al. (2015) used a discourse-dependency model with hand-crafted weighting for sentiment polarity, whereas this work automates the weighting through attention.The comparison identifies attention as the main methodological difference.
8 Conclusion
The paper concludes that automatically derived discourse structure can improve text categorization when parsing is accurate, but its benefit does not extend to legislative bills whose discourse differs from news.
- Conclusion: Automatically derived discourse structure can help text categorization, and the benefit increases with discourse-parsing accuracy.The conclusion motivates improving discourse parsing as a route to better categorization performance.
- Conclusion: The approach provides no benefit for legislative-bill categorization, whose discourse structure diverges from the news genre used by the parser.The genre mismatch defines an important boundary on the reported finding.
- Conclusion: These findings motivate further improvements to discourse parsing, especially for new genres.The recommendation follows from both the accuracy dependence and the legislative-bill limitation.
A Supplementary Material: An example text from the Bill corpus
The supplementary material presents a legislative-bill example from the Bill corpus: a 1994 congressional bill concerning housing assistance for homeless AFDC families.
- Bill-corpus example: The example is H. R. 4449 from the 103d Congress’s second session, introduced in the House of Representatives on May 18, 1994.The passage identifies the bill number, congressional session, chamber, and introduction date.
- Bill-corpus example: The bill proposes amending title IV of the Social Security Act to support permanent housing for homeless AFDC families.Its stated purpose includes constructing, rehabilitating, purchasing, or renting housing.
- Bill-corpus example: The proposal would use funds otherwise allocated for emergency assistance for those families.The passage describes the intended funding substitution.