Source-linked AI summary
Rationale-Augmented Convolutional Neural Networks for Text Classification
Ye Zhang, Iain Marshall, Byron C. Wallace
TL;DR
Text classification usually uses document labels, but this paper studies settings where annotators also mark supporting sentences or snippets as rationales. It introduces RA-CNN, which weights sentence representations using learned rationale probabilities, and reports consistent gains over strong baselines across five datasets while providing explanations.
Problem
Text classification models typically assume document labels alone, despite settings where annotators also provide sentence-level rationales supporting document categorizations.
Method
RA-CNN estimates each sentence's probability of being a rationale and scales its contribution to a document representation accordingly.
Results
RA-CNN consistently outperforms strong baseline models across five datasets, including vanilla, hierarchical, and attention-equipped CNN variants.
Takeaways & Limitations
RA-CNN uses rationale supervision to produce document classifications while naturally providing explanations for its test-time predictions.
Takeaways & Limitations
The paper identifies combining attention and direct sentence supervision through an auxiliary loss as future work.
Abstract
from arXiv · showhide
We present a new Convolutional Neural Network (CNN) model for text classification that jointly exploits labels on documents and their component sentences. Specifically, we consider scenarios in which annotators explicitly mark sentences (or snippets) that support their overall document categorization, i.e., they provide rationales. Our model exploits such supervision via a hierarchical approach in which each document is represented by a linear combination of the vector representations of its component sentences. We propose a sentence-level convolutional model that estimates the probability that a given sentence is a rationale, and we then scale the contribution of each sentence to the aggregate document representation in proportion to these estimates. Experiments on five classification datasets that have document labels and associated rationales demonstrate that our approach consistently outperforms strong baselines. Moreover, our model naturally provides explanations for its predictions.
1 Introduction
The paper addresses text classification with both document labels and sentence-level rationales, proposing a neural model that exploits this dual supervision. It reports broad empirical gains and interpretable predictions.
- Rationales are sentences or snippets that support document categorizations and can be collected alongside document labels with little additional annotator effort.
- The paper proposes a CNN that exploits both document labels and associated rationales, extending rationale-based supervision beyond prior sparse linear models.
- The model is presented as the first neural text-classification approach to incorporate rationales.
- The proposed model uniformly outperforms relevant rationale-based, CNN, and attention-equipped baselines across five datasets.
- The model naturally provides explanations for its classifications, supporting interpretability.
2 Related Work
Prior work studied rationales and sentence importance mainly through sparse linear models, while related neural work modeled document structure or attention. This paper positions rationale supervision as a distinct form of dual supervision.
- Hierarchical neural models represented sentences with recurrent networks or used two levels of attention for document classification.
- The paper assumes that some sentences in long documents directly support their categorization and calls these sentences rationales.
- Rationale-based classification previously modified SVM objectives so documents containing annotated rationales were classified more confidently than rationale-stripped pseudo-instances.
- Dual-supervision research has often used learning constraints based on prior feature-label affinities or rankings, rather than provided rationales.
3 Preliminaries: CNNs for text classification
The preliminary CNN converts word embeddings into convolutional feature maps, pools them into a fixed-size vector, and classifies the resulting representation. For documents, the basic variant treats the entire document as one long sentence.
- Each word is represented by a pretrained d-dimensional embedding, and the embeddings are stacked into an instance matrix A ∈ R^n×d.
- Filters share width d but vary in height, so each filter captures n-gram features corresponding to its height and produces a feature map.
- ReLU transforms each feature map, after which 1-max pooling extracts its maximum value.
- The pooled features are concatenated into vector o ∈ R^|F|, and a softmax performs classification on top of it.
- The basic CNN adapts sentence classification to documents by treating each document as one long sentence.
4 Rationale-Augmented CNN for Document Classification
RA-CNN uses sentence-level rationale supervision to weight sentence representations when constructing document vectors for classification. It trains sentence and document components in sequence and naturally identifies the sentences supporting each prediction.
- RA-CNN: RA-CNN estimates each sentence’s probability of being a positive or negative rationale and uses those estimates to inform document classification.The model jointly exploits sentence- and document-level supervision through a hierarchical architecture.
- Document structure: Doc-CNN applies a CNN independently to each sentence, sums sentence vectors into a document vector, and classifies the result with a softmax layer.Dropout is applied to both sentence vectors and the final document vector.
- RA-CNN: RA-CNN replaces Doc-CNN’s unweighted sum with a weighted sum in which likely rationale sentences have greater influence and neutral sentences have reduced influence.The document-level classifier receives the resulting document vector.
- Document structure: Summing sentence vectors performed better than averaging them in informal testing.
- Sentence supervision: The sentence model uses three classes: positive rationale, negative rationale, and neutral non-rationale; separating rationale polarities performed better than using only rationale and non-rationale classes.The document classification tasks considered are binary, while the model can be extended to multiclass settings.
- Training: Training proceeds sequentially: rationale identification is trained first, then document classification initializes shared embeddings and convolution parameters from that model and tunes them with document labels.The authors report that alternating multitask training did not work well in this setting.
- Interpretability: RA-CNN provides explanations by outputting the sentences with the highest rationale probabilities for the predicted class.These sentences are the ones the model relied on most for its disposition.
- Attention comparison: The model is related to supervised attention, while AT-CNN learns sentence importance using only document labels rather than rationale annotations.This comparison isolates the contribution of direct rationale-level supervision.
5 Datasets
The evaluation uses five text classification datasets with document-level labels and associated rationales: four biomedical datasets and one movie-review dataset. The movie-review data adapts sub-sentential rationale annotations into sentence-level labels and uses cross-validation because the held-out test set is small.
- Dataset overview: The study evaluates five datasets, four biomedical and one movie-review collection, all containing document categorizations with associated rationales.
- Biomedical datasets: The biomedical datasets concern Risk of Bias assessment for evidence from randomized controlled trials.The task includes assessing whether trial procedures, such as patient blinding, were properly conducted.
- Biomedical datasets: The Risk of Bias task is organized around domains defined by the Cochrane Risk of Bias Tool, including random sequence generation and allocation concealment.
- Movie-review dataset: The movie-review dataset contains 1000 positive and 1000 negative reviews, with rationale annotations added for 1800 documents and 200 documents reserved for testing.
- Dataset overview: The dataset summary reports N, average sentences per document, average tokens per sentence, and average rationales per document.
- Movie-review dataset: Because the movie-review test set contains only 200 documents, the authors also run 9-fold cross-validation on the 1800 annotated documents.Each fold contains 200 documents.
6 Experimental Setup
The experiments compare rationale-augmented CNNs with linear SVM, rationale-augmented SVM, multitask SVM, and CNN baselines across risk-of-bias and movie-review datasets. Models use sentence splitting, pretrained embeddings, staged sentence- and document-level training, and cross-validation.
- Baselines: The baselines include unigram and bigram SVMs, RA-SVM, RoB-specific MT-SVM, standard CNN, Doc-CNN, and attention-based AT-CNN.The comparison targets the value of directly incorporating rationale-level supervision into the CNN architecture.
- Preprocessing: Documents were split into sentences with the NLTK sentence splitter for Doc-CNN and RA-CNN.
- CNN configuration: CNN models used pretrained Word2Vec embeddings: 300-dimensional Google News vectors for movie reviews and 200-dimensional biomedical vectors for RoB datasets.
- Training: Training first learned sentence-level supervision with balanced sentence classes, then initialized document-level fitting from the sentence-level embedding and convolution parameters.Sentence classes were downsampled before each epoch because nearly all sentences were neutral.
- Optimization: RA-CNN document training tuned sentence-vector dropout, while all models used ADADELTA, mini-batches of 50, and early stopping.The RA-CNN dropout search ranged from 0 to .9; document-level dropout was fixed at 0.5 after tuning.
7 Results and Discussion
RA-CNN consistently outperforms the baseline models across five datasets and produces rationales that explain its classifications. The results also indicate that rationale supervision is especially valuable for long documents, where standard CNN variants may not surpass strong SVM baselines.
- Quantitative results: RA-CNN consistently outperforms all baseline models across all five datasets.The RoB results are reported in Table 2 and movie-review results in Table 3.
- Quantitative results: SVM-based models remain strong baselines for longer documents, where CNN and Doc-CNN do not necessarily improve on their results.This contrasts with earlier sentence-classification comparisons in which CNN outperformed SVM uniformly.
- Quantitative results: RA-CNN outperforms AT-CNN and MT-SVM on the RoB datasets while using direct rationale supervision without cross-domain multitask learning.AT-CNN often improves over vanilla CNN variants, especially on the longer RoB documents.
- Qualitative results: The model ranks the highest-scoring positive and negative rationales, which by construction are the sentences that most influenced the document classification.
- Qualitative results: Extracted rationales directly support classifications in examples from both risk-of-bias studies and movie reviews, providing insight into otherwise opaque neural predictions.Examples include evidence about double blinding, retrospective study design, and evaluative movie-review language.
8 Conclusions
The paper introduces RA-CNN, which directly uses rationales in CNN text classification and outperforms strong baselines across five datasets. It also automatically provides explanations for test-time classifications, while future work will explore additional lower-level supervision mechanisms.
- Contributions: RA-CNN extends CNN text classification to directly exploit available rationales and outperforms strong CNN, hierarchical CNN, attention-based CNN, and other baselines across five datasets.
- Contributions: RA-CNN automatically provides explanations for test-time classifications, providing interpretability.
- Future work: Future work will explore additional mechanisms for exploiting supervision at lower levels in neural architectures, including a possible hybrid of AT-CNN and RA-CNN.The proposed hybrid would add an auxiliary loss when attention outputs disagree with direct sentence supervision.