Source-linked AI summary
Dependency Sensitive Convolutional Neural Networks for Modeling Sentences and Documents
Rui Zhang, Honglak Lee, Dragomir Radev
TL;DR
Sentence and document models must represent meaning across variable-length text, while recursive models depend on parsing and conventional CNNs capture local windows. DSCNN combines LSTMs with convolution to model dependencies within sentences and across document sentences, and reports state-of-the-art results on several tasks.
Problem
Sentence and document modeling seeks accurate representations of meaning, but recursive models rely on parsing and conventional CNNs capture local features while losing long-distance dependencies.
Source: §1 · p. 1, §4 · p. 4, §5.4.1 · p. 8Method
DSCNN processes pretrained word embeddings with LSTMs and then convolution, using a second LSTM layer to encode dependencies across document sentences.
Source: §1 · p. 2, §4.1 · p. 4Results
DSCNN outperforms traditional CNNs and achieves state-of-the-art results on several sentiment, question-type, and subjectivity datasets, although Paragraph Vector and SA-LSTM achieve better results than DSCNN in document modeling.
Source: §1 · p. 2, §6 · p. 9, §5.4.2 · p. 9Takeaways & Limitations
DSCNN is a classification architecture for sentence and document inputs that requires root-level labels rather than phrase labels.
Source: §1 · p. 2, §4 · p. 4Reported sentence-modeling errors include missed numeric meaning, an out-of-vocabulary abbreviation, and a likely ambiguous or mistaken annotation.
Source: §5.4.1 · p. 9
Abstract
from arXiv · showhide
The goal of sentence and document modeling is to accurately represent the meaning of sentences and documents for various Natural Language Processing tasks. In this work, we present Dependency Sensitive Convolutional Neural Networks (DSCNN) as a general-purpose classification system for both sentences and documents. DSCNN hierarchically builds textual representations by processing pretrained word embeddings via Long Short-Term Memory networks and subsequently extracting features with convolution operators. Compared with existing recursive neural models with tree structures, DSCNN does not rely on parsers and expensive phrase labeling, and thus is not restricted to sentence-level tasks. Moreover, unlike other CNN-based models that analyze sentences locally by sliding windows, our system captures both the dependency information within each sentence and relationships across sentences in the same document. Experiment results demonstrate that our approach is achieving state-of-the-art performance on several tasks, including sentiment analysis, question type classification, and subjectivity classification.
1 Introduction
Sentence and document modeling must capture word order and dependencies, but recursive and conventional CNN approaches have limitations in parsing requirements and long-range information. DSCNN combines LSTMs with convolution for sentence and document classification, and reports top accuracies on several benchmarks.
1 Introduction
- Recursive models rely on parsers and phrase-level supervision, and their tree structures are difficult to extend from sentences to documents.The introduction also notes vanishing gradients as a concern for recursive models.Source: §1 · pp. 1–2
- Conventional CNNs process local n-grams, losing dependencies that span their sliding windows.The paper motivates a model that captures long-term dependencies without parsing for both sentence and document inputs.Source: §1 · pp. 1–2
- DSCNN processes word embeddings with LSTMs before convolution, adding a second LSTM layer for dependencies across document sentences.Sentence inputs use an LSTM for long-distance dependencies and variable-size convolution filters with max pooling for classification.Source: §1 · p. 2
- DSCNN achieves the highest accuracies among several competitive baselines on MR sentiment analysis, TREC question classification, and SUBJ subjectivity classification.The paper also reports state-of-the-art performance across several tasks.Source: §1 · p. 2
2 Related Work
Prior text-composition approaches range from order-insensitive models to sequence models and CNNs. They use word embeddings to construct phrase and sentence representations in different ways.
2 Related Work
- Word embeddings represent words as dense, low-dimensional vectors encoding semantic and syntactic features.The paper groups composition methods into unordered models, sequence models, and CNN models.Source: §2 · p. 2
- Bag-of-words averages embeddings without regard to token order, while neural bag-of-words adds a hidden layer before classification.These are examples of unordered phrase and sentence representations.Source: §2 · p. 2
- RNNs and LSTMs build order-sensitive representations, while tree-structured variants use parsing information to combine constituent or child representations.Examples include Recursive Neural Networks and Tree-LSTM.Source: §2 · pp. 2–3
- CNNs extract features from variable-length phrases using convolution; examples add dynamic k-max pooling or multichannel embeddings and unsupervised pretraining.The cited examples are DCNN and a model by Yin and Schütze.Source: §2 · p. 3
3 Preliminaries
The preliminaries introduce LSTMs for sequence modeling and convolution with max-over-time pooling for feature extraction. Together, these are the building blocks used to model sequences and extract features.
3 Preliminaries
- The preliminaries establish LSTM networks and convolution with max-over-time pooling as the system’s two building blocks.LSTM models sequence data; convolution and pooling extract features from sequence inputs.Source: §3 · p. 3
3.1 Long Short-Term Memory
- LSTMs use memory cells and gates to address the difficulty traditional RNNs have capturing long-distance dependencies due to exploding or vanishing gradients.Input, forget, and output gates regulate input updates, previous-cell forgetting, and memory exposure in the hidden state.Source: §3.1 · p. 3
3.2 Convolution and Max-over-time Pooling
- Wide convolution pads both ends of the hidden-state sequence, applies filters across consecutive windows, and max-over-time pooling selects each filter’s maximum activation.Multiple filter window sizes capture features from input windows of different lengths.Source: §3.2 · pp. 3–4
4 Model Architectures
DSCNN targets long-distance dependencies while avoiding parsing and phrase-level supervision, then organizes sentence and document representations hierarchically. Sentence inputs pass through LSTMs and convolution; document inputs add subsentence averaging and a higher-level LSTM.
4 Model Architectures
- DSCNN addresses local-window loss of long dependencies and tree models’ parser, supervision, and sentence-only constraints through parser-free hierarchical representations.The architecture can be restructured for document inputs.Source: §4 · p. 4
4.1 Sentence Modeling
- For sentences, LSTMs process multiple word-embedding versions into hidden-state sequences, followed by convolution, max-over-time pooling, and softmax classification.The convolution layer stacks the embedding-specific sequences as multiple feature-map channels.Source: §4.1 · pp. 4–5
4.2 Document Modeling
- For documents, punctuation-delimited subsentences are processed by independent LSTMs; averaged hidden states feed a higher-level LSTM, then convolution, pooling, and classification.The higher-level LSTM captures the joint meaning created by the sentences.Source: §4.2 · p. 5
5 Experiments
The experiments span sentence and document datasets, compare LSTM initialization choices, and report benchmark outcomes and dependency-length analyses.
5.1 Datasets
- The datasets cover MR and SST sentiment, six-class TREC question classification, SUBJ subjectivity, and multi-sentence IMDB sentiment.MR and SUBJ use 10-fold cross-validation; IMDB includes 25,000 training and 25,000 testing reviews.Source: §5.1 · pp. 5, 7
5.2 Training Details and Implementation
- Implementation uses two 300-dimensional embedding channels, 100 ReLU filters at widths 3–5, 0.5 dropout, Adadelta, validation early stopping, and about 4,000 tokens per second.On TREC, an epoch takes about one minute and training converges within 50 epochs.Source: §5.2 · p. 7
5.3 Pretraining of LSTM
- Sequence-autoencoder pretraining initializes task-specific DSCNN-Pretrain LSTMs; the alternative DSCNN variant uses random orthogonal LSTM weights.The pretrained encoders are trained on each task's train/validation/test splits.Source: §5.3 · p. 7
5.4.1 Sentence Modeling
- DSCNN and Dep-CNN outperform CNN-MC as dependency lengths grow, while Dep-CNN requires a parser and predefined patterns.The models are similar below a summed dependency-arc length of 15.Source: Figure 3 · p. 7, §5.4.1 · p. 8
- DSCNN beats all baselines on MR and TREC, ties MVCNN on SUBJ, trails MVCNN slightly on SST-2, and ranks behind only Tree-LSTM on SST-5.Table 1 reports accuracy comparisons; Tree-LSTM relies on parsers.Source: Table 1 · p. 6, §5.4.1 · pp. 7–8, §5.4 · p. 7
- CNN-MC errors include missed long dependencies, while DSCNN errors include an uncaptured numeric sense, an out-of-vocabulary term, and possibly ambiguous annotation.The examples include “point” and “TMJ.”Source: §5.4.1 · pp. 8–9
5.4.2 Document Modeling
- On IMDB, DSCNN outperforms bag-of-words, DAN, and WRRBM+BoW, while Paragraph Vector and SA-LSTM achieve better results.CNN-based results were not reported because of computation time; Paragraph Vector and SA-LSTM use 50,000 unlabeled examples.Source: Table 1 · p. 6, §5.4.2 · p. 9
6 Conclusion
DSCNN models text at sentence and document levels by processing word vectors through LSTMs and extracting classification features with convolutional operators. It consistently outperforms traditional CNNs and achieves state-of-the-art results on several tasks.
6 Conclusion
- DSCNN consistently outperforms traditional CNNs and achieves state-of-the-art results on several sentiment analysis, question type classification, and subjectivity classification datasets.Source: §6 · p. 9
- DSCNN models text at both sentence and document levels, capturing long-term inter-sentence and intra-sentence dependencies with LSTM layers.Source: §6 · p. 9
- Convolutional operators extract features for classification from the LSTM-processed word vectors.Source: §6 · p. 9