PaperlayerView exact revision on arXiv

Source-linked AI summary

A C-LSTM Neural Network for Text Classification

Chunting Zhou, Chonglin Sun, Zhiyuan Liu, Francis C. M. Lau

arXiv:1511.08630v2cs.CL

TL;DR

Sentence modeling requires representations that capture both local phrase information and sequential semantics, while CNNs and RNNs emphasize different strengths. The paper proposes C-LSTM, which feeds CNN-derived higher-level phrase sequences into LSTM, and reports strong results across sentiment and question classification, including superiority over individual CNN and LSTM models. The authors identify more structured convolutional representations as a future direction.

  • Problem

    Sentence modeling needs meaningful representations, but CNNs and RNNs emphasize different aspects of language: local features versus sequential correlations.

  • Method

    C-LSTM uses CNN to extract higher-level phrase representations and feeds their ordered sequence into LSTM to learn long-term dependencies.

  • Results

    C-LSTM achieves excellent results on sentiment and question classification and outperforms individual CNN and LSTM models.

  • Takeaways & Limitations

    Combining CNN and LSTM yields a unified sentence model that captures local phrase features together with global and temporal sentence semantics.

  • Takeaways & Limitations

    Future work could replace standard convolution with tensor-based operations or tree-structured convolutions to learn more structured higher-level representations.

Abstract

from arXiv · show

Neural network models have been demonstrated to be capable of achieving remarkable performance in sentence and document modeling. Convolutional neural network (CNN) and recurrent neural network (RNN) are two mainstream architectures for such modeling tasks, which adopt totally different ways of understanding natural languages. In this work, we combine the strengths of both architectures and propose a novel and unified model called C-LSTM for sentence representation and text classification. C-LSTM utilizes CNN to extract a sequence of higher-level phrase representations, and are fed into a long short-term memory recurrent neural network (LSTM) to obtain the sentence representation. C-LSTM is able to capture both local features of phrases as well as global and temporal sentence semantics. We evaluate the proposed architecture on sentiment classification and question classification tasks. The experimental results show that the C-LSTM outperforms both CNN and LSTM and can achieve excellent performance on these tasks.

1 Introduction

Sentence modeling must represent meaningful sentence features despite limitations of bag-of-words, composition-based methods, and architectures that specialize in either local or sequential information. C-LSTM combines CNN feature extraction with LSTM sequence modeling and is evaluated on sentiment and question classification.

  • Bag-of-words sentence modeling often suffers from the curse of dimensionality, while composition-based methods may lose word-order information.
  • CNNs extract n-gram features and learn local correlations, whereas RNNs handle variable-length sequences and long-term dependencies.
  • C-LSTM feeds higher-level representations learned by a one-layer CNN into an LSTM while preserving their sequence order.
  • The model learns higher-level n-gram representations from pretrained word vectors and uses LSTM to model their temporal structure.
  • C-LSTM is evaluated on sentiment classification and 6-way question classification, where it outperforms individual CNN and RNN models.

2 Related Work

Prior sentence-representation work uses CNNs or RNNs over word sequences or syntactic trees. C-LSTM instead feeds consecutive CNN window features directly into an LSTM, and experiments report superiority over single CNN, LSTM, and related sequence-based models.

  • Neural sentence-representation models commonly operate on word sequences or transformed syntactic parse trees, with CNN and RNN as prominent architectures.
  • CNNs model consecutive context windows by extracting local and higher-level correlations, while pooling selects important features.
  • RNNs process variable-length sequences and discover long-term dependencies, with variants designed to store and access memories more effectively.
  • C-LSTM feeds consecutive CNN window features directly into LSTM so higher-order sequential features retain sequence structure for long-range modeling.
  • Experiments on sentiment and 6-way question classification report superiority over single CNN, LSTM, and other related sequence-based models.

3 C-LSTM Model

C-LSTM combines CNN and LSTM: convolution produces a sequence of higher-level window representations, and LSTM models long-term dependencies over that sequence. Pooling is omitted to preserve the sequential organization required by LSTM.

  • The architecture has CNN and LSTM components: CNN extracts higher-level sequences of word features, and LSTM captures dependencies over window-feature sequences.
  • N-gram Feature Extraction through Convolution: For a sentence of length L, a filter of length k slides over consecutive word vectors to produce a feature at each valid position.
  • N-gram Feature Extraction through Convolution: Multiple filters generate feature maps whose corresponding outputs are rearranged into one higher-order representation for each window.
  • Long Short-Term Memory Networks: These successive window representations are fed into LSTM, whose gates update memory and hidden states to learn long-term dependencies.
  • N-gram Feature Extraction through Convolution: Pooling is omitted after convolution because selecting discontinuous features would break the sequence organization required by LSTM.

4 Learning C-LSTM for Text Classification

The model trains a softmax classifier by minimizing cross-entropy, with preprocessing, initialization, padding, and regularization choices specified for text inputs.

  • The final LSTM hidden state represents the document, and a softmax layer produces the classification output.
  • Training minimizes cross-entropy error using stochastic gradient descent with RMSprop optimization.
  • Sentences shorter than maxlen are padded, while longer test sentences are truncated to the training-set maximum length.
  • Word vectors are initialized from 300-dimensional Google News word2vec vectors and fine-tuned during training.
  • Dropout is applied to word vectors or LSTM outputs, while L2 regularization is applied to softmax weights.

5 Experiments

The experiments evaluate C-LSTM on sentiment and question-type classification using established benchmarks, with task-specific settings and baseline comparisons.

  • The evaluation covers sentiment classification and question type classification.
  • Datasets: The Stanford Sentiment Treebank contains 11,855 movie reviews split into training, development, and test sets.
  • Datasets: SST supports 5-label fine-grained and binary sentiment classification, with training on phrases and sentences but testing on sentences.
  • Datasets: TREC classifies questions into six categories using 5,452 labeled training questions and 500 testing questions.
  • Experimental Settings: The final configuration uses one convolutional layer and one LSTM layer for both tasks.
  • Results: Table 1 compares C-LSTM with recursive, convolutional, LSTM-based, and other baselines on fine-grained and binary SST classification.
  • Experimental Settings: SST uses 150 filters and LSTM memory dimension 150, while TREC uses 300 for both settings; both use dropout probability 0.5.

6 Results and Model Analysis

Experiments evaluate C-LSTM against diverse baselines on sentiment and TREC question classification, showing strong results and analyzing convolutional filter configurations. The analysis finds consistent benefits from combining CNN and LSTM and from a single length-3 convolutional layer.

  • Sentiment Classification: C-LSTM achieves the fourth best published result on 5-class sentiment classification and comparable state-of-the-art performance on binary classification.
  • Model Comparison: The CNN–LSTM combination outperforms individual multilayer CNN and RNN models, supporting LSTM modeling of higher-level representation sequences.
  • Question Type Classification: On TREC question classification, C-LSTM consistently outperforms published neural baselines and approaches an SVM using highly engineered features.The authors emphasize that C-LSTM requires no human-designed features.
  • Filter Configuration Analysis: A single convolutional layer with filter length 3 consistently outperforms alternative filter-size configurations.The same pattern appears in both TREC question classification and sentiment classification.
  • Filter Configuration Analysis: Among parallel multilayer configurations, those including length-3 filters perform better than configurations without trigram filters.The authors conjecture that LSTM learns better sentence representations from sequences of trigram features.

7 Conclusion and Future Work

The paper concludes that C-LSTM combines CNN phrase extraction with LSTM dependency modeling for sentence representation. It reports satisfactory results across sentiment and question classification while identifying more structured convolutional representations as future work.

  • C-LSTM combines a convolutional layer for phrase-level features with LSTM processing of their sequences to learn long-term dependencies.
  • The learned semantic sentence representations achieve very satisfactory results on sentiment classification and question type classification.
  • Future work could replace standard convolution with tensor-based operations or tree-structured convolutions to provide more structured higher-level representations.The authors believe LSTM would benefit from these representations.
Loading 1511.08630v2…