Source-linked AI summary
Comparative Study of CNN and RNN for Natural Language Processing
Wenpeng Yin, Katharina Kann, Mo Yu, Hinrich Schütze
TL;DR
The paper addresses the absence of consensus about selecting CNNs or RNNs for NLP tasks. It systematically compares CNNs, GRUs, and LSTMs across representative tasks and finds that performance depends on task requirements, while hidden and batch sizes also matter substantially.
Problem
NLP lacks systematic evidence for choosing between CNNs and RNNs across different language-processing tasks.
Method
The study systematically compares CNNs, GRUs, and LSTMs across sentiment and relation classification, entailment, answer selection, question-relation matching, path query answering, and part-of-speech tagging.
Results
CNNs and RNNs provide complementary information for text classification, and the better architecture depends on the need to understand global or long-range sequence semantics.
Takeaways & Limitations
RNNs perform well and robustly across many tasks except those essentially requiring keyphrase recognition, while hidden size and batch size require careful optimization.
Abstract
from arXiv · showhide
Deep neural networks (DNN) have revolutionized the field of natural language processing (NLP). Convolutional neural network (CNN) and recurrent neural network (RNN), the two main types of DNN architectures, are widely explored to handle various NLP tasks. CNN is supposed to be good at extracting position-invariant features and RNN at modeling units in sequence. The state of the art on many NLP tasks often switches due to the battle between CNNs and RNNs. This work is the first systematic comparison of CNN and RNN on a wide range of representative NLP tasks, aiming to give basic guidance for DNN selection.
1 Introduction
The paper addresses the lack of consensus on choosing CNNs or RNNs for NLP by systematically comparing CNNs, GRUs, and LSTMs across representative tasks. Experiments find complementary architecture strengths and strong sensitivity to hidden size and batch size.
- Research problem: NLP research lacks a clear architecture choice because CNNs and RNNs can each excel on tasks associated with the other architecture.RNNs perform well on document-level sentiment classification, while gated CNNs have outperformed LSTMs on language modeling.
- Approach: The study systematically compares CNNs, GRUs, and LSTMs across sentiment and relation classification, textual entailment, answer selection, question-relation matching, path query answering, and part-of-speech tagging.
- Findings: CNNs and RNNs provide complementary information for text classification, with the better architecture depending on the importance of understanding the whole sequence.
- Findings: Learning rate changes performance relatively smoothly, whereas hidden size and batch size produce large performance fluctuations.
2 Related Work
Prior work reports mixed CNN-versus-RNN outcomes across NLP tasks, including evidence for complementary information and comparable recurrent variants. This motivates systematic comparison rather than a universal architecture preference.
- Prior evidence: No prior work had systematically compared CNNs and RNNs across a large array of NLP tasks.
- Prior evidence: Relation-classification results found CNNs outperforming basic RNNs while also identifying complementary information from weighted word combinations and informative ngrams.
- Prior evidence: Other studies support CNNs for long-sentence classification, attention-based CNNs for answer selection, and gated CNNs for long-context language modeling.
- Prior evidence: Sentiment analysis of Russian tweets found GRUs outperforming both LSTMs and CNNs.
- Prior evidence: Empirical studies report no clear winner between GRUs and LSTMs, with comparable performance often making hyperparameter tuning more important than architecture selection.
3 Models
The paper presents CNN, GRU, and LSTM models for processing sequences. CNNs learn local representations with convolution and max pooling, while recurrent models update hidden representations sequentially, with LSTMs additionally maintaining gated history.
- CNN: The CNN represents an input sequence as a d × n feature map of dense token vectors.
- CNN: CNN convolution learns representations from sliding w-grams, concatenating w embeddings and applying weights W and bias b after zero padding outside the sequence.
- CNN: Max pooling combines all w-gram representations into one sequence representation by taking the maximum for each feature dimension.
- GRU: GRU processes text sequentially, using the hidden state h_t to encode the history x_1, · · ·, x_t and gates z and r to control updates.
- LSTM: LSTM uses input, forget, and output gates to combine a temporary result with prior history, update that history, and produce the current hidden state.
4 Experiments
The experiments compare CNNs, GRUs, and LSTMs across four NLP task categories, showing that architecture performance depends on sequence semantics and that hidden size and batch size are unstable hyperparameters.
- Experimental setup: The study evaluates CNN, GRU, and LSTM models on text classification, semantic matching, sequence-order, and context-dependency tasks.Tasks include sentiment and relation classification, textual entailment, answer selection, question-relation matching, path query answering, and POS tagging.
- Experimental setup: Experiments train basic architectures from scratch, without pretrained embeddings or complex tricks, while tuning hyperparameters separately for each task and model.For POS tagging, bidirectional RNNs are additionally reported because they encode context from both sides.
- Results: GRU performs best on sentiment classification, matches CNN on relation classification, and outperforms CNN on textual entailment; CNN leads on answer selection and question-relation matching.For path query answering, both GRU and LSTM outperform CNN.
- Results: CNN outperforms one-directional RNNs on POS tagging but lags behind bidirectional RNNs, while RNNs are well suited to sequence order and long-range context dependency.These results align with the role of RNNs in encoding order and context information.
- Results and analysis: CNNs and RNNs provide complementary information for text classification: GRU gains advantage on longer sentiment examples, whereas CNN and GRU are comparable for lengths below 10.Long examples often require comprehending inverse-semantic clauses rather than relying on a local sentiment indicator.
- Results and analysis: Learning-rate changes produce relatively smooth performance shifts, but hidden-size and batch-size changes cause large oscillations across models.The CNN curve is mostly below GRU and LSTM on sentiment classification but higher on answer selection.
5 Conclusions
The paper compares CNN, GRU, and LSTM across representative NLP tasks and finds that RNNs are broadly effective, except when tasks primarily require keyphrase recognition. It also identifies hidden size and batch size as crucial sources of performance variation.
- Conclusions: RNNs perform well and robustly across a broad range of NLP tasks, except when the task is essentially keyphrase recognition.The cited examples include some sentiment detection and question-answer matching settings.
- Conclusions: Hidden size and batch size can make CNN and RNN performance vary dramatically, making their optimization crucial for good performance.