Source-linked AI summary
Effects of padding on LSTMs and CNNs
Mahidhar Dwarampudi, N V Subba Reddy
TL;DR
The paper examines how pre- versus post-padding affects LSTM and CNN performance on tweet sentiment analysis. It compares both padding choices on the same task and reports that LSTMs perform better with pre-padding, while padding position matters little for CNNs.
Problem
Because LSTMs and CNNs require equal-length sequential inputs, the paper asks how padding position affects their performance on sentiment analysis.
Source: §1 · p. 1Method
The paper compares pre- and post-padding for LSTMs and CNNs on the same tweet sentiment-analysis task using preprocessed text and word vectors.
Source: §1 · p. 1, §2.1.2 · p. 3Results
For sentiment analysis, LSTMs are more efficient than CNNs and perform better with pre-padding, while pre- versus post-padding matters little for CNNs.
Source: §5 · p. 5Takeaways & Limitations
The paper concludes that pre-padding is preferable for LSTMs and may be better when combining multiple neural-network types for a task.
Source: §5 · p. 5
Abstract
from arXiv · showhide
Long Short-Term Memory (LSTM) Networks and Convolutional Neural Networks (CNN) have become very common and are used in many fields as they were effective in solving many problems where the general neural networks were inefficient. They were applied to various problems mostly related to images and sequences. Since LSTMs and CNNs take inputs of the same length and dimension, input images and sequences are padded to maximum length while testing and training. This padding can affect the way the networks function and can make a great deal when it comes to performance and accuracies. This paper studies this and suggests the best way to pad an input sequence. This paper uses a simple sentiment analysis task for this purpose. We use the same dataset on both the networks with various padding to show the difference. This paper also discusses some preprocessing techniques done on the data to ensure effective analysis of the data.
1 Introduction
LSTMs and CNNs require equal-length sequential inputs, so padding is used to align them. This paper compares pre- and post-padding for both networks on tweet sentiment analysis using word vectors.
1 Introduction
- The study compares pre- and post-padding effects on LSTMs and CNNs using the same sentiment-analysis task and dataset.Tweets are classified as positive or negative using word vectors.Source: §1 · p. 1
- Because LSTMs and CNNs take sequential inputs of equal length, inputs must be padded to align their lengths.Source: §1 · p. 1
- The paper also discusses preprocessing techniques used to prepare the data for analysis.Source: §1 · p. 1
2 Literature Survey
The literature survey situates LSTMs and CNNs across sequence, image, and text tasks, then introduces variable-length input preparation, word vectors, and the networks' differing sequence-processing mechanisms.
2 Literature Survey
- Prior examples use LSTMs for machine translation, image captioning, handwriting generation, and question answering [1] [2] [3] [4].CNNs are described as widely used for image classification and facial recognition [5] [6], and later for sentence classification and sentiment analysis [7] [8].Source: §2 · p. 1
2.1 Preprocessing Variable Length Input Sequences
- Input sequences can vary in length, so they must be made equal length before training.The survey presents preprocessing approaches for variable-length inputs.Source: §2.1 · p. 2, §2 · p. 2
2.1.1 Truncation
- Pre-sequence truncation removes values from sequence beginnings, while post-sequence truncation removes values from endings.Both approaches use the smallest sequence length or a chosen length; when that length exceeds the smallest sequence, that sequence is padded.Source: §2.1.1 · p. 2
2.1.2 Padding
- Pre-padding adds zeros at sequence beginnings and post-padding adds zeros at sequence endings to reach the longest or a chosen longer length.Because truncation causes data loss, the paper prefers padding and compares the two padding positions for LSTMs and CNNs.Source: §2.1.2 · p. 3
2.2 Word Vectors
- The paper uses ordered word vectors rather than bag-of-words representations, constructing vectors with the Word2Vec skip-gram model [9].Skip-gram trains on a word and words in its surrounding window, then uses the penultimate layer output as the word vector.Source: §2.2 · p. 3
2.3 LSTMs and CNNs
- LSTMs use a memory or cell state updated across a sequence, whereas CNNs seek patterns using filters and do not have memory.CNN filters produce feature maps; pooling reduces spatial representation, parameters, and computation before a fully connected classifier.Source: §2.3 · p. 4
3 Data
The study uses a subset of Thinknook tweets, preprocesses their text, and trains 100-dimensional skip-gram word vectors before padding sequences to the maximum tweet length.
3 Data
- The dataset contains about 157,860 tweets, split into 126,288 training tweets and 31,572 test tweets.Training data includes 63,001 positive and 63,287 negative tweets; only 10% of the published dataset was used.Source: §3 · p. 4
- Preprocessing replaces numbers, Twitter handles, and URLs with ‘0’, ‘1’, and ‘2’, lemmatizes words, and retains stop words.The paper retains stop words because negative stopwords can affect tweet sentiment.Source: §3 · p. 4
- Skip-gram training uses a five-word window, 100 hidden units, and five epochs to produce 100-dimensional word vectors.Source: §3 · p. 4
- All sequences are padded to length 93, the length of the longest tweet.Source: §3 · p. 4
4 Final Comparison
The paper describes four-layer LSTM and CNN models for comparison, with 100 units per layer and dropout; the supplied section passages specify model configurations but not comparison outcomes.
4 Final Comparison
- The LSTM has four hidden layers of 100 neurons, with 0.2 dropout and 0.2 recurrent dropout on each layer.It takes sequences of length 93, uses tanh except for a sigmoid output layer, and classifies with that output.Source: §4 · p. 4
- The supplied model descriptions do not report which network or padding configuration performs better.Source: §4 · p. 4
- The CNN has four hidden layers of 100 units, with 0.2 dropout on each layer and a sigmoid output neuron.The hidden layers use linear activation.Source: §4 · p. 4
5 Results
The results favor pre-padding for LSTMs, while padding choice matters little for CNNs. The post-padding LSTM peaked at six epochs, then overfit, and had lower accuracy than pre-padding.
5 Results
- The post-padding LSTM peaked at 6 epochs, then overfit, and its accuracy was lower than the pre-padding model.The paper compares the padding conditions using a conversational analogy: longer response delays correspond to less retained context and less relevant replies.Source: §5 · pp. 4–5
- Pre-padding was more efficient for LSTMs, while pre- versus post-padding mattered little for CNNs.The tables compare padding approaches and accuracy for both model types.Source: §5 · pp. 4–5, Table 1 · p. 4, Table 2 · p. 5
- For sentiment analysis, the paper concludes that pre-padding is generally preferable when combining multiple neural-network types.The stated rationale is that padding choice matters little to CNNs, unlike LSTMs.Source: §5 · p. 5