Source-linked AI summary
Cached Long Short-Term Memory Neural Networks for Document-Level Sentiment Classification
Jiacheng Xu, Danlu Chen, Xipeng Qiu, Xuangjing Huang
TL;DR
Document-level sentiment classification remains difficult because recurrent memory can lose sentiment information in long texts. CLSTM divides internal memory into groups with different forgetting rates, and the paper reports state-of-the-art performance across three document-level datasets. Its conclusion is supported within the evaluated review-dataset setting, while dynamically adjusting forgetting rates remains future work.
Problem
Long document-level sentiment classification challenges recurrent models because standard memory can lose significant sentiment information and suffer gradient vanishing across hundreds of words.
Method
CLSTM equips LSTM with a cache mechanism that divides memory into groups with distinct forgetting rates, using faster groups to transfer information toward slower groups.
Results
CLSTM outperforms state-of-the-art models by a large margin on the Yelp 2013, Yelp 2014, and IMDB document-level datasets.
Takeaways & Limitations
The model captures local and global semantic information within an RNN architecture while keeping computational complexity and parameter counts almost unchanged relative to counterpart models.
Takeaways & Limitations
The authors leave dynamically adjusting forgetting rates for fine-grained document-level sentiment classification as future work.
Abstract
from arXiv · showhide
Recently, neural networks have achieved great success on sentiment classification due to their ability to alleviate feature engineering. However, one of the remaining challenges is to model long texts in document-level sentiment classification under a recurrent architecture because of the deficiency of the memory unit. To address this problem, we present a Cached Long Short-Term Memory neural networks (CLSTM) to capture the overall semantic information in long texts. CLSTM introduces a cache mechanism, which divides memory into several groups with different forgetting rates and thus enables the network to keep sentiment information better within a recurrent unit. The proposed CLSTM outperforms the state-of-the-art models on three publicly available document-level sentiment analysis datasets.
1 Introduction
Document-level sentiment analysis must preserve global sentiment information across long, noisy texts, where simple recurrent models struggle to retain distant signals. CLSTM addresses this with grouped memory and different forgetting rates to capture local and long-range information.
- Document-level analysis must select and store global sentiment information from long texts containing noise and redundant local patterns.
- Simple RNNs struggle to retain key sentiment messages across relatively distant time steps.
- Hierarchical methods combine sentence representations into document embeddings but may impose structural assumptions or discard within-sentence order.
- CLSTM equips standard LSTM with a cache mechanism that divides internal memory into groups with different forgetting rates.
- The cache groups bridge information toward groups with lower forgetting rates, helping capture, remember, and forget semantic information at different timescales.
- CLSTM is reported to outperform state-of-the-art methods on Yelp 2013, Yelp 2014, and IMDB document-level datasets.
2 Related Work
Document-level sentiment classification requires inferring whole-document sentiment when informative content is unevenly distributed and distant context can be forgotten. CLSTM instead uses internally grouped memory with distinct forgetting rates so every group updates at every time step.
- Document-level sentiment classification infers the polarity or intensity of a whole document whose parts are not equally informative.
- Traditional sentiment methods often require effective handcrafted features, while neural methods learn discriminative features from data.
- Hierarchical approaches are commonly used for document-level classification, but long documents remain difficult for recurrent models to represent.
- Although LSTMs retain information longer than RNNs, they can still forget information far from the current point.
- External-memory approaches can suffer from poor time performance because of large memory matrices, whereas CLSTM adjusts LSTM internal memory.
- Unlike multiple-time-scale models that update slower groups intermittently, CLSTM updates every memory group at every time step.
3 Long Short-Term Memory Networks
LSTM is a recurrent architecture that uses memory and gates to model long dependencies while mitigating gradient diffusion and explosion. Its component updates are parameterized by gates, memory activation, and trainable transformations.
- LSTM alleviates gradient diffusion and explosion through a recurrent memory unit and gate mechanism.
- The memory unit and gates help LSTM capture long dependencies by controlling how stored information is used and updated.
- The formal LSTM update uses input, forget, and output gates together with memory-cell and hidden-state representations.
- The gates and memory-cell activation are parameterized by trainable input-to-hidden and hidden-to-hidden matrices.
4 Cached Long Short-Term Memory Neural Network
CLSTM extends LSTM with a cache mechanism that divides memory into groups with distinct forgetting rates, capturing information at multiple temporal scales for long-text sentiment classification. A bidirectional variant combines contextual information from both directions before document-level prediction.
- CIFG-LSTM: CLSTM uses the CIFG-LSTM variant, which couples the input and forget gates into one gate through i(t) = 1 − f(t).This design replaces the separate input and forget gates while retaining the output gate.
- Cached LSTM: Groups with low forgetting rates serve as long-term memories, while high-rate groups act as short-term caches that transfer information toward slower groups.The cache mechanism is intended to preserve both local and long-range dependencies.
- Cached LSTM: CLSTM divides LSTM memory cells into K groups, each with its own internal memory, output gate, and forgetting rate.The forgetting rates are constrained to distinct ranges so groups acquire different temporal roles.
- Cached LSTM: The squash function constrains each group’s forgetting rate to a distinct range, with G1 slowest and GK fastest.A rate near 0 indicates long-term memory, whereas a rate near 1 indicates short-term memory.
- Bidirectional CLSTM: Bidirectional CLSTM receives information from both context directions and concatenates the slowest group’s endpoint states to represent a document.The resulting document representation is passed through a fully connected layer and softmax classifier.
5 Training
Training minimizes cross-entropy between predicted and true label distributions while applying L2 regularization across the model parameters.
- Objective: The training objective combines cross-entropy error with an L2 regularization term over all parameters.The objective is defined for m training sentence-label pairs.
6 Experiment
The experiments evaluate CLSTM on three real-world document-level sentiment datasets because common sentiment benchmarks often contain only short paragraphs. The study reports comparisons with competing models and dataset-specific configurations.
- Results: The experiments report that the proposed model outperforms competitor models in several aspects when modeling long texts.The paper evaluates the model across three document-level sentiment datasets.
- Datasets: The evaluation uses Yelp 2013, Yelp 2014, and IMDB for document-level sentiment classification.These are publicly accessible datasets preprocessed and split following prior work.
- Configuration: Table 3 provides the optimal hyper-parameter configuration for each of the three datasets.The table is dataset-specific rather than a single shared configuration.
- Datasets: IMDB reviews average 394.6 words, making them substantially longer than the two Yelp review datasets.Yelp ratings range from 1 to 5 stars, while IMDB ratings range from 1 to 10.
6.2 Evaluation Metrics
The evaluation uses Accuracy and MSE for sentiment classification and compares CLSTM and B-CLSTM with multiple baseline models across document-level datasets.
- Accuracy measures the overall classification result, while MSE measures divergence between predicted sentiment labels and ground-truth labels.
- The experiments compare CLSTM and B-CLSTM with CBOW, JMARS, CNN, RNN, LSTM, BLSTM, and CIFG-based recurrent baselines.
- Table 2 reports classification accuracy and MSE for the models on IMDB, Yelp 2014, and Yelp 2013.
6.4 Hyper-parameters and Initialization
The models use validation-based hyper-parameter selection, pretrained word vectors, and recurrent initialization choices; results are reported against competitive models and established baselines.
- Hyper-parameters are selected mainly by validation accuracy, with 50-dimensional word vectors, 120 hidden units, Adagrad, learning rate 0.01, and batch sizes of 32, 64, or 128.Weight decay is selected from {5e−4, 1e−4, 1e−5}; CLSTM memory-group counts are chosen separately for each dataset.
- Recurrent matrices are initialized uniformly in [-0.1, 0.1], while GloVe embeddings are fine-tuned during training and the best validation configuration is evaluated on the test set.
- Figure 3 evaluates convergence speed on Yelp 2013 by plotting classification accuracy against iteration epochs.
- The reported classification results compare CLSTM and B-CLSTM with other competitive models using accuracy and MSE.
6.6 Rate of Convergence
The convergence and memory-group experiments examine how CLSTM training speed and classification accuracy vary with its memory organization.
- 6.6 Rate of Convergence: B-CLSTM converges faster than CIFG-LSTM, CIFG-BLSTM, LSTM, and BLSTM under approximately matched parameter counts.The authors attribute this to splitting memory groups acting as better initialization and training constraints.
- 6.7 Effectiveness on Grouping Memory: Figure 4 uses the number of memory groups on the x-axis and classification accuracy on the y-axis across three datasets.
- 6.8 Sensitivity on Document Length: Figure 5 ranks IMDB test samples by document length from 0% to 100% after dividing them into ten parts.
- 6.7 Effectiveness on Grouping Memory: In Yelp 2013, four memory groups produce the best result among the tested group counts, while performance drops beyond five groups.The study plots validation prediction accuracy for different memory-group counts across all datasets.
- 6.7 Effectiveness on Grouping Memory: More memory groups reduce the neurons assigned to each group because the total neuron count is held equal to vanilla LSTM.The reduced per-group capacity is identified as a reason for worse performance with excessive splitting.
6.8 Sensitivity on Document Length
The IMDB length-sensitivity experiment evaluates accuracy across ten groups of documents ordered by length, finding that B-CLSTM performs well on both short and long texts.
- IMDB test samples are divided into ten groups according to document length for evaluating model performance across text lengths.
- B-CLSTM performs better overall than CIFG-BLSTM and is described as adaptive to both short texts and long documents.
- Bidirectional models perform much better than their counterpart models across the document-length evaluation.
- CBOW is slightly better than CIFG-LSTM, which the authors associate with information loss during unidirectional propagation.
7 Conclusion
The model addresses document-level sentiment analysis in an RNN architecture and outperforms state-of-the-art models on three real-world review datasets. Future work will dynamically adjust forgetting rates for fine-grained analysis.
- The model outperforms state-of-the-art models by a large margin on three real-world document-level review datasets.
- Future work will design a strategy to dynamically adjust forgetting rates for fine-grained document-level sentiment analysis.