PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Learning for Sentiment Analysis : A Survey

Lei Zhang, Shuai Wang, Bing Liu

arXiv:1801.07883v2cs.CLcs.IRcs.LGstat.ML

TL;DR

Sentiment analysis must process large and diverse collections of online opinions that are difficult for people to monitor and summarize manually. This paper surveys deep learning architectures and their sentiment-analysis applications, reporting that many reviewed techniques achieve state-of-the-art results across sentiment-analysis tasks. The survey also identifies representation and sequence-length limitations relevant to these approaches.

  • Problem

    Large, diverse volumes of online opinion text make manual identification, extraction, and summarization difficult, motivating automated sentiment analysis.

  • Method

    The paper first overviews deep learning and then comprehensively surveys its applications to sentiment analysis, including architectures such as autoencoders and convolutional neural networks.

  • Results

    Many reviewed deep learning techniques show state-of-the-art results for various sentiment-analysis tasks.

  • Takeaways & Limitations

    Deep learning for sentiment analysis is a growing research area with applications spanning multiple architectures and sentiment-analysis tasks.

  • Takeaways & Limitations

    Bag-of-Words representations ignore word order and barely encode word semantics, while standard RNNs practically look back only a few steps because of vanishing or exploding gradients.

Abstract

from arXiv · show

Deep learning has emerged as a powerful machine learning technique that learns multiple layers of representations or features of the data and produces state-of-the-art prediction results. Along with the success of deep learning in many other application domains, deep learning is also popularly used in sentiment analysis in recent years. This paper first gives an overview of deep learning and then provides a comprehensive survey of its current applications in sentiment analysis.

INTRODUCTION

Sentiment analysis studies opinions and attitudes expressed about entities, while the Web and social media have created vast digital opinion data. This survey introduces deep learning and reviews its applications to sentiment analysis.

  • Sentiment analysis computationally studies opinions, sentiments, emotions, appraisals, and attitudes toward entities and their attributes.
  • Web and social media produced huge volumes of digitally recorded opinionated data, expanding sentiment analysis across NLP and other fields.
  • Organizations and individuals increasingly use online reviews and discussions instead of relying only on surveys, polls, focus groups, friends, or family.
  • Automated sentiment analysis is needed because diverse websites contain large, difficult-to-decipher volumes of opinion text.
  • Earlier sentiment-analysis research used supervised classifiers, feature combinations, sentiment lexicons, grammatical analysis, and syntactic patterns.

DEEP LEARNING

Deep learning applies multilayer neural networks to learn hierarchical representations, with lower layers extracting simple features and higher layers combining them into more complex ones. The survey describes these architectures and related NLP techniques, including word embeddings.

  • Deep learning overcame the earlier view that neural networks were practical only with one or two layers because deeper training was complicated and expensive.
  • Deep learning uses multiple nonlinear processing layers whose lower layers learn simple features and higher layers learn increasingly complex derived features.This creates a hierarchical feature representation.
  • The survey introduces major deep learning architectures and related techniques applied to NLP tasks.
  • Word embedding transforms vocabulary words from high-dimensional sparse vectors into lower-dimensional dense vectors whose dimensions represent latent word features.
  • Word2Vec learns word embeddings with CBOW predicting a target word from its context and Skip-Gram using the reverse prediction direction.

AUTOENCODER AND DENOISING AUTOENCODER

The paper surveys several deep neural architectures relevant to sentiment analysis, including autoencoders, convolutional networks, and recurrent networks. These models learn representations through reconstruction, local feature extraction, or sequence memory, while standard RNNs remain limited by gradient problems.

  • AUTOENCODER AND DENOISING AUTOENCODER: Autoencoders learn hidden representations by reconstructing their inputs through an encoder and decoder.The model maps an input to a latent hidden representation, then reconstructs the input and minimizes reconstruction error.
  • AUTOENCODER AND DENOISING AUTOENCODER: Stacked autoencoders provide multiple levels of representation that can initialize a supervised deep neural network.Higher-level autoencoders use lower-level outputs as training data and are trained unsupervised.
  • AUTOENCODER AND DENOISING AUTOENCODER: Denoising autoencoders corrupt inputs during training so hidden layers learn robust features rather than the identity mapping.Training minimizes denoising reconstruction error and requires reconstructing inputs despite noise.
  • CONVOLUTIONAL NEURAL NETWORK: CNN convolutional layers extract local features, while convolution and pooling detect local indicators regardless of their positions.This local connectivity pattern is useful in NLP because class clues can occur at different positions in the input.
  • RECURRENT NEURAL NETWORK: RNNs process sequences using a hidden state that depends on the current input and previous hidden state.The hidden state acts as memory, and shared parameters across time steps reduce the total number of parameters.
  • RECURRENT NEURAL NETWORK: Standard RNNs can theoretically use arbitrarily long sequences but practically look back only a few steps because of vanishing or exploding gradients.Bidirectional, deep bidirectional, and long short-term memory networks were developed to address shortcomings of the standard model.

LSTM NETWORK

LSTM is an RNN variant designed to learn long-term dependencies through interacting layers, hidden and cell states, and gated memory updates. Variants extend it to tree-structured data or simplify its gating and state design.

  • LSTM is a special type of RNN capable of learning long-term dependencies.
  • Its repeating module has four interacting layers and maintains both hidden and cell states.
  • The forget gate controls which cell-state information is retained or discarded from the previous time step.
  • The input gate selects values to update, while a tanh layer creates candidate values that are added to the cell state.
  • The forget gate regulates gradient flow and permits explicit memory deletion and updating, helping alleviate vanishing or exploding gradients.
  • Tree-LSTM generalizes LSTM to tree-structured data, while GRU combines forget and input gates and merges cell and hidden states.

ATTENTION MECHANISM WITH RECURRENT NEURAL NETWORK

Attention addresses persistent long-range-dependency problems by letting recurrent models focus selectively on input information. The section also situates attention alongside memory and recursive architectures.

  • Attention was proposed because bidirectional RNNs and LSTMs still have difficulty handling long-range dependencies in practice.
  • In NLP, attention learns which input text to focus on using the input and the model’s prior outputs.
  • In encoder-decoder translation, each output word depends on a weighted combination of input states rather than only the final state.
  • Attention weights indicate each input state’s contribution to an output and normally sum to 1.
  • End-to-End Memory Networks use recurrent attention over long-term memory and can be trained end to end with standard backpropagation.
  • Recursive neural networks compose representations bottom-up over directed acyclic structures such as sentence parse trees.

SENTIMENT ANALYSIS TASKS

The survey introduces sentiment analysis tasks at document, sentence, and aspect granularity, then reviews deep-learning approaches for document-level classification. These approaches replace or augment bag-of-words representations with dense, structured, and context-sensitive representations.

  • Sentiment analysis is studied mainly at document, sentence, and aspect levels, with additional work on emotion, sarcasm, and multilingual sentiment.
  • Document-level classification assigns an overall polarity, rating score, or one of five classes to an opinion document.
  • Bag-of-words ignores word order and poorly represents semantics, while bag-of-n-grams introduces sparsity and high dimensionality.
  • Word embeddings provide dense vectors that can encode some semantic and syntactic properties for neural document representations.
  • An empirical comparison found that artificial neural networks produced competitive results to SVMs in most document-level cases.
  • Document-level models use representations from paragraph vectors, autoencoders, CNN variants, recurrent encoders, user and product information, caching, and hierarchical attention.
  • Attention mechanisms can operate at word and sentence levels, allowing review models to weight individual words or sentences differently.

SENTENCE LEVEL SENTIMENT CLASSIFICATION

Sentence-level sentiment classification targets the sentiment of individual sentences, often using subjectivity and polarity distinctions. Deep-learning research combines learned representations with syntactic, linguistic, contextual, domain, social, and reader-related information.

  • Sentence-level classification determines the sentiment expressed in a single sentence through subjectivity and polarity classification.
  • Earlier models used parse trees with words, whereas CNNs and RNNs increasingly learn from word embeddings without requiring parse trees.
  • Recursive models include semi-supervised recursive autoencoders and matrix-vector recursive networks that build sentence representations over tree structures.
  • CNN-based approaches model sentence semantics using dynamic k-max pooling, character-to-sentence convolutions, and different word-embedding configurations.
  • LSTM and CNN-RNN hybrids model word interactions, local features, long-distance dependencies, and valence-arousal ratings in short texts.
  • Other models incorporate linguistic resources, weak supervision, cross-domain data, social relationships, user gaze, and cognitive features.

ASPECT LEVEL SENTIMENT CLASSIFICATION

Aspect-level sentiment classification predicts sentiment toward a specified target, so the same sentence can receive different polarities for different aspects. Deep learning approaches address this by modeling target–context relations through embeddings, attention, memory, gating, and interactive architectures.

  • Task definition: Aspect-level classification infers sentiment toward a target aspect rather than assigning one polarity to the whole sentence.For example, “screen” is positive while “battery life” is negative in the same sentence.
  • Challenges: The central challenge is that context words influence sentiment differently, requiring models to capture semantic connections between targets and their surrounding context.
  • Target and context representation: Neural models represent targets explicitly through target embeddings or target-conditioned features, including TD-LSTM and TC-LSTM.These models concatenate target information with context features to support aspect sentiment classification.
  • Attention and memory: Attention and memory architectures identify context words relevant to a target, using target-specific attention, external memory, or separate target and context attentions.Examples include attention-based LSTMs, end-to-end memory networks, and Interactive Attention Networks.
  • Interaction modeling: Other approaches model target–context interactions with gated, recurrent, tensor, or holographic structures, while some jointly learn target identification and sentiment classification.Joint models allow target-detection signals to inform polarity classification and predicted polarity to provide feedback for identifying targets.

ASPECT EXTRACTION AND CATEGORIZATION

Aspect extraction identifies opinion targets, and aspect categorization groups equivalent expressions into shared categories. Deep learning supports these tasks by learning feature representations and by combining neural, structured, attention-based, and semi-supervised methods.

  • Task definition: Aspect extraction identifies aspect terms or entities associated with opinions, while aspect categorization groups equivalent expressions into categories.For example, “image,” “photo,” and “picture” can belong to the Image category.
  • Motivation: Deep learning can capture aspect–context semantics by learning complicated feature representations in hidden layers.
  • Joint extraction: Neural sequence and structured models jointly extract aspects with opinion terms, sentiments, or relations between opinion entities.Reported approaches combine bidirectional LSTMs, RNNs, CRFs, neural layers, and information propagation between aspect and opinion terms.
  • Representation and grouping: Aspect extraction methods use convolutional tagging, dependency-aware embeddings, attention, and metric learning for identifying or grouping aspect expressions.These approaches include CNN tagging, dependency-path representations, attention-based embedding learning, and K-means over learned feature subspaces.
  • Cross-domain extraction: Cross-domain extraction can combine true labels with auxiliary rule-based label sequences in RNN-based models.

OPINION EXPRESSION EXTRACTION

Opinion expression extraction identifies sentiment-bearing expressions, while related work extends deep learning to compositional sentiment, opinion holders, and future opinion prediction. The surveyed methods use recurrent, recursive, convolutional, and memory-based architectures.

  • Task definition: Opinion expression extraction identifies sentiment expressions in sentences or documents and is treated similarly to other sequence-labeling tasks.
  • Motivation: Deep learning is applicable because opinion-expression characteristics can be represented in a learned feature space.
  • Extraction models: Deep bidirectional RNNs outperform traditional shallow RNNs with the same number of parameters and previous CRF methods in reported opinion-expression extraction experiments.
  • Extraction models: RNN-based discriminative models use pretrained word embeddings across Elman-type, Jordan-type, LSTM, and related architectures.
  • Related tasks: Recursive neural networks are suited to sentiment composition because their tree structures represent constituent meanings and grammatical structure.A deep RecNN was reported to capture different aspects of compositionality more accurately.
  • Related tasks: Neural models also address future opinion prediction by encoding past content and social relations or forecasting implied sentiments among event participants.

SENTIMENT ANALYSIS WITH WORD EMBEDDING

Word embeddings provide semantic representations for sentiment analysis, while sentiment-aware, paragraph-level, multilingual, and multi-sense variants adapt those representations to specific tasks. Deep learning with related representations is also applied to sarcasm and emotion analysis.

  • Embedding foundations: Word embeddings contribute to sentiment analysis both as inputs to deep models and as features for non-neural models.
  • Sentiment-aware embeddings: Regular context-based embeddings may place sentiment-opposite words near each other, motivating sentiment-encoded embeddings.Sentiment-aware methods aim to capture semantic information together with polarity.
  • Document representations: Paragraph vectors learn fixed-length representations for variable-length texts and produced gains on sentence- and document-level sentiment classification tasks.
  • Specialized embeddings: Embedding research extends to enriched features, multi-sense representations, bilingual embeddings, and cross-lingual aspect-based sentiment classification.Bilingual sentiment embeddings incorporate sentiment information into English–Chinese representations using labeled corpora and translation.
  • Sarcasm analysis: Deep learning models for sarcasm detection combine bidirectional recurrent or convolutional architectures with textual, contextual, emotional, sentiment, personality, psychological, or cognitive features.
  • Emotion analysis: Emotion analysis applies related deep learning strategies to code-switched prediction, conversation generation, distant-supervision datasets, emotion detection, and emotion-cause extraction.Attention, LSTM, GRU, and emoji-based pretraining are among the reported approaches.

MULTIMODAL DATA FOR SENTIMENT ANALYSIS

Deep learning extends sentiment analysis beyond text by jointly modeling textual, visual, and acoustic signals. The surveyed work applies CNNs, RNNs, attention, fusion, and multimodal-specific methods across diverse tasks.

  • Multimodal foundations: Multimodal sentiment analysis combines textual, visual, and acoustic inputs to learn shared or fused sentiment representations.Deep learning models project modalities into latent spaces using concatenation, joint latent spaces, or more sophisticated fusion approaches.
  • Text and acoustic modeling: CNNs extract text features for heterogeneous multimodal fusion, with multiple kernel learning used for classification.
  • Text and acoustic modeling: CNN-based models support emotion and sentiment recognition in acoustic data and sentiment, emotion, and personality recognition in dialogue systems.
  • Visual sentiment analysis: Visual sentiment methods use adjective and noun descriptions, conditional probability networks, hierarchical CNN features, bidirectional RNNs, and attention over local image regions.These approaches provide alternative supervision, feature integration, and region-selection mechanisms for image sentiment classification.
  • Video and multimodal dynamics: Video-based approaches model utterance-level sentiment, contextual information, physiological and video signals, and intra- and inter-modality dynamics.Examples include LSTM-based models, multimodal datasets, Tensor Fusion Networks, cognition-based attention, and Select-Additive Learning for confound removal.

RESOURCE-POOR LANGUAGE AND MULTILINGUAL SENTIMENT ANALYSIS

Deep learning has been applied to sentiment analysis in resource-poor and multilingual settings, as well as to related tasks. The surveyed applications span language-specific models, translation-based systems, lexicon expansion, recommendation, volatility prediction, and stance detection.

  • Resource-poor and multilingual languages: Deep learning has enabled progress in sentiment analysis for resource-poor languages and has been applied to multilingual sentiment settings.
  • Resource-poor and multilingual languages: Hindi, Arabic, and Hindi-English code-mixed sentiment classification use CNN, word-embedding, and sub-word LSTM architectures.
  • Resource-poor and multilingual languages: A multilingual system translates several languages into English before applying English embeddings, sentiment-lexicon polarities, and a CNN classifier.Experiments include Hindi, Marathi, Russian, Dutch, French, Spanish, Italian, German, and Portuguese.
  • Related sentiment tasks: Related applications include intersubjectivity modeling, opinion-lexicon expansion, financial volatility prediction, personalized opinion recommendation, and stance detection.
  • Survey conclusion: The survey reports state-of-the-art results for many sentiment analysis tasks and anticipates further research as deep learning advances.
Loading 1801.07883v2…