Source-linked AI summary
Robust Image Sentiment Analysis Using Progressively Trained and Domain Transferred Deep Networks
Quanzeng You, Jiebo Luo, Hailin Jin, Jianchao Yang
TL;DR
Visual sentiment analysis extends sentiment modeling beyond text but faces subjective interpretation, noisy labels, limited labeled data, and domain variation. The paper develops a CNN with progressive training and domain transfer, using large weakly labeled Flickr data and manually labeled Twitter images. The authors report notable improvements and better performance than competing feature-based approaches.
Problem
Visual sentiment analysis requires models that can learn from large-scale weakly labeled images and generalize across domains despite subjective, diverse visual content.
Method
The paper designs a CNN architecture and fine-tunes it using progressive training on weakly labeled Flickr images followed by domain transfer with manually labeled Twitter images.
Results
The proposed CNN achieves notable improvements and can outperform classifiers using predefined low-level features or mid-level visual attributes for visual sentiment analysis.
Takeaways & Limitations
Properly trained CNNs can transfer learned knowledge to other domains using a simpler fine-tuning technique than methods cited in the paper.
Abstract
from arXiv · showhide
Sentiment analysis of online user generated content is important for many social media analytics tasks. Researchers have largely relied on textual sentiment analysis to develop systems to predict political elections, measure economic indicators, and so on. Recently, social media users are increasingly using images and videos to express their opinions and share their experiences. Sentiment analysis of such large scale visual content can help better extract user sentiments toward events or topics, such as those in image tweets, so that prediction of sentiment from visual content is complementary to textual sentiment analysis. Motivated by the needs in leveraging large scale yet noisy training data to solve the extremely challenging problem of image sentiment analysis, we employ Convolutional Neural Networks (CNN). We first design a suitable CNN architecture for image sentiment analysis. We obtain half a million training samples by using a baseline sentiment algorithm to label Flickr images. To make use of such noisy machine labeled data, we employ a progressive strategy to fine-tune the deep network. Furthermore, we improve the performance on Twitter images by inducing domain transfer with a small number of manually labeled Twitter images. We have conducted extensive experiments on manually labeled Twitter images. The results show that the proposed CNN can achieve better performance in image sentiment analysis than competing algorithms.
Introduction
Visual sentiment analysis addresses the growing use of images and videos for expressing opinions, while confronting subjective labels, limited prior work, noisy data, and cross-domain generalization. This work develops a CNN architecture and training strategies for these challenges, alongside a manually labeled dataset.
- Online sentiment research has largely relied on textual content, although images and videos are increasingly used to express opinions and experiences.
- Visual sentiment analysis is challenging because sentiment requires subjective, high-level interpretation across diverse objects, scenes, actions, and events.
- Large-scale supervised learning is hindered by the tremendous labor required to collect millions of diverse labeled images.
- The work targets learning from large-scale weakly labeled data and generalizing the learned model across domains.
- The proposed CNN uses two convolutional layers and several fully connected layers to predict visual sentiment labels.
- Progressive training and domain transfer fine-tune the network to address machine-generated labels, while a Mechanical Turk dataset supports evaluation and future research.
Related Work
Prior sentiment-analysis research addressed textual social-media data, while visual sentiment work used pixel-level, visual-entity, or attribute features. CNNs provide a learned-feature framework that this paper applies to visual sentiment analysis.
- Social-media sentiment analysis must handle huge data volumes, informal short messages, and multimodal expression through text, images, and videos.
- Earlier visual sentiment methods used pixel-level features or visual entities and attributes to represent image sentiment.
- CNNs combine convolutional and fully connected layers, with parameters learned through back-propagation in supervised training.
- The paper uses CNNs to learn features useful for visual sentiment analysis, building on their success in image-related tasks.
Visual Sentiment Analysis
Visual sentiment analysis is difficult because sentiment classes contain diverse images, subjective labels, and noisy training instances. The proposed Progressive CNN addresses these challenges by training a CNN and progressively filtering weakly labeled data.
- The proposed framework develops a CNN architecture for visual sentiment analysis and uses progressive training to filter noisy instances.The CNN is trained on the original data before being fine-tuned on a selected subset.
- Visual sentiment analysis is challenging because sentiment classes contain diverse images and people may assign different sentiments to the same image.
- The CNN learns parameters by maximizing conditional log likelihood over image feature vectors and sentiment labels.
- Algorithm 1 trains a CNN, predicts sentiment scores, deletes selected images, fine-tunes on the remaining subset, and returns the PCNN.
- Progressive training removes instances probabilistically according to the similarity of their predicted sentiment scores for the two classes.Instances with similar scores are more likely to be removed, while those with sufficiently different scores are retained.
Experiments
The experiments evaluate CNN architectures and progressively trained models on weakly labeled Flickr images, then test domain transfer to manually labeled Twitter images. PCNN improves over CNN and baseline methods, including after fine-tuning with limited Twitter data.
- Flickr experiments: Half a million weakly labeled Flickr images are used for training, with 90% for training and 10% for testing.The Flickr labels come from 1200 adjective-noun pairs, and training uses 300,000 iterations before progressive fine-tuning.
- CNN architecture: The selected CNN architecture uses two convolutional layers and several fully connected layers, and performs slightly better than alternatives on F1 and accuracy.The experiments therefore focus on the architecture shown in Figure 2.
- Flickr experiments: PCNN outperformed CNN in Precision, Recall, F1, and Accuracy on the randomly selected Flickr testing data.The results and first-layer filters suggest progressive fine-tuning helps the network search for a better local optimum.
- Twitter dataset: The Twitter evaluation uses 1269 manually labeled images, with five Amazon Mechanical Turk workers assigned to each image.Only 153 images had significant disagreement between workers in the reported labeling process.
- Twitter evaluation: PCNN performs better than CNN on all three Twitter labeling sets in both F1 and accuracy, with both models performing better on the less ambiguous images.Top-ranked examples from both neural networks are correctly predicted, although the models select different images.
- Transfer learning: Both CNN models outperform baseline algorithms on Twitter images, while fine-tuned CNN models improve significantly using four fifths of the 1269 images for domain adaptation.The baseline comparison includes methods using color, visual-word, and mid-level sentiment features; the evaluation uses five-fold cross-validation.
Conclusions
The paper presents a CNN architecture and training strategies for visual sentiment analysis, using progressive training and target-domain transfer to address noisy large-scale data. These strategies improve performance and support broader model generalizability across domains.
- The paper designs a CNN architecture and training strategies specifically for the challenging visual sentiment analysis problem.
- Progressive training and transfer learning from a small number of confidently labeled target-domain images yield notable improvements.
- Properly trained CNNs outperform classifiers based on predefined low-level features or mid-level visual attributes.
- The approach leverages larger weakly labeled training data and manually labeled testing data to improve model generalizability and confidence.
- The authors identify multimodality combining textual and visual content as a future direction for social media sentiment analysis.