Source-linked AI summary
Visual Sentiment Prediction with Deep Convolutional Neural Networks
Can Xu, Suleyman Cetintas, Kuang-Chih Lee, Li-Jia Li
TL;DR
Visual sentiment analysis has received limited attention compared with text analysis, despite images conveying emotion in social posts. The paper transfers a CNN pretrained for image recognition to sentiment prediction, and reports that its models outperform state-of-the-art methods on Twitter and Tumblr.
Problem
Sentiment analysis has largely focused on text, while the emotional content of images in social posts remains relatively unexplored.
Source: §1 · p. 1Method
The framework transfers a CNN pretrained on large-scale image-classification data and uses its image representations for sentiment prediction.
Source: §1 · p. 1, §3.3 · p. 3Results
The proposed models outperform state-of-the-art methods on both the Twitter and Tumblr datasets.
Source: §6 · p. 6Takeaways & Limitations
Abstract
from arXiv · showhide
Images have become one of the most popular types of media through which users convey their emotions within online social networks. Although vast amount of research is devoted to sentiment analysis of textual data, there has been very limited work that focuses on analyzing sentiment of image data. In this work, we propose a novel visual sentiment prediction framework that performs image understanding with Deep Convolutional Neural Networks (CNN). Specifically, the proposed sentiment prediction framework performs transfer learning from a CNN with millions of parameters, which is pre-trained on large-scale data for object recognition. Experiments conducted on two real-world datasets from Twitter and Tumblr demonstrate the effectiveness of the proposed visual sentiment analysis framework.
1 Introduction
Sentiment analysis has largely focused on text, while social posts increasingly convey emotion through images with little or no text. This work predicts sentiment from visual information using transferred CNN representations and introduces a five-scale Tumblr image benchmark.
1 Introduction
- Visual sentiment analysis remains relatively unexplored, although many social-network photo and video posts contain little or no text and convey emotion visually.Existing work has concentrated on opinion mining and sentiment analysis of post text.Source: §1 · p. 1
- The framework transfers a CNN trained on large-scale image data to visual sentiment prediction, and its activations outperform state-of-the-art methods by a large margin.The approach does not require psychology or linguistics domain knowledge to define features, simplifying training for production deployment.Source: §1 · p. 1
- The paper introduces a five-scale Tumblr image sentiment rating that includes neutral images and different strengths of sentiment polarity.The authors analyze and validate the labeling scheme using statistics from multiple annotators.Source: §1 · p. 2
2 Related Work
Prior visual-sentiment work developed mid-level attributes and compared image descriptors, while deep CNNs and transfer learning offered a route to learned representations without extensive target-task data.
2 Related Work
- Prior visual sentiment methods used mid-level attributes, including scene and facial-expression cues or detectors of selected visual concepts.Jiang, Xu, and Xue (2014) compared low-level descriptors and mid-level attributes, finding semantic-level clues effective for visual sentiment classification.Source: §2 · p. 2
- These attribute-based approaches require psychology or linguistics knowledge to define features and human intervention to tune sentiment predictions.Source: §2 · p. 2
- Large-scale CNNs can learn deep image representations, and prior transfer-learning studies applied them to recognition and scene classification when target data were limited.This work applies that transfer-learning idea to sentiment prediction.Source: §2 · p. 2
3 Method
The method trains a CNN for image classification, transfers its parameters to generate image representations for sentiment prediction, and fits classifiers to those features.
3 Method
- The framework trains a CNN on large-scale image data, transfers its learned parameters to sentiment prediction, and trains classifiers on the resulting image-level features.Source: §3 · p. 2
3.1 Deep convolutional neural networks
- The CNN uses five convolutional layers followed by two fully connected hidden layers, with ReLU nonlinearities and a final softmax output.It takes 224 × 224 RGB images as input.Source: §3.1 · p. 2
3.2 Network training
- The CNN is trained on ILSVRC-2012, a subset of ImageNet, before its parameters are transferred to sentiment prediction.The architecture figure depicts parameter transfer from ImageNet classification and the use of fc7 and fc8 activations as sentiment features.Source: §3.2 · pp. 2–3, Figure 2 · p. 3
3.3 Transfer learning
- Transfer learning uses either fixed 4096-dimensional activations from the seventh layer or a 1000-dimensional distribution over ImageNet object categories as image representations.The latter representation retains the softmax classifier output and is treated as an attribute descriptor associated with objects in the image.Source: §3.3 · p. 3
3.4 Classification
- Separate logistic-regression classifiers are trained on the 4096-dimensional seventh-layer activations and 1000-dimensional softmax responses.Source: §3.4 · p. 3
4 Dataset
The evaluation uses real-world Twitter and Tumblr datasets, including a proprietary Tumblr collection of photo posts. Tumblr posts receive five-level sentiment labels, with high exact-label agreement among annotators.
4 Dataset
- The evaluation uses a public Twitter benchmark and a proprietary Tumblr dataset collected for this work.The reported methods were evaluated on both real-world microblogging datasets.Source: §4 · pp. 3–4
4.1 Twitter
- The Twitter benchmark contains 470 positive and 133 negative image tweets labeled using combined image-text inspection.The benchmark was introduced in prior work.Source: §4.1 · p. 4
4.2 Tumblr
- Tumblr contributes the authors’ first photo-post visual-sentiment dataset; Figure 3 pairs example images and tags/text with image-text sentiment-strength labels.Source: §4.2 · p. 4, Figure 3 · p. 4
Data collection
- Tumblr tags were screened with an 8,222-word subjectivity lexicon for strongly polarized terms; 1,179 photo posts were sampled, and reported data and results are deliberately incomplete and anonymized.Source: Data collection · p. 4
Ground truth labeling
- Under the five-level sentiment scheme, exact-label agreement was 94%, yielding 1,110 images with image-text combined ground truth from 1,179 images; the authors describe this agreement as high given the scheme’s granularity.Five annotators rated each image and its associated tags, with each assigned to exactly three annotators; an image was valid when at least two agreed on the exact label.Source: Ground truth labeling · pp. 4–5
5 Experiments
The experiments evaluate visual sentiment methods on Twitter and Tumblr using average AUC across five runs and dataset partitions. The transferred CNN methods outperform both baselines on both datasets, with the larger margin on Twitter.
Low-level visual features
- The Low-level features baseline combines RGB color histograms, GIST, LBP, and a spatial-pyramid Bag-of-Words descriptor.The Bag-of-Words descriptor uses a 1,000-word dictionary with max pooling over a two-layer spatial pyramid.Source: Low-level visual features · p. 5
SentiBank
- SentiBank represents images with 1,200 adjective-noun concepts, whose detectors are trained on Flickr images.Its ontology draws on psychology studies and web mining.Source: SentiBank · p. 5
5.2 Evaluation Metric
- AUC measures classifier discrimination independently of decision criteria and is less sensitive to class imbalance than accuracy.The authors use average AUC over five independent runs with five dataset partitions.Source: §5.2 · p. 5
5.3 Experimental Results
- SentiBank outperforms Low-level features on both datasets, with a larger performance difference on Tumblr than Twitter.The authors attribute this to noisier Twitter data and SentiBank benefiting more from cleaner data.Source: §5.3 · p. 5
- Both fc7 and fc8 outperform Low-level features and SentiBank on Twitter and Tumblr, with a larger margin on Twitter.Table 3 reports the Tumblr comparison using AUC.Source: §5.3 · pp. 5–6, Table 3 · p. 6
- On Twitter, fc7 outperforms fc8, while their results are comparable on Tumblr, where images are cleaner and objects easier to identify.The authors suggest the 7th-layer activations characterize more image aspects than 8th-layer object-detection scores.Source: §5.3 · p. 6
6 Conclusions
The paper transfers large-scale CNN image representations to visual sentiment prediction and reports that its models outperform state-of-the-art methods on Twitter and Tumblr. It also constructs a Tumblr benchmark with five-scale sentiment ratings and identifies directions for further work.
6 Conclusions
- The framework transfers image representations from a CNN trained on a large-scale dataset for sentiment analysis.The authors report that the proposed models outperform state-of-the-art methods on Twitter and Tumblr.Source: §6 · p. 6
- The Tumblr photo-post benchmark uses five-scale sentiment ratings, and the experiments report stronger performance than state-of-the-art methods on Twitter and Tumblr.The rating scale is described as more comprehensive than bipolar labeling.Source: §6 · p. 6
- Future directions include semisupervised domain-specific fine-tuning of CNN fully connected layers, applications in recommendation, advertising, and games, and extension to video.Source: §6 · p. 6