Source-linked AI summary
DeepSentiBank: Visual Sentiment Concept Classification with Deep Convolutional Neural Networks
Tao Chen, Damian Borth, Trevor Darrell, Shih-Fu Chang
TL;DR
The paper addresses the challenge of classifying visual sentiment concepts from noisy, biased web images, using ANPs as mid-level cues. It trains an ImageNet-initialized deep CNN, DeepSentiBank, on Flickr-derived ANP data. DeepSentiBank significantly improves annotation accuracy and retrieval performance over earlier SVM-based models.
Problem
Modeling visual sentiment is difficult because adjectives are separated from image-level features by an affective gap, while available Flickr labels are noisy and sentiment data is biased toward strong sentiment.
Method
DeepSentiBank represents sentiment with ANPs and trains a deep CNN on Flickr images, initializing its weights from ImageNet to address insufficient, biased training data.
Results
DeepSentiBank significantly improves ANP annotation accuracy and retrieval performance compared with predecessor models based mainly on binary SVM classifiers.
Takeaways & Limitations
ANPs provide statistical cues for detecting emotions in images, and the improved model supports applications built on SentiBank, including assistive commenting and Twitter sentiment prediction.
Abstract
from arXiv · showhide
This paper introduces a visual sentiment concept classification method based on deep convolutional neural networks (CNNs). The visual sentiment concepts are adjective noun pairs (ANPs) automatically discovered from the tags of web photos, and can be utilized as effective statistical cues for detecting emotions depicted in the images. Nearly one million Flickr images tagged with these ANPs are downloaded to train the classifiers of the concepts. We adopt the popular model of deep convolutional neural networks which recently shows great performance improvement on classifying large-scale web-based image dataset such as ImageNet. Our deep CNNs model is trained based on Caffe, a newly developed deep learning framework. To deal with the biased training data which only contains images with strong sentiment and to prevent overfitting, we initialize the model with the model weights trained from ImageNet. Performance evaluation shows the newly trained deep CNNs model SentiBank 2.0 (or called DeepSentiBank) is significantly improved in both annotation accuracy and retrieval performance, compared to its predecessors which mainly use binary SVM classification models.
1. INTRODUCTION
The paper addresses visual sentiment analysis by representing difficult-to-model sentiment concepts with ANPs and classifying them using deep CNNs. DeepSentiBank uses ImageNet initialization to handle biased, limited sentiment data and improves annotation and retrieval over earlier models.
- Visual sentiment analysis is motivated by the growing volume of social multimedia and its applications in communication, education, advertising, and entertainment.
- Modeling sentiment-related adjectives is difficult because of the affective gap between low-level visual features and high-level sentiment.
- Adjective Noun Pairs combine adjectives’ sentimental strength with nouns’ visual detectability and provide statistical cues for detecting depicted emotions.
- The training data contains about one million Flickr images across thousands of sentiment-concept categories, motivating a high-capacity CNN approach.
- DeepSentiBank initializes CNN weights from ImageNet and significantly improves ANP annotation accuracy while moderately improving retrieval over predecessor SVM models.
2. RELATED WORK
Related work spans text-based sentiment analysis, visual sentiment ontologies, broader visual concept modeling, deep CNNs, and transferable representations. The paper builds on these directions to apply deep learning to visual sentiment concepts.
- Most prior sentiment-analysis research uses textual information for applications such as human behavior prediction, business, and political science.
- Visual sentiment modeling has been less studied than text-based sentiment analysis, with prior work using ANP ontologies, one-vs-all SVMs, localization, and semantic similarity.
- Multimedia and computer vision research models objects, scenes, activities, fine-grained categories, aesthetics, memorability, interestingness, and emotions.
- Deep convolutional networks achieved successful results on digit recognition and later on ImageNet-scale datasets containing more than one million images.
- Supervised pre-training followed by domain-adaptive fine-tuning is described as an efficient paradigm for learning with scarce data.
3. VISUAL SENTIMENT ONTOLOGY AND CONCEPTS OVERVIEW
The paper uses a Flickr-based Visual Sentiment Ontology of ANPs as the basis for visual sentiment concept classification. It filters noisy, sparse categories and creates a manually annotated subset for retrieval comparisons.
- The Visual Sentiment Ontology was constructed as a database for sentiment analysis on visual content and contains more than 3,000 ANPs.
- Filtering removes ANPs with fewer than 120 images, leaving 2,089 ANPs and 867,919 images for classification experiments.
- Flickr tags serve as labels, but may be incomplete or falsely assigned, so the study treats them as pseudo ground truth.
- The retrieval subset covers six nouns and 135 ANPs, with 60 manually annotated test images per ANP comprising 20 positives and 40 negatives.
4. DEEP CONVOLUTIONAL NEURAL NETWORKS SOLUTION
DeepSentiBank uses an eight-layer CNN architecture with a 2089-way output, trained with augmentation, regularization, dropout, and ImageNet-initialized weights. Caffe provides the implementation framework.
- 4.1 Introduction of Caffe: Caffe is a deep learning framework emphasizing cleanliness, readability, and speed.
- 4.1 Introduction of Caffe: Caffe’s configuration-based networks and CPU/GPU switching support rapid deployment and training across GPU machines and commodity clusters.
- 4.2 CNN Architecture: The CNN contains eight weighted layers—five convolutional and three fully connected—followed by a 2089-way softmax over class labels.
- 4.2 CNN Architecture: Response normalization follows the first two pooling layers, and dropout is applied to the first two fully connected layers.
- 4.2 CNN Architecture: Training uses random 227 × 227 patches and horizontal reflections extracted from normalized 256 × 256 images to reduce overfitting.
- 4.2 CNN Architecture: ImageNet-pretrained weights are used except for the top layer because the sentiment dataset is insufficient and biased toward strong sentiment.
5. EXPERIMENTAL RESULTS
The experiments evaluate DeepSentiBank using annotation accuracy and retrieval performance, comparing it with alternative models across ANP concepts and noun categories.
- Annotation accuracy: Figure 2 compares smoothed ranked top-10 accuracy per ANP across different approaches.The comparison concerns fine-tuned DeepSentiBank, a non-fine-tuned deep CNN, and SentiBank 1.1.
- Annotation accuracy: Annotation accuracy is measured as the percentage of images whose pseudo ground-truth label appears among the top-k detected concepts.Top-1, top-5, and top-10 accuracies are computed for individual and all ANPs.
- Qualitative examples: Figure 3 shows the top 10 concepts detected from test images by fine-tuned DeepSentiBank, marking pseudo ground-truth concepts in red.The examples provide a qualitative view of concept detections.
- Retrieval performance: Figure 4 reports mean average precision for each noun category and for all categories in the 135-ANP retrieval subset.The figure organizes retrieval performance by noun category.
- Retrieval performance: Retrieval performance is measured by average precision at top 20 on a subset of 135 ANPs.The subset covers six nouns and uses manually annotated positive and negative test images.
- Retrieval performance: DeepSentiBank outperforms SentiBank 1.1 by 62.3% and SentiBank 1.5R by 8.9% in retrieval performance.Although not designed specifically for retrieval, DeepSentiBank achieves these improvements on the 135-ANP subset.
6. CONCLUSION
The paper presents DeepSentiBank, a deep CNN visual sentiment concept classifier initialized with ImageNet weights. It reports improved annotation and retrieval performance over earlier binary-SVM-based approaches, while identifying localization and concept relations as future directions.
- Conclusion: The paper presents a visual sentiment concept classification model based on deep convolutional neural networks.The model is trained using Caffe.
- Conclusion: ImageNet initialization addresses biased strong-sentiment training data and helps prevent overfitting.The model weights trained from ImageNet are used to initialize the deep CNN.
- Conclusion: DeepSentiBank significantly improves annotation and retrieval performance compared with previous approaches using independent binary SVM classifiers.The conclusion also identifies concept localization and concept relations as directions for future model improvements.