Building a Large Scale Dataset for Image Emotion Recognition: The Fine Print and The Benchmark

• Unlike affective-computing approaches based on collected physiological responses, visual emotion prediction seeks to predict reactions from general visual stimuli.Source: Introduction · p. 1
• Social-network text supports semantic retrieval but has limited ability to bridge the affective gap between image pixels and human feelings; Hanjalic calls this affective content analysis.Source: Introduction · p. 1
• Following prior work, the study examines the same eight emotions—Amusement, Awe, Contentment, Excitement, Anger, Disgust, Fear, and Sadness—and its example images are labeled by five Amazon Mechanical Turk workers.Source: Introduction · pp. 1–2
• Large labeled datasets have fueled deep-learning gains elsewhere, but visual emotion lacked strong labels; You et al. (You et al. 2015) used weak labels for binary sentiment, not eight-class prediction.Source: Introduction · p. 2
• They collect weakly labeled emotion images and have Amazon Mechanical Turk workers manually label them, creating a dataset intended to enable CNN training and be released to researchers.Source: Introduction · p. 2
• The authors report that CNNs significantly outperform state-of-the-art manually crafted visual features for visual emotion analysis, establishing a baseline for future research.Source: Introduction · p. 2

Related Work

• The authors report no prior CNN work on visual emotion analysis and identify the lack of well-labeled data as a barrier to training deep networks.Their work aims to provide such a dataset and test a widely used deep convolutional architecture on emotion recognition.Source: Related Work · p. 2
• Visual emotion analysis used dimensional approaches (Nicolaou, Gunes, and Pantic 2011; Lu et al. 2012) or categorical approaches.The authors focus on the categorical approach, where each emotion is treated as a distinct class.Source: Related Work · p. 2
• Categorical methods used color features with social relationships and hand-tuned visual features inspired by art and psychology.Zhao et al. (2014) reported the best performance on several publicly accessible emotion datasets using features such as balance, variety, and gradation.Source: Related Work · p. 2
• These hand-tuned features had been validated on small public datasets, motivating tests of deep learning on a much larger image set.The authors frame their dataset as addressing the shortage of well-labeled data for training deep neural networks.Source: Related Work · p. 2

Visual Emotion Data Sets

• The three existing datasets are small, imbalanced, and domain-limited; some categories, such as Anger, have fewer than 10 images, risking over- or underfitting in five-fold cross-validation.These datasets are IAPS-Subset, ArtPhoto, and Abstract Paintings.Source: Visual Emotion Data Sets · pp. 2–3, Table 1 · p. 3

Building An Image Emotion Dataset from the Wild

• The dataset begins with over 3 million weakly labeled Flickr and Instagram images, then removes images tagged with multiple emotions and duplicates before verification.The emotion categories follow the eight-emotion system derived from Mikels et al. (2005).Source: Building An Image Emotion Dataset from… · p. 3
• Five qualified AMT workers verify each image with yes/no responses, and images receiving at least three yes votes are retained from a sample of 11,000 per emotion.Workers first had to answer at least half of 20 qualification questions correctly; 225 of over 1,000 workers met the criteria.Source: Building An Image Emotion Dataset from… · p. 3
• About 23,000 images, roughly 30 times ArtPhoto, form the resulting dataset, with 1,000+ images in every emotion category.The total includes an additional 2,000 Fear images to ensure that category also exceeds 1,000 images.Source: Building An Image Emotion Dataset from… · pp. 3–4, Table 2 · p. 4

Fine-tuning Convolutional Neural Network for Visual Emotion Analysis

• The authors fine-tune the ImageNet reference network by replacing its 1000-class output layer with 8 classes, using 23,000 randomly split labeled samples.They also train Noisy-Fine-tuned-CNN on 156,000 images, excluding AMT-selected images and sampling 20,000 images for other categories because Contentment has about 16,000.Source: Fine-tuning Convolutional Neural… · p. 4

Performance of Convolutional Neural Networks on Visual Emotion Analysis

• Almost 60% overall accuracy for Fine-tuned-CNN exceeds about 30% for ImageNet-CNN features and about 46% for Noisy-Fine-tuned-CNN.The noisy model learned from weak labels but did not outperform the model trained on strongly labeled samples, despite using more training samples.Source: Table 3 · p. 4, Performance of Convolutional Neural… · p. 4
• Fine-tuned-CNN has the best true negative rates in most emotion categories, while ImageNet-CNN has a higher true positive rate for Fear.Noisy-Fine-tuned-CNN's confusion matrix is more balanced except for Contentment; the authors relate true positive rates to the number of labeled images per emotion.Source: Performance of Convolutional Neural… · p. 4, Figure 3 · p. 5
• t-SNE embeddings show ImageNet-CNN grouping images by similar scenes or objects, while fine-tuned features yield more varied scenes and objects.This is consistent with the authors' observation that an object such as a dog can evoke different emotions in different states.Source: Performance of Convolutional Neural… · p. 4

Performance of Convolutional Neural Networks on Public Existing Visual Emotion Data Set

• The evaluation extracts 4096-dimensional deep features, reduces them to 20 dimensions with PCA while retaining at least 90% variance, and uses linear one-vs-all SVMs with five-fold cross-validation.Anger true-positive rates are not reported for two datasets.Source: Performance of Convolutional Neural… · pp. 5–6
• Fine-tuned-CNN features outperform manually crafted features in some emotion categories, but performance varies by category and dataset, including poor Contentment and Fear results on ArtPhoto.The paper reports that Fine-tuned-CNN is relatively more consistent than ImageNet-CNN and Noisy-Fine-tuned-CNN, while visual emotion analysis remains challenging.Source: Performance of Convolutional Neural… · pp. 5–6

Conclusions

• Deep CNN features outperform state-of-the-art hand-tuned features for visual emotion analysis, and emotion-data fine-tuning further improves performance.The authors caution that these results are only a start for research using deep learning and other learning frameworks.Source: Conclusions · p. 6
• The authors introduce a large-scale dataset and intend to release it to promote research on visual emotion analysis with deep learning and other frameworks.They plan to submit an additional 1 million AMT-labeled images.Source: Conclusions · p. 6
• Future directions include understanding how emotional arousal relates to visual stimuli and extending that work to valence.The authors also hope their results encourage research on online user-generated multimedia content in the wild.Source: Conclusions · p. 6