Source-linked AI summary
Building a Large Scale Dataset for Image Emotion Recognition: The Fine Print and The Benchmark
Quanzeng You, Jiebo Luo, Hailin Jin, Jianchao Yang
TL;DR
Visual emotion analysis lacked the large, confidently labeled datasets that had supported deep learning in other vision tasks. The paper constructs such a dataset from weakly labeled web images and evaluates CNN-based features against hand-tuned alternatives, finding that deep CNN features outperform the state-of-the-art hand-tuned features while fine-tuned networks are more consistent across categories and datasets.
Problem
Visual emotion analysis lacked a large, well-labeled dataset for evaluating CNNs and training deep models.
Method
The authors collect weakly labeled emotion images, manually verify them with Amazon Mechanical Turk, and benchmark differently trained CNN features against hand-crafted features.
Results
Deep CNN features outperform state-of-the-art hand-tuned features, while fine-tuned CNN features perform more consistently across emotion categories and datasets.
Takeaways & Limitations
The dataset and benchmark provide a starting point for research using deep learning and other learning frameworks in visual emotion analysis.
Takeaways & Limitations
The performance of deep visual features is not consistent across emotion categories, and the authors plan to collect additional labeled data.
Abstract
from arXiv · showhide
Psychological research results have confirmed that people can have different emotional reactions to different visual stimuli. Several papers have been published on the problem of visual emotion analysis. In particular, attempts have been made to analyze and predict people's emotional reaction towards images. To this end, different kinds of hand-tuned features are proposed. The results reported on several carefully selected and labeled small image data sets have confirmed the promise of such features. While the recent successes of many computer vision related tasks are due to the adoption of Convolutional Neural Networks (CNNs), visual emotion analysis has not achieved the same level of success. This may be primarily due to the unavailability of confidently labeled and relatively large image data sets for visual emotion analysis. In this work, we introduce a new data set, which started from 3+ million weakly labeled images of different emotions and ended up 30 times as large as the current largest publicly available visual emotion data set. We hope that this data set encourages further research on visual emotion analysis. We also perform extensive benchmarking analyses on this large data set using the state of the art methods including CNNs.
Introduction
Visual emotion analysis seeks to predict emotional reactions from visual content, but progress has been limited by the lack of large, confidently labeled datasets. This work introduces a large manually labeled dataset and evaluates CNNs against established hand-crafted features.
- Visual emotion prediction infers emotional reactions from general visual stimuli rather than physiological signals collected from people.
- Social-network text supports semantic retrieval but does little to bridge the affective gap between image pixels and human feelings.
- Hand-crafted color, texture, composition, and art-inspired features had shown promise on emotion-analysis datasets.
- Large labeled datasets have supported deep-learning progress in other vision tasks, whereas visual emotion analysis lacked a comparable strongly labeled resource.
- The authors build a large emotion dataset, manually label weakly labeled images with Amazon Mechanical Turk, and release the dataset to the research community.
- CNN features outperform state-of-the-art manually crafted features, establishing CNN performance as a baseline for future visual emotion analysis.
Related Work
Prior visual emotion research used categorical or dimensional representations and mainly relied on hand-tuned features. The paper positions CNNs as an untested alternative whose evaluation requires a much larger, well-labeled dataset.
- Visual emotion analysis has been organized around dimensional representations or categorical emotion classes.
- Existing categorical methods use color, texture, composition, content, social relationships, and art-inspired properties such as balance, variety, and gradation.
- These hand-tuned features were validated on several small publicly available datasets, while well-labeled large-scale data for training deep networks remained unavailable.
- The paper therefore tests whether deep learning can address visual emotion analysis at a much larger scale than prior work.
Visual Emotion Data Sets
Earlier emotion datasets were small, imbalanced, and drawn from narrow domains, limiting training and evaluation. The authors construct a larger resource from weakly labeled web images, verify labels with workers, and obtain about 23,000 images across eight emotions.
- Existing datasets were small, imbalanced across eight categories, and drawn from a few specific domains, with some categories containing fewer than 10 images.
- The limited samples in prior datasets could leave only several training images under five-fold cross-validation, risking overfitted or underfitted models.
- The authors query Flickr and Instagram with eight emotion keywords to collect over 3 million weakly labeled images, then remove cross-emotion tags and duplicates.
- Amazon Mechanical Turk workers verify emotion labels, with five workers assigned per image and images retained after receiving at least three Yes votes.
- The resulting dataset contains about 23,000 images, approximately 30 times the size of ArtPhoto, the current largest emotion dataset described here.
- The current labeled dataset contains more than 1,000 images in every emotion category.
Visual Emotion Analysis
The paper evaluates CNN-based visual emotion analysis using pretrained and fine-tuned models, comparing them with hand-crafted features across several datasets. Fine-tuning generally helps, but performance varies by emotion and dataset.
- Experimental setup: The study compares ImageNet-CNN, Noisy-Fine-tuned-CNN, and Fine-tuned-CNN models with established hand-crafted visual features.The evaluation uses classification accuracy and per-class true positive rates across the authors’ AMT-labeled data and three public datasets.
- AMT-labeled evaluation: about 46% overall accuracy is achieved by Noisy-Fine-tuned-CNN, but it does not outperform Fine-tuned-CNN despite using more training samples.This comparison suggests that noisily labeled images provide learnable information, while strongly labeled samples remain more effective in this evaluation.
- AMT-labeled evaluation: Fine-tuned-CNN achieves the best true negative rates in most emotion categories on the AMT testing data.The confusion-matrix analysis reports that ImageNet-CNN performs worse overall, although its fear true positive rate is higher than Fine-tuned-CNN’s.
- Feature analysis: t-SNE embeddings show ImageNet-CNN grouping images by similar scenes or objects, whereas fine-tuned features produce more diverse object and scene neighborhoods.The authors relate this difference to emotion varying even for the same object in different states, such as an angry versus cute dog.
- Public-dataset evaluation: The public-dataset benchmark evaluates three CNN models against Wang, Yanulevskaya, Machajdik, and Zhao features on IAPS-Subset, Abstract Paintings, and ArtPhoto.The figures report per-class true positive rates for these methods; anger is omitted for IAPS-Subset and Abstract Paintings because those categories contain very few images.
- Public-dataset evaluation: Deep visual features significantly outperform state-of-the-art manually crafted features in some emotion categories, but their performance is not consistent across categories and datasets.Fine-tuned-CNN is relatively more consistent, yet performs poorly for Contentment and Fear in the ArtPhoto dataset.
Conclusions
The paper introduces a large visual-emotion dataset and benchmarks deep visual features for categorical emotion analysis. Its results support CNN use and emotion-specific fine-tuning, while positioning the findings as an initial step for further research.
- Conclusions: The work introduces a large-scale dataset and evaluates deep visual features from differently trained neural-network models for visual emotion analysis.The dataset is intended for release to promote research using deep learning and other learning frameworks.
- Conclusions: Deep convolutional neural-network features outperform state-of-the-art hand-tuned features, and fine-tuning on emotion-related data can further improve performance.These are the paper’s reported experimental conclusions across its visual-emotion evaluations.
- Future directions: The authors describe the results as a start for research on deep learning and other frameworks for visual emotion analysis.They plan to continue collecting AMT-labeled data and identify understanding emotion arousal and extending analysis to valence as future directions.