PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Multi-level Deep Representations for Image Emotion Classification

Tianrong Rao, Min Xu, Dong Xu

arXiv:1611.07145v2cs.CV

TL;DR

Image emotion classification must account for semantics, aesthetics, and low-level visual features, whereas existing handcrafted or deep-feature methods often focus on only part of this information. MldrNet combines multi-level representations from global and local views, and experiments on Internet images and abstract paintings report improved performance over state-of-the-art methods, including at least a 6% overall-accuracy gain.

  • Problem

    Existing image emotion-classification methods do not fully integrate semantic, aesthetic, and low-level visual information.

  • Method

    MldrNet fuses multi-level deep representations from global and local views through a CNN with side branches and a fusion layer.

  • Results

    MldrNet consistently improves emotion-classification accuracy across Internet-image and abstract-painting datasets, including at least a 6% overall-accuracy improvement over state-of-the-art methods.

  • Takeaways & Limitations

    The multi-level approach is especially useful for image types whose emotions depend on aesthetic or low-level visual cues, and it remains robust with noisy Internet labels.

Abstract

from arXiv · show

In this paper, we propose a new deep network that learns multi-level deep representations for image emotion classification (MldrNet). Image emotion can be recognized through image semantics, image aesthetics and low-level visual features from both global and local views. Existing image emotion classification works using hand-crafted features or deep features mainly focus on either low-level visual features or semantic-level image representations without taking all factors into consideration. The proposed MldrNet combines deep representations of different levels, i.e. image semantics, image aesthetics, and low-level visual features to effectively classify the emotion types of different kinds of images, such as abstract paintings and web images. Extensive experiments on both Internet images and abstract paintings demonstrate the proposed method outperforms the state-of-the-art methods using deep features or hand-crafted features. The proposed approach also outperforms the state-of-the-art methods with at least 6% performance improvement in terms of overall classification accuracy.

I. INTRODUCTION

Image emotion depends on high-level semantics, mid-level aesthetics, and low-level visual features, but existing handcrafted and CNN methods do not capture all factors effectively. MldrNet addresses these challenges by fusing multi-level representations from global and local views.

  • I. INTRODUCTION: Image emotion reflects complex high-level, mid-level, and low-level visual features from both global and local views.Examples include semantics, composition and emphasis, and color, shape, line, and texture.
  • I. INTRODUCTION: Handcrafted methods cannot comprehensively consider image semantics, image aesthetics, and low-level visual features together.
  • I. INTRODUCTION: AlexNet handles web-image semantics better than the mid-level and low-level features emphasized by abstract paintings.The paper attributes this limitation to AlexNet's focus on high-level semantic representations.
  • I. INTRODUCTION: MldrNet extracts different representation levels through side branches and integrates them with a fusion layer for classification.The network is designed to learn representations from both global and local views.
  • I. INTRODUCTION: Experiments on publicly available web-image and abstract-painting datasets compare MldrNet with existing methods and examine noisy labels.The paper reports improved emotion classification relative to state-of-the-art handcrafted- and deep-feature methods.

II. RELATED WORK

Prior visual emotion-classification research uses dimensional or categorical emotion models and features spanning multiple visual levels. CNN methods provide learned representations, motivating MldrNet's unified multi-level architecture.

  • II. RELATED WORK: Visual emotion classification research is commonly divided into dimensional emotion-space and categorical-emotion-state approaches.
  • II. RELATED WORK: Earlier methods extract low-level features such as color, shape, texture, and Gabor descriptors for emotion classification.
  • II. RELATED WORK: Other methods use mid-level aesthetic features, including composition, visual balance, and emphasis, to represent emotions in artistic images.
  • II. RELATED WORK: CNN-based methods learn image representations without manually designing visual features and have been applied to visual emotion analysis.
  • II. RELATED WORK: MldrNet unifies high-level semantics, mid-level aesthetics, and low-level visual features within one CNN structure using a fusion layer.

A. Convolutional Neural Network

A conventional CNN transforms an image through convolutional and fully connected layers, then predicts an emotion distribution with a softmax layer. AlexNet primarily captures high-level semantics and is less suited to paintings whose emotions depend on aesthetic and low-level features.

  • A. Convolutional Neural Network: A CNN extracts layer-wise image representations through convolutional and fully connected layers before softmax prediction over eight emotion categories.
  • A. Convolutional Neural Network: The softmax output represents a probability distribution over the possible image-emotion categories.
  • A. Convolutional Neural Network: AlexNet contains five convolutional layers, max-pooling layers, and three fully connected layers ending in eight output neurons.
  • A. Convolutional Neural Network: AlexNet is mainly trained for semantic-level classification and therefore does not effectively capture aesthetic and low-level emotion information in abstract paintings.

B. Analysis of different CNN models

CNN depth corresponds broadly to representation level, while different specialized CNNs use related structures to capture semantics, aesthetics, or texture. MldrNet shares these ideas in one multi-branch network and fuses representations across levels.

  • B. Analysis of different CNN models: Deeper CNN layers generally encode higher-level features, while lower layers retain color, texture, composition, and visual-balance information.
  • B. Analysis of different CNN models: A-CNN targets image aesthetics with fewer convolutional layers than AlexNet, while T-CNN is designed to extract texture representations.
  • B. Analysis of different CNN models: Related CNN models differ mainly in convolutional-layer count, allowing MldrNet to share parameters while extracting multiple representation levels.
  • B. Analysis of different CNN models: MldrNet uses a main network and four side branches to extract global and local representations from different convolutional layers.
  • B. Analysis of different CNN models: The architecture must balance depth against computational cost because deeper models may add parameters without materially improving emotion classification.
  • B. Analysis of different CNN models: Fusion choices include concatenation, minimum, maximum, and mean because representation levels may contribute differently across image types.

D. Fusion Layer

The fusion layer aggregates deep representations from different MldrNet layers because emotion-relevant features can have different importance. It evaluates fusion functions that assign equal or selective emphasis across layers.

  • D. Fusion Layer: The fusion layer combines multi-level deep representations using fusion functions rather than treating all layer features as equally important.This design addresses the differing impact of feature levels on image emotion analysis.
  • D. Fusion Layer: The entire-image representation is aggregated from layer representations h_i through a fusion function f(x).The paper defines h_i as the representation extracted from the ith layer.
  • D. Fusion Layer: Mean assigns equal weight to each convolutional-layer representation, whereas max and min emphasize one layer’s representation.The experiments compare these fusion choices because the function critically affects the method.
  • D. Fusion Layer: The evaluation covers a large-scale emotion dataset, three small datasets, and MART for binary abstract-painting classification.The datasets evaluate eight emotion categories or, for MART, positive versus negative emotions.

A. Experimental Settings

The experiments use one large-scale emotion dataset and several smaller datasets spanning natural images, art photographs, abstract paintings, and binary abstract-painting labels. The materials include noisy and manually labeled data and established affective-image benchmarks.

  • A. Experimental Settings: The large-scale dataset contains noisy-labeled images collected from Instagram and Flickr using eight emotion-category names.The categories include five positive and three negative emotions.
  • A. Experimental Settings: IAPS-Subset contains 395 natural images mapped from arousal and valence values to eight discrete emotion categories.The images depict documentary-style scenes such as portraits, animals, babies, and landscapes.
  • A. Experimental Settings: The well-labeled large-scale data include 23,164 manually labeled images, while 88,298 images remain noisy labeled.The counts reflect the available images after accounting for images no longer existing online.
  • A. Experimental Settings: ArtPhoto contains 806 art photographs whose emotion categories were determined by the artists who created and uploaded them.The artists intentionally manipulate emotional objects, lighting, and colors to evoke emotions.
  • A. Experimental Settings: Abstract contains 228 paintings whose emotion categories were selected from the category receiving the most votes among 14 viewers.These paintings express emotions through overall color and texture rather than emotional objects.
  • A. Experimental Settings: MART contains 500 abstract paintings labeled positive or negative using a relative-score method.The works come from the Museum of Modern and Contemporary Art of Trento and Rovereto.

3) Compared Methods:

The comparison includes hand-crafted feature methods, AlexNet-based approaches, deeper pretrained CNNs, and MldrNet fusion variants. The variants isolate concatenation, max, min, and mean fusion functions.

  • 3) Compared Methods:: The experiments compare MldrNet with state-of-the-art image-emotion methods and popular CNN models.This comparison spans both hand-crafted and deep-feature approaches.
  • 3) Compared Methods:: Machajdik, Zhao, and Rao represent hand-crafted approaches using art-theory features, principles-of-art features, or multi-scale image-patch features.Their feature designs target visual properties associated with image emotion.
  • 3) Compared Methods:: AlexNet+SVM extracts emotion-related deep features with AlexNet and classifies them using an SVM.The method separates feature extraction from classification.
  • 3) Compared Methods:: AlexNet, VGGNet-19, and ResNet-101 are pretrained on ImageNet and fine-tuned on the large-scale emotion dataset.These models provide CNN baselines based on different network architectures.
  • 3) Compared Methods:: MldrNet variants test concatenation, max, min, and mean as fusion functions.These variants are designed to quantify fusion-function effects and identify a suitable architecture.

B. Emotion Classification on Large Scale and Noisy Labeled Dataset

The layer-depth experiment evaluates how the number of convolutional layers affects MldrNet’s emotion-classification accuracy and computational cost. Four layers are selected because shallower models perform worse, while deeper models add little accuracy and increase training time.

  • B. Emotion Classification on Large Scale and Noisy Labeled Dataset: The large-scale experiment uses well-labeled images split into training, testing, and validation sets and also constructs a noisy-labeled training dataset.The well-labeled split uses 80% training, 15% testing, and 5% validation data.
  • B. Emotion Classification on Large Scale and Noisy Labeled Dataset: Four convolutional layers are selected as the best MldrNet configuration for subsequent experiments.Models with fewer layers perform worse, while models with more than four layers provide little additional accuracy.
  • B. Emotion Classification on Large Scale and Noisy Labeled Dataset: Fewer-layer models lack high-level image-feature information needed for stronger emotion classification.The paper identifies insufficient high-level information as the main reason for their weaker performance.
  • B. Emotion Classification on Large Scale and Noisy Labeled Dataset: Adding more than four layers does not significantly improve emotion-classification accuracy but increases the number of computed parameters and training time.The configuration choice balances classification performance against computational cost.

2) Choice of Fusion Function:

MldrNet evaluates fusion functions for combining multi-level representations and finds mean pooling most effective across well and noisy training data. Compared with hand-crafted and conventional deep models, it improves classification and remains robust to noisy labels.

  • Choice of Fusion Function: Mean pooling achieves the best MldrNet performance across both well and noisy training datasets.Compared with max pooling, min pooling, and concatenation, mean pooling preserves more emotional information from convolutional layers.
  • Comparison of Different Methods: Deep-representation methods outperform hand-crafted features, while MldrNet significantly improves accuracy despite using only four convolutional layers.VGGNet-19 and ResNet-101 provide only slight gains over AlexNet, whereas MldrNet incorporates mid-level and low-level representations.
  • Comparison of Different Methods: MldrNet remains competitive when trained on noisy data, supporting use of images collected directly from the Internet.The authors connect this robustness to applications including recommender systems, social networks, and personalized advertising.
  • Comparison of Different Methods: Across most emotional categories, MldrNet outperforms AlexNet and is more robust across different training datasets.AlexNet particularly confuses fear and sadness, whereas MldrNet extracts emotional information despite false labels.
  • Comparison of Different Methods: Images misclassified by AlexNet often convey emotion through color, texture, and aesthetics rather than semantics alone.Combining mid-level and low-level representations improves emotion classification accuracy.

C. Emotion Classification on small Scale Datasets

On the small-scale IAPS-Subset, ArtPhoto, and Abstract datasets, deep methods generally outperform hand-crafted approaches. MldrNet achieves the best performance in almost all emotion categories and remains robust where emotions depend strongly on mid-level and low-level features.

  • Emotion Classification on small Scale Datasets: The evaluation uses one-against-all classifiers and 5-fold cross-validation because emotion categories are imbalanced and contain limited images.True positive rate per class is used for comparison, with anger not reported for IAPS-Subset and Abstract.
  • Emotion Classification on small Scale Datasets: Deep learning methods significantly outperform state-of-the-art hand-crafted methods for most emotion categories.The evaluation reports per-category emotion classification accuracies for the three datasets.
  • Emotion Classification on small Scale Datasets: MldrNet achieves the best performance in almost all emotion categories across IAPS-Subset, ArtPhoto, and Abstract.The authors characterize this as a robust result across the three datasets.
  • Emotion Classification on small Scale Datasets: AlexNet performs relatively poorly on Abstract and ArtPhoto, where emotions are mainly conveyed by mid-level and low-level visual features.These datasets therefore expose limitations of relying primarily on high-level semantic representations.

D. Emotion Classification on Abstract Paintings

For abstract-painting emotion classification, MldrNet combines representations from different convolutional layers through a dedicated fusion design. The paper concludes that this multi-level approach consistently improves accuracy with fewer convolutional layers and is robust to noisy training data.

  • Emotion Classification on Abstract Paintings: MldrNet unifies high-level semantics, mid-level aesthetics, and low-level visual features within one CNN structure.A fusion layer aggregates representations from different levels according to image-emotion characteristics.
  • Emotion Classification on Abstract Paintings: MldrNet achieves consistent accuracy improvements with fewer convolutional layers than popular CNN models across different emotion datasets.The conclusion presents this as evidence for the value of multi-level representations.
  • Emotion Classification on Abstract Paintings: MldrNet is more robust when trained on noisy Internet-collected data, reducing the demand for reliable training data.The authors link this robustness to using larger collections of images.
  • Emotion Classification on Abstract Paintings: The authors propose extending MldrNet to other abstract-level vision tasks, including image aesthetics analysis and photo quality assessment.This is stated as a future research direction rather than an evaluated result.
Loading 1611.07145v2…