PaperlayerView exact revision on arXiv

Source-linked AI summary

Island Loss for Learning Discriminative Features in Facial Expression Recognition

Jie Cai, Zibo Meng, Ahmed Shehab Khan, Zhiyuan Li, James O'Reilly, Yan Tong

arXiv:1710.03144v3cs.CV

TL;DR

Facial expression recognition degrades in real-world settings because appearance, pose, illumination, and occlusion produce high intra-class variation and inter-class similarity. The paper proposes island loss, which jointly compacts same-expression features and separates different-expression centers, and evaluates it through IL-CNN. Across four benchmark databases, IL-CNN outperforms softmax- and center-loss CNN baselines and matches or exceeds state-of-the-art methods.

  • Problem

    Real-world facial expression recognition must handle high intra-class variations and inter-class similarities caused by appearance, pose, illumination, and occlusion changes.

  • Method

    Island loss pulls samples toward their class centers while pushing different expression centers apart, and is jointly minimized with softmax loss in IL-CNN.

  • Results

    Across four facial expression databases, IL-CNN outperforms CNN baselines using softmax or center loss and achieves comparable or better performance than state-of-the-art methods.

  • Takeaways & Limitations

    Island loss enhances the discriminative power of learned facial expression features by reducing intra-class variation and increasing inter-class differences.

Abstract

from arXiv · show

Over the past few years, Convolutional Neural Networks (CNNs) have shown promise on facial expression recognition. However, the performance degrades dramatically under real-world settings due to variations introduced by subtle facial appearance changes, head pose variations, illumination changes, and occlusions. In this paper, a novel island loss is proposed to enhance the discriminative power of the deeply learned features. Specifically, the IL is designed to reduce the intra-class variations while enlarging the inter-class differences simultaneously. Experimental results on four benchmark expression databases have demonstrated that the CNN with the proposed island loss (IL-CNN) outperforms the baseline CNN models with either traditional softmax loss or the center loss and achieves comparable or better performance compared with the state-of-the-art methods for facial expression recognition.

I. INTRODUCTION

Facial expression recognition remains difficult in real-world conditions because appearance, pose, illumination, occlusion, and personal-attribute changes create high intra-class variation and inter-class similarity. The paper proposes island loss and an IL-CNN to learn compact, separated expression features, with evaluations showing stronger performance than baseline CNNs.

  • Real-world facial expression recognition is challenging because head pose, illumination, occlusions, and personal attributes increase intra-class variation and inter-class similarity.
  • Softmax loss separates expression classes but can leave within-class features scattered and different-class clusters overlapping.
  • Island loss compresses each expression cluster while pushing different class centers apart as isolated islands.
  • The IL-CNN jointly uses island loss at the second fully connected layer and softmax loss at the decision layer to fine-tune CNN parameters.
  • The proposed work introduces island loss and develops IL-CNN to learn representations with lower intra-class variation and higher inter-class distance.
  • Across four expression databases, IL-CNN outperforms CNN baselines using softmax or center loss and achieves comparable or better performance than state-of-the-art methods.

II. RELATED WORK

Facial expression recognition has used both handcrafted and learned features, with CNNs becoming prominent for challenging real-world conditions. Existing CNN approaches commonly rely on softmax supervision, motivating losses that address remaining feature variation and class similarity.

  • Facial expression recognition features are designed to capture appearance and geometric changes in static images or videos.
  • Handcrafted approaches include Gabor, SIFT, HOG, LBP, LPQ, and LGBP features for facial activity analysis.
  • Features can be learned through unsupervised sparse coding or supervised deep learning, including CNN-based methods.
  • Deep CNNs have achieved promising facial expression recognition performance under real-world conditions in EmotiW2015 and EmotiW2016 challenges.
  • Most CNN-based approaches use softmax loss, but real-world recognition still involves high intra-class variation and inter-class similarity.
  • Center loss reduces intra-class variation by pulling samples toward class centers, but island loss is introduced to also increase inter-class differences.

A. A Brief Review of Center Loss

Center loss reduces intra-class feature variation by pulling samples toward class centers, while its weighted combination with softmax loss guides CNN training.

  • A. A Brief Review of Center Loss: The center loss is defined as the summed squared distances between mini-batch samples and their corresponding class centers.This objective penalizes each sample’s distance from the center associated with its class label.
  • A. A Brief Review of Center Loss: Center loss pulls samples toward their corresponding class centers to reduce intra-class variation.The feature vectors are taken from the fully connected layer before the decision layer, and centers are updated during training.
  • A. A Brief Review of Center Loss: During forward propagation, a weighted joint loss combines softmax loss and center loss to drive CNN fine-tuning.The scalar λ balances the two loss terms during backward propagation.
  • A. A Brief Review of Center Loss: The center-loss gradient with respect to each input feature is computed during backward propagation, while class centers are updated iteratively with SGD.The center updates occur as part of the CNN’s iterative optimization process.

B. An Island Loss for Facial Expression Recognition

Island loss extends center loss by simultaneously reducing within-expression variation and increasing separation between expression classes through center similarity penalties.

  • B. An Island Loss for Facial Expression Recognition: Island loss addresses overlapping expression clusters by reducing intra-class variation and increasing inter-class differences simultaneously.The design responds to center loss, which can reduce within-class variation while leaving clusters from different classes overlapped.
  • B. An Island Loss for Facial Expression Recognition: Island loss combines center loss with pairwise distances between class centers in feature space.Its two components respectively constrain sample-to-center distances and similarities between expression centers.
  • B. An Island Loss for Facial Expression Recognition: The first island-loss term penalizes sample-to-center distance, while the second penalizes similarity between different expressions.The parameter λ1 balances these two terms.
  • B. An Island Loss for Facial Expression Recognition: The overall CNN objective adds island loss to softmax loss, with λ balancing the two losses during training.The resulting joint loss is used in the CNN training process.
  • B. An Island Loss for Facial Expression Recognition: The island-loss derivative with respect to an input feature is backpropagated through fully connected and convolutional layers to fine-tune the CNN.The cited passage states that this derivative is further propagated to lower layers.

2) Backward propagation:

IL-CNN backward propagation updates island-loss centers iteratively with SGD and propagates the resulting error through the network; the complete process is summarized algorithmically.

  • 2) Backward propagation:: The island-loss gradient with respect to each input feature is the same as the center-loss gradient and is propagated to lower layers.This error reaches the lower fully connected and convolutional layers to drive CNN fine-tuning.
  • 2) Backward propagation:: The jth class center is updated using stochastic gradient descent during backward propagation.The update is performed for each class center as part of iterative optimization.
  • 2) Backward propagation:: The center-update formulation uses the total number of expressions, denoted |N|.This quantity appears as a condition in the center-update description.
  • 2) Backward propagation:: Class centers are updated iteratively in each mini-batch using a learning rate α1.The forward and backward learning process is summarized in Algorithm 1.

IV. EXPERIMENTS

The experiments evaluate IL-CNN on four benchmark expression databases and compare it with same-architecture CNN baselines trained using softmax loss or softmax plus center loss.

  • IV. EXPERIMENTS: Experiments use four benchmark databases: CK+, MMI, Oulu-CASIA, and the spontaneous SFEW dataset.The databases include three posed facial-expression datasets and one spontaneous dataset.
  • IV. EXPERIMENTS: The experiments set α = 1, λ = 0.01, and λ1 = 10 empirically.These are the reported experimental hyperparameter settings.
  • IV. EXPERIMENTS: IL-CNN training uses mini-batch forward-backward optimization with joint loss L = LS + λLIL.The algorithm initializes network, softmax-loss, and island-loss parameters before iterative updates.
  • IV. EXPERIMENTS: The training procedure outputs network parameters, island-loss parameters cj, and softmax-loss parameters θ.These outputs are listed at the end of the forward-backward learning algorithm.
  • IV. EXPERIMENTS: IL-CNN is compared with two same-structure baseline CNNs supervised by softmax loss or softmax loss plus center loss.These comparisons are designed to evaluate the effectiveness of the proposed island loss.

A. Preprocessing

Preprocessing uses landmark-based alignment to reduce face-scale and in-plane-rotation variation, followed by fixed resizing and contrast enhancement. The datasets include posed sequences and selected frames around expression peaks.

  • A. Preprocessing: Facial landmarks align each image using the centers of both eyes and the mouth.The aligned face regions are resized to 60 × 60.
  • A. Preprocessing: Histogram equalization is applied to improve contrast in the aligned facial images.
  • A. Preprocessing: CK+ contributes the final three frames of each sequence, producing 981 labeled images.The database contains 327 videos from 118 subjects and seven expression labels.
  • A. Preprocessing: MMI contributes three middle frames from each sequence because the peak-frame location is not provided.The experiment uses 208 frontal-view sequences from 31 subjects.

2) MMI dataset:

The supplied passages describe Oulu-CASIA and SFEW datasets and outline shared CNN initialization and training settings for benchmark evaluation.

  • 2) MMI dataset:: Oulu-CASIA contains 2,880 videos from 80 subjects across six expressions and varied imaging and illumination conditions.Only 480 VIS videos recorded under normal indoor illumination are used in the described experiment.
  • 2) MMI dataset:: SFEW contains 1,766 images split into 958 training, 436 validation, and 372 testing samples across seven expression categories.Testing labels are withheld and performance is provided by the challenge organizer.
  • 2) MMI dataset:: The baseline CNN is pretrained on FER-2013 and the other three datasets before models are fine-tuned separately on each benchmark.
  • 2) MMI dataset:: Training uses stochastic gradient descent with momentum 0.9, mini-batches of 300, weight decay 0.05, and dropout rate 0.6.The learning rate starts at 0.001 and is reduced tenfold every 2500 iterations.

D. Experimental Results

Across posed facial-expression benchmarks, IL-CNN is evaluated with confusion matrices and accuracy comparisons against CNN baselines and prior methods. The reported evidence shows strong CK+ performance and consistent advantages over softmax- and center-loss baselines.

  • D. Experimental Results: The proposed method is compared using average recognition accuracy and confusion matrices across CK+, MMI, and Oulu-CASIA.The supplied tables identify the corresponding dataset-specific comparisons and confusion-matrix evaluations.
  • D. Experimental Results: 94.35% average recognition accuracy is achieved on CK+ across 7 expressions.
  • D. Experimental Results: IL-CNN consistently outperforms CNNs using softmax loss or center loss across the CK+, MMI, and Oulu-CASIA datasets.The paper attributes this pattern to reducing intra-class variations and inter-class similarities.
  • D. Experimental Results: IL-CNN achieves better or at least comparable performance to state-of-the-art methods on the three posed datasets.
  • D. Experimental Results: Island loss can replace softmax loss in more advanced network structures.This is noted in comparison with the deeper Inception model on MMI.

4) Results on SFEW dataset:

On SFEW, IL-CNN outperforms the baseline CNNs on both validation and testing sets. A shallow single IL-CNN ranks third among compared testing methods, while an ensemble reaches performance comparable to the best methods.

  • 4) Results on SFEW dataset:: IL-CNN outperforms the baseline CNNs on both the SFEW validation and testing sets.The evaluation uses separate confusion matrices for validation and testing.
  • 4) Results on SFEW dataset:: A shallow single IL-CNN ranks third on the SFEW testing set among the compared methods.The two higher-ranked methods use CNN ensembles, and one also combines different network structures.
  • 4) Results on SFEW dataset:: An ensemble of IL-CNNs achieves performance comparable to the best methods on SFEW.
  • 4) Results on SFEW dataset:: t-SNE visualizations compare softmax, softmax-plus-center-loss, and softmax-plus-island-loss features using training, validation, and testing samples.Island-loss features form more compact clusters that are better separated in feature space.

F. A Study of the Distances

The study evaluates between-center and sample-to-center distances to assess whether island loss separates expression classes and compresses within-class features. Across four datasets, island loss increases center separation and yields the smallest sample-to-center distances.

  • Between-center distance: The analysis averages cosine distances between expression centers to measure inter-class separation across four databases.For softmax, distances are computed between expression means; for center loss and island loss, they are computed between learned centers.
  • Between-center distance: Larger average between-center distance indicates farther-separated expression centers.
  • Sample-to-center distance: Island loss consistently produces the smallest average sample-to-center distance on all four datasets.Smaller distance indicates that samples lie closer to their corresponding class centers.
  • Interpretation: These distance patterns show that island loss increases inter-class differences while further reducing intra-class variation compared with center loss.
  • Broader applicability: Island loss is presented as a general CNN loss function that can be adopted by other advanced network structures and machine-learning problems.
Loading 1710.03144v3…