PaperlayerView exact revision on arXiv

Source-linked AI summary

Efficient Facial Feature Learning with Wide Ensemble-based Convolutional Neural Networks

Henrique Siqueira, Sven Magg, Stefan Wermter

arXiv:2001.06338v1cs.CVcs.LGstat.ML

TL;DR

Deep-network ensembles are computationally costly, and the paper examines whether shared representations can reduce redundancy while preserving ensemble performance. It evaluates ESRs across facial-expression and affect tasks, reporting performance above state-of-the-art methods on AffectNet and FER+ and human-level facial-expression recognition on AffectNet.

  • Problem

    Deep-network ensembles require high computational power, while the effects of ESR branching level and scalability to large-scale facial-expression datasets need further examination.

    Source: Introduction · pp. 1–2

  • Method

    The paper evaluates convolutional ESRs that share representations across layers and use independent branches, examining branching levels and training on facial-expression and affect tasks.

    Source: Introduction · pp. 1–2, Training ESRs at Different Branching… · p. 4, Evaluation on the AffectNet Dataset · p. 8

  • Results

    ESRs outperformed state-of-the-art deep networks on AffectNet and FER+, and facial-expression recognition on AffectNet reached human-level performance.

    Source: Conclusions · p. 9

  • Takeaways & Limitations

    The authors report that balanced training subsets and collective classification help ESRs cope with unbalanced label distributions while retaining low redundancy.

    Source: Conclusions · p. 9

    Human-level affect inference under real-world conditions remains far from being reached.

    Source: Conclusions · p. 9

Abstract

from arXiv · show

Ensemble methods, traditionally built with independently trained de-correlated models, have proven to be efficient methods for reducing the remaining residual generalization error, which results in robust and accurate methods for real-world applications. In the context of deep learning, however, training an ensemble of deep networks is costly and generates high redundancy which is inefficient. In this paper, we present experiments on Ensembles with Shared Representations (ESRs) based on convolutional networks to demonstrate, quantitatively and qualitatively, their data processing efficiency and scalability to large-scale datasets of facial expressions. We show that redundancy and computational load can be dramatically reduced by varying the branching level of the ESR without loss of diversity and generalization power, which are both important for ensemble performance. Experiments on large-scale datasets suggest that ESRs reduce the remaining residual generalization error on the AffectNet and FER+ datasets, reach human-level performance, and outperform state-of-the-art methods on facial expression recognition in the wild using emotion and affect concepts.

Introduction

Deep-network ensembles can reduce residual generalization error, but their computational cost and redundancy remain challenges. The paper examines how shared representations and branching level affect ensemble efficiency, diversity, and performance, including on large-scale facial-expression datasets.

Introduction

  • Deep-network ensembles require high computational power, while independent models can retain substantial redundancy in low-level visual features.Prior approaches reduce redundancy through sample selection, input-space decomposition, or distillation, but the latter still requires training a traditional ensemble.Source: Introduction · p. 1
  • ESRs combine shared layers that reduce redundancy with independent convolutional branches that carry ensemble diversity.The paper identifies the branching level as a key factor in computational load, redundancy, diversity, and generalization.Source: Introduction · p. 1
  • The optimal branching level is hypothesized to lie between early branching, which duplicates low-level features, and late branching, which may reduce diversity.The proposed intermediate features are abstract facial movements such as smiling and frowning, before they are encoded as emotion concepts.Source: Introduction · pp. 1–2
  • The study examines branching effects on a small, structured lab dataset and tests ESR scalability and generalization on large-scale in-the-wild benchmarks.The authors also report that source code, trained networks, and supplementary material are available in their repository.Source: Introduction · p. 2

Ensembles with Shared Representations

An ESR shares low- and middle-level convolutional features across independent branches. A combined branch loss trains the shared base and branches together, while the branches make a collective prediction at inference.

Ensembles with Shared Representations

  • An ESR uses a convolutional base to learn low- and middle-level features, then shares those features with independent convolutional branches.The base corresponds to the shared layers, and the branches form the ensemble.Source: Ensembles with Shared Representations · pp. 2–3
  • Each branch can learn distinctive features while competing for the shared layers, and training minimizes the sum of the branches’ loss functions.The shared-layer parameters act as a regularizer, while each branch has its own parameters.Source: Ensembles with Shared Representations · p. 3
  • Branches are added sequentially during training, and the shared layers transfer features across branches as the ensemble grows.At inference, the ensemble classifies an input through a collective decision such as plurality or majority voting.Source: Ensembles with Shared Representations · p. 3

Experimental Datasets

The experiments use a controlled, posed-expression dataset and two in-the-wild datasets with varied facial configurations and annotation schemes. This selection supports evaluation under both scarce-data lab conditions and more challenging real-world variation.

Experimental Datasets

  • The lab and in-the-wild datasets assess ESRs under different conditions: scarce training data and structured annotations versus large-scale data, intraclass variation, occlusions, and imbalanced labels.The paper describes the in-the-wild datasets as capturing facial configurations across varied environmental conditions.Source: Experimental Datasets · p. 3
  • CK+ includes facial configurations portrayed by 123 subjects; onset expressions were recorded from frontal and 30-degree camera viewpoints, and peak expressions were annotated and validated for 30 action units and 8 discrete emotion concepts.Source: In-the-Lab Dataset · p. 4
  • AffectNet contains more than one million internet images, with half manually annotated for 8 discrete emotions, arousal, and valence.Its labels are heavily imbalanced; contempt constitutes 1% of annotated images.Source: In-the-Wild Datasets · p. 4
  • FER+ reannotates FER-2013’s 35,887 images through crowd-sourced labels from 10 annotators, using 8 emotion categories including contempt.The source dataset’s annotations were considered highly noisy.Source: In-the-Wild Datasets · p. 4

Redundancy and Diversity Analysis

The CK+ experiments compare ESR branching levels and training strategies for recognition performance, computational load, and feature diversity. ESR-4 Lvl. 3 matched the traditional ensemble’s generalization with fewer trainable parameters, while added branches converged faster and retained diverse facial-feature representations.

Training setup

  • The CK+ comparison used subject-independent 10-fold crossvalidation, with four networks as the traditional-ensemble baseline and ESRs evaluated across branching levels.The single-network baseline was selected after searching convolutional architectures and training strategies; Figure 3 illustrates the in-the-lab architecture and ESR-4 Lvl. 3.Source: Training ESRs at Different Branching… · p. 4, Methodology · p. 4, Figure 3 · p. 5

Training strategies

  • The tested ESR training strategies varied whether shared layers and existing branches continued learning at the same rate, a lower rate, or not at all; interleaved training was also tested.For interleaved training, branches were trained iteratively on random mini-batches from the four training folds.Source: Training ESRs at Different Branching… · pp. 4–5

Redundancy and Diversity Analysis

Training strategies

  • ESR ensembles reached accuracy as high as the traditional ensemble, whereas interleaved training performed worse among the ensemble approaches.The authors suggest the interleaved approach’s lower performance might reflect low ensemble diversity.Source: Training ESRs at Different Branching… · p. 5, Figure 4 · p. 5

Transfer Learning and Diversity Analysis

  • After the first update, an added ESR branch was already twice as accurate as its traditional-ensemble counterpart; the gap closed around epoch 50.The authors attribute faster ESR branch convergence to prior knowledge stored in the shared layers.Source: Figure 5 · p. 6, Transfer Learning and Diversity Analysis · p. 6
  • ESR learned the mouth’s relevance to happy expressions after one update of branch two, while the traditional ensemble took around 50 epochs; ESR branches also captured distinct facial cues.For happy expressions, branches captured the smile, nasolabial furrows, or outer-eye wrinkles.Source: Transfer Learning and Diversity Analysis · pp. 6–7

Training ESRs on Large-Scale Datasets

The experiments examine whether ESRs can make large-scale facial-expression ensembles affordable while maintaining strong recognition and affect prediction. ESR-9 achieves state-of-the-art results on AffectNet and FER+ and reaches human-expert performance in facial-expression annotations.

Training ESRs on Large-Scale Datasets

  • The large-scale experiments followed benchmark protocols, used a deeper ESR architecture to preserve feature-map spatial information, and targeted affordable ensemble training on limited compute.The AffectNet architecture added convolutional and batch-normalization layers, with max pooling after every two convolutional layers.Source: Evaluation on the AffectNet Dataset · p. 7, Methodology · p. 7, Training ESRs on Large-Scale Datasets · p. 7
  • 0.33 and 0.36 RMSEs for arousal and valence, respectively, outperform state-of-the-art methods; ESR-9 uses approximately 20M parameters, nine times fewer than the comparison approach.Only output-layer weights were trained for affect prediction, and ESR-9 retained discrete emotion perception.Source: Table 4 · p. 8, Evaluation on the AffectNet Dataset · p. 8
  • ESR-9 reached human experts’ performance in facial-expression annotations, whose disagreement levels were 0.36 and 0.34 RMSEs for arousal and valence prediction.The experiments also trained three AlexNets, one for each facial-expression perception problem.Source: Evaluation on the AffectNet Dataset · p. 8
  • 57.92% test accuracy on FER+ before fine-tuning ESR-9 indicates cross-dataset generalization from AffectNet training.FER+ images are grayscale, low-resolution, and less centralized than AffectNet images.Source: Fine-tuning on the FER+ Dataset · p. 8
  • A 5,000-sample-per-category upper bound on FER+ balanced overall accuracy with relatively strong recognition of under-represented categories; lower caps reduced overall accuracy, while higher caps reduced ensemble diversity.AffectNet branches were also trained on balanced subsets of up to 5,000 samples per emotion category.Source: Evaluation on the AffectNet Dataset · p. 7, Fine-tuning on the FER+ Dataset · p. 8
  • 87.153% average test accuracy (0.097% standard deviation) on FER+ and 59.3% accuracy on AffectNet put ESR-9 ahead of state-of-the-art methods on both datasets.The FER+ result follows sequential branch fine-tuning; the AffectNet ensemble outperformed the compared methods.Source: Table 3 · p. 7, Evaluation on the AffectNet Dataset · p. 7, Fine-tuning on the FER+ Dataset · p. 8

Conclusions

The conclusions attribute improved performance on AffectNet and FER+ to training branches on balanced subsets while sharing representations. They also identify catastrophic forgetting and the gap between facial-expression recognition and real-world affect inference as unresolved constraints.

Conclusions

  • ESRs trained branches on balanced subsets and collectively outperformed state-of-the-art deep networks on AffectNet and FER+, with low redundancy and transfer from shared layers.The authors report that branching level directly affects ensemble diversity, generalization, and computational load.Source: Conclusions · p. 9
  • Continual branch training can cause catastrophic forgetting; large learning-rate differences may erase learned information, while similar rates may foster co-adaptation and reduce diversity.The authors identify overcoming catastrophic forgetting in ESRs as future work.Source: Conclusions · p. 9
  • Despite human-level facial-expression recognition on AffectNet, human-level affect inference under real-world conditions remains distant and calls for cross-modal, temporal, and contextual information.The proposed next step is to use ESR-9’s high-level representations as “proto concepts” to guide learning of emotion concepts.Source: Conclusions · p. 9
Loading 2001.06338v1…