PaperlayerView exact revision on arXiv

Source-linked AI summary

Learn From All: Erasing Attention Consistency for Noisy Label Facial Expression Recognition

Yuhang Zhang, Chengrui Wang, Xu Ling, Weihong Deng

arXiv:2207.10299v2cs.CV

TL;DR

Noisy-label FER must handle inter-class similarity, annotation ambiguity, and models that memorize partial features tied to noisy labels. EAC uses flip semantic consistency, attention consistency, and image erasing to suppress this memorization automatically. It outperforms state-of-the-art noisy-label FER methods and generalizes to large-class image classification tasks.

  • Problem

    Noisy-label FER is difficult because similar facial expressions and ambiguous annotations make noisy labels especially challenging, while models can memorize label-related partial features.

  • Method

    EAC uses an imbalanced framework with classification loss on original images, flip attention consistency, and random erasing to prevent attention from concentrating on noisy-label features.

  • Results

    EAC outperforms state-of-the-art noisy-label FER methods and generalizes well to CIFAR100 and Tiny-ImageNet.

  • Takeaways & Limitations

    EAC learns useful information from all training samples without requiring the noise rate or label ensembling.

Abstract

from arXiv · show

Noisy label Facial Expression Recognition (FER) is more challenging than traditional noisy label classification tasks due to the inter-class similarity and the annotation ambiguity. Recent works mainly tackle this problem by filtering out large-loss samples. In this paper, we explore dealing with noisy labels from a new feature-learning perspective. We find that FER models remember noisy samples by focusing on a part of the features that can be considered related to the noisy labels instead of learning from the whole features that lead to the latent truth. Inspired by that, we propose a novel Erasing Attention Consistency (EAC) method to suppress the noisy samples during the training process automatically. Specifically, we first utilize the flip semantic consistency of facial images to design an imbalanced framework. We then randomly erase input images and use flip attention consistency to prevent the model from focusing on a part of the features. EAC significantly outperforms state-of-the-art noisy label FER methods and generalizes well to other tasks with a large number of classes like CIFAR100 and Tiny-ImageNet. The code is available at https://github.com/zyh-uaiaaaa/Erasing-Attention-Consistency.

1 Introduction

Noisy-label FER is difficult because facial expressions have inter-class similarity and ambiguous annotations, while models can memorize noisy samples through partial, label-related features. EAC addresses this through feature learning, flip-based attention consistency, and erasing rather than relying on loss-based sample filtering.

  • Noisy-label FER is especially challenging because facial expressions share visual similarities and annotations can be ambiguous.
  • Existing FER approaches primarily select or relabel samples using loss or uncertainty estimates to suppress suspected noisy labels.
  • FER models can memorize noisy samples by attending to features associated with noisy labels while neglecting features needed for the latent truth.
  • EAC uses an imbalanced framework that applies classification loss to original images and attention consistency loss between original and flipped images.
  • Randomly changing the erased area prevents the model from matching attention maps mechanically, making overfitting noisy samples produce larger consistency loss.
  • EAC learns from all training samples without requiring the noise rate, while avoiding the potential label changes introduced by relabeling.

2 Related Work

Prior noisy-label learning methods mainly modify the loss or select clean samples. These approaches use noise modeling, robust objectives, reweighting, or small-loss selection to reduce the influence of noisy data.

  • Noisy-label learning methods are mainly divided into loss modification and clean-sample selection.
  • Loss-based methods estimate noise transition matrices or design robust loss functions to reduce overfitting to noisy labels.
  • Sample-selection methods exploit the memorization effect by reweighting examples or training models to prioritize presumed clean samples.
  • Co-training and related methods select small-loss or agreement-based samples to suppress different types of label noise.

3 Proposed Method

EAC combines classification on original facial images with attention consistency between erased images and their flipped counterparts. This design uses CAM-based attention maps and consistency loss to discourage reliance on noisy-label-related features.

  • Attention Maps: CAM attention maps are weighted sums of last-convolutional-layer feature maps, revealing the features used for prediction.For class j, the attention value at each location is computed by weighting feature maps across channels with fully connected-layer weights.
  • Framework: EAC computes classification loss only on original images, while comparing attention maps between original and flipped counterparts.The framework randomly erases input images, obtains flipped counterparts, and uses consistency loss to prevent memorization of noisy labels.
  • Consistency Regularization: Randomly changing the erased area prevents the model from matching attention maps before and after flipping without learning consistent features.When the model focuses on noisy-label-related features, attention maps diverge under flipping and produce a larger consistency loss.
  • Loss Computation: The FC layer receives globally pooled features from the original feature maps to compute classification loss.Feature maps are passed through global average pooling, resized, and then processed by the fully connected layer using the given labels.
  • Consistency Regularization: The consistency-loss weight λ controls the contribution of erasing consistency loss in the total objective.The paper reports an ablation study of λ in Section 4.8.

4 Experiments

The experiments evaluate EAC on three in-the-wild FER benchmarks with differing dataset sizes and label structures. They also examine noisy-label settings, implementation behavior, learned representations, ablations, and generalization to large-class image classification.

  • Experimental Scope: Experiments evaluate EAC under different label-noise levels on three in-the-wild FER benchmarks.The study describes RAF-DB, FERPlus, and AffectNet before testing noisy-label performance and analyzing why EAC works.
  • Datasets: RAF-DB contains 12,271 training images and 3,068 test images across seven basic expressions.The expressions are neutral, happy, surprise, sad, angry, disgust, and fear.
  • Datasets: FERPlus contains 28,709 training images and 3,589 test images, with the most-voted category used as the annotation.FERPlus extends FER2013 with finer label annotations and is collected through Google search.
  • Datasets: AffectNet contains 286,564 training images and 4,000 test images manually labeled into eight classes.It is collected from the Internet using expression-related queries across three search engines.
  • Implementation: The default setup uses an MS-Celeb-1M-pretrained ResNet-18, aligned 224×224 facial crops, horizontal flips, and random erasing.Training uses batch size 256, an initial learning rate of 0.0002, and Adam optimization.

4.3 Evaluation of EAC on Noisy FER Datasets

EAC is evaluated against noisy-label FER methods across multiple datasets, noise levels, and implementation settings. It consistently improves performance without requiring label relabeling or prior knowledge of the noise rate.

  • Evaluation setup: EAC is evaluated on RAF-DB, FERPlus, and AffectNet with 10%, 20%, and 30% label noise.Other methods are re-implemented on the same randomly generated noisy datasets for fair comparison.
  • Results: EAC outperforms other state-of-the-art noisy-label FER methods by a large margin.Table 1 reports mean accuracy from the final five training epochs.
  • Results: 6.97%, 3.24%, and 4.31% are EAC’s improvements over SCN under 30% noise on RAF-DB, FERPlus, and AffectNet, respectively.The reported comparisons are accuracy differences across the three datasets.
  • Method properties: EAC does not modify training labels and does not require the noise rate to learn from all training samples.This avoids the relabeling risk and the need to distinguish hard samples from noisy samples during filtering.
  • Backbone analysis: Adding EAC to MobileNet or ResNet-50 improves performance, with ResNet-50 producing better results than ResNet-18 in the reported settings.The backbone experiments use erase and flip for baselines as well as EAC.

4.4 Why EAC works

The ablation study attributes EAC’s effectiveness to the interaction of flip attention consistency, imbalanced supervision, and dynamic erasing. Together, these components prevent shortcut memorization and encourage flip-consistent features.

  • Module contributions: Without flip attention consistency, the model cannot use the shared semantic meaning of flipped counterparts to regularize classification.This removes the consistency-based supervision supplied by the transformed image pair.
  • Module contributions: Without erasing, the model can remember attention maps from flipped images, reducing the effectiveness of consistency regularization.Dynamic changes to the erased area prevent this direct memorization strategy.
  • Module contributions: Without the imbalanced framework, noisy labels affect both original and flipped images, allowing joint memorization that makes consistency loss ineffective.The framework computes classification loss only on original images, while consistency links them to their flipped counterparts.

4.5 Whether flip and erase is sufficiently valid for EAC

EAC uses flipping because FER benefits from spatial attention consistency, and erasing because noisy-label memorization relies on partial feature views. The experiments and visualizations examine these design choices and their effects on learned features and attention.

  • Augmentation choices: Flip provides the spatial transformation needed for attention consistency, whereas rotation and scaling are less effective for predominantly frontal, similarly scaled FER faces.The reported rationale is tied to the geometry of FER test images.
  • Ablation comparison: The three-module ablation compares configurations of flip attention consistency, the imbalanced framework, and erasing on noisy RAF-DB.The table reports mean accuracy for RAF-DB with 30% label noise.
  • Augmentation choices: Erasing targets part-based noisy-label memorization by removing remembered feature regions and encouraging attention to the whole feature.Other augmentations are described as unable to directly address this part-view problem.
  • Feature visualization: EAC’s learned features mix samples with different noisy labels, place noisy samples near the classification boundary, and remain related to latent truth.The visualization uses t-SNE features from EAC trained on RAF-DB with 30% noisy labels.
  • Attention visualization: Attention-map comparisons show SCN focusing on noisy-label information in original images while retaining correct predictions on flipped counterparts; EAC regularizes both views.EAC uses flipped-image attention maps to regularize classification on the original and flipped images.

4.7 Visualization of the classification loss values

After 60 training epochs with noisy samples, the baseline and SCN retain evidence of noisy-sample overfitting, whereas EAC preserves a clear loss-based separation between clean and noisy samples.

  • Loss visualization: The baseline remembers nearly all noisy samples, with loss values for all samples concentrated around 0.
  • Loss visualization: SCN avoids overfitting only part of the noisy samples, while many noisy samples remain close to zero loss.
  • Loss visualization: After 60 epochs, EAC still clearly separates clean and noisy samples by classification loss.The loss distribution remains distinguishable after training, unlike the baseline's near-zero losses for nearly all samples.

4.8 Ablation Study

The experiments examine EAC's consistency-loss weighting and its behavior on large-class noisy-label classification tasks. EAC improves baseline accuracy on CIFAR100 and Tiny-ImageNet across tested noise ratios.

  • Ablation Study: EAC achieves state-of-the-art performance across a wide range of consistency-loss weights λ from 0.1 to 10.0.On RAF-DB with ResNet-18, the best λ is 3 under 10% and 20% noise and 5 under 30% noise.
  • Generalization ability: EAC is designed to generalize beyond FER because it does not rely on a class-number-dependent multi-branch model.DMUE requires a branch count equal to the number of classes plus one, which is described as unaffordable for very large class counts.
  • Generalization ability: EAC consistently improves baseline top-1 and top-5 accuracy on both CIFAR100 and Tiny-ImageNet.
  • Generalization ability: On CIFAR100, EAC exceeds the baseline by 6.37%, 9.40%, and 10.89% in top-1 accuracy under 10%, 20%, and 30% noise.
  • Generalization ability: On Tiny-ImageNet, EAC exceeds the baseline by 12.11%, 17.67%, and 22.19% in top-1 accuracy under 10%, 20%, and 30% noise.

4.10 Comparison with other state-of-the-art FER methods

On clean FER datasets, EAC achieves the best reported performance on RAF-DB and AffectNet with seven classes, while scoring slightly below KTN on FERPlus.

  • Clean-dataset comparison: EAC achieves the best performance among compared methods on RAF-DB and AffectNet with seven classes.
  • Clean-dataset comparison: EAC performs slightly below KTN on FERPlus.The comparison uses ResNet-18 for EAC, while method is excluded because it uses a Vision Transformer backbone.

5 Conclusion

The paper introduces EAC as a feature-learning approach for noisy-label FER that uses erasing and flip consistency to prevent noisy-label memorization. Experiments report improvements on FER and generalization to large-class noisy-label classification.

  • Conclusion: EAC addresses noisy-label FER from a feature-learning perspective rather than relying on traditional loss-based noisy-sample detection.
  • Conclusion: EAC combines an imbalanced framework with erasing and flip consistency loss to prevent the model from remembering noisy labels.
  • Conclusion: EAC does not require the noise rate or label ensembling.
  • Conclusion: Experiments show that EAC outperforms other state-of-the-art noisy-label FER methods on clean and noisy datasets.
  • Conclusion: EAC generalizes to noisy-label classification tasks with a large number of classes.
Loading 2207.10299v2…