Source-linked AI summary
Learning to Amend Facial Expression Representation via De-albino and Affinity
Jiawei Shi, Songhao Zhu, Zhiwei Liang
TL;DR
Padding can erode feature maps, weakening facial-expression representations, and the paper also considers affinities among expressions. It introduces ARM, which reduces the weight of eroded features and splits facial features for simpler learning. The authors report improved FER performance, including a gain over pre-trained ResNet-18 on RAF-DB.
Problem
Padding convolution erodes feature maps and affects downstream representations, while affinities among expressions have received little attention in FER.
Source: §1 · pp. 1–2Method
ARM replaces a pooling layer with blocks that reduce the weight of eroded features and split facial features into generic and unique parts.
Source: §1 · p. 2, §3.1 · p. 4Results
90.42 from 86.4: ARM-variant ResNet-18 on RAF-DB improves pre-trained ResNet-18 performance and exceeds the state-of-the-art methods.
Source: §1 · p. 2Takeaways & Limitations
The authors report that their lightweight module addresses potential padding erosion and significantly improves facial expression recognition.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Facial Expression Recognition (FER) is a classification task that points to face variants. Hence, there are certain affinity features between facial expressions, receiving little attention in the FER literature. Convolution padding, despite helping capture the edge information, causes erosion of the feature map simultaneously. After multi-layer filling convolution, the output feature map named albino feature definitely weakens the representation of the expression. To tackle these challenges, we propose a novel architecture named Amending Representation Module (ARM). ARM is a substitute for the pooling layer. Theoretically, it can be embedded in the back end of any network to deal with the Padding Erosion. ARM efficiently enhances facial expression representation from two different directions: 1) reducing the weight of eroded features to offset the side effect of padding, and 2) decomposing facial features to simplify representation learning. Experiments on public benchmarks prove that our ARM boosts the performance of FER remarkably. The validation accuracies are respectively 90.42% on RAF-DB, 65.2% on Affect-Net, and 58.71% on SFEW, exceeding current state-of-the-art methods. Our implementation and trained models are available at https://github.com/JiaweiShiCV/Amend-Representation-Module.
1. Introduction
FER matters for human-computer interaction, but limited datasets and padding-related feature erosion pose challenges for recognition. The paper introduces ARM to address padding erosion and use expression affinities to simplify representation learning.
1. Introduction
- Limited FER dataset scale constrains deep networks, while the cost of creating larger datasets makes improving recognition accuracy a major challenge.Facial expression recognition is also described as significant for human-computer interaction.Source: §1 · p. 1
- Padding can erode feature maps from the edges inward, producing albino features that affect downstream representations.The paper links this erosion to zeros used in padding convolution and notes that edge information is important.Source: §1 · pp. 1–2
- ARM replaces a pooling layer with De-albino and Feature Arrangement blocks, reducing eroded-feature weights and rearranging the feature map to support de-albino processing.The arrangement changes feature-point positions while preserving their relative positions; the channel count collapses by a ratio of four.Source: §1 · p. 2, Figure 2 · p. 2
- The Sharing Affinity block splits facial features into generic and unique parts to simplify representation learning, addressing affinities among expression features.The paper presents affinity features as a first attempt to optimize FER using relationships among different expressions.Source: §1 · p. 2
- 90.42 from 86.4: ARM-variant ResNet-18 on RAF-DB improves pre-trained ResNet-18 performance and exceeds the state-of-the-art methods.The passage reports these values without specifying a percent sign.Source: §1 · p. 2
- The paper also proposes minimal random resampling to address data imbalance.This contribution is listed alongside the ARM design contributions.Source: §1 · p. 2
2. Related Work
The related-work section traces CNN development from early architectures through later design advances, then reviews the transition in facial-expression recognition from engineered features to deep-learning methods.
2.1. Convolutional Neural Networks
- LeNet established a modern CNN structure with convolutional, pooling, and fully connected layers, while its complexity limited its use on complex problems.LeCun et al. introduced LeNet for handwritten digit recognition.Source: §2.1 · p. 2
- AlexNet won the 2012 ImageNet contest by 10.9 percentage points over the runner-up and used padding, ReLU, dropout, and data augmentation.The passage attributes the architecture to Hinton et al..Source: §2.1 · p. 3
- Later CNN advances included GoogLeNet and VGG, followed by ResNet identity connections and SENet channel recalibration.The passage reports that ResNet [14] addressed degradation in very deep networks and SENet [15] recalibrated channels using spatial features.Source: §2.1 · p. 3
2.2. Facial Expression Recognition
- FER research moved from early optical-flow and action-unit work toward deep learning as datasets grew and engineered features struggled with large, complex data.The passage also surveys deep-learning approaches for landmarks, pose, uncertain labels, occlusion, label inconsistency, and multi-label learning.Source: §2.2 · p. 3
3. Methodology
ARM replaces CNN pooling to address padding-related representation erosion, combining feature rearrangement, convolutional reweighting, and affinity-based feature splitting.
3.1. Overview
- ARM uses FA, DA, and SA blocks: FA supports DA, DA reweights features according to erosion, and SA splits facial features for simpler learning.The evaluated pipeline extracts features with ResNet-18’s front part up to GAP; the two-channel relationship requires a single-channel, unique de-albino kernel.Source: §3.1 · p. 4, Figure 3 · p. 4
3.2. Feature Arrangement
- Feature Arrangement gathers similarly eroded pixels at the feature-map periphery while maintaining their relative position distribution.The authors adopt a feature-map restructuring method to concentrate the most eroded pixels on the outer edge.Source: §3.2 · p. 4
3.3. Assigning Weights by Means of Convolution
- A dedicated no-padding convolution with a large kernel and stride lowers peripheral albino-feature weights while retaining key parts.The authors exploit convolution’s lower perception of edge pixels; kernel size and stride control how peripheral information is filtered.Source: §3.3 · pp. 4–5
3.4. Sharing Affinity
- Sharing Affinity models facial features as generic and unique parts, using a batch-average generic representation updated with exponential moving average.The generic component simplifies learning across similar faces, while λ controls the decrease in weighting of older observations.Source: §3.4 · p. 5
4. Experiments
The experiments evaluate ARM on three public FER datasets, examine padding erosion and block contributions, and compare the full method with prior approaches. The authors report higher benchmark accuracy than current state-of-the-art methods.
4. Experiments
- The experiments assess ARM on three public FER datasets collected in the wild and include padding-effect verification, block ablation, and comparisons with prior methods.The datasets are RAF-DB, AffectNet, and SFEW.Source: §4 · p. 5, §4.1 · p. 5
4.1. Datasets
- Evaluation uses RAF-DB's basic seven-category subset, AffectNet's seven- and eight-category settings, and seven-category SFEW.RAF-DB has 12,271 training and 3,068 test images; SFEW has 958 training, 436 validation, and 372 test samples.Source: §4.1 · p. 5
4.2. Metrics and Results
- Weighted accuracy reflects overall sample accuracy, while unweighted accuracy averages category accuracies; Figure 5 shows test-set confusion matrices across datasets.The authors report that neutral reaches 98% on RAF-DB, slightly exceeding happiness.Source: §4.2 · pp. 5–7, Figure 5 · p. 6
4.3. The Adverse Effect of Albino Features
- On RAF-DB, reducing peripheral albino-feature weights improves representation and slightly boosts performance, with gains varying hump-shaped as kernel size k increases.The authors relate the gain pattern to changes in the perception-frequency ratio between center and edge.Source: §4.3 · p. 7, Figure 6 · p. 7
4.4. Implementation Details
- In the described ARM, padding is omitted; its convolution kernel is 32x32x1 pixels with stride 8 (half the cluster size), and the two DA-block output channels are averaged as each expression’s representation.The feature-map channels are considered parallel, and feature clusters are 16x16x1.Source: §4.4 · p. 7
4.7. Method Comparison
- ARM outperforms current state-of-the-art methods with 90.42% on RAF-DB, 65.2% on AffectNet, and 58.71% on SFEW.Table 3 compares methods across datasets and reports Weighted Accuracy and Unweighted Accuracy on RAF-DB.Source: §4.7 · p. 8, Table 3
4.5. Fine-tuning on the AffectNet Dataset
- AffectNet training uses minimal random resampling to equalize category counts per epoch, selecting images inversely according to class size.The authors retain access to the original data across epochs and adjust the learning-rate attenuation rate from 0.9 to 0.78.Source: §4.5 · pp. 7–8
4.6. Ablation Study
- In RAF-DB ablations, DA and SA each improve performance separately; FA further helps DA, while SA benefits from representations purified by FA and DA.The authors report an additional 1% performance increase for DA with FA, but no apparent FA benefit for SA alone.Source: §4.6 · p. 8, Table 4 · p. 8
5. Conclusion
The conclusion presents ARM as a lightweight response to potential Padding Erosion in facial expression recognition.
5. Conclusion
- ARM is a lightweight module intended to address potential Padding Erosion.The conclusion says its role is amplified through an auxiliary module and reports significant improvement in facial expression recognition.Source: §5 · p. 8
- The module's role is amplified through an auxiliary module.Source: §5 · p. 8
- The authors report significant improvement in facial expression recognition.Source: §5 · p. 8