Source-linked AI summary
Feature Decomposition and Reconstruction Learning for Effective Facial Expression Recognition
Delian Ruan, Yan Yan, Shenqi Lai, Zhenhua Chai, Chunhua Shen, Hanzi Wang
TL;DR
FER methods may struggle when different expressions share facial actions and produce subtle distinctions. FDRL decomposes backbone features into facial action-aware latent features, models their relationships, and reconstructs expression features. It consistently outperforms several state-of-the-art methods across in-the-lab and in-the-wild databases, including 89.47% accuracy on RAF-DB and 62.16% on SFEW.
Problem
Existing FER methods can overlook high similarities between expressions, making subtle expression differences difficult to distinguish.
Method
FDRL jointly trains FDN to decompose backbone features into facial action-aware latent features and FRN to model their relationships and reconstruct expression features.
Results
FDRL consistently outperforms several state-of-the-art FER methods across in-the-lab and in-the-wild databases, achieving 89.47% accuracy on RAF-DB and 62.16% on SFEW.
Takeaways & Limitations
Feature decomposition and reconstruction provide fine-grained expression features for distinguishing subtle differences between facial expressions.
Abstract
from arXiv · showhide
In this paper, we propose a novel Feature Decomposition and Reconstruction Learning (FDRL) method for effective facial expression recognition. We view the expression information as the combination of the shared information (expression similarities) across different expressions and the unique information (expression-specific variations) for each expression. More specifically, FDRL mainly consists of two crucial networks: a Feature Decomposition Network (FDN) and a Feature Reconstruction Network (FRN). In particular, FDN first decomposes the basic features extracted from a backbone network into a set of facial action-aware latent features to model expression similarities. Then, FRN captures the intra-feature and inter-feature relationships for latent features to characterize expression-specific variations, and reconstructs the expression feature. To this end, two modules including an intra-feature relation modeling module and an inter-feature relation modeling module are developed in FRN. Experimental results on both the in-the-lab databases (including CK+, MMI, and Oulu-CASIA) and the in-the-wild databases (including RAF-DB and SFEW) show that the proposed FDRL method consistently achieves higher recognition accuracy than several state-of-the-art methods. This clearly highlights the benefit of feature decomposition and reconstruction for classifying expressions.
1. Introduction
FER must distinguish expressions that share facial actions and therefore have subtle visual differences. FDRL addresses this by decomposing features into shared, facial action-aware components and reconstructing expression-specific features, achieving strong benchmark accuracy.
- Motivation: Existing FER methods often disentangle pose, identity, and illumination but overlook high similarities between expressions.These similarities can leave extracted expression features difficult to distinguish.
- Motivation: Different expressions can share facial actions such as closing eyes, opening mouths, or frowning brows, making their distinctions subtle.Examples include fear versus surprise and other expression pairs with overlapping facial movements.
- Method: FDRL jointly trains a Feature Decomposition Network and Feature Reconstruction Network end-to-end.The method is designed to model both expression similarities and expression-specific variations.
- Method: FDN decomposes backbone features into facial action-aware latent features, while FRN models expression-specific variations and reconstructs the expression feature.FRN includes Intra-RM and Inter-RM for modeling intra-feature and inter-feature relationships.
- Results: FDRL consistently outperforms several state-of-the-art FER methods across in-the-lab and in-the-wild databases.The evaluation includes CK+, MMI, Oulu-CASIA, RAF-DB, and SFEW.
- Results: 89.47% and 62.16% recognition accuracy are achieved on RAF-DB and SFEW, respectively.These results support the reported benefit of feature decomposition and reconstruction for FER.
2. Related work
Prior FER work reduces disturbance effects, label noise, or holistic feature ambiguity, but often does not model subtle inter-expression differences. FDRL instead uses fine-grained, facial action-aware latent representations and relation modeling.
- Existing approaches: Deep FER methods primarily address disturbance disentangling and expression feature extraction.Disturbances include pose, identity, and illumination, while feature-extraction methods improve discriminability through losses or attention.
- Existing approaches: Some methods address noisy labels, including inconsistent annotations and uncertain facial images.These approaches focus on database-label reliability rather than subtle expression differences.
- Research gap: Existing disturbance- and label-focused methods do not account for subtle differences between facial expressions.FDRL formulates FER through feature decomposition and reconstruction to model expression similarities and expression-specific variations.
- FDRL architecture: FDRL uses a backbone, FDN, FRN, and EPN, with FRN reconstructing expression features from relation-aware latent features.The overview identifies Intra-RM and Inter-RM as FRN's two relation-modeling modules.
- FDRL distinction: Unlike holistic-feature methods, FDRL decomposes basic features into facial action-aware latent features and models their intra-feature and inter-feature relationships.This produces more fine-grained representations for identifying subtle expression differences.
3. Our Method
FDRL combines feature decomposition and reconstruction: FDN extracts compact facial action-aware latent features, while FRN models intra- and inter-feature relationships to reconstruct expression features.
- FDRL comprises a backbone, FDN, FRN, and EPN, jointly trained end-to-end for facial expression prediction.The backbone extracts basic CNN features; EPN predicts expression labels from reconstructed features.
- Feature Reconstruction Network: Intra-RM assigns importance weights to latent features, regularizes their class-specific distributions, and produces intra-aware features.Each latent feature is scaled by its Intra-W weight before inter-feature modeling.
- Feature Decomposition Network: FDN decomposes basic features into facial action-aware latent features and uses compactness loss to reduce intra-latent variation.The compactness loss learns a center for each latent feature and penalizes distances to that center.
- Feature Decomposition Network: The nine latent-feature groups visualize distinct facial actions, including closing eyes, opening mouths, frowning brows, and lip movements.Groups are formed using the highest intra-feature relation weight for the same latent feature.
- Feature Reconstruction Network: Inter-RM encodes intra-aware features as relation messages and models their pairwise relationships in a complete graph.The resulting inter-feature weights represent relation importance between different relation-message nodes.
4. Experiments
Experiments evaluate FDRL across controlled and real-world FER databases, then analyze latent-feature choices, module contributions, and feature representations. FDRL improves discriminability and achieves its best reported accuracies with integrated decomposition and reconstruction.
- Experimental setup: Experiments compare FDRL with state-of-the-art FER methods across five public databases spanning in-the-lab and in-the-wild settings.The databases include CK+, MMI, Oulu-CASIA, RAF-DB, and SFEW.
- Ablation studies: Nine latent features produce the best recognition accuracy on both the MMI and RAF-DB ablation evaluation.Too few latent features inadequately model expression similarities.
- Ablation studies: The compactness, balance, and distribution loss weights achieve their best reported setting at λ1 = 0.0001, λ2 = 1.0, and λ3 = 0.0001.Removing compactness loss decreases performance, while the reported best values are selected through ablation studies on MMI and RAF-DB.
- Ablation studies: Adding FDN improves the backbone, while adding either Intra-RM or Inter-RM further improves recognition accuracy; the complete FDRL variant performs best.FDN models shared expression information, whereas the relation modules model expression-specific variations.
- Visualization: Compared with baseline ResNet-18 features, FDRL reduces intra-class differences and enhances inter-class separability, especially for fear–surprise and disgust–sadness.Mean Intra-W vectors also show similar weights across some expressions, motivating Inter-RM to model inter-feature relationships.
5. Conclusion
The conclusion presents FDRL as a feature decomposition and reconstruction method that separates shared expression information from expression-specific information. Its components jointly support fine-grained facial expression recognition across laboratory and real-world databases.
- Conclusion: FDRL uses FDN to model shared information across expressions with a compactness loss.FDN produces compact latent features representing expression similarities.
- Conclusion: FRN models expression-specific information through Intra-RM and Inter-RM, then reconstructs the expression feature.Intra-RM captures within-feature relationships, while Inter-RM captures relationships between intra-aware features.
- Conclusion: Experimental results on in-the-lab and in-the-wild databases show the superiority of FDRL for facial expression recognition.The conclusion summarizes the method's reported performance across both database settings.