PaperlayerView exact revision on arXiv

Source-linked AI summary

From Static to Dynamic: Adapting Landmark-Aware Image Models for Facial Expression Recognition in Videos

Yin Chen, Jia Li, Shiguang Shan, Meng Wang, Richang Hong

arXiv:2312.05447v2cs.CV

TL;DR

DFER faces data scarcity, limited dataset representativeness, ambiguous labels, and underexplored temporal dependencies. S2D adapts an SFER image model with landmark-aware prompts, temporal adapters, and anchor-based self-distillation, and reports improvements over prior methods across DFER benchmarks.

  • Problem

    DFER has less data and lower annotation agreement than SFER, while its temporal expression dependencies remain insufficiently modeled.

    Source: §I · pp. 1–2

  • Method

    S2D transfers a pretrained SFER model to video using landmark-aware MCPs, temporal-modeling adapters, and Emotion-Anchors self-distillation.

    Source: §I · pp. 2–3

  • Results

    S2D outperformed previous best methods by 6.47% UAR and 6.73% WAR on DFEW, 5.05% UAR and 4.16% WAR on FERV39K, and 6.59% UAR and 9.19% WAR on MAFW.

    Source: 2) Results on DFER Datasets · p. 10

  • Takeaways & Limitations

    The results support using SFER prior knowledge and facial-landmark detections to enhance DFER performance on the evaluated benchmarks.

    Source: §V · p. 13

Abstract

from arXiv · show

Dynamic facial expression recognition (DFER) in the wild is still hindered by data limitations, e.g., insufficient quantity and diversity of pose, occlusion and illumination, as well as the inherent ambiguity of facial expressions. In contrast, static facial expression recognition (SFER) currently shows much higher performance and can benefit from more abundant high-quality training data. Moreover, the appearance features and dynamic dependencies of DFER remain largely unexplored. To tackle these challenges, we introduce a novel Static-to-Dynamic model (S2D) that leverages existing SFER knowledge and dynamic information implicitly encoded in extracted facial landmark-aware features, thereby significantly improving DFER performance. Firstly, we build and train an image model for SFER, which incorporates a standard Vision Transformer (ViT) and Multi-View Complementary Prompters (MCPs) only. Then, we obtain our video model (i.e., S2D), for DFER, by inserting Temporal-Modeling Adapters (TMAs) into the image model. MCPs enhance facial expression features with landmark-aware features inferred by an off-the-shelf facial landmark detector. And the TMAs capture and model the relationships of dynamic changes in facial expressions, effectively extending the pre-trained image model for videos. Notably, MCPs and TMAs only increase a fraction of trainable parameters (less than +10\%) to the original image model. Moreover, we present a novel Emotion-Anchors (i.e., reference samples for each emotion category) based Self-Distillation Loss to reduce the detrimental influence of ambiguous emotion labels, further enhancing our S2D. Experiments conducted on popular SFER and DFER datasets show that we achieve the state of the art.

I. INTRODUCTION †

DFER is constrained by smaller, less representative datasets, ambiguous annotations, and limited explicit modeling of temporal expression changes. S2D transfers SFER knowledge into a landmark-aware video model and reports strong results across SFER and DFER benchmarks.

I. INTRODUCTION †

  • DFER datasets have fewer samples and lower annotation agreement than SFER datasets, alongside deficiencies in pose, occlusion, illumination, and expression-label clarity.FERV39K has 38,935 samples versus approximately 450,000 for AffectNet; DFEW agreement is 0.50 versus 0.55 on FERPlus.Source: §I · pp. 1–2

II. RELATED WORK

  • The paper motivates transfer from SFER because its datasets are more abundant and diverse, while existing DFER approaches include 3D CNNs, CNN-RNN combinations, and Transformer models.The introduction also discusses vision-language and adapter-based transfer, noting their respective performance or facial-domain challenges.Source: §I · p. 2

I. INTRODUCTION †

  • Most DFER methods learn temporal relationships implicitly from sparse video-to-label supervision rather than explicitly modeling expression changes across a sequence.The introduction identifies this as limiting their ability to capture facial-expression temporal dynamics fully.Source: §I · p. 2
  • S2D adapts an SFER model using landmark-aware features and temporal adapters, while Emotion-Anchors Self-Distillation supplies auxiliary soft-label supervision for ambiguous expressions.The method uses static features learned on AffectNet and landmark-aware features from MobileFaceNet.Source: §I · pp. 2–3
  • S2D achieved superior or comparable accuracy on widely used SFER and DFER datasets; on DFEW it had the highest WAR with fewer than 10% tunable parameters.The SFER datasets were RAF-DB, AffectNet, and FERPlus; the DFER datasets were DFEW, FERV39K, and MAFW.Source: Figure 1 · p. 1, §I · p. 3
  • S2D reports state-of-the-art DFER performance while tuning less than 10% of the model parameters.The authors also describe it as a simple, user-friendly baseline that is easy to implement and follow.Source: §I · pp. 2–3

A. Dynamic Facial Expression Recognition in the Wild

The paper situates S2D among video-recognition and parameter-efficient transfer approaches, then formulates DFER with facial images and landmark-aware features as inputs. Its model embeds both streams and combines image tokens with landmark-guided prompts and temporal information.

A. Dynamic Facial Expression Recognition in the Wild

  • DFER methods include 3D CNNs, CNN-RNN combinations, and Transformer approaches; S2D distinguishes itself by using facial-landmark and SFER prior knowledge.The passage notes 3D CNN computational cost and possible scaling difficulty, while 2D CNN-RNN methods first extract frame features and then model time.Source: A. Dynamic Facial Expression… · p. 3

B. Parameter-Efficient Transfer Learning

  • Parameter-efficient transfer adapts image models to video tasks by tuning subsets of parameters; unlike AIM, S2D's TMA uses learnable temporal self-attention between transformer blocks.AIM modifies transformer layers directly and uses unlearnable self-attention for temporal modeling.Source: B. Parameter-Efficient Transfer Learning · p. 3

III. METHODOLOGY

  • S2D first trains an SFER model, then adds and tunes a small fraction of adaptation parameters to transfer it to DFER.The paper describes this as an alternative to fully fine-tuning a video model or training a DFER model from scratch.Source: §III · pp. 3–4

1) Problem Formulation:

  • The DFER mapping takes facial image sequences and landmark-aware features extracted with an off-the-shelf detector as inputs, and predicts an emotion label.Source: 1) Problem Formulation · p. 4

2) S2D Model:

  • S2D separately patch-embeds facial-image and landmark-feature sequences into expression and landmark tokens for transformer processing.The two input streams have T frames, and each frame is divided into N patches mapped to a D-dimensional latent space.Source: 2) S2D Model · p. 4
  • At each transformer layer, S2D adds MCP guiding prompts and TMA temporal information to the expression tokens as residual inputs.The resulting prompted tokens are then fed into the transformer layer for image-level representation extraction.Source: 2) S2D Model · p. 4

B. Image-Level Representation Enhancement

The section strengthens image-level representations by combining SFER features with facial landmark-aware features. MCP uses both feature views to generate guiding prompts for transformer layers.

1) Selection of Static Facial Expression Feature:

  • AffectNet is used to pre-train the model for a robust expression-appearance representation before fine-tuning on other FER datasets.The dataset contains more than 1 million Internet face images, including about 450,000 manually annotated images.Source: 1) Selection of Static Facial… · p. 4

2) Facial Landmark-Aware Feature:

  • MobileFaceNet landmark-aware features provide an auxiliary view that is sensitive to key facial regions and implicitly encodes dynamic facial changes across a video sequence.The features help focus on local expression details such as the mouth, nose, and eyes.Source: 2) Facial Landmark-Aware Feature · pp. 4–5

3) Guiding Prompts Generation:

  • MCP takes facial-expression tokens Hl and landmark tokens Al as input and generates guiding prompts Pl+1 for the next transformer layer.The features are reshaped and projected to a lower-dimensional latent space before prompt generation.Source: 3) Guiding Prompts Generation · p. 5
  • A spatial-attention-like operation on the projected facial-expression tokens emphasizes expression-related details before the guiding prompts are obtained.The prompts are produced through a 1×1 convolution.Source: 3) Guiding Prompts Generation · p. 5

C. Expanding the Image Model to Efficient Video Model

TMA extends the image model to videos by adding temporal modeling between transformer layers. Its temporal and vanilla adapters capture frame relationships and reduce the SFER–DFER domain gap, while only a limited set of parameters is trained.

C. Expanding the Image Model to Efficient Video Model

  • TMA inserts a Temporal Adapter, LayerNorm, and Vanilla Adapter between transformer layers to add temporal modeling to the image model.The Temporal Adapter captures temporal information, while the Vanilla Adapter is used to reduce the SFER–DFER domain gap.Source: C. Expanding the Image Model to… · p. 5, Figure 3 · p. 5
  • The T-Adapter applies Temporal Multi-Head Self-Attention to model relationships between video frames.The video embeddings are reshaped and downsampled before entering temporal self-attention.Source: C. Expanding the Image Model to… · p. 5
  • Only the TMA, MCP, and classifier are tunable during video training, with 9M learnable parameters.All other image-model parameters remain frozen.Source: C. Expanding the Image Model to… · p. 6

D. Emotion-Anchors based Self-Distillation Loss

The self-distillation loss uses emotion-category reference samples to provide auxiliary supervision when video-expression annotations are ambiguous. Its anchors supply similarity-based pseudo-labels alongside the original labels during training.

D. Emotion-Anchors based Self-Distillation Loss

  • Emotion anchors provide auxiliary supervision intended to prevent ambiguous expression labels from deteriorating FER performance.The method assumes most manual annotations are reliable and that the model can gradually learn to recognize correct samples.Source: D. Emotion-Anchors based… · p. 6
  • Per-emotion queues P and Q store reference-sample output probabilities and feature vectors, respectively, and are dynamically updated using ground-truth labels from each mini-batch.The queues maintain a fixed size by adding new samples and removing earlier ones.Source: D. Emotion-Anchors based… · p. 6
  • For each current sample, the method computes cosine similarities to emotion-anchor features and selects the top K samples in each emotion-anchor group.The selected similarities form scores used in the anchor-based supervision process.Source: D. Emotion-Anchors based… · p. 6
  • Training supervision combines the provided label Yi with the pseudo-label Ysoft using cross-entropy and binary cross-entropy terms weighted by η.The text identifies CE and BCE as cross-entropy and binary cross-entropy, respectively.Source: D. Emotion-Anchors based… · p. 6

A. Datasets

The evaluation covers three static-expression datasets and three dynamic-expression datasets, with dataset-specific train/test splits or cross-validation protocols.

A. Datasets

  • SFER evaluation uses RAF-DB, AffectNet-7/8, and FERPlus, with dataset-specific training, validation, and testing splits.RAF-DB has 29,672 images; AffectNet-7 and AffectNet-8 use 283,901 and 287,568 training images, respectively; FERPlus has 28,709 training images.Source: A. Datasets · p. 6
  • DFER evaluation uses DFEW, FERV39K, and MAFW, with DFEW and MAFW evaluated through five-fold cross-validation.FERV39K uses 31,088 training clips and 7,847 test clips; MAFW's experiments use 9,172 clips.Source: A. Datasets · pp. 6–7

B. Evaluation Metrics

The section defines the evaluation metrics and experimental settings, then examines static-feature transfer and landmark-detector choices. AffectNet-7 pretraining generalizes best among the tested SFER models, and MobileFaceNet is selected for landmark-aware features.

B. Evaluation Metrics

  • SFER performance is reported as Accuracy; DFER performance is reported as UAR and WAR, with WAR indicating overall accuracy.UAR is the averaged accuracy across classes.Source: B. Evaluation Metrics · p. 7

C. Implementation Details

  • For DFER, the model starts from AffectNet-7-pretrained weights and is fine-tuned after sampling 16 resized frames per video.The backbone is ViT-B/16, and the training strategy mainly follows MAE-DFER.Source: C. Implementation Details · p. 7

D. Ablation Studies

  • The ablations cover AffectNet-7/8, RAF-DB, FERPlus, DFEW, and FERV39K, reporting only DFEW fold 1.This follows the practice of prior work.Source: D. Ablation Studies · p. 7

1) Image-Level Representation Enhancement:

  • AffectNet-7-pretrained SFER models show smaller cross-dataset accuracy drops than RAF-DB-pretrained models, including a 2.1% drop versus 14.86% on FERPlus.AffectNet-Small, matched to RAF-DB in data size, shows a 5.02% accuracy drop on AffectNet-7; results with MAE-ImageNet1K initialization are consistent with these observations.Source: 1) Image-Level Representation… · p. 7
  • AffectNet-7-pretrained SFER models outperform the alternatives on DFEW and FERV39K, especially when initialized with general-domain weights.The authors recommend AffectNet-7 pretraining for downstream generalization, citing its larger size and more diverse samples.Source: 1) Image-Level Representation… · p. 8
  • MobileFaceNet is selected because it has the lowest inter-ocular normalization error and the highest performance on FERV39K.The reported performance decline with lower landmark-localization precision is particularly noticeable on DFEW.Source: 1) Image-Level Representation… · p. 8

2) Ablation study on the Multi-View Complementary Prompter:

The ablations test landmark-feature fusion, temporal adapters, and Emotion-Anchors-based self-distillation. MCP improves results across SFER and DFER datasets, while the temporal adapter and SDL provide further reported gains.

2) Ablation study on the Multi-View Complementary Prompter:

  • MCP improves accuracy by +1.14% on RAF-DB, +0.33% on AffectNet-7, +0.83% on FERPlus, +0.68% WAR on DFEW, and +1.12% WAR on FERV39K.Table IV compares no landmark-aware features, MCP fusion, and concatenation-plus-projection (CAP), with SFER and DFER results produced by the specified model variants.Source: 2) Ablation study on the Multi-View… · p. 8, Table IV · p. 8
  • CAP causes performance declines across all tested datasets relative to the baseline, whereas MCP is reported as more effective for fusing static and landmark-aware features.The authors suggest CAP may disrupt the structure of the original static features.Source: 2) Ablation study on the Multi-View… · p. 8

3) Ablation study on the Temporal-Modeling Adapter:

  • The Vanilla Adapter improves DFEW by 4.67% UAR and 3.80% WAR, and FERV39K by 5.44% UAR and 4.88% WAR.The authors characterize the gains as limited because the Vanilla Adapter lacks temporal-information modeling.Source: 3) Ablation study on the… · p. 8
  • The Temporal Adapter exceeds the Vanilla Adapter by 2.65% UAR and 3.59% WAR on DFEW, and 0.39% UAR and 0.24% WAR on FERV39K.The passage attributes this enhancement to the temporal modeling capabilities of T-MSA.Source: 3) Ablation study on the… · p. 8

4) The Effectiveness of SDL:

  • Table VI evaluates SDL by comparing it with the original one-hot supervised signal and the commonly used label-smoothing method.Source: Table VI · p. 9

5) Contribution of Proposed Components:

  • Table VII evaluates the contributions of MCP, TMA, and SDL against a baseline.Its caption expands the component names as Multi-View Complementary Prompter, Temporal-Modeling Adapter, and Emotion-Anchors-based Self-Distillation Loss.Source: Table VII · p. 9

3) Ablation study on the Temporal-Modeling Adapter:

  • Combining the Vanilla and Temporal Adapters into TMA yields further gains, particularly on FERV39K (+2.28% UAR and +1.41% WAR).The table caption identifies TMA as the best-performing adapter among those compared.Source: Table V · p. 8, 3) Ablation study on the… · p. 9

4) The Effectiveness of SDL:

The reported SDL ablation shows WAR gains on DFEW and FERV39K, and component ablations attribute further improvements to MCP and TMA. The SFER comparison reports comparable or better performance than previous state-of-the-art methods.

4) The Effectiveness of SDL:

  • SDL improves WAR by 0.04% on DFEW and 0.43% on FERV39K, while Label Smoothing lowers WAR by 0.13% and 0.20%, respectively.The authors report that SDL improves discriminability for ambiguous samples and attribute a FERV39K UAR decline to data imbalance.Source: 4) The Effectiveness of SDL · p. 9

5) Contribution of Proposed Components:

  • MCP improves DFEW by 3.22% UAR and 2.95% WAR, and FERV39K by 1.21% UAR and 1.22% WAR versus the baseline.The authors link these gains to enhanced image-level representation from landmark-aware features.Source: 5) Contribution of Proposed Components · p. 9
  • In ablations on the DFEW and FERV39K testing sets, integrating the Temporal-Modeling Adapter (TMA; line 3) further enhanced performance by 7.23% UAR and 7.39% WAR on DFEW, and 9.36% UAR and 6.10% WAR on FERV39K.Source: 5) Contribution of Proposed Components · p. 9
  • The component comparison uses DFEW and FERV39K testing sets and includes class-level WAR and emotion-category accuracy visualizations.The figure compares S2D with the baseline defined as the model without TMA, MCP, and SDL.Source: Figure 5 · p. 9, 5) Contribution of Proposed Components · p. 9

E. Comparison with State-of-the-Art Methods

  • The SFER model achieves comparable or better performance than previous state-of-the-art methods on RAF-DB, AffectNet-7/8, and FERPlus.The comparison highlights the best and second-best results; the cited table caption does not provide the individual values.Source: Table VIII · p. 9

5) Contribution of Proposed Components:

  • SDL adds 0.04% WAR on DFEW and 0.43% WAR on FERV39K after S2D improves over the baseline.The authors identify mitigating interference from ambiguous annotations as a particular role of SDL.Source: 5) Contribution of Proposed Components · p. 9

E. Comparison with State-of-the-Art Methods

1) Results on SFER Datasets:

S2D reports leading results on the supplied SFER and DFER benchmarks, while using landmark-aware features to focus attention on facial regions and expression changes. Its DFER results include comparisons with supervised, self-supervised, and vision-language methods.

1) Results on SFER Datasets:

  • S2D achieved the best results on RAF-DB, AffectNet-7, and FERPlus, including 92.57% accuracy on RAF-DB, 0.50% above existing methods.The authors also report the highest accuracy on FERPlus despite its limited data and 48x48 grayscale images.Source: 1) Results on SFER Datasets · p. 10

2) Results on DFER Datasets:

  • S2D surpassed prior supervised leaders across DFEW, FERV39K, and MAFW, including +6.59% UAR and +9.19% WAR on MAFW.The authors attribute the gains to learning representations by pre-training on a large-scale SFER dataset.Source: 2) Results on DFER Datasets · p. 10
  • Against MAE-DFER, S2D improved WAR by 1.60%, 0.49%, and 3.06% on DFEW, FERV39K, and MAFW, respectively.With oversampling, it also improved UAR by 2.04%, 0.85%, and 1.54% on those datasets, respectively.Source: 2) Results on DFER Datasets · p. 10
  • S2D improved on CLIP-based methods by +5.84% UAR and +4.78% WAR on DFEW, +2.70% UAR and +0.91% WAR on FERV39K, and +4.51% UAR and +4.82% WAR on MAFW.The compared methods, CLIPER and DFER-CLIP, use language and general vision knowledge from CLIP.Source: 2) Results on DFER Datasets · p. 10
  • On DFEW, S2D exceeded M3DFEL by 14.87% on sad, 7.43% on neutral, and 9.95% on angry.Oversampling improved disgust by 24.14% and fear by 15.51%, while maintaining the highest UAR and WAR.Source: 2) Results on DFER Datasets · p. 10, Table X · p. 11

1) Attention Visualization:

  • Landmark-aware attention concentrated on key facial regions and shifted with muscle movements, whereas attention without landmarks was more dispersed and relatively unchanged.The visualizations show attention moving toward the lips as a woman opens her mouth, then diminishing as it closes.Source: 1) Attention Visualization · pp. 11–12, Figure 6 · p. 11

2) Visualization of Feature Distribution:

Visualizations indicate that MCP, TMA, and SDL improve feature discrimination, while SDL makes output distributions more discriminative for ambiguous expressions. The conclusion describes S2D as a parameter-efficient transfer approach with comparable performance to prior state-of-the-art methods.

2) Visualization of Feature Distribution:

  • Adding MCP, TMA, and SDL made DFEW high-level features more discriminative and separable than the baseline; TMA notably improved discriminability.SDL made category distributions more concentrated, particularly for happy, but surprise and fear remained intermixed and disgust features dispersed, which the authors attribute to data shortage.Source: 2) Visualization of Feature Distribution · p. 12

3) Visualization of Output Probability Distribution:

  • For a subtle smile, SDL widened the happy-over-neutral score difference from 0.02 to 0.23, illustrating greater discriminability for an ambiguous expression.The authors also report that SDL distinguishes angry from sad in examples where the model without SDL assigns similar scores and predicts incorrectly.Source: 3) Visualization of Output Probability… · p. 12, Figure 8 · p. 12

V. CONCLUSION

  • The conclusion reports comparable performance to prior state-of-the-art methods and 9M tunable parameters for S2D.It identifies exploring more effective ways to use prior facial and other potential knowledge as future work.Source: §V · p. 13

Related papers

Loading 2312.05447v2…