PaperlayerView exact revision on arXiv

Source-linked AI summary

MAE-DFER: Efficient Masked Autoencoder for Self-supervised Dynamic Facial Expression Recognition

Licai Sun, Zheng Lian, Bin Liu, Jianhua Tao

arXiv:2307.02227v2cs.CVcs.AIcs.HCcs.MM

TL;DR

Supervised DFER is constrained by limited labeled data, while collecting and annotating larger datasets is labor-intensive. MAE-DFER pretrains on unlabeled facial videos with efficient local-global attention and joint appearance-motion modeling. Across six datasets, it outperforms supervised methods by substantial margins.

  • Problem

    Supervised DFER is constrained by limited training samples, and collecting and annotating larger high-quality datasets is time-consuming and labor-intensive.

    Source: §1 · pp. 1–2

  • Method

    MAE-DFER uses large-scale self-supervised pretraining on unlabeled facial videos, with LGI-Former and joint masked appearance and frame-difference reconstruction.

    Source: §1 · p. 2, §3.3 · p. 3

  • Results

    Across six DFER datasets, MAE-DFER consistently beats previous best supervised methods (+5∼8% UAR on in-the-wild datasets; +7∼12% WAR on lab-controlled datasets).

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors position MAE-DFER as a strong DFER baseline and identify scaling and related facial tasks as future directions.

    Source: §5 · p. 8

Abstract

from arXiv · show

Dynamic facial expression recognition (DFER) is essential to the development of intelligent and empathetic machines. Prior efforts in this field mainly fall into supervised learning paradigm, which is severely restricted by the limited labeled data in existing datasets. Inspired by recent unprecedented success of masked autoencoders (e.g., VideoMAE), this paper proposes MAE-DFER, a novel self-supervised method which leverages large-scale self-supervised pre-training on abundant unlabeled data to largely advance the development of DFER. Since the vanilla Vision Transformer (ViT) employed in VideoMAE requires substantial computation during fine-tuning, MAE-DFER develops an efficient local-global interaction Transformer (LGI-Former) as the encoder. Moreover, in addition to the standalone appearance content reconstruction in VideoMAE, MAE-DFER also introduces explicit temporal facial motion modeling to encourage LGI-Former to excavate both static appearance and dynamic motion information. Extensive experiments on six datasets show that MAE-DFER consistently outperforms state-of-the-art supervised methods by significant margins (e.g., +6.30\% UAR on DFEW and +8.34\% UAR on MAFW), verifying that it can learn powerful dynamic facial representations via large-scale self-supervised pre-training. Besides, it has comparable or even better performance than VideoMAE, while largely reducing the computational cost (about 38\% FLOPs). We believe MAE-DFER has paved a new way for the advancement of DFER and can inspire more relevant research in this field and even other related tasks. Codes and models are publicly available at https://github.com/sunlicai/MAE-DFER.

1 INTRODUCTION

DFER uses video sequences to capture temporal expression cues, but supervised progress is constrained by small labeled datasets and costly annotation. MAE-DFER uses unlabeled facial videos for self-supervised pretraining, pairing efficient local-global attention with joint appearance-motion reconstruction.

1 INTRODUCTION

  • Unlike static FER, DFER recognizes expressions from dynamic image sequences or videos, where temporal information is important for interpretation.Source: §1 · p. 1
  • Existing DFER datasets typically have around 10K training samples, and collecting and annotating larger high-quality datasets is time-consuming and labor-intensive.Source: §1 · pp. 1–2
  • LGI-Former uses local intra-region attention, global inter-region attention, and local-global interaction, while frame differences add a motion target alongside appearance reconstruction.The design replaces global space-time self-attention with staged local-global information exchange.Source: §1 · p. 2
  • Compared with VideoMAE, MAE-DFER has comparable or better performance while reducing FLOPs by about 38%.Source: §1 · p. 2
  • Pretraining on more than 1M unlabeled VoxCeleb2 clips is followed by fine-tuning on six in-the-wild and lab-controlled DFER datasets.Source: §1 · p. 2
  • +5∼8% UAR on three in-the-wild datasets and +7∼12% WAR on three lab-controlled datasets over the previous best supervised methods.The paper reports consistent gains across six DFER datasets.Source: §1 · p. 2

2 RELATED WORK

Prior DFER work has used convolutional, recurrent, and Transformer-based supervised approaches, while masked autoencoders provide a self-supervised pretraining route that MAE-DFER adapts for DFER.

2.1 Dynamic Facial Expression Recognition

  • DFER methods include 3D CNNs for joint spatiotemporal features, 2D CNNs combined with RNNs, and Transformer-based approaches that augment CNN/RNN models.Source: §2.1 · pp. 2–3
  • These DFER approaches remain supervised and are constrained by limited training samples, unlike MAE-DFER’s use of massive unlabeled facial video data.Source: §2.1 · p. 3

2.2 Masked Autoencoders

  • Masked autoencoders use mask-then-predict pretraining; MAE’s asymmetric encoder-decoder enabled efficient end-to-end training, and MAE-DFER builds on VideoMAE with two DFER-focused designs.Source: §2.2 · p. 3

3 METHOD

MAE-DFER adapts VideoMAE for DFER with joint appearance-and-motion reconstruction and an efficient LGI-Former encoder, addressing missing explicit motion modeling and costly ViT fine-tuning.

3.1 Revisiting VideoMAE

  • VideoMAE embeds video cubes, tube-masks 90% of tokens, encodes visible tokens, and reconstructs masked video with a lightweight decoder.The reconstruction loss is calculated between the original and reconstructed video at masked positions.Source: §3.1 · p. 3

3.2 MAE-DFER: Overview

  • MAE-DFER targets VideoMAE’s missing explicit motion modeling and costly ViT fine-tuning with joint reconstruction and LGI-Former.Source: §3.2 · p. 3

3.3 MAE-DFER: Joint Masked Appearance and Motion Modeling

  • A parallel motion branch reconstructs frame differences alongside appearance, sharing the decoder backbone but using separate heads and odd/even-frame targets.The total loss weights the two mean squared errors; the appearance target uses odd frames and the motion target uses even frames.Source: §3.3 · p. 3

3.4 MAE-DFER: Efficient LGI-Former

  • LGI-Former replaces global ViT attention with local-region attention, inter-region representative-token attention, then cross-attention that returns global information to local tokens.Representative tokens aggregate local information before exchanging information across regions; the figure depicts the three-stage flow.Source: §3.4 · pp. 4–5, Figure 2 · p. 4
  • LGI-Former’s block complexity is O((1/N)K^2), compared with O(K^2) for a standard ViT Transformer block.The authors state that M and N are much smaller than K in practice, reducing computational cost relative to ViT.Source: §3.4 · p. 5

4 RESULTS

Across six DFER benchmarks, MAE-DFER improves on supervised methods and is comparable to or better than VideoMAE with substantially lower fine-tuning computation. Ablations and visualizations examine its pre-training, architecture, joint appearance-motion modeling, and learned representations.

4.1 Datasets; 4.2 Implementation Details

  • Experiments use VoxCeleb2 development data for pre-training and evaluate on three in-the-wild and three lab-controlled datasets; clips generally use 16 frames at 160 × 160 resolution.The pre-training set contains 1,092,009 clips from 145,569 videos; most fine-tuning uses temporal stride 4, except FERV39k, which uses stride 1.Source: §4.1 · p. 5, §4.2 · p. 5

4.3 Ablation Studies

  • Longer pre-training generally helps until performance saturates around 50 epochs, while training from scratch performs nearly at random.The authors attribute the scratch result largely to limited labeled DFER samples and the data demands of large vision Transformers.Source: §4.3 · p. 5, Table 2 · p. 6
  • LGI-Former performs best when global inter-region attention and local-global interaction are combined, with about 38% fewer FLOPs than ViT and under 0.6% performance drop.Global inter-region attention adds about 0.1G FLOPs using eight representative tokens.Source: §4.3 · p. 6, Table 3 · p. 6
  • Joint masked appearance and motion modeling performs best around a loss weight of 0.5 and beats either standalone target on DFEW fold 1.The joint model exceeds appearance-only results by 1.69% UAR and 0.92% WAR, and motion-only results by 1.77% UAR and 1.02% WAR.Source: §4.3 · p. 6, Figure 3 · p. 6
  • Performance and computation vary little across local region sizes; the 2 × 5 × 10 setting with eight representative tokens gives the best performance-computation trade-off.This result is reported in Table 4.Source: §4.3 · p. 6, Table 4 · p. 6

4.4 Comparison with State-of-the-art Methods

  • 6.30% UAR and 5.18% WAR on DFEW, 5.36% UAR and 3.53% WAR on FERV39k, and 8.34% UAR and 6.13% WAR on MAFW are improvements over each dataset’s previous best supervised methods.The results are reported in Tables 5–7.Source: §4.4 · pp. 6–7, Table 5 · p. 7, Table 6 · p. 7
  • With VoxCeleb2 pre-training, MAE-DFER reduces fine-tuning computation by about 38% FLOPs versus VideoMAE, with comparable DFEW and FERV39k performance and better MAFW performance.The reported changes are −0.11% to 0.19% UAR and −0.17% to 0.32% WAR on DFEW and FERV39k, and +0.75% UAR and +0.80% WAR on MAFW.Source: §4.4 · pp. 7–8
  • On CREMA-D, RAVDESS, and eNTERFACE05, MAE-DFER reports larger improvements than on in-the-wild datasets, including over 12% UAR and 10% WAR over the best unimodal methods on CREMA-D.It also slightly exceeds the state-of-the-art multimodal method on CREMA-D, improves RAVDESS by over 12% WAR, and surpasses Graph-Tran by about 7% WAR on eNTERFACE05.Source: §4.4 · p. 8, Table 8 · p. 8

5 CONCLUSION

The paper presents MAE-DFER as a self-supervised framework for using unlabeled facial videos in DFER. It identifies scaling the data and models, and applying the approach to related tasks, as future directions.

5 CONCLUSION

  • The authors present MAE-DFER as a framework for exploiting unlabeled facial videos to address the limitations of supervised DFER methods.They describe it as a potential strong baseline for further DFER research.Source: §5 · p. 8
  • Future work will explore scaling MAE-DFER with more data and larger models, and applying it to dynamic micro-expression recognition and facial action unit detection.Source: §5 · p. 8

A DATASETS

The evaluation spans three large in-the-wild datasets and three small lab-controlled datasets, with each dataset differing in clip counts, labels, or experimental subset.

DFEW

  • DFEW contains 16,372 clips from over 1,500 movies, with seven basic-emotion labels; this paper uses 11,697 single-labeled clips.Source: A DATASETS · p. 11

FERV39k

  • FERV39k contains 38,935 real-world clips averaging 1.5 seconds, annotated with seven basic emotions.Source: A DATASETS · p. 11

MAFW

  • MAFW contains 10,045 clips annotated with 11 compound emotions; experiments use 9,172 single-labeled clips.Source: A DATASETS · p. 11

CREMA-D

  • CREMA-D is an audiovisual dataset of 7,442 clips labeled with six emotions.Source: A DATASETS · p. 11

RAVDESS

  • RAVDESS contains 2,880 audiovisual clips labeled with eight emotions, while this paper uses its 1,440-clip speech subset.Source: A DATASETS · p. 11

eNTERFACE05

  • eNTERFACE05 contains approximately 1,200 audiovisual clips simulating six emotions.Source: A DATASETS · p. 11

B MORE ABLATION STUDIES

The ablations examine model scaling, adding temporal facial motion modeling to VideoMAE, and the classification token used for downstream fine-tuning.

Model Size

  • Smaller LGI-Former versions moderately reduce performance, especially on FERV39k; even the tiny model outperforms supervised methods with similar parameter counts and computational cost.The small version has roughly half the base model's parameters and FLOPs, and the tiny version is similar in size to small.Source: B MORE ABLATION STUDIES · p. 11, Table 9 · p. 11

VideoMAE with Joint Masked Appearance and Motion Modeling

Role of Classification Token Type

  • Mean pooling representative tokens for final classification slightly outperforms mean pooling original tokens.The authors speculate that representative tokens are more compact and high-level.Source: B MORE ABLATION STUDIES · p. 11, Table 11 · p. 11

C DETAILED RESULTS

Fine-grained results show gains across most expressions, including rare classes, and comparable or better performance than VideoMAE under the same pre-training setting.

DFEW

MAFW

  • Over 5% on anger, 7% on disgust, 8% on contempt, 8% on anxiety, 6% on helplessness, and 7% on disappointment: MAE-DFER's gains over the best-performing supervised methods on MAFW.Source: C DETAILED RESULTS · p. 11, Table 14 · p. 12

Cross-dataset findings

  • MAE-DFER significantly outperforms state-of-the-art supervised methods on most facial expressions across DFEW, FERV39k, and MAFW, especially on rare expressions.Compared with VideoMAE pre-trained under the same setting, MAE-DFER has comparable or better fine-grained performance while reducing fine-tuning computation; Kinetics-400-pre-trained VideoMAE performs poorly on some rare expressions but best on some dominant ones.Source: C DETAILED RESULTS · p. 11, Table 12 · p. 12, Table 13 · p. 12

Related papers

Loading 2307.02227v2…