PaperlayerView exact revision on arXiv

Source-linked AI summary

SwinFace: A Multi-task Transformer for Face Recognition, Expression Recognition, Age Estimation and Attribute Estimation

Lixiong Qin, Mei Wang, Chao Deng, Ke Wang, Xi Chen, Jiani Hu, Weihong Deng

arXiv:2308.11509v1cs.CV

TL;DR

Prior transformer-based methods typically target a single face task, while face-analysis tasks have limited training data and separate networks are inefficient. SwinFace shares a Swin Transformer across recognition and grouped analysis subnets, with MLCA for task-specific feature selection. Experiments report state-of-the-art results for facial expression recognition and age estimation.

  • Problem

    Prior transformer-based models typically target one task, while face-analysis tasks have limited training data and separate networks for different tasks are inefficient.

    Source: §I · p. 1

  • Method

    SwinFace shares a Swin Transformer backbone across face recognition and grouped face-analysis subnets, using MLCA to adaptively select features from different levels and channels.

    Source: §I · pp. 1–2, 3) Face Analysis Subnets · p. 4

  • Results

    Experiments report state-of-the-art results for facial expression recognition and age estimation, including 90.97% accuracy on RAF-DB and 0.22 test ϵ-error on CLAP2015.

    Source: §I · p. 2, §V · p. 10

  • Takeaways & Limitations

    The conclusion emphasizes robust face-recognition initialization for facial expression recognition and age estimation.

    Source: §V · p. 10

    The current model does not include localization tasks such as head pose estimation, face alignment, face parsing, or 3D face reconstruction.

    Source: §V · p. 10

Abstract

from arXiv · show

In recent years, vision transformers have been introduced into face recognition and analysis and have achieved performance breakthroughs. However, most previous methods generally train a single model or an ensemble of models to perform the desired task, which ignores the synergy among different tasks and fails to achieve improved prediction accuracy, increased data efficiency, and reduced training time. This paper presents a multi-purpose algorithm for simultaneous face recognition, facial expression recognition, age estimation, and face attribute estimation (40 attributes including gender) based on a single Swin Transformer. Our design, the SwinFace, consists of a single shared backbone together with a subnet for each set of related tasks. To address the conflicts among multiple tasks and meet the different demands of tasks, a Multi-Level Channel Attention (MLCA) module is integrated into each task-specific analysis subnet, which can adaptively select the features from optimal levels and channels to perform the desired tasks. Extensive experiments show that the proposed model has a better understanding of the face and achieves excellent performance for all tasks. Especially, it achieves 90.97% accuracy on RAF-DB and 0.22 $\epsilon$-error on CLAP2015, which are state-of-the-art results on facial expression recognition and age estimation respectively. The code and models will be made publicly available at https://github.com/lxq1000/SwinFace.

I. INTRODUCTION

The paper targets data sparsity and model inefficiency in face analysis by sharing a transformer across face recognition and analysis tasks. It introduces SwinFace with task-grouped subnets and MLCA, and reports state-of-the-art results for facial expression recognition and age estimation.

I. INTRODUCTION

  • Prior transformer-based models typically target one task, leaving face-analysis tasks with limited training data and no knowledge sharing from face recognition.The paper identifies memory and inference inefficiency from learning separate networks for different tasks.Source: §I · p. 1
  • The model combines a shared Swin Transformer backbone with a face-recognition subnet and grouped face-analysis subnets.The design aims to share representations across related tasks to improve performance and reduce training time.Source: §I · pp. 1–2
  • MLCA combines features from different levels and adaptively weights them, allowing analysis tasks to select features suited to their needs and mitigating task conflict.The paper contrasts this adaptive selection with methods that fuse empirically selected layers.Source: §I · p. 2
  • SwinFace is presented as the first single-transformer model to jointly handle face recognition and 42 face-analysis tasks.The analysis tasks are grouped into 11 subnets, with related tasks sharing a subnet to reduce computation.Source: §I · p. 2
  • 90.97% accuracy on RAF-DB and 0.20 validation and 0.22 test ϵ-error on CLAP2015 are reported as state-of-the-art results for expression recognition and age estimation, respectively.The CLAP2015 values refer to its validation and test sets.Source: §I · p. 2

A. Face Recognition

The related work reviews methods for face recognition, expression recognition, age and attribute estimation, and multi-task learning. It describes gaps in task synergy and feature selection that motivate SwinFace’s approach.

A. Face Recognition

  • Face recognition methods use margin-based softmax losses and large in-the-wild datasets; transformer-based work also leaves downstream analysis benefits from recognition largely unexplored.The paper describes million-scale training and margin-based softmax loss as the current state-of-the-art approach, and cites ViT advantages on larger datasets such as WebFace42M.Source: A. Face Recognition · p. 2

B. Facial Expression Recognition

  • Facial expression recognition faces inter-class similarity, intraclass variation, and label uncertainty; prior work addresses these with specialized losses, attention, or synthesis rather than broad task synergy.The paper distinguishes its target-conflict motivation from AMP-Net’s focus on robustness in real-world scenarios.Source: B. Facial Expression Recognition · pp. 2–3

C. Age Estimation

  • Age estimation spans classification, regression, ranking, and distribution learning, while the paper argues that face-recognition initialization has been under-emphasized.The cited methods generally pre-train on IMDB-WIKI; the authors report that their model performed well without dedicated structures for age inhomogeneity,.Source: C. Age Estimation · p. 3

D. Face Attribute Estimation

  • Face-attribute methods exploit attribute relationships or auxiliary tasks, and the paper also groups attributes to improve performance and reduce computation.Source: D. Face Attribute Estimation · p. 3

E. Multi-task Learning

  • Multi-task learning is described as a route to efficiency and performance gains, while SwinFace differs from HyperFace [14] and AIO by adaptively selecting feature levels rather than fusing empirically chosen ones.The paper’s longer-term goal is a generic model for face-related tasks, with localization tasks and semi-supervised learning identified as future directions.Source: E. Multi-task Learning · pp. 3–4

III. METHOD

SwinFace uses a shared hierarchical Swin Transformer to provide features for recognition and analysis subnets. The recognition subnet uses only the top-level feature map to form its embedding.

A. Overall Architecture

  • A single Swin Transformer extracts shared feature maps at different levels for a face-recognition subnet and 11 face-analysis subnets.The analysis subnets process groups of related tasks.Source: A. Overall Architecture · p. 4

1) Shared Backbone:

  • The shared backbone processes 112 × 112 × 3 face crops into four feature maps, from 28×28×192 through 7×7×768, using patch partition, embedding, patch merging, and Swin blocks.The four maps are denoted FM1 to FM4.Source: 1) Shared Backbone · p. 4

2) Face Recognition Subnet:

  • Face recognition uses only FM4 and produces a 512-D embedding through a structure including BN, similar to ArcFace.The paper reports that an FC-BN-FC-BN recognition subnet outperforms its counterpart without that structure in Section IV-D.Source: 2) Face Recognition Subnet · p. 4

3) Face Analysis Subnets:

The 42 face-analysis tasks are grouped into 11 related-task subnets, each using MLCA to select useful features and address conflicts among tasks. MLCA combines information across feature levels and uses channel attention to weight the resulting features.

3) Face Analysis Subnets:

  • Each of 11 face-analysis subnets handles a group of related tasks and includes MLCA, pooling, ReLU, and fully connected layers.Grouping tasks lets related analyses share a subnet and reduces computation.Source: 3) Face Analysis Subnets · p. 4, Table II · p. 5

B. Multi-Level Channel Attention

  • MLCA is integrated into each analysis subnet to address conflicting task targets through multi-level feature fusion and channel attention.The method is motivated by conflicts such as identity-invariant face recognition versus expression recognition that ignores identity.Source: B. Multi-Level Channel Attention · pp. 4–5

1) Multi-Level Feature Fusion:

  • MLFF aligns feature-map scales, reduces channels with convolutions, and concatenates features from four levels into a 512-dimensional map.Average pooling downsamples FM1 and FM2; the combined representation draws on both local and global face information.Source: 1) Multi-Level Feature Fusion · p. 5, Figure 3 · p. 5

2) Channel Attention:

  • CA derives channel weights from average- and max-pooled descriptors passed through a shared MLP, then applies the sigmoid attention vector to the feature map.The weighted map emphasizes channels according to the task group's preferences.Source: 2) Channel Attention · p. 5, Figure 4 · p. 5

C. Training

Training shares the Swin Transformer backbone across face recognition and 42 analysis tasks, using datasets divided into four categories by label type. The described recognition and expression training use CosFace and merged expression datasets, respectively.

C. Training

  • The multi-task phase shares the backbone between face recognition and 42 face-analysis tasks, with training datasets divided into four categories by label type.The system begins from a pretrained model containing the backbone and face-recognition subnet.Source: C. Training · pp. 5–6, Table III · p. 6

1) Face Recognition:

  • Face recognition is trained on MS-Celeb-1M with CosFace; its normalized embeddings are scaled by 64 and use a margin penalty of 0.4.The implementation also uses PFC with a sampling ratio of 0.3 to conserve computing resources.Source: 1) Face Recognition · p. 6

2) Facial Expression Recognition:

  • Basic expression recognition merges AffectNet and RAF-DB by labels into seven classes, and uses labeled-sample class probabilities for training.The classes are surprise, fear, disgust, happiness, sadness, anger, and neutral.Source: 2) Facial Expression Recognition · p. 6

3) Age Estimation:

Age estimation is trained as regression using three datasets and a loss combining Gaussian and Euclidean terms. Face-attribute estimation comprises 40 binary classification tasks, with additional age-dataset labels used for gender recognition.

3) Age Estimation:

  • Age estimation is formulated as regression and trained with IMDB+WIKI, Adience, and MORPH using a linear combination of Gaussian and Euclidean loss.The loss weight λ starts at 0 and increases to 1; σ is fixed at 3 when the dataset does not provide it.Source: 3) Age Estimation · p. 6

4) Face Attribute Estimation:

  • Face-attribute estimation comprises 40 binary classification problems trained on CelebA, while gender training also uses IMDB+WIKI, Adience, and MORPH labels.For each attribute, the training description defines a binary presence label and predicted probability.Source: 4) Face Attribute Estimation · p. 6

5) Total Loss function:

The paper combines task-specific losses into a weighted total loss and describes the datasets used to train and evaluate face recognition, expression recognition, and age estimation.

5) Total Loss function:

  • The final loss is a weighted sum of the individual task losses, with each task weight set to 1.0 in the reported experience.The task set includes recognition, expression, age, and attribute tasks.Source: 5) Total Loss function · p. 6

1) Face Recognition:

  • Face recognition is pretrained and evaluated on six verification benchmarks, including cross-pose, cross-age, and challenging unconstrained-face protocols.The benchmarks are LFW, CFP-FP, AgeDB-30, CALFW, CPLFW, and IJB-C.Source: 1) Face Recognition · p. 6, Table IV · p. 7

2) Facial Expression Recognition:

  • Expression recognition training merges AffectNet and RAF-DB by labels, balancing AffectNet’s larger sample count against RAF-DB’s more accurate labels.Overall accuracy is reported on the RAF-DB test set.Source: 2) Facial Expression Recognition · p. 6

3) Age Estimation:

  • Age estimation uses IMDB+WIKI, Adience, and MORPH for training, then reports performance on both CLAP2015 validation and test splits.Adience age groups are converted to regression labels using each group’s average age.Source: 3) Age Estimation · pp. 6–7

4) Face Attribute Estimation:

The CelebA attribute evaluation preserves hair and neck information, while the implementation uses a Swin-T backbone and separately initialized analysis subnets.

4) Face Attribute Estimation:

  • CelebA testing uses 19,962 images, and gender training also draws labels from IMDB+WIKI, Adience, and MORPH.Face alignment is not done in the recognition manner because CelebA includes hair and neck attributes.Source: 4) Face Attribute Estimation · p. 7

B. Implementation Details

  • The Swin-T backbone has 28.5M parameters; the recognition subnet has about 1M excluding PFC, and each analysis subnet has about 3.5M.Training uses four NVIDIA Tesla T4 GPUs; the backbone and recognition subnet are pretrained before 11 analysis subnets are randomly initialized.Source: B. Implementation Details · p. 7

1) Pre-training:

  • Pretraining runs for 40 epochs with AdamW, cosine decay, five warm-up epochs, and horizontal-flip augmentation.The initial learning rate is 5×10^-4 and batch size is 512.Source: 1) Pre-training · p. 7

2) Multi-task Learning:

  • Multi-task training runs for 80k steps, sampling 128 examples from each of four categories per step and 256 gender examples.Non-recognition samples receive horizontal flips, Randaugment, and Random Erasing; recognition samples receive horizontal flips only.Source: 2) Multi-task Learning · p. 7

C. Performance Evaluation

SwinFace reports strong performance across face recognition, expression recognition, age estimation, and face attribute estimation. Results include benchmark comparisons and task-specific accuracy or error margins.

1) Face Recognition:

  • SwinFace outperforms Swin-T on every face-recognition benchmark and other models in almost all reported protocols, despite a much smaller backbone.The comparison uses models trained on MS-Celeb-1M.Source: 1) Face Recognition · p. 7

2) Facial Expression Recognition:

  • 90.97% accuracy on RAF-DB exceeds the reported state of the art using standard cross-entropy, while the expression subnet totals 32M backbone and subnet parameters.The comparison with TransFER notes that it uses 65.2M parameters and experimentally selects third-stage features; SwinFace selects levels adaptively.Source: 2) Facial Expression Recognition · pp. 7–8

3) Age Estimation:

  • 0.45 (0.30) MAE margin and 0.06 (0.03) ϵ-error margin over conventional algorithms are reported on CLAP2015 validation (test) splits.The age subnet uses MLCA; ϵ-error accounts for the standard deviation of each sample’s age annotations.Source: Table VI · p. 8, 3) Age Estimation · p. 8

4) Face Attribute Estimation:

  • 91.32% average accuracy on CelebA is reported as comparable to other state-of-the-art methods, with alignment preserving hair and neck information.The paper relates this result to generalization across faces of different scales.Source: 4) Face Attribute Estimation · p. 8, Figure 5 · p. 8

D. Ablation Study

The ablations report benefits from multi-task training, face-recognition initialization, and multi-level channel attention across face-analysis tasks. Face-recognition initialization improves results on RAF-DB and CLAP2015, while MLCA's largest reported gain is for facial expression recognition.

1) Multi-task Framework:

  • Multi-task training improves age estimation over single-task learning by a 0.039 ϵ-error margin, while “Smiling” and “Young” accuracy rise by 0.78 and 0.67.Table VIII compares single-task and multi-task training for analysis tasks.Source: 1) Multi-task Framework · p. 8, Table VIII · p. 9

2) Model Initialization from Face Recognition Task:

  • 0.025 lower ϵ-error on CLAP2015 validation and 4.6 higher RAF-DB accuracy result from initialization with face-recognition pretraining rather than ImageNet-1K.The CLAP2015 validation evaluation does not fine-tune on the CLAP2015 training set.Source: 2) Model Initialization from Face… · p. 8, Table IX · p. 9

3) Multi-Level Channel Attention:

4) Importance of Feature Maps for Different Subnets:

Feature-map importance varies across analysis subnets, while the face-recognition experiments favor CosFace and an FC-BN-FC-BN design. Running all 43 outputs takes 36% more time than calculating the recognition feature alone; the conclusion reports state-of-the-art results for expression recognition and age estimation.

4) Importance of Feature Maps for Different Subnets:

  • Deeper feature maps generally receive higher importance, but the top layer is not always most useful; the third map benefits the six-task face-attribute subnet most.The six tasks are Attractive, Blurry, Chubby, Heavy Makeup, Oval Face, and Pale Skin.Source: 4) Importance of Feature Maps for… · p. 10

5) Loss Function Selection and Subnet Design for Face Recognition:

  • CosFace performs better than ArcFace for Swin Transformer-based face recognition, and the FC-BN-FC-BN structure also contributes to performance improvement.The experiments use CASIA-WebFace, which contains 494K images of 10,572 celebrities.Source: 5) Loss Function Selection and Subnet… · p. 10

6) Model Running Efficiency:

  • 3.205ms calculates the recognition feature, versus 4.357ms for all 43 outputs, a 36% time-overhead increase.Measurements use one NVIDIA Tesla T4 GPU with batch size 32.Source: 6) Model Running Efficiency · p. 10

V. CONCLUSION

  • SwinFace performs face recognition, expression recognition, age estimation, and attribute estimation simultaneously; the authors report state-of-the-art results for expression recognition and age estimation.The conclusion emphasizes robust face-recognition initialization for the latter two analysis tasks.Source: §V · p. 10
  • The model's functionality remains limited; extending it to localization tasks is future work, and pseudo-labeling is proposed as a possibility but was not validated.Suggested extensions include head-pose estimation, face alignment, face parsing, and 3D face reconstruction.Source: §V · p. 10
Loading 2308.11509v1…