PaperlayerView exact revision on arXiv

Source-linked AI summary

FaceXFormer: A Unified Transformer for Facial Analysis

Kartik Narayan, Vibashan VS, Rama Chellappa, Vishal M. Patel

arXiv:2403.12960v3cs.CV

TL;DR

FaceXFormer addresses the difficulty of integrating specialized facial-analysis systems into one efficient pipeline. It uses a transformer encoder-decoder with task tokens and a lightweight bi-directional cross-attention decoder, achieving competitive multi-task performance at 33.21 FPS. The paper demonstrates real-time unified facial analysis while noting that distinct tasks can require conflicting preprocessing.

  • Problem

    Specialized facial-analysis architectures and preprocessing techniques limit integration, while deploying multiple models is computationally intensive for real-time use.

    Source: §1 · p. 1

  • Method

    FaceXFormer uses a transformer encoder-decoder with one learnable token per task and a lightweight FaceX decoder that jointly processes face and task tokens through bi-directional cross-attention.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    FaceXFormer achieves state-of-the-art or competitive performance across multiple benchmarks for ten tasks while operating at 33.21 FPS.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    A single unified model can provide broad facial-analysis coverage with real-time inference and reduced computational cost compared with prior multi-task models.

    Source: §1 · p. 2, §5.3 · p. 8, §6 · p. 8

    The tasks can require distinct and sometimes conflicting preprocessing steps, such as alignment for landmarks versus natural head-pose variability.

    Source: §3.4 · p. 5

Abstract

from arXiv · show

In this work, we introduce FaceXFormer, an end-to-end unified transformer model capable of performing ten facial analysis tasks within a single framework. These tasks include face parsing, landmark detection, head pose estimation, attribute prediction, age, gender, and race estimation, facial expression recognition, face recognition, and face visibility. Traditional face analysis approaches rely on task-specific architectures and pre-processing techniques, limiting scalability and integration. In contrast, FaceXFormer employs a transformer-based encoder-decoder architecture, where each task is represented as a learnable token, enabling seamless multi-task processing within a unified model. To enhance efficiency, we introduce FaceX, a lightweight decoder with a novel bi-directional cross-attention mechanism, which jointly processes face and task tokens to learn robust and generalized facial representations. We train FaceXFormer on ten diverse face perception datasets and evaluate it against both specialized and multi-task models across multiple benchmarks, demonstrating state-of-the-art or competitive performance. Additionally, we analyze the impact of various components of FaceXFormer on performance, assess real-world robustness in "in-the-wild" settings, and conduct a computational performance evaluation. To the best of our knowledge, FaceXFormer is the first model capable of handling ten facial analysis tasks while maintaining real-time performance at 33.21 FPS. Code: https://github.com/Kartik-3004/facexformer

1. Introduction

FaceXFormer addresses the difficulty of integrating specialized facial-analysis systems by unifying ten tasks in one transformer model. Its lightweight FaceX decoder supports robust multi-task representations and real-time performance while retaining competitive benchmark results.

1. Introduction

  • Specialized models are difficult to integrate because they use task-specific designs and preprocessing, while deploying them together increases complexity and computational cost.Source: §1 · pp. 1–2
  • FaceXFormer represents each task with a learnable token in a transformer encoder-decoder, enabling simultaneous processing of ten facial-analysis tasks.Source: Figure 1 · p. 1, §1 · p. 2
  • FaceX is a lightweight decoder using bi-directional cross-attention to jointly process face and task tokens and learn generalized facial representations.Source: §1 · p. 2
  • FaceXFormer achieves state-of-the-art or competitive performance across multiple benchmarks while supporting more tasks than previous multi-task models.Source: §1 · p. 2
  • 33.21 FPS enables FaceXFormer to perform ten facial-analysis tasks in real time, with a 69.44% speed boost over prior multi-task models.Source: §1 · p. 2

2. Related Work

Prior facial-analysis systems are specialized or computationally heavy when unified, motivating a lightweight multi-task transformer. FaceXFormer uses bi-directional cross-attention and a two-layer decoder to improve efficiency while handling multiple tasks.

2. Related Work

  • Specialized models excel at designated tasks but require task-specific preprocessing and generally underperform outside their specialization.Source: §2 · p. 2
  • Earlier multi-task frameworks use auxiliary tasks or learnable-token architectures, but QFace and Faceptor employ heavier feature-fusion and decoder designs.Source: §2 · p. 3
  • FaceXFormer replaces those heavier designs with bi-directional cross-attention and a two-layer lightweight decoder for efficient task-specific feature extraction.Source: §2 · p. 3
  • Unified transformer models can be competitive across tasks but are often computationally intensive for facial-analysis applications requiring real-time performance.Source: §2 · p. 3
  • FaceXFormer is presented as the first lightweight transformer-based model for multiple facial-analysis tasks, operating at 33.21 FPS.Source: §2 · p. 3

3. FaceXFormer

FaceXFormer unifies ten facial analysis tasks through a transformer encoder-decoder that combines multi-scale face features with task-specific tokens. FaceX, its lightweight decoder, jointly models task and face tokens before unified task heads produce predictions.

3. FaceXFormer

  • FaceXFormer combines multi-scale face features with learnable task tokens in a unified encoder-decoder for ten facial analysis tasks.The task tokens are processed with the fused face representation and then passed to corresponding prediction heads.Source: §3 · p. 3

3.1. Multi-scale Encoder

  • Multi-scale encoding produces coarse-to-fine features, while MLP-Fusion aggregates them into a unified representation with 983k parameters.This design supports differing feature requirements, such as global information for age estimation and fine-grained information for face parsing.Source: §3.1 · pp. 3–4

3.2. FaceX Decoder

  • FaceX models task interactions and bidirectional task-face information flow through task self-attention, task-to-face cross-attention, and face-to-task cross-attention.The inverse attention step augments the face representation with task-specific details, supporting unified facial representations.Source: §3.2 · pp. 4–5
  • FaceX is a lightweight decoder designed to reduce computational cost while jointly processing face and task tokens for real-time facial analysis.The decoder is described as a two-layer lightweight module that models relationships among task tokens and face tokens.Source: §3.2 · pp. 4–5

3.3. Unified-Head

  • The unified head converts refined tokens into task predictions using task-specific heads, including hourglass, regression, classification, recognition, and segmentation components.Token counts and head types vary by task, including 68 landmark tokens and 9 head-pose tokens representing a 3×3 rotation matrix.Source: §3.3 · p. 5

3.4. Multi-Task Training

  • Multi-task training uses task-specific tokens and combines distinct losses to accommodate conflicting preprocessing requirements across facial analysis tasks.The objective includes Dice and cross-entropy losses for parsing, STAR loss for landmarks, geodesic loss for pose, ArcFace loss for recognition, and task-specific classification losses.Source: §3.4 · p. 5

4. Experiments and Results

FaceXFormer is trained and evaluated across ten facial-analysis datasets using task-specific metrics, with comparisons against specialized and multi-task models. It achieves strong multi-task results, real-time performance, and robust predictions on challenging in-the-wild images.

4.2. Implementation Details

  • The models are trained with ImageNet-initialized backbones at 224 × 224 resolution using distributed PyTorch training on eight A6000 GPUs.Training uses AdamW, 12 epochs, task-balanced upsampling, and augmentation including blur, grayscale, occlusion, flipping, and affine transformations.Source: §4.2 · pp. 5–6

4.3. Main results

  • FaceXFormer achieves state-of-the-art or competitive performance across multiple facial-analysis tasks while unifying evaluation across ten task-specific datasets.The evaluation covers parsing, landmarks, head pose, attributes, expression, age/gender/race, recognition, and visibility using standard task metrics.Source: §4.1 · p. 5, §4.3 · p. 6
  • 92.01 mean F1 on CelebAMaskHQ establishes state-of-the-art face parsing at 224 × 224 resolution, half the input size required by other state-of-the-art methods.The result is reported in the comparative analysis of the main task benchmarks.Source: §4.3 · p. 6, Table 3 · p. 6
  • FaceXFormer reports mean MAE 3.52 for head pose, mean NME 4.67 for landmarks, 91.83% attribute accuracy, and 72.56% visibility accuracy.It also achieves 4.17 for age estimation, 88.24% expression-recognition accuracy, and 95.94% face-recognition accuracy versus Faceptor’s 95.28%.Source: §4.3 · p. 6, Table 5 · p. 7, Table 6 · p. 7
  • 33.21 FPS supports real-time multi-task inference, while FaceXFormer outperforms prior multi-task models across segmentation, pose, landmarks, attributes, and visibility.The comparison also reports second-best performance in age estimation and competitive results under the broader ten-task setting.Source: §4.3 · p. 6

4.4. Qualitative “in-the-wild” results

  • On randomly selected in-the-wild images, FaceXFormer handles segmentation, pose, landmarks, age, gender, race, and attributes under extreme poses, occlusions, and blur.The model can also generate multiple annotations for each image for downstream use.Source: §4.4 · p. 7

5. Ablation Study

The ablations identify multi-scale fusion and bi-directional cross-attention as important design choices, while additional analyses examine bias and computational efficiency. FaceXFormer combines strong cross-task behavior with faster, lower-cost inference than Faceptor.

5. Ablation Study

  • The ablation study evaluates component effects across tasks, while the broader study also examines backbone size, demographics, and computational performance.Additional backbone comparisons are provided in Appendix C.Source: §5 · p. 7

5.1. Impact of various components in FaceXFormer

  • Bi-directional cross-attention improves the standard-cross-attention decoder by 1.11 MAE in head pose, 0.63 NME in landmarks, 1.85 attribute-accuracy points, and 0.67 MAE in age.Removing decoder cross-attention produces extremely poor performance, while removing MLP fusion also reduces results.Source: §5.1 · pp. 7–8, Table 7 · p. 7

5.2. Bias Analysis and Ethical Considerations

  • FaceXFormer shows the smallest performance discrepancy across racial groups and minimal bias compared with other models on FairFace.The analysis notes that the training datasets are imbalanced across age, gender, and race, while FairFace provides balanced evaluation groups.Source: §5.2 · p. 8, Table 8 · p. 8

5.3. Computational Performance Analysis

  • 33.2 FPS in FP32 and 100.1 FPS in FP16 make FaceXFormer the fastest evaluated multi-task face-analysis model.It requires 114 GFLOPs versus Faceptor’s 167 GFLOPs and reduces latency from 69.9 to 30.1 ms in FP32 and from 23.7 to 10.0 ms in FP16.Source: §5.3 · p. 8, Table 9 · p. 8
  • The lightweight FaceX decoder enables two decoder layers while maintaining effective face-feature extraction and reducing computational cost.The paper attributes the lower latency and GFLOPs to the bi-directional cross-attention design.Source: §5.3 · p. 8

6. Conclusion

FaceXFormer concludes that a unified transformer can handle diverse facial-analysis tasks in real time through task tokens and a lightweight FaceX decoder. The model is positioned as both a multi-task analysis system and a tool for expanding facial datasets and existing applications.

6. Conclusion

  • FaceXFormer integrates ten facial-analysis tasks through a transformer encoder-decoder with learnable task tokens and achieves 33.21 FPS.FaceX uses bi-directional cross-attention to learn generalized face representations across tasks.Source: §6 · p. 8
  • The paper identifies surveillance, subject analysis, and image retrieval as application areas for the real-time unified model.These uses are described as broader applications of FaceXFormer’s task-specific query outputs.Source: §6 · p. 8, B. Discussion · p. 15

A. Overview

  • The appendix extends the paper with broader impact, ablations, cross-dataset evaluation, in-the-wild visualizations, and dataset details.These extensions are listed as supplementary sections accompanying the main paper.Source: A. Overview · p. 15

B. Discussion

  • The model can serve as an annotator for large-scale face datasets and provide additional information to existing facial-analysis systems.The discussion connects this use to the shortage of broadly annotated datasets, including missing annotations in WebFace42M.Source: B. Discussion · p. 15

C. Ablation Study

The ablation compares backbone architectures across segmentation, regression, classification, and inference speed to identify a suitable FaceXFormer backbone.

Experimental Setup

Backbone Comparison

  • Swin Transformer achieves the strongest regression and classification results, with a mean error of 4.12 and mean accuracy of 90.03%.These results motivate selecting Swin Transformer as FaceXFormer’s backbone.Source: C. Ablation Study · p. 15
  • ConvNeXt achieves the best segmentation performance, reaching an F1 score of 92.08%.Source: C. Ablation Study · p. 15
  • MobileNet records the lowest reported metrics, including an F1 score of 91.21% and a mean error of 4.64.The passage attributes this limitation to MobileNet’s smaller receptive field compared with Swin Transformer.Source: C. Ablation Study · p. 15

D. Cross-Dataset Evaluation

Cross-dataset testing evaluates FaceXFormer on unseen, unconstrained faces, while qualitative web images probe robustness under occlusion, extreme pose, and accessories.

D. Cross-Dataset Evaluation

E. In-the-wild Visualization

F. Datasets and Implementation Details

FaceXFormer is trained across datasets with unequal sizes using balanced sampling and a shared optimization setup, including CelebAMask-HQ for face parsing.

F. Datasets and Implementation Details

  • Upsampling smaller datasets ensures that each training batch includes samples from every dataset, reducing domination by larger datasets.Training uses 12 epochs, batch size 384, AdamW, and a learning rate decayed at epochs 6 and 10.Source: F. Datasets and Implementation Details · p. 16

F.1. Face Parsing

  • CelebAMask-HQ provides 30,000 high-resolution images with 19 annotated classes for face parsing, resized to 224 × 224 during training.The listed classes include facial regions such as skin, face, nose, eyes, eyebrows, lips, and mouth.Source: §F.1 · p. 16

F.2. Landmarks Detection

Landmark detection uses 300W for training and evaluation, 300VW for cross-dataset testing, and augmentations covering geometric, photometric, blur, and occlusion variations.

F.2. Landmarks Detection

  • 300W contains 3,148 training images and 689 test images with 68 annotated landmarks, while 300VW supplies three cross-dataset test categories.The 300VW categories span well-lit, mildly unconstrained, and challenging video conditions.Source: §F.2 · p. 16
  • Training augmentation applies rotation, scaling, translation, flipping, grayscale, blur, occlusion, gamma adjustment, and five-landmark image alignment.Source: §F.2 · pp. 16–17

F.3. Head Pose Estimation

The paper evaluates facial-analysis tasks using task-specific datasets, preprocessing choices, and augmentation strategies across head pose, attributes, demographics, expressions, recognition, and visibility.

F.3. Head Pose Estimation

  • Head pose estimation trains on 300W-LP and evaluates on BIWI, spanning approximately ±75° yaw and ±60° pitch.Training uses loosely cropped faces with grayscale, blur, resized-crop, and gamma augmentations.Source: §F.3 · p. 17

F.4. Attributes Prediction

  • Attributes prediction trains on CelebA and evaluates cross-dataset on LFWA, using 40 binary facial-attribute labels.Training applies geometric, grayscale, blur, and gamma augmentations.Source: §F.4 · p. 17

F.5. Age/Gender/Race Estimation

  • Age, gender, and race estimation uses FairFace, UTKFace, and MORPH-II with cross-dataset testing on FFHQ and age evaluation on MORPH-II.The study uses five race labels, decade-based age bins, and two gender labels.Source: §F.5 · p. 17

F.6. Facial Expression Recognition

  • Facial expression recognition trains on RAF-DB and AffectNet and evaluates within RAF-DB across diverse subjects, poses, lighting conditions, and occlusions.RAF-DB contains seven basic emotions, while AffectNet contains eight discrete expressions.Source: §F.6 · p. 17

F.7. Face Recognition

  • Face recognition trains on approximately 5.1 million MS1MV3 images covering 93,000 identities and evaluates verification under pose and age variation.Evaluation uses LFW, CFP-FP, AgeDB, CALFW, and CPLFW, with five-keypoint alignment and no training augmentation.Source: §F.7 · pp. 17–18

F.8. Visibility Prediction

  • Visibility prediction uses COFW, where 29 facial landmarks each have binary visibility annotations, with loosely cropped faces and basic augmentations.Training applies horizontal flipping, grayscale conversion, Gaussian blur, and gamma adjustment.Source: §F.8 · p. 18
Loading 2403.12960v3…