Source-linked AI summary
General Facial Representation Learning in a Visual-Linguistic Manner
Yinglin Zheng, Hao Yang, Ting Zhang, Jianmin Bao, Dongdong Chen, Yangyu Huang, Lu Yuan, Dong Chen, Ming Zeng, Fang Wen
TL;DR
Face analysis commonly relies on separate supervised models requiring costly manual annotations, motivating a transferable facial representation. FaRL combines image-text contrastive learning with masked image modeling and pre-trains on LAION-FACE. It transfers better than previous pre-trained models, including in low-data settings, and surpasses state-of-the-art methods on face parsing and face alignment.
Problem
Face analysis relies heavily on task-specific supervised training with costly manual labels, while transferable pre-training across diverse face tasks remains insufficiently explored.
Method
FaRL learns general facial representations by combining image-text contrastive learning for semantics with masked image modeling for complementary low-level information.
Results
FaRL achieves superior transfer performance across face analysis tasks, performs better in the low-data regime, and surpasses state-of-the-art methods on face parsing and face alignment.
Takeaways & Limitations
A single general facial representation can support multiple downstream face-analysis tasks and is particularly useful when labeled data or device resources are limited.
Takeaways & Limitations
FaRL exhibits some race-related bias and has not yet been adapted to face detection, face anti-spoofing, or face forgery detection.
Abstract
from arXiv · showhide
How to learn a universal facial representation that boosts all face analysis tasks? This paper takes one step toward this goal. In this paper, we study the transfer performance of pre-trained models on face analysis tasks and introduce a framework, called FaRL, for general Facial Representation Learning in a visual-linguistic manner. On one hand, the framework involves a contrastive loss to learn high-level semantic meaning from image-text pairs. On the other hand, we propose exploring low-level information simultaneously to further enhance the face representation, by adding a masked image modeling. We perform pre-training on LAION-FACE, a dataset containing large amount of face image-text pairs, and evaluate the representation capability on multiple downstream tasks. We show that FaRL achieves better transfer performance compared with previous pre-trained models. We also verify its superiority in the low-data regime. More importantly, our model surpasses the state-of-the-art methods on face analysis tasks including face parsing and face alignment.
1. Introduction
FaRL addresses the costly data demands of supervised face analysis by learning transferable facial representations from large-scale image-text pairs. It combines high-level semantic learning with low-level visual modeling and achieves stronger transfer across face tasks, including state-of-the-art results for face parsing and alignment.
- Motivation: Supervised face-analysis models are typically trained separately with large manually annotated datasets that are expensive and difficult to acquire.The data challenge is especially pronounced for face parsing and face alignment.
- Data: LAION-FACE provides 20 million face images filtered from a large openly available image-text dataset for FaRL pre-training.The dataset is constructed without manually designed face-related web queries.
- Approach: FaRL uses image-text contrastive learning for high-level semantics and masked image modeling for complementary low-level information.Contrastive learning aligns matched image-text embeddings and separates non-matched pairs, while masked modeling predicts visual tokens for masked patches.
- Results: FaRL achieves better transfer performance than previous pre-trained models and shows superiority in the low-data regime across several downstream face tasks.Evaluated tasks include face parsing, face alignment, and face attribute prediction.
- Approach: FaRL studies transfer performance across versatile face-analysis tasks using visual-linguistic pre-training, an area described as relatively unexplored.The framework is evaluated as a general facial representation rather than a task-specific model.
- Results: FaRL surpasses state-of-the-art methods on face parsing and face alignment.This is presented as the paper’s most important task-level result.
2. Related Works
Visual representation learning increasingly uses pre-training, including contrastive and visual-language objectives, while face analysis still relies heavily on task-specific supervised data. FaRL addresses this gap with a large face image-text dataset and a framework combining semantic and low-level representation learning.
- Visual Representation Pre-training: Pre-training has become a major strategy for visual representation learning, with contrastive methods and Transformer-based objectives achieving promising results.
- Visual-Language Pre-training: Visual-language pre-training learns from large, freely available image-text datasets and can improve transfer while reducing dependence on manually labeled data.
- Face Representation Learning: Face analysis methods commonly train separate supervised models for individual tasks using large manually annotated datasets, which are expensive and difficult to obtain.
- Face Representation Learning: Prior face transfer and few-shot studies generally target specific tasks, leaving weakly supervised pre-training across diverse face tasks comparatively unexplored.
- LAION-FACE: LAION-FACE contains 20 million face image-text pairs sampled from LAION after filtering images with RetinaFace detection scores above 0.9.
- FaRL Framework: FaRL combines image-text contrastive learning for high-level semantics with masked image modeling for complementary low-level information, using a shared image encoder for downstream tasks.
4. Experiments
Experiments evaluate frozen pretrained backbones across face parsing, alignment, and attribute recognition, using shared downstream settings and comparisons. FaRL shows consistent transfer advantages, strong few-shot performance, and state-of-the-art results on several face tasks.
- Experimental setup: Downstream heads combine features from selected intermediate and final Transformer layers rather than using only the last layer.The selected layers are K = {4, 6, 8, 12} for all downstream tasks.
- Experimental setup: All pretrained models use the same ViT-B/16 backbone structure and identical heads and training hyperparameters, differing only in backbone weights.This controls the comparison across pretrained Transformers.
- Pretrained Transformer comparison: FaRL consistently outperforms self-supervised, supervised, CLIP, and Face Transformer pretrainings across LaPa parsing, AFLW-19 alignment, and CelebA attributes.The comparison uses frozen backbones with only downstream heads fine-tuned.
- Few-shot evaluation: In few-shot evaluation, FaRL achieves the best results for LaPa parsing and CelebA attributes, but CLIP leads alignment with 1% and 10% training data.The authors associate CLIP’s alignment advantage with its much larger private pretraining dataset, but state this as a conjecture.
- Ablation studies: Adding MIM1 to ITC improves parsing and alignment but not attributes, while combining ITC, MIM1, and ALIGN becomes FaRL’s default configuration.A deeper MIM6 head performs worse than MIM1, and ALIGN is especially critical for CelebA.
- Ablation studies: Face-specific pretraining matters: LAION-FACE consistently beats equal-sized LAION-RANDOM, with face-image ratio especially important for attribute recognition.The authors suggest that higher-level facial attributes and identity require exposure to many face images during pretraining.
- Comparison with state-of-the-art methods: Fine-tuned FaRL448 reaches new state-of-the-art alignment results on AFLW-19 and WFLW, while vanilla FaRL consistently beats training from scratch.On 300W, FaRL448 is comparable with the cited prior method and does not assume landmark co-boundary relationships.
- Comparison with state-of-the-art methods: FaRL outperforms competing methods in all reported few-shot face-attribute settings and ranks second under full-shot evaluation.These comparisons use the CelebA and LFWA benchmarks.
5. Discussions
FaRL is presented as a general facial representation framework with strong downstream performance and practical benefits, while the authors acknowledge bias and task-coverage limitations.
- Benefits: FaRL provides a general facial representation that can support different face tasks through a shared feature extraction module.The authors highlight its usefulness for resource-restricted mobile devices.
- Benefits: Joint contrastive learning and masked image modeling make the representation applicable to both low-level and high-level face tasks.The contrastive component captures semantics, while masked image modeling harnesses low-level information.
- Benefits: The framework achieves satisfactory results using 20M samples, fewer than general image pre-training approaches using hundreds of millions.The authors state that this smaller pre-training scale may facilitate adoption.
- Bias analysis: FairFace is balanced across race, gender, and age, with gender divided into 2 groups, race into 7, and age into 9.The analysis uses FairFace as a face attribute dataset.
- Limitations: FaRL exhibits race-group performance gaps, although its gap values are relatively moderate among the compared methods.The authors associate such gaps with possible social bias.
- Limitations: The model remains biased to a certain degree and has not yet been adapted to face detection, anti-spoofing, or forgery detection.The authors identify dataset bias and face-detector performance bias as possible sources.
6. Conclusion
The paper proposes FaRL, which combines image-text contrastive learning with masked image modeling to learn a general facial representation. FaRL transfers well across face analysis tasks, outperforming previous pre-trained models and surpassing state-of-the-art methods on face parsing and alignment.
- FaRL combines image-text contrastive learning and masked image modeling to learn a more general facial representation.
- FaRL transfers well to face parsing, face alignment, and face attribute recognition.
- FaRL achieves superior transfer performance compared with previous pre-trained models.
- FaRL surpasses state-of-the-art methods on face parsing and face alignment.
A. Evaluation on Face Editing Tasks.
The paper evaluates FaRL in text-driven face editing by replacing the pre-trained CLIP component with an equal-size FaRL model. Figure 4 presents the comparison between FaRL and CLIP.
- FaRL replaces the pre-trained CLIP in a text-driven face editing framework for visual-language reasoning.The replacement uses models with equal size.
- Figure 4 compares FaRL with CLIP in text-driven face editing.
B. Visualizing the Pre-trained Image Encoder
Grad-CAM visualizations examine how FaRL’s pre-trained image encoder responds to different text queries. The encoder localizes corresponding image regions, showing high correlation with human attention.
- Grad-CAM visualizations are generated for FaRL’s image encoder under different text queries.Gradients are calculated at the output of the first LayerNorm within the last Transformer block.
- FaRL’s image encoder successfully localizes regions corresponding to the query texts.The visualizations show high correlation with human attention.
C. Features on Different Backbone Levels
Features from different backbone levels support different downstream face tasks, while fusing multiple levels performs best overall. Deep features favor attributes recognition, whereas an intermediate level favors face parsing.
- Experimental setup: Single-level backbone features show divergent performance patterns across face parsing and face attributes recognition.The experiment replaces multi-level features with repeated features from one level while keeping the backbone frozen.
- Task-dependent feature semantics: Deep level-9 features are most effective for face attributes recognition on CelebA, but perform poorly on face parsing.The most effective face-parsing feature is instead on the 5-th level.
- Multi-level fusion: 92.32 F1-mean on LaPa and 91.39 mAcc on CelebA are achieved by multi-level feature fusion.Fusion outperforms all single-level settings, indicating complementary information across levels.
- Visualization: Grad-CAM visualizations examine EI responses under different text queries using gradients from the first LayerNorm in the final Transformer block.This visualization probes the model's query-dependent activations.
- Evaluation: Figure 6 reports the performance of different backbone levels on LaPa face parsing and CelebA face attributes recognition.The two tasks provide the comparison axes for evaluating level-specific features.
D. Ratios of Face Images in Pre-training
Increasing the proportion of face images in pre-training generally improves downstream face-task performance, with larger gains for attributes recognition than for parsing or alignment. The pattern is consistent with different tasks requiring different semantic information.
- Overall trend: Higher face-image ratios generally produce better downstream performance across face tasks.All compared pre-training datasets contain 20M images and differ in their face-image proportions.
- Task differences: The gains are significant for face attributes recognition but subtle for face parsing and face alignment.The paper relates this difference to the semantic requirements of the downstream tasks.
- Semantic interpretation: High-level facial semantics needed for attributes recognition can only be learned on face images.Low-level semantics such as corners and edges can be learned from face and non-face images.
- Experimental design: Table 11 presents downstream performances for different face-image ratios using image-text contrastive learning only.This isolates the effect of face-image proportion within the stated experiment.
- Comparison with general-image pre-training: FaRL's advantages over Transformers pre-trained on general images are smaller for face parsing and alignment than for attributes recognition.The cited comparison includes ImageNet- and WIT-pre-trained Transformers.
E. Comparison with Self-supervised Methods on Face Dataset
FaRL is compared with self-supervised methods under matched face-data, network, and fine-tuning conditions, and it performs better on all evaluated tasks. Longer pre-training provides additional benefits, while implementation and augmentation details define the evaluation setting.
- Self-supervised comparison: FaRL achieves better performance than SwAV and SimCLR on all evaluated face tasks.The comparison uses LAION-Face, the same network structure, and the same fine-tuning strategy.
- Self-supervised comparison: SwAV on face images is equivalent to Bulat because Bulat adopts SwAV for face pre-training.This identifies the relationship between the compared self-supervised baseline and the cited prior method.
- Pre-training duration: Additional benefits are observed from longer FaRL pre-training.The longer-pretraining fine-tuning results are reported in Table 13.
- Computational setting: During pre-training, FaRL has an extra text encoder and an additional masked image modeling stage compared with CLIP.These additions generally double computational complexity relative to CLIP, while remaining comparable to self-supervised contrastive methods.
- Downstream evaluation: α = 0.8 is selected as the default warping setting for face parsing.The cited table evaluates F1 scores under different warping factors.
- Downstream evaluation: On CelebAMask-HQ, the alignment matrix is replaced by simple resizing and Tanh-warping is disabled because the faces are already aligned.The remaining augmentations are retained and applied consistently to other pre-trained models for fair comparison.
H. Data Usage
The paper documents the sources, sizes, and usage restrictions of datasets used in its face-analysis evaluations. Several datasets are limited to research or non-commercial use, while WFLW has no stated license.
- Dataset licensing: LAION contains 400M Internet-collected image-text pairs and is licensed under Creative Commons CC-BY 4.0.The dataset providers do not claim copyright of the images.
- Dataset licensing: LaPa contains over 22K face images and is available for academic and non-commercial purposes under its stated license.The license permits uses including research, teaching, scientific publications, and personal experimentation.
- Dataset licensing: AFLW licensing information is referenced through an external download page.The cited passage provides the URL but no license terms.
- Dataset licensing: WFLW contains about 10K face images and does not mention any licenses.The passage provides no further usage terms.
- Dataset licensing: 300W contains over 4K face images and prohibits commercial use, including training commercial algorithms.The data are provided for research purposes only.
- Dataset sizes: CelebA contains 202,599 images and LFWA contains 13,143 images; CelebA is available for non-commercial research purposes only.LFWA is based on the original LFW dataset.
I. Code Usage
The code uses publicly available pretrained models and supporting libraries for downstream evaluation, with licensing and model variants documented for reproducibility.
- ViT and DeiT pretrained Transformers are loaded through the timm library using specified base patch16 configurations.The ViT checkpoint is vit base patch16 224 in21k, while DeiT uses deit base distilled patch16 224.
- MoCo v3 weights are converted to DeiT format before loading through timm.The downloaded ViT-Base model is converted using the provided script.
- The cited codebases and model weights carry varied licenses, including Apache 2.0, CC-BY-NC 4.0, MIT, and an unspecified license for FaceTransformer.The MoCo v3 repository is identified by a footnote URL, while FaceTransformer is reported without a license.
- BEiT-base and CLIP ViT-B/16 serve as pretrained model variants.
- FaceTransformer uses the ViT-P8S8 variant, while RetinaFace provides face detection.
- MMSegmentation supplies the UperNet implementation for downstream tasks including face parsing and face alignment.