Source-linked AI summary
CASIA-SURF CeFA: A Benchmark for Multi-modal Cross-ethnicity Face Anti-spoofing
Ajian Li, Zichang Tan, Xuan Li, Jun Wan, Sergio Escalera, Guodong Guo, Stan Z. Li
TL;DR
Face anti-spoofing lacks sufficient evidence for studying ethnic bias because existing datasets generally lack ethnic labels and have limited modality and attack coverage. The paper introduces CeFA and a static-dynamic, partially shared multi-modal baseline, with experiments reporting strong performance across several face anti-spoofing benchmarks. Its visual analysis also identifies difficult cases, especially 3D attacks whose depth resembles real faces.
Problem
Ethnic bias remains an open face anti-spoofing problem, while existing datasets lack ethnic labels and provide limited modalities and attack types.
Method
The paper releases CeFA with four evaluation protocols and proposes PSMM-Net, combining static-dynamic processing within modalities with partially shared multi-modal fusion.
Results
The proposed method achieves state-of-the-art results on CASIA-SURF, OULU-NPU, SiW, and CeFA.
Takeaways & Limitations
CeFA provides a public benchmark for evaluating cross-ethnicity, cross-attack, and cross-modality face anti-spoofing generalization.
Takeaways & Limitations
Depth images of 3D attacks can be extremely similar to real faces, creating difficult misclassified cases.
Abstract
from arXiv · showhide
Ethnic bias has proven to negatively affect the performance of face recognition systems, and it remains an open research problem in face anti-spoofing. In order to study the ethnic bias for face anti-spoofing, we introduce the largest up to date CASIA-SURF Cross-ethnicity Face Anti-spoofing (CeFA) dataset (briefly named CeFA), covering $3$ ethnicities, $3$ modalities, $1,607$ subjects, and 2D plus 3D attack types. Four protocols are introduced to measure the affect under varied evaluation conditions, such as cross-ethnicity, unknown spoofs or both of them. To the best of our knowledge, CeFA is the first dataset including explicit ethnic labels in current published/released datasets for face anti-spoofing. Then, we propose a novel multi-modal fusion method as a strong baseline to alleviate these bias, namely, the static-dynamic fusion mechanism applied in each modality (i.e., RGB, Depth and infrared image). Later, a partially shared fusion strategy is proposed to learn complementary information from multiple modalities. Extensive experiments demonstrate that the proposed method achieves state-of-the-art results on the CASIA-SURF, OULU-NPU, SiW and the CeFA dataset.
1 Introduction
Face anti-spoofing protects face-recognition systems, but ethnic bias remains underexplored because existing datasets lack ethnic labels and offer limited modalities and attack diversity. CeFA addresses this gap with a large, cross-ethnicity benchmark and a partially shared multi-modal baseline.
- Motivation: Ethnic bias remains an open face anti-spoofing problem, and existing datasets lack ethnic labels for evaluating it.Existing datasets also provide limited samples and mostly RGB data.
- Motivation: CASIA-SURF provides a large dataset but covers only one ethnicity and 2D print attacks.
- Dataset contribution: CeFA is released as a large cross-ethnicity face anti-spoofing dataset spanning ethnicities, modalities, subjects, and diverse attack types.Its attacks include cloth printing, video replay, 3D print, and silica gel attacks.
- Dataset contribution: CeFA is presented as the first public dataset designed to study cross-ethnicity effects in face anti-spoofing.
- Method contribution: PSMM-Net combines modality interaction with static-dynamic processing in each modality to address ethnic and attack-pattern bias.The dynamic image is generated by rank pooling.
2 Related work
Prior face anti-spoofing work includes static, temporal, and multi-modal approaches, but existing methods face limitations with replay attacks, handcrafted features, and restricted modality interaction. PSMM-Net instead uses modality-specific and shared networks to learn fused features.
- Dataset limitations: Earlier datasets were small, mostly RGB-only, and lacked publicly available ethnic labels for face anti-spoofing.
- Static and temporal methods: Static and constrained temporal methods can become vulnerable to replay attacks.
- Static and temporal methods: Handcrafted temporal features limited the accuracy of some face anti-spoofing methods.Other work used CNN-RNN or spatio-temporal attention to model temporal and spatial information.
- Multi-modal fusion: Earlier multi-modal methods primarily concatenated modality features, whereas PSMM-Net connects modality-specific and shared networks through a partially shared structure.The proposed structure is designed to learn discriminative fused features.
3 CeFA dataset
CeFA combines synchronized RGB, Depth, and IR capture with 2D and 3D attacks across three ethnicities. Its four protocols evaluate cross-ethnicity, cross-PAI, cross-modality, and combined generalization under subject-disjoint splits.
- Acquisition: CeFA captures RGB, Depth, and IR videos simultaneously at 30fps with 1280 × 720 resolution per frame.Subjects move their heads to provide varied pose views.
- Dataset statistics: The 2D subset covers print and video-replay attacks across three ethnicities, with 500 subjects per ethnicity and 18,000 videos overall.Each subject has one real sample, two print samples, and one video-replay sample.
- Dataset statistics: The 3D subset includes 3D print masks and silica gel face attacks captured under multiple lighting conditions.The 3D print-mask subset contains 5,346 videos.
- Evaluation protocols: CeFA defines four evaluation protocols for cross-ethnicity, cross-PAI, cross-modality, and cross-ethnicity with PAI generalization.The protocols comprise 11 sub-protocols for the 2D attack subset.
4 Proposed Method
The method combines static and rank-pooled dynamic information within each modality, then partially shares features across RGB, Depth, and infrared networks. PSMM-Net uses modality-specific SD-Nets and a shared branch to learn complementary multimodal features while preserving static-dynamic representations.
- 4.1 SD-Net for Single Modality: Rank pooling maps K consecutive RGB, Depth, or infrared frames to a single dynamic vector or dynamic image.Applied directly to frame pixels, the resulting dynamic image retains the input-frame size.
- 4.1 SD-Net for Single Modality: SD-Net processes each modality with static, dynamic, and static-dynamic branches that learn complementary features.The network uses ResNet-18, with separate static and dynamic pathways and a fused static-dynamic branch.
- 4.1 SD-Net for Single Modality: The SD-Net objective sums static, dynamic, static-dynamic, and combined-feature losses to jointly optimize complementary representations.Each branch has an independent binary cross-entropy loss, alongside a loss on summed features.
- 4.2 PSMM-Net for Multi-modal Fusion: PSMM-Net combines three modality-specific SD-Nets with a shared branch that learns complementary information across RGB, Depth, and infrared modalities.The shared branch uses ResNet-18 and exchanges information with the modality-specific networks.
- 4.2 PSMM-Net for Multi-modal Fusion: Static and dynamic features from all modality branches are fused into shared blocks, then shared features are fed back to the SD-Nets.This exchange occurs only for static and dynamic features to avoid disturbing hybrid static-dynamic representations with multimodal semantics.
- 4.2 PSMM-Net for Multi-modal Fusion: PSMM-Net training combines modality-specific losses for color, depth, and infrared with an overall loss based on summed network features.The total objective includes Lcolor, Ldepth, Lir, and Lwhole.
5 Experiments
Experiments evaluate ethnic bias, cross-condition generalization, and the proposed fusion components across CeFA and established benchmarks. Results show that static-dynamic features, three-modality fusion, and CeFA pretraining improve anti-spoofing performance, while cross-ethnicity and cross-attack settings remain challenging.
- Ethnic bias: 11.4% versus 19.6% ACER for MS-SEF across East Asian and Central Asian subjects, respectively, exposing ethnic performance disparities.The corresponding African-versus-Central Asian comparison for MS-SEF is 14.2% versus 26.1%.
- CeFA benchmark evaluation: 24.5%, 43.2%, and 27.7% ACER are obtained for Protocol 4 sub-protocols, making simultaneous cross-ethnicity and cross-PAI evaluation the most difficult setting.Protocol 2 also varies substantially across attack conditions, from 0.4% to 7.5% ACER.
- Static-dynamic ablation: 12.6%, 6.1%, and 6.4% ACER for SD-Net on RGB, Depth, and IR outperform both static-only and dynamic-only branches.Relative to S-Net, the improvements are 4.6%, 1.6%, and 3.4%; relative to D-Net, they are 7.3%, 3.3%, and 4.9%.
- Multi-modal ablation: 2.2% APCER, 2.2% BPCER, and 2.2% ACER are achieved when PSMM-Net uses RGB, Depth, and IR together.The reported result attributes the benefit partly to smaller inter-ethnicity skin-color differences in infrared imagery.
- Fusion ablation: PSMM-Net outperforms naive halfway fusion and its no-backward-feeding variant through information exchange between SD-Nets and the shared branch.The comparison evaluates NHF, PSMM-Net-WoBF, and PSMM-Net as distinct fusion strategies.
- Methods comparison: At least 0.4% ACER improvement over competing CASIA-SURF fusion methods is reported, while CeFA pretraining raises TPR@FPR = 10^-4 by 2.4%.On SiW, the proposed SD-Net achieves the best results on all protocols; on OULU-NPU, it is best on protocols 3 and 4.
6 Conclusion
The paper releases CeFA, a large multi-modal face anti-spoofing dataset with ethnic labels, and introduces four protocols plus a partially shared baseline for studying generalization and bias.
- CeFA is the largest face anti-spoofing dataset up to date in modalities, subject count, and attack types.
- CeFA is the only public face anti-spoofing dataset with ethnic labels.
- Four protocols assess face anti-spoofing generalization under varied conditions.
- The PSMM-Net baseline learns complementary information from multi-modal video data using partially shared fusion and single-modality static-dynamic feature learning.