Source-linked AI summary
Masked Face Recognition for Secure Authentication
Aqeel Anwar, Arijit Raychowdhury
TL;DR
Masked faces degrade facial-recognition authentication while suitable masked training data is unavailable. The paper introduces MaskTheFace to augment existing datasets with synthetic masks and retrain Facenet. The retrained system improves true positive rate and achieves similar accuracy on the real-world MFR2 dataset.
Problem
Masked faces make systems trained on unoccluded images difficult to recognize reliably, while masked-face training data is unavailable.
Method
MaskTheFace applies configurable masks to existing face datasets, which are used to retrain an existing Facenet recognition system.
Results
∼38% increase in true positive rate is reported for the existing Facenet system across masked and unmasked faces.
Takeaways & Limitations
The retrained system also achieves similar accuracy when evaluated on the real-world MFR2 masked-face dataset.
Abstract
from arXiv · showhide
With the recent world-wide COVID-19 pandemic, using face masks have become an important part of our lives. People are encouraged to cover their faces when in public area to avoid the spread of infection. The use of these face masks has raised a serious question on the accuracy of the facial recognition system used for tracking school/office attendance and to unlock phones. Many organizations use facial recognition as a means of authentication and have already developed the necessary datasets in-house to be able to deploy such a system. Unfortunately, masked faces make it difficult to be detected and recognized, thereby threatening to make the in-house datasets invalid and making such facial recognition systems inoperable. This paper addresses a methodology to use the current facial datasets by augmenting it with tools that enable masked faces to be recognized with low false-positive rates and high overall accuracy, without requiring the user dataset to be recreated by taking new pictures for authentication. We present an open-source tool, MaskTheFace to mask faces effectively creating a large dataset of masked faces. The dataset generated with this tool is then used towards training an effective facial recognition system with target accuracy for masked faces. We report an increase of 38% in the true positive rate for the Facenet system. We also test the accuracy of re-trained system on a custom real-world dataset MFR2 and report similar accuracy.
I. INTRODUCTION
Face masks undermine existing facial-recognition authentication because systems trained on unoccluded faces struggle with occluded identities and masked-face training data is unavailable. The paper introduces MaskTheFace to generate masked datasets from existing face data for retraining.
- Motivation: Face masks make existing facial-recognition systems inefficient because they are commonly trained on unoccluded faces.Such systems learn features including the eyes, nose, lips, and face edges, which are partly occluded by masks.
- Motivation: The paper identifies the unavailability of masked-face training data as the central problem.
- Contributions: MaskTheFace is an open-source tool for generating masked-face datasets from existing face datasets.
- Contributions: The paper also introduces MRF2, a small real-world dataset of aligned masked faces.
- Tool design: MaskTheFace uses facial landmarks and face tilt to select and transform an appropriate mask template.
- Tool design: The tool supports five mask types, custom masks and patterns, multiple faces, varied face angles, and bulk dataset masking.
II. EXPERIMENTATION
The experimentation evaluates whether retraining a face-recognition system with masked images can recover performance lost on masked-face recognition.
- Experimental objective: The experiments examine degradation on masked images and evaluate a retraining-based solution using selected data, metrics, and a face-recognition system.
A. Face recognition system:
The paper uses Facenet, which represents faces as embeddings and makes identity decisions by comparing distances between image representations. Training uses triplet loss to separate same-identity from different-identity pairs.
- Face recognition system: Facenet maps faces to unified embeddings and compares their distances for identity decisions.
- Face recognition system: During training, triplets contain an anchor, a positive same-identity image, and a negative different-identity image.
- Face recognition system: Triplet loss imposes a distance margin that separates positive pairs from negative pairs.
- Face recognition system: Online triplet mining generates training triplets during optimization to minimize triplet loss.
B. Training data:
Training compares Facenet networks trained on unmasked and synthetically masked versions of VGGFace2-mini, while testing uses separate datasets. The synthetic masking nearly doubles the training dataset without adding new source-image information.
- Training data: VGGFace2 contains about 3 million images of 9131 identities, with pose, age, ethnicity, and illumination variation.
- Training data: VGGFace2-mini samples 42 images per identity to create an unmasked training subset.
- Training data: VGGFace2-mini-SM applies randomly selected surgical-green, surgical-blue, N95, and cloth masks, nearly doubling the dataset size.
- Training data: The study trains one Facenet on VGGFace2-mini and another on VGGFace2-mini-SM for comparison.
- Dataset roles: Table I summarizes four datasets, with the first two used for training and the last two used for testing.
- Training data: The authors consider the comparison fair because the additional masked images are generated from the original images and add no extra source information.
C. Performance Metrics:
The paper evaluates face-recognition networks using maximum accuracy, accuracy at FAR=0.1%, and true positive rate at FAR=0.1%. These metrics emphasize both overall pair classification and performance under a constrained false-acceptance rate.
- Max Accuracy measures the network’s maximum accuracy when classifying test image pairs as positive or negative.
- ACC @ FAR=0.1% measures pair-classification accuracy at the threshold where the false acceptance rate is 0.1%.
- TPR @ FAR=0.1% measures the percentage of positive pairs correctly identified at the threshold where the false acceptance rate is 0.1%.
- MaxAccuracy is computed as (TP + TN) / (TP + TN + FP + FN).
- ACC@FAR0.1% and TPR@FAR0.1% impose FP / (TP + FP) = 0.001, with TP, TN, FP, and FN denoting true and false positives and negatives.
III. RESULTS
The results evaluate no-mask and mask networks across unmasked and simulated-mask variants of LFW. The LFW-SM-Mixed dataset extends LFW with randomly selected simulated masks for detailed pairwise testing.
- III. RESULTS: The evaluation compares trained no-mask and mask networks using the previously defined performance metrics on several test datasets.
- LFW datasets: LFW contains 5,749 identities and 13,233 images and serves as the standard benchmark underlying the masked-face variations.
- LFW datasets: The tested variants include original LFW, surgical-green, surgical-blue, N95, cloth, and randomly masked images.
- LFW datasets: LFW-SM contains only images with simulated masks, and its variations support detailed performance analysis.
- Evaluation protocol: The Facenet evaluation selects an L2-distance threshold using nine dataset splits and reports metrics on the tenth, following a leave-one-out approach.
- LFW-SM-Mixed: LFW-SM-Mixed contains 5,749 identities, 64,973 images, and 29,235 pairs generated with surgical, cloth, and N95 masks.
B. MFR2 - Dataset with real masks
MFR2 is a small, pre-processed real-world masked-face dataset created to evaluate systems trained on simulated masks. It provides a real-image test of transfer beyond the simulated mask types.
- Motivation: The paper evaluates networks trained on simulated masks against real-world masked faces because a diverse, aligned, ready-to-use dataset was unavailable.
- Dataset construction: MFR2 contains 53 celebrity and politician identities with 269 internet-collected images, and it is intended to demonstrate simulated-mask effectiveness rather than replace the Wuhan dataset.
- Dataset construction: Each identity has an average of five images, and the dataset includes both masked and unmasked faces processed for alignment and fixed image dimensions.
- Evaluation protocol: The evaluation uses 848 MFR2 image pairs: 424 positive pairs and 424 negative pairs.
A. LFW-SM Dataset
On LFW-SM, the no-mask network degrades substantially for masked inputs under a fixed false-acceptance constraint. The mask-trained network performs more consistently across cross-dataset pair combinations.
- Maximum accuracy: ∼97% is the best pair-wise maximum accuracy for pairs where both template and unknown images are unmasked; other combinations range from 86 to 93%.
- Accuracy at fixed FAR: At FAR=0.1%, accuracy drops to 76 to 85% when masked faces are presented to the network.
- Accuracy at fixed FAR: The worst accuracy occurs when the template is unmasked and the unknown image is N95-masked.
- True positive rate: The no-mask-trained network’s worst-case TPR@FAR=0.1% falls from ∼86% to ∼30%.
- Network comparison: The mask-trained network performs almost equally across cross-dataset pair combinations and slightly outperforms the no-mask network in some cases.
- Network comparison: The mask network significantly outperforms the no-mask network across all three reported performance metrics on average.
B. MFR2 dataset
On the MFR2 real-world dataset, the mask network produced more consistent identity clusters than the no-mask network, with comparable accuracy despite previously unseen mask appearances.
- The MFR2 masks included cloth masks with textures and colors absent from training, yet performance remained comparable to LFW-SM.
- 34%, 17% and 6% improvements were observed in TPR@FAR=0.2%, Accuracy @ FAR=0.2% and maximum accuracy, respectively.
- The trained network showed a 2 to 4% decrease in performance when tested on real-world images.
- The mask network produced mostly correct MFR2 identity clusters, with only a few images mapped to wrong identities.The no-mask network showed several discrepancy types, including split identities and merged clusters.
- Training data combined randomly selected cloth, surgical-green, surgical-blue and N95 masks with the original un-masked images.Including original images was intended to support performance on both masked and un-masked inputs.
VI. CONCLUSION
The paper presents MaskTheFace as an open-source tool for generating masked-face data to improve recognition through existing systems. The retrained Facenet system improved true positive rate and showed similar accuracy on the MFR2 real-world dataset.
- The paper addresses masked-face recognition through existing face recognition systems with reliable accuracy.
- MaskTheFace generates a large masked-face dataset that can be used to train an effective facial recognition system.
- 38% increase in true positive rate was reported for the existing Facenet system on both masked and un-masked faces.
- The retrained system achieved similar accuracy when tested on the custom real-world MFR2 dataset.
- MaskTheFace and the MFR2 dataset are available to other researchers through the project repository.