Source-linked AI summary
Efficient Masked Face Recognition Method during the COVID-19 Pandemic
Walid Hariri
TL;DR
Masked faces make recognition difficult because masks hide facial regions and reduce the effectiveness of existing methods. The paper removes the masked region, extracts features from visible areas with pretrained CNNs, quantizes them with Bag-of-Features, and classifies identities using an MLP, outperforming other state-of-the-art methods in accuracy and time complexity.
Problem
Masks hide substantial facial regions, making face authentication, access control, and existing recognition methods difficult or ineffective.
Method
The method crops masked faces to retain forehead and eye regions, extracts features using pretrained CNNs, quantizes last-convolutional-layer maps with Bag-of-Features, and uses an MLP for classification.
Results
The proposed method outperformed other state-of-the-art methods in recognition accuracy and time complexity.
Takeaways & Limitations
Using only unmasked regions and lightweight Bag-of-Features representations supports high generalization and applicability to real-time applications.
Takeaways & Limitations
The approach assumes that occlusions occur in one predictable facial region, primarily the nose and mouth.
Abstract
from arXiv · showhide
The coronavirus disease (COVID-19) is an unparalleled crisis leading to a huge number of casualties and security problems. In order to reduce the spread of coronavirus, people often wear masks to protect themselves. This makes face recognition a very difficult task since certain parts of the face are hidden. A primary focus of researchers during the ongoing coronavirus pandemic is to come up with suggestions to handle this problem through rapid and efficient solutions. In this paper, we propose a reliable method based on occlusion removal and deep learning-based features in order to address the problem of the masked face recognition process. The first step is to remove the masked face region. Next, we apply three pre-trained deep Convolutional Neural Networks (CNN) namely, VGG-16, AlexNet, and ResNet-50, and use them to extract deep features from the obtained regions (mostly eyes and forehead regions). The Bag-of-features paradigm is then applied to the feature maps of the last convolutional layer in order to quantize them and to get a slight representation comparing to the fully connected layer of classical CNN. Finally, Multilayer Perceptron (MLP) is applied for the classification process. Experimental results on Real-World-Masked-Face-Dataset show high recognition performance compared to other state-of-the-art methods.
1 Introduction
COVID-19 mask wearing creates security and authentication challenges because much of the face is hidden. The paper addresses masked face recognition by extracting features from unmasked facial regions.
- Face recognition avoids physical contact, but masks make authentication and access control difficult by hiding a large part of the face.
- The paper distinguishes face mask recognition from masked face recognition, focusing on recognizing masked faces using the eyes and forehead.
- The proposed task uses a deep learning-based model to extract features from facial regions outside the mask.
- The paper presents related work, motivation, the proposed method, experiments, and conclusions in successive sections.
2 Related works
Prior work on face recognition under occlusion includes matching, restoration, and occlusion removal approaches. This paper adopts occlusion removal with deep CNN features for masked face recognition.
- Occlusion is a key limitation of real-world 2D face recognition, and masks are a particularly difficult occlusion because they hide much of the face.
- Matching approach: Matching approaches compare local patches or keypoint features between probe and gallery faces while avoiding overlap with occluded regions.
- Restoration approach: Restoration approaches estimate or reconstruct occluded facial regions using gallery information, PCA, statistical shape models, or ICP.
- Occlusion removal approach: Occlusion removal approaches detect occluded regions and discard them before feature extraction and classification.
- Deep CNN methods can support occluded-face recognition by focusing on non-occluded features or discarding masked-region elements.
- The paper proposes combining occlusion removal with a deep CNN-based model to address masked face recognition during the COVID-19 pandemic.
3 Motivation and contribution of the paper
The paper proposes efficient masked-face recognition by cropping away masked regions, extracting deep CNN features, and quantizing them with Bag-of-Features. The design targets lightweight representations and practical recognition.
- Motivation and contribution: The method localizes the mask, crops informative forehead and eye regions, and extracts their features with a pre-trained deep learning model.
- Motivation and contribution: Fully connected CNN layers require millions of parameters, powerful hardware, and substantial processing capacity and memory.
- Motivation and contribution: The proposed quantization-based pooling method uses feature maps from the last convolutional layers of three pre-trained models through Bag-of-Features.
- Motivation and contribution: Bag-of-Features extracts local features, quantizes them into a codebook, and represents test images with histograms of assigned codes.
- Motivation and contribution: Deep quantization provides lightweight representations and supports classification of informative masked-face images with different sizes.
- Motivation and contribution: The differentiable quantization scheme jointly trains the quantizer and network, without training on the removed mask region.
4 The proposed method
The proposed method normalizes and crops masked-face images to retain non-masked regions, extracts deep features with pretrained CNNs, quantizes them using a BoF layer, and classifies identities with an MLP.
- Preprocessing: Face images are normalized to 240 × 240 pixels, divided into 100 fixed-size 24 × 24 blocks, and restricted to blocks 1–50 containing non-masked regions.
- Feature extraction: Three pretrained CNNs—VGG-16, AlexNet, and ResNet-50—extract deep features from the selected informative regions.The method uses the last convolutional feature maps for VGG-16, the fifth convolutional layer for AlexNet, and the last convolutional layer for ResNet-50.
- Deep bag of features: The BoF layer uses RBF neurons to measure similarity between extracted feature vectors and learned codeword centers.The RBF centers are initialized through k-means and subsequently learned to obtain final codewords.
- Deep bag of features: A quantization layer aggregates RBF outputs into a histogram representing each face as a global quantized feature vector.The histogram uses a predefined number of bins, with each bin corresponding to a codeword.
- Classification: The resulting term vector is passed to an MLP classifier to assign each test image to a face identity.The experiments use backpropagation and gradient descent, with 10-fold cross-validation on RMFRD.
5 Experimental results
Experiments on RMFRD and SMFRD evaluate quantized deep features from masked-face regions using three pre-trained CNNs and MLP classification. Performance varies by dataset and architecture, while comparisons report advantages in recognition performance and computation time.
- Datasets: RMFRD contains 5,000 masked and 90,000 unmasked images of 525 subjects, whereas SMFRD contains 500,000 simulated masked faces of 10,000 subjects.RMFRD is imbalanced and was oversampled by cropping non-masked faces before feature extraction.
- Experimental setup: The pipeline extracts last-convolutional-layer features from VGG-16, AlexNet, and ResNet-50, quantizes them with codebooks, and classifies faces using MLP.Evaluation used repeated 10-fold cross-validation on both datasets.
- RMFRD results: 91.3% was the best RMFRD recognition rate, achieved by VGG-16 using the third feature maps and 60 codewords.AlexNet reached 86.6% with 100 codewords, while ResNet-50 reached 89.5% with 70 codewords.
- SMFRD results: 88.9% was the highest SMFRD recognition rate, achieved by ResNet-50 using quantized DRF features, 70 codewords, and an MLP classifier.AlexNet’s highest rate was 86.0%, compared with 85.6% for VGG-16.
- State-of-the-art comparison: The proposed method outperformed the tested transfer-learning and covariance-based recognizers on RMFRD and SMFRD.The comparison included transfer learning with fine-tuned pre-trained models and covariance descriptors built from 2D features.
- Efficiency: The Bag-of-Features paradigm reduced deep-feature extraction and classification time compared with a method using the same pre-trained models on RMFRD.AlexNet had lower training and testing time than VGG-16 and used less GPU memory.
- Discussion: The authors attribute the method’s accuracy and efficiency to last-convolutional-layer features, lightweight BoF representations, and processing only unmasked regions.They state that this design supports applicability in real-time applications.
6 Conclusion
The paper proposes a deep learning and quantization-based method for masked-face recognition using unmasked regions and MLP classification. It reports improved accuracy and time complexity relative to other state-of-the-art methods, with future extension to ensemble models.
- Method: The method uses three pre-trained models’ last convolutional layers, Bag-of-Features quantization, and an MLP classifier for masked-face recognition.
- Conclusion: The proposed method improves face-recognition generalization in the presence of masks.
- Conclusion: The method outperformed other state-of-the-art methods in accuracy and time complexity.
- Future work: Future work considers deep ensemble models with additional pre-trained models to enhance accuracy.