PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-restrained Triplet Loss for Accurate Masked Face Recognition

Fadi Boutros, Naser Damer, Florian Kirchbuchner, Arjan Kuijper

arXiv:2103.01716v3cs.CV

TL;DR

Face masks pose a challenge for recognition systems, which are mostly designed for unoccluded faces. The paper applies EUM and SRT atop existing models; evaluations report improved performance in most settings across three models and four masked datasets.

  • Problem

    Face masks significantly degrade recognition accuracy compared with unmasked faces, while most systems are designed for unoccluded faces.

    Source: §I · p. 1

  • Method

    EUM maps masked-face embeddings toward same-identity unmasked embeddings, with SRT guiding training by dynamically focusing on genuine-pair distances.

    Source: §I · p. 1, §III · p. 2, 1) Self-restrained Triplet Loss · p. 4

  • Results

    Across most experimental settings, the approach improved verification; with masked probes on MobileFaceNet, FMR100 measures fell by ∼28% and 26% on two real masked datasets.

    Source: §I · p. 1

  • Takeaways & Limitations

    The evaluations span three face-recognition models and four datasets, including real in-the-wild and collaborative datasets and synthetic masked-face datasets.

    Source: G. Discussion · p. 13

Abstract

from arXiv · show

Using the face as a biometric identity trait is motivated by the contactless nature of the capture process and the high accuracy of the recognition algorithms. After the current COVID-19 pandemic, wearing a face mask has been imposed in public places to keep the pandemic under control. However, face occlusion due to wearing a mask presents an emerging challenge for face recognition systems. In this paper, we present a solution to improve masked face recognition performance. Specifically, we propose the Embedding Unmasking Model (EUM) operated on top of existing face recognition models. We also propose a novel loss function, the Self-restrained Triplet (SRT), which enabled the EUM to produce embeddings similar to these of unmasked faces of the same identities. The achieved evaluation results on three face recognition models, two real masked datasets, and two synthetically generated masked face datasets proved that our proposed approach significantly improves the performance in most experimental settings.

I. INTRODUCTION

Face masks impair recognition systems largely designed for unoccluded faces. The paper adds an embedding-space model and loss atop existing recognizers, reporting improved verification in most experimental settings.

I. INTRODUCTION

  • Masks significantly degrade face-recognition accuracy versus unmasked faces, despite face recognition’s contactless capture, high accuracy, and broad deployment,,.Source: §I · p. 1
  • EUM transforms masked-face embeddings toward same-identity unmasked embeddings atop existing recognizers, while SRT focuses on genuine-pair distance once impostor separation is sufficient.The approach avoids modifying or retraining the existing face-recognition model.Source: §I · p. 1
  • Across most settings, EUM with SRT improved verification; with masked probes on MobileFaceNet, FMR100 measures fell by ∼28% and 26% on the two real masked datasets.FMR100 is the lowest false non-match rate for false match rate ≤1.0%.Source: §I · p. 1

II. RELATED WORK

  • Earlier occlusion studies addressed sunglasses or scarves,,,, while mask-specific evaluations found degradation,,,; NIST’s synthetic masks may not reflect real protective-mask effects.DHS testing found most best-performing systems dropped from 100% to 96% accuracy with masks; Damer et al. evaluated 24 participants across three collaborative sessions.Source: §II · p. 2
  • Prior methods used simulated-mask fine-tuning or periocular recognition,, whereas this approach runs atop existing face-recognition models without retraining.The cited FaceNet fine-tuning work used 269 images, and the periocular approach was tested only on simulated masked-face datasets,,.Source: §II · p. 2
  • The proposed approach targets mask recognition directly and is designed to improve performance on top of existing models without retraining them.Source: §II · p. 2

III. METHODOLOGY

The method maps masked-face embeddings to same-size embeddings intended to resemble unmasked embeddings of the same identity, using an EUM that can sit on existing recognition models. Its SRT loss adjusts triplet learning to prioritize genuine-pair similarity once impostor separation is sufficient.

A. Embedding Unmasking Model Architecture

  • EUM takes a masked-face embedding and outputs a same-size feature vector intended to resemble the unmasked embedding of that identity, without modifying or retraining the recognition model.The EUM uses a fully connected network with an input layer, two hidden layers, and an output layer.Source: §III · p. 2, A. Embedding Unmasking Model… · p. 2

B. Unmasked Face Embedding Learning

  • SRT minimizes genuine-pair distance while maintaining impostor separation, switching to genuine-pair minimization alone when inter-class separation is sufficient.When anchor-negative distance falls below positive-negative distance, SRT behaves like conventional triplet loss; otherwise, it substitutes the positive-negative distance, comparing distances at the mini-batch level to reduce sensitivity to impostor-pair outliers.Source: §III · p. 2, 1) Self-restrained Triplet Loss · p. 4
  • The learning objective targets reduced FNMR for genuine masked-unmasked pairs, reflecting reports that masks affect genuine scores more than impostor scores.The paper cites reports of increased FNMR and relatively less affected FMR when probes are masked.Source: B. Unmasked Face Embedding Learning · p. 3, 1) Self-restrained Triplet Loss · p. 4
  • SRT-trained EUMs learned significantly smaller anchor-positive distances than naive-triplet-trained EUMs across ResNet-100, ResNet-50, and MobileFaceNet.The smaller d1 indicates greater similarity between the output embedding and the unmasked embedding of the same identity.Source: 1) Self-restrained Triplet Loss · p. 4

IV. EXPERIMENTAL SETUP

This section introduces the experimental setup and implementation details. The accompanying triplet-loss illustration contrasts two possible distance-learning scenarios.

IV. EXPERIMENTAL SETUP

  • The section presents the experimental setups and implementation details used in the paper.Source: §IV · p. 4
  • In the illustrated triplet-loss scenarios, scenario 2 learns greater anchor-positive similarity, while scenario 1 learns a much larger anchor-negative distance.The figure describes the anchor-negative distance as d2 and the anchor-positive distance as d1.Source: Figure 3 · p. 5
  • Scenario 2 illustrates stronger anchor-positive similarity than scenario 1.The figure identifies this distance as d1.Source: Figure 3 · p. 5

A. Face Recognition Model

The evaluation uses three ArcFace-trained recognition models, with shared training data and loss for ResNet-50 and MobileFaceNet. The models produce embeddings from aligned, cropped 112 × 112 face images.

A. Face Recognition Model

  • The study evaluates ResNet-100, ResNet-50, and MobileFaceNet, using cosine distance for comparison.Source: A. Face Recognition Model · p. 5
  • ResNet-50 and MobileFaceNet use ArcFace loss and MS1MV2 training data, while ResNet-100 uses the pretrained ArcFace model released by.Source: A. Face Recognition Model · p. 5
  • The models process aligned and cropped 112 × 112 pixel faces, producing 512-D embeddings for ResNet-100 and ResNet-50 and 128-D embeddings for MobileFaceNet.MTCNN is used to detect and align input faces.Source: A. Face Recognition Model · p. 5

B. Synthetic Mask Generation

Because large-scale paired masked and unmasked face data are unavailable, the authors train using synthetically masked images generated from facial landmarks.

B. Synthetic Mask Generation

  • Synthetic masks are drawn using 68 facial landmarks detected with Dlib, with six mask types varying in shape, height, and coverage.The listed types include wide and round masks at high, medium, and low coverage.Source: B. Synthetic Mask Generation · p. 5
  • Synthetic mask generation is used for training because no large-scale dataset of paired unmasked and masked faces is available.Source: B. Synthetic Mask Generation · p. 5

C. Dataset

The approach is trained on a synthetically masked version of MS1MV2 and evaluated using two real masked-face datasets and two benchmarks with synthetic masks. The evaluation datasets range from a small in-the-wild collection to large-scale verification benchmarks.

C. Dataset

  • MS1MV2-Masked was created from MS1MV2, with mask type and color selected randomly for each image.MS1MV2 contains 58 million images of 85,000 identities; 426,000 images without extracted facial landmarks were excluded.Source: C. Dataset · p. 5
  • The synthetic-mask examples vary in shape and coverage.Source: Figure 4 · p. 6
  • The evaluation covers two real masked-face datasets, MRF2 and MFR, plus synthetically masked LFW and IJB-C.Masks in LFW and IJB-C were generated with varied types and colors.Source: C. Dataset · p. 6
  • MRF2 contains 269 real masked and unmasked images from 53 identities, collected from the internet.The dataset averages about five images per identity and is treated as in-the-wild data.Source: C. Dataset · p. 6
  • The extended MFR dataset includes 48 participants recorded across three sessions on different days, with masked and unmasked videos under differing lighting conditions.Session 1 serves as the reference; sessions 2 and 3 serve as probes.Source: C. Dataset · p. 6
  • LFW contains 13,233 images of 5,749 identities and uses 6,000 comparison pairs in its stated protocol.Those pairs comprise 3,000 genuine and 3,000 impostor comparisons.Source: C. Dataset · p. 6
  • IJB-C contains 31,334 still images and 117,542 video frames from 3,531 identities, with over 15 million impostor comparisons in its 1:1 mixed protocol.The protocol includes 19,557 genuine and 15,638,932 impostor comparisons.Source: C. Dataset · p. 6

D. Evaluation Settings

The evaluation compares seven settings that vary whether references and probes are masked and whether EUM processes masked images. It reports verification performance and feature-extraction failures across the datasets.

D. Evaluation Settings

  • Seven settings are evaluated, with verification performance reported for each dataset and failure-to-extract rate (FTX) used to capture mask effects on face detection.LFW and IJB-C have 0.0% FTX because dataset bounding boxes are used; FTX is reported for MFR and MRF2.Source: D. Evaluation Settings · p. 6
  • UMR-UMP compares unmasked references with unmasked probes, providing the unmasked baseline condition.MFR uses first-session references and later-session probes; MRF2 uses all-to-all comparisons.Source: D. Evaluation Settings · pp. 6–7
  • UMR-MP compares unmasked references with masked probes, while UMR-MP(T) and UMR-MP(SRT) apply EUM trained with conventional triplet loss or SRT to those probes.The EUM variants retain the UMR-MP comparison pairs and FTX values.Source: D. Evaluation Settings · p. 7
  • MR-MP compares masked references with masked probes; MR-MP(T) and MR-MP(SRT) process both sides with EUM trained using conventional triplet loss or SRT.The EUM variants retain the MR-MP comparison pairs and FTX values.Source: D. Evaluation Settings · p. 7

E. Model Training Setup

Six EUM instances were trained across three face-recognition backbones, with SRT and conventional triplet loss used for comparison. Training used a shared optimizer and schedule, while early stopping produced different training durations.

E. Model Training Setup

  • Six EUM instances pair ResNet-100, ResNet-50, and MobileFaceNet embeddings with either SRT or conventional triplet loss.The triplet-loss instances serve as an ablation comparison for SRT.Source: E. Model Training Setup · p. 7
  • All models used SGD with an initial learning rate of 1e-1 and batch size 512, with the learning rate divided by 10 at 30k, 60k, and 90k iterations.The models were implemented in PyTorch and trained on an Nvidia GeForce RTX 2080 GPU.Source: E. Model Training Setup · p. 7
  • Early stopping used patience 3 and stopped the six instances after 10k, 80k, 70k, 10k, 60k, and 10k iterations, respectively.Source: E. Model Training Setup · p. 7

F. Evaluation Metric

The evaluation reports verification metrics, score-distribution separability, and ROC curves across experimental settings. The MFR results report significant verification improvement from SRT-based EUM in most evaluation cases.

F. Evaluation Metric

  • Verification performance is reported using EER, FMR100, and FMR1000, with FMR100 and FMR1000 defined as the lowest FNMR at FMR ≤1.0% and ≤0.1%, respectively.The evaluation also reports errors at thresholds set from unmasked-reference/unmasked-probe performance.Source: F. Evaluation Metric · p. 7
  • FDR measures genuine–impostor score separability; larger FDR indicates greater separation and therefore better expected verification performance.The evaluation also reports genuine and impostor score means and examines their distribution shifts.Source: F. Evaluation Metric · p. 7
  • The log-scale ROC plots distinguish SRT EUM, triplet-loss EUM, and models without EUM by color, and identify masked-probe versus masked-reference-and-probe settings by line style.AUC is listed inside each ROC plot.Source: Figure 5 · p. 8

V. RESULT

  • The results section examines mask effects, score separability, masked-face verification gains, and an SRT-versus-triplet-loss ablation.Source: §V · pp. 8–9
  • SRT-based EUM significantly improves verification performance in most MFR evaluation cases across the reported models and settings.The table covers ResNet-100, ResNet-50, and MobileFaceNet, with EER and thresholded average errors identified as lowest where marked bold.Source: Table I · p. 9

A. Impact of Masked Face on the Face Recognition Verification Performance

Masked faces reduce verification performance compared with unmasked faces, though the impact varies across models, datasets, and masking conditions. The paper also reports that its EUM with SRT improves score separability in most tested settings.

A. Impact of Masked Face on the Face Recognition Verification Performance

  • On MFR, ResNet-50 EER rises from 1.2492% (UMR-MP) to 1.2963% (MR-MP), while on LFW it is 1.4667% and 1.3667%, respectively.Across datasets and models, the relative effect of masking both reference and probe versus masking only the probe varies.Source: A. Impact of Masked Face on the Face… · p. 9
  • At FMR100 ThUMR-UMP, wearing a face mask (UMR-MP and MR-MP) has a greater effect on FNMR than FMR.At FMR1000 ThUMR-UMP, MR-MP yields lower FMR and FNMR than UMR-MP in most evaluation settings.Source: A. Impact of Masked Face on the Face… · p. 9
  • MR-MP imposter scores shift toward genuine scores, making masked pairs of different identities more similar and affecting verification performance.The paper links this shift to the generally higher G-mean but lower verification performance reported for MR-MP than UMR-MP.Source: A. Impact of Masked Face on the Face… · p. 10
  • Face masks negatively affect verification across three models and four datasets, supporting earlier findings on masked-face recognition.The evaluation covered two real masked datasets and two synthetically masked datasets.Source: A. Impact of Masked Face on the Face… · p. 10
  • With EUM and SRT, FDR increased in 20 of 24 experimental settings, indicating improved separation between genuine and imposter scores in most settings.Examples include ResNet-50 on MFR, where FDR rises from 12.6189 to 13.4045, and ResNet-100 on synthetic IJB-C, where it rises from 9.7516 to 9.9005.Source: B. Impact of our EUM with SRT on the… · p. 10

D. Impact of our EUM with SRT on in-the-wild Masked Face Recognition

On real MRF2 data and synthetic LFW and IJB-C data, EUM with SRT generally improves masked-face verification, with some EER exceptions. The reported gains vary across models and evaluation settings.

D. Impact of our EUM with SRT on in-the-wild Masked Face Recognition

E. Impact of our EUM with SRT on Simulated Masked Face Recognition

  • On synthetic IJB-C, ResNet-100 MR-MP EER rises from 2.7356% to 2.9197%, although FMR100-threshold average error decreases across evaluation cases.The authors state that the synthetic-dataset results support their findings on real masked datasets.Source: E. Impact of our EUM with SRT on… · p. 11
  • On synthetic LFW, EUM with SRT improves verification performance across all reported evaluation cases.Table III compares the three base models and EUM trained with triplet loss or SRT loss.Source: E. Impact of our EUM with SRT on… · p. 11, Table III · p. 12

F. Ablation Study on Self-restrained Triplet Loss

The ablation contrasts naive triplet loss with SRT for training EUM on masked-face embeddings. SRT outperforms naive triplet loss in the reported experiments by focusing on genuine-pair distances while maintaining imposter-pair distances near the base model’s.

F. Ablation Study on Self-restrained Triplet Loss

  • On MFR with masked probes, ResNet-50 EER is 1.9789% with naive triplet loss (UMR-MP(T)) versus 0.9611% with SRT (UMR-MP(SRT)).The paper reports that EUM trained with naive triplet loss is inefficient across the evaluated masked datasets.Source: F. Ablation Study on Self-restrained… · p. 12
  • Naive triplet loss can have few violating triplets because masked anchors may already be similar to positives and dissimilar to negatives.Increasing the margin may create more violations but can further optimize imposter distances even when they are already sufficient.Source: F. Ablation Study on Self-restrained… · p. 12
  • SRT reduces genuine-pair distances while keeping imposter-pair distances closer to those learned by the base model, outperforming naive triplet loss.The reported I-mean values with SRT are closer to unmasked-data I-mean values than those obtained with naive triplet loss.Source: F. Ablation Study on Self-restrained… · p. 13
  • EUM is intended for masked embeddings, assumes mask presence is known, and is not intended to process unmasked faces.Applying it to unmasked embeddings slightly degrades verification in several settings, including the reported IJB-C experiments.Source: F. Ablation Study on Self-restrained… · p. 13

G. Discussion

The discussion and conclusion report that EUM with SRT improves masked-face verification across three models and four datasets, while avoiding retraining the existing recognizers. The authors also identify demographic effects and realistic mask generation as future research directions.

G. Discussion

  • The authors suggest commercial face-recognition developers could use the embedding-space approach with systems that store face templates.The paper notes that template storage supports faster identification searches and matching by avoiding repeated embedding generation.Source: G. Discussion · p. 13

VI. CONCLUSION

  • Across three face-recognition models and four masked datasets, EUM with SRT improves masked-face verification in most experimental settings.The datasets include real collaborative and in-the-wild captures, plus synthetically masked LFW and IJB-C.Source: G. Discussion · p. 13, §VI · p. 14
  • EUM operates in embedding space and uses SRT to increase genuine-pair similarity while decreasing imposter-pair similarity, without retraining existing face-recognition models.SRT focuses on genuine-pair distance optimization when imposter-pair distance is deemed sufficient.Source: §VI · p. 14
  • Future work includes studying demographic effects of masked images and generating realistic masked faces with varied textures.Source: §VI · p. 14
Loading 2103.01716v3…