PaperlayerView exact revision on arXiv

Source-linked AI summary

A survey of face recognition techniques under occlusion

Dan Zeng, Raymond Veldhuis, Luuk Spreeuwers

arXiv:2006.11366v1cs.CV

TL;DR

Occluded face recognition remains difficult because occlusions can occur in unknown locations and vary widely, while suitable large-scale training and evaluation data remain limited. This survey organizes detection and recognition methods into major strategy categories, compares representative approaches, and discusses future dataset and research challenges. It concludes that newer large-scale datasets and deep-learning methods are important directions for advancing the field.

  • Problem

    Occluded face recognition is challenged by unknown occlusion locations, sizes, and shapes, while realistic large-scale training data are difficult to collect.

  • Method

    The paper surveys occluded face detection and recognition methods, categorizing recognition approaches as occlusion robust feature extraction, occlusion aware recognition, and occlusion recovery based recognition.

  • Results

    The survey provides comparative performance evaluations for occluded face detection and face recognition while reviewing representative approaches and their trade-offs.

  • Takeaways & Limitations

    Future progress is tied to large-scale real-life occluded-face datasets, standardized evaluation, and increased use of deep-learning methods.

Abstract

from arXiv · show

The limited capacity to recognize faces under occlusions is a long-standing problem that presents a unique challenge for face recognition systems and even for humans. The problem regarding occlusion is less covered by research when compared to other challenges such as pose variation, different expressions, etc. Nevertheless, occluded face recognition is imperative to exploit the full potential of face recognition for real-world applications. In this paper, we restrict the scope to occluded face recognition. First, we explore what the occlusion problem is and what inherent difficulties can arise. As a part of this review, we introduce face detection under occlusion, a preliminary step in face recognition. Second, we present how existing face recognition methods cope with the occlusion problem and classify them into three categories, which are 1) occlusion robust feature extraction approaches, 2) occlusion aware face recognition approaches, and 3) occlusion recovery based face recognition approaches. Furthermore, we analyze the motivations, innovations, pros and cons, and the performance of representative approaches for comparison. Finally, future challenges and method trends of occluded face recognition are thoroughly discussed.

I. INTRODUCTION

Occlusion remains a difficult face-recognition problem because the missing facial region can occur anywhere and vary in size or shape. The survey organizes existing approaches into three categories and reviews testing scenarios, datasets, and future needs.

  • Problem: Occlusion creates a missing-data problem because probe faces contain fewer available facial parts than high-quality gallery images.This induces a mismatch between gallery and probe representations.
  • Problem: Unknown occlusion location, size, and shape make it impractical to collect training data covering every realistic occlusion.The authors identify this as a central reason occluded face recognition remains challenging.
  • Testing scenarios: OFR testing commonly compares occluded probes with occlusion-free gallery faces across real, partial, synthetic, rectangular, and unrelated-image occlusions.These scenarios range from more realistic to less realistic settings.
  • Method taxonomy: The survey classifies methods into occlusion robust feature extraction, occlusion aware face recognition, and occlusion recovery based face recognition.An OFR system also requires decisions about cross-occlusion strategy, feature extraction, and comparison strategy.
  • Survey scope: A new survey is motivated by recent deep-learning contributions and newly available large-scale datasets such as MAFA and IJB-C.The earlier survey covered work before 2017, while newer data support updated evaluation and training discussions.

II. OCCLUDED FACE DETECTION

Occluded face detection is reviewed alongside general face detection, with methods spanning rigid templates, deformable part models, and deep convolutional networks. Deep detectors and newer benchmarks provide promising directions, but performance and computational trade-offs remain relevant.

  • Organization: Occluded face detection is divided into general face detection methods and methods specifically designed to address occlusion.The survey presents this distinction as the organizing principle for the section.
  • Method categories: General face detection methods comprise rigid templates, deformable part models, and deep convolutional neural networks.These categories include Viola-Jones-style cascades, DPMs, and CNN-based detectors.
  • Method categories: Rigid-template detectors can operate in real time but may drop dramatically in real-world conditions, whereas DPMs improve performance at higher computational cost.The comparison highlights an efficiency-versus-performance trade-off.
  • Deep learning: CNN face detectors include two-stage region-proposal frameworks and one-stage proposal-free frameworks such as SSD and YOLO.Some methods jointly perform detection and alignment to exploit their correlation.
  • Benchmarks: Widerface supports evaluation under pose, scale, expression, illumination, makeup, and occlusion variation, while FANet reports state-of-the-art performance on several face-detection benchmarks.The passage identifies Widerface as a challenging benchmark for unconstrained detection.

B. Detecting occluded face

Occluded face detection methods locate visible facial structure, fuse sub-region detections, or use occlusion information directly. Representative approaches improve detection robustness, but some require labeled attributes or specialized training data.

  • Challenges: Occluded face detection must locate faces despite unknown occlusion locations and types, and existing evaluations are often concentrated on MAFA.The survey notes that this creates an uneven comparison with general face detection methods.
  • Locating visible facial segments: Attribute-aware CNNs score visible facial parts and their spatial arrangement to estimate face likelihood and regress precise bounding boxes.The method jointly trains face classification and bounding-box regression from facial-part responses.
  • Locating visible facial segments: Faceness-net improves feature-representation robustness on Widerface but requires labeled facial attributes, limiting practical use.The cited passage links its performance to a more effective CNN design and identifies the labeling requirement as a constraint.
  • Fusing detection results: MAFA enables masked-face detection research, while LLE-CNNs use CNN descriptors and locally linear embedding to create similarity-based descriptors.The dataset contains varied masks and supports benchmarking masked-face detection.
  • Fusing detection results: 88.5% average precision on Widerface hard and 88.3% on MAFA are reported for the face attention network detector.FAN integrates anchor-level attention into a single-stage detector to emphasize face-region features.

3) Occlusion information adversarially used for detection:

Occlusion-aware detection methods use facial subregions or occlusion information to reduce the adverse effects of occlusion during face detection. The review distinguishes independent sub-block losses from adversarial masking and segmentation strategies.

  • 3) Occlusion information adversarially used for detection:: Grid loss treats occluded face detection as single-class object detection and minimizes errors independently across face sub-blocks.This exploits the observation that some detectors may remain correct when others are confused by partial occlusions.
  • 3) Occlusion information adversarially used for detection:: Adversarial occlusion-aware face detection detects faces and segments occlusion areas simultaneously while masking distinctive facial parts during training.The masking strategy mimics different occlusion situations and encourages the detector to learn what may be occluded.
  • Occlusion robust feature extraction: Occlusion-robust feature extraction seeks features that suppress occlusion outliers while preserving discriminative identity information.The review groups these methods into engineered and learning-based features.
  • A. Patch based engineered features: Engineered features are typically extracted from facial patches and concatenated, offering efficient extraction and relatively compact feature spaces.Examples include local descriptors such as LBP, SIFT, and HOG.
  • A. Patch based engineered features: Patch-based engineered methods often require precise registration, but occlusion can hide the eyes used for alignment.This dependence makes them less realistic when robust alignment is unavailable.

2) Patch-based matching:

Patch-based matching addresses occlusion by emphasizing local correspondences and downweighting unreliable regions. The reviewed methods span elastic graph matching, statistical local-feature weighting, and sparse representations.

  • 2) Patch-based matching:: Dynamic graph representations enable elastic and partial matching by matching nodes with adjacent regions, improving robustness to partially occluded faces.In EBGM, graph nodes represent facial regions and edges encode spatial relationships.
  • 2) Patch-based matching:: Learned features are more flexible than engineered features when occlusion types and locations vary.Face-specific structural constraints support subspace learning for feature extraction.
  • 2) Patch-based matching:: ICA-based part representations can outperform PCA and LDA under partial occlusions and local distortions by using salient facial-part information.The method focuses on locally informative components rather than only a global linear subspace.
  • 2) Patch-based matching:: Statistical local-feature methods estimate feature reliability and use weighted distances to reduce the influence of occluded regions.Gaussian models can estimate descriptor probabilities, while low-posterior regions are de-emphasized during matching.
  • 2) Patch-based matching:: Sparse representation methods model a test face as training samples plus sparse errors, while later variants combine local features, low-rank structure, and multiscale outlier detection.RDLRR reports improved performance under illumination changes, real occlusion, and block occlusion.

4) Deep learning:

Deep-learning approaches address occlusion through synthetic augmentation, region-focused training, explicit occlusion detection, and learned masking. These methods either improve representations or exclude corrupted regions from recognition.

  • 4) Deep learning:: Synthetic augmentation with hairstyle and glasses templates relieves data deficiency and improves performance, but remains limited to sunglasses and hair occlusions.The described method uses 87 hairstyle templates and 100 glasses templates.
  • 4) Deep learning:: Region-focused CNN training covers high-effect facial regions more frequently, reducing reliance on central features when those regions are occluded.The approach instead encourages learning from outer facial regions, although predefined occlusions may impose limitations.
  • 4) Deep learning:: Occlusion-aware face recognition methods either detect occlusions before extracting non-occluded representations or assume visible facial parts are already available.The review classifies these as occlusion detection-based recognition and partial face recognition.
  • 1) Occlusion detection:: Occlusion detection methods often use predefined scarves and sunglasses as representative occlusions to simplify arbitrary-location detection.A 1-NN threshold classifier and selective local NMF are described as examples.
  • 4) Deep learning:: Adaptive occlusion estimation can avoid prior occlusion knowledge and handle large continuous occlusions through reconstruction-error information.This is presented as an alternative to selecting fixed visible subregions.
  • 4) Deep learning:: Deep models can learn occlusion masks or region-specific probabilities, including separate predictions for the eyes, nose, and mouth.MaskNet downweights hidden units activated by occluded areas, while PDSN links occluded blocks to corrupted feature elements and improves results on synthesized and realistic datasets.

B. Partial face recognition

Partial face recognition treats visible face fragments as the recognition input and uses feature-aware, comparison-aware, and learning-based methods to address missing regions and misalignment. Recent approaches combine deep feature learning with sparse representation or sliding-window matching.

  • B. Partial face recognition: Partial face recognition assumes a partial face is available and focuses on recognition without necessarily performing explicit occlusion detection.Such inputs commonly arise in unconstrained surveillance or handheld-device imagery.
  • B. Partial face recognition: Multiscale patch networks extract features from aligned facial regions at different scales and combine their learned weights for recognition.MDSCNN crops patches using eye-corner alignment.
  • B. Partial face recognition: Comparison-aware methods use voting, masks, or learned classifiers to tolerate missing information and alignment difficulties.The review includes ensemble RBF networks and Lophoscopic PCA among these approaches.
  • B. Partial face recognition: Multi-keypoint descriptors enable recognition of holistic or partial faces with variable-length representations without preliminary face alignment.This work is described as an early systematic study of arbitrary face patches.
  • B. Partial face recognition: Feature-set matching can combine geometric and textural similarity, but fails when occlusion removes keypoints and can be computationally slow.Robust point-set matching additionally constrains affine transformations to prevent unrealistic warping.
  • B. Partial face recognition: Recent methods combine deep learning with sparse representation, including sliding-window matching and multiscale dynamic feature learning.These combinations improve tolerance to misalignment between partial probes and gallery faces.
  • B. Partial face recognition: Occlusion recovery reconstructs a whole face from an occluded input so conventional face recognition algorithms can be applied afterward.The review identifies reconstruction-based and inpainting-based recovery as the main approaches.

A. Reconstruction based face recognition

Reconstruction-based approaches recover occlusion-free faces so conventional recognition algorithms can be applied. The survey covers linear reconstruction, sparse representation, and deep-learning techniques, including methods that detect occlusions and iteratively recover corrupted regions.

  • A. Reconstruction based face recognition: Reconstruction methods model relationships between occluded and occlusion-free faces using linear reconstruction, sparse representation, or deep learning.These categories include PCA-based methods, dictionary learning, and neural reconstruction models.
  • Linear reconstruction: PCA-based methods detect or compensate for occlusions before reconstructing occlusion-free face images.One approach combines PCA reconstruction, recursive error compensation, a Markov Random Field, and parse representations for glasses-induced eye occlusions.
  • Sparse representation: Sparse representation models represent a test face with training samples plus sparse errors that account for occlusions or corruption.The SRC introduces an identity matrix as an occlusion dictionary under the assumption that occlusion has a sparse representation.
  • Sparse representation: Several methods combine occlusion detection with reconstruction, including low-complexity localization, overdetermined reconstruction, structured sparse dictionaries, and modular weighting.Modular weights reduce the influence of low-discriminant regions and help detect occluded modules.
  • Deep learning reconstruction: Deep reconstruction methods use stacked sparse denoising autoencoders or LSTM-autoencoder architectures to encode occlusions and reconstruct faces.The robust LSTM-autoencoder uses spatial and dual-channel LSTM components, while adversarial CNNs enhance discriminative information in recovered faces.

B. Inpainting

Inpainting methods repair occluded image regions, generally without directly considering face recognition. The survey distinguishes non-blind methods, which receive corrupted-pixel locations, from blind methods, for which deep learning is effective.

  • B. Inpainting: Image inpainting repairs occluded images and is divided into non-blind and blind inpainting according to whether corrupted-pixel locations are known.The survey treats inpainting as an image-repair strategy distinct from direct face-recognition modeling.
  • Non-blind inpainting: Non-blind inpainting fills missing regions using surrounding pixels, including exemplar-based texture synthesis that samples and copies source colors.One method determines a target-region fill order by combining pixel confidence with image isophotes.
  • Generative models: Generative models synthesize samples through explicit or implicit density estimation, with GANs generating realistic-looking images through generator-discriminator competition.GAN training is described as powerful but unstable, motivating adversarial autoencoder variants.
  • Blind inpainting: Blind-inpainting methods combine sparse coding and deep neural networks for denoising and inpainting, while attributes such as hairstyles and eyeglasses can be toggled during generation.The cited approaches include stacked sparse denoising autoencoders and attribute-based face editing.
  • Blind inpainting: ERGAN targets eyeglasses removal, whereas ID-GAN combines a CNN recognizer with GAN-based generation to preserve realism and identity.ERGAN is described as unsupervised and evaluated in terms of removal realism and diversity.

B. Existing OFR benchmark datasets

OFR benchmarks combine a small number of real-occlusion datasets with synthetic occlusions generated from general face-recognition datasets. Comparisons remain difficult because protocols and occlusion configurations vary, while severe occlusion and single-sample settings remain challenging.

  • B. Existing OFR benchmark datasets: General face-recognition datasets are often converted into OFR benchmarks by adding synthetic occlusions because occluded faces are barely represented.The survey organizes testing scenarios from real occlusions to less realistic synthetic settings.
  • Dataset construction: The benchmark summary records subjects, images, real-occlusion availability, and the synthetic occlusions used to create each D-occ dataset.The survey denotes the original dataset as D and its occluded derivative as D-occ.
  • Real-occlusion datasets: AR is one of the few datasets with real occlusions, containing sunglasses and scarf images across two sessions for 126 individuals.The dataset includes more than 4,000 faces and varies expression, illumination, and partial occlusion.
  • Synthetic-occlusion datasets: Extended Yale B contains no occlusions originally, so researchers synthesize occluded versions by overlaying unrelated images or rectangles on occlusion-free faces.These evaluations typically use occlusion-free gallery images and randomly occluded test images.
  • Evaluation protocols: AR results are difficult to interpret because papers use different experimental setups and the dataset lacks standard protocols.Commonly used protocols include S-TR-7IPS, S-TR-8IPS, and S-TR-SSPP.
  • Evaluation protocols: OFR performance still has room to improve under the SSPP protocol, while deep-learning methods increasingly rely on large general datasets rather than AR-only training splits.SSPP denotes the single-sample-per-person setting.
  • Benchmark difficulty: Occluded versions of otherwise well-addressed datasets remain challenging, with severe degradation when 90% of the face is covered.Partial-face benchmarks are also difficult to compare because arbitrary patches may contain different amounts of discriminative information.

2) Evaluation based on the face recognition pipeline for categorization:

The survey reclassifies occlusion methods according to where they operate in the face-recognition pipeline. This view separates data handling, feature extraction, feature comparison, and fusion while acknowledging that named method categories can span multiple components.

  • 2) Evaluation based on the face recognition pipeline for categorization:: Pipeline-based categorization places occlusion handling in data augmentation or recovery, feature extraction, feature comparison, and fusion strategies.The framework offers a perspective complementary to the ORFE, OAFR, and ORecFR categories.
  • Data Augmentation or Data Recovery: Data augmentation incorporates possible real-life occlusions into training data, whereas recovery explicitly fills or implicitly de-occludes occluded regions.Encoder-decoder structures are cited as an example of implicit recovery.
  • Feature extraction: Feature extraction exploits occlusion locality through patch-based engineered features, statistical learning, sparse representation, or deep learning.The goal is to obtain features that remain useful despite localized corruption.
  • Feature comparison: Feature comparison establishes semantic correspondence between fixed-length whole-face features and variable-length occluded-face features.Directly matching features from different semantic parts is described as meaningless.
  • Fusion strategy: Fusion strategies combine decisions from multiple classifiers or use majority voting under the assumption that visible facial regions are more useful for identity recognition than occluded regions.The strategy aims to reduce the influence of features extracted from occlusions.
  • Future challenges: Future dataset and research challenges are interdependent because dataset requirements often reflect underlying real-world problems.The survey treats datasets as indicators of unresolved research challenges.

A. Future Dataset Challenges

Future progress in occluded face recognition depends on larger, more diverse datasets, standardized evaluation, and methods suited to unconstrained occlusions. The survey also identifies deep-learning directions for augmentation, recovery, and detection.

  • Future Dataset Challenges: Current datasets are limited by small scale, narrow occlusion diversity, and inconsistent evaluation protocols.AR includes only 126 individuals; common synthetic occlusions such as rectangles and Baboon images are not representative of real-life occlusions.
  • Future Dataset Challenges: Future benchmark datasets should include more individuals, varied and sufficient occlusions, and well-defined evaluation protocols.
  • Future Dataset Challenges: Large-scale training data remain necessary, but web-sourced occluded faces are scarce and severely occluded images can be difficult or impossible for humans to label.The paper identifies efficient labeling of occluded faces as an open issue for deep-learning development.
  • Future Dataset Challenges: IJB-C provides many natural occlusions and annotated occlusion locations, supporting future evaluation on larger real-life datasets.The survey expects standard evaluation on AR and newly developed real-life, large-scale IJB-C datasets.
  • Future Research Challenges: Unconstrained occluded face recognition remains open because existing AR and Extended Yale B datasets are aligned and captured in constrained environments.The challenge also intersects with pose variation, age gaps, and low resolution.
  • Future Research Challenges: Future methods are expected to improve robust feature extraction, occlusion-aware recognition, and occlusion recovery, increasingly using CNN-based deep networks.
  • Future Research Challenges: GAN-based augmentation could transfer occlusion types from real occluded faces to occlusion-free faces, avoiding reliance on manually aligned templates.Manual templates depend on accurate facial landmarking and proper alignment.
  • Future Research Challenges: Recovery models could jointly optimize reconstruction with identification or verification, while unified occlusion detection and recognition could automate preprocessing and recognition.Existing recovery systems emphasize visually pleasing rather than accurate reconstruction, and patch-based detection often yields rough occlusion areas.

APPENDIX

The appendix documents terminology and evaluation summaries for representative occluded face recognition algorithms across datasets and application-oriented components.

  • APPENDIX: Table IV summarizes representative algorithms’ identification rates on the AR dataset across ORFE, OAFR, and ORecFR categories.The table also records experiment-setting abbreviations, occlusion conditions, and whether training used session one.
  • APPENDIX: Tables V and VI summarize identification and verification rates on other benchmarks, but verification methods cannot be compared properly because experimental settings differ.
  • APPENDIX: Table VII organizes algorithms by the components they use during occluded face recognition, including data augmentation or recovery and fusion strategies.Fusion includes feature-level and decision-level fusion.
  • APPENDIX: Table VIII summarizes algorithms by application-oriented purpose, including occluded face detection and the three occluded face recognition classes.The table marks partial-face detection as unavailable according to the survey.
  • APPENDIX: Table IX provides a glossary of abbreviations and terminology expansions.
Loading 2006.11366v1…