PaperlayerView exact revision on arXiv

Source-linked AI summary

Facial Expression Analysis under Partial Occlusion: A Survey

Ligang Zhang, Brijesh Verma, Dian Tjondronegoro, Vinod Chandran

arXiv:1802.08784v1cs.CV

TL;DR

Partial occlusion remains insufficiently understood for real-world facial expression analysis, where existing studies often rely on non-occluded or artificially occluded data. This survey reviews datasets, algorithms, and occlusion effects, finding that current systems remain early-stage and that occlusion effects are context-dependent. It identifies challenges and opportunities for more robust and better-benchmarked FEA.

  • Problem

    Partial occlusion is a major obstacle for accurate real-world FEA, while comprehensive datasets and focused synthesis of occlusion-handling efforts remain limited.

  • Method

    The paper surveys occlusion-related FEA datasets, automatic recognition approaches, investigations of occlusion effects, and associated challenges and opportunities.

  • Results

    Current occlusion-capable FEA systems remain early-stage, relying mainly on limited prototypical emotions, artificially generated occlusion, visual face data, and frontal 2D/3D faces.

  • Takeaways & Limitations

    The survey provides a focused reference for benchmarking and developing FEA techniques under partial occlusion.

  • Takeaways & Limitations

    Creating comprehensive occluded-expression datasets is constrained by the difficulty of collecting spontaneous uncommon expressions and reliably annotating occlusion and emotions.

Abstract

from arXiv · show

Automatic machine-based Facial Expression Analysis (FEA) has made substantial progress in the past few decades driven by its importance for applications in psychology, security, health, entertainment and human computer interaction. The vast majority of completed FEA studies are based on non-occluded faces collected in a controlled laboratory environment. Automatic expression recognition tolerant to partial occlusion remains less understood, particularly in real-world scenarios. In recent years, efforts investigating techniques to handle partial occlusion for FEA have seen an increase. The context is right for a comprehensive perspective of these developments and the state of the art from this perspective. This survey provides such a comprehensive review of recent advances in dataset creation, algorithm development, and investigations of the effects of occlusion critical for robust performance in FEA systems. It outlines existing challenges in overcoming partial occlusion and discusses possible opportunities in advancing the technology. To the best of our knowledge, it is the first FEA survey dedicated to occlusion and aimed at promoting better informed and benchmarked future work.

1 Introduction

Automatic facial expression analysis remains difficult in real-world conditions because partial occlusion is often overlooked and substantially degrades recognition. This survey addresses the gap through a focused review of datasets, approaches, occlusion effects, challenges, and opportunities.

  • Motivation: Occlusion is often assumed to be handled by controlled acquisition, but that assumption applies only to laboratory or prescribed conditions.
  • Motivation: Partial occlusion is a major obstacle to accurate FEA in real-world conditions and can severely deteriorate system performance.Occlusion changes facial appearance, disrupts feature extraction and alignment, and introduces noise and outliers.
  • Research gap: Earlier FEA surveys reported no specifically designed techniques for overcoming facial occlusion.
  • Research gap: Current literature lacks a comprehensive, focused survey of efforts to overcome partial occlusion in FEA.
  • Scope and contribution: The paper reviews databases, automatic approaches, occlusion effects, and challenges and opportunities for robust FEA.

2 Background

The background introduces how facial expressions are represented and why partial occlusion is difficult to handle. It also summarizes the field’s history and the varied properties of occlusion.

  • Expression representation: FEA represents expressions using message-based methods or movements of facial components.
  • Expression representation: Message-based representation includes discrete categorical labels and continuous dimensional spaces.Categorical methods use prototypical emotions, whereas dimensional methods place expressions in spaces such as arousal and valence.
  • Expression representation: Facial-component methods encode muscle movements through systems such as FACS and action units, while micro-expressions capture brief involuntary expressions.Micro-expressions last approximately 1/25 to 1/3 second and involve subtle muscle movements.
  • Occlusion types: Facial occlusion is generally classified as systematic or temporary, including occlusion from components, adornments, or transient objects.
  • Occlusion characteristics: Occlusion varies in type, combinations, location, duration, and visual properties, making its effects difficult to predict.Temporary hand occlusion is less predictable in location, while objects such as glasses differ in shape, appearance, and size.
  • Occlusion characteristics: Because occlusion usually affects only part of the face, unoccluded regions may compensate for its impact.Prior knowledge about likely occlusion parameters can also support techniques intended to overcome its effects.

3 Related Databases

The survey reviews databases used to evaluate FEA under occlusion, emphasizing the limited availability of datasets created specifically for this purpose. Existing resources vary in realism, modality, expression labels, and occlusion coverage.

  • Dataset landscape: Well-annotated public datasets with facial occlusion are a prerequisite for evaluating FEA systems, yet few were created specifically for FEA.
  • Existing databases: JAFFE, CK, and BHUFE provide benchmark expression data, including posed non-occluded faces and selected hand-obscured expressions.
  • Existing databases: Realistic wild datasets include varied conditions, but available occlusion may be limited or lack explicit occlusion annotations.HAPPEI contains faces with partial occlusion without occlusion labels, while other wild datasets are not specifically designed for evaluating FEA under occlusion.
  • Existing databases: Bosphorus, Cam3D, UMB, BU-3DFE, and BP4D add 3D, multimodal, spontaneous, or naturally occluded facial-expression data.
  • Dataset landscape: Most reviewed databases use discrete categories or action units, are laboratory-collected, and focus on hands, glasses, hair, or pose-related self-occlusion.

4 Automatic Facial Expression Analysis Approaches

The survey organizes occlusion-robust FEA methods into six approach categories and compares their detection, features, classifiers, emotion targets, occlusion settings, and performance. Reported methods span handcrafted, geometric, statistical, 3D, sparse, and deep-learning strategies.

  • Scope: The survey limits coverage to methods that use or investigate partially occluded face data, excluding general non-occluded FEA surveys.
  • Approach taxonomy: Existing occlusion-aware FEA studies are grouped into feature reconstruction, sparse representation, sub-region, statistical model, 3D data, and deep learning approaches.
  • Comparative framework: The approaches are compared by occlusion detection, feature type, classifier, emotion category, occlusion type, simulation, and performance.
  • Evaluation: Reported evaluations cover benchmark datasets including JAFFE, CK/CK+, BU4DFE, BP4D, and FERA 2017, with performance varying by task and setting.

4.1 Feature Reconstruction Approach

Feature reconstruction approaches address partial occlusion by recovering missing geometric or texture features from facial configuration, but remain dependent on accurate detection, tracking, alignment, or manual initialization.

  • 4.1 Feature Reconstruction Approach: Feature reconstruction approaches recover missing geometric or texture features caused by partial occlusion using the visual configuration of the face.They include geometry-based and texture-based methods.
  • 4.1 Feature Reconstruction Approach: Conditional mean PCA reconstruction produced the best accuracy for recovering random subsets of 22 missing feature points on CK images.
  • 4.1.1 Geometry based Approach: Enhanced Kanade-Lucas tracking reconstructs drifting or lost facial points in video, then uses local spatiotemporal vectors and rank-weighted KNN classification.
  • 4.1.1 Geometry based Approach: Infrared-sensitive cameras can detect pupil locations under unfavorable lighting for normalizing eyes and brow regions before extracting landmark-based shape parameters.
  • 4.1.2 Texture based Approach: Texture reconstruction combines RPCA-based face reconstruction with saliency detection, replacing detected occluded regions before feature extraction and classification.
  • 4.1 Feature Reconstruction Approach: These approaches remain difficult to automate because they depend on reliable face detection, feature tracking, precise alignment, or manual annotation and omission of occluded features.

4.2 Sparse Representation Approach

Sparse representation methods model occluded faces using training samples and explicit or robust residual handling, achieving strong recognition results while depending on representative dictionaries and accurate preprocessing.

  • 4.2 Sparse Representation Approach: Sparse representation treats training samples as a dictionary and represents a test image as a linear combination of samples from the same emotion class.
  • 4.2 Sparse Representation Approach: For moderate occlusion, an identity-matrix error component can be isolated with l1 minimization and subtracted to recover a clearer test image.
  • 4.2 Sparse Representation Approach: 95% accuracy was obtained with 50% noise corruption, while block occlusion ranging from 10×10 to 40×40 pixels exceeded 91% accuracy on JAFFE images.
  • 4.2 Sparse Representation Approach: MLESR outperformed SRC and GSRC across simulated random occlusion levels from 0 to 90% on JAFFE and CK databases.
  • 4.2 Sparse Representation Approach: GSNMF with nearest-neighbor classification achieved 93.3%, 94.0%, and 91.4% accuracy under eyes, nose, and mouth occlusion, respectively, on CK.
  • 4.2 Sparse Representation Approach: LBP-map and SRC fusion achieved 87.0% accuracy with 35% face replacement and 72.4% accuracy with 30% eye-region occlusion on CK.
  • 4.2 Sparse Representation Approach: Sparse representation performance depends on representative training dictionaries, sufficient emotion-specific variation, and precise face location, alignment, and normalization.

4.3 Sub-region based Approach

Sub-region approaches divide faces into local areas and fuse information from regions considered less affected by occlusion, but their effectiveness depends on occlusion extent and subdivision granularity.

  • 4.3 Sub-region based Approach: Sub-region methods fuse features or classification decisions from local facial areas while attempting to exclude information from occluded regions.
  • 4.3.1 Feature Fusion Approach: Gabor-template methods use multi-scale local texture features sampled from gallery images, anticipating that only a small subset will be affected by occlusion.
  • 4.3.2 Decision Fusion Approach: Maximum decision fusion combines WLDH-and-SVM outputs from equally sized facial sub-regions for expression classification.
  • 4.3.2 Decision Fusion Approach: The decision-fusion approach achieved more than 87% accuracy for mouth, eye, and left- or right-side occlusion on JAFFE images.
  • 4.3.2 Decision Fusion Approach: Local-expression prediction methods used masked random-forest trees and autoencoder reconstruction errors to weight local predictions under occlusion.
  • 4.3 Sub-region based Approach: Sub-region methods often perform satisfactorily for small occlusion, while face-subdivision granularity remains open, especially for random occlusion without fixed location.

4.4 Statistical Model based Approach

Statistical model approaches infer or track occluded facial features using spatial dependencies in images or temporal correlations in video. Video-based methods can use neighboring frames and handle missing frames, but benchmarking and ground-truth creation remain limited.

  • Model principle: Statistical models infer occluded features from spatial dependencies in non-occluded regions or temporal correlations across neighboring video frames.They do not directly reconstruct occluded regions or divide the face into sub-regions.
  • Video methods: Bayesian trackers augmented KLT tracking with Bayesian feedback to track facial points under temporal occlusion from head motions or hands.Tracked distances were subsequently used by HMMs and ANNs for expression classification.
  • Strengths and limitations: Video-based statistical methods can exploit temporal expression patterns, handle missed frames, and may be more robust than static-image approaches.Their practical drawbacks include the need for suitable full-expression video ground truth and robust tracking in occluded frames.
  • Strengths and limitations: Performance advantages over other approaches remain insufficiently validated because suitable benchmark databases are lacking.Direct comparisons with other approach families are still largely unexplored.

4.5 3D Data based Approach

3D-based FEA adds depth and facial-structure information that can support view-independent features under occlusion, especially pose-related self-occlusion. Methods include view-specific classification, pose-invariant feature mapping, and multimodal 2D–3D fusion, but they require substantial computation and accurate mapping.

  • Motivation: 3D data adds depth information that can generate more robust, discriminative, and view-independent features under head-pose or missing-part occlusion.The survey identifies self-occlusion from head-pose changes as a particularly relevant case.
  • Multiple views: Multiple-view methods first classify the face view and then apply a view-dependent emotion classifier.Hu et al. used five view classes and trained a separate classifier for each view.
  • Simulated 3D features: Pose-invariant feature mapping transfers non-frontal 3D facial features to corresponding frontal-view representations before expression recognition.Vieriu et al. used a cylindrical representation and treated self-occlusion as missing information.
  • Reported results: Over 80% accuracy was achieved at raw and 60-degree pitch angles, while accuracy fell to zero at 150-degree pitch or yaw when useful facial information was completely occluded.Temporal approaches outperformed image-based statistical approaches in the reported evaluation.
  • Multimodal fusion: Fusing 2D texture and 3D geometry at feature and score levels improved FER with an SVM classifier.The descriptors represented local texture around 2D landmarks and local geometry around 3D landmarks.
  • Limitations: 3D approaches require heavy computation and accurate algorithms for mapping non-frontal features into frontal representations.These requirements constrain practical deployment despite the richer structural, texture, and depth information.

4.6 Deep Learning Approach

Deep learning approaches learn abstract expression patterns directly from facial data and have achieved strong results in occlusion-relevant FEA evaluations. Their adoption remains constrained by data, tuning, and computational requirements.

  • Approaches: Deep learning approaches include boosted deep belief networks, CNNs, LSTMs, and multiple-network fusion for FEA under occlusion.Existing occlusion-focused studies remain limited and mainly use a single deep architecture.
  • FERA 2017: AUMPNet outperformed the FERA 2017 baseline, achieving mean F1 scores of 0.521 and 0.506 for AU detection on validation and test subsets, respectively.It jointly detected AUs and estimated intensity from multi-view images using a pose-aware multi-task loss.
  • Potential for occlusion handling: Deep architectures can learn discriminative features from raw faces without separate occlusion detection or reconstruction processes.Occlusion information can be embedded in the features learned by the architecture.
  • Limitations: Deep learning requires large training datasets, extensive parameter tuning, and expensive computation.These constraints are identified as major challenges for applying deep architectures to FEA.

4.7 Summary of FEA Approaches

The surveyed literature covers many occlusion types but relies mainly on artificial occlusion, non-occluded benchmark datasets, frontal single-occlusion settings, and 2D features. The field lacks agreed descriptors and broad evaluations of natural, multiple, multimodal, 3D, and dimensional-expression settings.

  • Occlusion types: Primary occlusion types include eyes, mouth, face regions, blocks, hands, glasses, noise, and self-occlusion from head-pose variation.Few studies also examine brows, hair, blur, or missed frames.
  • Data realism: Most studies simulate occlusion by removing features or superimposing masks, while few use naturally occurring masks, sunglasses, or hand occlusion.Natural occlusion has received comparatively limited investigation.
  • Evaluation coverage: Most evaluations use JAFFE, CK, and CK+ with non-occluded faces, and natural-occlusion studies generally report sample demonstrations rather than whole-dataset accuracy.The survey found no work investigating occlusion in 3D face data.
  • Scenario coverage: Existing studies focus on one occlusion type in frontal faces, leaving co-existing occlusion and multiple-face settings uninvestigated.This limits coverage of more complex real-world conditions.
  • Pipeline design: Most systems extract features directly from occluded images and depend on accurate face alignment and robust facial-feature tracking.Only a few integrate occlusion detection as a preprocessing stage.
  • Feature modalities: Features are largely restricted to 2D texture or geometry, with few studies fusing modalities or using skin-color features.3D work primarily addresses self-occlusion caused by varying head poses.
  • Expression targets: Most work targets six basic emotions plus neutral, while dimensional spaces have not received occlusion-specific study.Only a few studies address contempt, grammatical expressions, or action units.
  • Open questions: No agreement exists on the most effective feature descriptor for handling facial occlusion.The survey identifies automatic feature learning as a potential advantage of deep learning.

4.8 Top Five Benchmark Approaches

This section identifies five top-performing benchmark approaches on JAFFE and CK/CK+ and adds recent FERA 2017 challenge approaches to support direct future comparisons.

  • The survey selects the top five approaches separately for JAFFE and CK/CK+ based on the highest overall accuracy for six basic and neutral emotions from 2001–2016.The selection targets direct comparison with state-of-the-art results.
  • Recent approaches from the FERA 2017 challenge are also included among the benchmark comparisons.
  • Table 4 reports benchmark accuracy percentages under different types of facial occlusion.

5 Effect of Occlusion on Facial Expression

Occlusion effects vary by facial region, emotion, database, and evaluation setup. Across computer-vision and human-perception studies, the mouth and eyes often provide especially important information, but no absolute consensus exists.

  • Computer Vision Investigation: Occluding the mouth generally reduces accuracy more than occluding the eyes for anger, fear, happiness, sadness, and overall performance.For disgust, the relative effect reverses across the CK and JAFFE databases.
  • Computer Vision Investigation: Lower-face occlusion significantly reduces classification accuracy across the six basic emotions, neutral, and overall performance.
  • Computer Vision Investigation: Upper-face occlusion produces large accuracy reductions for anger and happiness but small reductions for neutral and fear.
  • Computer Vision Investigation: Nose occlusion usually causes little accuracy reduction, whereas random-block occlusion causes a large reduction in overall accuracy.
  • Human Perception Investigation: Human-perception studies identify the mouth and eyes as crucial regions for recognizing genuine emotions, while temporal displays improve recognition over static peak expressions.
  • Summary of Effect of Occlusion: The mouth is summarized as the most important facial region, with large effects on six basic emotions and overall performance but a relatively small effect on neutral.
  • Summary of Effect of Occlusion: The eyes are the second most important region, strongly affecting sadness, disgust, and surprise but only slightly affecting happiness and neutral.
  • Summary of Effect of Occlusion: Occlusion effects lack absolute consensus because features, classifiers, participants, emotional stimuli, and evaluation protocols differ across experiments.System preprocessing and train-test strategies can also substantially affect results on the same database and occlusion type.

6 Challenges and Opportunities

FEA under partial occlusion remains constrained by limited datasets, difficult labeling, incomplete preprocessing, and insufficiently realistic evaluation. The survey identifies opportunities spanning occlusion-aware features, human perception studies, context, and broader competitions.

  • Database creation and labeling: Most studies use non-occluded databases with limited artificial occlusions and mainly six basic emotions plus neutral.Comprehensive datasets with natural occlusions and well-annotated expressions remain scarce.
  • Database creation and labeling: Collecting spontaneous, especially subtle or context-dependent, expressions under occlusion is difficult because occlusion can alter subjects’ natural reactions.Strategies that improve elicitation and annotation can increase recruitment difficulty and restrict collectible occlusion types.
  • Database creation and labeling: Occlusion annotation lacks a standard criterion beyond location, while object type, intensity, material, transparency, and texture may also affect FEA.Pixelwise region labels support localization but may not enable detailed analysis of occlusion properties.
  • Database creation and labeling: Human emotion labeling becomes less reliable when informative regions are obscured, particularly for uncommon emotions, action units, and continuous dimensions.Occlusion makes subtle local muscle changes difficult to annotate when the relevant facial part is invisible.
  • Algorithm development: Most approaches process occluded faces directly, although detecting occlusion first can support reconstruction or discarding affected features and has improved FEA performance.Reliable parameters include occlusion type, location, shape, appearance, and temporal behavior.
  • Other features: Skin color can assist occluded-region segmentation and provide complementary information for facial-expression and emotional-state analysis.It has been used to detect occluded regions and infer emotional valence.
  • Additional issues: Comparing computer-vision and human-perception experiments on the same occlusions and datasets may provide insights into robust FEA design.The survey notes unresolved questions about holistic versus component-based human recognition under occlusion.
  • Additional issues: International competitions could broaden evaluation beyond non-occluded faces toward discrete emotions, action units, and continuous affect dimensions.Existing FEA challenges have largely focused on non-occluded conditions.

7 Conclusion

The survey reviews state-of-the-art efforts for FEA under partial occlusion and identifies persistent gaps in datasets, detection, multimodal modeling, and realistic emotion evaluation. It proposes broader benchmarks, modalities, representations, and architectures as directions for future work.

  • Conclusion: Sparse representation and deep learning approaches demonstrate the most impressive reviewed results for combating facial occlusion.Current systems nevertheless remain early-stage, with limited prototypical emotions, artificial occlusion, visual-only features, and mostly frontal 2D/3D faces.
  • Conclusion: Existing work lacks comprehensive benchmarks covering natural occlusions and well-annotated discrete emotions, action units, and dimensional axes.The survey identifies benchmark construction as a central unresolved issue.
  • Conclusion: Reliable automatic detection of occlusion type and location remains insufficiently developed.The survey presents occlusion detection as a distinct research gap.
  • Conclusion: Multimodal temporal correlations and fused audio, visual, and physiological features remain underexplored for handling facial occlusion.Future systems are expected to move beyond a single face modality.
  • Conclusion: The effects of occlusion on non-prototypical spontaneous emotions require investigation across multiple realistic datasets.This gap complements the survey’s broader call for naturally occurring rather than artificially imposed occlusion.
  • Conclusion: Future systems should expand from artificial to natural occlusion, 2D to 3D data, manual preprocessing to automatic detection, and prototypical emotions to action units, continuous affect, and micro-expressions.The survey also highlights deeper architectures, multiple faces, temporal 3D color features, and multimodal inputs.
  • Conclusion: Progress in psychology, cognitive science, neuroscience, face recognition, detection, and tracking may contribute techniques and knowledge to occlusion-tolerant FEA.The survey also proposes context-sensitive algorithms and further study of deep learning for inherent occlusion handling.
Loading 1802.08784v1…