PaperlayerView exact revision on arXiv

Source-linked AI summary

Robust LSTM-Autoencoders for Face De-Occlusion in the Wild

Fang Zhao, Jiashi Feng, Jian Zhao, Wenhan Yang, Shuicheng Yan

arXiv:1612.08534v1cs.CV

TL;DR

Partial facial occlusion remains difficult for recognition, while existing de-occlusion methods often assume constrained or closed-set conditions. The paper proposes Robust LSTM-Autoencoders, which encode facial patches robustly and recurrently reconstruct faces while detecting occlusions, with identity-preserving adversarial training. Experiments on synthetic and real occlusion datasets show effective restoration and improved recognition performance over other de-occlusion methods.

  • Problem

    Existing face recognition and de-occlusion methods struggle with partial occlusion and often work only under constrained conditions such as closed subject sets.

  • Method

    RLA uses a multi-scale spatial LSTM encoder, a dual-channel recurrent decoder for face reconstruction and occlusion detection, and identity-preserving supervised and adversarial training.

  • Results

    Experiments on synthetic and real occlusion datasets show that RLA outperforms other de-occlusion methods in recovered-face quality and occluded-face recognition accuracy.

  • Takeaways & Limitations

    RLA effectively restores varied facial occlusions in open-test-set settings while improving recognition performance on partially occluded faces.

Abstract

from arXiv · show

Face recognition techniques have been developed significantly in recent years. However, recognizing faces with partial occlusion is still challenging for existing face recognizers which is heavily desired in real-world applications concerning surveillance and security. Although much research effort has been devoted to developing face de-occlusion methods, most of them can only work well under constrained conditions, such as all the faces are from a pre-defined closed set. In this paper, we propose a robust LSTM-Autoencoders (RLA) model to effectively restore partially occluded faces even in the wild. The RLA model consists of two LSTM components, which aims at occlusion-robust face encoding and recurrent occlusion removal respectively. The first one, named multi-scale spatial LSTM encoder, reads facial patches of various scales sequentially to output a latent representation, and occlusion-robustness is achieved owing to the fact that the influence of occlusion is only upon some of the patches. Receiving the representation learned by the encoder, the LSTM decoder with a dual channel architecture reconstructs the overall face and detects occlusion simultaneously, and by feat of LSTM, the decoder breaks down the task of face de-occlusion into restoring the occluded part step by step. Moreover, to minimize identify information loss and guarantee face recognition accuracy over recovered faces, we introduce an identity-preserving adversarial training scheme to further improve RLA. Extensive experiments on both synthetic and real datasets of faces with occlusion clearly demonstrate the effectiveness of our proposed RLA in removing different types of facial occlusion at various locations. The proposed method also provides significantly larger performance gain than other de-occlusion methods in promoting recognition performance over partially-occluded faces.

I. INTRODUCTION

The paper targets face de-occlusion in the wild, where existing methods are constrained by closed-set assumptions and varied real-world occlusions. RLA progressively restores occluded regions using recurrent encoding, decoding, detection, and identity-preserving training.

  • Motivation: Existing de-occlusion methods often work only in constrained settings, limiting their applicability to complex scenarios such as surveillance.Prior approaches commonly assume faces come from a predefined set or share subjects between training and testing.
  • Proposed approach: RLA addresses open-test-set face de-occlusion with varied occlusion types by restoring occluded face parts through several successive processes.Each process can use the recovered output from the previous step.
  • Proposed approach: The model uses a multi-scale spatial LSTM encoder and a dual-channel LSTM decoder for robust representation, face reconstruction, and occlusion detection.The decoder jointly reconstructs the face and detects occluded regions, while information is exchanged between the two channels.
  • Identity preservation: The proposed framework adds supervised and adversarial learning to preserve facial details and identity information in recovered faces.These components are intended to support recognition accuracy over de-occluded faces.

III. ROBUST LSTM-AUTOENCODERS FOR FACE DE-OCCLUSION

The method builds on LSTM sequence modeling to process facial information recurrently. Its architecture combines spatial patch encoding with recurrent decoding and uses LSTM memory to retain long-range context.

  • A. Long Short-Term Memory: LSTM uses memory units and input, forget, and output gates to regulate information over sequential inputs.Its memory formulation is designed to distribute gradient contributions across time steps.
  • B. Robust LSTM-Autoencoders: The proposed framework consists of a multi-scale spatial LSTM encoder and a dual-channel LSTM decoder for concurrent face reconstruction and occlusion detection.The framework and training process are illustrated in Fig. 2.
  • A. Long Short-Term Memory: LSTM memory helps preserve long-range context because its memory unit accumulates activity across time steps, mitigating vanishing gradients in long sequences.This property supports the paper’s recurrent treatment of face de-occlusion.
  • A. Long Short-Term Memory: LSTM is used for face de-occlusion because the task can be treated as sequential processing of facial information.The paper connects recurrent modeling to progressively removing facial occlusion.

B. Robust LSTM-Autoencoders

Face de-occlusion is formulated as recovering an occlusion-free face from an occluded observation. RLA parameterizes this recovery with an autoencoder but performs restoration progressively because large contiguous occlusions are difficult to remove in one step.

  • Problem formulation: Face de-occlusion seeks a function that minimizes the difference between the recovered face and its corresponding occlusion-free face.The input is an occluded face, and the target is its clean counterpart.
  • Autoencoder formulation: The recovering function is parameterized as an autoencoder composed of encoder and decoder functions with learned weights and biases.This formulation follows the autoencoder approach used for image denoising and inpainting.
  • Motivation for recurrent restoration: Direct autoencoder recovery is difficult for large spatially contiguous occlusions under unconstrained variations in resolution, illumination, pose, and expression.The paper motivates a recurrent, progressive restoration strategy rather than one-step removal.

1) Multi-scale Spatial LSTM Encoder:

The model encodes an occluded face through sequential multi-scale spatial patches, then repeatedly reconstructs the face while detecting and compensating occluded regions.

  • 1) Multi-scale Spatial LSTM Encoder:: Sequential multi-scale spatial LSTM processing limits occlusion contamination because some patches remain occlusion-free during encoding.The encoder also incorporates coarse contextual regions and controls noisy patch information through its gates.
  • 1) Multi-scale Spatial LSTM Encoder:: The encoder uses neighboring patch memory states to incorporate spatial context and outputs its final hidden state as the face representation.The resulting representation is recurrently decoded for face recovery.
  • 2) Dual-Channel LSTM Decoder:: The reconstruction decoder refines its output across multiple steps, producing the final face after sigmoid transformation at step T.Its recurrent updates use the previous output and the encoder’s learned representation.
  • 2) Dual-Channel LSTM Decoder:: A second LSTM detection decoder uses reconstruction hidden states to localize occlusion, then combines reconstructed and observed pixels according to occlusion scores.The recovered face is a weighted sum of the reconstructed face and the occluded input.

3) Optimization:

RLA is optimized in stages rather than training all three networks simultaneously, using separate pre-training before joint refinement.

  • 3) Optimization:: The joint reconstruction objective is a mean square error optimized through standard stochastic gradient descent.The resulting gradients update the parameters of each LSTM network.
  • 3) Optimization:: The reconstruction gradients are zeroed on non-occluded regions using occlusion scores, directing the reconstruction network toward occluded parts.These gradients are used in backpropagation to update the LSTM networks.
  • 3) Optimization:: Multi-stage optimization first pre-trains the encoder and reconstruction decoder, then trains occlusion detection with fixed parameters, and finally jointly retrains all networks.The separate pre-training rounds provide initial parameters for subsequent joint optimization.

C. Identity-Preserving Face De-Occlusion

Identity-preserving RLA adds supervised identity guidance and adversarial training to improve recovered-face details while reducing artifacts.

  • C. Identity-Preserving Face De-Occlusion: The supervised CNN jointly guides pixel reconstruction and identity classification, preserving high-level facial identity while recovering low-level structure.Its parameters are fixed during fine-tuning after learning filters from original occlusion-free faces.
  • C. Identity-Preserving Face De-Occlusion: Adversarial training makes RLA generate recovered faces that the discriminator cannot distinguish from original occlusion-free faces.The generator and discriminator are optimized alternately.
  • C. Identity-Preserving Face De-Occlusion: The adversarial component addresses artifacts that arise when the model fits recovered faces to the classification network.Pixelwise fitting alone can produce mean-valued appearances when multiple facial completions are plausible.
  • C. Identity-Preserving Face De-Occlusion: The identity-preserving RLA pipeline initializes from pre-trained RLA, supervised CNN, and adversarial CNN components before end-to-end fine-tuning.The pipeline includes fixed network parameters during the fine-tuning stage.

IV. EXPERIMENTS

The evaluation uses one synthesized-occlusion dataset and one real-occlusion dataset, combining qualitative de-occlusion results with quantitative face-recognition evaluation.

  • IV. EXPERIMENTS: RLA is evaluated on both synthesized and real occluded-face datasets using qualitative occlusion-removal results and quantitative face-recognition measurements.The supplied passage describes the evaluation scope but does not report numerical outcomes.

A. Datasets

Training data are synthesized from CASIA-WebFace because paired real-world occluded and clean faces are difficult to collect. The synthesis uses multiple object types and held-out templates to model varied occlusions.

  • Training Data:: Around 380,000 near-frontal CASIA-WebFace faces are used to synthesize occlusions from 9 types of common objects.CASIA-WebFace contains 10,575 subjects and 494,414 images; the selected faces span −45° to +45°.
  • Training Data:: Each occluding object has 100 templates, split between training and testing to prevent template reuse across splits.

2) Test Data:

Testing uses LFW and the real-occlusion 50OccPeople dataset, with occlusion categories and implementation settings designed to evaluate varied facial corruption.

  • Test Data:: LFW contains 13,233 images from 5,749 subjects and has no subject overlap with CASIA-WebFace.
  • Test Data:: 50OccPeople contains 50 subjects and 1,200 real-occlusion images, including one normal image and 23 occluded images per subject.
  • Settings and Implementation Details: The encoder and decoder each use two LSTM layers with 2,048 hidden units, while the decoder runs for 8 steps.
  • Test Data:: IP-RLA is qualitatively evaluated across quarter-face, half-face, upper-face, and lower-face occlusions on occluded LFW.
  • Settings and Implementation Details: PCA, AE, SRC, and SSDA serve as comparison baselines alongside the proposed model.

C. Results and Comparisons

RLA progressively removes occlusion and preserves subject-specific facial structure across synthetic and real occlusion settings. IP-RLA recovers varied facial appearances, though exact reconstruction is harder when large upper or lower regions are missing.

  • Occlusion Removal:: RLA progressively adds facial structures and sharpens predicted occlusion regions across decoding steps.Intermediate outputs show a coarse face profile and occlusion estimate refined using previous-step states.
  • Occlusion Removal:: On occluded LFW, RLA restores occluded parts while retaining non-occluded appearance, and IP-RLA sharpens discriminative edges and textures.PCA blurs occlusions, SRC damages non-occluded appearance, while the reconstruction-only channel damages fine-grained structures.
  • Occlusion Removal:: IP-RLA recovers diverse subject-specific appearances rather than producing a mean occluded facial part under identical occlusion conditions.
  • Occlusion Removal:: IP-RLA handles quarter- and half-face occlusions with high similarity, while upper- or lower-face recovery can differ from the original but preserves general attributes.The reported attributes include gender, size, skin color, and expression.
  • Occlusion Removal:: On the real 50OccPeople dataset, RLA and IP-RLA still obtain good de-occlusion results despite training only on synthetic occluded faces.

2) Face Recognition:

RLA is evaluated by whether de-occluded faces improve verification across synthetic and real occlusion settings. Its spatial LSTM encoder, dual-channel decoder, and identity-preserving training support progressively recovered faces while retaining discriminative information.

  • Verification results: IP-RLA achieves better verification performance for every occlusion type on occluded LFW than PCA, AE, SRC, and SSDA.The comparison uses equal error rates on 12,000 generated test pairs.
  • Verification results: Occlusion detection lowers error rates beyond face reconstruction alone by preserving discriminative information in non-occluded regions.The reported comparison attributes the improvement to retaining non-occluded parts during recovery.
  • Occlusion patterns: Quarter and left/right-half occlusions yield larger recognition gains, while upper or lower-face recovery still improves over unrecovered occluded faces.The paper relates these patterns to facial symmetry and learned relations between upper and lower facial regions.
  • Verification results: Our model outperforms other methods on overall verification across all occlusion types in the 50OccPeople dataset, generalizing to real occluded face data.The evaluation averages equal error rates over 10,000 sampled positive and negative pairs.
  • Model design: RLA progressively reconstructs faces and detects occlusion with spatial and dual-channel LSTM networks, while supervised and adversarial CNNs preserve person identity information.The encoder reads face patches sequentially, and the decoder reconstructs and detects occlusion step by step.
Loading 1612.08534v1…