PaperlayerView exact revision on arXiv

Source-linked AI summary

Robust Face Recognition via Multimodal Deep Face Representation

Changxing Ding, Dacheng Tao

arXiv:1509.00244v1cs.CV

TL;DR

Multimedia face recognition must handle substantial appearance variation and large-scale identity matching. MM-DFR addresses this challenge by combining specialized CNNs across complementary modalities with SAE-based feature fusion, achieving over 99.0% recognition rate on LFW.

  • Problem

    Multimedia face recognition is difficult because social-network images exhibit rich pose, illumination, expression, and occlusion variations, while accuracy degrades with increasing identities.

  • Method

    MM-DFR uses specialized CNNs to extract complementary features from multiple face-image modalities, concatenates them, and compresses them with a three-layer SAE.

  • Results

    Higher than 99.0% recognition rate was achieved on LFW using a small multimodal ensemble trained with a publicly available training set.

  • Takeaways & Limitations

    Complementary multimodal information substantially improves face-recognition performance over a single-modal approach.

Abstract

from arXiv · show

Face images appeared in multimedia applications, e.g., social networks and digital entertainment, usually exhibit dramatic pose, illumination, and expression variations, resulting in considerable performance degradation for traditional face recognition algorithms. This paper proposes a comprehensive deep learning framework to jointly learn face representation using multimodal information. The proposed deep learning structure is composed of a set of elaborately designed convolutional neural networks (CNNs) and a three-layer stacked auto-encoder (SAE). The set of CNNs extracts complementary facial features from multimodal data. Then, the extracted features are concatenated to form a high-dimensional feature vector, whose dimension is compressed by SAE. All the CNNs are trained using a subset of 9,000 subjects from the publicly available CASIA-WebFace database, which ensures the reproducibility of this work. Using the proposed single CNN architecture and limited training data, 98.43% verification rate is achieved on the LFW database. Benefited from the complementary information contained in multimodal data, our small ensemble system achieves higher than 99.0% recognition rate on LFW using publicly available training set.

I. INTRODUCTION

Face recognition in multimedia applications is challenged by large-scale matching and substantial pose, illumination, expression, and occlusion variation. The paper proposes MM-DFR, which combines complementary multimodal CNN features and SAE compression to improve robust face representation.

  • Multimedia face recognition is difficult because real-world images contain rich pose, illumination, expression, and occlusion variations.
  • Recognition accuracy typically degrades dramatically as the number of face identities increases.
  • Traditional local-descriptor representations are too shallow to distinguish complex nonlinear facial appearance variations.
  • MM-DFR uses an ensemble of CNNs to extract complementary features from holistic, 3D-rendered frontal, and uniformly sampled patch modalities.
  • The extracted CNN features are concatenated and compressed into a compact representation using a three-layer stacked auto-encoder.
  • Using a publicly available training set, MM-DFR achieves higher than 99.0% recognition rate on LFW.

II. RELATED STUDIES

Prior face-recognition methods commonly use local descriptors or single deep representations, limiting their ability to capture diverse facial information. MM-DFR combines multiple modalities, specialized CNN processing, and nonlinear SAE compression to address these limitations.

  • Face representations are broadly grouped into local descriptor-based and deep learning-based approaches.
  • Local descriptors include handcrafted and learning-based features, but complex facial appearance changes call for nonlinear representation techniques.
  • Single face representations are restrictive because they cannot fully capture the diverse information contained in face images.
  • MM-DFR extracts features from holistic images, 3D-rendered frontal faces, and uniformly sampled patches.
  • The framework uses SAE rather than PCA to compress high-dimensional deep features through nonlinear transformations.
  • MM-DFR differs from earlier methods by using 3D rendering for frontal-face extraction and sparse-landmark-guided uniform patch sampling.

III. MULTIMODAL DEEP FACE REPRESENTATION

MM-DFR extracts complementary facial features from multiple image modalities using an ensemble of CNNs, then combines them for face representation. Its modalities include holistic images, a 3D pose-normalized face, and uniformly sampled patches processed by CNN architectures of different sizes.

  • Multimodal inputs: MM-DFR uses CNNs to extract features from an original holistic face image, a 3D pose-normalized holistic image, and six image patches.The holistic image is 165 × 120 pixels, while each patch is 100 × 100 pixels.
  • CNN architectures: Two CNN designs are used: NN2 is deeper and wider for CNN-H1, while the smaller, more efficient NN1 is applied to the other seven CNNs.The design assigns the larger model to the original holistic image and the smaller model to pose-normalized and patch inputs.
  • CNN architectures: NN1 contains 10 convolutional layers, four max-pooling layers, one mean-pooling layer, and two fully connected layers, whereas NN2 contains 12 convolutional layers.Both architectures use 3 × 3 convolutional filters; NN2 is described as more robust to highly nonlinear facial appearance variations but more vulnerable to overfitting.
  • CNN architectures: The CNN feature representation uses a 512-dimensional Fc6 output and a 9,000-unit Fc7 layer matching the number of training subjects.Dropout with ratio 0.4 is used on the first fully connected layer to regularize the model.
  • Multimodal inputs: Nine uniformly distributed 3D landmarks are projected onto 2D face images to guide semantic patch sampling.A square 100 × 100-pixel patch is cropped around each projected landmark; six patches are retained using facial symmetry and horizontal-flip augmentation.

B. Combination of CNNs using Stacked Auto-Encoder

The framework concatenates the features from eight CNNs into a high-dimensional vector and compresses it with a three-layer stacked auto-encoder. The SAE uses nonlinear transformations to produce a compact face signature.

  • Feature fusion: Features from eight CNNs are concatenated into a single high-dimensional feature vector.Each CNN feature has dimension 512, so the concatenated representation has dimension 8 × 512 before compression.
  • Feature fusion: A three-layer SAE reduces the concatenated vector to a compact face signature.The three auto-encoder layers contain 2048, 1024, and 512 neurons, respectively; the last encoder output is used as the signature.
  • Feature fusion: The SAE is motivated by the impractical dimensionality of the concatenated vector for real-world face recognition applications.Unlike PCA, the SAE is described as learning nonlinear feature transformations.
  • SAE nonlinearities: Sigmoid and tanh nonlinearities are evaluated for the SAE, with input normalization matched to each activation's output range.The concatenated vector is normalized to [0, 1] for sigmoid and [−1, +1] for tanh.

IV. FACE MATCHING WITH MM-DFR

MM-DFR matches faces using either cosine similarity without supervision or Joint Bayesian modeling with nearest-neighbor identification.

  • The unsupervised mode measures similarity between two MM-DFR features using cosine distance.
  • The supervised mode employs Joint Bayesian to model face generation and evaluate pair similarity.
  • Nearest-neighbor classification is used for face identification in both evaluation modes.
  • Joint Bayesian computes a log-likelihood ratio between same-subject and different-subject hypotheses.
  • The resulting log-likelihood ratio is treated as the similarity score in experiments.

V. EXPERIMENTAL EVALUATION

The evaluation uses internet-collected LFW and CASIA-WebFace data, with training focused on 9,000 CASIA-WebFace subjects and augmented CNN inputs.

  • Experiments evaluate MM-DFR on the large-scale unconstrained LFW and CASIA-WebFace databases.
  • LFW contains 13,233 images from 5,749 subjects with substantial pose, illumination, and expression variation.
  • CASIA-WebFace training uses 471,592 images from the top 9,000 subjects, while 22,822 images from 1,575 subjects form testing data.
  • NN1 training data totals 943,184 images after horizontal flipping, while NN2 uses more aggressive augmentation.
  • NN2 augmentation includes horizontal flipping, image jittering, and downsampling, producing about 1.8 million training images.
  • The multi-stage CNN training combines softmax classification with two further epochs of triplet-loss fine-tuning.

A. Performance Comparison with Single CNN Model

The single-CNN evaluation studies ReLU placement and compares the proposed model with prior CNNs on LFW under unsupervised and supervised protocols.

  • The comparison evaluates NN1, NN1+C52R, and NN1+C52R+Fc6R using unsupervised and supervised paradigms.
  • ReLU after Conv52 or Fc6 harms CNN performance in the reported experiments.
  • The results support dense features and a linear projection from Conv52 to the low-dimensional Fc6 subspace.
  • The proposed CNN consistently outperforms state-of-the-art CNN models under both evaluation paradigms.
  • Triplet-loss fine-tuning mainly improves performance in the unsupervised mode in this experiment.

B. Performance of the Eight CNNs in MM-DFR

The eight MM-DFR CNNs are evaluated individually, revealing modality-specific performance and complementary contributions to the fused representation.

  • Table IV evaluates each of the eight CNNs using both unsupervised and supervised face-verification paradigms.
  • Supervised evaluation uses a unified PCA subspace dimension of 110 for all eight CNNs.
  • Feature normalization typically improves supervised-paradigm performance by 0.1% to 0.4%.
  • CNN-H1 outperforms CNN-H2 when both use the NN1 structure, despite extracting features from holistic face images.
  • Excluding CNN-H2 slightly reduces MM-DFR performance on View 1, indicating complementary information from its modality.
  • The performance of patch-level CNNs CNN-P1 through CNN-P6 varies with the discriminative power of their corresponding patches.

C. Fusion of CNNs with SAE

The SAE fuses features from eight CNNs after feature averaging and L2 normalization, while tanh nonlinearity outperforms sigmoid because it preserves feature sign information.

  • C. Fusion of CNNs with SAE: Eight CNN features are averaged across original and horizontally flipped images, L2-normalized, concatenated, and fused through a three-layer SAE.The SAE performs feature-level fusion after preprocessing each CNN's averaged feature.
  • C. Fusion of CNNs with SAE: The multimodal system is evaluated against a single-modality CNN on face verification.The comparison is identified in the supplied figure caption.
  • C. Fusion of CNNs with SAE: 97.90% verification rate is reported on LFW View1 for the better-performing SAE nonlinearity.The supplied passage identifies SAE-TANH as considerably outperforming SAE-SIG.
  • C. Fusion of CNNs with SAE: SAE-TANH outperforms SAE-SIG because sigmoid removes feature-element sign information that remains available under tanh.The passage reports feature ranges of [0, 1] for sigmoid and [−1, 1] for tanh.

D. Performance of MM-DFR with Joint Bayesian

MM-DFR features are used as face signatures with Joint Bayesian, outperforming a single-modal CNN and showing gains from multimodal fusion, while identification remains difficult at scale.

  • D. Performance of MM-DFR with Joint Bayesian: MM-DFR-JB uses the MM-DFR output as the face-image signature and compares it with CNN-H1 plus Joint Bayesian.The comparison is reported in Table V and illustrated with ROC curves in Fig. 9.
  • D. Performance of MM-DFR with Joint Bayesian: 99.10% actual performance is reported for MM-DFR-JB after excluding five labeling errors in LFW.This is an adjusted result based on removing the stated labeling errors.
  • D. Performance of MM-DFR with Joint Bayesian: The 8-net ensemble outperforms DeepID2, despite DeepID2 using as many as 25 CNNs.The comparison is presented alongside the authors' smaller training and model scale.
  • D. Performance of MM-DFR with Joint Bayesian: 76.53% rank-1 identification rate is achieved on the CASIA-WebFace test setting, indicating that large-scale real-world identification remains difficult.The experiment uses 7,875 gallery images and 14,947 probe images from 1,575 subjects.
  • D. Performance of MM-DFR with Joint Bayesian: MM-DFR-JB exceeds CNN-H1 plus JB by as much as 4.27%, while combining two modalities adds 2.25% over CNN-H1.CNN-H1 plus JB itself outperforms CNN-H2 plus JB, partly because CNN-H1 uses a larger architecture and more aggressive augmentation.

VI. CONCLUSION

The paper addresses multimedia face-recognition challenges by combining carefully designed CNNs with complementary multimodal data and SAE-based feature fusion. Experiments report advantages over single-modal algorithms and some deep models.

  • VI. CONCLUSION: Multimedia face recognition is challenged by appearance changes caused by pose, expression, and illumination variations.These variations motivate the proposed deep architecture.
  • VI. CONCLUSION: MM-DFR extracts features from holistic faces, 3D-rendered frontal poses, and sampled patches using a set of CNNs.The modalities are designed to provide complementary information.
  • VI. CONCLUSION: A stacked auto-encoder performs feature-level fusion and nonlinear dimension reduction for the CNN-extracted features.The conclusion identifies SAE fusion as part of the framework.
  • VI. CONCLUSION: Experiments on face verification and identification show clear advantages for MM-DFR over single-modal algorithms and some state-of-the-art deep models.The conclusion attributes the reported advantage to effective use of multimodal information.
Loading 1509.00244v1…