PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Task Convolutional Neural Network for Pose-Invariant Face Recognition

Xi Yin, Xiaoming Liu

arXiv:1702.04710v2cs.CV

TL;DR

Face recognition is challenged by entangled pose, illumination, and expression variations, while comprehensive evidence for multi-task learning remains limited. The paper combines identity classification with PIE side tasks, dynamic weighting, and pose-directed routing, finding improved recognition and broad benchmark performance, including 79.55% rank-1 identification for p-CNN on Multi-PIE.

  • Problem

    Comprehensive study of multi-task learning for face recognition is lacking, despite MTL's use in other vision problems and the challenge posed by PIE variations.

  • Method

    The paper learns identity with PIE side tasks using CNN-based dynamic loss weighting, pose-specific feature routes, stochastic routing, and energy-based weight analysis.

  • Results

    79.55% rank-1 identification was achieved by p-CNN on Multi-PIE, while dynamic weighting improved performance to 79.35% from 77.59% with fixed weighting.

  • Takeaways & Limitations

    MTL side tasks serve as regularizations that disentangle PIE variations from learned identity features, especially improving recognition for large poses.

Abstract

from arXiv · show

This paper explores multi-task learning (MTL) for face recognition. We answer the questions of how and why MTL can improve the face recognition performance. First, we propose a multi-task Convolutional Neural Network (CNN) for face recognition where identity classification is the main task and pose, illumination, and expression estimations are the side tasks. Second, we develop a dynamic-weighting scheme to automatically assign the loss weight to each side task, which is a crucial problem in MTL. Third, we propose a pose-directed multi-task CNN by grouping different poses to learn pose-specific identity features, simultaneously across all poses. Last but not least, we propose an energy-based weight analysis method to explore how CNN-based MTL works. We observe that the side tasks serve as regularizations to disentangle the variations from the learnt identity features. Extensive experiments on the entire Multi-PIE dataset demonstrate the effectiveness of the proposed approach. To the best of our knowledge, this is the first work using all data in Multi-PIE for face recognition. Our approach is also applicable to in-the-wild datasets for pose-invariant face recognition and achieves comparable or better performance than state of the art on LFW, CFP, and IJB-A datasets.

I. INTRODUCTION

The paper frames face recognition as a multi-task problem in which identity classification is supported by pose, illumination, and expression estimation. It proposes dynamic task weighting and pose-directed learning to disentangle PIE variations from identity features, and evaluates the approach comprehensively across Multi-PIE and in-the-wild benchmarks.

  • Large pose, illumination, and expression variations increase intra-person variation and challenge face recognition systems.
  • The proposed MTL formulation uses identity classification as the main task and PIE estimations as side tasks to improve face recognition.
  • MTL is intended to disentangle PIE variations from identity features, allowing identity to exclude dimensions modeling pose variation.
  • A dynamic-weighting scheme automatically assigns loss weights to side tasks because their contributions may differ and brute-force search is impractical.
  • The pose-directed CNN groups poses into separate routes, learns generic and pose-specific identity features, and fuses them through stochastic routing.
  • The study uses the full PIE range in Multi-PIE and reports comparable or superior performance to state-of-the-art methods on Multi-PIE, LFW, CFP, and IJB-A.

II. RELATED WORK

The paper reviews pose-invariant face recognition, multi-task learning, and Multi-PIE usage, positioning its work as joint pose-invariant feature extraction across poses.

  • The review focuses on CNN-based methods related to pose variation, multi-task learning, and the Multi-PIE dataset.
  • Pose-Invariant Face Recognition: Pose-invariant face recognition methods include multi-view subspace learning, pose-invariant feature extraction, face synthesis, and hybrids.
  • Pose-Invariant Face Recognition: This work extracts pose-invariant identity features for all poses jointly in one CNN framework rather than learning separate models for each pose.

B. Multi-Task Learning

The paper situates its contribution within multi-task learning research and addresses how to determine task importance without independently searching every task weight.

  • Multi-task learning has been studied broadly across machine learning, natural language processing, and computer vision.
  • Regularizations: Prior task-wise early stopping can permanently remove a task after its loss stops decreasing.
  • Task Weighting: The proposed dynamic-weighting scheme searches only the overall side-task weight and lets the CNN distribute it among side tasks.
  • Task Weighting: When one task saturates, its dynamic weight decreases without explicitly stopping that task.

III. THE PROPOSED APPROACH

The proposed approach combines multi-task CNN learning with dynamic side-task weights and pose-directed grouping to learn pose-specific identity features jointly.

  • The m-CNN treats identity classification as the main task and pose, illumination, and expression estimations as side tasks.
  • The m-CNN uses dynamic weights for the main and PIE estimation tasks.
  • The p-CNN separates poses into groups and jointly learns pose-specific identity features for each group.

A. Multi-Task CNN

The multi-task CNN shares convolutional features across identity and PIE tasks, attaches task-specific classifiers, and uses identity features for cosine-similarity face matching.

  • The network uses five blocks, each containing two convolutional layers and a pooling layer, with batch normalization and ReLU after each convolution.
  • Training data contain images with identity labels and side-task labels, including pose labels.
  • The m-CNN maps each input image to shared high-level features x through a nonlinear convolutional and pooling representation.
  • The shared features are pool5 and feed fully connected task classifiers, including identity classification over the training identities.
  • Figure 2 indicates that blocks reduce spatial dimensions, increase channels, and identify the striped identity features used during testing.
  • The identity output is passed through softmax to obtain subject probabilities and an estimated identity.
  • The combined loss weights identity and PIE tasks; setting all side-task weights to zero yields a single-task CNN.
  • At testing, features before the identity softmax layer are used for face recognition with cosine similarity.

B. Dynamic-Weighting Scheme

The dynamic-weighting scheme avoids separately searching all side-task weights by learning their relative contributions during training. A searched overall side-task weight is distributed among pose, illumination, and expression estimation losses.

  • CNN-based MTL requires task-loss weights, but equal weighting and brute-force search are unsuitable when side tasks contribute differently and training is expensive.
  • The scheme automatically assigns loss weights to each side task during training.
  • The method fixes the main-task weight at αd = 1 and searches only the summed side-task weight ϕs = αp + αl + αe on a validation set.
  • A fully connected layer and softmax layer learn positive side-task weight percentages µp, µl, and µe that sum to 1.
  • Multiplying ϕs by the learned percentages produces the dynamic loss weight for each side task.

C. Pose-Directed Multi-Task CNN

The pose-directed CNN uses pose labels to route training samples through pose-specific branches while jointly learning generic and pose-specific identity features. At testing, stochastic routing combines pose-probability-weighted comparisons to reduce sensitivity to pose-estimation errors.

  • C. Pose-Directed Multi-Task CNN: Pose-directed MTL applies divide-and-conquer by grouping samples according to pose and routing them through different network paths.
  • C. Pose-Directed Multi-Task CNN: The pose-directed branch separates training images into left-profile, frontal, and right-profile groups using pose labels.
  • C. Pose-Directed Multi-Task CNN: Each pose group learns a pose-specific mapping to a joint space, producing separate weight sets while retaining valid inputs for empty groups.
  • C. Pose-Directed Multi-Task CNN: The model learns generic identity features robust to all poses and pose-specific features robust within a small pose range.
  • C. Pose-Directed Multi-Task CNN: Figure 4 depicts joint learning of pose-specific identity features across all pose groups.
  • C. Pose-Directed Multi-Task CNN: During testing, stochastic routing averages generic-feature distance with pose-specific distances weighted by pose probabilities across all pair comparisons.

IV. EXPERIMENTS

The experiments evaluate the proposed models on controlled and in-the-wild face datasets, while analyzing how multi-task learning affects the learned representation.

  • IV. EXPERIMENTS: Experiments cover Multi-PIE identification with pose, illumination, and expression side tasks, plus LFW, CFP, and IJB-A evaluation with pose estimation as the side task.
  • IV. EXPERIMENTS: The Multi-PIE analysis finds that side tasks regularize the network toward a disentangled identity representation for PIE-invariant face recognition.

A. Face Identification on Multi-PIE

On the entire Multi-PIE dataset, multi-task learning improves identity recognition, particularly for profile faces. Dynamic weighting and pose-directed routing further improve performance over single-task and fixed-weight baselines.

  • Multi-PIE contains 754,200 images of 337 subjects across four sessions, 15 cameras, 19 illuminations, and six expressions.
  • Effects of MTL: 79.35% rank-1 identification is achieved by m-CNN with all side tasks and dynamic weights, compared with 77.59% using fixed weights.
  • Effects of MTL: 79.55% rank-1 identification is achieved by p-CNN, which adds pose-directed routing and fuses generic with pose-specific identity features.
  • Dynamic-Weighting Scheme: Dynamic weighting emphasizes the easiest or most helpful side task, with pose receiving greater weight as its classification becomes useful and expression or illumination losses decrease.
  • Compare to Other Methods: 91.27% performance versus 76.89% in prior work corresponds to a 62% relative error reduction in the reported comparison.

B. How our m-CNN works?

The energy analysis indicates that MTL allocates shared-feature dimensions differently across tasks. Identity-specific weighting can suppress dimensions modeling pose, illumination, and expression variation, improving identity representation.

  • Energy vectors quantify the importance of each shared-feature dimension to identity, pose, illumination, and expression tasks.
  • The concatenated task-weight matrix shows a block-wise allocation of shared dimensions, with separate feature subsets associated with different tasks.
  • Identity performance using the n largest raw-energy dimensions rises and then drops as dimensions modeling PIE variations are added.
  • During training, energy-vector means decrease while standard deviations increase, reflecting a shift from evenly distributed to more concentrated feature usage.
  • Increasing the overall side-task loss weight changes how many shared dimensions are assigned to side tasks.

C. Unconstrained Face Recognition

On unconstrained datasets, the proposed models improve most when pose variation is large. The approach remains effective when estimated pose labels replace unavailable manual side-task labels.

  • Estimated yaw angles from PIFA supply three pose-group labels because manual side-task labels are unavailable for the training data.
  • Performance on CFP: On CFP frontal-to-profile verification, p-CNN reaches 94.57%, while the reported accuracy of 94.39% gives a 14.8% relative error reduction over DR-GAN at 93.41%.
  • Performance on CFP: On CFP frontal-to-frontal verification, m-CNN and p-CNN improve only slightly over s-CNN because pose variation is limited.
  • Performance on LFW and IJB-A: The paper reports state-of-the-art performance on LFW, CFP, and IJB-A, while IJB-A evaluation includes retraining without overlapping subjects and fold-specific fine-tuning.

V. CONCLUSIONS

The paper presents MTL as an effective approach to pose-invariant face recognition, combining dynamic task weighting, pose-directed routing, and energy-based analysis. Results across Multi-PIE and in-the-wild benchmarks support disentangling PIE variations from identity features.

  • MTL with PIE side tasks, dynamic weighting, and pose-directed stochastic routing improves pose-invariant face recognition, especially for large poses.
  • Experiments use the entire Multi-PIE dataset and report state-of-the-art performance on LFW, CFP, and IJB-A.
  • The energy analysis supports the view that side tasks disentangle PIE variations from learned identity features.
Loading 1702.04710v2…