PaperlayerView exact revision on arXiv

Source-linked AI summary

Trunk-Branch Ensemble Convolutional Neural Networks for Video-based Face Recognition

Changxing Ding, Dacheng Tao

arXiv:1607.05427v2cs.CV

TL;DR

Video face recognition is challenged by blur, pose variation, occlusion, scarce training data, and demanding efficiency requirements. The paper combines artificial-blur training, the efficient TBE-CNN architecture, and MDR-TL, with experiments reporting advantages on three video face databases. Its scope also includes the computational cost of ensemble methods and the choice of artificial blur kernels.

  • Problem

    Video face recognition requires robust recognition under blur, pose variation, occlusion, limited video data, and higher efficiency demands than still-image recognition.

  • Method

    The framework trains CNNs with clear and artificially blurred still faces, uses TBE-CNN to combine holistic and facial-component features efficiently, and applies MDR-TL end to end.

  • Results

    The proposed algorithm achieves superior performance on PaSC, COX Face, and YouTube Faces across S2V, V2S, and V2V evaluation tasks.

  • Takeaways & Limitations

    The approach provides a CNN-based VFR method that addresses blur, pose variation, and occlusion while reducing the computational cost of ensemble modeling.

  • Takeaways & Limitations

    The method retains simpler artificial blur kernels because the benefit of more realistic kernels was slight on PaSC, and ensemble fusion otherwise incurs significant time cost.

Abstract

from arXiv · show

Human faces in surveillance videos often suffer from severe image blur, dramatic pose variations, and occlusion. In this paper, we propose a comprehensive framework based on Convolutional Neural Networks (CNN) to overcome challenges in video-based face recognition (VFR). First, to learn blur-robust face representations, we artificially blur training data composed of clear still images to account for a shortfall in real-world video training data. Using training data composed of both still images and artificially blurred data, CNN is encouraged to learn blur-insensitive features automatically. Second, to enhance robustness of CNN features to pose variations and occlusion, we propose a Trunk-Branch Ensemble CNN model (TBE-CNN), which extracts complementary information from holistic face images and patches cropped around facial components. TBE-CNN is an end-to-end model that extracts features efficiently by sharing the low- and middle-level convolutional layers between the trunk and branch networks. Third, to further promote the discriminative power of the representations learnt by TBE-CNN, we propose an improved triplet loss function. Systematic experiments justify the effectiveness of the proposed techniques. Most impressively, TBE-CNN achieves state-of-the-art performance on three popular video face databases: PaSC, COX Face, and YouTube Faces. With the proposed techniques, we also obtain the first place in the BTAS 2016 Video Person Recognition Evaluation.

1 INTRODUCTION

Video-based face recognition must handle blur, pose variation, occlusion, limited video training data, and strict efficiency demands. The paper addresses these challenges with simulated blur training, an efficient trunk-branch CNN ensemble, and a regularized triplet loss, achieving superior performance across three databases and three matching tasks.

  • Challenges: VFR is harder than still-image recognition because video frames have lower quality, severe blur, broad pose variation, and occlusion.Low-cost surveillance and mobile cameras further reduce frame clarity.
  • Challenges: Limited real-world video training data and blur-free still-image databases hinder CNN-based learning for video face recognition.Existing video databases are small and highly redundant, increasing overfitting risk during direct CNN training.
  • Proposed approach: Artificially blurred still images are paired with clear images from the same class so CNN learns representations insensitive to blur.The method uses two training streams: still faces and simulated video frames created with random artificial blur.
  • Proposed approach: TBE-CNN combines holistic-face and facial-component branch representations while sharing low- and middle-level layers to reduce ensemble computation.The trunk and branches optimize high-level layers independently, and their feature maps are concatenated.
  • Proposed approach: MDR-TL regularizes triplet loss using label information so inter- and intra-class distance distributions become uniform.It is designed to enhance the discriminative power of TBE-CNN representations during end-to-end training.
  • Evaluation: Extensive evaluation on PaSC, COX Face, and YouTube Faces reports superior performance for S2V, V2S, and V2V matching.The paper evaluates the proposed algorithm systematically across all three settings.

2 RELATED WORKS

Prior VFR methods address frame quality, inter-frame redundancy, or robust per-frame feature extraction, but CNN-based approaches remain limited by video-specific distortions and scarce data. This paper positions TBE-CNN as an efficient CNN-based feature extractor robust to blur, pose variation, and occlusion.

  • VFR literature: Existing VFR research includes frame-quality evaluation, redundant-information exploitation, and robust feature extraction from individual frames.These categories cover key-frame selection, sequence or set modeling, and per-frame representation learning.
  • VFR literature: Frame-quality methods select high-quality subsets for efficient recognition, while sequence-based methods rely on robust face trackers to model facial dynamics.Other redundancy-based approaches use dictionaries or image sets.
  • Blur robustness: Deblur methods estimate blur kernels before feature extraction, whereas blur-robust methods extract features using descriptors such as LPQ.The paper notes that blur-kernel estimation is ill-posed and that no CNN-based method had yet handled VFR image blur to its knowledge.
  • Positioning: The proposed approach automatically learns representations robust to blur, pose variation, and occlusion through an efficient CNN model.It belongs to the robust per-frame feature-extraction category.
  • CNN-based VFR: CNN face-recognition advances have mainly targeted still images, while VFR studies are limited because video data are small, redundant, and prone to overfitting.The paper contrasts direct video training with still-image pretraining followed by limited video fine-tuning.
  • Architectural novelty: TBE-CNN integrates holistic and facial-component networks by sharing low- and middle-level layers rather than maintaining separate models.Its design differs from approaches that share all convolutional layers and crop parts from final feature maps.

3 TRUNK-BRANCH ENSEMBLE CNN

The section presents simulated video-like training data and TBE-CNN, combining holistic and facial-component representations while sharing lower-level computation. It also describes blur simulation choices and the resulting 512-dimensional frame representation.

  • 3.1 Artificially Simulated Video Data: Training data are expanded by generating video-like face images from still images with simulated motion and out-of-focus blur.The method assumes uniform blur kernels because faces occupy small areas in video frames.
  • 3.1 Artificially Simulated Video Data: 38 blur types are sampled by combining 12 motion-blur choices, 2 out-of-focus choices, and 24 sequential combinations.Motion blur varies kernel length and direction; out-of-focus blur varies Gaussian scale.
  • 3.1 Artificially Simulated Video Data: More realistic blur models exist, but the paper retains its simpler kernels because a more complex kernel produced only slight additional gains on PaSC.The retained kernels are those used for motion and Gaussian out-of-focus blur.
  • 3.2 Trunk-Branch Ensemble CNN: TBE-CNN combines one holistic-face trunk with two facial-component branches to learn complementary representations for video face recognition.The networks share low- and middle-level layers but use separate high-level layers.
  • 3.2 Trunk-Branch Ensemble CNN: The trunk and branches fuse their output feature maps by concatenation, then reduce the over-complete representation to a 512-dimensional frame feature.The final feature is produced by one fully connected layer.

4 TBE-CNN TRAINING

TBE-CNN is trained in stages before end-to-end optimization with MDR-TL, which combines triplet learning with constraints on subject mean representations. The strategy targets discriminative face representations while addressing non-uniform distance distributions.

  • Stage-wise training: The trunk is pretrained with softmax loss, branches are then trained separately, and the complete TBE-CNN is fine-tuned with softmax loss.Each branch includes a 256-dimensional fully connected layer during standalone training.
  • Optimization: MDR-TL directly optimizes representation discriminativeness, although its convergence is slower than softmax loss.The loss uses online triplet sampling within each batch.
  • Mean Distance Regularized Triplet Loss: MDR-TL augments triplet loss by constraining distances between mean representations of different subjects.The constraint enforces a margin between each subject mean and its nearest mean representation in the batch.
  • Mean Distance Regularized Triplet Loss: Triplet constraints and mean-representation margins jointly regularize intra-class and inter-class distance distributions.The mean-distance margin addresses non-uniform distributions that make face-verification threshold selection difficult.
  • Optimization: The final loss is optimized with equal weights for triplet and mean-distance terms using stochastic gradient descent with momentum.The number of violating mean representations is tracked within the current batch, and nearest subject means determine the active constraints.

5 VFR WITH TBE-CNN

For video face recognition, TBE-CNN processes augmented frames and averages their outputs into a compact video representation. Still images are handled as single-frame videos, and cosine similarity is used for S2V, V2S, and V2V matching.

  • Representation extraction: Horizontal flipping augments video frames before TBE-CNN extracts their representations.The same extraction procedure applies to a still image by treating it as a single-frame video.
  • Representation extraction: Average pooling fuses frame outputs into a compact video representation.This pooled representation is used for matching against video or still-image representations.
  • Face matching: Cosine similarity measures representation similarity consistently for S2V, V2S, and V2V matching.The similarity is calculated between the two resulting representations.

6 EXPERIMENTS

Experiments evaluate TBE-CNN across three video face databases and multiple matching tasks, testing simulated-video training, MDR-TL, and overall comparisons with prior methods. The proposed approach consistently performs strongly, including leading results on PaSC and advantages in still-to-video and video-to-still matching.

  • Experimental Settings: Experiments use PaSC, COX Face, and YouTube Faces, covering S2V, V2S, V2V, and verification settings.PaSC includes control and handheld conditions; COX Face includes three camera subsets; YouTube Faces contains low-resolution videos with compression artifacts.
  • Simulated Video Training: 5.6%: simulated-video training improves over still-image training on PaSC’s handheld set.The advantage is attributed to the severe blur in handheld video frames, which better matches the simulated training data.
  • Simulated Video Training: 5.9% and 5.6%: combined still and simulated-video training outperforms fine-tuning on PaSC control and handheld sets, respectively.The combined strategy directly optimizes CNN parameters for VFR and avoids relying on a small, redundant real-world video training set.
  • MDR-TL: 2.6%: MDR-TL outperforms triplet loss in the most challenging PaSC handheld setting with still-image training.With simulated-video training, the margin is around 1%, corresponding to about 20% reduction in verification error rate.
  • Training-Data Composition: Around 1.0% and 5.0%: combined still and simulated-video training exceeds still-image training on PaSC control and handheld sets, respectively.Comparisons across multiple experiments identify the proposed training-data composition as essential for blur robustness.
  • Performance Comparisons: 98.0% and 97.0%: the four-model ensemble achieves verification rates at 1% FAR on PaSC control and handheld sets, respectively.TBE-CNN achieves the best performance under the reported PaSC settings and obtains first place in the BTAS 2016 Video Person Recognition Evaluation.
  • Performance Comparisons: TBE-CNN achieves the best performance under all reported COX Face settings and has especially strong advantages in S2V and V2S tasks.The compact representation is also described as efficient for retrieval and robust to changes in the number of video frames.
  • Performance Comparisons: 2.24%: TBE-CNN* outperforms VGG Face* on YouTube Faces, while TBE-CNN remains slightly below FaceNet trained on 260 million face images.The paper notes that fair comparison with FaceNet is difficult because no similarly sized public database is available.

7 CONCLUSION

The paper addresses blur, pose variation, occlusion, and efficiency challenges in video face recognition through artificial blur augmentation, TBE-CNN, and MDR-TL. Extensive S2V, V2S, and V2V experiments show advantages over state-of-the-art methods on three video face databases.

  • Artificially blurred still images supplement limited real-world video data and encourage blur-robust CNN representations.The training set combines still images with blurred versions to address the shortage of real-world video training data.
  • TBE-CNN extracts holistic-face and facial-component representations efficiently by sharing low- and middle-level CNN layers.The architecture improves over a single CNN with only marginal increases in time and memory costs.
  • MDR-TL enhances the discriminative power of TBE-CNN representations and outperforms widely adopted triplet loss by a considerable margin.
Loading 1607.05427v2…