PaperlayerView exact revision on arXiv

Source-linked AI summary

Real-Time Facial Segmentation and Performance Capture from RGB Input

Shunsuke Saito, Tianye Li, Hao Li

arXiv:1604.02647v1cs.CV

TL;DR

Occlusions make RGB facial performance capture prone to visual artifacts and tracking loss, and modeling their appearance is challenging. The paper segments facial regions in real time, uses masked data for regression-based tracking, and demonstrates uninterrupted capture under severe occlusions and side views.

  • Problem

    RGB facial tracking remains susceptible to occlusions, while the variation in occlusion appearance makes modeling them challenging.

    Source: §1 · pp. 1–2

  • Method

    A two-stream CNN segments RGB face regions, and the resulting masked images are used to train a regression-based facial tracker.

    Source: §1 · p. 2, §3 · p. 5

  • Results

    The approach demonstrates accurate, uninterrupted facial tracking under severe occlusions and side views while retaining real-time performance.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The segmentation masks can also composite partial 3D facial models under hair and other occluding objects.

    Source: §1 · p. 2

    Limited training data can still yield flickering segmentation boundaries.

    Source: §7 · p. 14

Abstract

from arXiv · show

We introduce the concept of unconstrained real-time 3D facial performance capture through explicit semantic segmentation in the RGB input. To ensure robustness, cutting edge supervised learning approaches rely on large training datasets of face images captured in the wild. While impressive tracking quality has been demonstrated for faces that are largely visible, any occlusion due to hair, accessories, or hand-to-face gestures would result in significant visual artifacts and loss of tracking accuracy. The modeling of occlusions has been mostly avoided due to its immense space of appearance variability. To address this curse of high dimensionality, we perform tracking in unconstrained images assuming non-face regions can be fully masked out. Along with recent breakthroughs in deep learning, we demonstrate that pixel-level facial segmentation is possible in real-time by repurposing convolutional neural networks designed originally for general semantic segmentation. We develop an efficient architecture based on a two-stream deconvolution network with complementary characteristics, and introduce carefully designed training samples and data augmentation strategies for improved segmentation accuracy and robustness. We adopt a state-of-the-art regression-based facial tracking framework with segmented face images as training, and demonstrate accurate and uninterrupted facial performance capture in the presence of extreme occlusion and even side views. Furthermore, the resulting segmentation can be directly used to composite partial 3D face models on the input images and enable seamless facial manipulation tasks, such as virtual make-up or face replacement.

1 Introduction

RGB facial tracking remains vulnerable to occlusions, while reliably masking non-face regions could make 3D tracking a more constrained problem. The paper introduces real-time facial segmentation and uses masked data for robust facial performance capture.

1 Introduction

  • Occlusions and uncontrolled illumination make 3D face fitting highly nonlinear, but reliable dense segmentation could reduce tracking to a more constrained problem.The authors motivate this approach as a way to address the appearance variability that makes occlusion modeling difficult.Source: §1 · pp. 1–2
  • The approach segments facial regions in RGB images and processes the masked data to support robust tracking under severe occlusions.Its two-stream deconvolution network shares lower convolution layers; a final convolution combines the outputs, and graph cut refines the mask.Source: §1 · p. 2
  • The authors report superior tracking accuracy and robustness over the current state of the art, with uninterrupted tracking under challenging occlusions and side views.The approach retains real-time performance and requires no calibration process.Source: §1 · pp. 2–3
  • The contributions include a real-time pure-RGB segmentation framework, a shared-convolution two-stream architecture, and training data augmentation for difficult occlusions such as hands.The authors also report improved segmentation-network efficiency and accuracy and tracking performance using masked training data.Source: §1 · p. 3

2 Related Work

Prior work spans controlled high-end capture, unconstrained 2D tracking, depth-based performance capture, and RGB-based 3D tracking. The cited approaches leave challenges in occlusion robustness, deployment, or computational cost.

2 Related Work

  • High-end production capture can achieve exceptional tracking accuracy, but typically depends on sophisticated sensors, controlled settings, substantial computation, and full face visibility.The section contrasts this with more unconstrained approaches.Source: §2 · p. 3
  • Unconstrained 2D landmark methods handle varied poses and lighting, but remain sensitive to occlusions and detect only a limited number of facial features.The section traces a progression from parametric models to data-driven methods and explicit shape regression.Source: §2 · p. 3
  • Depth-sensor systems enabled real-time 3D performance capture, and explicit facial segmentation for severe occlusions was demonstrated using depth and RGB cues.The cited segmentation approach relies on depth sensor input.Source: §2 · pp. 3–4
  • RGB-based 3D tracking has advanced through shape regression and larger training data, but the cited methods remain sensitive to large occlusions and cannot segment facial regions.Some approaches also use more expensive non-real-time computation.Source: §2 · p. 4
  • Existing facial segmentation methods either parse components rather than the whole face, require sufficient visibility, or incur high computation and remain vulnerable to similar-colored occluders.Other cited approaches improve occlusion handling for landmarks, including methods trained with occlusion data.Source: §2 · p. 4

3 Overview

The pipeline takes an RGB image and produces both a binary facial mask and a tracked 3D face model. It uses a CNN to segment a cropped face region and passes the mask to tracking and compositing.

3 Overview

  • The pipeline takes an RGB image and outputs a binary segmentation mask alongside a tracked 3D face model.Figure 2 depicts the facial segmentation and performance capture pipeline.Source: §3 · p. 4, Figure 2 · p. 5
  • The tracked model's shape vector describes rigid head motion and facial expression, while identity shape and focal length are solved during capture.The shape vector can also retarget a digital character with compatible animation controls.Source: §3 · p. 5
  • A CNN predicts a 128 × 128 probability map for a cropped face region, which graph cut converts to a binary mask before upsampling to input resolution.The crop is positioned using the previous frame's projected-model bounding box; training uses synthetic occlusions, hand compositings, and no-face negative samples.Source: §3 · p. 5
  • The resulting mask feeds the facial tracker, supplies masked faces for DDE regression training, and supports compositing partial 3D facial models during occlusions.The pipeline is based on a DDE method that does not require individual-user calibration.Source: §3 · p. 5

4 Facial Segmentation

The segmentation pipeline combines complementary deconvolution streams to produce a face/non-face probability map, then refines it into a binary mask. Training uses varied face data and synthetic occlusions, while architecture and training choices target real-time performance.

Architecture

  • Shared convolution features feed FCN and DeconvNet streams whose outputs are fused into a probability map, combining coarse clean shapes with detailed but noisier segmentations.A final 1 × 1 convolution learns the blend instead of simply averaging the streams; graph cut converts the map to a binary mask.Source: §4 · pp. 6–7, Figure 3 · p. 6

Architecture and training

  • Reducing the first fully connected layer's kernel from 7 × 7 to 4 × 4 addresses the runtime bottleneck to support real-time GPU performance.The network was initialized with pretrained weights and trained using losses on DeconvNet, FCN, and fused score maps; training took 9 hours over 50,000 SGD iterations.Source: §4 · pp. 7–8

Training

  • The training set combines 2,927 LFW images with 5,094 FaceWarehouse images, whose segmentations were refined and manually checked for occlusions.FaceWarehouse masks were initialized from fitted templates and refined with graph cuts before manual inspection and correction.Source: §4 · p. 7
  • Random translation, rotation, and scale perturbations expanded the data, and 82,770 rectangle-occluded samples were added to improve robustness to partial occlusions.The source photographs mostly avoided hair, hands, and other object occlusions.Source: §4 · p. 7, Figure 4 · p. 7
  • Compositing 1,092 segmented hand images onto 8,021 face images produced 41,380 additional samples, for 132,426 generated training images overall.The hand images varied in skin tone, lighting, and pose.Source: §4 · p. 8
  • Fine-tuning added 4,699 hand images without faces so negative samples equaled positive ones, targeting false face detections on hands and backgrounds.The fine-tuning converged after 10,000 SGD iterations and took an additional 1.5 hours.Source: §4 · p. 8

Segmentation refinement

  • Graph cut uses facial probabilities for unary costs and neighboring-pixel intensities for pairwise costs, removing minor boundary artifacts before upsampling the binary mask.The stated settings are λ = 10 and σ = 5.Source: §4 · pp. 8–9

5 Facial Tracking

The facial tracker regresses a 3D face from RGB frames using a DDE framework trained and run on segmented face regions. Masking and simulated occlusions reduce the appearance variation the regression must handle.

5 Facial Tracking

  • Accurately segmented images significantly improve tracking robustness and accuracy, including for skin-toned hand occlusions.The regression then needs to account only for facial appearance and lighting variations.Source: §5 · p. 9
  • Following the DDE model, tracking regresses facial-shape displacements from the current RGB image and the previous frame’s initial shape.The shape combines a linear 3D blendshape model, rigid head motion, and 2D residual displacements for 73 landmarks.Source: §5 · p. 9
  • The cascade regressor updates pose, expression, and landmark displacements, while identity coefficients and focal length are estimated only at key frames on a concurrent thread.Its two-level structure uses sequential regressors and fern-based feature comparisons.Source: §5 · pp. 9–10
  • Ground-truth shape parameters are fitted from landmarks, with blendshape and identity coefficients solved by constrained nonlinear least squares and rigid motion estimated through iterative PnP.The coefficient optimization uses three iterations and an L-BFGS-B solver.Source: §5 · pp. 10–11
  • Training uses 14,460 labeled examples from FaceWarehouse [10], LFW [16], and GTAV [63], expanded to 506,100 samples through parameter perturbations.Blendshape coefficients are perturbed 15 times and other parameters 5 times.Source: §5 · pp. 10–11
  • Non-face pixels are masked during training and inference, and random crops of segmented faces augment training to improve robustness under occlusion.Each of the 506,100 training sets receives an additional crop covering up to 80% of the face bounding box in width and height.Source: §5 · p. 11

6 Results

The results show that the proposed segmentation approach compares favorably with several facial-segmentation methods, while supporting real-time tracking under challenging occlusions. The authors also report real-time pipeline performance and successful tracking when only a small portion of the face is visible.

Segmentation Evaluation and Comparison

  • Reliable 3D facial fitting remains possible when only a small portion of the face is visible, and segmentation handles occlusions including hair, hands, headwear, and props.The reported examples include objects with skin tones similar to the face, and boundaries are correctly estimated in most cases.Source: §6 · p. 11, Figure 6 · p. 12
  • IOU improves by 12.7% over FCN-8s and 1.4% over DeconvNet; performance is comparable to EDeconvNet, with nearly double the performance enabling 30 fps.The method also contains much less noise than FCN-8s and DeconvNet in Figure 7.Source: §6 · p. 12, Figure 7 · p. 12, Table 1 · p. 13
  • IOU is 0.833 vs 0.724 against RPP, while global is 0.882 vs 0.839 and ave(face) is 0.929 vs 0.886 against structured forest; results are comparable to SAPM.Against SAPM, IOU is 0.833 vs 0.835 and ave(face) is 0.929 vs 0.871; SAPM requires up to 30s per frame.Source: §6 · p. 12, Table 1 · p. 13

Tracking Evaluation and Comparison

  • Masked training improves tracking robustness under occlusion, and additional synthetic occlusions further improve performance; the method handles larger occlusions and head rotations than robust 2D landmark estimation.The tracking comparison uses three 180-frame sequences with increasing synthetic occlusion size.Source: §6 · p. 13, Figure 8 · p. 13, Figure 9 · p. 13

Performance

  • The full facial tracking pipeline runs at 30 fps, with tracking and segmentation stages running in parallel across two computers.Tracking takes 18 ms for 3D face regression and 5 ms for identity and focal-length optimization; segmentation takes 23 ms for probability-map computation, 4 ms for refinement, and 1 ms for transmission.Source: §6 · pp. 13–14

7 Conclusion

The paper concludes that real-time pixel-level facial segmentation from unconstrained RGB images can support accurate tracking under occlusion. Its approach combines a shared-convolution, two-stream deconvolution network with data augmentation, while limited training data still leaves segmentation boundaries prone to flickering.

7 Conclusion

  • Real-time pixel-level facial segmentation from unconstrained RGB images supports significantly improved facial tracking accuracy and robustness to occlusion.Training the regressor on facial regions and augmenting with synthetic occlusions supports continuous tracking through challenging occlusions such as hair and hands.Source: §7 · p. 14
  • A shared-convolution, two-stream deconvolution network extracts overall shape and fine-scale detail, while carefully designed augmentation addresses limited segmentation ground truth and skin-colored occlusions.Composited hand datasets are used in both positive and negative training samples.Source: §7 · p. 14
  • Limited training data still yields flickering segmentation boundaries; temporal information and domain-specific priors are proposed for future work on this issue.The authors also propose extending segmentation to other body parts for more complex compositing.Source: §7 · p. 14
Loading 1604.02647v1…