PaperlayerView exact revision on arXiv

Source-linked AI summary

Style Aggregated Network for Facial Landmark Detection

Xuanyi Dong, Yan Yan, Wanli Ouyang, Yi Yang

arXiv:1803.04108v4cs.CV

TL;DR

Facial landmark detectors have overlooked intrinsic image-style variation, which can affect predictions as faces are collected from diverse sources. SAN generates style-aggregated images with a GAN and jointly uses them with original images as complementary inputs. The method is reported as robust to style variation and achieves state-of-the-art performance on 300-W and AFLW, while its style-transfer datasets support evaluation across styles.

  • Problem

    Intrinsic image-style variation from diverse collection sources is overlooked in facial landmark detection, although it can change predictions for faces with otherwise identical content.

  • Method

    SAN uses a GAN to generate style-aggregated face images and combines each with its original image to exploit complementary facial appearance and environmental information.

  • Results

    SAN is reported to be robust to large image-style variance and achieves state-of-the-art performance on the 300-W and AFLW benchmark datasets.

  • Takeaways & Limitations

    Style aggregation is presented as a way to address image-style variance in facial landmark detection, with potential extension to other vision tasks such as object detection and person re-identification.

  • Takeaways & Limitations

    The proposed style-analysis datasets use synthetic styles generated by selected Photoshop filters, whose realism is justified by their use in real-world photo editing.

Abstract

from arXiv · show

Recent advances in facial landmark detection achieve success by learning discriminative features from rich deformation of face shapes and poses. Besides the variance of faces themselves, the intrinsic variance of image styles, e.g., grayscale vs. color images, light vs. dark, intense vs. dull, and so on, has constantly been overlooked. This issue becomes inevitable as increasing web images are collected from various sources for training neural networks. In this work, we propose a style-aggregated approach to deal with the large intrinsic variance of image styles for facial landmark detection. Our method transforms original face images to style-aggregated images by a generative adversarial module. The proposed scheme uses the style-aggregated image to maintain face images that are more robust to environmental changes. Then the original face images accompanying with style-aggregated ones play a duet to train a landmark detector which is complementary to each other. In this way, for each face, our method takes two images as input, i.e., one in its original style and the other in the aggregated style. In experiments, we observe that the large variance of image styles would degenerate the performance of facial landmark detectors. Moreover, we show the robustness of our method to the large variance of image styles by comparing to a variant of our approach, in which the generative adversarial module is removed, and no style-aggregated images are used. Our approach is demonstrated to perform well when compared with state-of-the-art algorithms on benchmark datasets AFLW and 300-W. Code is publicly available on GitHub: https://github.com/D-X-Y/SAN

1. Introduction

Facial landmark detection has largely overlooked intrinsic image-style variation, even though collection sources introduce substantial style differences that can alter predictions. SAN addresses this by combining original and GAN-generated style-aggregated faces, and experiments report robustness to style variation and strong benchmark performance.

  • Motivation: Image-style variation, including grayscale versus color and light versus dark, is overlooked despite causing different landmark predictions for otherwise identical face contents.Figure 1 illustrates prediction deviations across styled versions of the same face.
  • Approach: SAN generates style-aggregated face images with a GAN and combines them with original images as complementary inputs for landmark prediction.Original images preserve undistorted facial appearance, while aggregated images provide more stationary environments but may lose some shape fidelity.
  • Contributions: The study explicitly investigates style variation as a previously overlooked problem in facial landmark detection and empirically verifies its performance degradation.The paper identifies this issue as increasingly relevant for faces collected from varied web sources.
  • Contributions: SAN introduces style-analysis datasets by transferring 300-W and AFLW images into multiple styles, with approximately 12,000 and 80,000 images, respectively.The datasets are named 300W-Styles and AFLW-Styles.
  • Results: Experiments report that SAN is insensitive to large image-style variance and achieves state-of-the-art performance on AFLW and 300-W benchmarks.These findings are presented as empirical evidence for the method’s effectiveness.

2. Related Work

Prior facial landmark detectors commonly use regression, deep CNNs, cascades, heatmaps, and geometric constraints, but focus mainly on facial shape and pose. This paper instead targets intrinsic image-style variation using GAN-based style generation and complementary original and aggregated inputs.

  • Facial Landmark Detection: Facial landmark detection predicts predefined facial points such as eye corners, eyebrows, and the nose tip, supporting applications including recognition, pose estimation, reenactment, and 3D reconstruction.Accurate landmark localization is therefore useful across multiple computer-vision tasks.
  • Facial Landmark Detection: Existing methods commonly learn regressors with deep CNNs or handcrafted features, often using cascaded architectures to progressively update landmark estimates.Other approaches incorporate geometric constraints or staged reinitialization.
  • Style Aggregation: SAN combines original and style-aggregated images so their complementary appearance and environmental information supports more robust landmark predictions.The architecture fuses information from both streams in a cascaded prediction process.
  • Facial Landmark Detection: Heatmap-based methods learn robust landmark-response maps and infer coordinates from the highest responses, with some methods adding multitask learning or specialized network structures.These methods provide an alternative to direct coordinate regression.
  • Facial Landmark Detection: Most existing algorithms emphasize facial shape information such as extreme head poses and deformation, while few consider intrinsic style differences such as grayscale, color, light, or darkness.The paper identifies style variance as its central research focus.
  • Style Aggregation: The proposed style-generation component uses a trained GAN generator to transform faces into different styles and combat style variance.The related-work discussion connects this design to GAN-based face applications and CycleGAN.

3. Methodology

SAN learns a neutral style from unlabeled style variation, then jointly uses original and style-aggregated faces for landmark prediction. Its generation pipeline combines style-discriminative clustering with CycleGAN, while the detector fuses complementary streams through staged belief-map prediction.

  • Network design: SAN combines a face generation module that aggregates styles with a landmark prediction module using original and neutral-style faces.The original stream preserves undistorted facial appearance, while the aggregated stream provides more stationary environments but may lose shape fidelity.
  • Style-aggregated face generation: Unsupervised style learning transfers the original dataset into three styles, adds the original images as a fourth class, and fine-tunes ResNet-152 for style-discriminative features.The paper identifies the transferred styles as Light, Gray, and Sketch.
  • Style-aggregated face generation: The style-discriminative features cluster original images into k groups that potentially encode hidden styles, which are then used to train CycleGAN style-transformation models.The trained models are combined to average transferred faces into the final style-aggregated images while preserving input structure through cycle consistency.
  • Facial Landmark Prediction Module: The landmark detector extracts features from both input streams, predicts separate first-stage belief maps, and feeds their concatenation into later fully convolutional stages.The second stage receives Fo, Fs, Ho, and Hs; the output contains K landmark channels plus one additional channel.
  • Facial Landmark Prediction Module: Training minimizes belief-map losses for each face, with H* denoting the ideal belief map, and obtains landmark coordinates by upsampling H3 before applying argmax.Bicubic interpolation restores the belief map to the original image size before coordinate extraction.

4. Experiments

Experiments evaluate SAN on 300-W and AFLW, test style variation using synthetic datasets, and examine the contributions of its two input streams. SAN achieves strong benchmark performance and remains robust across style changes.

  • Experiment Settings: 300-W contains 3,148 training images, 554 common-set test images, and 135 challenging-set test images with 68 annotated landmarks.The common and challenging subsets together form the full testing set.
  • Experiment Settings: AFLW contains 21,997 real-world images with 25,993 faces and evaluates splits that include AFLW-Full and AFLW-Frontal.Faces include pose, expression, occlusion, and illumination variation; ear landmarks are excluded.
  • Evaluation Metrics: SAN is evaluated with NME, CED curves, and AUC @ 0.08, using interocular distance for 300-W and face size for AFLW normalization.These metrics compare facial landmark prediction errors under dataset-specific normalization settings.
  • Benchmark Results: 21.8% relative NME improvement on the 300-W common set is reported over the state-of-the-art method using the same official detector bounding box.The comparison uses official detector boxes; ground-truth boxes provide better initialization.
  • Benchmark Results: More than 11% improvement on AFLW-Full and more than 14% on AFLW-Front are reported over prior state-of-the-art results.The default style-generation setting uses k = 3 because more clusters and generation models produced similar results.
  • Style-Robustness Analysis: 7% relative improvement is obtained on 300W-Style when training and testing styles differ, with improvements across all 16 train-test style combinations.The gain is larger when the style variance between training and testing sets is larger.

5. Conclusion & Future Work

SAN addresses overlooked image-style variation in facial landmark detection by pairing each original face image with a GAN-generated style-aggregated image. The method is reported as robust to style variance, achieves state-of-the-art performance on 300-W and AFLW, and may extend to other vision tasks.

  • 5. Conclusion & Future Work: SAN takes two complementary images for each face: one in the original style and one in a GAN-generated aggregated style.The original image preserves appearance content, while the aggregated image provides a more stationary environment.
  • 5. Conclusion & Future Work: Empirical studies verify that image-style variation degenerates facial landmark detection performance, while SAN is robust to this variance.
  • 5. Conclusion & Future Work: SAN’s style-aggregation step is decoupled from the landmark detector and may improve other landmark detectors.
  • 5. Conclusion & Future Work: Because style variance also occurs in object detection and person re-identification, style aggregation may extend beyond facial landmark detection.
Loading 1803.04108v4…