PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Face Data Augmentation

Xiang Wang, Kai Wang, Shiguo Lian

arXiv:1904.11685v1cs.CVcs.AIcs.LG

TL;DR

Face-related deep-learning tasks depend on training-set quality and size, but collecting and labeling adequate, balanced real-face samples is costly. This survey reviews face-augmentation transformations and methods, with attention to deep-learning approaches and GANs, and discusses their applications, evaluation, and limitations. It describes broad application and potential across face detection and recognition while identifying generation-control and diversity challenges.

  • Problem

    Deep-learning face tasks depend on training-data quality and size, yet collecting and labeling large, varied real-face datasets is laborious and expensive.

    Source: §I · p. 1

  • Method

    The paper systematically reviews face-augmentation transformation types and methods, including traditional and current approaches, and discusses applications, evaluation metrics, challenges, and opportunities.

    Source: §I · p. 2, §VIII · p. 17

  • Results

    The survey describes face data augmentation as having wide application and substantial potential across face detection and recognition tasks.

    Source: §VIII · p. 17

  • Takeaways & Limitations

    The review aims to give beginners a general understanding of face data augmentation and researchers an account of its challenges and opportunities.

    Source: §I · p. 2, §VIII · p. 17

    Deep neural generation remains limited in controllability and diversity, and latent-vector interpretation cannot completely disentangle facial features.

    Source: §VII · p. 17

Abstract

from arXiv · show

The quality and size of training set have great impact on the results of deep learning-based face related tasks. However, collecting and labeling adequate samples with high quality and balanced distributions still remains a laborious and expensive work, and various data augmentation techniques have thus been widely used to enrich the training dataset. In this paper, we systematically review the existing works of face data augmentation from the perspectives of the transformation types and methods, with the state-of-the-art approaches involved. Among all these approaches, we put the emphasis on the deep learning-based works, especially the generative adversarial networks which have been recognized as more powerful and effective tools in recent years. We present their principles, discuss the results and show their applications as well as limitations. Different evaluation metrics for evaluating these approaches are also introduced. We point out the challenges and opportunities in the field of face data augmentation, and provide brief yet insightful discussions.

I. INTRODUCTION

The survey reviews traditional and current face data augmentation methods, their transformation types, applications, evaluation, and open challenges. It frames augmentation as a response to costly data collection and limited variation in face datasets.

I. INTRODUCTION

  • Collecting and labeling large quantities of real face samples is laborious, expensive, and error-prone, while existing datasets lack real-world variation; insufficient or imbalanced data can lead to overfitting and weaker learning.The paper notes that deep learning performance depends heavily on training-data amount and quality.Source: §I · p. 1
  • Face data augmentation enlarges training or testing data by transforming collected real or simulated virtual face samples.The survey distinguishes augmentation from data synthesis by its basis in existing data, while noting that synthesis techniques can also be applied to augmentation.Source: §I · pp. 1–2
  • Augmentation can provide inexpensive annotated synthetic data, controllable facial attributes, faces without self-occlusion, and more balanced datasets with greater intra-class variation.The paper cites self-occlusion removal and balanced datasets with more intra-class variation as specific advantages.Source: §I · p. 1
  • Generated faces can lack realistic appearance variation and differ in distribution from real data; high-quality detail, resolution, identity preservation, and large-pose variation remain challenges.The cited appearance gaps include lighting, makeup, skin color, occlusion, and complex backgrounds; domain adaptation and transfer learning are used to improve synthetic-data utility.Source: §I · pp. 1–2

II. RELATED WORK

  • Earlier work studied augmentation methods and virtual-sample generation, while this survey covers a wider range of methods and more recent research.Related work includes analyses of face-augmentation techniques [10] [12] and a virtual-sample-generation survey.Source: §II · p. 2

III. TRANSFORMATION TYPES

  • Generic transformations typically alter whole images, whereas face-specific methods target components or attributes such as age, makeup, and hairstyle; Table I overviews common transformation types.Source: §III · p. 2, Table I · p. 2

A. Geometric and Photometric Transformation

Face augmentation ranges from whole-image geometric and photometric changes to face-specific transfers such as hairstyle and makeup. The reviewed work describes both traditional operations and learned methods, including GAN-based approaches.

A. Geometric and Photometric Transformation

  • Geometric transformations reposition image pixels, while photometric transformations change RGB values; reported classification studies found gains from augmentation and stronger performance from geometric than photometric schemes.One benchmark also found flipping and cropping more beneficial than random scaling and occlusion for fine-grained aircraft classification.Source: A. Geometric and Photometric… · p. 3

B. Hairstyle Transfer

  • Hairstyle transfer changes hair color, shape, or bangs, using methods ranging from templates and image compositing to GAN-based transfer.StarGAN performs multi-domain translation in one model; uses a text query such as curly hair to generate new photos retaining the input person's identity.Source: B. Hairstyle Transfer · p. 3

C. Facial Makeup Transfer

  • Makeup transfer applies, removes, or exchanges cosmetics while aiming to preserve identity, using traditional image editing or deep learning methods.Traditional component-based methods can produce artifacts at adjacent regions, while end-to-end networks are described as offering improved output quality and diversity; BeautyGAN combines global and local losses.Source: C. Facial Makeup Transfer · p. 4

D. Accessory Removal and Wearing

The reviewed face-specific transformations address glasses, pose, and expression variation. They include template and 3D-model methods as well as learning-based approaches that manipulate attributes or synthesize controlled outputs.

D. Accessory Removal and Wearing

  • Glasses wearing and removal are studied because glasses cover facial regions and can affect recognition accuracy; proposed approaches include templates, augmented reality, InfoGAN, and residual-image manipulation.The residual-image method uses inverse image transformation networks to produce residuals for adding or removing glasses.Source: D. Accessory Removal and Wearing · p. 4

E. Pose Transformation

  • Pose augmentation addresses datasets dominated by near-frontal faces, with methods ranging from 2D and 3D models to GANs that generate arbitrary poses or frontalize faces.UV-GAN completes self-occluded regions in a facial UV map, while LB-GAN separates frontalization from subsequent rotation and was reported to perform better in identity preservation in its comparison.Source: E. Pose Transformation · pp. 4–5

F. Expression Synthesis and Transfer

  • Expression synthesis targets emotion and expression variation for tasks including emotion classification and expression recognition, using 2D geometry, 3D geometry, or learning-based methods.The earlier 2D and 3D approaches do not require large amounts of training samples; learning-based GANs condition generation on references, expression codes, action-unit labels, or geometry.Source: F. Expression Synthesis and Transfer · pp. 5–6
  • Learning-based expression methods support different controls, including intensity, continuous action-unit coefficients, and facial geometry, with examples designed for identity preservation or distracting backgrounds.ExprGAN controls expression intensity; G2-GAN synthesizes expressions from a single image using facial geometry as a condition.Source: F. Expression Synthesis and Transfer · p. 6

G. Age Progression and Regression

The reviewed transformations span age changes, other facial attributes, and general image-processing operations. The paper also classifies augmentation methods into broader methodological categories.

G. Age Progression and Regression

  • Age progression and rejuvenation synthesize faces across age ranges while aiming to retain personal features, and the resulting age-span variation can improve a model's robustness to age variation.Traditional prototype-based transfer can lose individual features, while model-based biological-change approaches can be computationally complex; later GAN methods use latent-space transformations and identity-preservation mechanisms.Source: G. Age Progression and Regression · p. 6

H. Other Styles Transfer

  • Other reviewed transfers include illumination, gender, skin and eye color, eyebrows, facial hair, landmarks, context, and background; unified models can transform multiple attributes.StarGAN supports expression, gender, age, and skin-color transformations in one model with one-time training; Fig. 10 shows mustache, beard, and skin-color transfer examples.Source: H. Other Styles Transfer · pp. 6–7, Figure 10 · p. 7

IV. TRANSFORMATION METHODS

  • The paper groups transformation methods as basic image processing, model-based transformation, realism enhancement, generative-based transformation, augmented reality, and auto augmentation.The overview is presented in Table II.Source: §IV · p. 7, Table II · p. 7

A. Basic Image Processing

  • Basic image processing represents image transformation as applying T to input image f to produce output g; affine transforms move pixels, and convolutional or color operations support further changes.The described operations include translation, rotation, scaling, reflection, and shearing, as well as blurring, sharpening, embossing, and color-space transformations.Source: A. Basic Image Processing · pp. 7–8
  • Some face augmentation methods combine image manipulations, including landmark perturbation, face deformation with wrinkle mapping, face-part compositing, and stitching triangles from different images.These examples apply sequences of operations to produce more complex transformations.Source: A. Basic Image Processing · p. 8

B. Model-based Transformation

Model-based augmentation fits a face model to an input and varies its parameters to synthesize appearances. The section covers 2D AAMs and 3DMMs, their fitting and rendering, and challenges in 3D face synthesis.

B. Model-based Transformation

  • Model-based augmentation fits a face model to an input image and varies its parameters to synthesize faces with different appearances.AAMs and 3DMMs are the representative 2D and 3D models, respectively.Source: B. Model-based Transformation · p. 8
  • A 2D AAM represents shape as landmark coordinates and models shape variation with a mean shape plus PCA-derived modes.Fig. 11 illustrates the mean shape and shape variations.Source: B. Model-based Transformation · p. 8, Figure 11 · p. 8
  • AAM texture is represented by pixel intensities and modeled with PCA after piecewise-affine warping and photometric normalization.Fig. 11 illustrates mean texture and texture variations.Source: B. Model-based Transformation · p. 8
  • A 3DMM uses PCA to model scan geometry and texture, and LSFM illustrates a mean shape with its five leading shape components.The LSFM example was constructed from 9,663 distinct facial identities.Source: B. Model-based Transformation · p. 8, Figure 12 · p. 9
  • 3D models provide surface normals for lighting simulation, and model variants differ in whether they represent pose, illumination, expression, or occlusion.AAM instances use 2D normalization and similarity transforms, whereas 3DMMs use scaled orthographic or weak perspective models.Source: B. Model-based Transformation · pp. 8–9
  • 3D fitting is ill-posed, while changing head pose can expose missing-region artifacts and models omit eyes, teeth, and mouth cavity.Prior work addressed these issues with landmarks, analysis-by-synthesis, neural fitting, teeth proxies, mouth warping, and inpainting.Source: B. Model-based Transformation · p. 9

C. Realism Enhancement

Realism enhancement addresses the limited detail and realism of rendered virtual faces. The section also describes generative models as learning image distributions and distinguishes adversarial learning from likelihood-based approaches.

C. Realism Enhancement

  • Simplified rendering and limited illumination and surface modeling make photorealistic virtual faces difficult to generate, including detailed features such as wrinkles.The section presents realism enhancement as a way to improve simulated samples.Source: C. Realism Enhancement · p. 9
  • Realism enhancement methods include displacement maps for facial detail transfer, GAN-based refinement, dual-path generators, and unsupervised refiners.The GAN refinement approach uses unlabeled real faces and two discriminators; dual-path GANs separate global structure from local details.Source: C. Realism Enhancement · p. 9, Figure 14 · p. 9, Figure 15 · p. 10

D. Generative-based Transformation

  • A generative model maps a unit Gaussian distribution through a neural network to a generated image distribution intended to match the real-data distribution.Changing the network parameters changes the generated distribution.Source: D. Generative-based Transformation · p. 10, Figure 16 · p. 10
  • Autoregressive models and VAEs minimize KL divergence, whereas GANs use adversarial learning without specifying an explicit density for each data point.GANs are described as implicit generative models.Source: D. Generative-based Transformation · p. 10

1) Autoregressive Generative Models:

The section surveys autoregressive models, VAEs, and GANs as generative approaches. It outlines their modeling principles and examples of conditional face generation and augmentation frameworks.

1) Autoregressive Generative Models:

  • PixelRNN and PixelCNN model an image likelihood as a product of conditional pixel probabilities, predicting each pixel from previously generated pixels.PixelRNN uses two-dimensional LSTMs, while PixelCNN uses convolutional networks; later variants include Gated PixelCNN, Conditional PixelCNN, and PixelCNN++ [102].Source: 1) Autoregressive Generative Models · p. 10

2) Variational Autoencoders:

  • VAEs encode training data into latent representations and decode them back to data, using variational inference because the true posterior and relevant integrals are intractable.The objective uses a lower bound on marginal likelihood, with the KL divergence between approximate and true posterior as the other term.Source: 2) Variational Autoencoders · pp. 10–11
  • Conditional VAE variants control generation or capture conditional distributions, while other VAE-based work uses disentangled foreground and background representations or self-evaluation.Examples include CVAE, CMMA, a layered model, and IntroVAE.Source: 2) Variational Autoencoders · p. 11

3) Generative Adversarial Network:

  • GANs train a generator and discriminator adversarially: the generator maps prior noise to data, while the discriminator estimates whether samples are real or generated.The generator and discriminator are adjusted through a two-player min-max objective.Source: 3) Generative Adversarial Network · p. 11
  • GAN extensions address stability, conditioning, image resolution, and semantic structure, and many later approaches target face augmentation.Examples include DCGAN [109], Wasserstein-based training [111], conditional GANs [112], StackGAN, and InfoGAN.Source: 3) Generative Adversarial Network · pp. 11–12
  • DAGAN uses an encoder, random-vector concatenation, decoder, and adversarial discriminator for augmentation, while CycleGAN-based augmentation transfers between unpaired domains.The CycleGAN framework uses cycle consistency and adversarial losses; applied it to augment an underrepresented class.Source: 3) Generative Adversarial Network · pp. 11–12, Figure 18 · p. 12, Figure 19 · p. 12

4) Flow-based Generative Models:

Flow-based generative models map data to a latent space through a sequence of invertible transformations. The survey notes their inference and likelihood properties, along with progress in image synthesis that still trails GAN results.

4) Flow-based Generative Models:

  • Flow-based models transform high-dimensional data into a tractable latent distribution through a sequence of invertible nonlinear transformations.The composition of these transformations is called a flow.Source: 4) Flow-based Generative Models · p. 12
  • Compared with VAEs and GANs, flows provide exact latent-variable inference and log-likelihood evaluation, and compared with autoregressive models they allow efficient inference and synthesis.Glow demonstrated high-resolution face synthesis, although its results still had a gap with GANs.Source: 4) Flow-based Generative Models · p. 12
  • Flow-GAN combines a normalizing-flow generator with a GAN framework so that exact likelihoods can be evaluated tractably.The approach was intended to bridge high-quality generated samples and the ill-defined likelihood of GANs.Source: 4) Flow-based Generative Models · p. 12

5) Generative Models Comparison:

The section compares generative models, describes augmented reality and learned augmentation policies, and reviews evaluation measures. It emphasizes that model strengths differ and that fair comparison depends on each method’s task and application.

5) Generative Models Comparison:

  • Autoregressive models offer tractable likelihoods and stable training but slow sequential sampling; VAEs sample efficiently but often produce blurrier images than GANs.GANs can produce sharp images without Markov-chain sampling, but lack explicit density and have unstable training.Source: 5) Generative Models Comparison · pp. 12–13
  • GAN samples appear more realistic than the compared VAE and flow samples, though PGGAN images retain cloth and earring asymmetries.IntroVAE samples show unnatural eyebrows and wrinkles, while Glow samples have a painting-like hairline and lighting.Source: 5) Generative Models Comparison · p. 13, Figure 20 · p. 13

E. Augmented Reality

  • Augmented reality can expand training data by adding missing scene elements, precise virtual elements, and diversity.Face applications include fitting virtual eyeglasses to real faces and synthesizing eyeglass images for recognition.Source: E. Augmented Reality · p. 13, Figure 21 · p. 13

F. Auto Augmentation

  • Auto-augmentation methods learn or search for task-suitable transformations, but current techniques usually target simple operations such as rotation, translation, and cropping.Approaches include learned augmentation networks and reinforcement-learning search over augmentation policies.Source: F. Auto Augmentation · pp. 13–14

V. EVALUATION METRICS

  • Qualitative image inspection and quantitative statistical measures are commonly used together, with metrics chosen to suit the transformation’s purpose.IS and FID are commonly used for image generation, while identity-focused work calls for recognition or verification tests.Source: §V · p. 14
  • A uniform fair comparison is difficult because methods target different problems and applications, and task outcomes are tied to particular algorithms and datasets.FID is described as more robust to noise and more sensitive to intra-class mode collapse than IS.Source: §V · p. 14

VI. CHALLENGES AND OPPORTUNITIES

The section identifies identity preservation and disentangled representation as challenges in conditional face generation. It reviews loss-based and network-based strategies for retaining identity or controlling facial attributes.

A. Identity Preserving

  • Identity remains difficult to preserve during conditional face generation, despite being a key label for face recognition and verification.The review notes that retaining identity-relative features while changing other attributes remains difficult for end-to-end networks.Source: A. Identity Preserving · pp. 14–15
  • Existing identity-preservation strategies include identity losses, cycle consistency, and networks that explicitly classify identity or pose.Methods include perceptual identity losses and feature-distance constraints; some combine perceptual and cycle-consistency losses.Source: A. Identity Preserving · p. 15

B. Disentangled Representation

  • Disentangled representations aim to adjust selected facial factors while keeping other attributes fixed, enabling more controlled conditional generation.Reviewed approaches separate identity and attributes into distinct representations or use separate latent factors for facial properties.Source: B. Disentangled Representation · p. 15

C. Unsupervised Data Augmentation

The section surveys unsupervised augmentation, GAN improvements, and broader challenges in face-data transformation. It concludes that augmentation methods can increase training-data variation, while identity preservation, controllability, and synthetic-data diversity remain concerns.

C. Unsupervised Data Augmentation

  • Unsupervised methods reduce the need for labeled or paired training data, but can be less accurate and stable, and the gap from paired-data results may be difficult to close.The review describes unsupervised translation as ill-posed and notes that prior or expert knowledge may be needed.Source: C. Unsupervised Data Augmentation · pp. 15–16

D. Improvement of GANs

  • GAN improvements include a third-player identity classifier, style control through an intermediate latent space, and multi-scale discrimination for aged-face generation.The pyramidal discriminator is reported to make generated aged faces more natural and detailed.Source: D. Improvement of GANs · p. 16, Figure 22 · p. 16

VII. DISCUSSION

  • Face augmentation methods can increase training-data variation and improve learned-model robustness, but deep networks still have limits in controllability and synthetic-data diversity.The discussion notes that greater facial change makes identity preservation more difficult.Source: §VII · p. 17

VIII. CONCLUSION

  • The survey reviews transformation types, commonly used methods, evaluation metrics, and challenges and opportunities in face data augmentation.The authors frame the review as a resource for beginners and researchers considering future studies.Source: §VIII · p. 17

APPENDIX

The appendix table summarizes recent face-augmentation work by transformation type, method, and evaluation. Its transformation labels reflect only what the original papers explicitly report.

APPENDIX

  • The table summarizes recent works by transformation type, method, and evaluation of their augmentation algorithms.Source: APPENDIX · p. 24, Table III · p. 24
  • Listed transformation categories include hairstyle, makeup, accessory, pose, expression, age, and other transformations.Source: APPENDIX · p. 25
  • Gu et al.’s D-GAN [158] appears in the table alongside transformation marks and an illumination-related entry.Source: APPENDIX · p. 25
Loading 1904.11685v1…