PaperlayerView exact revision on arXiv

Source-linked AI summary

Triple consistency loss for pairing distributions in GAN-based face synthesis

Enrique Sanchez, Michel Valstar

arXiv:1811.03492v1cs.CV

TL;DR

The paper addresses a mismatch between input and generated-image distributions that makes self-consistent face translation unsuitable for progressive reuse. It introduces triple consistency loss, which aligns one-step and multi-step routes to a target, and applies it in GANnotation for landmark-guided synthesis with broad pose and expression changes.

  • Problem

    Self-consistency can yield realistic images whose distributions differ from the input domain, causing networks to recover inputs instead of applying new targets during progressive translation.

    Source: §1 · pp. 1–2, §3.3.4 · p. 5

  • Method

    Triple consistency loss enforces similar outputs for direct and intermediate routes to a target, and GANnotation conditions face synthesis on target landmark heatmaps.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    GANnotation makes target landmarks become ground-truth points in generated images and supports synthesis across a wide range of poses and expressions.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The approach supports reusable generated images for progressive translation and landmark-guided face synthesis with simultaneous pose and expression changes.

    Source: §1 · p. 2

    Identity preservation can fail when target landmarks conflict with identity or are substantially displaced from the input landmarks.

    Source: §5.3 · p. 8

Abstract

from arXiv · show

Generative Adversarial Networks have shown impressive results for the task of object translation, including face-to-face translation. A key component behind the success of recent approaches is the self-consistency loss, which encourages a network to recover the original input image when the output generated for a desired attribute is itself passed through the same network, but with the target attribute inverted. While the self-consistency loss yields photo-realistic results, it can be shown that the input and target domains, supposed to be close, differ substantially. This is empirically found by observing that a network recovers the input image even if attributes other than the inversion of the original goal are set as target. This stops one combining networks for different tasks, or using a network to do progressive forward passes. In this paper, we show empirical evidence of this effect, and propose a new loss to bridge the gap between the distributions of the input and target domains. This "triple consistency loss", aims to minimise the distance between the outputs generated by the network for different routes to the target, independent of any intermediate steps. To show this is effective, we incorporate the triple consistency loss into the training of a new landmark-guided face to face synthesis, where, contrary to previous works, the generated images can simultaneously undergo a large transformation in both expression and pose. To the best of our knowledge, we are the first to tackle the problem of mismatching distributions in self-domain synthesis, and to propose "in-the-wild" landmark-guided synthesis. Code will be available at https://github.com/ESanchezLozano/GANnotation

1. Introduction

The paper identifies a distribution mismatch caused by self-consistency losses, which makes generated faces unsuitable for progressive translation, and introduces triple consistency loss and GANnotation to address it.

Motivation

  • Self-consistency can produce realistic images that fail when reused for further attribute translation, preventing progressive image translation.The network may recover the input even when given a different target attribute, rather than applying the requested transformation.Source: §1 · pp. 1–2

Triple consistency loss

  • Retrained StarGAN correctly places the Blonde Hair attribute after the input has first been translated to Black Hair.The bottom rows of Figure 2 contrast with the original StarGAN behavior, which largely recovers the prior image.Source: Figure 2 · p. 2, §1 · p. 2

Motivation

  • The paper attributes this mismatch to a hidden footprint in generated images that lets the network recover inputs regardless of the returned target attribute.This behavior preserves identity under self-consistency but separates generated images from the intended input distribution.Source: §1 · p. 2

Triple consistency loss

  • Triple consistency loss requires one-step and two-step routes to the same target to produce the same generated image.It is designed to bridge input and target distributions and support progressive translation.Source: Figure 1 · p. 1, §1 · p. 2

GANnotation

  • GANnotation performs landmark-guided face synthesis with simultaneous pose and expression changes across unconstrained target landmarks.The target landmarks become ground-truth points in generated images, supporting face-alignment applications and person-specific datasets.Source: §1 · pp. 2–3

2. Related Work

Prior face-synthesis methods preserve identity or appearance but generally cannot customize pose and expression together. Self-consistency loss contributes to a mismatch between input and target distributions, limiting progressive translation.

Identity-preserving synthesis

  • Face frontalisation, profile synthesis, and multi-view methods preserve identity, but do not support customized expressions and poses simultaneously.CR-GAN provides multiple views, but those views cannot include additional synthesized expressions.Source: §2 · p. 3

Landmark-guided synthesis

  • GP-GAN and GAGAN generate random landmark-conditioned faces, so they cannot provide person-specific data augmentation.GP-GAN is limited to expression variation, while GAGAN samples from a latent space and does not preserve the input person.Source: §2 · p. 3
  • CMN-Net and GC-GAN synthesize landmark-guided expressions but are limited to frontal or fixed-pose settings.CMN-Net expects a neutral frontal input, while GC-GAN does not account for pose changes.Source: §2 · p. 3

Pose and expression synthesis

  • CAPG-GAN handles pose-specific rotation with heatmaps, but its pose representation removes the ability to synthesize expressions; GANimation has the opposite limitation.GANimation supports expression changes but is restricted to near-frontal faces and does not change pose.Source: §2 · p. 3

Self-consistency limitations

  • Self-consistency loss preserves identity but mismatches input and target distributions, preventing generated images from reliably supporting further translation steps.Existing methods typically perform only one forward pass, such as neutral-to-expression or non-frontal-to-frontal translation.Source: §2 · p. 3

3. Proposed approach

The proposed approach targets person-specific landmark-driven face synthesis while reducing the distribution gap between generated and input images. It is designed to support simultaneous pose and expression changes.

3. Proposed approach

  • The approach aims to generate person-specific images whose landmarks match prescribed targets while allowing simultaneous changes in pose and expression.The generated images are intended to remain photo-realistic and distribution-wise close to the input images.Source: §3 · pp. 3–4
  • Figure 3 feeds an input image and target landmark points, encoded as heatmaps, into a generator that produces the synthesized face.The target points are overlaid on the generated image to illustrate the network’s task.Source: Figure 3 · p. 4, §3 · p. 4

3.1. Notation

The notation defines landmark-conditioned generation as a mapping from an input face and target-shape heatmaps to a warped output image. It also distinguishes real and generated image distributions for adversarial training.

3.1. Notation

  • The generator G maps an input image I and heatmaps H(s_t) encoding a target shape to an estimated warped image Î.The input image has w × h pixels, and the target shape contains n indexed points.Source: §3.1 · p. 4
  • H(s) represents heatmaps whose dependence is on shape s, with H(s) ∈ R^{n×w×h}.Source: §3.1 · p. 4
  • The formulation distinguishes real images drawn from P_I from generated images drawn from P_Î, with the discriminator classifying inputs as real or fake.The intended objective is for P_Î to match P_I as closely as possible.Source: §3.1 · p. 4

3.2. Architecture

The architecture uses a convolutional generator conditioned on the RGB input and landmark heatmaps, with a mask-based output blend. A PatchGAN discriminator scores the resulting images.

3.2. Architecture

  • The generator uses two downsampling convolutions, residual blocks, and two upsampling blocks, accepting 3 + n channels for RGB data and landmark heatmaps.Its design is adapted from neural transfer and image-to-image translation architectures.Source: §3.2 · p. 4
  • The generator splits its final layer into a color image C and mask M, producing Î through an element-wise blend of the generated color and input image.The output is defined by Î = (1 − M) ◦ C + M ◦ I.Source: §3.2 · p. 4
  • The PatchGAN discriminator uses convolutional downsampling blocks and LeakyReLU layers, producing a 4×4×512 feature volume at 128 × 128 resolution before final scoring.The feature volume is forwarded to a fully convolutional network for the final score.Source: §3.2 · p. 4

3.3. Training

The training objective combines adversarial, reconstruction, consistency, identity, perceptual, and regularization terms, with triple consistency addressing mismatched input and target distributions. The method also uses identity-preserving and perceptual feature constraints alongside the full weighted generator loss.

3.3.1 Adversarial loss

  • The training objective uses hinge adversarial loss, pixel reconstruction, and total-variation regularization to learn target representations and smooth generated images.The hinge formulation is described as requiring fewer discriminator updates per generator update and enabling faster learning.Source: §3.3.1 · p. 4, §3.3.2 · pp. 4–5

3.3.3 Consistency Loss

  • Self-consistency trains the generator to invert a target transformation by passing the generated output back with the original landmarks and recovering the input.This loss is also called identity loss in [26].Source: §3.3.3 · p. 5, §3.3.4 · p. 5

3.3.4 Triple Consistency Loss

  • Triple consistency addresses the observation that self-consistency can recover the input under unintended targets, indicating mismatched input and target domains.The proposed loss aims to make outputs from sequential and direct routes to a target similar, supporting progressive image generation.Source: §3.3.4 · p. 5

3.3.5 Identity preserving loss

  • The identity-preserving loss compares Light CNN features from generated and real images at the fully connected and last pooling layers.The loss uses the l1 norm between those feature representations.Source: §3.3.5 · p. 5

3.3.6 Perceptual loss

  • The perceptual loss matches generated and real images through VGG-19 feature reconstruction and style reconstruction based on Gram matrices.Feature reconstruction uses selected VGG layers, while style reconstruction compares Gram matrices from relu3_3 features.Source: §3.3.6 · p. 5

3.3.7 Full loss

  • The generator loss combines adversarial, pixel, self-consistency, triple-consistency, identity, perceptual, and total-variation terms.Its weights are λadv = 1, λpix = 10, λself = 100, λtriple = 100, λid = 1, λpp = 10, and λtv = 10^-4.Source: §3.3.7 · p. 5

4. Training Datasets

Training uses paired triplets from annotated videos and augments them with unpaired images transformed through landmark perturbations. The two sources provide non-rigid deformation learning alongside identity preservation and robustness to rigid perturbations.

4. Training Datasets

  • The training set uses triplets from 300VW and BP4D videos to compare one- and two-pass outputs with ground-truth images.300VW contains 50 people with 3000 triplets per video, while BP4D contains 40 subjects performing 8 tasks with 500 triplets per video.Source: §4 · p. 6
  • Approximately 8000 unpaired images from five facial datasets are augmented with affine landmark transformations and mirroring to reduce overfitting and preserve identity.This augmentation teaches non-rigid deformation from 300VW and robustness to rigid perturbations from unpaired data.Source: §4 · p. 6

5. Experiments

Experiments show that triple consistency preserves plausible results across progressive passes and supports landmark-guided multi-view synthesis, while identity preservation remains constrained by training-loss and landmark conditions.

5.1. On the use of a triple consistency loss

  • The triple-consistency StarGAN retains high-level image quality and target attributes during progressive translation, unlike the original model, whose images degrade with burning-like artifacts.The comparison uses Celeb-A attributes including hair, gender, and age, with outputs from sequential attribute passes.Source: §5.1 · p. 6
  • After a second pass, the model without triple consistency recovers the input with subtle contrast changes, whereas the triple-consistency model correctly maps to the target.Both models produce similar plausible images after the first pass; the divergence appears during subsequent forwarding.Source: §5.1 · p. 6, Figure 5 · p. 7

5.2. GANnotation

  • GANnotation generates landmark-guided frontalised and pose-specific faces through progressive or one-to-one mappings, and its evaluation compares these outputs with CR-GAN on 300VW.The experiments manipulate pose, in-plane rotation, and expression-related shape parameters before synthesising target views.Source: §5.2 · p. 7, Figure 6 · p. 8
  • The experiments report photo-realistic synthesis with consistency across multiple passes, including under the landmark-guided GANnotation setting.Figure 6 covers progressive generation, one-to-one mapping, and comparison with CR-GAN.Source: §5.3 · p. 8, Figure 6 · p. 8

5.3. Remarks

  • Removing self-consistency can eliminate degradation when using triple consistency alone, but it makes identity preservation prone to failure; both losses therefore serve different roles.Triple consistency pulls the input image out of the target domain, while self-consistency better preserves identity.Source: §5.3 · p. 8
  • Landmark-guided identity preservation can fail when target points force implausible facial geometry or are substantially displaced from the input landmarks.The network prioritises target locations and lacks an attention mechanism to determine which information should transfer.Source: §5.3 · p. 8

6. Conclusion

The paper identifies a mismatch between realistic generated faces and their suitability for reuse, then introduces triple consistency and GANnotation to address progressive translation and landmark-guided synthesis.

6. Conclusion

  • The paper concludes that triple consistency makes outputs more consistent across different numbers of translation steps and supports high-quality landmark-guided synthesis.It reports that target landmarks become ground-truth points, including for low-resolution inputs.Source: §6 · p. 8
Loading 1811.03492v1…