PaperlayerView exact revision on arXiv

Source-linked AI summary

High-Quality Facial Photo-Sketch Synthesis Using Multi-Adversarial Networks

Lidan Wang, Vishwanath A. Sindagi, Vishal M. Patel

arXiv:1710.10182v2cs.CV

TL;DR

Photo-sketch synthesis is challenging because photos and sketches differ, and GAN methods have limitations generating realistic high-resolution images. PS2-MAN uses multi-resolution adversarial supervision within a bidirectional CycleGAN framework. The paper reports improved image-quality measures and photo-sketch matching rates relative to other methods.

  • Problem

    Photo-sketch synthesis is challenging because photos and sketches differ, and existing GAN-based methods have limitations generating realistic high-resolution images.

    Source: §I · p. 2, B. Objective · p. 3

  • Method

    PS2-MAN uses multiple discriminators to supervise hidden generator layers across resolutions, with bidirectional CycleGAN training and cycle-consistency and L1 losses.

    Source: §I · p. 2, B. Objective · p. 4

  • Results

    PS2-MAN preserves high-frequency details while minimizing artifacts and achieves the best SSIM and FSIM results and improved photo-sketch matching rates compared with other methods.

    Source: D. Comparison with state-of-the-art… · pp. 6–7

  • Takeaways & Limitations

    The experiments support PS2-MAN as a method for realistic photo-sketch synthesis with improved image-quality measures and matching rates.

    Source: D. Comparison with state-of-the-art… · p. 7, §V · p. 7

Abstract

from arXiv · show

Synthesizing face sketches from real photos and its inverse have many applications. However, photo/sketch synthesis remains a challenging problem due to the fact that photo and sketch have different characteristics. In this work, we consider this task as an image-to-image translation problem and explore the recently popular generative models (GANs) to generate high-quality realistic photos from sketches and sketches from photos. Recent GAN-based methods have shown promising results on image-to-image translation problems and photo-to-sketch synthesis in particular, however, they are known to have limited abilities in generating high-resolution realistic images. To this end, we propose a novel synthesis framework called Photo-Sketch Synthesis using Multi-Adversarial Networks, (PS2-MAN) that iteratively generates low resolution to high resolution images in an adversarial way. The hidden layers of the generator are supervised to first generate lower resolution images followed by implicit refinement in the network to generate higher resolution images. Furthermore, since photo-sketch synthesis is a coupled/paired translation problem, we leverage the pair information using CycleGAN framework. Both Image Quality Assessment (IQA) and Photo-Sketch Matching experiments are conducted to demonstrate the superior performance of our framework in comparison to existing state-of-the-art solutions. Code available at: https://github.com/lidan1/PhotoSketchMAN.

I. INTRODUCTION

Photo-sketch synthesis is practically useful, but differences between photos and sketches make direct matching impractical and high-resolution GAN outputs prone to artifacts. PS2-MAN combines multi-level adversarial supervision with CycleGAN-based bidirectional synthesis, and the paper reports experiments and ablations evaluating the approach.

I. INTRODUCTION

  • Photo-sketch synthesis has applications in suspect retrieval and digital entertainment, while differences between the modalities make direct photo-sketch matching impractical.A common response is to synthesize one modality before matching.Source: §I · p. 1
  • Prior GAN-based image translation methods had limitations generating higher-resolution images, where artifacts can occur.The paper adopts CycleGAN's two generators for photo-to-sketch and sketch-to-photo synthesis, while adding multiple discriminators and L1 reconstruction error.Source: §I · pp. 1–2
  • PS2-MAN applies adversarial supervision to hidden generator layers to refine outputs across resolutions.Feature maps from deconvolution layers are converted into outputs at three resolutions, each with a discriminator.Source: §I · p. 2
  • The authors describe their work as the first detailed study of adversarial networks specifically for face photo-sketch synthesis.They distinguish it from prior generic image-to-image translation methods with preliminary results on this task.Source: §I · pp. 1–2
  • The paper presents detailed synthesis experiments and ablations to evaluate the method and the role of iterative synthesis.Source: §I · p. 2

II. RELATED WORK

This section reviews face photo-sketch synthesis methods and generative modeling for image-to-image translation. It surveys method categories and representative approaches to synthesis.

A. Face photo-sketch synthesis

  • Photo-sketch synthesis methods have been classified by model construction and representation strategy.The cited taxonomies include subspace learning, sparse representation, Bayesian inference, holistic image, independent local patch, and spatially constrained local patch approaches.Source: A. Face photo-sketch synthesis · p. 2
  • Earlier synthesis approaches include subspace and patch-based methods, sparse representations, Bayesian models, Markov methods, and transductive learning.The cited LLE patch method averages overlapping sketch patches, which can blur outputs and ignore relationships among patches.Source: A. Face photo-sketch synthesis · p. 2
  • Later work includes multi-representation and superpixel synthesis, spatial-neighbor Bayesian models, CNN methods, and attribute-based sketch-to-photo synthesis.Source: A. Face photo-sketch synthesis · pp. 2–3

B. Image-to-image translation

The section reviews GAN-based image-to-image translation and formulates face photo-sketch synthesis as paired, bidirectional translation. Its formulation uses two generators for the photo-to-sketch and sketch-to-photo directions.

B. Image-to-image translation

  • Image-to-image translation methods include fully convolutional photo-sketch synthesis and related tasks such as sketch synthesis, photo-caricature translation, and parameterized avatars.Source: B. Image-to-image translation · p. 3
  • GANs use a generator and discriminator to learn realistic image distributions, with conditional GANs incorporating inputs such as images; CycleGAN and related methods support unpaired translation.Source: B. Image-to-image translation · p. 3

A. Formulation

  • The formulation treats photo-sketch synthesis as two mappings between paired photos and sketches, using generators GA and GB for the two directions.For a photo input, GA produces a synthesized sketch and GB maps that sketch back to a reconstructed photo; the inverse direction is also defined.Source: A. Formulation · p. 3

B. Objective

The objective combines adversarial, synthesis, and cycle-consistency losses. Multi-resolution outputs receive discriminator and target-image supervision, with hidden-layer feedback used to refine synthesis and address artifacts.

B. Objective

  • Adversarial loss can sharpen outputs compared with L1 or L2-only training, but artifacts in generated images can adversely affect face-sketch matching.The paper links high-resolution artifacts to training instabilities that become more severe as image resolution increases.Source: B. Objective · p. 3
  • Three discriminators supervise each generator's 64×64, 128×128, and 256×256 outputs, providing feedback to hidden layers for iterative refinement.For photo input, GA produces three sketch outputs; the final output is passed to GB for three reconstructions, with an analogous path for sketch input.Source: B. Objective · p. 4, Figure 2 · p. 4
  • The synthesis loss is an L1 difference between synthesized images and their corresponding targets, minimized at all three resolution levels.Source: B. Objective · p. 4
  • Cycle-consistency losses at different resolution stages regularize the mappings by enforcing forward-backward consistency.The final objective also combines L1 synthesis error and adversarial loss; the paper describes cycle consistency as additional regularization.Source: B. Objective · p. 4

C. Network Architecture

The generators combine stride-2 convolutions, residual blocks, and fractionally strided convolutional layers. The discriminators use 70 × 70 PatchGANs.

C. Network Architecture

  • Generator sub-networks use stride-2 convolutions, residual blocks, and fractionally strided convolutional layers.The listed configuration is C7S1-64, C3-128, C3-256, RB256×9, TC64, TC32, C7S1-3.Source: C. Network Architecture · pp. 4–5
  • The generator is C7S1-64, C3-128, C3-256, RB256×9, TC64, TC32, C7S1-3; C7S1 is 7×7 convolution-BatchNorm-ReLU, stride 1, and Ck is 3×3 convolution-BatchNorm-ReLU, stride 2.RBk×m denotes m residual blocks, each containing two 3×3 convolutional layers with the same number of filters; TC denotes 3×3 transposed-convolution-BatchNorm-ReLU with k filters and stride 1.Source: C. Network Architecture · p. 5
  • The discriminator uses a 70 × 70 PatchGAN that classifies overlapping image patches as real or fake.Its configuration is C64-C128-C256-C512, with 4 × 4 convolution, batch normalization, and LeakyReLU layers at stride 2.Source: C. Network Architecture · p. 5

IV. EXPERIMENTAL RESULTS

The experiments evaluate the proposed method on CUFS and CUFSF, comparing it with state-of-the-art methods and examining its components through ablation studies. Both visual and quantitative comparisons are presented.

IV. EXPERIMENTAL RESULTS

  • The proposed method is evaluated on the CUFS and CUFSF datasets.Source: §IV · p. 5
  • The evaluation compares the proposed method's performance with state-of-the-art methods.Source: §IV · p. 5
  • Ablation studies examine various components, with visual and quantitative comparisons presented for both the ablations and performance evaluation.Source: §IV · p. 5

A. Datasets

The experiments use the CUFS and CUFSF viewed-sketch datasets. CUFSF includes lighting variation and shape exaggeration, while both datasets provide facial landmarks for alignment.

A. Datasets

  • CUFS contains 188 CUHK student faces, 123 AR faces, and 295 XM2VTS faces, each paired with an artist-drawn sketch.The photos are frontal, taken under normal lighting, and show neutral expressions.Source: A. Datasets · p. 5
  • CUFSF contains 1,194 FERET persons and is described as particularly challenging because its photos vary in illumination and its sketches exaggerate shape.The sketches are drawn by an artist while viewing the photo, and the dataset is closer to a forensic-sketch scenario.Source: A. Datasets · p. 5
  • Both datasets include facial landmark coordinates that can be used for alignment.For datasets without landmarks, recent face-alignment algorithms can be applied during preprocessing.Source: A. Datasets · p. 5

B. Training Details

Training uses 256 × 256 inputs, random-noise augmentation, and a learning rate that decays over 200 epochs. The section also reports the dataset splits and an ablation figure.

B. Training Details

  • Training images are resized to 256 × 256, augmented online with random noise, and trained with a learning rate of 0.0002 for 100 epochs followed by linear decay to 0 over 100 epochs.The model is trained from scratch using Adam, with weights initialized from a Gaussian distribution of mean 0 and standard deviation 0.02.Source: B. Training Details · p. 5
  • CUHK uses 60 training, 28 validation, and 100 test pairs, while CUFSF uses 600 training, 297 validation, and 297 test pairs.Horizontally flipping the 60 CUHK training pairs yields 120 training images.Source: B. Training Details · p. 5
  • The ablation figure compares configurations with one, two, or three discriminator resolutions and shows the three-level configuration with minimized artifacts and more realistic results.Source: Figure 3 · p. 5

C. Ablation Study

The ablation compares single- and multi-level discriminator supervision on CUHK. Visual quality improves as supervision is added at more resolution levels, with similar trends in SSIM and FSIM.

C. Ablation Study

  • C−D256 uses a single discriminator at the final 256×256 resolution level.Source: C. Ablation Study · p. 5
  • C−D256,128 and C−D256,128,64 add discriminators at the 128×128 and 64×64 levels, respectively.The latter configuration uses three resolution levels: 256×256, 128×128, and 64×64.Source: C. Ablation Study · p. 5
  • Visual quality improves as adversarial supervision is added at more resolution levels, with similar trends in SSIM and FSIM.The comparison includes quantitative measurements, but their values are not supplied here.Source: C. Ablation Study · p. 5

D. Comparison with state-of-the-art methods

Across the reported synthesis and matching evaluations, PS2-MAN compares favorably with existing methods. The section reports improved SSIM and FSIM, matching rates at all ranks, and fewer visual artifacts.

D. Comparison with state-of-the-art methods

V. CONCLUSION

The conclusion presents PS2-MAN as a high-resolution photo-sketch synthesis method using multi-adversarial supervision and iterative bidirectional training. Evaluations on two datasets show improvements in visual quality and photo-sketch matching rates.

V. CONCLUSION

  • PS2-MAN targets high-resolution photo-sketch synthesis by applying adversarial supervision to hidden generator layers.Source: §V · p. 7
  • Forward and backward synthesis are trained iteratively in a CycleGAN framework using L1 reconstruction and cycle-consistency losses.The conclusion describes these losses as providing regularization for high-quality, high-resolution synthesis.Source: §V · p. 7
  • Evaluations on two datasets show significant improvements in visual quality and photo-sketch matching rates over recent generative methods.Source: §V · p. 7

Related papers

Loading 1710.10182v2…