PaperlayerView exact revision on arXiv

Source-linked AI summary

Perceptual Adversarial Networks for Image-to-Image Transformation

Chaoyue Wang, Chang Xu, Chaohui Wang, Dacheng Tao

arXiv:1706.09138v2cs.CV

TL;DR

Image-to-image transformation needs methods that avoid the blur and artifacts associated with pixel-wise losses. PAN combines generative and perceptual adversarial losses in alternating training of transformation and discriminative networks, and experiments report effectiveness across several transformation tasks.

  • Problem

    Pixel-wise image-to-image losses can produce blur and artifacts, while existing perceptual-loss methods depend on external pretrained classification networks.

  • Method

    PAN combines generative adversarial loss with a perceptual adversarial loss using discriminator hidden layers, and alternately trains transformation network T and discriminative network D.

  • Results

    Experiments on several image-to-image transformation tasks demonstrate that PAN is effective and promising for practical applications.

  • Takeaways & Limitations

    PAN provides a generic framework for learning mappings between paired images across multiple image-to-image transformation tasks.

Abstract

from arXiv · show

In this paper, we propose a principled Perceptual Adversarial Networks (PAN) for image-to-image transformation tasks. Unlike existing application-specific algorithms, PAN provides a generic framework of learning mapping relationship between paired images (Fig. 1), such as mapping a rainy image to its de-rained counterpart, object edges to its photo, semantic labels to a scenes image, etc. The proposed PAN consists of two feed-forward convolutional neural networks (CNNs), the image transformation network T and the discriminative network D. Through combining the generative adversarial loss and the proposed perceptual adversarial loss, these two networks can be trained alternately to solve image-to-image transformation tasks. Among them, the hidden layers and output of the discriminative network D are upgraded to continually and automatically discover the discrepancy between the transformed image and the corresponding ground-truth. Simultaneously, the image transformation network T is trained to minimize the discrepancy explored by the discriminative network D. Through the adversarial training process, the image transformation network T will continually narrow the gap between transformed images and ground-truth images. Experiments evaluated on several image-to-image transformation tasks (e.g., image de-raining, image inpainting, etc.) show that the proposed PAN outperforms many related state-of-the-art methods.

I. INTRODUCTION

Image-to-image transformation methods map inputs to desired outputs across vision applications, but pixel-wise losses can produce blur and artifacts. PAN addresses this setting by combining generative and perceptual adversarial losses, with experiments across several tasks showing strong transformation capability.

  • Image-to-image transformation maps an input image to a desired output across image processing, graphics, and computer vision applications.Examples include denoising, inpainting, super-resolution, colorization, and segmentation.
  • Pixel-wise losses provide reasonable outputs but can introduce image blur and artifacts.
  • Generative adversarial losses compare generated and real-world distributions and can improve image realism in image-to-image transformation.
  • Existing perceptual-loss approaches use hidden features from external pretrained classification networks to compare output and ground-truth images.
  • PAN combines generative adversarial loss with perceptual adversarial loss, which uses discriminator hidden layers to evaluate output–ground-truth discrepancy dynamically.The transformation and discriminative networks are trained adversarially, with the discriminator continually exploring discrepancies and the transformation network reducing them.
  • Experiments evaluate PAN on several image-to-image transformation tasks and report strong capability for accomplishing these transformations.

II. BACKGROUND

Feed-forward CNNs learn image-to-image mappings by encoding inputs and decoding outputs, while GAN-based methods model realistic image distributions. Related work spans paired and cross-domain translation methods, including cycle-based mappings between image domains.

  • Feed-forward CNNs encode input images into hidden representations and decode them into transformed outputs using back-propagation training.
  • Image-to-image methods use pixel-wise losses alone or with additional losses for tasks such as super-resolution, de-raining, and inpainting.
  • GANs learn generative models through a minimax game between generative and discriminative networks to reproduce real-world data distributions.
  • GAN-based image-to-image works include video prediction, super-resolution, de-raining, photo modification, and image inpainting.
  • Pix2pix-cGANs perform translations such as semantic labels to street scenes, object edges to pictures, and aerial photos to maps.
  • Cross-domain GAN variants explore cyclic mappings so images can be translated between domains and then reconstructed.

C. Perceptual loss

PAN builds on perceptual features from hidden network representations while using adversarial learning for paired image transformation. Its transformation network generates outputs from inputs, and its discriminator evaluates both distributional and perceptual discrepancies.

  • Hidden features from well-trained image classification networks can capture perceptual information, semantics, and image style distributions.
  • PAN is introduced as a framework combining generative and perceptual adversarial losses with two convolutional neural networks.
  • The generative adversarial loss trains a discriminator to distinguish real samples from generated samples and a generator to produce increasingly realistic outputs.
  • The transformation network T generates T(x) from input x, while each input is paired with a corresponding ground-truth image y.
  • The generative adversarial loss measures the distributional discrepancy between transformed images and ground-truth images.

B. Perceptual adversarial loss

The perceptual adversarial loss uses discriminator hidden layers to measure and continually update high-level discrepancies between transformed and ground-truth images. Unlike pixel-wise and conventional perceptual losses, it learns dynamic measurements through adversarial training.

  • Pixel-wise losses can produce blurry images and artifacts despite generating reasonable outputs.
  • The perceptual adversarial loss evaluates differences between transformed and ground-truth images using hidden layers of the discriminative network D.The hidden-layer representations are compared using a least absolute loss in the described experiments.
  • The transformation network T and hidden layers of D form a non-zero-sum game with separate losses LT and LD.LT trains T, while LD trains the hidden layers of D.
  • When hidden-layer discrepancy falls below margin m, LD updates D to explore new latent feature spaces that preserve discrepancy.This dynamic update distinguishes the loss from fixed pixel-wise and conventional perceptual measurements.
  • Conditional GAN loss evaluates whether generated images form appropriate pairs with inputs, whereas perceptual adversarial loss directly minimizes transformed-ground-truth differences across perspectives.

C. The perceptual adversarial networks

PAN combines generative and perceptual adversarial losses in alternating training of an image transformation network T and a discriminative network D. D distinguishes transformed images from ground truth using static and dynamic perceptual aspects, while T reduces their discrepancy.

  • PAN consists of an image transformation network T and a discriminative network D trained alternately in an adversarial process.
  • D distinguishes transformed image T(x) from ground-truth image y using both generative and dynamic perceptual aspects.
  • T is trained to generate increasingly better images by reducing the discrepancy between its output and the ground-truth image.

D. Network architectures

PAN uses two CNNs: transformation network T generates outputs, while discriminative network D measures transformed–ground-truth discrepancies through selected hidden layers. T uses an encoder–decoder architecture, and D supplies both perceptual measurements and adversarial discrimination.

  • Network architectures: PAN comprises image transformation network T and discriminative network D.T generates transformed images, while D computes discrepancies between transformed and ground-truth images.
  • Network architectures: T encodes inputs with convolutional layers and decodes outputs with transposed-convolutional layers, using mirrored skip-connections.The output layer uses Tanh rather than ReLU and omits batch normalization.
  • Network architectures: D extracts high-level features with convolutional layers and measures perceptual adversarial loss at its 1st, 4th, 6th, and 8th layers.Its final convolutional output is flattened and fed into a single subsequent component.
  • Network architectures: The same D architecture is applied across all demonstrated tasks, whereas T uses different architectures for loss-function analysis and image inpainting.The architecture note identifies transposed convolution as the deconvolution operation used in the framework.

IV. EXPERIMENTS

PAN is evaluated across diverse paired image-to-image transformation tasks using settings aligned with prior work. Experiments use alternating optimization of T and D with fixed training hyperparameters and task-dependent epoch counts.

  • Experimental setting up: Experiments cover single-image de-raining, image inpainting, semantic labels↔images, edges→images, and aerial→map transformation.The tasks use datasets associated with ID-CGAN, Context-Encoder, Cityscapes, pix2pix, and aerial-to-map translation.
  • Experimental setting up: The experiments adopt the same settings as existing works and report multiple evaluation metrics on the test sets.The supplied experimental setup states that comparisons were designed for fairness.
  • Experimental setting up: The shared hyperparameters are θ = 1, λ1 = 5, λ2 = 1.5, λ3 = 1.5, λ4 = 1, and batch size 4.Task-dependent epoch counts accommodate substantially different dataset sizes, with approximately 100k total training iterations.

B. Evaluation metrics

The paper evaluates PAN with qualitative comparisons and standard image-quality metrics, while analyzing loss components and hidden-layer choices. Results indicate trade-offs between perceptual detail, color preservation, and the balance of adversarial losses.

  • Evaluation metrics: Evaluation combines visual input–output comparisons with PSNR, SSIM, UQI, and Visual Information Fidelity measurements over test sets.These measures are used to assess transformed-image performance quantitatively.
  • Analysis of the loss functions: Pixel-wise loss alone performs worst in de-snowing, while adding cGAN loss improves de-snowing but introduces artifacts and lowers PSNR.Combining pixel-wise, cGAN, and VGG-16 perceptual losses further improves observed and quantitative quality, although color distortion remains.
  • Analysis of the loss functions: PAN removes most streaks without color distortion and achieves better quantitative performance than the compared loss configurations.Adding perceptual adversarial loss to cGAN substantially improves performance, while replacing the original GAN loss with cGAN does not further improve PAN.
  • Analysis of the loss functions: Lower discriminator layers emphasize patch-to-patch and color transformations but produce blurrier images, whereas higher layers capture high-frequency information while losing color information.PAN integrates these differing hidden-layer properties in its final configuration.
  • Analysis of the loss functions: Increasing θ initially improves label-to-facade results, but continued increases weaken the perceptual adversarial loss and can produce visual artifacts.Using only perceptual adversarial loss at θ = 0 already synthesizes visually reasonable images.

D. Comparing with existing works

PAN is compared with Context-Encoder and other image-to-image transformation approaches across inpainting, de-raining, and related tasks. The reported comparisons describe stronger semantic detail and reconstruction performance for PAN.

  • Context-Encoder: PAN better understood surrounding context and synthesized semantic contents and more details than Context-Encoder in inpainting examples.Context-Encoder more often copied the nearest region, usually background, into the missing part.
  • Pix2pix-cGAN: The section also presents comparisons involving edge-to-image, semantic-label, and aerial-photo transformation tasks using pix2pix-cGAN and PAN.The supplied passages identify the relevant visual comparisons and tables but do not report their numerical values.
  • ID-CGAN: PAN and ID-CGAN were evaluated for single-image de-raining using the same experimental setting, with synthetic training and test data plus real-world rainy images.The synthesized training set contained 700 images; testing used 100 synthetic and 50 real-world rainy images.

2) ID-CGAN:

The de-raining comparison evaluates PAN against ID-CGAN under matched settings and reports qualitative and quantitative advantages for PAN. The discussion links these differences to how perceptual information is learned.

  • ID-CGAN: PAN removed more rain streaks with less color distortion than ID-CGAN in zoomed comparisons.Both methods were reported to perform well overall on single-image de-raining.
  • ID-CGAN: On synthetic test images, PAN’s de-rained outputs were reported as much more similar to the corresponding ground-truth images than ID-CGAN’s.The passage does not provide the numerical values from Table III.

3) Pix2pix-cGAN:

PAN is compared with pix2pix-cGAN on inverse semantic-label translation, edge-to-object-photo synthesis, semantic-label generation, and aerial-photo-to-map translation. The reported visual results emphasize detail and reduced deformation, while task ambiguity limits gains in edges2images.

  • Semantic labels to cityscapes: For semantic labels to cityscapes images, PAN captured more details with less deformation than pix2pix-cGAN, producing more realistic synthetic images.The task is ill-posed because the transformation network must learn prior knowledge from training data.
  • Edges to images: PAN was trained to translate object edges into shoes and handbags and was compared with pix2pix-cGAN using visual examples and test-set quantitative results.The supplied passage identifies the comparison but does not state the numerical values or specific outcome.
  • Other pix2pix-cGAN tasks: PAN was also compared with pix2pix-cGAN for generating semantic labels from cityscapes photos and maps from aerial photos.Examples and quantitative results were presented in Fig. 10 and Table V.
  • Task difficulty: Edges2images is more difficult than cityscapes and aerial2map because one handbag sketch can correspond to many outputs with different colors and textures.The paper reports relatively small visual improvement on edges2images compared with the other two tasks.

E. Extension to unpaired image translations

The paper extends perceptual adversarial loss to unpaired image translation, where corresponding target images are unknown. In a horse-to-zebra experiment, adding the loss produced some performance improvement and demonstrated feasibility.

  • Motivation: In unpaired translation, the target image corresponding to a generated image is unknown, preventing direct paired-image discrepancy measurement.The proposed perceptual adversarial loss was originally designed to explore and minimize discrepancies between generated images and corresponding targets.
  • Method: The extension measures perceptual similarity across image domains using discriminator hidden-layer representations in the CycleGAN framework.The supplied passages describe generator outputs and discriminator-layer representations as the quantities used in the domain comparison.
  • Experiment: For horse↔zebra translation, the authors used CycleGAN defaults and discriminator layers 3 and 4 to measure perceptual adversarial loss.The experiment used λ3 = λ4 = 0.5, m = 0.1, and batch size 4.
  • Result: Considering perceptual similarity in unpaired translation improved model performance more or less in the reported qualitative experiment.The authors present this as evidence of the possibility of improving unpaired translation through cross-domain perceptual similarity.
  • PAN framework: PAN combines generative adversarial and perceptual adversarial losses while alternately updating discriminative network D and transformation network T.D explores discrepancies against ground truth, while T is trained to narrow them.
Loading 1706.09138v2…