PaperlayerView exact revision on arXiv

Source-linked AI summary

Gated-GAN: Adversarial Gated Networks for Multi-Collection Style Transfer

Xinyuan Chen, Chang Xu, Xiaokang Yang, Li Song, Dacheng Tao

arXiv:1904.02296v1cs.CV

TL;DR

Collection style transfer seeks to render photographs in an artist’s or genre’s style, while unpaired GANs face instability and semantic-consistency challenges. Gated-GAN routes images through style-specific branches in one network and adds reconstruction loss; its reported image quality is similar to CycleGAN while producing four styles from one network.

  • Problem

    Collection style transfer seeks to imitate an artist’s or genre’s style, while unpaired GANs can produce outputs without the input’s semantic content and prior cycle-consistent methods transfer only one style per network.

    Source: §I · pp. 1–2

  • Method

    Gated-GAN uses an encoder, style-selecting gated transformer, and decoder, with adversarial training and an auto-encoder reconstruction loss.

    Source: §I · p. 2, §III · p. 4

  • Results

    Image quality was similar to CycleGAN, while four styles were produced from a single network.

    Source: 4) Comparison with Collection Style… · p. 8

  • Takeaways & Limitations

    Weighted connections between transformer branches provide an opportunity to explore new styles learned from artists or genres.

    Source: §VI · p. 12

Abstract

from arXiv · show

Style transfer describes the rendering of an image semantic content as different artistic styles. Recently, generative adversarial networks (GANs) have emerged as an effective approach in style transfer by adversarially training the generator to synthesize convincing counterfeits. However, traditional GAN suffers from the mode collapse issue, resulting in unstable training and making style transfer quality difficult to guarantee. In addition, the GAN generator is only compatible with one style, so a series of GANs must be trained to provide users with choices to transfer more than one kind of style. In this paper, we focus on tackling these challenges and limitations to improve style transfer. We propose adversarial gated networks (Gated GAN) to transfer multiple styles in a single model. The generative networks have three modules: an encoder, a gated transformer, and a decoder. Different styles can be achieved by passing input images through different branches of the gated transformer. To stabilize training, the encoder and decoder are combined as an autoencoder to reconstruct the input images. The discriminative networks are used to distinguish whether the input image is a stylized or genuine image. An auxiliary classifier is used to recognize the style categories of transferred images, thereby helping the generative networks generate images in multiple styles. In addition, Gated GAN makes it possible to explore a new style by investigating styles learned from artists or genres. Our extensive experiments demonstrate the stability and effectiveness of the proposed model for multistyle transfer.

I. INTRODUCTION

Collection style transfer aims to render photographs in the style of an artist or genre, a different goal from imitating a single reference image. Gated-GAN addresses this with a single multi-style network and an encoder-decoder reconstruction objective.

II. RELATED WORK

  • Earlier style-transfer work progressed from texture and optimization methods to feedforward networks, including approaches that learned multiple styles from individual style images.The cited feedforward approaches include methods for faster transfer and multi-style transfer.Source: §I · p. 1

I. INTRODUCTION

  • Collection style transfer learns an artist’s or genre’s style from a painting collection, rather than transferring the style of a single reference image.The motivating examples include rendering a landscape as if painted by Monet or in the style of an animation.Source: §I · p. 1
  • Unpaired GANs can map inputs to target images with arbitrary semantic content, while cycle-consistent approaches use an additional network to map outputs back to the original domain.The added cycle-consistency networks make training more complicated as the number of styles increases.Source: §I · p. 2
  • Gated-GAN combines multiple style branches in one network with an encoder-decoder reconstruction loss to preserve semantic consistency in one-sided transfer.The authors describe this design as using fewer parameters and being readily generalized to multiple styles.Source: §I · p. 2

A. Traditional Texture Transfer Method

The related-work section traces style transfer from texture synthesis and optimization to feedforward and adversarial networks. It distinguishes collection-level style learning from methods that derive a style from one image.

A. Traditional Texture Transfer Method

  • Texture-transfer methods use low-level target-image features and can take a long time to migrate a style from one image to another.Texture transfer renders an object with texture taken from another object.Source: A. Traditional Texture Transfer Method · p. 2

B. Optimization-based Methods

  • Optimization-based methods can produce high-quality results but are computationally expensive because each step requires forward and backward passes through a pretrained network.Source: B. Optimization-based Methods · p. 2

C. Feedforward Networks-based Methods

  • Feedforward methods accelerate transfer, and multi-style variants include conditional or adaptive normalization and networks with style-specific filter banks.The cited multi-style methods generally extract style representations from style images; this paper instead targets styles learned from collections.Source: C. Feedforward Networks-based Methods · p. 3

D. Adversarial Network-based Methods

  • Adversarial methods include cycle-consistent unpaired translation and class-conditional image generation, while this work applies adversarial learning to migrate collection styles to content images.Source: D. Adversarial Network-based Methods · pp. 3–4

III. PROPOSED ALGORITHM

Gated-GAN uses an encoder, a gated transformer, and a decoder to transfer collection styles, with adversarial learning and reconstruction loss. The reconstruction objective is intended to stabilize training and preserve input structure.

III. PROPOSED ALGORITHM

  • The generator encodes an input, routes its features through a gated transformer, and decodes them; separate gates integrate multiple styles in one network.The training setup uses unpaired content images and target collection images.Source: §III · p. 4

A. Adversarial Network for Style Transfer

  • Adversarial loss trains the generator to produce images resembling the target style distribution while the discriminator distinguishes generated images from genuine target images.The paper uses least-squares GAN loss for a smooth, non-saturating discriminator gradient.Source: A. Adversarial Network for Style… · p. 4

B. Auto-encoder Reconstruction Loss for Training Stabilization

  • The auto-encoder reconstruction loss compares Dec(Enc(x)) with x to restrict possible mappings, while the encoder-decoder subnetwork is intended to preserve input structure and output diversity.The authors motivate reconstruction loss as a way to address GAN instability and mode collapse under unpaired training.Source: B. Auto-encoder Reconstruction Loss… · p. 4

C. Adversarial Gated Network for Multi-Collection Style Transfer

Gated-GAN routes images through style-specific branches in a gated transformer while sharing an encoder and decoder across styles. An auxiliary classifier guides the generator toward recognizable style categories.

1) Gated Generated Network:

  • The gated transformer switches among style-specific branches between a shared encoder and decoder, whose parameters need not be duplicated for each style.Each branch uses a residual transfer module.Source: 1) Gated Generated Network · p. 4

2) Auxiliary Classifier for Multiple Styles:

  • The discriminator is trained to distinguish real from stylized images, while the auxiliary classifier predicts style categories.The classifier has fully connected output layers separate from the discriminator's.Source: Figure 2 · p. 5, 2) Auxiliary Classifier for Multiple… · p. 5
  • Adversarial loss alone can mix styles, while LabelGAN's simultaneous encouragement of all real-class logits creates weighted-average gradients.Each sample is pushed toward multiple label predictors at once.Source: 2) Auxiliary Classifier for Multiple… · p. 5
  • Auxiliary classifier C shares low-level convolutional layers with the discriminator, predicts K style classes, and guides the generator toward high-confidence correct-style outputs.The generator's entropy objective is described as a way to overcome the overlaid-gradient problem.Source: 2) Auxiliary Classifier for Multiple… · p. 5

IV. IMPLEMENTATION

The implementation uses a convolutional encoder and decoder around gated and residual transfer blocks, with PatchGAN discriminators and classifiers focused on local image patches.

IV. IMPLEMENTATION

  • The generator uses two stride-2 encoder convolutions, one gated residual transfer block, five residual blocks, and two fractionally strided decoder convolutions.Instance normalization follows the convolutional layers.Source: 1) Network Configuration · p. 5
  • PatchGAN discriminators and classifiers process overlapping 70×70 patches rather than full images, focusing on local high-frequency texture and style features.The passage also notes PatchGAN has fewer parameters and accepts any input size.Source: 1) Network Configuration · p. 5

2) Training Strategy:

Training combines adversarial and style-classification objectives with image smoothing and an autoencoder reconstruction objective. The experiments assess image quality with FID and also examine training stability and functionality.

2) Training Strategy:

  • Total variation loss is used to smooth generated images, and the generator objective combines it with adversarial and style-classification losses.The encoder and decoder are also trained as an autoencoder using weighted reconstruction loss.Source: 2) Training Strategy · p. 5
  • Training uses 128×128 inputs, randomly flipped and cropped after scaling samples to 143×143, to augment the training set and reduce computational cost.At test time, images are evaluated at their original resolution.Source: 2) Training Strategy · p. 6
  • Discriminator updates sample from a buffer of 50 previously generated images to stabilize training; the reported discriminator and generator update counts are both one per iteration.The buffer is sampled during each discriminator training step.Source: 2) Training Strategy · p. 6

V. EXPERIMENTS

  • The experiments evaluate image quality, stability, and functionality, including texture synthesis, gated-transformer filter visualization, and comparisons with state-of-the-art methods.The assessment section uses FID, comparing generated-image embeddings with those of authentic paintings.Source: §V · p. 6, A. Assessment of Image Quality · p. 6

A. Assessment of Image Quality

  • FID is computed between generated stylized images and authentic paintings, using the means and covariances of their Inception-feature embeddings.The generated and real sample distributions are treated as continuous multivariate Gaussians in the embedding space.Source: A. Assessment of Image Quality · p. 6

B. Texture synthesis

The texture-synthesis experiment probes what the gated transformer learns by training on texture samples and visualizing synthesized images alongside decoded features.

B. Texture synthesis

  • Each style is trained using a textured image, and Gaussian noise is supplied as the generative network input for texture synthesis.The figure shows three synthesized outputs for different noise samples alongside each texture example.Source: B. Texture synthesis · pp. 6–7, Figure 3 · p. 7
  • After adversarial training, the generator produces realistic textured images from the noise inputs.Figure 3 presents four cases, with three noise-conditioned results per texture example.Source: B. Texture synthesis · p. 7, Figure 3 · p. 7
  • Decoded gated-transformer features reveal style representations such as color and stroke, interpreted as bases combined using encoder-learned coefficients.Feature visualizations activate one of 128 channels with Gaussian noise before passing it through a style-specific transformer filter and shared decoder.Source: B. Texture synthesis · p. 7, Figure 4 · p. 7

C. Style Transfer

Gated-GAN generates images in four artist or genre styles from one network. Its collection-transfer outputs adapt to the content scene, while comparisons highlight the difference between learning a collection and applying a selected style image.

1) Multi-Collection Style Transfer:

  • For Photo → Monet, Figure 5 pairs input photos with similar-landscape Monet paintings and stylized outputs that adapt to different themes.The columns show input photos, selected Monet paintings, and the corresponding stylized images.Source: Figure 5 · p. 7, 1) Multi-Collection Style Transfer · p. 7
  • The model produces collection-style images resembling Monet’s paintings across different scenes and is evaluated in Monet, Van Gogh, Cezanne, and Ukiyo-e styles.The evaluation uses collections from, with 1,073 Monet, 400 Van Gogh, 526 Cezanne, and 563 Ukiyo-e paintings.Source: 1) Multi-Collection Style Transfer · p. 7, C. Style Transfer · p. 7

2) Comparison with Image Style Transfer:

  • Unlike Gatys et al., which requires a closely matched target image or yields an average collection style, Gated-GAN produces diverse, reasonable samples from an artist’s style distribution.The comparison uses two representative artworks per content image and the collection’s average Gram matrix as the collection-style target.Source: 2) Comparison with Image Style Transfer · p. 8

3) Comparison with Universal Style Transfer:

The comparisons distinguish collection-style transfer from applying a specific style image and test Gated-GAN against collection-transfer and conditional-GAN approaches. The reported results show scene-adaptive outputs, comparable quality to CycleGAN with four styles in one network, and greater stylistic diversity than the conditional-GAN variants.

3) Comparison with Universal Style Transfer:

  • Gated-GAN adapts Monet-like rendering to the scene, while applies a particular style image and may carry its colors into content that does not fit that subject.For a cityscape, Gated-GAN produces foggy strokes associated with Monet’s London paintings; using “Flowers at Vetheuil” as the style image gives green and red spots.Source: 3) Comparison with Universal Style… · p. 8

4) Comparison with Collection Style Transfer:

5) Comparison with Conditional GAN:

  • The conditional GAN fails to produce meaningful results without paired examples; adding cycle-consistent loss yields styles that differ mainly in color, unlike Gated-GAN’s varied strokes and textures.Figure 11 compares the input, conditional GAN, conditional GAN with cycle-consistent loss, and the proposed method across Monet, Ukiyo-e, and Cezanne styles.Source: 5) Comparison with Conditional GAN · p. 9

D. Analysis of Loss Function

The loss analysis selects λCLS = 1 and λR = 10, and examines how reconstruction loss and its norm affect training and output quality.

1) Influence of Parameters in Loss Function:

2) Analysis of Auto-encoder Reconstruction Loss:

  • The authors report no significant difference between L1 and L2 reconstruction losses and adopt L1, as in CycleGAN’s cycle-consistent loss.The comparison is evaluated in terms of FID.Source: 2) Analysis of Auto-encoder… · pp. 10–11
  • Removing autoencoder reconstruction loss leads to random-texture outputs that become less diverse over training, whereas the full model produces satisfying results.The reconstruction objective encourages semantic structure in the output to align with the input, which the authors connect to diversity across different inputs and avoiding mode collapse.Source: 2) Analysis of Auto-encoder… · p. 11

E. Analysis of network architecture

The architecture analysis finds that skip connections help preserve input structure and that one gated-transformer residual block is sufficient. Incremental training adds a style branch while keeping the encoder-decoder subnetworks fixed, with a reported competitive FID score.

E. Analysis of network architecture

F. Incremental Training

  • For a new Monet style, incremental training adds a residual-block branch in the gated-transformer while keeping encoding/decoding subnets fixed; it scores FID 57.27 versus 55.13 for Gated-GAN and 64.14 for baseline CycleGAN.Source: F. Incremental Training · pp. 11–12

G. Linear Interpolation of Styles

Gated branches can be blended to create transitions between learned styles, including a smooth Monet-to-Van Gogh transition. The paper concludes that its single-network model supports multi-collection transfer and exploration of new styles.

G. Linear Interpolation of Styles

  • Adjusting gate weights blends learned styles and can create new styles or transitions between artists or genres.The model switches among style branches using gates applied to the encoded input.Source: G. Linear Interpolation of Styles · p. 12
  • Convex combinations of Monet and Van Gogh style branches produce a smooth transition between the styles.Figure 17 varies α from 0 to 1.Source: G. Linear Interpolation of Styles · p. 12, Figure 17 · p. 13
  • Figure 16 compares original inputs, CycleGAN results, simultaneous-style training results, and incremental-training results.The caption identifies the comparison conditions but does not state which approach performs better.Source: Figure 16 · p. 12

VI. CONCLUSIONS AND FUTURE WORK

  • The conclusion presents Gated GAN as a single-network approach for multi-collection transfer, with experiments reporting stability, functionality, effectiveness, and satisfactory results against a single-style state-of-the-art algorithm.The authors propose future applications to object transfiguration, season transfer, and photo enhancement, and exploration of diversified style-transfer results.Source: §VI · p. 12

Related papers

Loading 1904.02296v1…