PaperlayerView exact revision on arXiv

Source-linked AI summary

ArtGAN: Artwork Synthesis with Conditional Categorical GANs

Wei Ren Tan, Chee Seng Chan, Hernan Aguirre, Kiyoshi Tanaka

arXiv:1702.03410v2cs.CV

TL;DR

Prior GAN image generation emphasized images with structured shapes, while artwork can be abstract and difficult to separate into foreground and background. ARTGAN feeds label information back to the generator and adds pixel-wise reconstruction loss. The paper reports visually compelling artwork and natural-looking CIFAR-10 samples with clear object structures.

  • Problem

    Prior GAN image-generation work focused on images with structured shapes, whereas artwork may be abstract, lack clear foreground-background separation, and depart from natural shapes.

    Source: §1 · p. 1

  • Method

    ARTGAN conditions the generator on labels, back-propagates label-classification errors from the discriminator, and adds an L2 pixel-wise reconstruction loss to the generator's adversarial loss.

    Source: §1 · p. 2, §2.3 · p. 3

  • Results

    ARTGAN's generated artwork is reported as more visually natural than DCGAN and GAN/VAE, and its CIFAR-10 samples as natural-looking with clear object structures.

    Source: §1 · p. 2, §3.3 · p. 4

  • Takeaways & Limitations

    The reported experiments show ARTGAN generating artwork and CIFAR-10 images with object-specific appearances and shapes despite substantial shape variability.

    Source: §1 · p. 2, §6.3 · p. 6

Abstract

from arXiv · show

This paper proposes an extension to the Generative Adversarial Networks (GANs), namely as ARTGAN to synthetically generate more challenging and complex images such as artwork that have abstract characteristics. This is in contrast to most of the current solutions that focused on generating natural images such as room interiors, birds, flowers and faces. The key innovation of our work is to allow back-propagation of the loss function w.r.t. the labels (randomly assigned to each generated images) to the generator from the discriminator. With the feedback from the label information, the generator is able to learn faster and achieve better generated image quality. Empirically, we show that the proposed ARTGAN is capable to create realistic artwork, as well as generate compelling real world images that globally look natural with clear shape on CIFAR-10.

1. INTRODUCTION

Prior GAN image-generation work focused on datasets with distinguishable foregrounds and backgrounds, usually one object, and structured shapes, while artwork can lack these properties. ARTGAN addresses this challenge with label feedback to the generator and an added reconstruction loss.

1. INTRODUCTION

  • Earlier GAN image-generation datasets mostly featured distinguishable foregrounds and backgrounds, one object per image, and objects with structured shapes.The cited examples include MNIST, CIFAR-10, CUB-200, and LFW.Source: §1 · p. 1
  • Artwork presents a harder generation target because backgrounds and foregrounds can be difficult to distinguish and some works do not follow natural shapes.The introduction includes abstract and Cubist paintings as examples.Source: §1 · p. 1
  • The paper motivates subject-focused GAN training by analogy to art instruction, which recommends practicing a particular skill or subject at a time.It contrasts this with GAN training on unlabeled data, then notes that prior conditional GANs feed a vector to both generator and discriminator without label feedback to intermediate layers.Source: §1 · p. 1
  • ARTGAN feeds labels to the generator and back-propagates discriminator errors to it, while adding an L2 pixel-wise reconstruction loss to the adversarial loss.The discriminator outputs K + 1 classes and assigns generated images a label based on the input label.Source: §1 · p. 2

2. APPROACH

ARTGAN extends GANs with label information that lets the generator receive discriminator feedback about generated-image labels. Its architecture combines adversarial training with a pixel-wise reconstruction loss used to improve training stability.

2.1. Preliminaries

  • The approach builds on the GAN minmax setup, in which the discriminator distinguishes generated from training samples and the generator seeks to match the training-data distribution.The discriminator is updated by minimizing its loss, and the generator’s adversarial objective is formulated as a minimization problem.Source: §2.1 · p. 2, §2.3 · p. 3

2.2. ARTGAN

  • ARTGAN back-propagates discriminator errors based on generated-image labels to the generator, using label feedback to help it learn better.Unlike CondGAN, ARTGAN feeds the label information to the generator only; the discriminator assigns generated images labels based on that information and uses sigmoid outputs.Source: §2.2 · pp. 2–3

2.3. Details and Formulation of Architecture

  • The generator takes concatenated random noise and label information as input, while the discriminator maps an image to a label-probability distribution.The generator comprises a zNet followed by a decoder; the discriminator comprises an encoder followed by a classifier.Source: §2.3 · p. 3
  • The formulation uses K real classes and an additional FAKE class, and randomly chooses a label for each generated image.The training procedure samples noise and labels alongside real-image minibatches.Source: §2.3 · p. 3
  • ARTGAN adds an L2 pixel-wise reconstruction loss to the generator’s adversarial loss to improve training stability.The generator loss is the sum of the adversarial and reconstruction losses; the reconstruction loss updates the generator, not the encoder, because using it to update the encoder degraded generated-image quality.Source: §2.3 · p. 3

3. EXPERIMENTS

The experiments evaluate ARTGAN on annotated Wikiart and CIFAR-10, including artwork samples conditioned on genre, artist, and style. ARTGAN ranks best among compared models on Parzen-window log-likelihood, and nearest-neighbor comparisons indicate it does not simply memorize training paintings.

3.1. Dataset

  • Wikiart contains around 80,000 annotated artworks; all have style labels, about 60,000 genre labels, and around 20,000 artist labels.The dataset was split into 30% testing and the remainder training.Source: §3.1 · pp. 3–4

3.2. Experiment Settings

  • Training ran for 100 epochs with minibatches of 128, and the learning rate was reduced tenfold at epoch 80.The experiments used RmsProp with an initial learning rate of 0.001.Source: §3.2 · p. 4

3.3. Artwork Synthesis Quality

  • Artist-conditioned samples reflect reported preferences: Gustave Dore’s are greyish, while Vincent van Gogh’s are colourless and sombre.The paper relates these appearances to engraving and Sketch-and-Study examples in the dataset.Source: Figure 3 · p. 4, §3.3 · p. 4
  • Ukiyo-e samples retain a yellowish appearance and mimic subject patterns, although their subjects are hardly recognizable.Source: Figure 4 · p. 4, §3.3 · p. 4
  • ARTGAN’s genre-conditioned artwork looks more natural than DCGAN outputs, which contain many artifacts, and GAN/VAE outputs, which are less compelling.The GAN/VAE samples are described as less noisy and slightly more natural than DCGAN samples.Source: §3.3 · p. 4

3.4. Drawing CIFAR-10 with ARTGAN

  • On CIFAR-10, ARTGAN produces clearer automobile and horse shapes than DCGAN, despite the dataset’s high shape variability.The paper describes DCGAN outputs as more recognizable than its artwork outputs, but ARTGAN improves the CIFAR-10 images.Source: Figure 5 · p. 4, §3.4 · pp. 4–5

3.5. Quantitative Analysis

  • ARTGAN performs best among compared models on Parzen-window log-likelihood, while nearest-neighbor comparisons indicate it does not simply memorize training paintings.The authors caution that the likelihood measurements might be misleading.Source: Table 1 · p. 5, §3.5 · p. 5, Figure 6 · p. 5

4. CONCLUSIONS

ARTGAN synthesizes challenging, complex images, and the experiments attribute improved artwork quality to label feedback during back-propagation. The authors identify deeper models and joint learning of modes as future directions.

4. CONCLUSIONS

  • The authors propose deeper ARTGAN to encode more detail concepts and jointly learning modes to generate artwork from combinations of modes.Source: §4 · p. 5
  • The experiments show that label information fed back during back-propagation improves the quality of generated artwork.Source: §4 · p. 5
  • ARTGAN is proposed for synthesizing challenging and complex images.Source: §4 · p. 5

6. APPENDIX

The appendix lists the generator and discriminator configurations and reports additional conditional-generation results on Wikiart and CIFAR-10, alongside a nearest-neighbour check of generated paintings.

6.1. Model Configurations

6.2. More Results for Wikiart Dataset

  • Wikiart examples show genre- and artist-conditioned paintings, including mountain-like Roerich outputs and forest landscapes associated with Shishkin.The mountain-like outputs are described as unrealistic.Source: §6.2 · p. 6, Figure 7 · p. 7, Figure 8 · p. 8

6.3. More Results on CIFAR-10

  • ARTGAN generates CIFAR-10 images with object-specific appearances and shapes despite substantial shape variability.Figure 9 shows generated images from each class.Source: §6.3 · p. 6, Figure 9 · p. 9

6.4. More Results on Neighbourhood

  • Nearest-neighbour examples are presented to support the claim that ARTGAN does not simply memorize the Wikiart training set.Figure 10 shows generated paintings alongside their nearest neighbours.Source: §6.4 · p. 6, Figure 10 · p. 10

Related papers

Loading 1702.03410v2…