PaperlayerView exact revision on arXiv

Source-linked AI summary

GAN Computers Generate Arts? A Survey on Visual Arts, Music, and Literary Text Generation using Generative Adversarial Network

Sakib Shahriar

arXiv:2108.03857v2cs.AIcs.CLcs.CVcs.LGstat.ML

TL;DR

The paper addresses how GANs have been used to generate computer art across visual arts, music, and literary text. It surveys architectures and recent applications, then synthesizes reported results and challenges, including gaps in music and literary-text evaluation.

  • Problem

    Earlier computer-generated art often depended on extensive programming and human supervision, while the survey identifies challenges in generating music and literary text with GANs.

    Source: §I · p. 1, A. Challenges · p. 12

  • Method

    The paper surveys GAN architectures and recent work generating visual arts, music, and literary texts, and reviews their results and challenges.

    Source: §I · p. 2, §V · p. 12

  • Results

    The survey presents recent GAN art-generation work across visual arts, music, and literary text, alongside architecture descriptions and a discussion of results.

    Source: §I · p. 2, §V · p. 12

  • Takeaways & Limitations

    The survey highlights challenges and future research directions for GAN-based art generation across the reviewed forms.

    Source: §I · p. 2, §V · p. 12

    Reviewed music work largely uses MIDI and lacks standard quality evaluation, while literary-text work focuses mostly on poetry and lacks consistent qualitative evaluation.

    Source: A. Challenges · p. 12

Abstract

from arXiv · show

"Art is the lie that enables us to realize the truth." - Pablo Picasso. For centuries, humans have dedicated themselves to producing arts to convey their imagination. The advancement in technology and deep learning in particular, has caught the attention of many researchers trying to investigate whether art generation is possible by computers and algorithms. Using generative adversarial networks (GANs), applications such as synthesizing photorealistic human faces and creating captions automatically from images were realized. This survey takes a comprehensive look at the recent works using GANs for generating visual arts, music, and literary text. A performance comparison and description of the various GAN architecture are also presented. Finally, some of the key challenges in art generation using GANs are highlighted along with recommendations for future work.

I. INTRODUCTION

The introduction frames computer-generated art as a long-standing creative question, noting that earlier systems often relied on extensive programming and human supervision. It presents GANs as a deep-learning approach and surveys their use across visual arts, music, and literary text.

I. INTRODUCTION

  • Computer-generated art includes work produced at least partly automatically by a computer program.The introduction distinguishes visual arts from performing arts and uses generative art and computer art for its focus.Source: §I · p. 1
  • Earlier computer-art systems produced uniform work and relied on programming and extensive human supervision, limiting independent generation.The introduction contrasts this with learning from data and experience without explicit programming.Source: §I · p. 1
  • GANs use a generator trained from noise and a discriminator that distinguishes generated samples from real data in an adversarial process.The introduction describes the networks as trained together toward fooling the discriminator about half the time.Source: §I · p. 2
  • The introduction identifies a scope gap between a broad deep-learning survey of visual arts and a focused review of free-hand sketching [13].The survey responds by focusing on GANs across three art forms.Source: §I · p. 2
  • The survey compares recent GAN-generated visual arts, music, and literary text, and discusses architectures, limitations, and future directions.It also describes primary GAN architectures for generative arts.Source: §I · p. 2

II. BACKGROUND

The background introduces GANs as a two-network adversarial model and notes that conditioning can make generation category-specific rather than noise-only.

II. BACKGROUND

  • The background section covers GAN architectures and common loss functions before the paper reviews art-generation work.The next section compares GAN applications in visual arts, music, and literary texts.Source: §II · p. 2

A. Generative Adversarial Netowrks

  • A basic GAN trains a generator to produce samples resembling real data and a discriminator to distinguish generated samples from real ones.The generator receives discriminator feedback but has no access to real data; the discriminator has access to ground-truth labels.Source: A. Generative Adversarial Netowrks · p. 2
  • Conditional GANs add auxiliary information, such as class labels, to the generator and discriminator so generation can depend on an input condition.In the basic GAN, generation depends only on input noise, which is unsuitable for some applications.Source: A. Generative Adversarial Netowrks · p. 2

1) Conditional GAN:

The background outlines conditional, convolutional, and recurrent GAN architectures, alongside several other variants relevant to art generation.

1) Conditional GAN:

  • Conditional GANs combine auxiliary information with noise to control generation, supporting multimodal outputs such as different music genres.The examples include jazz, rock, and classical music.Source: 1) Conditional GAN · p. 3

2) Deep Convolutional GAN:

  • DCGAN uses convolutional generator and discriminator networks, with strided and fractionally strided convolutions for downsampling and upsampling.The passage describes mapping between image space and a lower-dimensional latent space and identifies DCGAN as suitable for image generation.Source: 1) Conditional GAN · p. 3

3) Recurrent Adversarial Netwokrs:

  • The recurrent architecture uses a convolutional encoder to extract the current canvas and a decoder to decide whether to update it.Recurrent architectures are suited to sequential, time-dependent data and are generally used for text and audio applications.Source: 2) Deep Convolutional GAN · p. 3
  • Other architectures mentioned include InfoGAN, Laplacian GAN, and Wasserstein GAN.The paper directs readers to comparison surveys for a comprehensive architecture comparison.Source: 3) Recurrent Adversarial Netwokrs · p. 3

B. Common Loss Functions

The section introduces common GAN losses, explaining their roles and the distributions or prediction errors they measure.

B. Common Loss Functions

  • Cross-entropy loss is presented as a common objective for optimizing GAN models, with binary cross-entropy used for two categories.The passage identifies the true distribution as p(x) and the estimated distribution as q(x).Source: B. Common Loss Functions · p. 3
  • Mean squared error measures prediction error and is typically used in least square GANs.The passage defines the actual and predicted values over N training examples.Source: B. Common Loss Functions · p. 3
  • KL divergence compares two probability distributions and indicates how much one differs from the other.The passage identifies P(x) and Q(x) as the distributions being compared.Source: B. Common Loss Functions · p. 3
  • Wasserstein distance is presented as a loss function that has gained interest in GAN training.The passage describes it using distributions u and v and the set of couplings Γ(u,v).Source: B. Common Loss Functions · p. 3
  • The following section compares GAN applications in visual arts, music, and literary texts.It discusses recent works generating these art forms.Source: §III · p. 3

A. Organization

The survey is organized around visual arts, music, and literary text generation. For each area, it covers research challenges, compares results, and highlights GAN losses and architectures.

  • The survey is organized around three artwork-generation areas: visual arts, music, and literary text.Source: A. Organization · p. 3
  • Within each area, the survey examines research challenges and compares results.Source: A. Organization · p. 3
  • The survey also highlights loss functions, GAN types, and architectural details.Source: A. Organization · p. 3

B. Visual Arts Generation using GANs

This section surveys GAN approaches to visual-art generation, including sketch translation, painting, Japanese facial art, and handwriting. It compares their architectures and reported evaluations, while noting cases with limited or absent evaluation.

Sketch-to-image generation

  • 63.3% of participants preferred the style-conditioned sketch-to-image model over its baselines; it also outperformed existing works on FID and style-classification accuracy.Figure 4 shows generations conditioned on an input sketch and a specified artist or style.Source: B. Visual Arts Generation using GANs · p. 4, Figure 4 · p. 4

Sketch-conditioned generation

  • For [24], paired sketch/color training and a U-Net generator outperformed three existing models in volunteer preferences; ’s shoe-from-sketch samples were low quality and lacked appropriate evaluation.The [24] study used 55 volunteers to choose favorite and least-favorite generations.Source: B. Visual Arts Generation using GANs · pp. 3–4

Brushstroke generation

  • [27] used painting-environment controls such as brush size and color as generator inputs, producing rougher, more realistic strokes, but reported no evaluation.The paper does not elaborate on the generator and discriminator’s architectural parameters.Source: B. Visual Arts Generation using GANs · p. 4

Unconditional art generation

Japanese-art facial expressions

  • [29] introduced a Japanese-art facial-expression dataset of more than 5,500 labeled RGB images and explored StyleGAN, but did not evaluate generation performance.The labels include gender and social status; StyleGAN’s mapping network controls style while noise vectors add variation.Source: B. Visual Arts Generation using GANs · pp. 4–5, Figure 6 · p. 5

Face and sketch translation

  • [31] trained conditional GANs for both face-photo-from-sketch and sketch-from-photo generation, but reported no evaluation beyond generator and discriminator losses.The authors compared selected face sketches from prior works, an artist, and their model.Source: B. Visual Arts Generation using GANs · p. 5

Stroke generation

  • 91% classification accuracy was reported for StrokeNET’s preprocessed generated images.StrokeNET encodes stroke coordinates and pressure alongside brush radius, then generates a 265 by 265 image.Source: B. Visual Arts Generation using GANs · p. 5

Handwriting and calligraphy

C. Music and Melody Generation using GANs

The survey reviews GAN approaches to music and melody generation, covering their architectures, datasets, and evaluations. Reported findings include strong results against baselines as well as studies without quantitative quality evaluation.

  • The hybrid VAE-GAN in generated melodies from 1,998 ten-second MIDI samples across five genres, but its evaluation reported no quantitative results.Source: C. Music and Melody Generation using… · p. 6
  • The proposed model in averaged 3.27 across pleasingness, realism, and interest, and evaluators identified its samples as synthesized 48.1% of the time.Source: C. Music and Melody Generation using… · p. 6
  • MidiNet [37] used chord and prior-measure conditioning, while [41] outperformed random and maximum-likelihood baselines on BLEU and received a 4.1/5 melody score.For MidiNet, ratings were around 3 for pleasantness and realism; interest averaged 4 among musically trained participants and 3.4 among those without musical backgrounds.Source: C. Music and Melody Generation using… · p. 7
  • MuseGAN's jamming model performed best on intra-track metrics, while its composer and hybrid models performed better on inter-track metrics; the hybrid model scored 2.86/5 in a qualitative study.Source: C. Music and Melody Generation using… · p. 7

D. Poetry and Literary Text Generation using GANs

The survey describes GAN methods for poetry and literary text, including approaches to discrete sequences and image-conditioned generation. Reported evaluations show gains against baselines in some studies, while others lack qualitative assessment or generated examples.

  • SeqGAN used discriminator rewards and Monte Carlo search to train a stochastic-policy generator, outperforming maximum-likelihood estimation on Chinese-poem BLEU and expert ratings.The reported BLEU difference was statistically significant at p < 10^-6; the expert evaluation also favored SeqGAN with a statistically significant 0.54 average score.Source: D. Poetry and Literary Text Generation… · pp. 8–9
  • I2P-GAN outperformed the compared models, with a 77.2% normalized average score and a human overall rating of 7.18/10, near the ground-truth 7.37.Source: D. Poetry and Literary Text Generation… · p. 9
  • The Shakespearean prose approach [46] generated image-linked poems before style transfer; scores averaged 3.7 for content and 3.9 for both creativity and Shakespearean style.BLEU decreased as source sentences lengthened; style transfer may suffer when its dataset lacks similar words, motivating dataset expansion.Source: D. Poetry and Literary Text Generation… · p. 9
  • The maximum-likelihood-augmented discrete GAN in [47] introduced a low-variance objective using discriminator output after the corresponding log-likelihood to address discrete text generation.Source: D. Poetry and Literary Text Generation… · p. 9
  • For Poem-5 and Poem-7, the LSTM model in [48] outperformed prior work, but the study reported BLEU evaluations without qualitative analysis or sample generations.Source: D. Poetry and Literary Text Generation… · p. 10
  • RankGAN [50] outperformed SeqGAN and maximum-likelihood estimation on BLEU; human ratings were 4.6 for RankGAN, 3.4 for SeqGAN, and 6.4 for human poems.On Romeo and Juliet, RankGAN reached BLEU-2 0.91, but that application had no qualitative evaluation or generated samples.Source: D. Poetry and Literary Text Generation… · p. 10
  • Creative-GAN [51] used a discriminator reward encouraging creative tokens and outperformed existing work on perplexity, but provided no qualitative analysis or generated text samples.Source: D. Poetry and Literary Text Generation… · p. 10

IV. CHALLENGES AND FUTURE WORK RECOMMENDATIONS

The survey notes that promising GAN-based art-generation developments leave challenges to address, and frames this section around challenges and future-work recommendations for each art form.

  • Recent GAN-based art-generation developments are described as promising, but the survey identifies remaining challenges.Source: §IV · p. 12
  • The discussion considers major challenges separately for each art form.Source: §IV · p. 12
  • The section presents recommendations for future work alongside those challenges.Source: §IV · p. 12

A. Challenges

GAN-generated arts face distinct evaluation and data challenges across visual art, music, and literary text. The reviewed works lack consistent qualitative assessment, while music and text also have format, scope, and comparison constraints.

A. Challenges

  • Visual-art studies lack consistent qualitative validation, and reviewed GAN training datasets used at least 5,000 images, creating time and resource demands for new applications.The survey calls for artist assessments of creativity; gathering datasets before training can be a substantial project constraint.Source: A. Challenges · p. 12
  • Music generation must model sequential notes, while MIDI-focused work can deter non-specialists and the lack of standard evaluation makes comparisons difficult.Most reviewed works use MIDI, and human evaluation is common for melody quality.Source: A. Challenges · p. 12
  • Literary-generation research centers on poetry, with few prose or lyrics applications, and lacks consistent qualitative evaluation despite accepted BLEU and Perplexity measures.The survey recommends involving poets and writers in assessing creativity.Source: A. Challenges · p. 12

B. Recommendations and Future Work

The survey recommends addressing data and evaluation constraints in visual art, music, and literary generation. It also calls for expanding music outputs beyond MIDI and literary work beyond short forms.

B. Recommendations and Future Work

  • Test smaller datasets and GAN architectures for visual art, and standardize qualitative assessment with specialist and non-specialist feedback and adequate sample sizes.This is especially relevant because pre-modern artworks are rare and large-scale GAN architectures may not suit such data.Source: B. Recommendations and Future Work · p. 12
  • Generate music in raw audio rather than MIDI and develop quantitative quality metrics, while accounting for the computational demands of raw audio.The survey notes that MIDI makes it challenging for non-specialists to participate in improving music generation.Source: B. Recommendations and Future Work · p. 12
  • Explore longer literary works, including novels and dramas, and develop comprehensive qualitative evaluation alongside BLEU and Perplexity scores.The proposed qualitative method would support more meaningful comparison of generated text.Source: B. Recommendations and Future Work · p. 12

V. CONCLUSION

The survey reviews GANs for visual arts, music, and literary text generation, from their historical context and architectures to results, challenges, and future directions.

V. CONCLUSION

  • The survey covers computer-art history and its limitations, GAN types and loss functions, recent work across visual arts, music, and literary text, and related results, challenges, and future directions.Source: §V · p. 12
Loading 2108.03857v2…