Source-linked AI summary
Style Transfer for Anime Sketches with Enhanced Residual U-net and Auxiliary Classifier GAN
Lvmin Zhang, Yi Ji, Xin Lin
TL;DR
The paper addresses the difficulty of applying a specific painting style to anime sketches, a task for which existing style-transfer methods can fail. It combines a residual U-net with AC-GAN to generate styled paintings from sketches and style maps in a fast feed-forward process, while acknowledging limitations from the ImageNet-trained VGG and large model size.
Problem
Existing neural style transfer and sketch-colorization methods do not reliably apply a chosen painting style to an anime sketch.
Method
The method combines an enhanced residual U-net generator, two guide-decoder losses, and an auxiliary-classifier discriminator using sketch maps and style maps.
Results
The network directly generates a combination of sketch and style image through a fully feed-forward process designed for high-speed synthesis.
Takeaways & Limitations
The approach provides an automatic way to draw sketches according to user-supplied style maps.
Takeaways & Limitations
The method relies on an ImageNet photograph-classification VGG rather than a painting-specific classifier, and its deep residual network limits training batch size to no more than 4.
Abstract
from arXiv · showhide
Recently, with the revolutionary neural style transferring methods, creditable paintings can be synthesized automatically from content images and style images. However, when it comes to the task of applying a painting's style to an anime sketch, these methods will just randomly colorize sketch lines as outputs and fail in the main task: specific style tranfer. In this paper, we integrated residual U-net to apply the style to the gray-scale sketch with auxiliary classifier generative adversarial network (AC-GAN). The whole process is automatic and fast, and the results are creditable in the quality of art style as well as colorization.
1. Introduction
The paper targets style-specific painting of anime sketches, where existing style-transfer and sketch-colorization methods do not adequately combine a chosen painting style with sketch content. It proposes a fast feed-forward system built from residual U-net and AC-GAN components.
- Neural style transfer can produce strange results when applying painting styles to sketches.
- Sketch colorization tools such as Paintschainer and pix2pix can produce paintings but cannot exploit existing paintings as style references.
- The proposed feed-forward network applies a painting’s style directly to a sketch at high speed.
- An enhanced residual U-net is designed to handle paired images with unbalanced information quantity.
- The method adds two additional losses to train the residual U-net and uses an AC-GAN-derived discriminator for paintings of different styles.
2. Related Works
Prior methods either fail to transfer a specific style onto sketches or depend on input-output relationships that are too limited for sketch-to-painting translation. The paper motivates an adversarial design that distinguishes realism while handling style information.
- Neural style transfer combines content and style images through deep-feature statistics but produces improper paintings when the content image is a sketch.
- cGAN-based paired image translation is reliable when input and output contain similar amounts of information, but quality declines as their information gap increases.
- Sketch-to-painting translation is more complex than simple paired transformations because the generator must learn semantic and low-level features simultaneously.
- Paintschainer uses an unconditional discriminator and supports sketch colorization with optional color hints, but lacks a mature solution for style-specific colorization.
3. Methods
The method combines an enhanced residual U-net generator with style-conditioned adversarial discrimination to transfer painting styles onto sketches. Guide decoders supply gradients to mid-level layers, enabling the network to use an informative 1×1×4096 style hint while preserving color and style transfer objectives.
- The generator receives sketch maps and style maps, while the discriminator judges realism and classifies the corresponding style.
- 3.1. Architecture and Objective of Generator: The residual U-net adds losses to potentially lazy mid-level layers so they continue receiving stable gradients during training.Guide Decoder 1 and Guide Decoder 2 provide the additional losses.
- 3.1. Architecture and Objective of Generator: A 1×1×4096 VGG 19 fc1 hint is used because a 1×1×256 hint is insufficient for sketch painting, although the larger hint can make mid-level outputs noisy.
- 3.1. Architecture and Objective of Generator: The guide-decoder loss combines the final U-net output with intermediate decoder outputs, using recommended weights α=0.3 and β=0.9.Feeding grayscale maps to the guide decoder at the entry of the mid-level layers improves color distribution.
- 3.2. Architecture and Objective of Discriminator: The system also employs a traditional DCGAN discriminator for faster speed and larger batch sizes, then fine-tunes the networks by shifting between the two loss functions.
4. Limitations and Discussions
The network’s style-guided sketch rendering depends on VGG’s classification ability, while its deep residual architecture limits training batch size.
- Limitations and Discussions: The method relies on a pretrained VGG classifier designed for ImageNet photographs rather than paintings.The authors plan to train a painting-specific classification network to improve results.
- Limitations and Discussions: The residual network’s large number of layers restricts training batch size to no more than 4.The authors identify balancing batch size against network depth as future work.