PaperlayerView exact revision on arXiv

Source-linked AI summary

User-Guided Deep Anime Line Art Colorization with Conditional Adversarial Networks

Yuanzheng Ci, Xinzhu Ma, Zhihui Wang, Haojie Li, Zhongxuan Luo

arXiv:1808.03240v2cs.CV

TL;DR

Scribble-guided line-art colorization lacks grayscale cues, and synthetic training data can leave models struggling to generalize to authentic art. The paper combines a conditional adversarial model with a local-features network and reports better qualitative and quantitative results than state-of-the-art methods. It also provides collected datasets for training and evaluation.

  • Problem

    Scribble-guided line-art colorization must work without grayscale cues, while synthetic-sketch training can overfit and produce artifacts on authentic art.

    Source: §1 · p. 2

  • Method

    The paper combines a conditional adversarial network conditioned on pretrained local features with WGAN-GP and perceptual loss for stroke-guided anime line-art colorization.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    The approach outperforms state-of-the-art methods in both qualitative and quantitative evaluations.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The collected datasets provide quality anime illustrations and authentic line arts for training and rigorous evaluation.

    Source: §4.1 · p. 6, §5 · p. 8

Abstract

from arXiv · show

Scribble colors based line art colorization is a challenging computer vision problem since neither greyscale values nor semantic information is presented in line arts, and the lack of authentic illustration-line art training pairs also increases difficulty of model generalization. Recently, several Generative Adversarial Nets (GANs) based methods have achieved great success. They can generate colorized illustrations conditioned on given line art and color hints. However, these methods fail to capture the authentic illustration distributions and are hence perceptually unsatisfying in the sense that they often lack accurate shading. To address these challenges, we propose a novel deep conditional adversarial architecture for scribble based anime line art colorization. Specifically, we integrate the conditional framework with WGAN-GP criteria as well as the perceptual loss to enable us to robustly train a deep network that makes the synthesized images more natural and real. We also introduce a local features network that is independent of synthetic data. With GANs conditioned on features from such network, we notably increase the generalization capability over "in the wild" line arts. Furthermore, we collect two datasets that provide high-quality colorful illustrations and authentic line arts for training and benchmarking. With the proposed model trained on our illustration dataset, we demonstrate that images synthesized by the presented approach are considerably more realistic and precise than alternative approaches.

1 INTRODUCTION

Line-art colorization is often manual, and existing approaches struggle with sparse stroke guidance, artifacts, and generalization to authentic art. The paper introduces a conditional adversarial approach and reports better qualitative and quantitative results than state-of-the-art stroke-based methods.

1 INTRODUCTION

  • Existing methods struggle with sparse line-art inputs, artifacts, synthetic-data overfitting, and preserving authentic illustrations’ textures and global shading.Colorization is often done manually, making it labor-intensive, particularly in animation.Source: §1 · p. 2
  • A conditional adversarial architecture uses a pretrained local-features network, WGAN-GP, perceptual loss, and simulated user interactions to colorize sparse authentic line art.The simulated interactions train the network to propagate sparse stroke colors to semantic scene elements.Source: §1 · p. 2
  • The proposed method outperforms state-of-the-art stroke-based user-guided line-art colorization methods in both qualitative and quantitative evaluations.The reported model is trained on the proposed illustration dataset with minimal augmentation.Source: §1 · p. 2
  • The paper introduces an illustration synthesis architecture and collects two datasets for quality illustration training and line-art testing.The contributions also include a deeper generator with specially designed layers to increase receptive field and network capacity.Source: §1 · p. 2

2 RELATED WORK

Prior work spans interactive color propagation, learned colorization, sketch-to-image synthesis, and GAN-based generation. The section describes the capabilities and limitations of these approaches, including challenges with arbitrary line art and GAN training stability.

2.1 User-guided colorization

  • User-guided work spans low-level propagation, learned priors [61], and reference-color transfer [14] [18]; line-art systems still struggled with arbitrary art and shading [13] [34].PaintsChainer produced pleasing results but did not capture authentic illustration distributions or accurate shading.Source: §2.1 · p. 2

2.2 Automatic colorization

  • Automatic grayscale colorizers learn image-to-color mappings [60], while sketch GANs and image-retrieval systems map sketches to realistic scenes [23] [62] [6].Some learned colorization methods can generate multiple results, but cover only a small subset of possibilities.Source: §2.2 · pp. 2–3

2.3 Generative Adversarial Networks

  • GANs support broad image synthesis [43], but vanilla training faces mode collapse and vanishing gradients [47]; WGAN-GP and side information address stability and conditional sample quality [3] [17] [37] [52].The section also notes that mismatches between training and test patterns can limit existing frameworks.Source: §2.3 · p. 3

3 PROPOSED METHOD

The method trains a deep colorizer from line art and color hints, conditioning both generator and discriminator on pretrained local features. Its objectives combine perceptual content and adversarial losses, while sparse stroke inputs are simulated for training.

3.1 Learning Framework for Colorization

  • Local features from line art condition both generator and discriminator, while color hints and line art guide the generated color image.The discriminator compares generated and real color outputs paired with the same local features; raw line art is excluded from its input.Source: Figure 2 · p. 3, §3.1 · p. 3

3.2 Loss Function

  • The generator combines perceptual content loss with a conditional WGAN-GP adversarial objective, while the discriminator uses critic and penalty terms.The perceptual loss compares generated and target CNN feature maps; the discriminator’s penalty combines gradient penalty and a drift constraint.Source: §3.2 · p. 4

3.3 Network Architecture

  • The generator uses a deep U-Net with ResNeXt blocks, dilation, and sub-pixel upsampling; the discriminator uses local features and no dilation.Figure 4 compares automatic and user-guided results on authentic line arts: selected examples show higher quality, while others exhibit failed background propagation or gridding artifacts.Source: §3.3 · pp. 4–5, Figure 3 · p. 4, Figure 4 · p. 5

3.4 User Interaction

  • Training simulates stroke hints using randomly sampled points at quarter resolution; user strokes are downsampled and thinned to reduce redundancy while preserving spatial information.The authors describe these choices as a way to cover the input space and train an effective model.Source: §3.4 · pp. 5–6

4 EXPERIMENTS

The experiments evaluate the datasets and training setup, compare colorization quality with FID and MOS, and test generalization to authentic line art. The final method outperforms reference methods in both reported metrics, while architecture and user-guided results are examined separately.

4.1 Dataset

  • The proposed datasets contain 21,930 colored anime illustrations and 2,779 authentic line arts; XDoG filtering also produces synthetic sketch–drawing pairs.XDoG parameters vary line thickness to support line-width variation.Source: §4.1 · p. 6

4.2 Experimental Settings

  • Training used 250,000 iterations, with the learning rate reduced after 125,000; data augmentation included sketch intensity scaling, random cropping, and horizontal flipping.Training used batch size 4 on one GTX 1080ti GPU.Source: §4.2 · p. 6

4.3 Qantitative Comparisons

  • The FID comparison uses automatic colorizations without hints because two PaintsChainer models were not open-sourced, preventing identical user inputs during testing.FID compares automatic results on 2,779 authentic line arts with 21,930 illustrations; the study also uses MOS to assess perceptual plausibility to human observers.Source: §4.3.1 · p. 6, §4.3 · p. 6, Table 1 · p. 6

4.3.1 Fr´echet Inception Distance (FID).

  • The final method achieves smaller FID scores than other methods and significantly higher MOS scores (p < 0.01).FID is reported for automatic colorization without color hints; MOS results are from a rating study.Source: Table 1 · p. 6, §4.3.1 · p. 7, §4.3.2 · p. 7

4.3.2 Mean opinion score (MOS) testing.

  • The MOS study asked 16 raters to score six versions of 94 line arts; the final method outperformed all reference methods in a one-tailed test at p < 0.01.Raters were not calibrated; canna performed better on MOS than FID, which the authors interpret as favoring eye-pleasing results over matching the authentic illustration distribution.Source: §4.3.2 · p. 7, Table 2 · p. 7, Figure 7 · p. 7

4.4 Analysis of the Architectures

  • The method with adversarial loss and local features performs better on authentic line art; ablations show the two components address sharpness and generalization, respectively.Without adversarial loss, outputs lack saturation; without local features, outputs are unusually colorized. All methods overfit synthetic line-art artifacts.Source: §4.4 · p. 8, Figure 8 · p. 8

4.5 Color Strokes Guided Colorization

  • The network predicts user-intended actions from learned priors, infers missing details, produces diverse results, and propagates stroke colors within object boundaries.Figure 6 shows examples colorized by a novice user.Source: Figure 6 · p. 7, §4.5 · p. 8

5 CONCLUSION

The paper presents a conditional adversarial architecture and loss for user-guided anime line-art colorization, alongside local features and two datasets. Experiments report that the approach outperforms state-of-the-art methods qualitatively and quantitatively.

5 CONCLUSION

  • The conditional adversarial architecture and loss target stroke-based user-guided anime line-art colorization.Source: §5 · p. 8
  • A local features network addresses the gap between synthetic and authentic data, while specially designed subnetworks increase capacity and receptive fields.Source: §5 · p. 8
  • Two datasets of quality anime illustrations and line arts support efficient training and rigorous evaluation.Source: §5 · p. 8
  • Experiments report that the approach outperforms state-of-the-art methods in qualitative and quantitative evaluations.Source: §5 · p. 8

Related papers

Loading 1808.03240v2…