PaperlayerView exact revision on arXiv

Source-linked AI summary

Coloring With Limited Data: Few-Shot Colorization via Memory-Augmented Networks

arXiv:1906.11888v1

TL;DR

Animation and cartoon datasets are often limited, yet deep colorization models require substantial training data. MemoPainter retrieves color information from external memory trained without labels using a threshold triplet loss. It works across varied images and consistently produces accurate, vibrant colorizations in one-shot and few-shot comparisons.

  • Problem

    Animation and cartoon datasets are often limited, yet training deep colorization models requires substantial data.

    Source: §1 · p. 1

  • Method

    MemoPainter retrieves relevant color information from external memory and trains that memory without labels using a threshold triplet loss.

    Source: §1 · p. 2

  • Results

    MemoPainter works across varied images and consistently produces accurate, vibrant colorizations in one-shot and few-shot comparisons with existing methods.

    Source: §5 · p. 8, §4.1.4 · p. 8

  • Takeaways & Limitations

    The results suggest memory-augmented colorization is promising for practical applications requiring little data, including animations and cartoons.

    Source: §5 · p. 8

    The threshold triplet loss rests on an assumption involving images with both similar spatial features and similar color features.

    Source: §4.1.3 · p. 7

Abstract

from arXiv · show

1. Introduction

MemoPainter uses external neural memory to enable one-shot or few-shot colorization and reduce the dominant color effect. Its threshold triplet loss allows memory training without labeled data.

1. Introduction

  • Scarce, hand-created animation and cartoon data conflicts with deep colorization models’ large training-data requirements, motivating colorization methods that learn from little data.Existing deep colorization models [33] [7] [11] [34] remain limited for animation and cartoon applications; many are trained on more abundant real-world photographs.Source: §1 · p. 1
  • MemoPainter enables one-shot or few-shot colorization by storing useful training-set color information in external memory and retrieving color information relevant to each input.To the authors’ knowledge, this is the first colorization network augmented with external neural memory.Source: §1 · p. 2
  • MemoPainter captures rare classes and suffers less from the dominant color effect, in which models favor frequent colors and lose class-specific color identity.Color identity means the distinctive colors separating one object class from another; dominant-color predictions can make rare side characters resemble the main character.Source: §1 · p. 2
  • A novel threshold triplet loss enables unsupervised training of the memory networks, allowing colorization without labeled data.Source: §1 · p. 2

2. Related Work

The section identifies large training-data requirements and failures on rare instances as limitations of deep colorization. It positions memory-augmented few-shot learning and adversarial generation conditioned on grayscale images and memory-derived color features as the paper’s response.

Deep Learning-Based Colorization

  • Existing deep colorization methods [33] [34] [11] require large training datasets and struggle with rare instances, motivating the paper’s previously unexplored few-shot colorization setting.Obtaining abundant animation training data is particularly expensive because professional animators must produce it.Source: §2 · p. 2
  • Prior colorization approaches emphasize rare colors [33], incorporate local and global color hints [34], or use conditional adversarial networks [11].Zhang et al. [33] reweight the training loss for more vibrant results; Zhang et al. [34] enable interactive colorization at test time.Source: §2 · p. 2

Memory Networks

  • The paper claims the first augmentation of colorization networks with memory networks to enable few-shot image colorization.Earlier memory-augmented networks supported lifelong and one-shot learning, especially remembering rare events [12], and stored image information for other visual tasks.Source: §2 · p. 2

Conditional Generative Adversarial Networks

  • The method conditions its adversarial loss on a grayscale image and a color feature extracted from memory to generate realistic colored images.GANs train a discriminator to distinguish real from generated images while the generator learns to fool it.Source: §2 · p. 2

3. Proposed Method

MemoPainter augments colorization networks with memory to retain rare instances when data are limited. Threshold triplet loss enables memory training without class labels, while adaptive instance normalization (AdaIN) [9] injects color information.

3.1. Memory Networks

  • Memory retrieves the color feature associated with the most cosine-similar spatial key to condition the colorization network.Queries use ImageNet -pretrained ResNet18-pool5 features [8], followed by a learned linear transformation and normalization. When retrieved and target colors match within a threshold, memory averages and normalizes the key with the query; otherwise, it replaces a randomly chosen oldest slot with the query–color pair, resetting the affected age.Source: §3.1 · pp. 3–5
  • 313-bin Lab color distributions suit diverse colors and intricate drawings, while ten dominant RGB colors work better for one-shot colorization.Distributions quantize ab values using the parametrization in [33]; Color Thief extracts the dominant RGB values.Source: §3.1 · p. 4
  • Threshold triplet loss trains memory without class labels by selecting positive and negative neighbors using a color-distance threshold.Positive neighbors fall within threshold δ and negative neighbors exceed it, pulling queries toward positive keys and away from negative keys. Color distance uses symmetric KL divergence for distributions and CIEDE2000 for RGB features converted to Lab.Source: §3.1 · p. 4

3.2. Colorization Networks

  • AdaIN [9] injects color into a conditional GAN, producing more vivid colorizations than element-wise color conditioning in existing models [2] [34].A multilayer perceptron adapted from [10] predicts scaling and shifting parameters for normalized activations from the color feature. Training uses ground-truth color features with adversarial and smooth L1 losses; testing conditions the generator on memory-retrieved colors.Source: §3 · p. 3, §3.2 · p. 5

4. Experiments

MemoPainter outperforms the compared methods in user-study and LPIPS evaluations and produces accurate, vibrant colorization with few-shot and one-shot training. Memory ablations and retrieval analyses support its memory augmentation and threshold triplet loss, which requires consistent colors within classes.

4.1.1 Datasets

  • Five datasets cover abundant-data, few-shot, and one-shot colorization settings.Superheroes contains seven categories with fewer than five images per category. Pokemon contains 819 classes with one image per class, with additional internet images collected for testing.Source: §4.1.1 · pp. 5–6

4.1.4 Qualitative Comparisons

  • MemoPainter alone consistently produces accurate, vibrant few-shot and one-shot colorizations, outperforming Deep Priors [34], CIC [33], and Pix2pix [11] on rare instances.It preserves a normally green character’s rare pink-painted appearance and remembers minor details such as clothing, objects, and backgrounds.Source: Figure 6 · p. 6, §4.1.4 · p. 8

4.1.2 Analysis on Memory Networks

  • Memory augmentation improves colorization across cartoons and real-world images, particularly when only one or a few examples are available.Without memory, Res-cGAN loses rare instances’ ground-truth colors or fails entirely in one-shot settings such as Pokemon.Source: §4.1.2 · p. 7, Figure 7 · p. 7

4.1.3 Analysis on Threshold Triplet Loss (TTL)

  • Threshold triplet loss uses spatial similarity and a color-distance threshold to retrieve content-relevant color features, including instances seen only once.Its assumption is that spatially similar images with sufficiently similar colors likely share a class. The illustrated top-three retrieved features correspond to images sharing the query’s class and finer contextual details.Source: §4.1.3 · pp. 7–8, Figure 8 · p. 7

4.1.2 Analysis on Memory Networks

  • LPIPS remains stable across broad memory-size and color-threshold ranges, with quality degrading only at excessively small or large settings.The analysis finds no overfitting to a particular memory size or color threshold.Source: §4.1.2 · p. 7, Figure 9 · p. 8

4.1.5 Quantitative Evaluation

  • MemoPainter outperforms the compared state-of-the-art methods on both user-study judgments and LPIPS perceptual distance.The user study involved 30 participants answering 40 questions each, judging output quality while maintaining the source image’s color identity.Source: Table 1 · p. 8, §4.1.5 · p. 8

4.1.3 Analysis on Threshold Triplet Loss (TTL)

  • Unsupervised TTL matches supervised memory retrieval’s class accuracy across varying class counts and training sets, but requires consistent class colors rather than arbitrary colors.The supervised upper bound stores class labels in value memory and updates memory using the lifelong memory module for few-shot classification [12].Source: Table 2 · p. 8, §4.1.3 · p. 8

5. Conclusions

The paper concludes that memory-augmented colorization is promising for practical applications, emphasizing few-shot operation and MemoPainter’s applicability to a wide variety of images.

5. Conclusions

  • MemoPainter works on a wide variety of images, suggesting potential for applications requiring few-shot colorization.Source: §5 · p. 8
  • The conclusion emphasizes working with little data so that colorization models can be used to color animations and cartoons.Source: §5 · p. 8
  • The paper’s results suggest that combining colorization networks with memory networks is a promising approach for practical colorization applications.Source: §5 · p. 8

Related papers

Loading 1906.11888v1…