Source-linked AI summary
Coloring With Limited Data: Few-Shot Colorization via Memory-Augmented Networks
TL;DR
Deep colorization models remain limited for real-world animation and cartoon applications because relevant data are scarce and dominant colors can obscure color identity. MemoPainter addresses this with external memory networks that store and retrieve useful color information, and the paper reports accurate colorization with extremely limited data across varied images.
Problem
Deep colorization models remain difficult to apply to animations and cartoons because training data are limited and dominant colors can obscure distinctive object colors.
Method
MemoPainter augments colorization networks with external neural memory that stores and retrieves color information, trained using an unsupervised threshold triplet loss.
Results
MemoPainter produces accurate colorization with extremely limited data, including fewer than five training images or one item per class, across varied images.
Takeaways & Limitations
Memory-augmented colorization is presented as a promising approach for practical applications requiring few-shot colorization, including animations and cartoons.
Takeaways & Limitations
The threshold triplet loss assumes that images with similar spatial features also have similar color features.
Abstract
from arXiv · showhide
1. Introduction
Deep colorization models remain impractical for animations and cartoons because training data are limited and dominant colors suppress rare instances. MemoPainter addresses these problems with external memory networks for few-shot colorization, rare-class capture, and unsupervised memory training.
- Motivation: MemoPainter targets animation and cartoon colorization, where limited training data make existing deep colorization models difficult to apply in practice.Cartoon images are laborious to create, whereas real-world images are easier to obtain and commonly used for training.
- Motivation: Existing colorization models favor frequent colors, causing rare side characters to be ignored and similarly colored to dominant main characters.This dominant color effect can minimize overall loss but produces unsatisfactory results when stories contain diverse characters.
- Contributions: MemoPainter uses external neural memory networks to extract, store, and query relevant color information, enabling one-shot or few-shot colorization.The authors present this as the first colorization network augmented by external neural memory networks, to their knowledge.
- Contributions: MemoPainter captures rare classes and suffers less from the dominant color effect than previous methods.This capability is intended to preserve diverse character colors that existing models tend to overlook.
- Contributions: A novel threshold triplet loss trains the memory networks unsupervised, so labeled data are unnecessary for successful image colorization.The loss is introduced specifically to enable unsupervised memory-network training.
2. Related Work
Prior work improves colorization with deep networks, external memory, and conditional GANs. MemoPainter combines memory and colorization networks to retrieve and inject a matching color feature into grayscale images during generation.
- Deep Learning-Based Colorization: Deep colorization methods [33] [34] [11] use neural networks to improve results through rare-color reweighting, color hints, and conditional generation.Zhang et al. [33] emphasize rare colors for more vibrant results, while Zhang et al. [34] use local and global hints for interactive test-time colorization.
- Memory Networks: Memory networks augment neural networks with external storage for algorithmic problems [6], question answering [28] [15] [17], and lifelong or one-shot learning [12].Related approaches also store image data for tasks including image captioning, summarization, and image generation.
- Conditional Generative Adversarial Networks: Conditional GANs generate samples conditioned on inputs such as classes [18] [20] [21] and text descriptions [31].GANs rely on adversarial loss, with a discriminator distinguishing real from fake images and a generator producing realistic samples.
- Proposed Model: MemoPainter retrieves a top-1 matching color feature from memory and supplies it as a condition to a trained generator.During training, memory networks learn feature retrieval while colorization networks learn to inject the retrieved feature into the grayscale target.
3. Proposed Method
MemoPainter combines memory networks with conditional colorization networks to retrieve rare color features and improve colorization with limited data. Its memory uses spatial keys, color values, and age-based updates, while threshold triplet loss enables unsupervised memory training and AdaIN conditions vivid generation.
- 3. Proposed Method: MemoPainter augments colorization networks with memory networks that remember rare instances for high-quality colorization with limited data.The model introduces a new threshold triplet loss for unsupervised memory-network training.
- Memory Networks: Memory stores spatial feature keys, color-feature values, and item ages; queries retrieve the nearest value by cosine similarity for conditioning the colorization networks.Queries are extracted from ResNet18-pool5 features [8] pretrained on ImageNet, transformed by a learnable linear layer, and normalized.
- Color Features: Value memory represents color either as 313-bin color distributions [33] or ten dominant RGB values, with direct RGB features working better for one-shot colorization.The RGB representation is extracted with Color Thief, whereas distributions are computed in CIE Lab space.
- Threshold Triplet Loss for Unsupervised Training: Threshold triplet loss replaces unavailable class labels by treating color features within threshold δ as positive neighbors and farther features as negative neighbors.It minimizes query-to-positive-key distance while maximizing query-to-negative-key distance; supervised alternatives [12, 26] require labels that are often unavailable for colorization data.
- Memory Update and Objective Function: The system averages matching retrieved keys and resets their age, or replaces the least-recently-used slot when no stored color matches the new query.During training, ground-truth color features condition the generator and discriminator; at test time, retrieved memory colors condition the generator, whose AdaIN conditioning produces vivid colorizations [2] [34].
4. Experiments
Experiments evaluate MemoPainter across abundant-, few-, and one-shot settings using ablations, threshold-triplet-loss analysis, and comparisons with three baseline models. MemoPainter consistently handles rare instances and extremely limited data, while memory retrieval remains effective across diverse datasets and hyperparameters.
- Memory-network ablation: Memory augmentation improves colorization across diverse cartoon and real-world datasets, especially in few-shot settings, enabling accurate results for single or rare training instances.Figure 7 compares MemoPainter with Res-cGAN without memory networks; the memory-augmented model retrieves relevant color features and handles rare instances presented once during training.
- Hyperparameter analysis: LPIPS remains stable across a wide range of memory sizes and color thresholds, with quality declining only for excessively small or large hyperparameters.This indicates the model does not overfit to a particular memory size or color threshold.
- Threshold triplet loss: Threshold triplet loss retrieves top-ranked color features whose corresponding images share the query image’s class, supporting content-relevant color retrieval.The assumption combines similar spatial features with similar color features within a threshold; classification accuracy evaluates agreement between the top-1 memory slot and the query label.
- Baseline comparisons: MemoPainter consistently produces accurate, vibrant colorization with fewer than five or only one training image per class, outperforming Deep Priors [34], CIC [33], and Pix2pix [11].The user study and LPIPS comparisons in Table 1 report superiority of the proposed method, while qualitative results cover multiple datasets.
5. Conclusions
The results identify memory-augmented colorization networks as promising for practical few-shot colorization, particularly when limited data is required for animations, cartoons, and diverse image applications.
- Memory-augmented colorization networks show promise for practical colorization applications.
- MemoPainter addresses the need for colorization models that work with little data, supporting animation and cartoon coloring.
- MemoPainter works across a wide variety of images, suggesting potential for applications requiring few-shot colorization.