PaperlayerView exact revision on arXiv

Source-linked AI summary

Reference-Based Sketch Image Colorization using Augmented-Self Reference and Dense Semantic Correspondence

Junsoo Lee, Eungyeup Kim, Yunsung Lee, Dongjun Kim, Jaehyuk Chang, Jaegul Choo

arXiv:2005.05207v1cs.CV

TL;DR

Sketch colorization is difficult because sketches are information-scarce, while reference-guided training lacks reliable semantic pairs and correspondence labels. The paper generates geometrically distorted, color-perturbed self-references and uses their dense correspondence to supervise attention-based color transfer. Qualitative and quantitative evaluations report state-of-the-art performance across diverse datasets and introduce SC-PSNR for measuring faithful reference-color transfer.

  • Problem

    Reference-based sketch colorization lacks sufficient semantically meaningful training pairs and reliable dense correspondence labels, while sketches provide scarce visual information.

  • Method

    The method generates augmented-self references from the original image and uses spatially corresponding feature transfer with correspondence supervision from geometric transformation weights.

  • Results

    Qualitative and quantitative evaluations indicate state-of-the-art performance for reference-based sketch colorization across diverse datasets.

  • Takeaways & Limitations

    SC-PSNR measures whether colors are faithfully transferred from the reference into corresponding sketch regions.

Abstract

from arXiv · show

This paper tackles the automatic colorization task of a sketch image given an already-colored reference image. Colorizing a sketch image is in high demand in comics, animation, and other content creation applications, but it suffers from information scarcity of a sketch image. To address this, a reference image can render the colorization process in a reliable and user-driven manner. However, it is difficult to prepare for a training data set that has a sufficient amount of semantically meaningful pairs of images as well as the ground truth for a colored image reflecting a given reference (e.g., coloring a sketch of an originally blue car given a reference green car). To tackle this challenge, we propose to utilize the identical image with geometric distortion as a virtual reference, which makes it possible to secure the ground truth for a colored output image. Furthermore, it naturally provides the ground truth for dense semantic correspondence, which we utilize in our internal attention mechanism for color transfer from reference to sketch input. We demonstrate the effectiveness of our approach in various types of sketch image colorization via quantitative as well as qualitative evaluation against existing methods.

1. Introduction

Reference-based sketch colorization addresses the information scarcity of sketches but remains difficult because models must align semantically corresponding regions without sufficient labeled training pairs. The paper uses augmented-self references to provide correspondence supervision and reports state-of-the-art performance.

  • Motivation: Sketches contain less information than grayscale images, making their colorization intrinsically challenging.Existing approaches impose additional conditions through user hints or reference images.
  • Motivation: Reference-guided colorization requires visual correspondence between the sketch and colored reference, but labeled correspondence datasets are scarce and costly to create.The information discrepancy between sketches and references further limits this task.
  • Approach: The method generates an augmented-self reference through color perturbation and geometric distortion of the original image.Because the sketch and reference derive from the same image, the construction provides correspondence information and colored-output ground truth.
  • Approach: A spatially corresponding feature transfer module moves contextual reference representations into corresponding sketch positions using attention-based pixel-wise transfer.Similarity-based triplet supervision uses the resulting ground-truth correspondence without manually annotated source-reference labels.
  • Results: Both qualitative and quantitative experiments indicate state-of-the-art performance for reference-based colorization of information-scarce sketches.The paper evaluates the approach across the target task and presents its practical potential across domains.

2. Related work

Sketch-based colorization is challenging because sketches lack texture and color information. Reference-based methods reduce the need for direct user color input but still face correspondence and domain-sensitivity challenges, motivating self-supervised augmented references.

  • Sketch-based tasks: Sketches are useful because they are simple and easy to draw and edit, but their lack of texture and color makes colorization difficult.Reference-based sketch colorization remains comparatively under-explored.
  • Conditional colorization: User-conditioned colorization methods use scribbles, palettes, or text tags but require precise color information and geometric hints.These requirements can make user control inconvenient.
  • Reference-based colorization: Reference-based methods use an already-colored image, but the absence of input-level geometric correspondence requires visual matching.Earlier work used hand-crafted features, whereas later work formed semantically close pairs with pretrained features or color histograms.
  • Proposed direction: The paper learns visual correspondence by generating augmented-self references during training in a self-supervised manner.It demonstrates scalability across various dataset types.

3. Proposed method

The proposed method trains sketch colorization with an augmented-self reference and SCFT, which transfers semantically corresponding reference features into the sketch. It also uses correspondence supervision derived from geometric transformations to guide reference-to-sketch color transfer.

  • Overall workflow: The model converts a color image into a sketch, generates an augmented-self reference, encodes both inputs, and decodes transferred features into a colored output.The reference is produced from the same original image using appearance and spatial transformations, while the sketch is obtained with an outline extractor.
  • Spatially Corresponding Feature Transfer: SCFT computes dense correspondences between reference and sketch regions through attention, then transfers reference context into semantically matching sketch positions.The transferred context is fused with sketch features before residual processing and U-Net decoding.
  • Attention-based transfer: Attention weights are computed from query-key similarities, determining how much information each sketch position receives from each reference position.The value projection contains color features from semantically related reference regions, and the resulting context vector enriches the sketch feature before decoding.
  • Feature representation: The method aggregates multi-level encoder activations by spatially downsampling them to a common size and concatenating their channels.This combines low- to high-level features into the representations used for correspondence computation.
  • Objective functions: Geometric transformations expose pixel-level correspondence weights that supervise a similarity-based triplet loss for matching reference and sketch features.The loss pulls query representations toward positive keys and separates them from negative samples; unlike L1 loss, it avoids penalizing semantically close but spatially distant pairs.
  • Objective functions: Additional objectives include reconstruction, conditional adversarial, and perceptual losses to constrain color accuracy, realism, content preservation, and perceptual plausibility.The adversarial loss conditions discrimination on the input sketch, while perceptual loss compares multilayer activations of generated and ground-truth images.

4. Experiments

The experiments evaluate the method across real-image, face, anime, and sketch domains using qualitative comparisons, FID, SC-PSNR, user studies, and loss-function and attention analyses. Across these evaluations, the method transfers reference colors to corresponding sketch regions, achieves favorable quantitative comparisons, and benefits from its loss and attention designs.

  • Experimental setup: Experiments cover real images, human faces, anime, and sketch domains, with comparisons against colorization, exemplar-guided translation, and style-transfer baselines.The evaluation uses qualitative and quantitative comparisons across multiple datasets.
  • Datasets: The Tag2pix dataset contributes 54,317 training images and 6,036 test images for comic-domain training, while SPair-71k evaluates SC-PSNR on cat, dog, and car categories.SPair-71k is used only for evaluation, whereas Tag2pix is used to train the model.
  • Qualitative comparisons: The method transfers reference colors into corresponding sketch positions across five datasets, including real-image and sketch-image domains.Qualitative results report exact color transfer from references, including facial regions and shoe interiors and soles.
  • Quantitative comparisons: The method outperforms existing baselines on FID across seven datasets and on SC-PSNR over cat, dog, and car datasets.The reported FID comparison emphasizes realistic and diverse outputs, while SC-PSNR evaluates correspondence-aware color transfer.
  • Human evaluation: User studies assign the method a large share of Top1 and Top2 votes, indicating stronger reference-color reflection and more realistic outputs than the baselines.The reported percentages are averaged across the experimented datasets.
  • Analysis: Ablations show that removing adversarial or triplet loss produces unrealistic or incorrectly colorized regions, while attention maps retrieve relevant reference regions for sketch pixels.For example, the attention mechanism retrieves highly attentive reference regions for a queried left-eye region and supports blue-eye colorization.

5. Conclusions

The paper presents a training scheme combining augmented-self references with attention-based feature transfer to learn semantic correspondence for reference-based sketch colorization. SCFT achieves state-of-the-art performance across diverse datasets and SC-PSNR measures faithful exemplar-style reflection.

  • The proposed training scheme integrates augmented-self references and an attention-based feature transfer module to directly learn semantic correspondence.
  • SCFT achieves state-of-the-art performance over diverse datasets, while SC-PSNR measures how faithfully outputs reflect an exemplar’s style.

A. Supplementary Material

The supplementary document provides additional analyses, human evaluations, and implementation details for the proposed approach. It also includes network, reference-generation, and training information.

  • Section A.1 analyzes the spatially corresponding feature transfer mechanism quantitatively, while Section A.2 reports human evaluations against baseline methods.
  • Section A.3 gives implementation details covering network architectures, augmented-self reference generation, and other training details.

A.1. Effects of Aggregation Methods

The aggregation study compares addition, AdaIN, and SCFT for combining reference and sketch features. SCFT better aligns corresponding local regions and generally produces more realistic colorizations, though it performs worse on Yumi’s Cells.

  • SCFT combines spatially aligned reference features with content features to reflect reference colors in corresponding positions.
  • Addition can mislocate corresponding regions and transfer background colors, producing an overly yellowish car in the qualitative example.
  • AdaIN ignores spatially varying color information and instead colorizes using the reference’s dominant color.
  • SCFT produces the most realistic results over most datasets by aligning local sketch and reference regions with attention matrix A.
  • On Yumi’s Cells, SCFT obtains a worse FID score than the other aggregation methods, potentially because randomly paired sketches and references contain different object types.

A.2. User Study

The user study evaluates realism and detailed reference-color reflection across datasets. The proposed approach receives over 50% rank-one votes across datasets for both criteria, with a larger advantage over Style2Paints for faithful colorization.

  • The study evaluates overall colorization quality and realism, requiring perceptually realistic outputs without artifacts or color bleeding.
  • It also evaluates whether reference colors are reflected in corresponding sketch regions part by part.
  • Our approach receives rank-one votes above 50% across all adopted datasets for both realism and faithful colorization questions.
  • Style2Paints achieves comparable realism on Comics datasets, but the gap increases when users judge faithful colorization.
  • Triplet-loss supervision yields more realistic and faithful outputs than L1-loss or no correspondence supervision.

A.3. Implementation Details

The implementation extracts sketches with XDoG after Gaussian blurring, uses specified encoder architecture conventions, and trains with Patch-GAN and LSGAN components. A qualitative comparison also shows a reference-colorization failure by Zhang et al. on sketch inputs.

  • Augmented-Self Reference Generation: Sketches are generated with XDoG after applying Gaussian blurring with σ = 0.7 to suppress superfluous edges.The blurred image is processed before outline extraction.
  • Network Architecture: Encoder architecture tables use I, O, K, P, and S for input channels, output channels, kernel size, padding, and stride.Conv denotes a convolutional layer.
  • Discriminator and Training: The discriminator adopts Patch-GAN, while the LSGAN objective is used for stable training.
  • Qualitative Comparison: Zhang et al.’s sketch-input result fails to transfer the reference eye color and spreads yellow across the face.

A.4. Comparison to Zhang et al. (2019) [47].

Compared with Zhang et al., the method differs through direct supervision of semantic correspondence and a generalized attention module. The comparison identifies these as the principal architectural distinctions between the two approaches.

  • Direct Supervision: The method directly supervises its attention module through a triplet loss, unlike Zhang et al.’s approach.
  • Attention Module: The method uses a generalized attention module as a second key difference from Zhang et al.

A.5. Colorization without reference.

Although the method primarily targets reference-guided colorization, it can also operate without reference information by training with zero-filled reference images. The reported examples cover Yumi’s Cells, Tag2pix, and CelebA, while the figures document qualitative comparisons, attention maps, and user studies across datasets.

  • Training Without Reference: The model is trained with a zero-filled reference at a 9:1 ratio to support colorization without an informative reference.The zero-filled reference provides no color guidance during inference.
  • Inference Without Reference: With a sketch and zero-filled reference, the model synthesizes an output using colors that often appear in the training set, conditioned on the sketch.
  • Dataset Examples: Qualitative no-reference examples use models trained for Yumi’s Cells, Tag2pix, and CelebA.The figure assigns these datasets to the first, middle, and last example groups, respectively.
  • Evaluation Visualizations: The evaluation figures include dataset-specific results, baseline comparisons, attention-map visualizations, and user studies of quality, reference reflection, and loss choices.The listed figures span CelebA, Tag2pix, Edges→Shoes, ImageNet, and Yumi’s Cells.
Loading 2005.05207v1…