Source-linked AI summary
SketchyGAN: Towards Diverse and Realistic Sketch to Image Synthesis
Wengling Chen, James Hays
TL;DR
Sketch-to-image synthesis must handle sparse, imperfect sketches despite limited paired training data and the difficulty of supporting diverse categories. SketchyGAN uses an end-to-end GAN with automatically augmented edge-map data, gradual training adaptation, and a Masked Residual Unit that repeatedly conditions the network on the input sketch. It generates plausible objects across 50 categories and reports higher quality than existing deep generative models, while remaining generally non-photorealistic and not always faithful to sketch intent.
Problem
Sketchy image synthesis is challenging because sketches are sparse and imperfect, paired sketch-photo data are limited, and prior neural methods did not support diverse categories.
Method
SketchyGAN combines an end-to-end conditional GAN with automatic edge-map data augmentation, gradual edge-to-sketch training adaptation, and Masked Residual Units that repeatedly use the input sketch.
Results
SketchyGAN generates plausible objects from 50 categories and demonstrates increased quality over existing deep generative models.
Takeaways & Limitations
The framework supports sketch-to-image synthesis across diverse categories without image retrieval at test time.
Takeaways & Limitations
Results are generally not photorealistic or high enough resolution, and the method sometimes fails to balance realism with faithfulness to sketch intent.
Abstract
from arXiv · showhide
Synthesizing realistic images from human drawn sketches is a challenging problem in computer graphics and vision. Existing approaches either need exact edge maps, or rely on retrieval of existing photographs. In this work, we propose a novel Generative Adversarial Network (GAN) approach that synthesizes plausible images from 50 categories including motorcycles, horses and couches. We demonstrate a data augmentation technique for sketches which is fully automatic, and we show that the augmented data is helpful to our task. We introduce a new network building block suitable for both the generator and discriminator which improves the information flow by injecting the input image at multiple scales. Compared to state-of-the-art image translation methods, our approach generates more realistic images and achieves significantly higher Inception Scores.
1. Introduction
SketchyGAN addresses the difficulty of synthesizing realistic images from sparse, imperfect sketches by combining a GAN, automatic data augmentation, and a new network building block. It generates plausible objects across 50 categories without retrieval at test time, while improving quality over existing deep generative models.
- Motivation: Sketch-to-image synthesis is difficult because novice sketches are sparse, imperfect, and often fail to precisely reflect object boundaries.Realistic outputs should preserve artist intent while potentially deviating from coarse strokes to remain on the natural image manifold.
- Approach: SketchyGAN is an end-to-end GAN-based approach that generates realistic images containing objects in poses similar to input sketches across 50 classes.The task is challenging because paired sketch-photo data are difficult to acquire and prior neural methods generally covered only one or a few categories.
- Approach: The method augments the Sketchy database with automatically synthesized edge-map and photo pairs, then gradually shifts training inputs from edge maps toward sketches.This strategy transfers knowledge from edge-image synthesis to sketch-image synthesis while addressing the lack of sufficient human-annotated training data.
- Contributions: Unlike retrieval-based methods and deep translation methods that effectively colorize input edges, SketchyGAN generates plausible objects without image retrieval at test time.The authors report increased quality compared with existing deep generative models, while noting that results are not generally photorealistic.
- Contributions: The model combines additional objective functions with a new network building block, and the authors report that removing any of these components reduces result quality.The contribution is framed as a GAN model with both loss-function additions and architectural changes.
2. Related Work
Prior work spans sketch-based retrieval, small paired-sketch datasets, GAN-based image translation, and automated edge-map processing. These approaches motivate SketchyGAN’s focus on learning mappings from imperfect sketches with broader paired data.
- Sketch-Based Image Retrieval and Synthesis: Sketch-based retrieval methods commonly use representations intended to be invariant across sketches and photos, but struggle with fine-grained retrieval and badly drawn sketch edges.Deep CNN approaches were introduced to relate sketches and photos more directly.
- Sketch-Based Datasets: Existing human-drawn sketch datasets are often small or limited in categories, while Sketchy contains 75,471 sketches of 12,500 objects across 125 categories.The paper selects Sketchy because it is described as the only substantial paired sketch-photograph dataset spanning diverse categories.
- Image-to-Image Translation with GANs: GAN-based image translation uses a discriminator to distinguish unrealistic images from real ones, encouraging sharper generated outputs than direct pixel reconstruction.The related work positions pix2pix and later translation methods as relevant baselines for conditional image synthesis.
- Edge-Map Processing: The edge-map creation pipeline applies edge detection, binarization, thinning, cleanup, erosion, and spur removal to produce sketch-like edge maps.The cited figure caption states that intermediate steps remove artifacts and make edge maps more sketch-like.
- Image-to-Image Translation with GANs: Unsupervised image translation methods such as CycleGAN extend the translation framework by enforcing cycle-consistency losses.This work is presented alongside coupled GAN and variational-autoencoder approaches in the related-work discussion.
3. Sketchy Database Augmentation
SketchyGAN augments the limited Sketchy database with filtered Flickr images and automatically synthesized, sketch-like edge maps, then gradually shifts training from edge-map pairs to sketch pairs. The network uses multi-scale inputs through MRU blocks, and this gradual schedule outperforms discrete edge pre-training followed by sketch fine-tuning.
- 3. Sketchy Database Augmentation: The augmented dataset combines Flickr photos with machine-generated edge maps to supplement the Sketchy database.The paper presents this augmentation as a way to address limited paired sketch-photo training data.
- 3.2. Data Collection: The augmentation pipeline collects and filters Flickr images across 50 categories before synthesizing edge maps as sketch approximations.Filtering uses category-specific classifiers and object detection, while edge maps undergo binarization, thinning, component removal, erosion, and spur removal.
- 3.3. Edge Map Creation: MRU blocks let both the generator and discriminator accept multi-scale inputs.This multi-scale design is part of the complete network structure.
- 3.4. Training Adaptation from Edges to Sketches: Training gradually increases sketch-image pairs while reducing edge-map-image pairs, avoiding separate edge-map pre-training.The sketch proportion grows from 0.1 to 0.9 using λ = 1 in the experiments.
- 3.4. Training Adaptation from Edges to Sketches: 6.73 vs 7.90 Inception Scores: discrete edge pre-training followed by sketch fine-tuning performs worse than the gradual transition.The comparison is reported on the test set.
4. SketchyGAN
SketchyGAN is an end-to-end GAN framework that transforms sketches into images, using masked residual units, augmented training data, conditional supervision, and diversity-oriented losses. It compares favorably with pix2pix variants and alternative generator architectures while producing more focused and diverse outputs.
- Framework: SketchyGAN transforms input sketches into output images with a generator and discriminator in an end-to-end GAN framework.The generator maps x to y, while the discriminator distinguishes generated from real images.
- Masked Residual Unit: Masked Residual Units repeatedly condition convolutional feature maps on the input image through learned masks and residual connections.The same image can be injected at different scales along the computation path.
- Results: Compared with pix2pix variants, SketchyGAN produces substantially higher test-image scores, while MRU significantly outperforms ResNet under similar parameter counts.Visual comparisons report that pix2pix outputs are blurry and noisy, whereas MRU emphasizes the main object more than DCGAN, CRN, and ResNet.
- Network Structure: The generator uses an encoder-decoder built from MRU blocks, feeds resized sketches into each block, and can concatenate encoder outputs with corresponding decoder outputs.The discriminator is also built with MRU blocks.
- Objective Function: Conditional instance normalization and discriminator class prediction incorporate sketch labels into the generator and discriminator.The generator maximizes the same auxiliary classification log-likelihood with the discriminator fixed.
- Objective Function: L1 supervision is supplemented with perceptual and diversity losses because directly minimizing L1 discourages diverse outputs.Perceptual loss uses intermediate Inception-V4 responses, while Gaussian noise is concatenated at the generator bottleneck.
5. Experiments
Experiments evaluate SketchyGAN against pix2pix variants, component ablations, alternative network structures, and human judgments of realism and faithfulness. SketchyGAN produces sharper, more realistic outputs, while balancing sketch faithfulness against necessary deviations.
- Experimental Setup: The experiments use Sketchy test-set evaluation, 64×64 images, Inception Scores, and comparisons across pix2pix models trained with Sketchy or augmented data.
- Comparison to Baselines: SketchyGAN generates sharper images, more accurate object colors, richer textures, meaningful backgrounds, and more colorful scenes than the pix2pix baselines.
- Comparison to Baselines: Edge-map augmentation improves pix2pix outputs from unidentifiable color patches toward the general object shape, although results remain blurry and artifact-prone.
- Component Analysis: Removing GAN, auxiliary, L1, or perceptual losses substantially harms quality, while removing diversity loss causes a smaller decrease.
- Component Analysis: MRU models outperform ResNet, CRN, and DCGAN structures in visual quality and Inception Scores, including when using only standard GAN losses.
- Human Evaluation of Realism and Faithfulness: SketchyGAN is more faithful than retrieval but less faithful than pix2pix, while participants judge it more realistic than pix2pix variants.
6. Conclusion
The paper presents SketchyGAN as a promising approach to sketch-to-image synthesis, supported by automatic data augmentation and a network structure that improves over existing structures. Its outputs remain limited in photorealism, resolution, and consistent faithfulness to sketch intent.
- SketchyGAN combines data augmentation with a GAN framework to synthesize diverse, more realistic images from sketches.
- The proposed network structure performs better than existing structures in the generative task.
- Results generally are not photorealistic or high enough resolution, and many sketches fail to achieve both realism and faithfulness.
Supplementary Material Outline
The supplementary material lists the training categories, compares MRU with other models on CIFAR-10, and shows generated samples across all 50 categories.
- The supplementary material covers category lists, MRU comparisons on CIFAR-10, and generated images from all 50 categories.
1. Category list
The models are trained and tested across 50 object categories spanning animals, everyday objects, foods, vehicles, and landmarks.
- The category list contains 50 classes, including airplane, cat, couch, horse, motorcycle, pizza, tiger, volcano, and zebra.
2. Evaluation of MRU on CIFAR-10
The MRU is evaluated against recent architectures on CIFAR-10, with performance reported through error rates where lower values are better.
- MRU performance on CIFAR-10 is reported as comparable to ResNet.The comparison includes an improved ResNet variant called “ResNet-v2.”
- The evaluation also tests an MRU-108 variant using LeakyReLU gates instead of sigmoid activations.The resulting masks are normalized to the range [0, 1].
- The table compares error rates across CIFAR-10 models, with lower error rates indicating better performance.
3. Samples from all 50 categories
The paper presents samples from all 50 categories, comparing multiple pix2pix variants and SketchyGAN outputs against input sketches and ground-truth images.
- Each category contains three input samples, with the third sample serving as a failure case for the proposed method.
- The comparison includes pix2pix on Sketchy, pix2pix on Augmented Sketchy, Label-supervised pix2pix, the proposed method, and ground-truth images.
- The figures organize six columns as the input sketch, three baselines or methods, and the ground-truth image.