Source-linked AI summary
Image Generation from Sketch Constraint Using Contextual GAN
Yongyi Lu, Shangzhe Wu, Yu-Wing Tai, Chi-Keung Tang
TL;DR
Badly drawn sketches can force conditional image translation to follow edges that do not match photographic objects. This paper uses a contextual GAN to generate images through joint sketch-image completion, treating sketches as weak context. It reports more realistic outputs than state-of-the-art conditional GANs on challenging sketches and comparable results on good sketches.
Problem
Hard-conditioned image translation makes outputs follow input edges, a problem when freehand sketches do not align with photographic objects.
Source: §1 · pp. 1–2Method
A contextual GAN learns a joint sketch-image distribution and generates the image by completing a joint image with the sketch as weak context.
Source: §1 · pp. 2–3, §3.1 · p. 5Results
Contextual GAN generates more realistic and natural images than state-of-the-art conditional GANs on challenging sketches and comparable results on good sketches.
Source: §1 · p. 4Takeaways & Limitations
The joint representation avoids separate sketch and image domains, allows appearance beyond strict sketch-edge alignment, and supports image-to-sketch generation with the same network.
Source: §1 · p. 3Freehand sketches do not guarantee identity-preserving face generation, and some input attributes such as glasses or beards may be missed.
Source: §5.4 · p. 13
Abstract
from arXiv · showhide
In this paper we investigate image generation guided by hand sketch. When the input sketch is badly drawn, the output of common image-to-image translation follows the input edges due to the hard condition imposed by the translation process. Instead, we propose to use sketch as weak constraint, where the output edges do not necessarily follow the input edges. We address this problem using a novel joint image completion approach, where the sketch provides the image context for completing, or generating the output image. We train a generated adversarial network, i.e, contextual GAN to learn the joint distribution of sketch and the corresponding image by using joint images. Our contextual GAN has several advantages. First, the simple joint image representation allows for simple and effective learning of joint distribution in the same image-sketch space, which avoids complicated issues in cross-domain learning. Second, while the output is related to its input overall, the generated features exhibit more freedom in appearance and do not strictly align with the input features as previous conditional GANs do. Third, from the joint image's point of view, image and sketch are of no difference, thus exactly the same deep joint image completion network can be used for image-to-sketch generation. Experiments evaluated on three different datasets show that our contextual GAN can generate more realistic images than state-of-the-art conditional GANs on challenging inputs and generalize well on common categories.
1 Introduction
Hard-conditioned image translation can force generated edges to follow badly drawn sketches, while realistic images may require some deviation from sketch shape. The paper instead treats the sketch as weak context for joint image completion, allowing generated images to remain faithful without strict alignment.
1 Introduction
- Hard pixel-wise conditions make translated images follow input edges, which can prevent realistic results when freehand sketches and photographic objects do not align.The goal is to respect sparse sketch content while allowing shape deviations that may be needed for realism.Source: §1 · pp. 1–2
- The method poses sketch-to-image generation as joint image completion, using the sketch as context for generating the entire missing image rather than as a hard translation condition.The joint representation places the sketch and corresponding image side-by-side.Source: §1 · pp. 2–3, Figure 2 · p. 2
- A GAN learns the joint sketch-image distribution, and a modified objective combines contextual and traditional GAN losses to map the corrupted joint image to a nearby uncorrupted latent representation.The resulting latent encoding is fed to the generator to produce the image portion.Source: §1 · p. 3
- Joint sketch-image learning uses one network rather than separate domains, permits appearance and shape variation beyond sketch edges, and supports reversing the generation direction with the same network.From the joint-image perspective, sketch and image can be swapped as completion context.Source: §1 · p. 3
- The contextual GAN generates more realistic images than state-of-the-art conditional GANs on challenging sketches while producing comparable results on good sketches.The figure compares CE [17], pix2pix [9], and the proposed method on freehand bird and car sketches.Source: Figure 1 · p. 2, §1 · p. 4
2 Related Work
Prior work uses image completion, retrieval, and cross-domain embeddings to relate images to sketches. This paper treats sketch and image as parts of one joint image rather than separate domains or a retrieval pair.
2 Related Work
- Image-completion methods use the uncorrupted region as context to reconstruct cropped image regions, while another completion approach uses autoencoders with reconstruction and adversarial losses.The paper analogizes this setup to using the entire sketch as context for generating the natural-image portion of a joint image.Source: §2 · p. 4
- Sketch retrieval and embedding methods use sketch-photo collections or separate networks to learn cross-domain representations,, [4], whereas this approach treats sketch and image as one joint image.The paper states that this avoids requiring sketch-photo collections or treating sketch and image as separate domains.Source: §2 · p. 4
3 Approach
The approach treats sketch-guided generation as joint image completion: the sketch supplies context, while the generated image can depart from its edges. A GAN learns sketch-image correspondence in a shared representation, and completion searches the generator’s latent space for a fitting output.
3 Approach
- Sketch-guided generation is framed as image completion, with the sketch serving as weak context rather than a hard pixel-aligned condition.Source: §3 · p. 5
3.1 Joint Sketch-Image Representation
- Joint images spatially concatenate aligned sketches and photographs, letting a GAN learn their correspondence and constrain output indirectly through the sketch embedding.Source: §3.1 · p. 5
3.2 Objective Function
- The latent objective combines KL-based sketch similarity with adversarial perceptual loss, weighted by λ to constrain generated appearance.Source: §3.2 · p. 6
3.3 Contextual GAN
- During completion, back-propagation updates the input latent vector—not generator or discriminator weights—to find a fitting joint image, then the generator fills the masked image region.Initialization selects the sampled sketch with the lowest KL-divergence to the input sketch.Source: §3.3 · p. 7
Network Architecture
- The generator upsamples a 100-D noise vector to a 64 × 128 image, while the discriminator downsamples image features through four convolutional layers.Source: Network Architecture · pp. 7–8
3.4 Network Generalization
- The model is pretrained on XDoG sketches and finetuned on other sketch styles; the authors report that style augmentation improves generalization while allowing appearance freedom.The additional styles include photocopy, simplification, and FDoG sketches.Source: §3.4 · pp. 8–9
4 Datasets and Implementation
Experiments use face, bird, and car categories, with sketches generated from image datasets because available sketch datasets were limited. Preprocessing and training vary by category, followed by category-specific pretraining and style finetuning.
4 Datasets and Implementation
- The study tests face, bird, and car categories using images from CelebA, Caltech-UCSD Birds-200-2011, and Stanford Cars, respectively.Source: §4 · p. 9
4.1 Data Preprocessing
- Preprocessing yields 1,200K face pairs from two landmark maps and three sketch styles, around 100K augmented bird pairs, and car pairs from 16K images with one sketch style.Bird images are cropped around annotated bounding boxes, randomly cropped four times per image, and horizontally flipped.Source: §4.1 · p. 9
4.2 Implementation
- Each category model is trained with Adam for 200 epochs, then other sketch styles are finetuned sequentially at a lower learning rate; completion optimizes the input latent vector with fixed generator and discriminator weights.Training uses batch size 64 and learning rate 0.0002; completion typically stabilizes after 100 iterations.Source: §4.2 · p. 9
5 Results
Results cover sketch-to-image generation across CelebA, CUB, and car images, including free-hand sketches not used in training. The experiments also examine quantitative metrics, reverse generation, and limitations.
5.1 Benchmark Datasets
- The benchmark strategy trains first on XDoG sketches, then fine-tunes on Photocopy and simplification styles; this outperforms training on all styles together.The authors propose that semantically correct pretrained models may generalize to arbitrary sketch styles.Source: §5.1 · p. 10
CelebA
- On CelebA, results across three sketch styles retain important sketch details while allowing more appearance freedom than pixel-to-pixel generation.The comparison is with a method whose outputs strictly follow the input sketch’s edge profile.Source: CelebA · p. 10, Figure 7 · p. 11
- For CUB birds, the method is evaluated on images with more texture, using relative total variation smoothing to separate sketch structure from texture before forming joint images.Results are shown across XDoG, Photocopy, and simplified sketch styles.Source: CelebA · p. 10, Figure 8 · p. 11
- On cars, identical input sketches can yield different shapes and colors while still constraining generated features without requiring strict edge alignment.The dataset also poses challenges from cluttered backgrounds and varied car poses.Source: CelebA · p. 10, Figure 9 · p. 11
Comparison on Hand-Drawn Sketches
- Without fine-tuning on free-hand sketches, our method produces natural-looking outputs from ugly or abstract inputs, while pix2pix follows their shapes and CE performs worse on cars.The comparison uses the same style model and default settings across methods.Source: Figure 10 · p. 12, Comparison on Hand-Drawn Sketches · p. 12
5.2 Quantitative Evaluation
- On typical CelebA sketches, our method matches pix2pix on SSIM and exceeds it in face-verification accuracy, while both measures exceed CE’s results.SSIM may not capture human perception for badly drawn sketches.Source: Table 1 · p. 13, §5.2 · p. 13
5.3 Bi-directional Generation
- The model generates sketches from photographs by swapping which part of the joint image is held out, using the same architecture and implementation as sketch-to-image generation.The paper reports convincing reverse-generation results.Source: §5.3 · p. 13, Figure 11 · p. 14
5.4 Limitations
- Face generation cannot guarantee identity preservation from sparse free-hand sketches; outputs may omit attributes or fail on irregular face shapes outside the data subspace.The authors suggest face-attribute constraints as a possible guide for generation.Source: §5.4 · p. 13, Figure 12 · p. 14
6 Conclusion and Future Work
The paper formulates sketch-to-image generation as joint image completion, with the sketch providing context for predicting the image. It identifies longer test-time inference and the need for perceptual correspondence measures as remaining concerns.
6 Conclusion and Future Work
- The contextual GAN learns sketch-image correspondence in a joint distribution, then maps a corrupted joint image to a nearby uncorrupted latent representation to generate its image portion.The joint-image formulation avoids separate image and sketch domains.Source: §6 · p. 14
- Compared with end-to-end methods, the two-stage approach requires longer test-time inference, and the paper calls for better measures of perceptual correspondence on badly drawn sketches.Future work also includes investigating more powerful generative models and additional applications.Source: §6 · p. 14