Source-linked AI summary
SketchyCOCO: Image Generation from Freehand Scene Sketches
Chengying Gao, Qi Liu, Qi Xu, Limin Wang, Jianzhuang Liu, Changqing Zou
TL;DR
Scene-level freehand sketches offer flexible expression of user intent but vary in style and may leave background areas unspecified. The paper introduces EdgeGAN and SketchyCOCO, and reports realistic, faithful scene-image generation alongside superior qualitative and quantitative performance over existing sketch-to-image approaches.
Problem
Scene-level freehand sketches vary widely in abstraction and detail, making it difficult to generate realistic images that match user intent across specified foregrounds and loosely sketched backgrounds.
Source: §1 · p. 1Method
EdgeGAN maps images and varied-style edge maps into a shared attribute space, while a two-stage pipeline generates foreground objects and then the background.
Source: §1 · p. 2Results
Takeaways & Limitations
SketchyCOCO provides a composite dataset for evaluating scene-level freehand-sketch image generation.
Source: §1 · p. 2Foreground object-level training images in the current SketchyCOCO version come from COCO Stuff, excluding objects with major parts occluded.
Source: §6 · p. 8
Abstract
from arXiv · showhide
We introduce the first method for automatic image generation from scene-level freehand sketches. Our model allows for controllable image generation by specifying the synthesis goal via freehand sketches. The key contribution is an attribute vector bridged Generative Adversarial Network called EdgeGAN, which supports high visual-quality object-level image content generation without using freehand sketches as training data. We have built a large-scale composite dataset called SketchyCOCO to support and evaluate the solution. We validate our approach on the tasks of both object-level and scene-level image generation on SketchyCOCO. Through quantitative, qualitative results, human evaluation and ablation studies, we demonstrate the method's capacity to generate realistic complex scene-level images from various freehand sketches.
1. Introduction
The paper tackles controllable scene-level image generation from freehand sketches, which can express object location, pose, and shape but vary widely in style and detail. It introduces EdgeGAN and SketchyCOCO, and reports superior performance over existing sketch-to-image approaches.
1. Introduction
- Freehand sketches offer intuitive, flexible control and convey user intent through details such as object location, pose, and shape.Prior work conditions image generation on signals including text, semantic maps, layouts, and scene graphs.Source: §1 · p. 1
- Scene-level sketches vary in abstraction and may specify foreground objects in detail while leaving background objects rough or blank.This variability makes it difficult to generate images that meet users' intentions across both foreground and background.Source: §1 · p. 1
- Progressive generation first synthesizes foreground objects, then uses them to constrain background generation and reduce the problem's complexity.Foreground content is intended to meet specific user requirements, while background content may align more loosely with the sketch.Source: §1 · pp. 1–2
- EdgeGAN bridges images and varied-style edge maps through shared attribute vectors, avoiding the need for foreground freehand sketches as training data.The vectors encode pose and appearance information from cross-domain data.Source: §1 · p. 2
- EdgeGAN achieves significantly superior performance to existing sketch-to-image approaches in qualitative and quantitative comparisons.The evaluation uses SketchyCOCO, which includes scene-level and object-level examples.Source: §1 · p. 2
- SketchyCOCO is a large-scale composite dataset built to evaluate the approach and facilitate related research.It contains scene-level image-sketch pairs, foreground sketch-image-edge triplets, background sketch-image pairs, and scene-sketch segmentation ground truth.Source: §1 · p. 2
2. Related Work
Prior sketch-to-image work focused mainly on single objects, while other scene-generation approaches commonly infer multiple appearances together. This paper instead generates foreground objects and background sequentially, using EdgeGAN for object generation and pix2pix for backgrounds.
2. Related Work
- Earlier sketch-to-image methods include retrieval and compositing, while SketchyGAN [9] and ContextualGAN [26] generate from object-level sketches; scene-level methods often use coarse-to-fine generation from layouts or other conditions [20] [38].The paper distinguishes its scene-sketch focus and sequential foreground-background strategy from these approaches.Source: §2 · p. 2
3. Method
- The framework locates and recognizes scene-sketch objects, generates foreground instances individually in random order, then generates the background in one pass from background sketches and synthesized foreground.The two modules are trained separately.Source: §3 · p. 3
3.1. Foreground Generation
- EdgeGAN learns a shared object representation from images and varied-style edge maps, then generates an image from an attribute vector capturing the sketch's expression intent.Its image and edge channels share noise and a category label; a joint discriminator checks whether generated images match generated edge maps.Source: §3.1 · p. 3
3.2. Background Generation
- Pix2pix generates backgrounds from background sketches and synthesized foreground, allowing uncertain or blank regions to be filled consistently with the foreground.The paper gives grass and ground below zebras as an example of a region filled without sketch constraints.Source: §3.2 · p. 3
4. SketchyCOCO Dataset
SketchyCOCO combines instance sketches with COCO Stuff images and associated edge, layout, and segmentation information. Its examples cover 14 foreground and 3 background categories, with training and test splits constructed for both sketches and images.
4. SketchyCOCO Dataset
- The dataset uses 14 foreground and 3 background categories, with per-category example counts and train/test splits shown alongside representative sketch-image pairs.Instance sketches were collected from Sketchy, Tuberlin, and QuickDraw datasets.Source: §4 · p. 3, Figure 4 · p. 4
- From 14,081 COCO Stuff images, the construction process produces 27,683 background sketch-image pairs by placing background sketches in corresponding background regions.The images are split 80% for training and 20% for testing.Source: §4 · p. 4
- Retrieving same-class sketches for foreground objects yields 20,198 foreground sketch-image-edge triplets, with layout and segmentation information for synthesized scene sketches.The data are split into 18,869 training and 1,329 test triplets.Source: §4 · p. 4
- Training and test scene sketches are built only from instance sketches assigned to their respective split.Source: §4 · p. 4
5. Experiments
Experiments find that EdgeGAN produces realistic object-level images and high-quality scene-level images from sketches, while its scene-level results are comparable to the state of the art overall. The evaluations also reveal limits: some baselines score better on particular faithfulness or scene-level metrics, and sketch-augmented training can distort global object shapes.
5.1. Object-level Image Generation
- EdgeGAN’s learned attribute vectors support better realism and faithfulness than ContextualGAN’s low-level sketch-edge similarity supervision, which is sensitive to sketch abstractness.Figure 6 presents the object-level input sketches and model outputs for comparison.Source: §5.1 · p. 5, Figure 6 · p. 5
5.4. Ablation Study
- Training SketchyGAN and pix2pix with added freehand sketches can yield realistic local patches but distort global shapes.Figure 8 tests sketches and edge maps from several drawing styles; EdgeGAN works with a wide variety, including some styles absent from training.Source: §5.1 · p. 5, §5.4 · p. 8, Figure 8 · p. 8
5.1. Object-level Image Generation
- EdgeGAN achieves the best object-level realism metrics, but pix2pix-E-SEP, pix2pix-E-MIX, and SketchyGAN-E score better on faithfulness.Qualitative comparisons report that EdgeGAN is more realistic than ContextualGAN; edge-only pix2pix and SketchyGAN tend to colorize sketches without changing their shapes.Source: §5.1 · pp. 5–6
5.2. Scene-level Image Generation
- Directly training pix2pix or SketchyGAN on scene-sketch/image pairs often collapses, whereas background generation with foreground images and background sketches converges using 80% of those pairs.The authors suggest that avoiding the foreground-sketch-to-image mapping and using foreground images as prior information help explain the difference.Source: §5.2 · p. 6
- EdgeGAN produces more realistic scene images than GauGAN and Ashual et al.’s method, with overall visual quality comparable to the state-of-the-art system.The qualitative comparison especially favors EdgeGAN on foreground-object realism.Source: §5.2 · p. 6, Figure 7 · p. 7
- Scene-level FID and SSIM favor GauGAN-semantic-map and Ashual-layout models on some comparisons, while EdgeGAN has an advantage on foreground FID (local).The authors attribute the global-metric differences partly to tighter semantic-map constraints and broader category coverage in GauGAN, and to explicit spatial constraints in Ashual’s layouts.Source: §5.2 · p. 7
5.3. Human Evaluation
- In human evaluation, EdgeGAN has the highest scene-level realism score, despite FID disagreement, while GauGAN leads scene-level faithfulness.Thirty participants evaluated realism and faithfulness across object-level and scene-level samples.Source: §5.3 · p. 7
6. Discussion and Limitation
Background sketches can control generated regions even when large areas remain unconstrained, while foreground placement changes also affect the generated background. The authors note view-angle bias in SketchyCOCO and segmentation failures on highly abstract object sketches.
6. Discussion and Limitation
- Adding background categories changes corresponding image regions, and outputs remain reasonable despite large unconstrained blank areas.The authors also report that changing foreground location and size alters the background, with foreground and background blending well.Source: §6 · p. 8
- SketchyCOCO foreground views are imbalanced, with some angles such as side views dominant; the authors are considering balancing the dataset.View angles were measured by sampling 50 examples per class across eight x-y-plane angle ranges.Source: §6 · p. 8, Figure 10 · p. 8
- The adopted instance-segmentation algorithm may fail on scene sketches whose object sketches are too abstract.The authors are considering tailoring a more effective scene-sketch segmentation algorithm.Source: §6 · p. 8
7. Conclusion
The paper presents a neural-network framework for generating scene-level images from freehand sketches and builds SketchyCOCO to evaluate it. Experiments demonstrate realistic and faithful image generation across a wide range of freehand sketches.
7. Conclusion
- The paper presents a neural-network framework for generating scene-level images from freehand sketches.The authors describe it as the first framework to tackle this problem.Source: §7 · p. 8
- SketchyCOCO is a large-scale composite dataset based on MS COCO Stuff, built to evaluate the proposed solution.Source: §7 · p. 8
- Experiments demonstrate that the approach can generate realistic and faithful images from a wide range of freehand sketches.Source: §7 · p. 8
1. Objective Function
The objective functions combine adversarial training with classification and encoder losses. The generator losses are summarized separately for edge and image generation.
1. Objective Function
- WGAN-GP provides the basis for network training, with separate losses for the edge-image discriminator, edge discriminator, and image discriminator.The edge and image discriminators use the same architecture as the edge-image discriminator.Source: §1 · p. 11
- The classifier uses focal loss, and the generator also maximizes the classification loss while the classifier is fixed.Source: §1 · p. 11
- The encoder is trained with an L1 loss between the sampled latent vector z and ˜z, obtained by encoding the generated edge.Source: §1 · p. 11
- The edge generator GE minimizes the combined generator loss LDJ(G) + LDE(G).Source: §1 · p. 11
- The image generator GI has a separately defined generator loss that it minimizes.Source: §1 · p. 11
2. Implementation Details
Implementation details specify training duration, latent-vector sampling, network components, and optimizer settings for instance and background generation.
2. Implementation Details
- Instance generation is trained for 100 epochs using latent vectors sampled from a normal distribution with zero mean and variance 1.0.Source: §2 · p. 11
- The encoder uses ResNet blocks with instance normalization and ReLU, while the classifier uses MRU blocks; generators use ReLU and Tanh, and discriminators use Leaky ReLU.DCGAN uses Adam at learning rate 0.0002 with beta 0.5; WGAN and WGAN-GP use RMSprop at 0.0002, with WGAN weight clamping at −0.01 to 0.01 and WGAN-GP gradient-penalty weight λ = 10.Source: §2 · p. 11
- Background generation uses pix2pix trained for 110 epochs, and XDoG produces the training-data edge maps.Source: §2 · p. 11
3. Representative Samples from SketchyCOCO
This section presents additional SketchyCOCO examples, comparisons with other sketch-to-image methods, and generated object- and scene-level results at multiple resolutions.
3. Representative Samples from SketchyCOCO
- Figure 11 presents additional SketchyCOCO examples, including five-tuple ground-truth data.The five-tuple examples are identified as SketchyCOCO ground truth.Source: §3 · p. 11
4.1. More object level comparison results
- Figure 13 supplements the paper’s comparison of EdgeGAN with ContextualGAN, SketchyGAN, and pix2pix under different training strategies.The section does not state comparative outcomes for these additional examples.Source: §4.1 · p. 11
4.2. Some 128 × 128 results in the object level
- The 128 × 128 object-level model adds a discriminator copied from DI and uses 64 × 64 and 256 × 256 discriminator inputs to capture global information and local detail.The section points to Figure 14 for example results.Source: §4.2 · p. 11
5. Scene Level Results
- Scene-level examples include 128 × 128 results generated from 64 × 64 object-level results and 256 × 256 results generated from 128 × 128 object-level results.The examples appear in Figures 15 and 16, respectively.Source: §5 · p. 11
6. Ablation Study
The ablations indicate that multi-scale and joint discriminators support image detail and image-edge consistency. Replacing WGAN-gp with WGAN or DCGAN worsens realism and faithfulness.
6. Ablation Study
- Without DJ, generated image pose and shape are incorrect; DJ’s adversarial loss constrains the generated edge and image to correspond.DJ receives the generated edge and image together as a fake input and the real edge-image pair as a real input.Source: §6 · pp. 11–12
- The full model uses WGAN-gp with DJ, DI, and DE; replacing WGAN-gp with WGAN or DCGAN worsens faithfulness and realism.Figure 12 depicts the network ablation.Source: §6 · p. 12, Figure 12 · p. 13
- The full model outperforms the model without multi-scale discriminators on both realism and faithfulness metrics.Table 2 reports the object-level ablation scores.Source: Table 2 · p. 13, §6 · p. 13
- Without DI or DE, local-detail quality is worse; their enlarged inputs let the 64 × 64 model learn smaller receptive fields.The enlarged discriminator inputs are 128 × 128.Source: §6 · p. 13