Source-linked AI summary
Scribbler: Controlling Deep Image Synthesis with Sketch and Color
Patsorn Sangkloy, Jingwan Lu, Chen Fang, Fisher Yu, James Hays
TL;DR
Deep image synthesis can generate detailed imagery, but users have limited control over its outputs. The paper introduces a feed-forward adversarial architecture conditioned on sketches and sparse color strokes, producing realistic and diverse images across faces, cars, and bedrooms. It also supports imperfect sketch inputs and controllable grayscale colorization, although realism objectives can override unusual user controls.
Problem
Deep image synthesis methods generate detailed and diverse images, but it remains difficult for users to control their outputs.
Method
A feed-forward adversarial architecture generates images from sketched boundaries and sparse color strokes, learning to recover detail and extrapolate colors to semantic scene elements.
Results
The method produces more realistic and diverse sketch-to-image results across faces, cars, and bedrooms, generalizes to imperfect sketches, and supports controllable colorization.
Takeaways & Limitations
Sparse sketches can guide high-level visual structure while sparse color strokes control object color patterns in feed-forward image synthesis.
Takeaways & Limitations
Adversarial loss can override rare requested colors or unusual sketch shapes, and color may leak across semantic regions.
Abstract
from arXiv · showhide
Recently, there have been several promising methods to generate realistic imagery from deep convolutional networks. These methods sidestep the traditional computer graphics rendering pipeline and instead generate imagery at the pixel level by learning from large collections of photos (e.g. faces or bedrooms). However, these methods are of limited utility because it is difficult for a user to control what the network produces. In this paper, we propose a deep adversarial image synthesis architecture that is conditioned on sketched boundaries and sparse color strokes to generate realistic cars, bedrooms, or faces. We demonstrate a sketch based image synthesis system which allows users to 'scribble' over the sketch to indicate preferred color for objects. Our network can then generate convincing images that satisfy both the color and the sketch constraints of user. The network is feed-forward which allows users to see the effect of their edits in real time. We compare to recent work on sketch to image synthesis and show that our approach can generate more realistic, more diverse, and more controllable outputs. The architecture is also effective at user-guided colorization of grayscale images.
1. Introduction
The paper addresses the difficulty of controlling neural image synthesis by combining editable sketches with sparse color strokes. It proposes a fast adversarial architecture that generates realistic, diverse images across several domains and supports sketch variation and colorization.
- Motivation: Neural image synthesis can produce detailed and diverse images, but controlling what it generates remains difficult for everyday users.Sketches are easy to draw and edit, while sparse color strokes resolve ambiguity about object colors.
- Method: The proposed deep adversarial architecture generates realistic images from sparse sketched boundaries and color strokes.It is trained with synthetic sketches, optionally augmented by randomly sampled color strokes, and tested on faces, cars, and bedrooms.
- Method: The feed-forward architecture is fast and interactive, allowing users to see the effects of sketch edits in real time.The paper emphasizes interactive control through simple edits to sketches and sparse color indications.
- Results: The method improves sketch-to-image synthesis with higher-resolution, more diverse images across faces, bedrooms, and cars.The paper compares its quality with existing sketch-to-image work.
- Results: Training with multiple sketch styles enables realistic synthesis from imperfect hand-drawn sketches and edited synthetic sketches.The approach is presented as general across diverse sketch styles.
- Results: The same architecture also supports grayscale-to-RGB conversion and controllable colorization using sparse color strokes.Color strokes provide the control signal for user-guided colorization.
2. Related Work
Related work spans non-parametric image-fragment reuse, latent and conditional deep generative models, and user-controlled synthesis. The paper distinguishes its feed-forward sketch-and-color architecture from optimization-based and non-user-controlled alternatives.
- Image synthesis: Earlier image synthesis methods often reused existing image fragments through non-parametric approaches.These methods were among the previously successful ways to synthesize images from collections.
- Deep generative models: Deep convolutional models offer implicit generalization, while prior methods still struggled to create realistic, high-resolution images.The paper contrasts their generalization with data-driven non-parametric methods.
- Deep generative models: Latent-variable models such as VAEs and GANs learn low-dimensional representations, and deep synthesis can be conditioned on inputs including attributes, viewpoints, identity, or grayscale images.These conditioning signals provide structured alternatives to unconstrained latent vectors.
- Conditional GANs: Conditional GANs generate images from meaningful inputs such as text, low-resolution images, incomplete images, or images from other domains.Prior applications include super-resolution, inpainting, product transformation, and artistic style transfer.
- Controllable synthesis: Controllable synthesis research has used signals including 3D pose, natural language, semantic attributes, segmentation, keypoints, and bounding boxes.These approaches share the goal of adding user-editable control to deep image generation.
- Sketch control: Zhu et al. optimize for an image matching a sketch within a learned image manifold, whereas this paper learns a direct sketch-and-color-to-image mapping.The paper states that optimization-based mapping is slower and does not learn the user-input-to-output mapping during training.
3. Overview
The paper develops a feed-forward encoder-decoder that transforms user control signals into realistic, detailed images. Its objective combines reconstruction, feature, adversarial, and smoothness losses to balance control fidelity with plausible, diverse synthesis.
- The network directly transforms fine-grained controls, such as hand-drawn sketches and color strokes, into high-resolution photos with realistic texture.
- The architecture uses an encoder-decoder with three downsampling steps, seven bottleneck residual blocks, and three upsampling steps.Downsampling uses stride-2 convolutions; upsampling uses bilinear interpolation followed by residual blocks.
- Residual connections and architectural modifications support higher-resolution inputs and more challenging categories such as cars and bedrooms.The design adds an up/downsampling layer, doubles intermediate filters, and replaces deconvolutions with bilinear upsampling plus residual blocks to reduce checkerboard artifacts.
- Objective Function: Pixel loss measures average per-pixel L2 error, while feature loss measures L2 distance between high-level representations extracted by a pretrained network.
- Objective Function: Pixel and feature losses alone can average over multimodal solutions, producing conservative, blurry, desaturated, or implausible outputs under coarse user control.The problem is especially visible for categories with varied foreground and background shapes and colors, such as cars and bedrooms.
- Objective Function: Adversarial loss encourages vivid, varied, realistic outputs with plausible deviations from ground truth rather than exact reproduction.A weak, input-unconditioned discriminator is used to improve adversarial training stability.
- Objective Function: The final objective combines pixel, feature, adversarial, and total-variation losses.Total variation loss encourages smoothness in the generated output.
4. Sketch-based Photo Synthesis
The paper trains a feed-forward adversarial network to reconstruct realistic photos from synthetic or hand-drawn sketches across faces, cars, and bedrooms. Multi-style training improves generalization, while adversarial loss sharpens, diversifies, and beautifies outputs but can weaken fidelity to exaggerated sketches.
- Training Data: The network is trained on synthetic sketch-photo pairs generated from 200k photos across faces, cars, and bedrooms, with augmentations for robustness.Training includes random crops, brightness changes, and removed sketch lines; additional face data covers multiple sketch styles.
- Training Data: Because real hand-drawn sketches vary substantially from synthetic sketches, the authors augment training with multiple algorithmic and hand-drawn sketch styles.The multi-style set contains 21,848 images across six sketch styles and is used to fine-tune the network.
- Results: Compared with Sketch Inversion, the deeper network produces sharper bedroom results and better hallucinates missing facial details, even without adversarial loss.The comparison uses matched training data and settings across three image categories.
- Results: Adversarial loss encourages sharper edges, higher contrast, realistic lighting and color, and more diverse outputs than content loss alone.It reduces repetitive skin tones and hair colors in portraits and dull colors in bedrooms and cars.
- Results: After multi-style training and augmentation, the network generalizes to arbitrary hand-drawn sketches across head poses, backgrounds, and textures.The authors report that missing object parts remain challenging and outside the paper’s scope.
- Results: Adversarial training can realistify caricatured inputs, reducing exaggerated facial features or smoothing unusual face shapes instead of strictly following sketch constraints.This behavior reflects reliance on the learned natural-image manifold.
5. User-guided Colorization
The colorization extension addresses the under-determined nature of grayscale or sketch-based synthesis by adding sparse color strokes during training and inference. The network propagates user-specified colors within semantic regions while preserving plausible scene colors and details.
- Motivation: A grayscale sketch can correspond to photos with many colorings, so the authors add color controls to reduce this under-determination.The same framework supports guided sketch colorization and guided image colorization.
- Training: The network is trained with synthetic color strokes sampled from ground-truth images and overlaid on input sketches.Randomized stroke locations, lengths, thicknesses, and colors emulate user behavior.
- Training: Pixel loss is retained because it makes the output more precise with respect to the input color strokes, alongside feature and adversarial losses.The modified training data combines sketches with color strokes.
- Guided Sketch Colorization: The network propagates stroke colors to relevant bedroom and car regions while respecting object boundaries, even when strokes differ substantially from ground-truth colors.Simple edits indicating a picture frame and yellow lamp produce plausible bedroom details.
- Guided Image Colorization: For grayscale car images, the system uses sparse color strokes to guide foreground colors while adversarial training supplies plausible semantic colors for regions such as sky and trees.The reported results use strokes at random locations, mostly on the car body.
6. Applications
The proposed controls support interactive design, portrait editing, forensic reconstruction, and visual search. Real-time feed-forward inference lets users inspect and revise generated imagery immediately.
- Interactive Design: Given a 256x256 input, the network transforms it into a photo-like result in about 20ms, enabling instant feedback after edits.The paper links this speed to interactive image-generation and editing applications.
- Interactive Design: Interior and car designers can sketch rough shapes, specify regional colors, generate details and textures, then iteratively modify the design.The workflow is presented as a practical use of fine-grained sketch and color control.
- Portrait Editing: Artists can edit virtual portraits by changing eyes or hairstyles and adding glasses or head decorations.The paper also identifies portrait reconstruction as potentially useful for suspect identification.
- Visual Search: A reconstructed scene can be passed to a visual search engine to retrieve high-resolution photos with similar visual content.The workflow begins with sketched object boundaries and color constraints.
7. Network Training Details
Training is separated into content-focused pretraining and application-specific fine-tuning. The first stage prioritizes pixel and feature fidelity, while the second adjusts loss weights for reconstruction or colorization.
- Two-Stage Training: The authors use two training stages because adversarial training has unpredictable behavior.The first stage sets the adversarial weight to zero.
- Content Pretraining: During the first stage, content loss combines pixel and feature loss, with VGG-19 ReLU2-2 features chosen to preserve fine sketch details.Higher-level features may ignore details such as exact pupil locations.
- Application Fine-Tuning: The network is pretrained for content loss and then fine-tuned with different loss settings for grayscale-sketch reconstruction and colorization.Reconstruction emphasizes feature and adversarial losses, whereas colorization emphasizes feature and pixel losses.
8. Conclusion and Future Work
The framework enables sketch-guided structure and sparse color-stroke control for image synthesis, while exposing limitations in realism, rare controls, and scale generalization.
- The framework uses sketches to guide high-level visual structure and sparse color strokes to control object color patterns.
- Blurry boundaries between object parts or differently colored regions can diminish the realism of generated results.
- Color leakage can occur, such as a car hood’s color bleeding into the background.
- Adversarial loss can override rare user-specified colors or shapes in favor of outputs judged more realistic.
- The system expects objects at training-time scales, motivating future multi-scale support through randomized crop-to-image ratios.