Source-linked AI summary
Interactive Sketch & Fill: Multiclass Sketch-to-Image Translation
Arnab Ghosh, Richard Zhang, Puneet K. Dokania, Oliver Wang, Alexei A. Efros, Philip H. S. Torr, Eli Shechtman
TL;DR
Creating images from scratch is difficult for novice users when systems require complete, accurate sketches. The paper proposes a two-stage interactive GAN that completes partial sketches, renders the completed shapes, and uses gating to condition one generator across object classes, with successful multi-class generation reported.
Problem
Existing interactive image-translation systems require complete edge or label maps, while novice users struggle to draw accurate object proportions and shapes from scratch.
Method
The method completes sparse user sketches before synthesizing images and uses class-conditioned soft gating to support multiple object classes with one generator.
Results
Soft gating achieves 89.6%–99.6% classification accuracy, comparable to per-class generators at 97.0%, while channel-wise multiplication reaches a 23.4% AMT fooling rate versus 17.7%.
Takeaways & Limitations
The two-stage design provides interactive coarse-shape feedback while supporting realistic image generation across multiple classes.
Abstract
from arXiv · showhide
We propose an interactive GAN-based sketch-to-image translation method that helps novice users create images of simple objects. As the user starts to draw a sketch of a desired object type, the network interactively recommends plausible completions, and shows a corresponding synthesized image to the user. This enables a feedback loop, where the user can edit their sketch based on the network's recommendations, visualizing both the completed shape and final rendered image while they draw. In order to use a single trained model across a wide array of object classes, we introduce a gating-based approach for class conditioning, which allows us to generate distinct classes without feature mixing, from a single generator network. Video available at our website: https://arnabgho.github.io/iSketchNFill/.
1. Introduction
The paper introduces an interactive system that completes partial novice sketches, renders corresponding images, and supports multiple object classes with one gated conditional GAN.
- Novice-oriented interaction addresses the difficulty of incrementally constructing accurate object outlines and proportions from scratch.Existing translation interfaces generally require complete edge or label maps, while untrained users struggle with accurate proportions, 3D shapes, and perspective.
- The system generates full images from partial strokes and recommends plausible completions while users draw.It leaves exact object proportions to the model, which continually predicts plausible sketch completions.
- The method first completes sparse outlines or sketches, then synthesizes an image from the completed shape.This intermediate shape lets users inspect and refine coarse geometry, while additional supervision separates completion from image generation.
- A gating mechanism conditions one generator on object class while keeping class-specific activations separated.The shared generator and discriminator support multiple classes in a finite-size deployable model.
- The evaluation uses ten simple object classes, including six visually similar round classes, and also demonstrates faces and shoes.The round-class setup tests whether class conditioning can provide texture information beyond similar shapes.
2. Related Work
Related work spans interactive sketching, generative modeling, conditional image translation, and network gating; the paper combines these directions in a two-stage interface.
- Interactive Generation: Prior interactive systems include sketch completion, retrieval-based sketch-to-image methods, and optimization interfaces, but some are limited to one class or lack shape recommendations.The paper positions its interactive contribution against ShadowDraw, PhotoSketcher, Sketch-RNN, and related methods.
- Generative Modeling: GANs and VAEs learn mappings from low-dimensional latent codes to high-dimensional images, while conditional models additionally use class, noise, or image inputs.Conditional GANs established image-to-image translation, but injecting noise often fails to produce multimodality because the generator may ignore the latent code.
- Interactive Generation: Figure 2 illustrates two interface modes: quickly generating multiple objects and exploring multimodal shape completions during drawing.The visual is presented as a video of the interactive interface rather than a quantitative comparison.
- Conditioned Image Generation: The proposed interface decouples shape completion from appearance synthesis, using shape and appearance generators with separate discriminators.The shape generator completes sparse inputs before the appearance generator produces the final image.
- Gating Mechanisms: Gating mechanisms modulate network computation for category-specific specialization and have precedents in residual, normalization, and sequential architectures.The paper adapts this idea for image generation and analyzes gating mechanisms systematically.
3. Method
The method separates interactive sketch-to-image generation into shape completion and appearance synthesis, using multi-scale conditioning and class-dependent gating to support multimodal, multiclass generation.
- 3. Method: The system decouples generation into a shape generator for completing sparse user sketches and an appearance generator for producing the final image.
- 3.1. Shape completion: The shape generator repeatedly proposes completed shapes from partial strokes, using simulated missing patches and multiscale conditioning to preserve the user input.Training inputs are created by removing random square patches, while conditioning reaches the generator and discriminator at multiple scales.
- 3.2. Appearance synthesis: A single multiclass generator is conditioned by an object label and uses gating to focus activations on class-relevant network components.This gating approach is intended to separate classes while sharing one generator and discriminator across them.
- 3.2. Appearance synthesis: The gating network predicts layer-wise or channel-wise modulation parameters, allowing conditioning to switch blocks or channels on and off.The method applies gating to both generator and discriminator, with channel-wise gating reported as strongest empirically.
- 3.2. Appearance synthesis: The appearance generator evaluates several conditioning strategies, including input concatenation, all-layer conditioning, and an AC-GAN-like latent regressor.
4. Experiments
Experiments evaluate the two-stage pipeline on single- and multi-class sketch-to-image generation, then test class-conditioning strategies for a shared generator. The results support shape completion plus channel-wise gating as effective components for interactive multiclass generation.
- 4.1. Single Class Generation: Across faces and shoes, the 2-stage pipeline produces stronger results than directly mapping partial sketches to completed images.The method completes simplified edge maps before rendering realistic images from them.
- 4. Experiments: The interactive system updates completed shapes and generated images as users add or change strokes, with local edits producing coherent changes elsewhere.The multiclass results show that a few input strokes can suffice for class-specific outline and appearance completion.
- 4.2. Multi-Class Generation: Naive concatenation achieves only 64.5% accuracy when concatenating at all layers and 62.6% at the input layer, with both settings remaining low.The experiments report confusion among visually similar classes such as basketballs, oranges, cupcakes, pineapples, and fried chicken.
- 4.2. Multi-Class Generation: 89.6%–99.6% classification accuracy from soft-gated variants approaches the 97.0% per-class baseline, while ChannelGate reaches a 23.4% AMT fooling rate versus 17.7%.Channel-wise multiplication produces the most realistic images among the gating mechanisms tested.
- 4.2. Multi-Class Generation: Channel-wise gating improves both accuracy and realism across the EncoderDecoder and SkinnyResNet architectures.The SkinnyResNet results are quantitatively and qualitatively better overall, while gating enables successful pineapple generation with EncoderDecoder.
- 4.2. Multi-Class Generation: A circle outline can generate both circular objects and noncircular objects such as strawberries, pineapples, and cupcakes, including pineapple parts absent from the outline.This tests generalization to unseen shape and class combinations.
5. Discussion
The discussion centers on a two-stage interactive generation strategy that uses shape completion as an intermediary. This intermediary stabilizes training and provides coarse geometric feedback users can accept or revise.
- 5. Discussion: The shape-completion intermediary both stabilizes training and gives users coarse geometric feedback during interactive object generation.Users can choose whether to integrate the suggested geometry into their drawings.
6. Insights on Gating Mechanism
A toy mixture-of-Gaussians experiment illustrates how residual blocks represent distribution modes in the proposed gating framework. Removing blocks removes corresponding modes while samples remain mostly on the target manifold.
- 6. Insights on Gating Mechanism: Removing one residual block removes one predicted Gaussian mode, while removing two blocks removes two modes.The full residual network closely approximates the five-component training distribution.
- 6. Insights on Gating Mechanism: The residual-network samples remain mostly on the ground-truth distribution manifold despite block removal.The toy setup uses residual-block generator and discriminator architectures conditioned on a latent vector.
7. Shape Completion Details
Shape completion is trained by masking portions of complete outlines or sketches with randomly placed square occluders at three sizes. The completion architecture injects sparse conditioning through residual blocks at multiple scales.
- 7. Shape Completion Details: Training and testing use 75 partial versions per sketch, created from three occluder sizes: 64×64, 128×128, and 192×192.Each size is applied at 25 random locations over the full sketch or outline.
- 7. Shape Completion Details: Sparse ResNet blocks resize partial user strokes and convert them to matching channel counts before adding them to feature activations.Conditioning is injected before generator upsampling and discriminator average pooling.
8. Outline→Image Network Architecture
The architecture uses residual convolutional blocks and a class-conditioned gating hypernetwork to control generator and discriminator computation at block or channel level.
- Residual architecture: The generator and discriminator are built from convolutional residual blocks, including UpConvResblock, DownConvResblock, and gated Resnet components.The supplied architecture tables identify the principal residual-block variants used in the network.
- Gating hypernetwork: The gating network uses ResNet blocks with 1D convolutions and normalization to reduce parameters and accelerate prediction of gating parameters.Class conditioning first passes through an embedding layer before processing by the ResNet blocks.
- Class-specific gating: Learned gating differs across categories, with nonuniform channel usage indicating that different classes rely more heavily on different channels.The visualization compares soft-gating parameters for generator and discriminator blocks.
- Gating hypernetwork: A class-conditioned hypernetwork predicts gating parameters for the main network, with output size determined by the selected blockwise, affine, or channelwise gating scheme.Channelwise gating predicts parameters for every channel in every residual block, while blockwise gating predicts one parameter per block.
- Gating mechanism: Gating coefficients α are constrained to [0,1] to select or reject blocks, while β parameters are constrained to [-1,1] when affine transformations are used.The parameter constraints were chosen to support effective network performance.
9. Distribution of Alphas
The learned α coefficients concentrate near 0 and 1 for both blockwise and channelwise gating, despite having no explicit sparsity constraint.
- Alpha distribution: α values are pushed toward the extremes rather than intermediate values in both blockwise and channelwise gating.This distribution is shown in Figure 16 for the two gating settings.
10. Unusual Shapes for Various Classes
The gated generator generalizes beyond training shapes and can produce unusual object classes from outlines that do not structurally indicate their distinctive parts.
- Generalization: The gated generative techniques extend to shapes never shown during training.This result is reported for the channelwise gating setting.