Source-linked AI summary
SketchDreamer: Interactive Text-Augmented Creative Sketch Ideation
Zhiyu Qu, Tao Xiang, Yi-Zhe Song
TL;DR
AIGC has advanced in realistic image generation, but the paper identifies creativity in the content-creation process as a remaining gap. SketchDreamer combines text-conditioned diffusion with a differentiable Bézier renderer and initial-sketch conditioning for interactive sketch ideation. The authors report qualitative results and a human study, while noting that performance can be affected by dataset bias and quality issues.
Problem
Existing AIGC approaches focused on high-quality images are described as limited in capturing human creativity, while interactive edits often generate a new image rather than extend a sketch.
Source: §1 · pp. 1–2Method
SketchDreamer combines a differentiable Bézier renderer and score distillation sampling with ControlNet conditioning on an initial sketch, enabling text- and sketch-guided ideation.
Source: §1 · p. 3Results
The authors report qualitative results and a human study that validates SketchDreamer’s effectiveness.
Source: §1 · p. 3Takeaways & Limitations
The approach offers an interactive sketch-ideation process in which users can expand prompts and adjust sketches to develop a storyboard.
Source: §1 · p. 2, §6 · p. 10Performance can be affected by dataset biases and quality issues stemming from the use of the LDM algorithm.
Source: Limitations and Discussion · p. 10
Abstract
from arXiv · showhide
Artificial Intelligence Generated Content (AIGC) has shown remarkable progress in generating realistic images. However, in this paper, we take a step "backward" and address AIGC for the most rudimentary visual modality of human sketches. Our objective is on the creative nature of sketches, and that creative sketching should take the form of an interactive process. We further enable text to drive the sketch ideation process, allowing creativity to be freely defined, while simultaneously tackling the challenge of "I can't sketch". We present a method to generate controlled sketches using a text-conditioned diffusion model trained on pixel representations of images. Our proposed approach, referred to as SketchDreamer, integrates a differentiable rasteriser of Bezier curves that optimises an initial input to distil abstract semantic knowledge from a pretrained diffusion model. We utilise Score Distillation Sampling to learn a sketch that aligns with a given caption, which importantly enable both text and sketch to interact with the ideation process. Our objective is to empower non-professional users to create sketches and, through a series of optimisation processes, transform a narrative into a storyboard by expanding the text prompt while making minor adjustments to the sketch input. Through this work, we hope to aspire the way we create visual content, democratise the creative process, and inspire further research in enhancing human creativity in AIGC. The code is available at \url{https://github.com/WinKawaks/SketchDreamer}.
1 Introduction
The paper frames human sketches as a creative AIGC modality and proposes interactive ideation in which text and sketch jointly guide creation. SketchDreamer uses an initial sketch and text prompts to support iterative sketching, with qualitative results and a human study offered as evidence of effectiveness.
Motivation
- Sketches are presented as a creative form of visual communication, offering a way to explore AIGC beyond realistic image generation.The paper identifies a lack of creativity in existing AIGC as motivation for this direction.Source: §1 · p. 1
- Unlike multi-stage AIGC edits that produce a new image, the proposed process adds strokes to the existing sketch, resembling how people draw.The authors argue this makes sketching a more interactive and dynamic creative process.Source: §1 · p. 2
Interactive ideation
- SketchDreamer lets text and sketch jointly inform ideation, addressing the “I can’t sketch” challenge for novice users.The authors describe this as making ideation more accessible and inclusive.Source: §1 · p. 2
Proposed approach
- SketchDreamer generates controllable sketches from captions and initial sketches, then supports further ideation through both text and sketch prompts.The introduction contrasts this with VectorFusion [23], which optimises SVG paths using a differentiable rasteriser but is text-only.Source: §1 · p. 3
- A differentiable Bézier-curve renderer and score distillation sampling guide sketch generation, while ControlNet uses the initial sketch as a scribble condition.The initial sketch also serves as the renderer’s default image; the authors report qualitative results and a human study validating the method.Source: §1 · p. 3
2 Related Work
Related work spans diffusion-based generation, sketch synthesis and completion, text-guided sketch methods, and differentiable vector rendering. SketchDreamer is positioned against text-only VectorFusion by conditioning sketch generation on both text and an initial sketch.
Diffusion models
Sketch generation
- Sketch generation research includes SketchRNN [15] for stroke sequences, convolutional approaches [5], sketch completion with SketchGAN and Sketch-BERT [34], and compositional sketches from DoodlerGAN [13].Earlier work also commonly translated images into sketches.Source: §2 · pp. 3–4
Guided sketch generation
Comparison with VectorFusion
Vector graphics
- Differentiable rendering enables vector-content manipulation and synthesis through raster-based losses; this work uses a rasteriser that computes gradients for Bézier curves and other strokes.The renderer supports optimising vector primitives from rendered images.Source: §2 · p. 5
3 Methodology
The section introduces diffusion and Score Distillation Sampling (SDS) as a caption-alignment signal, then uses them to optimize differentiable Bézier sketches conditioned on text and an initial sketch.
3.1 Preliminaries
- Diffusion models learn data distributions through denoising, optimizing a variational likelihood bound expressed as a weighted combination of denoising objectives.The timestep is sampled uniformly and indexes the noise schedules; the denoiser predicts noise content.Source: §3.1 · p. 5
- Caption-conditioned diffusion models may produce outputs that lack caption coherence; superconditioning scales conditional outputs away from an unconditional prior that disregards the caption.The caption condition is typically provided through cross-attention layers and language-model text features.Source: §3.1 · p. 5
- SDS uses a pretrained text-to-image diffusion model as a loss to optimize an image or differentiable image parameterization toward the teacher model’s conditional distribution.It requires a pixel-space prior and computes a per-pixel gradient from the difference between added and predicted noise.Source: §3.1 · pp. 5–6
3.2 SketchDreamer
- The initial sketch may be simple strokes or a storyboard subplot, while only Bézier control-point positions are optimized and stroke width, color, and opacity remain fixed.Stroke parameters are rendered as a rasterized sketch by the differentiable rasterizer.Source: §3.2 · p. 6
- SketchDreamer represents an initialized sketch as n Bézier strokes, rasterizes them differentiably, and uses the initial sketch as an extra condition alongside the caption in ControlNet.The initial sketch can be simple strokes or a storyboard subplot; optimization changes control-point positions while fixing stroke width, color, and opacity.Source: §3.2 · p. 6
- Each optimization step augments the rasterized and initial sketches, encodes the result into latents, applies noise and conditioned denoising, then backpropagates SDS to the curve parameters.The gradient passes through the augmentations and differentiable rasterizer; the rasterizer, augmentation, and frozen LDM encoder are treated as one parameterized image generator.Source: §3.2 · pp. 6–7, Figure 3 · p. 6
4 Experiments
The experiments compare SketchDreamer with VectorFusion variants using QuickDraw initial sketches and evaluate text consistency and resemblance to those sketches. Qualitative comparisons favor SketchDreamer’s controllability, while the user study finds greater initial-sketch consistency despite lower CLIP scores.
4.1 Experimental settings
- The comparison included VectorFusion with text alone and VectorFusion Init with text plus an initial sketch; the user study involved 50 participants.Sketches were optimised for 1000 iterations using Adam, with random perspective and resized-crop augmentations.Source: §4.1 · p. 7, §4.3 · p. 9
4.2 Qualitative evaluation
- With QuickDraw seeds, SketchDreamer produced more controllable generations than VectorFusion and VectorFusion Init, completing simple sketches with intricate details.QuickDraw sketches are iconic and abstract, and some are unfinished.Source: §4.2 · p. 7
- Against VectorFusion’s completed-caption generations, SketchDreamer offered progressive interaction; VectorFusion showed weak appearance control and sometimes missed objects in multi-object sketches.The comparison used four completed captions because VectorFusion does not support progressive interaction.Source: §4.2 · p. 7, Figure 5 · p. 8
4.3 Quantitative evaluation
- SketchDreamer was significantly more consistent with initial sketches in the user study, although it did not lead CLIP R-Precision or similarity.The study asked participants to select sketches resembling the initial sketches from 20 sets; the table also reports CLIP metrics.Source: §4.3 · p. 9, Table 1 · p. 9
Limitations and Discussion
SketchDreamer’s performance is constrained by dataset biases and quality issues associated with LDM, and text-based Bézier-curve initialization is challenging. The authors leave stroke initialization to users in future applications.
Limitations and Discussion
- Dataset biases and quality issues stemming from LDM can influence SketchDreamer’s performance.The authors anticipate improvement as text-to-image models advance.Source: Limitations and Discussion · p. 10
- Text is challenging to use for Bézier-curve initialization, so the authors say future applications should leave stroke placement to users.Their current approach periodically removes low-opacity or small-area paths and randomly reinitializes curves.Source: Limitations and Discussion · p. 10
6 Conclusion
The conclusion presents interactive sketch ideation, informed by both text and sketches, as a way to explore creativity in AIGC through human sketches and widen access to visual content creation.
6 Conclusion
- SketchDreamer explores creativity in AIGC through an interactive process in which text and sketches inform ideation, with the stated aim of widening access to visual content creation.The authors hope this approach will inspire further directions in AIGC.Source: §6 · p. 10