PaperlayerView exact revision on arXiv

Source-linked AI summary

SketchFlow: Zero-Shot Vector Sketch Generation via GMM Prior Flow in CLIP Latent Space

Jin Zhou, Hongliang Yang, Pengfei Xu, Hui Huang

arXiv:2608.21659v1cs.CV

TL;DR

SketchFlow tackles zero-shot vector sketch generation despite scarce fine-grained text–sketch pairs. It uses CLIP-space OT-CFM transport from a continuous GMM prior and a hybrid diffusion decoder, achieving high visual quality, human-style adherence, and promising local zero-shot generalization. Its mapping remains imperfect for some unseen prompts, with shape omissions, text-like strokes, and inconsistent results.

  • Problem

    Fine-grained text–sketch pairs are scarce, while existing methods struggle to generate human-like strokes and generalize beyond fixed category vocabularies.

  • Method

    SketchFlow uses CLIP-mediated cross-modal alignment, a noise-expanded GMM prior, OT-CFM transport, and a hybrid 1D U-Net–Transformer diffusion decoder.

  • Results

    SketchFlow achieves high visual quality, follows human drawing styles, and shows promising local zero-shot generalization beyond the QuickDraw vocabulary.

  • Takeaways & Limitations

    Optimal Transport geometry enables CLIP-mediated zero-shot transport using label-only textual supervision, reducing reliance on exhaustive fine-grained annotations.

  • Takeaways & Limitations

    The mapping remains imperfect: some unseen-prompt outputs omit shape parts, resemble prompt words, or remain inconsistent despite increased search variance.

Abstract

from arXiv · show

Vector sketches remain one of the most concise and immediate mediums for abstract human expression. However, generating high-quality vector strokes that exhibit human-like drawing styles remains an open challenge due to the severe scarcity of fine-grained, high-quality text-to-sketch paired data. Existing text-conditioned generation methods often rely on unstable, time-consuming optimization or struggle to generalize to unseen categories in a zero-shot manner. To address these limitations, we present SketchFlow, a novel generative framework rooted in Optimal Transport (OT) theory and flow matching. By leveraging pre-trained CLIP models to bypass labor-intensive image-level text annotations, we formulate cross-modal alignment as a continuous mapping problem directly within the CLIP latent space. To bridge the inevitable modality gap between discrete text concepts and continuous sketch features, we first inject noise into discrete category embeddings to construct a continuous Gaussian Mixture Model (GMM) prior. We then utilize an Optimal Transport Conditional Flow Matching (OT-CFM) model to learn a deterministic vector field mapping from this continuous GMM prior to the target sketch feature distribution. Finally, a Hybrid Diffusion Decoder, fusing 1D U-Net and Transformer architectures, is designed to decode these features into fast and high-fidelity stroke trajectories. Extensive experiments demonstrate that SketchFlow substantially outperforms existing baselines in visual quality and adherence to natural human drawing styles. Furthermore, our geometry-preserving framework demonstrates promising local zero-shot synthesis for prompts beyond the QuickDraw training vocabulary, including unseen concept labels and semantic modifiers, while enabling smooth, continuous semantic interpolation between distinct concepts. Source code is available at: https://github.com/QiuHong-1202/SketchFlow.

1 Introduction

SketchFlow addresses zero-shot text-to-sketch synthesis under scarce fine-grained text–sketch supervision by aligning modalities in CLIP space and generating human-like vector strokes. It targets unseen concepts beyond QuickDraw’s fixed vocabulary.

  • Motivation: High-quality human-like vector sketch generation remains difficult because paired text–vector sketch annotations are scarce.QuickDraw provides many trajectories but only 345 discrete category labels and no fine-grained textual descriptions.
  • Motivation: Existing optimization, raster-based, and direct sequence methods are slow, unstable, artifact-prone, or inconsistent with authentic human drawing habits.The cited limitations include noise, topology violations, unintended branching, and deviations in style and abstract logic.
  • Approach: SketchFlow formulates zero-shot synthesis as continuous cross-modal mapping in CLIP latent space using Optimal Transport and flow matching.Here, zero-shot means prompts whose concept labels are absent from the 345-class QuickDraw vocabulary.
  • Approach: A continuous GMM prior bridges discrete text concepts and continuous sketch features, while an OT-CFM vector field maps between their distributions.The framework uses pre-trained CLIP representations to avoid exhaustive fine-grained text annotations.
  • Results: A hybrid diffusion decoder combining a 1D U-Net and Transformer reconstructs high-fidelity stroke trajectories.The decoder is reported to improve visual quality and adherence to human drawing styles relative to existing baselines.
  • Results: Experiments report superior visual quality and human-style adherence, plus promising local zero-shot synthesis and smooth semantic interpolation.These capabilities extend to unseen concept labels and semantic modifiers beyond the QuickDraw vocabulary.

2 Related Work

Related work spans trajectory models, differentiable vector optimization, diffusion-prior methods, and language-model-based SVG generation. SketchFlow instead combines CLIP-space transport with category-labeled sketches and a continuous GMM prior.

  • Trajectory Generation: Trajectory-based sketch models predict ordered strokes using RNNs, ODEs, or sequence diffusion, beginning with SketchRNN and QuickDraw.These approaches model the inherent temporal structure of vector sketches.
  • Text-Conditioned Synthesis: CLIPDraw, CLIPasso, and CLIPascene optimize Bézier curves through differentiable rendering but are unstable and do not reproduce authentic hand-drawn style without sketch references.Their optimization objective is text-image similarity rather than direct human-stroke modeling.
  • Diffusion Priors: Diffusion-prior methods optimize predefined vector primitives, an initialization described as ill-suited to sparse human sketches.The related methods include latent diffusion, vector-particle optimization, stroke-order constraints, and implicit curves.
  • Language Models: Language-model-based vector generators provide semantics or logical drawing order, but off-the-shelf models can produce rigid SVG graphics and paired-data methods require condition–SVG data.SketchAgent is cited as lacking learned human drawing conventions.
  • Flow Matching: OT-CFM and GMM priors motivate efficient probability transport and modeling of complex distributions.SketchFlow embeds a continuous GMM prior in CLIP space to translate isolated textual categories into a continuous semantic distribution.

3 Method

SketchFlow represents sketches as continuous trajectories, decodes target CLIP features with a hybrid diffusion architecture, and transports a continuous text-derived GMM prior to sketch features using OT-CFM.

  • Sketch Representation: The sketch trajectory is represented as an equidistant sequence of 2D coordinates and continuous pen-state variables.The pen-state range avoids early commitment to discrete states during high-noise diffusion stages.
  • Hybrid Diffusion Decoder: Interleaved 1D U-Net blocks extract local geometry, while Transformer layers model long-range dependencies across resolution scales.This hybrid backbone is the central architecture of the diffusion decoder.
  • Hybrid Diffusion Decoder: Diffusion timestep and target rendered-sketch CLIP embedding are projected, summed, and injected into decoder blocks through AdaLN.The decoder is trained with a denoising score matching objective.
  • Cross-Modal Conditioning: Continuous CLIP embeddings let the decoder translate visual semantics into coherent strokes, including for unseen concepts.The method therefore focuses on obtaining target sketch CLIP embeddings from text prompts.
  • GMM Prior: Noise injection expands discrete category text embeddings into a continuous GMM prior in CLIP latent space.This creates a smooth semantic manifold for prompts outside the training vocabulary.
  • OT Flow Matching: OT-CFM deterministically transports the GMM prior p0 to the target sketch distribution p1 through an ODE vector field.For each category, the initial state is sampled around its text embedding and the target state from the sketch-feature distribution.
  • OT Flow Matching: The intermediate state follows the straight interpolation z_t = (1 − t)z_0 + tz_1.OT coupling yields straight-path, constant-velocity trajectories intended to preserve semantic directions and non-crossing organization.
  • Adaptive Prior Variance: A fixed noise scale is suboptimal across categories, motivating variance-conditioned flow fields and adaptive inference control.Training samples standard deviations, while inference exposes a scaling parameter γ for prompt-specific adjustment.

4 Experiments

SketchFlow is evaluated on in-domain QuickDraw categories, unseen prompts, semantic modifiers, interpolation, ablations, and image-conditioned abstraction. Across these experiments, it improves fidelity and human-like sketch qualities while preserving zero-shot and continuous semantic behavior.

  • Comparison with Existing Methods: SketchFlow significantly improves FID over all learning-based baselines across the four evaluated categories.Lower RDP scores from some baselines reflect overly simplistic sketches, as shown by qualitative comparisons.
  • Comparison with Existing Methods: SketchFlow achieves CLIP scores comparable to NeuralSVG and SketchAgent while producing concise, coherent trajectories that better capture abstract hand-drawn characteristics.Higher baseline RDP scores can correspond to cluttered strokes that inflate recognition scores without matching human drawing behavior.
  • Zero-Shot Generation Beyond the Training Vocabulary: On all 44 prompts absent from the QuickDraw vocabulary, SketchFlow reaches a mean CLIP Score of 0.2480 versus 0.2024 for Gaussian-Prior and 0.2034 for Interp CLIP.It outperforms Interp CLIP on every evaluated unseen prompt without sample selection.
  • Zero-Shot Generation with Semantic Modifiers: The model generates distinct dynamic poses for running, jumping, and stretching cats while retaining an abstract hand-drawn style.These results provide qualitative evidence that CLIP-space transport retains modifier semantics beyond discrete category-name training.
  • Ablation and Semantic Interpolation: Interpolation between concepts produces smooth, topologically coherent transitions, whereas raw-text and Gaussian-Prior ablations yield unstructured strokes or abrupt structural shifts.Flow-mapped conditioning progressively self-organizes into recognizable sketches, while the baseline snaps between isolated training modes.

5 Conclusion & Limitations

SketchFlow bridges discrete text concepts and continuous sketch features through OT-CFM and a continuous GMM prior in CLIP space, producing fast, human-style vector sketches with promising local zero-shot generalization. Its limitations include incomplete shapes, text-like strokes, and inconsistent results for some unseen categories.

  • SketchFlow uses OT-CFM and a continuous GMM prior in CLIP space to bridge discrete text concepts and continuous sketch features.
  • The fast forward-pass generator achieves high visual quality, follows human drawing styles, and generalizes locally beyond the QuickDraw vocabulary.
  • The framework reduces reliance on exhaustive fine-grained textual annotations by enabling CLIP-mediated zero-shot transport under label-only textual supervision.
  • Some beyond-vocabulary outputs retain concept-level structure but omit parts of the intended shape or drift toward text-like strokes resembling the prompt word.
  • Some unseen categories require increased sampling variance and still produce inconsistent results, which the authors associate with locally uniform Gaussian assumptions and non-Gaussian semantic manifolds.

C.1 Network Architectures

The OT-Flow Matching model uses a FiLM-equipped residual MLP, while the Hybrid Diffusion Decoder combines a 1D U-Net backbone with specified multiscale channel widths.

  • The OT-Flow Matching velocity field uses a residual MLP with FiLM layers, hidden width 1024, and depth 10.
  • The Hybrid Diffusion Decoder’s 1D U-Net uses base dimension 128 and channel multipliers (1, 1.5, 2, 4).

C.2 Training Setup

SketchFlow is jointly trained with a large-batch AdamW setup and uses Heun transport integration followed by DDPM denoising at inference; prompt and scenario-specific settings are documented.

  • Training Setup: The OT-Flow Matching model and Hybrid Diffusion Decoder are jointly trained for 1,221,444 optimizer steps on 8 NVIDIA RTX 3090 GPUs.Training takes approximately three days, with total batch size 256 and AdamW learning rate 1e-4.
  • Training Setup: Inference integrates the transport ODE with Heun’s method for 60 steps, followed by 60 DDPM denoising steps.
  • Prompt Configurations: The supplementary material specifies descriptive prompts for stylized letters, numbers, and architectural sketches used in teaser results.
  • Evaluation: Creative-dataset FID comparisons use lower scores to indicate better visual fidelity and structural similarity to the target distribution.
  • Hyperparameter Settings: Default prior variance and sampling-noise settings vary by scenario, including known categories, beyond-vocabulary prompts, action prompts, and image-conditioned generation.

D Cross-Domain Transfer Evaluation

SketchFlow is evaluated on TU-Berlin and Creative-Mix to assess cross-domain transfer, stylistic adaptation, semantic control, and visual quality beyond QuickDraw.

  • The cross-domain evaluation assesses SketchFlow on three supplementary datasets beyond the primary QuickDraw training corpus.
  • TU-Berlin contains 250 object categories and roughly 20,000 freehand black-and-white sketches, while Creative-Mix combines Creative Birds and Creative Creatures.
  • TU-Berlin: On TU-Berlin, SketchFlow captures the distinctive freehand style, generates coherent sketches, and includes categories absent from the primary training corpus.
  • Creative-Mix: On Creative-Mix, the method adapts to a highly creative style while maintaining semantic control and surpassing Doodleformer in structural clarity and logical stroke composition.
  • Quantitative Results: The reported FID reduction indicates superior visual quality and closer distributional alignment to target domains, including imaginative and abstract contexts.

E Extended Zero-Shot Generation Beyond the Training Vocabulary

SketchFlow generates clear sketches for unseen concepts beyond the QuickDraw training vocabulary, with diversity across samples and a recurring hand-drawn style across random seeds.

  • SketchFlow produces clear sketches for a range of prompts outside the QuickDraw training vocabulary.
  • Multiple samples per prompt demonstrate generation diversity across random seeds.
  • The generated samples retain a recurring hand-drawn style across random seeds.

E.1 Unseen-Label Quantitative Evaluation

This evaluation reports per-prompt unseen-label results using nearest QuickDraw text anchors, cosine similarity, and mean CLIP Score, alongside the comparison setup for T2I+CLIPasso.

  • Unseen-Label Quantitative Evaluation: The evaluation reports complete per-prompt results for 44 unseen labels.
  • Unseen-Label Quantitative Evaluation: Each prompt is paired with its nearest QuickDraw text anchor and cosine similarity.
  • Unseen-Label Quantitative Evaluation: Mean CLIP Score is computed over 32 generated samples without sample selection.
  • Unseen-Label Quantitative Evaluation: The T2I+CLIPasso comparison evaluates six unseen single-concept prompts using fixed SDXL generation settings.

E.3 Additional Analysis on NeuralSVG

The analysis examines NeuralSVG across stroke counts and explains the evaluation focus on abstraction, semantics, human-likeness, stroke rationality, and aesthetics, while visualizing SketchFlow’s drawing order.

  • E.3 Additional Analysis on NeuralSVG: Fewer NeuralSVG strokes reduce RDP but degrade CLIP alignment, whereas additional strokes gradually improve CLIP and sharply increase RDP complexity.
  • E.3 Additional Analysis on NeuralSVG: At 32 strokes, NeuralSVG approaches SketchFlow’s CLIP score but retains a much higher RDP value.
  • E.3 Additional Analysis on NeuralSVG: The user questionnaire evaluates abstraction fidelity, semantic alignment, human-likeness, stroke rationality, and aesthetic quality.
  • E.3 Additional Analysis on NeuralSVG: SketchFlow’s stroke trajectories are color-coded from light blue initial strokes to dark blue final strokes.
  • E.3 Additional Analysis on NeuralSVG: The visualization is intended to show global geometric structure and a coherent drawing sequence aligned with plausible human drawing patterns.

F.3 Extended Qualitative Analysis

The qualitative analysis contrasts baseline trajectory characteristics with SketchFlow’s user-preferred style, emphasizing concise abstraction, drawing nuances, and expressive treatment of amorphous concepts.

  • F.3 Extended Qualitative Analysis: NeuralSVG outputs tend to appear dense and image-like because numerous short Bézier curves overlap at edges.
  • F.3 Extended Qualitative Analysis: SketchAgent produces semantically clear outputs, but its trajectories often appear rigid, symmetric, or repetitive.
  • F.3 Extended Qualitative Analysis: SketchFlow’s strong user preference is consistent with slight stroke variations that contribute to human-likeness.
  • F.3 Extended Qualitative Analysis: For amorphous concepts such as water and fire, SketchFlow conveys an abstract impression without being tied to a specific visual form.
  • F.3 Extended Qualitative Analysis: Extended zero-shot examples show generation diversity and a recurring hand-drawn style across random seeds.
Loading 2608.21659v1…