PaperlayerView exact revision on arXiv

Source-linked AI summary

StrokeFusion: Vector Sketch Generation via Joint Stroke-UDF Encoding and Latent Sequence Diffusion

Jin Zhou, Yi Zhou, Hongliang Yang, Pengfei Xu, Hui Huang

arXiv:2503.23752v4cs.GRcs.CV

TL;DR

Sketch-generation methods can lose stroke trajectories or struggle to balance common stroke patterns with accurate positions. StrokeFusion jointly encodes vector strokes and UDF maps, then generates strokes through latent diffusion; the authors report that it outperforms baseline approaches. Its stated limitations include challenges with semantically mixed strokes in algorithmically generated sketches.

  • Problem

    Raster models may produce pixelation or non-stroke noise and discard stroke trajectories, while vector models trade off stroke-pattern commonality against accurate positioning.

    Source: Introduction · p. 1

  • Method

    StrokeFusion jointly encodes vector strokes and rasterized UDFs, then uses stroke-level latent diffusion to denoise position, scale, and trajectory.

    Source: Introduction · p. 1

  • Results

    StrokeFusion significantly outperforms baseline approaches, with quantitative and qualitative comparisons and ablations supporting its design.

    Source: Introduction · p. 2

  • Takeaways & Limitations

    The authors report that StrokeFusion produces diverse, high-quality sketches with natural stroke fluidity, structural coherence, and fine-grained detail.

    Source: Conclusion · p. 7

    Sketches whose algorithmic strokes span multiple semantic parts can prevent high-quality generation because the method requires strokes to contain semantic information.

    Source: Limitations · p. 7

Abstract

from arXiv · show

In the field of sketch generation, raster-format trained models often produce non-stroke artifacts, while vector-format trained models typically lack a holistic understanding of sketches, leading to compromised recognizability. Moreover, existing methods struggle to extract common features from similar elements (e.g., eyes of animals) appearing at varying positions across sketches. To address these challenges, we propose StrokeFusion, a two-stage framework for vector sketch generation. It contains a dual-modal sketch feature learning network that maps strokes into a high-quality latent space. This network decomposes sketches into normalized strokes and jointly encodes stroke sequences with Unsigned Distance Function (UDF) maps, representing sketches as sets of stroke feature vectors. Building upon this representation, our framework exploits a stroke-level latent diffusion model that simultaneously adjusts stroke position, scale, and trajectory during generation. This enables high-fidelity sketch generation while supporting stroke interpolation editing. Extensive experiments on the QuickDraw dataset demonstrate that our framework outperforms state-of-the-art techniques, validating its effectiveness in preserving structural integrity and semantic features. Code and models will be made publicly available upon publication.

Introduction

Sketch generation methods face challenges balancing holistic structure, stroke-level control, and visual coherence. StrokeFusion combines dual-modal feature encoding with latent-space diffusion to address these challenges in vector sketch generation.

Raster Sketch Generation

  • Raster-based methods can use mature image-learning techniques but may produce pixelation or non-stroke noise and discard stroke trajectories.Prior raster methods yield visually appealing results but do not produce editable vector outputs; StrokeFusion combines image-level supervision with vector-format outputs.Source: Introduction · p. 1, Raster Sketch Generation · p. 2

Vector Sketch Generation

  • Relative-coordinate vector models capture common stroke patterns but accumulate positional error, while absolute-coordinate models weaken stroke-wise commonality.Prior vector-generation approaches also include fixed-length stroke embeddings treated as unordered sets to accommodate variation in stroke count and order.Source: Introduction · p. 1, Vector Sketch Generation · p. 2

Introduction

  • Its two-stage framework combines vector strokes and rasterized UDFs, then jointly denoises stroke position, scale, and trajectory in a structured latent space.The contributions also describe separating stroke layout prediction from shape synthesis to improve structural consistency and generation controllability.Source: Introduction · p. 1
  • StrokeFusion significantly outperforms baseline approaches, with quantitative and qualitative comparisons and ablations supporting its design.The reported ablations examine dual-modal encoding, stroke latent space, and spatial information disentanglement.Source: Introduction · p. 2

Continuous Vector Sketch Generation

  • Continuous vector methods using primitives, Bézier curves, or curve diffusion have limited expressiveness, while fully continuous approaches can miss holistic coherence.StrokeFusion instead encodes entire strokes as learnable embeddings while maintaining clean vector trajectories.Source: Continuous Vector Sketch Generation · p. 2

Diffusion Models

  • The paper situates its approach among continuous- and discrete-space diffusion advances and aligns diffusion modeling with progress in 3D shape representation.The passage describes diffusion models as involving forward and reverse processes.Source: Diffusion Models · p. 2

Method

StrokeFusion combines stroke-UDF joint encoding with latent diffusion to generate unordered, non-autoregressive sketches. It represents strokes with vector geometry and distance-field features, then denoises stroke latents together with their spatial configuration.

Problem Formulation and Representation

  • StrokeFusion bridges point-sequence and raster representations with stroke-UDF joint encoding, then generates sketches through unordered latent diffusion.The framework jointly denoises stroke position, scale, and trajectory in a structured latent space.Source: Problem Formulation and Representation · p. 2

Vector Representation

  • Strokes are resampled into evenly spaced points and normalized to their tight bounding squares, making stroke-wise features scale-invariant.Sketch-level and stroke-level normalization standardize bounding boxes while preserving aspect ratios.Source: Vector Representation · p. 2, Preprocessing · p. 3

Visual Representation

  • Each stroke’s unsigned distance field aggregates exponentially decaying contributions from interpolated stroke segments.The field takes the maximal contribution over segments; γ controls its sharpness.Source: Visual Representation · pp. 2–3

Stroke Embedding

Preprocessing

  • Preprocessing centers and scales sketches to [−1, 1], normalizes stroke boxes to [0, 1], and renders distance fields with a 0.8 stroke scale factor.These steps preserve aspect ratios and leave a margin against canvas-edge truncation.Source: Preprocessing · p. 3

Vector Encoder

  • The vector encoder pools Transformer-processed point features, while the image encoder compresses convolutional distance-field features before fusion.The vector path uses six Transformer layers; the image path uses six convolutional blocks.Source: Vector Encoder · p. 3, Image Encoder · p. 3

Total Loss.

  • Multi-level vector, image, and KL supervision combines stroke-geometry and perceptual-plausibility objectives for the fused representation.The total loss weights the vector, image, and KL terms.Source: Vector-level Supervision · p. 4, Total Loss · p. 4

Training Process

  • The diffusion stage augments each stroke latent with its bounding box and a validity flag, then denoises the unordered set to model strokes and spatial layout jointly.A Transformer without positional encoding preserves permutation invariance; generation retains strokes with positive predicted visibility and decodes their inverse-normalized trajectories.Source: Training Process · p. 4, Generation Process · pp. 4–5

Experiments

Experiments evaluate StrokeFusion across sketch datasets and against vector and raster baselines. Results report increasing advantage with stroke complexity and generally stronger performance across additional datasets.

Datasets.

  • Evaluation spans 14 QuickDraw classes, full TU Berlin, front-facing FaceX samples, and separate Creative Birds and Creative Creatures subsets.Source: Datasets · p. 5

Baseline Methods.

  • Comparisons include vector generators SketchRNN, SketchKnitter, and ChiroDiff, plus raster Doodleformer, which composes semantic-part boxes into raster images.Source: Baseline Methods · p. 5

Evaluation Metrics.

  • FID, Precision, and Recall assess generated-sample fidelity and diversity over 10,000 samples, with stroke simplification applied before evaluation.Source: Evaluation Metrics · p. 5

Quantitative Comparison

  • Across Creative Birds, Creative Creatures, FaceX, and TU Berlin, StrokeFusion outperforms the baselines in general, with better FID, Precision, and Recall reported across all datasets.Source: Table 2 · p. 6, Quantitative Comparison · p. 6
  • As stroke count increases, StrokeFusion becomes increasingly advantageous; it may slightly trail ChiroDiff on some simple sketches but improves fidelity and consistency on complex sketches.Source: Quantitative Comparison · p. 6, Table 1 · p. 6

Qualitative Comparison

  • On QuickDraw, StrokeFusion captures broader sketch variation, local detail, and global structural consistency than the baseline methods.Source: Qualitative Comparison · p. 6
  • On creative datasets, StrokeFusion balances structure and detail; against Doodleformer it has comparable visual quality while preserving vector output and stroke-level controllability.Source: Qualitative Comparison · pp. 6–7
  • On FaceX, StrokeFusion produces more regular, semantically aligned features; on low-data TU Berlin, it yields coherent, interpretable sketches, though results remain improvable.Source: Qualitative Comparison · p. 7

Ablation Study

The ablations find that stroke normalization and image-level supervision support stable, coherent stroke generation. The method also has efficiency and input-structure limitations.

Stroke Normalization

  • Without stroke normalization, visual quality and quantitative metrics decline, while fine structures such as spider legs and television antennas become distorted or misplaced.The comparison uses raw strokes without normalization or explicit bounding-box input.Source: Stroke Normalization · p. 7

Image-level Supervision

  • Removing image-level supervision substantially lowers all metrics and reduces diversity and coherence, with spider sketches degenerating into scattered fragments.The ablation sets γ = 0, removing visual input.Source: Image-level Supervision · p. 7

Limitations

  • High-frequency complex strokes challenge learning efficiency and generation quality, particularly for low-stroke sketches; UDF decoding also adds computational overhead.The authors propose exploring efficient implicit-field decoding or alternative UDF representations.Source: Limitations · p. 7
  • The stroke-level generation unit requires semantically meaningful dataset strokes; strokes spanning multiple semantic parts can prevent high-quality results.The authors suggest more flexible semantic segmentation as a potential improvement.Source: Limitations · p. 7

Conclusion

StrokeFusion combines dual-modal stroke encoding with continuous diffusion modeling for vector sketch generation. Its evaluations report diverse, high-quality sketches across complexity levels.

Conclusion

  • StrokeFusion combines dual-modal stroke encoding with continuous diffusion modeling for vector sketch generation.The dual-branch encoder captures geometric and visual features from point sequences and stroke distance fields in a unified latent representation.Source: Conclusion · p. 7
  • The unified embedding space facilitates smooth feature transitions during diffusion, supporting natural stroke fluidity, structural coherence, and fine-grained detail.Source: Conclusion · p. 7
  • Qualitative and quantitative evaluations report diverse, high-quality sketches across varying levels of complexity.Source: Conclusion · p. 7

Data Preprocessing

Preprocessing resamples and normalizes strokes for consistent model inputs, constructs UDFs for raster guidance, and prepares fixed-size stroke sequences for diffusion.

Data Preprocessing

  • Each stroke is uniformly resampled to 64 points and normalized to ensure geometric consistency across sketches.Inputs may begin as SVG primitives or polylines.Source: Data Preprocessing · p. 9
  • UDFs use exponential decay over interpolated stroke segments with γ = 50, avoiding large empty regions from hard rasterization and reducing aliasing.The reported performance is largely insensitive to the exact γ value; incorporating raster visual information is the key factor.Source: Data Preprocessing · p. 9
  • Diffusion training discards sketches with more than 32 strokes and pads shorter sequences with dummy strokes to a fixed size.The same filtering and padding strategy is applied to all baselines for fairness.Source: Data Preprocessing · p. 9
  • Visibility flags are noised and denoised as continuous attributes, then thresholded at zero to determine which strokes are decoded.Visible strokes use 0.1 and absent strokes use −0.1; negative decoded flags are skipped.Source: Data Preprocessing · p. 9
  • Figure 7 shows that larger γ values sharpen UDF stroke boundaries, while smaller values produce more diffuse fields.The default γ = 50 provides sufficient precision without introducing artifacts.Source: Figure 7 · p. 9

Model Architecture

The architecture encodes stroke sequences and distance fields into a fused latent representation, then decodes vector and image outputs. Validation selects a larger Transformer configuration for the final model.

Model Architecture

  • The vector branch encodes 64-point stroke sequences with six Transformer layers and attention pooling, while the image branch uses six convolutional blocks to produce visual embeddings.The input projection hidden dimension and visual embedding dimension are both 64.Source: Model Architecture · p. 9
  • The fused latent concatenates vector and visual embeddings, predicts variational parameters, and feeds vector and image decoders that reconstruct per-point outputs and the distance field.Encoder-decoder training applies KL regularization; without it, training is less stable and latent distributions are more dispersed.Source: Model Architecture · p. 9
  • Training combines vector reconstruction, image reconstruction, and KL-divergence losses, with the image loss including a perceptual term computed using frozen VGG-16.Default weights are λCE = 0.1, λL1 = 10, λimg = 10, and λKL = 0.001.Source: Model Architecture · pp. 9–10
  • Capacity ablations vary embedding dimension, Transformer depth, and attention-head count across five architectures, keeping hidden dimension 64 and sequence length 64 fixed.Each variant is trained from scratch with the same optimization settings.Source: Model Architecture · p. 10
  • The 512-dimensional, 16-layer, 16-head final model outperformed smaller models in reconstruction accuracy and perceptual quality on held-out validation.A 10-epoch linear warm-up stabilized this configuration and was unnecessary for smaller variants.Source: Model Architecture · p. 10

Sketch Generation via Diffusion

StrokeFusion generates sketches by denoising latent stroke sequences with a Transformer, treating strokes as an unordered set. A visibility channel determines which strokes are decoded, allowing variable-length outputs.

Sketch Generation via Diffusion

  • The generator denoises latent stroke sequences with a Transformer without positional encodings, maintaining permutation invariance across strokes.Each vector contains a stroke embedding, bounding-box parameters, and a visibility flag; all attributes are treated as continuous values and noised.Source: Sketch Generation via Diffusion · p. 10
  • The denoising network uses 16 Transformer layers, 16 attention heads, and 512-dimensional embeddings, selected over a larger alternative for efficiency.The 1024-dimensional alternative showed diminishing returns in validation loss and generation quality.Source: Sketch Generation via Diffusion · p. 10
  • Training uses the DDPM framework with a linear β schedule over 1000 timesteps and mean squared error to predict the original noise.The model receives a timestep, a noise-corrupted sequence, and an optional conditional embedding.Source: Sketch Generation via Diffusion · p. 10
  • After the final denoising timestep, the effective stroke count is determined by thresholding visibility: strokes with predicted visibility > 0 are decoded, while others are ignored, integrating variable-length outputs naturally into diffusion.Source: Sketch Generation via Diffusion · p. 10

Training Infrastructure and Runtime

Training and experiments ran on a four-GPU Linux server, with separate reported runtimes for the encoder-decoder and diffusion generator.

Training Infrastructure and Runtime

Metric Computation

Evaluation uses 10,000 generated samples per method, a shared rendering and simplification pipeline, and complementary fidelity and coverage metrics.

Metric Computation

  • Each method is evaluated on 10,000 generated sketches after RDP simplification and shared vector-to-raster rendering.These steps precede feature extraction for metric computation.Source: Metric Computation · p. 10
  • FID measures distributional discrepancy between generated and real sketches in Inception feature space, using their empirical means and covariances.The cited passage introduces FID as the evaluation metric and defines the real and generated feature statistics.Source: Metric Computation · p. 10
  • Precision measures the fraction of generated samples within threshold δ of the real-data manifold.The threshold is adaptively set using the average k-nearest-neighbor distance among real evaluation samples, with k = 20.Source: Metric Computation · pp. 10–11
  • Recall measures the proportion of real samples closely matched by generated outputs, capturing output diversity.The metric is computed by comparing real samples with the generated set.Source: Metric Computation · p. 10

Additional Generation Results.

Additional results show generations across QuickDraw categories and other datasets, emotion-conditioned facial sketches, and a comparison with a set-based raster method.

Additional Generation Results.

Conditional Generation with Emotion Labels.

  • A single emotion-conditioned diffusion model generates facial sketches for anger, disgust, happy, and sad with semantically aligned features.The authors report emotionally consistent sketches reflecting the target emotion.Source: Conditional Generation with Emotion… · p. 11

Comparison with Set-based Raster Methods

  • Against DoodleFormer, StrokeFusion produces comparable visual quality while retaining executable vector trajectories and stroke-level controllability.StrokeFusion treats strokes as set elements without explicit part labels; DoodleFormer uses head/body annotations and produces raster images.Source: Comparison with Set-based Raster Methods · p. 11

Related papers

Loading 2503.23752v4…