PaperlayerView exact revision on arXiv

Source-linked AI summary

ChiroDiff: Modelling chirographic data with Diffusion Models

Ayan Das, Yongxin Yang, Timothy Hospedales, Tao Xiang, Yi-Zhe Song

arXiv:2304.03785v1cs.LGcs.AIcs.CV

TL;DR

Autoregressive models for chirographic data have limited whole-sequence visibility and are commonly tied to fixed temporal sampling rates. CHIRODIFF applies diffusion in a non-autoregressive model that retains sequence structure; the authors report better reconstruction and generation metrics and demonstrate several downstream uses.

  • Problem

    Autoregressive chirographic models have limited visibility of the full sequence and commonly represent continuous-time data at a fixed sampling rate.

    Source: §1 · p. 2

  • Method

    CHIRODIFF uses a non-autoregressive DDPM framework to model continuous-time chirographic data while retaining its sequential nature.

    Source: §1 · pp. 2–3

  • Results

    The authors report better reconstruction and generation metrics, and consistently superior generation FID to SketchRNN and CoSE across three datasets.

    Source: §1 · p. 3, §5.3 · p. 7

  • Takeaways & Limitations

    CHIRODIFF supports conditional sampling, stochastic vectorization, creative mixing, denoising or healing, and controlled abstraction for chirographic data.

    Source: §1 · pp. 2–3

    Vector representations are more susceptible to noise than raster images, and accumulated absolute positions accumulate noise in proportion to their cardinality.

    Source: §6 · p. 9

Abstract

from arXiv · show

Generative modelling over continuous-time geometric constructs, a.k.a such as handwriting, sketches, drawings etc., have been accomplished through autoregressive distributions. Such strictly-ordered discrete factorization however falls short of capturing key properties of chirographic data -- it fails to build holistic understanding of the temporal concept due to one-way visibility (causality). Consequently, temporal data has been modelled as discrete token sequences of fixed sampling rate instead of capturing the true underlying concept. In this paper, we introduce a powerful model-class namely "Denoising Diffusion Probabilistic Models" or DDPMs for chirographic data that specifically addresses these flaws. Our model named "ChiroDiff", being non-autoregressive, learns to capture holistic concepts and therefore remains resilient to higher temporal sampling rate up to a good extent. Moreover, we show that many important downstream utilities (e.g. conditional sampling, creative mixing) can be flexibly implemented using ChiroDiff. We further show some unique use-cases like stochastic vectorization, de-noising/healing, abstraction are also possible with this model-class. We perform quantitative and qualitative evaluation of our framework on relevant datasets and found it to be better or on par with competing approaches.

1 INTRODUCTION

CHIRODIFF applies non-autoregressive diffusion to continuous-time chirographic data while retaining sequence order. It addresses limits of autoregressive visibility and sampling-rate flexibility, and supports conditional sampling and other downstream uses.

1 INTRODUCTION

  • Autoregressive models preserve stroke order but restrict full-sequence visibility, while point-set models discard ordering; CHIRODIFF uses non-autoregressive density while retaining sequence order.The paper identifies this as a middle ground for chirographic data.Source: §1 · p. 2
  • Prior models often represent continuous-time drawings as discrete sequences at one sampling rate, limiting spatial and temporal scalability.Neural ODE approaches address continuous time but are described as computationally difficult to train beyond simple temporal structures.Source: §1 · p. 2
  • CHIRODIFF uses DDPMs to model continuous-time entities, transforming discretized Brownian motion with unit step size into chirographic samples.The paper presents it as an early diffusion-model application to continuous-time entities.Source: §1 · p. 2
  • CHIRODIFF supports conditional sampling without an explicit encoder and considers both unconditional and conditional generation.The stated encoder-free conditional sampling applies when conditioned on homogeneous data.Source: §1 · pp. 2–3
  • The framework supports stochastic vectorization, DDIM latent interpolation, creative mixing, denoising or healing, and controlled abstraction.Creative mixing does not require training only on the interpolation endpoints; the paper also notes loss of stochastic completion capabilities found in autoregressive models.Source: §1 · p. 3
  • The authors report better reconstruction and generation metrics and present downstream applications for continuous-time chirographic data.They describe CHIRODIFF as a diffusion framework for variable-length temporal data.Source: §1 · p. 3

2 RELATED WORK

Prior chirographic models use raster images, autoregressive vector generation, or point-set representations, each with trade-offs. CHIRODIFF combines non-autoregressive generation with retained ordering.

2 RELATED WORK

  • Autoregressive vector models handle variable-length sequences but limit full-data visibility and holistic understanding of temporal concepts.Set representations avoid ordering but fit chirographic data poorly because of its unstructured nature.Source: §2 · p. 3
  • Raster representations are static, whereas topology-specific vector formats aim to model the dynamic drawing process.Set-based chirographic methods require explicit accounting for misalignment.Source: §2 · p. 3
  • Diffusion models are prominent in image generation, but the paper notes comparatively little focus on non-image modalities.The cited related work includes diffusion models for sequential data, often treated as fixed-length entities.Source: §2 · p. 3

3 DENOISING DIFFUSION PROBABILISTIC MODELS (DDPM)

DDPMs generate data by learning a reverse diffusion process from Gaussian noise, trained using samples from a forward noising process. For chirographic data, the paper illustrates the forward and reverse transformations.

3 DENOISING DIFFUSION PROBABILISTIC MODELS (DDPM)

  • A DDPM transforms an isotropic Gaussian prior into the model distribution through T reverse denoising steps with Markov transition distributions.The reverse conditional at each step depends on the current noisy state.Source: §3 · p. 3
  • Sampling starts from the prior and iteratively applies trained reverse conditionals until t = 0.Figure 3 depicts forward and reverse diffusion on chirographic data; its disconnected-line effect comes from diffusing pen-bits together.Source: §3 · p. 4, Figure 3 · p. 4
  • Training simulates latent noisy states with a forward diffusion process because the model log-likelihood is difficult to optimize directly.The forward process is specified by a monotonically decreasing function α_t.Source: §3 · p. 4
  • The forward process forms a noisy sample from the original data and noise, while the model estimates the noise at each step.The passage gives X_t(X_0, ϵ) = √α_tX_0 + √(1 − α_t)ϵ and defines β_t ≜ 1 − α_t/α_(t−1).Source: §3 · p. 4

4 DIFFUSION MODEL FOR CHIROGRAPHIC DATA

CHIRODIFF applies diffusion to variable-length chirographic sequences, representing strokes through velocities and using non-causal sequence context. Its reverse process can generate sequences at lengths beyond those seen in training.

4 DIFFUSION MODEL FOR CHIROGRAPHIC DATA

  • CHIRODIFF represents strokes as point-and-pen-state sequences, models velocities rather than absolute points, and recovers positions by cumulative summation.Using velocities focuses the model on higher-order structure rather than local temporal details.Source: §4 · p. 4
  • The forward sequence-diffusion process noises sequence elements independently, treats pen state as a continuous analog bit, and thresholds it at zero during generation.The terminal prior has standard-normal velocity elements.Source: §4 · pp. 4–5
  • The reverse process uses Gaussian conditional kernels and can be sampled iteratively with DDPM or deterministically with DDIM.The paper interprets generation as transforming a discretized Brownian motion into structured motion.Source: §4 · p. 5
  • A bidirectional sequence encoder gives the denoiser access to the entire sequence, allowing reverse diffusion to correct any part rather than generate causally.The authors report that a Bi-RNN works well and converges faster and better than their Transformer choice; absolute positions are also concatenated with noisy velocities.Source: §4 · p. 5
  • Because the model has no hard sequence-cardinality conditioning, generation can start from a prior of any length, including lengths above training length.The authors hypothesize and empirically show resilience to increased temporal sampling, attributing it to access to the full sequence.Source: §4 · p. 5

5.1 DATASETS

The evaluation uses vectorized digits, complex Kanji characters, and crowd-sourced Quick, Draw! sketches to test performance across different chirographic data.

5.1 DATASETS

  • VectorMNIST contains 10K samples of the digits 0–9 in polyline format and is split into 80-10-10 partitions.It is a vector analog of the traditional MNIST dataset.Source: §5.1 · p. 6
  • KanjiVG is used to evaluate performance on complex chirographic structures with many strokes.The experiments use a preprocessed version converted from SVGs into polyline sequences.Source: §5.1 · p. 6
  • Quick, Draw! supplies noisy, crowd-sourced sketches; the experiments use cat, crab, bus, mosquito, fish, yoga, and flower categories.The paper describes it as an ideal dataset for evaluating effectiveness on real noisy data.Source: §5.1 · p. 6

5.2 IMPLEMENTATION DETAILS

Implementation uses a linear diffusion schedule and bidirectional GRU denoisers, with model size varied by dataset. Training uses AdamW and step-wise learning-rate scheduling.

5.2 IMPLEMENTATION DETAILS

  • The forward process uses a linear noise schedule and T = 1000 diffusion steps, a standard value chosen because performance changed little across diffusion lengths.The schedule scales β_min = 10^-4 and β_max = 2 × 10^-2 by 1000/T.Source: §5.2 · p. 6
  • The noise estimator is a bidirectional GRU: 2 layers with 48 hidden units for VMNIST, and 3 layers with 128 or 96 units for QuickDraw or KanjiVG, respectively.The authors report that Transformers with positional encoding did not achieve reasonable results.Source: §5.2 · p. 6
  • Training uses AdamW, step-wise learning-rate decay, sinusoidal diffusion-time embeddings, and reverse-process variance σ^2_t = 0.8β̃_t.The authors identify this variance choice as effective for generation quality in most cases.Source: §5.2 · p. 6

5.3 QUANTITATIVE EVALUATIONS

The evaluations compare CHIRODIFF with relevant approaches across representation learning, generation, and computational efficiency, using approximately matched model capacity. CHIRODIFF reports stronger generation FID, resilience to higher-rate reconstruction, and easier training with faster convergence.

Evaluation setup

  • The evaluation covers representation learning, generative modelling, and computational efficiency, with competing models chosen for approximately equal parameter counts.The supplied evaluation passages report generation and reconstruction findings, but no separate representation-learning result.Source: §5.3 · p. 6

Reconstruction

  • Reconstruction CD remains resilient as sampling rate rises; SketchRNN fails at high rates, while CoSE and SketchODE have relatively flat curves.SketchODE results are omitted for KanjiVG because of computational and convergence issues; Figure 6 pairs sampling-rate factors 1 and 2.Source: §5.3 · p. 7, Figure 6 · p. 7

Generation

  • CHIRODIFF consistently outperforms SketchRNN and CoSE on generation FID across all three datasets.QuickDraw FID values are averaged over the individual categories used.Source: §5.3 · p. 7

Computational Efficiency

  • CHIRODIFF has easy training dynamics and relatively faster convergence than competing methods.The paper also reports approximate unconditional-generation sampling times.Source: §5.3 · p. 7

5.4 DOWNSTREAM APPLICATIONS

The section applies ChiroDiff to reconstruct plausible vector topologies, sample condition-related outputs, mix concepts, and control abstraction through sampling.

5.4.1 STOCHASTIC VECTORIZATION

  • Stochastic vectorization conditions on a perceptive point-set input to generate plausible chirographic structures and evaluates the reconstructions with Chamfer Distance against Das et al. (2021).The figure highlights samples with different topologies.Source: §5.4.1 · p. 7, Figure 7 · p. 8

5.4.2 IMPLICIT CONDITIONING

  • Implicit conditioning starts reverse denoising from a noisy condition at Tc; higher Tc yields more similar samples, which match the condition’s class 93% of the time on average.This method requires no explicit encoder and is described as stochastic rather than suitable for reconstruction.Source: §5.4.2 · p. 8

5.4.4 CREATIVE MIXING

  • Creative mixing uses DDIM latent interpolation for KanjiVG and VMNIST, while Quick, Draw uses a low-pass-modified DDPM reverse process with equal-length sequences.The Quick, Draw method resamples the conditioning sequence to match the reverse process length.Source: §5.4.4 · pp. 8–9

5.4.5 CONTROLLED ABSTRACTION

  • Controlled abstraction lowers reverse-process variance through k, producing samples with fewer high-frequency details but retained abstract concepts, without retraining for different controls.Figure 10 shows qualitative results on Quick, Draw!.Source: §5.4.5 · p. 9, Figure 10 · p. 9

6 CONCLUSIONS, LIMITATIONS & FUTURE WORK

The conclusion presents ChiroDiff as a non-autoregressive generative model for chirographic data, while identifying noise sensitivity in vector representations as a limitation.

6 CONCLUSIONS, LIMITATIONS & FUTURE WORK

  • ChiroDiff uses a non-autoregressive diffusion model to capture holistic concepts and support downstream tasks, including some described as infeasible for competing autoregressive models.The authors report better holistic modelling that benefits downstream tasks.Source: §6 · p. 9
  • Vector representations are more susceptible to noise than raster images, and reverse-process noise can overwhelm the model-predicted mean.The authors identify adaptive noising based on generation or data cardinality as a possible future improvement.Source: §6 · p. 9, Figure 11 · p. 9
  • Because the representation uses velocities, accumulated absolute positions also accumulate noise in proportion to sequence cardinality.The paper leaves adaptive noising as a potential future improvement.Source: §6 · p. 9

Related papers

Loading 2304.03785v1…