PaperlayerView exact revision on arXiv

Source-linked AI summary

CanvasVAE: Learning to Generate Vector Graphic Documents

Kota Yamaguchi

arXiv:2108.01249v1cs.CV

TL;DR

Vector-document generation remains relatively underexplored, in part because vector data lacks the pixel grid used by mainstream convolutional approaches. CanvasVAE learns document representations from canvas and element attributes and is evaluated with a newly collected design-template dataset. The reported results establish it as a strong baseline for vector-document reconstruction and generation.

  • Problem

    Generative modeling of vector documents is relatively underexplored, and vector data lacks canvas pixels for direct use by mainstream convolutional approaches without typically non-differentiable rasterization.

    Source: §1 · p. 1

  • Method

    CanvasVAE uses a VAE with Transformer-based encoding and decoding to model documents as canvas attributes plus a variable-length sequence of element attributes.

    Source: §1 · p. 1

  • Results

    CanvasVAE successfully reconstructs and generates vector documents on Crello and RICO, with results reported as a strong baseline for vector-document generative modeling.

    Source: §6 · p. 8

  • Takeaways & Limitations

    The work provides a baseline for unconditional generative modeling of vector graphic documents.

    Source: §1 · p. 1

    The mIoU evaluation ignores content attributes and canvas aspect ratio, evaluating only relative element position and size.

    Source: §5.1.1 · p. 5

Abstract

from arXiv · show

Vector graphic documents present visual elements in a resolution free, compact format and are often seen in creative applications. In this work, we attempt to learn a generative model of vector graphic documents. We define vector graphic documents by a multi-modal set of attributes associated to a canvas and a sequence of visual elements such as shapes, images, or texts, and train variational auto-encoders to learn the representation of the documents. We collect a new dataset of design templates from an online service that features complete document structure including occluded elements. In experiments, we show that our model, named CanvasVAE, constitutes a strong baseline for generative modeling of vector graphic documents.

1. Introduction

Vector graphics are compact, resolution-free, editable documents, but generative modeling of them has received relatively little study and poses challenges in representation, learning, and evaluation. CanvasVAE addresses unconditional generation by learning from structured document attributes and is presented as a strong baseline.

1. Introduction

  • Generative modeling of vector documents remains relatively underexplored compared with raster-image generation, despite vector graphics’ importance in creative applications.Vector documents lack canvas pixels, so mainstream convolutional approaches require rasterization, which is typically not differentiable.Source: §1 · p. 1
  • The task entails challenges in representing complex vector formats uniformly, formulating the learning problem, and evaluating document quality.These challenges are stated for generative learning of vector graphics.Source: §1 · p. 1
  • CanvasVAE models unconditional vector-document generation with a VAE over canvas and element attributes, using Transformer-based encoding and decoding for variable-length element sequences.The decoder can sample latent codes to generate new documents; the study also introduces normalized metrics for reconstruction and generation quality.Source: §1 · p. 1
  • The Transformer-based CanvasVAE is reported as a strong baseline for reconstruction and generation, and the Crello dataset supplies complete vector information, including occluded elements.The contributions also include a dataset of design templates and an empirical evaluation of the model.Source: §1 · p. 1

2. Related work

Prior work includes conditional layout generation and vector modeling at stroke or path level. CanvasVAE instead targets unconditional document generation, including layout and other attributes, in symbolic vector data.

Generative layout modeling

  • Conditional layout methods generate bounding-box arrangements from inputs, with prior work spanning label-conditioned layouts, partial relations, text-to-scene generation, and UI layouts.Examples include magazine layouts conditioned on elements and metadata, and models that refine given element geometry.Source: §2 · pp. 1–2
  • Unlike conditional layout generation, CanvasVAE targets unconditional document generation that includes layout and other attributes, operating on symbolic vector data without explicit rasterization.The paper contrasts this with prior approaches that use rasterization and with models that generate layouts without a learned latent representation or without content attributes.Source: §2 · p. 2

Vector graphic generation

  • Other vector-generation work models strokes or paths, including drawing strokes, font strokes, Bézier curves, and sets of paths.The paper draws inspiration from DeepSVG’s hierarchical autoencoder for its oneshot decoding architecture.Source: §2 · p. 2

3. Vector graphic representation

The paper represents each document as a single-page canvas with an ordered sequence of visual elements, described through global canvas attributes and element-specific attributes. Its datasets supply complementary design-template and UI-document settings for modeling and evaluation.

3.1. Document structure

  • A document is represented as a single-page canvas and an ordered sequence of elements, with canvas attributes for global properties and element attributes for each visual element.Element order reflects depth; element attributes include position, size, type, opacity, color, or a texture image, while canvas attributes include properties such as size or category.Source: §3.1 · p. 2

3.2. Datasets

  • Crello numeric attributes are quantized to one-hot representations, while image features support content-aware modeling; color and image attributes are used exclusively for different element types.The paper says discretization implicitly enforces element alignment and accommodates attributes that may not follow a normal distribution.Source: §3.2 · p. 2
  • Crello provides design templates across display formats and complete vector structure, including the appearance of occluded elements, enabling modeling beyond layout alone.The dataset also contains canvases with varied aspect ratios, which the paper identifies as a challenge for raster-based models.Source: §3.2 · pp. 2–3
  • RICO supplies manually annotated mobile UI designs and is used to evaluate CanvasVAE’s generalization ability, with canvas attributes limited to element length.Numeric attributes are also quantized to one-hot representations.Source: §3.2 · p. 3

4. CanvasVAE

CanvasVAE uses a variational auto-encoder to represent vector graphic documents with a probabilistic encoder and decoder. Its encoder maps canvas and element attributes to a latent distribution, while the decoder reconstructs document attributes and handles variable sequence length.

VAE basics

  • The model uses a variational lower bound, a Gaussian approximate posterior with diagonal covariance, and a unit multivariate Gaussian prior over latent codes.The encoder predicts the posterior parameters from the input document.Source: §4 · p. 3

Encoder

  • Canvas and element attributes are projected into embeddings, processed by Transformer blocks, pooled into an internal representation, and mapped to posterior parameters.Element embeddings include learned absolute-position information, and the Transformer blocks use a side input.Source: §4 · pp. 3–4

Decoder

  • The decoder conditions on a sampled latent code and uses a Transformer block to reconstruct document attributes at absolute sequence positions.For generation, it predicts the element count and masks positions beyond that length.Source: §4 · p. 4

Loss function

  • The per-document objective combines attribute reconstruction losses with KL and L2 regularization terms.Categorical attributes use cross-entropy, numerical attributes use mean squared error, and training uses teacher forcing with the ground-truth sequence length.Source: §4 · p. 4

5. Experiments

The experiments evaluate CanvasVAE on reconstruction and generation, using document-similarity measures and comparisons among model variants. Test results vary by dataset and metric, while qualitative examples show reconstruction, sampling, interpolation, and generation behavior.

5.1.1 Reconstruction metrics

  • Reconstruction evaluation combines multimodal structural similarity with box-based mIoU; mIoU omits content attributes and canvas aspect ratio.mIoU evaluates relative element position and size using labeled bounding boxes.Source: §5.1.1 · p. 5

5.1.2 Generation metric

  • Generation similarity compares per-attribute statistics across real and generated document sets, using histogram intersection for categorical attributes and cosine similarity for numerical ones.The method uses histogram aggregation for categorical attributes and average pooling for numerical attributes.Source: §5.1.2 · p. 5

5.2. Baselines

  • Baselines replace Transformer blocks with LSTMs or use autoregressive decoding; no directly comparable prior work was reported.The study also compares one- and four-block Transformer configurations.Source: §5.2 · pp. 5–6

5.3. Quantitative evaluation

  • Crello’s best structure reconstruction and layout mIoU are 88.67% and 47.02% with oneshot Transformer x1; its best generation score is 88.15% with x4, while RICO’s respective winners are autoregressive Transformer x1 (94.96%), oneshot Transformer x1 (60.42%), and x4 (95.14%).Table 2 reports these test results; the best configuration differs across metrics.Source: Table 2 · p. 6, §5.3 · p. 7
  • A smaller λKL improves generation but hurts reconstruction, and oneshot models generally outperform autoregressive counterparts across both datasets.The authors report oneshot Transformer x4 as consistently strong except for structural reconstruction on RICO, where scores largely saturate.Source: Figure 2 · p. 6, §5.3 · p. 7

5.4. Qualitative evaluation

  • For documents with many elements, oneshot models tend to reconstruct better than other baselines.The comparison uses Crello and RICO test examples; Crello visualizations include retrieved image and shape content alongside element-type maps.Source: Figure 3 · p. 6, §5.4 · p. 7
  • Sampling from the encoded latent distribution can change layout arrangement or canvas size; interpolated Crello designs transition gradually despite categorical or retrieved-image changes.The interpolation example spans documents differing in element count and canvas size.Source: §5.4 · p. 7, Figure 6 · p. 8
  • Generated Crello documents vary in layout and aspect ratio and are editable, while RICO samples show diverse, semantically meaningful arrangements that sometimes overlap.The authors describe Crello outputs as inspirational rather than ready for immediate real-world creative use.Source: §5.4 · pp. 7–8, Figure 7 · p. 8, Figure 8 · p. 8

6. Conclusion

CanvasVAE models vector graphic documents through canvas and element attributes, including layout geometry, and learns to reconstruct and generate documents. The results on Crello and RICO form a strong baseline; future work includes text and font generation and appearance-aware objectives.

6. Conclusion

  • CanvasVAE encodes and decodes vector documents represented by canvas and element attributes, including layout geometry.Source: §6 · p. 8
  • CanvasVAE learns to reconstruct and generate vector graphic documents on Crello and RICO, providing a strong baseline for generative modeling.p16Source: §6 · p. 8
  • Future extensions include text content and font styling generation, image-feature pretraining, appearance-aware raster loss, and partial-input constraints.Source: §6 · p. 8

Related papers

Loading 2108.01249v1…