Source-linked AI summary
A Learned Representation for Scalable Vector Graphics
Raphael Gontijo Lopes, David Ha, Douglas Eck, Jonathon Shlens
TL;DR
The paper addresses how to learn structured, higher-level representations for visual generation rather than relying only on exhaustive pixel modeling. It combines a class-conditioned VAE with a sequential SVG decoder to model font drawing processes. The resulting scale-invariant latent representation supports style manipulation and propagation on a 14 M-character font dataset, while the authors also identify failures from the model’s sequential, stochastic nature.
Problem
The paper studies how to learn structured representations for visual concepts and generate drawings as SVG command sequences rather than pixels.
Method
The model combines a class-conditioned VAE for rendered fonts with an autoregressive decoder that generates SVG commands from class labels and latent style representations.
Results
The model provides a scale-invariant latent representation for font styles and is demonstrated on a dataset of 14 M font characters, including style propagation across fontsets.
Takeaways & Limitations
The model may serve as an assistive agent for helping humans design fonts more efficiently through style propagation.
Takeaways & Limitations
The sequential, stochastic representation can accumulate drawing errors over time and fail to correct early mistakes.
Abstract
from arXiv · showhide
Dramatic advances in generative models have resulted in near photographic quality for artificially rendered faces, animals and other objects in the natural world. In spite of such advances, a higher level understanding of vision and imagery does not arise from exhaustively modeling an object, but instead identifying higher-level attributes that best summarize the aspects of an object. In this work we attempt to model the drawing process of fonts by building sequential generative models of vector graphics. This model has the benefit of providing a scale-invariant representation for imagery whose latent representation may be systematically manipulated and exploited to perform style propagation. We demonstrate these results on a large dataset of fonts and highlight how such a model captures the statistical dependencies and richness of this dataset. We envision that our model can find use as a tool for graphic designers to facilitate font design.
1. Introduction
The paper frames font generation as learning structured, higher-level representations rather than exhaustively modeling pixels. It proposes SVG-based generation to capture font diversity and support systematic style manipulation and propagation.
- Structured visual representations can support higher-level primitives for generalization, efficient learning, and creative exploration.
- SVG represents drawings as compact, scale-invariant sequences of higher-level commands paired with numerical arguments.
- The paper builds a generative model for SVG images using a large-scale dataset of 14 M font characters.
- The model is intended to provide a perceptually smooth latent representation that captures font-style diversity consistently across characters.
- The latent representation is used to infer complete SVG fontsets from one or multiple characters of a font.
- The model identifies semantically meaningful latent directions for globally manipulating font style.
2. Related Work
Related work spans high-quality image generation, probabilistic and autoregressive sequence models, graphics-program induction, and font-style representations. The paper positions its SVG approach within these efforts while addressing sequential generation of vector commands.
- Generative image models have produced high-resolution imagery that is nearly indistinguishable from real photographs.
- Invertible probabilistic models provide tractable objectives such as log-likelihood without saddle-point training instabilities.
- Conditional autoregressive prediction decomposes joint generation into sequential tasks and is commonly trained with teacher forcing.
- Autoregressive models combined with density modeling have been used for sequential handwriting generation.
- Graphics-program induction methods infer programs for graphics engines, but non-differentiable setups may use REINFORCE and can draw over earlier content.
- Prior font-style methods use simplified features or parametric controls, while manifold-learning approaches model rasterized styles or character outlines.
3. Methods
The method combines a class-conditioned VAE over rendered fonts with an autoregressive SVG decoder that generates command sequences. Training uses normalized SVG data, separate optimization stages, and stochastic sampling for evaluation.
- The SVG-Fonts dataset contains 14 M examples across 62 characters drawn from roughly 220 K fonts, with residual label noise.
- Each example is mapped from SFD to SVG using four commands: moveTo, lineTo, cubicBezier, and EOS.
- Commands are normalized by starting at the top-most command, ordering clockwise, and using relative argument positions.
- The dataset stores discrete SVG commands with normalized floating-point arguments and limits examples to fewer than 50 commands.
- A class-conditioned convolutional VAE learns a largely class-independent latent code z, while an autoregressive decoder generates SVG commands.
- The decoder uses four stacked LSTMs and a Mixture Density Network, combining command cross-entropy with a loss for real-valued arguments.
- The VAE is trained on pixel renderings for 3 epochs, then frozen while the SVG decoder is trained with teacher forcing.
- Evaluation samples the probabilistic VAE and MDN repeatedly, reporting the best result from 10 samples.
4. Results
The model learns a smooth, style-focused latent representation that supports consistent font generation, style propagation across character labels, and controllable manipulation of font attributes. Its performance varies with character class, sequence length, stylistic complexity, and stochastic decoding errors.
- 4.1. Learning a smooth, latent representation of font style: The learned latent space exhibits smooth transitions in font style, including changes in serif amount and boldness.UMAP visualization of 1 M examples reveals perceptually smooth regions with interpretable stylistic variation.
- 4.1. Learning a smooth, latent representation of font style: Linear interpolation between latent codes produces smooth SVG transitions despite each character requiring 15–30 commands.The decoded SVGs remain perceptually smooth even though their underlying command sequences are discrete and multi-step.
- 4.2. Exploiting the latent representation for style propagation: A single character yields a perceptually similar fontset across labels, while different latent codes capture substantial style diversity.The same latent value generates visually consistent characters without explicit cross-label consistency training, and the latent space preserves variety across fonts.
- 4.2. Exploiting the latent representation for style propagation: Conditioning on more characters systematically improves style consistency and the quality of generated icons.The method averages latent representations from multiple characters before propagating style to new characters.
- 4.3. Building style analogies with the learned representation: Latent-space directions support smooth, global manipulation of boldness, italics, and condensed style.Adding or subtracting concept vectors produces controlled changes in the corresponding font attributes across decoded SVGs.
- 4.4. Quantifying the quality of the learned representations: The model performs less reliably for longer sequences, certain character classes, and highly stylized characters, with errors accumulating during stochastic generation.Low-likelihood decisions can trigger unrecoverable drawing errors, while higher latent uncertainty corresponds to lower-quality SVG outputs.
5. Discussion
The model offers a scale-invariant latent representation for manipulating font styles and propagating styles from a few designed characters. The discussion also identifies sequence length, stochastic decoding, and latent-space variance as practical limitations and directions for broader vector-graphics modeling.
- The model provides a scale-invariant latent representation that can be manipulated for font-style propagation.The authors envision using it to synthesize remaining characters after a human designs a small set.
- Style propagation can synthesize complete fontsets from a single or multiple characters, supporting more time-efficient font design.
- The model’s limitations arise from its sequential, stochastic structure when capturing the dataset’s statistical dependencies and richness.
- SVG modeling beyond fonts remains an open direction because icons and human drawings contain greater diversity and longer stroke sequences.
- Adding colors, brush strokes, and other illustration tools could increase the expressivity of learned vector-graphics models.
A.1. Dataset details
The dataset contains millions of font characters converted into constrained SVG command sequences for learning. Preprocessing limits command complexity and introduces rendering and label-related boundaries that affect coverage.
- The dataset contains 14M font characters across the classes 0-9, a-z, and A-Z, collected in the SFD format.Characters were retained according to Unicode identifiers, although this filtering permits some label noise.
- SFD icons are converted to SVG and modeled as single paths using moveTo, lineTo, cubicBezierCurve, and EOS commands.
- Characters with more than 50 commands are filtered out, while relative positioning, rescaling, and standardized command ordering aid learning.
- Rendering characters as 64x64 pixel images requires a crop that captures descenders but cannot simultaneously preserve all size and style differences.
- Each character is represented as command-type one-hot vectors paired with normalized numerical arguments, using four command types including EOS.
A.2. Details of network architecture
The network combines a convolutional variational autoencoder with an autoregressive SVG decoder. The encoder produces the latent image representation, while stacked LSTMs and a mixture density output generate SVG commands and arguments.
- The model consists of a convolutional variational autoencoder and an autoregressive SVG decoder.
- The image encoder uses convolution, conditional instance normalization, and ReLU blocks to produce z, with stochastic sampling during training and z = µ at test time.
- The convolutional image encoder and decoder contain 416,672 and 516,865 parameters, respectively.
- The SVG decoder uses four stacked LSTMs and a top Mixture Density Network conditioned on z, the class, and the previous sampled output.
- The decoder has 34,875,272 parameters.
A.3. Training details
Training uses separate reconstruction, regularization, command-classification, and argument-density objectives for the VAE and SVG decoder. The models are optimized with teacher forcing, Adam, dropout-related settings, and specified batch sizes.
- The image VAE optimizes log-likelihood reconstruction and KL losses with KL-beta 4.68, 4.8 free bits, and batch size 64.
- The SVG decoder combines softmax cross-entropy for command types with an MDN loss for real-valued arguments.The softmax term is scaled by 10 during training.
- The SVG decoder is trained with teacher forcing and batch size 128.
- The models use He et al. initialization and the Adam optimizer with ϵ = 10^-6.
A.4. Visualization details
The latent space is visualized by reducing activations from one million examples to two UMAP components, discretizing the result into buckets, and decoding grid-cell averages.
- UMAP reduces activations z from 1M dataset examples to 2 components using cosine similarity.The visualization uses a minimum distance of 0.5 and 50 nearest neighbors.
- The resulting 2D space is discretized into 50 buckets for grid-based visualization.
- The image decoder generates a visualization by decoding the average z in each grid cell.
A.5. Samples from generated font sets
Additional generated-font samples illustrate both successful and poor outcomes, with examples selected as the best of 10 samples and failures associated with a high-variance latent region.
- Additional samples highlight both successes and failures of the font-generation model.These examples supplement selected figures that demonstrate model performance and other results.
- The displayed results are selected as the best out of 10 generated samples.
- Figure 10 provides additional examples of randomly generated fonts.
- Highly stylized characters cluster in a high-variance latent-space region associated with poor-quality SVG font samples.