Source-linked AI summary
Im2Vec: Synthesizing Vector Graphics without Vector Supervision
Pradyumna Reddy, Michael Gharbi, Michal Lukac, Niloy J. Mitra
TL;DR
Im2Vec targets vector-graphics generation without the difficult and non-unique vector supervision required by earlier methods. It learns variable-topology vector outputs from raster images through differentiable rendering and compositing, and achieves better reconstruction fidelity than supervised vector baselines. The method also supports vector-space reprojection, sampling, and interpolation, including MNIST vectorization without ground-truth vectors.
Problem
Vector-graphics datasets with high-quality ground-truth sequences are difficult to collect, while multiple vector representations can yield the same appearance.
Method
Im2Vec encodes raster images and decodes them into variable numbers of closed Bézier paths, training through differentiable rasterization and compositing without vector supervision.
Results
Im2Vec achieves better reconstruction fidelity than methods requiring vector supervision across evaluated graphics, including fonts, emojis, and icons.
Takeaways & Limitations
The learned vector representation supports projection, sampling, and interpolation, including topological variations and MNIST vectorization without ground-truth vectors.
Takeaways & Limitations
Raster-based training can lose fine features below the training resolution and can produce degenerate or semantically meaningless shapes with plausible raster appearances.
Abstract
from arXiv · showhide
Vector graphics are widely used to represent fonts, logos, digital artworks, and graphic designs. But, while a vast body of work has focused on generative algorithms for raster images, only a handful of options exists for vector graphics. One can always rasterize the input graphic and resort to image-based generative approaches, but this negates the advantages of the vector representation. The current alternative is to use specialized models that require explicit supervision on the vector graphics representation at training time. This is not ideal because large-scale high quality vector-graphics datasets are difficult to obtain. Furthermore, the vector representation for a given design is not unique, so models that supervise on the vector representation are unnecessarily constrained. Instead, we propose a new neural network that can generate complex vector graphics with varying topologies, and only requires indirect supervision from readily-available raster training images (i.e., with no vector counterparts). To enable this, we use a differentiable rasterization pipeline that renders the generated vector shapes and composites them together onto a raster canvas. We demonstrate our method on a range of datasets, and provide comparison with state-of-the-art SVG-VAE and DeepSVG, both of which require explicit vector graphics supervision. Finally, we also demonstrate our approach on the MNIST dataset, for which no groundtruth vector representation is available. Source code, datasets, and more results are available at geometry.cs.ucl.ac.uk/projects/2021/Im2Vec/
1. Introduction
Im2Vec addresses the difficulty of generating vector graphics by learning from raster images alone while preserving flexible, composable vector structure. Its differentiable representation supports complex topologies, generation, interpolation, and reconstruction without vector supervision.
- Motivation: Vector graphics offer compact, infinitely scalable, and editable representations, but creating them remains largely limited to manual expert workflows.Their irregular structure is also poorly suited to convolution-based generative architectures.
- Capabilities: Im2Vec supports reprojection, sampling, and interpolation in the learned vector space, including morphing across topological variations.The learned representation is demonstrated on fonts, emojis, icons, and other graphics.
- Motivation: Existing generative approaches such as SVG-VAE and DeepSVG require ground-truth vector sequences, which are difficult to collect and non-unique for a given appearance.Different drawing sequences can produce the same visual result, introducing structural constraints and bias.
- Approach: Im2Vec learns a vector-graphics latent space from image supervision, using differentiable rasterization and compositing to connect generated shapes with raster appearance.The approach treats the representation–appearance relationship as explicit and differentiable.
- Approach: Its representation uses variable-complexity closed Bézier paths and composable primitives, enabling graphics with varying numbers of elements, complexities, and topologies.Primitive closed shapes are modeled as deformations of a unit circle conditioned on a common latent vector.
- Results: Im2Vec achieves better reconstruction fidelity than SVG-VAE and DeepSVG while retaining vector output’s editability and compactness benefits.The method is also demonstrated for vectorizing MNIST without ground-truth vector representations.
2. Related Work
Prior work covers image vectorization, parametric shape estimation, drawing programs, and supervised generative vector models. Im2Vec differs by learning complex, variable-topology vector graphics from raster data alone.
- Learning-based image vectorization: Image-vectorization methods predict vector primitives from raster inputs, including technical drawings and variable-length parametric curves.These approaches include transformer-based prediction, optimization refinement, and hierarchical recurrent curve generation.
- Parametric shape estimation: Parametric shape-estimation methods commonly assemble primitives with fixed topology or cardinality and often supervise directly on shape parameters.Examples include quadratic Bézier curves, cuboids, and class-dependent templates.
- Shape-generating programs: Shape-generating programs use simulated rendering and image-based rewards to avoid direct drawing-program supervision, but some methods remain limited in primitive complexity or raster resolution.CSGNet, for example, outputs 32 × 32 raster images, while other systems use reinforcement-learning drawing agents.
- Generative vector graphics: Earlier generative vector-graphics models such as SVG-VAE and DeepSVG focus on vector representations but require vector supervision or operate in constrained settings.SVG-VAE uses two-stage training, whereas DeepSVG demonstrates hierarchical vector generation and interpolation.
- Generative vector graphics: Im2Vec uses differentiable rasterization as a building block to generate closed curves with complex, varying topologies without artifacts such as overlapping paths.This extends beyond the generative models described in the related work.
3. Method
Im2Vec uses an encoder–decoder architecture that maps raster images to latent codes and decodes them into vector graphics with variable numbers of paths and unconstrained topology.
- Architecture: The encoder maps a raster image I to a latent variable z, which the decoder transforms into a vector graphic structure.The model is designed to train using raster images alone rather than vector supervision.
- Vector graphics decoder: The decoder can generate a variable number T of paths with varying lengths and no predetermined topology.An RNN produces per-path latent codes, which a path decoder converts into closed paths with cubic Bézier segments.
3.1. Vector Graphics Decoder
The decoder represents graphics as depth-ordered collections of closed Bézier paths with variable counts and lengths. Circular convolutions deform unit-circle samples into paths, while recurrent path generation supports multiple components and adaptive complexity.
- Im2Vec represents each graphic as a depth-ordered set of T closed Bézier paths, enabling variable path counts, lengths, and topologies.
- An RNN decomposes the global latent code into per-path codes, which individual path decoders convert into closed cubic Bézier paths.
- Each path begins from 3k samples on a unit circle, with every third point marking a Bézier segment endpoint.
- Cyclic 1D convolutions share information between neighboring circle samples while producing final spatial locations for the path control points.
- Sampling density directly controls path complexity, allowing the number of Bézier segments to vary across paths and designs.
- Adaptive sampling deforms the unit-circle positions to allocate control points non-uniformly and improves reconstruction, especially with few segments.Its advantage over uniform sampling diminishes as more curve segments are added.
3.2. Predicting the number of path control points
The model predicts how many Bézier segments each path needs by modeling the complexity–fidelity trade-off. A user-selected derivative threshold converts this estimate into a segment count.
- Latent-space deformations of a topological disk provide point-to-point correspondences across single-path and multi-path graphics.
- An auxiliary network predicts each path’s complexity–fidelity curve and automatically determines its number of Bézier segments.
- The network outputs a, b, and c for the curve x 7→ae^-bx + c, approximating reconstruction loss as segment count changes.
- A user sets the quality–complexity trade-off through a derivative threshold, which is solved and rounded up to obtain the segment count.
3.3. Multi-resolution raster loss
Im2Vec supervises vector generation by rasterizing decoded paths and comparing the resulting image with the raster input across multiple resolutions. The image pyramid stabilizes optimization while retaining fine-scale accuracy.
- The decoder rasterizes each closed Bézier path, composites the paths into output raster O, and compares O with the input image I.
- Rasterization gradients have a small spatial support, which can hinder convergence when the generated and target images differ substantially.
- The method addresses this by rendering an image pyramid and aggregating losses against a Gaussian pyramid of the target image.
- Coarse-resolution gradients provide stable signals for large mismatches, while fine-scale gradients support high spatial accuracy.
3.4. Shape correspondences by segmentation
To make latent representations more interpretable, the method segments raster training images, clusters segments by spatial position, and assigns consistent colors to corresponding clusters.
- An automatic segmentation tool divides raster training images into components that can be compared across the dataset.
- Segments are clustered by spatial position and each cluster receives a unique RGB color, creating consistent labels across instances.
- These labels help the model learn latent correspondences between recurring parts, such as letter openings or facial features.
3.5. Training details
Im2Vec is trained end-to-end with dataset-dependent settings and randomizes the number of path segments to improve generalization.
- Training lasts 100–1000 epochs with batch sizes of 2–256 and learning rates between 10^-3 and 10^-4, depending on the dataset.The model uses the Ranger optimizer.
- The path decoder is trained with a randomly selected number of segments k ∈ {7, . . . , 25} at every iteration.This evaluates generalization to graphics containing different numbers of segments.
4. Evaluation
The evaluation compares Im2Vec with raster- and vector-based baselines across reconstruction, generation, and interpolation. Im2Vec produces vector outputs and achieves strong reconstruction and generative quality, including on MNIST, which lacks vector supervision.
- Evaluation tasks: Im2Vec is evaluated on reconstruction, generation, and interpolation against ImageVAE, SVG-VAE, and DeepSVG.The comparisons cover datasets including FONTS, MNIST, EMOJIS, and ICONS.
- Reconstruction: Reconstruction quality is measured with pixel-space L2 loss after rasterizing vector predictions from SVG-VAE and DeepSVG.This evaluates how accurately each latent space captures the training dataset in image space.
- Qualitative results: On FONTS, Im2Vec captures complex topologies and produces vector outputs, while ImageVAE produces limited-resolution raster outputs and SVG-VAE and DeepSVG often miss complex fonts.All methods were trained on the same font set.
- Reconstruction: Im2Vec achieves significant reconstruction improvement because it optimizes image appearance directly rather than vector parameters.The paper attributes the baselines’ difficulty to the nonlinear relationship between vector parameters and pixel appearance.
- Qualitative results: On MNIST, Im2Vec produces vector outputs and convincing interpolations despite training data containing only raster images.SVG-VAE and DeepSVG cannot be trained on MNIST because vector ground truth is unavailable.
- Generation and interpolation: Im2Vec enables plausible interpolation across substantial shape changes, including a digit ‘9’ whose stem naturally curls along the path.The learned latent space supports interpolation across EMOJIS, ICONS, and FONTS.
- Generation and interpolation: Generation and interpolation results on FONTS and MNIST are more accurate than both vector-supervised methods and the image-based baseline.This comparison is summarized in Table 2.
- Generation and interpolation: Generation quality is assessed using the average closest distance from generated shapes to training samples, based on 1000 variationally sampled shapes.Interpolation quality is evaluated using four evenly spaced samples from 250 random training-image pairs.
5. Limitations
Im2Vec’s raster-based supervision limits its ability to preserve fine features and prevent degenerate geometries. Higher-resolution supervision and geometric constraints are proposed as mitigations.
- Fine features may fall below the training resolution and be lost.Increasing resolution could address this limitation, but at the expense of computational efficiency.
- Generated shapes can reach local optima containing degenerate or semantically meaningless parts that still produce plausible raster images.The paper attributes this failure mode to the absence of vector supervision.
- Strengths alongside limitations: Latent-space interpolation can remain plausible even across significant shape changes, such as a digit ‘9’ whose stem curls naturally.This capability is demonstrated on FONTS, EMOJIS, and ICONS.
- Strengths alongside limitations: Random samples from the Fonts-trained latent space show substantial topological variation across font types.The paper provides 1000 random latent-space samples in supplementary material.
- Geometric constraints on generated paths could mitigate semantically non-meaningful or degenerate features.The proposed remedy targets failures that remain visually plausible after rasterization.
6. Conclusion
Im2Vec generates vector graphics of varying complexity and topology using only image supervision, while supporting projection, sampling, and interpolation. Its evaluations report better reconstruction fidelity than methods requiring vector supervision.
- Im2Vec produces vector graphics with varying complexity and topology without vector sequence guidance.The model is trained using only image supervision.
- The generative setup supports projection, sampling, and interpolation, including morphing between vector sequences with topological variations.Projection converts images to vector sequences, while sampling generates new vector shapes.
- Im2Vec achieves better reconstruction fidelity than methods requiring vector supervision.The conclusion states this as the outcome of the evaluations.
- The authors hope Im2Vec can become a building block for neural processing of vector graphics and similar parametric shapes.