PaperlayerView exact revision on arXiv

Source-linked AI summary

Vitruvion: A Generative Model of Parametric CAD Sketches

Ari Seff, Wenda Zhou, Nick Richardson, Ryan P. Adams

arXiv:2109.14124v2cs.LG

TL;DR

Parametric CAD design can be challenging and tedious, with engineers repeating similar operations many times. Vitruvion autoregressively generates primitives and constraints, and the authors demonstrate conditional synthesis including converting hand-drawings to parametric designs.

  • Problem

    Parametric CAD design is challenging and tedious, with engineers sometimes repeating similar operations many times per design.

    Source: §1 · p. 2

  • Method

    Vitruvion autoregressively samples geometric primitives and constraints to generate CAD sketches, with optional conditioning on contexts such as primers or images.

    Source: §1 · p. 2, §3.3 · p. 6

  • Results

    The authors demonstrate conditional synthesis capabilities, including converting hand-drawings to parametric designs.

    Source: §5 · p. 9

  • Takeaways & Limitations

    Generated constraint graphs can be handed to a solver and allow automatic edit propagation in standard CAD software.

    Source: §1 · p. 2

    The authors frame deployment as part of a human-in-the-loop process, with human expertise needed for validation and deployment, particularly for high-stakes mechanical systems.

    Source: §6 · p. 10

Abstract

from arXiv · show

Parametric computer-aided design (CAD) tools are the predominant way that engineers specify physical structures, from bicycle pedals to airplanes to printed circuit boards. The key characteristic of parametric CAD is that design intent is encoded not only via geometric primitives, but also by parameterized constraints between the elements. This relational specification can be viewed as the construction of a constraint program, allowing edits to coherently propagate to other parts of the design. Machine learning offers the intriguing possibility of accelerating the design process via generative modeling of these structures, enabling new tools such as autocompletion, constraint inference, and conditional synthesis. In this work, we present such an approach to generative modeling of parametric CAD sketches, which constitute the basic computational building blocks of modern mechanical design. Our model, trained on real-world designs from the SketchGraphs dataset, autoregressively synthesizes sketches as sequences of primitives, with initial coordinates, and constraints that reference back to the sampled primitives. As samples from the model match the constraint graph representation used in standard CAD software, they may be directly imported, solved, and edited according to downstream design tasks. In addition, we condition the model on various contexts, including partial sketches (primers) and images of hand-drawn sketches. Evaluation of the proposed approach demonstrates its ability to synthesize realistic CAD sketches and its potential to aid the mechanical design workflow.

1 INTRODUCTION

Parametric CAD encodes design intent in constraints, so edits can propagate coherently, but repetitive manual design and converting rough drawings into editable models remain challenging. Vitruvion autoregressively generates primitives and constraints, and explores conditioning on incomplete sketches and hand-drawn images.

1 INTRODUCTION

  • Parametric CAD represents design as a constraint program, allowing numerical modifications to propagate coherently while preserving geometric relationships.The workflow supports parameterized variations and structured representations for downstream tasks such as simulation and manufacturing.Source: §1 · pp. 1–2
  • Conditioned on a hand-drawn image, the model generates primitives, then constraints, yielding a parametric sketch that can be imported and edited in CAD software.The workflow can ultimately produce a 3D model.Source: Figure 1 · p. 1
  • The approach targets tedious repetitive design work and the sought-after conversion of hand drawings or noisy inputs into parametric, editable models.Design motifs may recur across related parts, while engineers may repeat similar operations many times per design.Source: §1 · p. 2
  • Vitruvion autoregressively samples geometric primitives and constraints, then passes the resulting constraint graph to a solver for the final sketch configuration.Constraint supervision comes from SketchGraphs, which pairs millions of real-world sketches with ground-truth geometric constraint graphs.Source: §1 · p. 2
  • The model is evaluated with log-likelihood and distributional statistics and demonstrated in autocomplete, autoconstrain, and image-conditional synthesis scenarios.The paper reports realistic sketch synthesis and potential to aid the mechanical design workflow.Source: §1 · p. 2

2 RELATED WORK

Related work spans CAD sketch generation, geometric program synthesis, vector graphics, and graph modeling. Vitruvion’s distinction is to model primitives and constraints together, including hand-drawn conditioning and noisy-input constraint inference.

2 RELATED WORK

  • In CAD, a “sketch” means the combination of geometric primitives and constraints, rather than a colloquial drawing.Source: §2 · p. 2
  • SketchGraphs provides ground-truth geometric constraint graphs and original design steps for millions of real-world Onshape sketches.Earlier models used message-passing networks without learned primitive positions, so output configurations were determined by constraints alone.Source: §2 · p. 3
  • Concurrent transformer-based CAD work varied in scope: Willis et al. modeled primitives without constraints, while Ganin et al. and Para et al. modeled both primitives and constraints.Unlike Vitruvion, the latter works did not condition on hand-drawn sketches or train constraint inference with noisy inputs.Source: §2 · p. 3
  • Geometric program synthesis includes inferring geometry-reconstruction instructions from input specifications, but prior work generally did not consider CAD constraints.Vitruvion’s hand-drawing setting instead infers intended primitives and constraints from a raster image, learning to ignore input noise rather than trace it faithfully.Source: §2 · pp. 3–4
  • Self-attention lets Vitruvion model geometric constraint graphs without edge-specific message passing, a relevant choice because these graphs are sparse.The passage notes that constraints grow at most linearly with the number of primitives and that neighbor-to-neighbor communication can impede learning when distant-node communication is needed.Source: §2 · p. 4

3 METHOD

Vitruvion models CAD sketches as ordered sequences of primitives and constraints, then uses a geometric solver to produce the final sketch. Its primitive model can condition on prefixes or hand-drawn images, while the constraint model generates relationships from the primitives.

3 METHOD

  • The pipeline factorizes sketch generation into primitive sampling, constraint sampling conditioned on primitives, and solving the resulting pair with a standard geometric constraint solver.The model assumes constraints are conditionally independent of context given the primitives.Source: §3 · p. 4

3.1 PRIMITIVE MODEL

  • The primitive model autoregressively generates ordered primitive tokens, using SketchGraphs design-step order to support prefix-conditioned completion.It compresses Onshape parameters to canonical forms, normalizes sketch coordinates, and applies 6-bit quantization before modeling.Source: §3.1 · p. 5

3.2 CONSTRAINT MODEL

  • Given primitives, the constraint model autoregressively generates constraint tuples and uses variable-length pointer distributions to reference primitives or their sub-primitives.The model covers categorical constraints with one or two references, excluding numerical-parameter and hyperedge constraints.Source: §3.2 · p. 6

3.3 CONTEXT CONDITIONING

  • For primer conditioning, the model continues an incomplete primitive prefix; for image conditioning, an image encoder supplies patch embeddings to the primitive decoder.Training images are augmented with simulated hand-drawing distortions, including primitive translations and rotations and Gaussian-process raster noise.Source: §3.3 · pp. 6–7

4 EXPERIMENTS

The experiments assess Vitruvion on SketchGraphs using held-out predictive and distributional measures, then test primitive generation with image and prefix context and constraint inference from noisy primitives. Results show that primitive ordering and noise augmentation matter, while conditional models can interpret hand drawings and complete partial sketches.

4.1 TRAINING

  • Models are evaluated on held-out SketchGraphs data using NLL, predictive accuracy, generated-sketch statistics, and conditional synthesis tasks.The filtered dataset contains sketches with four common primitive types, at most 16 primitives, at least six primitives, and non-empty constraints; deduplication yields 1.7 million unique sketches.Source: §4 · p. 7, §4.1 · p. 7

4.2 BASELINES

  • Shuffling primitive sequences reduces next-step prediction performance, indicating that true ordering supplies useful local hierarchical and global temporal structure.The unconditional model achieves substantially lower NLL than the uniform and compression baselines; Figure 5 also shows lower NLL at later positions and a four-step staircase associated with rectangle prevalence.Source: §4.3 · p. 8, Figure 5 · p. 8

4.3 PRIMITIVE MODEL

  • Hand-drawn simulation and its affine augmentations substantially improve image-conditional performance, and the model interprets hand drawings despite never seeing them during training.The evaluation pairs tablet-drawn sketches in a 128×128-pixel bounding box with their ground-truth primitive sequences.Source: §4.3 · p. 8
  • With 40% of a test sketch’s primitives removed from the suffix, the remaining prefix can prompt plausible completions, sometimes recovering the original sketch among six samples.The authors envision users querying a CAD program for multiple completions and selecting one that matches their intended sketch.Source: §4.3 · pp. 8–9

4.4 CONSTRAINT MODEL

  • Noise-augmented constraint training substantially improves performance on noisy primitives, while both models perform similarly on noiseless inputs.The model trained on original primitive locations fails to generalize outside that distribution; noise augmentation is important for imprecise placements such as those in the image-conditional setting.Source: §4.4 · p. 9

5 CONCLUSION AND FUTURE WORK

Vitruvion models and synthesizes parametric CAD sketches using self-attention to propagate sketch information during autoregressive generation. The paper demonstrates conditional synthesis, including converting hand drawings to parametric designs, and identifies 3D inputs as future work.

5 CONCLUSION AND FUTURE WORK

  • Vitruvion models and synthesizes parametric CAD sketches, using self-attention during autoregressive generation and demonstrating conditional synthesis including hand-drawing conversion.Future work includes modeling 3D construction operations and synthesis conditioned on noisy 3D scans or voxel representations.Source: §5 · p. 9

6 ETHICS STATEMENT

The authors acknowledge possible societal impacts from labor elimination and risks from automatically synthesizing mechanical systems that fail human structural standards. They situate the work within a human-in-the-loop process involving expert validation and deployment.

6 ETHICS STATEMENT

  • The authors identify possible labor displacement and high-stakes structural risks, and frame human expertise as part of validation and deployment.They describe the work as part of a larger human-in-the-loop process.Source: §6 · p. 10

A ADDITIONAL EVALUATION

Additional evaluation examines primer-based completion, constraint correction under coordinate noise, and distributional statistics of generated sketches. The reported comparisons show that noise-augmented constraint training corrects perturbed placements more effectively than training without noise.

A ADDITIONAL EVALUATION

  • Primer-conditioned evaluation deletes 40% of test-sketch primitives and uses the remaining prefix to generate completions until the stop token.Only the primitive model is used in these examples.Source: Figure 7 · p. 12
  • Noise-augmented constraint training more often corrects Gaussian-perturbed primitive locations than training without noise.The additional comparison uses test sketches with perturbed coordinates; the unaugmented model often fails to correct their locations.Source: Figure 8 · p. 12

A.1 DISTRIBUTIONAL STATISTICS

  • The distributional comparison samples 10K unconditional sketches and measures primitive counts, constraint counts, and total, removed, and net degrees of freedom.Figure 9 shows generated-versus-test-set statistics with bootstrapped 95% confidence intervals.Source: §A.1 · p. 12, Figure 9 · p. 13

A.2 ABLATION: TEST-TIME ORDERING SHUFFLING

The ordering-shuffling ablation shows that primitive modeling depends on sequence order. The accompanying appendix details failure cases, hand-drawn noise modeling, tokenization and reference assignments, and the effect of quantization bit width on reconstruction.

A.2 ABLATION: TEST-TIME ORDERING SHUFFLING

  • 104.7 bits/sketch on randomly permuted test sequences versus 66.6 bits/sketch on the original test set shows substantial sensitivity to primitive order.The model was trained on standard data and evaluated on randomly permuted sequences.Source: §A.2 · p. 13, Table 5 · p. 13

A.3 TYPICAL FAILURE CASES

  • Typical failures include non-axis-aligned hand-drawn lines, which are less prevalent in the data, and incorrect constraint references on noise-injected primitives.A tighter nucleus sampling parameter of 0.7 substantially mitigated incorrect references.Source: §A.3 · p. 13

B HAND-DRAWN NOISE MODEL

  • The hand-drawn noise model renders lines and arcs as Matérn Gaussian-process paths, with scale choices made consistent across primitive lengths.Lines use a zero-mean Matérn-3/2 process rotated to the correct orientation; arcs use paths in polar coordinates.Source: B HAND-DRAWN NOISE MODEL · p. 14

C SYSTEM DETAILS

  • Primitive and constraint types use reserved value tokens, while token sequences encode primitive values, IDs, and positions.The token mapping figures illustrate parameter-to-token assignments and a line-point-line sequence.Source: Table 6 · p. 14, C SYSTEM DETAILS · p. 14, Figure 11 · p. 15
  • Constraint generation uses learned primitive encodings to represent whole primitives or specific sub-primitives as reference targets.For a line, indices 0, 1, and 3 represent the line, first endpoint, and second endpoint; the mapping is described as arbitrary.Source: C SYSTEM DETAILS · pp. 15–16

D QUANTIZATION STUDY

  • The model uses 6-bit quantization; the illustrated reconstructions span 2-bit values through single-precision floating point.The figure notes that radius uses half the bins because it must be positive, while each bin represents an invariant numerical interval across parameters.Source: Figure 10 · p. 14, Figure 13 · p. 16

E EXPERIMENTAL DETAILS

The experimental-details section describes shared transformer components and the additional image encoder used for image-to-primitive generation.

E EXPERIMENTAL DETAILS

E.1 MODEL ARCHITECTURE

  • All models share an identical standard transformer decoder for processing primitive or constraint sequences, with 12 layers, 8 attention heads, and a total embedding dimension of 256; no significant hyperparameter optimization was performed.Source: §E.1 · p. 16
  • The image-to-primitive model uses a vision-transformer encoder for 128 × 128 images, split into 64 flattened 16 × 16 patches and projected to dimension 256.The primitive decoder cross-attends to the encoded image representations.Source: §E.1 · p. 17

E.2 TRAINING

Training used Adam with decoupled weight decay and a one-cycle learning-rate schedule, with model-specific epochs and batch sizes.

E.2 TRAINING

  • Primitive and constraint models trained for 30 epochs, while the image-to-primitive model trained for 40; epoch count did not appear to significantly affect performance.Source: §E.2 · p. 17
  • Training took just under 2 hours for the primitive model and 5.5 hours for both the image-to-primitive and constraint models.Runs used servers equipped with four Nvidia V100 GPUs.Source: §E.2 · p. 17
  • The models used Adam with decoupled weight decay, a one-cycle learning-rate schedule, and an initial learning rate of 3e-5 at reference batch size 128.Batch sizes per GPU were 1024 for the primitive model, 512 for image-to-primitive, and 384 for the constraint model.Source: §E.2 · p. 17

E.3 IMAGE-TO-PRIMITIVE DATA

The SketchGraphs generative baseline averages 124.0 bits per sketch, compared with 85.3 bits per sketch for Vitruvion's combined primitive and constraint models. The appendix also describes image-data preparation, evaluation scope, and qualitative comparisons with prior systems.

E.3 IMAGE-TO-PRIMITIVE DATA

  • Five hand-drawn noise-model images were generated per sketch, and one random image was selected each training epoch, with affine image augmentation.The transformations included translation, rotation, shear, and scaling within specified bounds.Source: §E.3 · p. 17

F COMPRESSION BASELINE

  • The compression baseline tokenizes primitive sequences as 6-bit integers and applies LZMA across training, validation, and test data.The reported averages cover the full dataset, so the entropy rate may be lower than reported.Source: F COMPRESSION BASELINE · p. 17

G SKETCHGRAPHS BASELINE

  • SketchGraphs baselines were adapted for constraint inference and joint unconditional modeling, but not for image-conditional or primer-conditional generation.The comparison uses the current work’s featurization for fairness.Source: G SKETCHGRAPHS BASELINE · p. 17

G.1 GENERATIVE MODEL

  • 85.3 bits per sketch for Vitruvion’s combined models compares with 124.0 bits per sketch for the SketchGraphs generative model.Vitruvion’s result comprises 66.6 bits per sketch for primitives and 18.7 for constraints conditioned on primitives.Source: §G.1 · pp. 17–18

G.2 CONSTRAINT MODEL

  • The constraint-model comparison is limited to sketch-level log-likelihood because the baseline uses a different constraint factorization and tokenization.A comparable accuracy metric cannot be computed.Source: §G.2 · p. 18

H QUALITATIVE BASELINES

  • The SketchGraphs qualitative baseline lacks learned primitive coordinates, making solver initialization difficult and often causing solver failure.Source: H QUALITATIVE BASELINES · p. 18
  • Willis et al.’s CurveGen samples appear more sophisticated but sometimes lack coherence; their models output primitives without constraints.A direct visual comparison is muddled because Willis et al. do not model construction primitives.Source: H QUALITATIVE BASELINES · p. 18
  • None of the cited prior models target image-conditional generation, which Vitruvion explores using hand-drawn images.Source: H QUALITATIVE BASELINES · p. 18
Loading 2109.14124v2…