PaperlayerView exact revision on arXiv

Source-linked AI summary

Computer-Aided Design as Language

Yaroslav Ganin, Sergey Bartunov, Yujia Li, Ethan Keller, Stefano Saliceti

arXiv:2105.02769v1cs.CVcs.LG

TL;DR

Structured 2D CAD sketches are difficult to create, and CAD work still requires engineers to attend to design details. The paper models these sketches with language-modeling techniques and Protocol Buffers, demonstrating generation both unconditionally and from bitmap references.

  • Problem

    CAD requires years of training, and its highly structured 2D sketches are particularly difficult to create.

    Source: §1 · p. 1

  • Method

    The authors represent structured objects with Protocol Buffers and model their distributions using techniques inspired by language modeling.

    Source: §1 · p. 2

  • Results

    The system demonstrated 2D CAD sketch generation both unconditionally and using a bitmap as a reference.

    Source: §6 · p. 18

  • Takeaways & Limitations

    The proof of concept points to interface extensions for conditioning on sketch properties, inferring constraints, and completing drawings.

    Source: §6 · p. 18

    Training imitates valid dataset examples without accounting for the sketch solver’s response, so solvability and stability may be overlooked.

    Source: §6 · p. 18

Abstract

from arXiv · show

Computer-Aided Design (CAD) applications are used in manufacturing to model everything from coffee mugs to sports cars. These programs are complex and require years of training and experience to master. A component of all CAD models particularly difficult to make are the highly structured 2D sketches that lie at the heart of every 3D construction. In this work, we propose a machine learning model capable of automatically generating such sketches. Through this, we pave the way for developing intelligent tools that would help engineers create better designs with less effort. Our method is a combination of a general-purpose language modeling technique alongside an off-the-shelf data serialization protocol. We show that our approach has enough flexibility to accommodate the complexity of the domain and performs well for both unconditional synthesis and image-to-sketch translation.

1. Introduction

CAD sketches combine geometric entities with constraints that encode design intent, making them difficult to construct consistently. This work adapts language-modeling ideas to generate structured sketches and contributes a Protocol Buffers representation, modeling techniques, and a large training dataset.

1. Introduction

  • CAD sketches comprise geometric entities and constraints on a single plane, with enclosed regions used by later operations such as lofts and extrusions.These operations generate complex 3D geometry from the sketch.Source: §1 · pp. 1–2
  • Constraints can interact across a sketch, making it difficult to choose a consistent system that preserves design intent as entity parameters change.Invalid constraint combinations can be impossible for a solver to satisfy, while multiple valid systems may produce similar behavior.Source: §1 · p. 2
  • Sketch generation resembles language generation: each next entity or constraint must be structurally consistent while contributing to a coherent design intent.The authors identify Transformers as a promising architecture for adapting language modeling to sketch generation.Source: §1 · p. 2
  • The authors collect over 4.7M preprocessed parametric CAD sketches and use the dataset to validate their generative models.They report that the experiments surpass prior literature in training-data scale and model capacity.Source: §1 · p. 2
  • The method describes structured objects with Protocol Buffers and proposes techniques for modeling distributions over their serialized representations.The approach draws inspiration from language modeling while aiming to eliminate data redundancy.Source: §1 · p. 2

2. Related work

Related work addresses constraint prediction, CAD datasets and construction sequences, image-to-CAD conversion, and vector generation. The paper positions its approach as a flexible sequence model for complex sketches, contrasting it with methods limited by domain-specific rules, data duplication, or task scope.

2. Related work

  • Constraint-prediction studies prescribe metaconstraints or iteratively suggest missing constraints, while differentiable-renderer approaches do not straightforwardly handle discrete entity and constraint choices.The related work also describes a custom raster-vectorization pipeline that is not end-to-end trainable and does not infer constraints.Source: §2 · pp. 2, 4
  • The ABC dataset emphasizes meshes and provides incomplete symbolic shape representations, while Seff et al. (2020) provide 2D sketches with substantial duplication and a baseline limited to simple sketches.The Seff et al. baseline relies heavily on an external solver to place primitives.Source: §2 · pp. 3–4
  • Fusion 360 Gallery recovers extrusion sequences while assuming sketches are given, leaving the scalability of the approach to real-world scenarios unclear (Willis et al., 2020).Its model chooses which sketch to extrude and by how much.Source: §2 · p. 4
  • Earlier image-to-CAD systems mostly used heuristic recognition rather than learning, whereas this paper presents an image-conditional model that learns recognition from training data.The paper says its approach makes no assumptions about visual appearance and programs no object-recognition heuristics.Source: §2 · p. 4
  • Vector-art work includes hierarchical VAE generation of SVG icons and Im2Vec, which uses differentiable rendering to avoid requiring vector-format training data (Carlier et al., 2020; Reddy et al., 2021).The paper also adopts Transformers as its sequence-modeling backbone (Vaswani et al., 2017).Source: §2 · p. 4
  • Compared with PolyGen’s Transformer-based mesh generation, this framework handles non-homogeneous structures of arbitrary complexity and uses one neural network for the entire object.The authors present these changes as a fit for CAD sketches and potentially other CAD construction components.Source: §2 · p. 4

3. Data

CAD sketches comprise geometric entities and constraints that preserve design intent as geometry changes. The paper serializes these structured objects as sequences, using Protocol Buffers and a custom token representation for modeling.

3.1. Sketches

  • Sketches combine geometric entities with constraints that preserve relationships as dimensions change; entity parameters serve as initial guesses for the constraint solver.Sketches are 2D drawings used to create or refine 3D geometry, and entity-only examples may look invalid outside their construction context.Source: §3.1 · p. 5, Figure 3 · p. 5
  • Because the solver treats entities and constraints holistically, different object orderings and parameter values can represent the same sketch.The order of objects in the corresponding collections does not matter to the solver.Source: §3.1 · p. 5

3.2. Data layout

  • Protocol Buffers replace raw JSON to reduce storage and specify complex objects with varied structure.The format represents entities and constraints as structured objects; constraints use pointers to refer to entities.Source: §3.2 · pp. 5–6, Figure 4 · p. 6
  • The sequence can place entities before constraints, or insert each constraint after the entities it references.The first ordering follows the Onshape API's entity and constraint lists; the alternative is explored in the experiments.Source: §3.2 · p. 6
  • A custom interpreter maps valid decision-step tokens to Protocol Buffer messages, ensuring every sequence in this format yields a valid message.This uses the sketch format's structure rather than a generic parser.Source: §3.2 · p. 7
  • Tokens use triplets for discrete values, continuous values, and repetition-ending flags, while field identifiers and locations provide model context.For each token, only the discrete or continuous value is active; the flag marks the end of a repeated field.Source: §3.2 · pp. 7–8
  • The token sequence encodes mutually exclusive fields with a discrete choice and ends repeated fields with a boolean flag.A sketch is a repeated list of objects, and nested repeated fields can produce multiple end flags.Source: §3.2 · p. 7

4. Model

The model treats CAD sketches as token sequences and uses autoregressive Transformer modeling to generate them. Byte and triplet representations support different token types, while a state-machine interpreter handles triplet sampling and image conditioning extends generation to visual inputs.

4. Model

  • The model estimates sketch distributions by treating sketches as token sequences and maximizing dataset log-likelihood with an autoregressive Transformer decoder.Each next token is modeled conditionally on preceding tokens.Source: §4 · p. 8

4.1. Byte representation

  • The byte representation models each byte as a categorical value from 0 to 255, with EOS marking the end of the sequence.The output projection has 257 logits.Source: §4.1 · pp. 8–9

4.2. Triplet representation

  • Triplet representation groups related token types to share embeddings and projections, and models continuous values as 8-bit quantized categoricals for improved training stability and performance.The paper also masks triplet end tokens except at valid list-ending positions.Source: §4.2 · p. 9
  • Triplet pointers refer only to prior tokens, with added referrable tokens allowing constraints to target entity parts absent from the entity-definition tokens.The referrable tokens are inserted after each entity and distinguished by part index.Source: §4.2 · p. 10

4.3. Sampling from the model

  • Triplet sampling uses a specification-derived state-machine interpreter to select the active token group, populate the corresponding field, and stop at the outermost end token.The generated token is fed back into the model at each step.Source: §4.3 · p. 10

4.4. Conditional generation

  • Conditional generation adds an image-derived input sequence to the Transformer, enabling it to attend to relevant image regions while generating sketch objects.A dataset pairing sketches with drawings for training this setting is left for future work.Source: §4.4 · pp. 10, 12

5. Experiments

The experiments evaluate the sketch dataset and compare unconditional and image-conditional generation using likelihoods and qualitative samples. They report plausible unconditional sketches and image reconstructions, while noting generation and sample-validity limitations.

5.1. Dataset

  • Duplicate filtering removes simple axis-aligned rectangles, groups sketches by object-type sequence, then keeps one sketch per visual cluster.Within each bin, the procedure clusters 128 × 128 binary renders using hierarchical clustering with a 0.1 threshold and Jaccard distance.Source: §5.1 · p. 14
  • 4,671,037 training sketches and 50,000 each for validation and test remain after excluding sketches with fewer than 4 or more than 100 entities.The collection came from a 6TB scrape of public Onshape documents and retained the most common 4 entity types and 16 constraint types, excluding constraints referring to external entities.Source: §5.1 · pp. 12, 14

5.2. Training details

  • Training uses Adam with a 10^-4 learning rate, gradient-norm clipping at 1.0, and 0.1 dropout.Batches use 128 dynamically packed lanes and are processed in parallel by 32 TPU cores.Source: §5.2 · p. 15

5.3. Unconditional generation

  • The proposed models (the last 4 rows) significantly outperform the weak baselines on test log-likelihoods.Table 2 reports test likelihoods, including mean bits per object and bits per sketch; the supplied passages do not give the cell values.Source: Table 2 · p. 12, §5.3 · p. 15
  • Interleaved ordering, which inserts each constraint immediately after the entities it operates on, consistently performs better than concatenated ordering.The authors suggest that earlier explicit information about entity relations makes the next generation decision easier.Source: §5.3 · p. 15
  • 25% of byte-model samples and 6% of triplet-model samples are invalid with Nucleus Sampling at top-p = 0.9.Without Nucleus Sampling, the invalid-sample rates are 36% for byte and 14% for triplet; sampling skews the sample distribution.Source: §5.3 · pp. 15–16
  • Random unconditional samples are often plausible, with closed regions, regularity, symmetry, and fine detail, but early sequence mistakes can make sketches nonsensical.Triplet-model constraints are usually reasonable, include relations spanning more than two primitives, and can be passed to an external solver to adjust entity parameters.Source: §5.3 · p. 16

5.4. Conditional generation

  • The image-conditional model fits better than the unconditional variant, nearly perfectly reconstructs simpler sketches, and mostly errs on sketches with many fine details.For the smiley-face input, it stopped generation early because most training sequences contained fewer examples; forcing generation and selecting among 64 samples produced a nearly perfect sketch.Source: §5.4 · pp. 16–17

6. Discussion

The paper demonstrates CAD sketch generation with language modeling and Protocol Buffers, including unconditional synthesis and bitmap-conditioned generation. It identifies solver-aware training and richer model–software interaction as directions for improving the system.

6. Discussion

  • The system synthesizes geometric primitives and their relations both unconditionally and with a bitmap reference.The authors describe these experiments as an initial proof of concept.Source: §6 · p. 18
  • The authors argue the approach is largely domain-agnostic, requiring a suitable Protocol Buffer specification or a more powerful interpreter if needed.They suggest 3D as a future direction, noting that operations such as extrusion can reference existing geometry in nested messages.Source: §6 · p. 18
  • Potential architecture improvements include visual input from a partially constructed sketch and tighter interaction with the CAD software.The authors suggest visual input may improve performance by giving the model more direct access to the generation state.Source: §6 · p. 18
  • Training imitates valid dataset examples without accounting for the sketch solver’s response, potentially overlooking solvability and stability.The authors propose objectives tied to desired behavior, including reinforcement learning with feedback from the software environment.Source: §6 · p. 18
  • The authors hope the work will support further advances in automated CAD and generative modeling of structured data.They also mention conditioning on sketch properties, inferring constraints from entities, and automatically completing drawings as possible applications.Source: §6 · p. 18

A. Object specifications

The object specifications use Protocol Buffers to represent sketch entities and constraints, with language features and options adapted to CAD data. The schemas also adapt Onshape structures and compress some redundant representations.

A. Object specifications

  • The framework supports nested messages, repeated fields, and oneof clauses; handlers can choose a branch from preceding fields, and options can set minimum repeated-field counts.CircleArcEntity uses oneof for mutually exclusive circle and arc configurations.Source: A. Object specifications · p. 21
  • The specifications adapt most Onshape API structures, remove redundant or irrelevant sections, and compress coincident pairwise constraints into one object..Source: A. Object specifications · p. 21
  • CircleArcEntity stores a center and uses oneof for circle or arc parameters; arc parameters include start and end vectors and a clockwise flag..Source: A. Object specifications · p. 21
  • Interpolated spline fields include construction and periodicity flags, interpolation points, endpoint derivatives, and trimmed or untrimmed parameters..Source: A. Object specifications · p. 21
  • The constraint schemas include coincident, concentric, equal, parallel, tangent, and perpendicular constraints..Source: A. Object specifications · pp. 21–22
  • Other schemas cover mirror, distance, length, diameter, radius, and angle constraints.The distance schema includes direction and length fields; the angle schema includes two pointers and an angle.Source: A. Object specifications · p. 22
  • Horizontal, vertical, and midpoint constraints are also specified, while mirror constraints include entity pointers and mirrored pairs..Source: A. Object specifications · p. 22
  • The distance handler selects a oneof branch using preceding fields; horizontal and vertical directions return branch 0..Source: A. Object specifications · pp. 22–23

B. Token groups

The token groups used in the experiments are defined by regular expressions over dotted field identifiers, with a range of possible values provided for each group.

B. Token groups

  • Token groups match subsets of dotted field identifiers with regular expressions, and each group has a specified range of possible values..Source: B. Token groups · p. 23

C. Additional statistics

Figure 11 presents distributions of additional sketch statistics.

C. Additional statistics

Loading 2105.02769v1…