PaperlayerView exact revision on arXiv

Source-linked AI summary

GenCAD: Image-Conditioned Computer-Aided Design Generation with Transformer-Based Contrastive Representation and Diffusion Priors

Md Ferdous Alam, Faez Ahmed

arXiv:2409.16294v2cs.CVcs.GRcs.LG

TL;DR

CAD design is time-consuming, and existing 3D representations and mostly unconditional CAD generation do not meet the need for editable, user-aligned models. GenCAD learns image-CAD representations and uses conditional latent diffusion and autoregressive decoding to generate CAD programs. The paper reports gains in conditional-generation measures and image-based retrieval over image-to-image search.

  • Problem

    CAD design is manual and iterative, while common 3D-generation representations do not provide editable, manufacturable CAD and most CAD-program generation is unconditional.

    Source: §1 · p. 2

  • Method

    GenCAD learns joint CAD-command and image latents with contrastive learning, samples image-conditioned CAD latents with a diffusion prior, and decodes them autoregressively into parameterized commands.

    Source: §1 · p. 3, §3.3.3 · p. 8

  • Results

    Image- and sketch-conditioned generation outperforms unconditional models on diversity, fidelity, and statistical distance, and retrieval is more than 15× as accurate as image-to-image search.

    Source: §1 · p. 3

  • Takeaways & Limitations

    GenCAD generates editable CAD programs from image or sketch inputs and supports image-based CAD-program retrieval from large databases.

    Source: §1 · p. 3, §7 · p. 16

    The CAD programs are simpler than industrial design tasks, the vocabulary omits operations such as revolve and fillet, and valid CAD generation is not guaranteed.

    Source: §6 · p. 15

Abstract

from arXiv · show

The creation of manufacturable and editable 3D shapes through Computer-Aided Design (CAD) remains a highly manual and time-consuming task, hampered by the complex topology of boundary representations of 3D solids and unintuitive design tools. While most work in the 3D shape generation literature focuses on representations like meshes, voxels, or point clouds, practical engineering applications demand the modifiability and manufacturability of CAD models and the ability for multi-modal conditional CAD model generation. This paper introduces GenCAD, a generative model that employs autoregressive transformers with a contrastive learning framework and latent diffusion models to transform image inputs into parametric CAD command sequences, resulting in editable 3D shape representations. Extensive evaluations demonstrate that GenCAD significantly outperforms existing state-of-the-art methods in terms of the unconditional and conditional generations of CAD models. Additionally, the contrastive learning framework of GenCAD facilitates the retrieval of CAD models using image queries from large CAD databases, which is a critical challenge within the CAD community. Our results provide a significant step forward in highlighting the potential of generative models to expedite the entire design-to-production pipeline and seamlessly integrate different design modalities.

1 INTRODUCTION

CAD design is labor-intensive, while common 3D-generation representations do not provide the editable, manufacturable models engineering requires. GenCAD generates parameterized CAD command sequences from images or sketches, with contributions in sequence reconstruction, conditional generation, and image-based retrieval.

1 INTRODUCTION

  • CAD modeling remains manual and iterative, and CAD software has a steep learning curve for many practitioners.The paper motivates learning-based generation as a way to automate part of the design process.Source: §1 · p. 2
  • Meshes, voxels, and point clouds lack CAD's editable, manufacturable representation, while B-rep topology is not directly suitable as neural-network input.B-reps encode parametric surfaces, edges, and vertices; a network-friendly intermediate representation is therefore needed.Source: §1 · p. 2
  • Prior CAD-program generation is mostly unconditional, whereas GenCAD conditions generation on CAD images to align outputs with user input.Source: §1 · pp. 2–3
  • GenCAD generates complete parameterized CAD command sequences aligned with image inputs, rather than only producing a 3D solid.The sequences can be converted to B-reps or other representations using an off-the-shelf geometry kernel.Source: §1 · p. 3
  • A transformer-based autoregressive model reconstructs CAD sequences more accurately than state-of-the-art models.Source: §1 · p. 3
  • Image- and sketch-conditioned CAD generation outperforms unconditional models on diversity, fidelity, and statistical distance.Source: §1 · p. 3
  • Contrastive learning enables image-based CAD retrieval, reported as more than 15× as accurate as image-to-image search.Source: §1 · p. 3

2 RELATED WORK

Related work frames CAD generation around sequential command representations, available design-history datasets, and generative approaches that are mostly unconditional. The cited literature leaves image-conditioned generation of an entire CAD program unaddressed.

2.1 CAD as a language modeling problem

  • Earlier language-like CAD formulations include 2D sketch representations and transformer-based sequential 3D generation, but the latter is unconditional and limited in operations.The figure illustrates CAD commands as a sequence processed through a geometry kernel.Source: §2.1 · p. 3, Figure 2 · p. 4

2.2 Datasets for CAD

  • The ABC dataset contains one million CAD models but lacks design histories and labels; DeepCAD adds design histories, while Fusion 360 includes histories, assemblies, and face segmentation but is comparatively small.Source: §2.2 · p. 4

2.3 Generative models for CAD

  • Generative CAD work includes direct B-rep synthesis and unconditional sketch or 3D generation; prior approaches mostly lack user-conditioned generation.The cited figure depicts command vectors passed sequentially to a geometry kernel to create a solid.Source: §2.3 · pp. 4–5, Figure 3 · p. 5

3 METHOD

GenCAD represents CAD designs as parametric command sequences and combines sequence modeling, image–CAD representation learning, and a diffusion prior to generate image-conditioned CAD programs. Its four-stage framework decodes generated CAD latents into command sequences for conversion to 3D solids.

3.1 From natural language to CAD language

  • CAD models are represented as fixed-dimensional sequences of parameterized commands whose operation types are discrete and whose parameters preserve geometric information.The commands are fed sequentially into a geometry kernel to create the solid.Source: §3.1 · pp. 5–6

3.2 Dataset

  • The DeepCAD dataset is filtered to 168,674 CAD models with command sequences, with 152,530 for training, 8,515 for validation, and 7,629 for testing.Its operations are limited to sketches and extrusions; the image and sketch datasets each contain 845,105 examples.Source: §3.2 · p. 6

3 METHOD

  • GenCAD is presented as a four-step framework for conditional CAD program generation, from learning CAD latents through image conditioning and latent generation to command-sequence decoding.The resulting commands can be passed to a geometry kernel to obtain a 3D solid.Source: §3.3 · pp. 6–7

3.3 GenCAD: conditional generation of CAD programs

  • The framework combines a command-sequence autoencoder, contrastive CAD-image pretraining, a CAD diffusion prior, and a pretrained decoder to generate CAD programs from images.The pretrained CAD encoder and decoder are reused in later stages.Source: §3.3 · pp. 6–7

3.3.1 Command Sequence Reconstruction (CSR): Latent representation learning for CAD programs

  • CSR quantizes command parameters into 256 levels, embeds commands, and average-pools encoder outputs into one CAD latent vector for each sequence.The latent is expanded into a sequence for decoding into command parameters.Source: §3.3.1 · p. 7

3.3.2 Contrastive CAD-Image Pre-training (CCIP): Joint representation learning of the CAD program and CAD-image

  • CCIP uses a ResNet-18 image encoder to map CAD images into the CAD latent space while keeping the trained CAD encoder frozen.Images are resized and center-cropped to 256 × 256 pixels before encoding.Source: §3.3.2 · p. 8

3.3.3 CAD Diffusion Prior (CDP): Conditional generative model for CAD

  • CDP denoises a noised CAD latent conditioned on an image latent, using a ResNet-MLP denoiser; the paper also develops a deterministic prior.The diffusion prior takes CAD latent and optional image latent as input.Source: Figure 4 · p. 7, §3.3.3 · p. 8

3.3.4 CAD Decoder model: Generating CAD sequences from latent representations

  • At inference, the pretrained CSR decoder converts generated CAD latents into command sequences, with a start-of-loop token prepended to the output.The decoder is reused rather than trained from scratch.Source: §3.3.4 · p. 8

4 EXPERIMENTS

The experiments describe training configurations and evaluate CAD command reconstruction, image-based CAD retrieval, and generated-shape quality and alignment. The supplied passages specify evaluation procedures and metrics.

4.1 Training details

  • The models use transformer encoder-decoder components for CSR, image encoders for CCIP, and a ResNet-MLP denoiser for the CAD diffusion prior.The ResNet-MLP was selected over an MLP denoiser for the reported results.Source: §4.1 · pp. 8–9

4.2 Training

  • CSR, CCIP, and CDP are trained with separate schedules, including 1,000 CSR epochs, 300 CCIP epochs, and 1 million CDP training timesteps.All models are trained on an 80GB NVIDIA A100 GPU.Source: §4.2 · p. 9

4.3.1 Command Sequence Reconstruction (CSR) model:

  • CSR reconstruction is evaluated using command and parameter accuracy, chamfer distance, and invalid-shape ratio, with chamfer distance computed from 2,000-point shape clouds.The evaluation also considers performance by command-sequence length.Source: §4.3.1 · p. 10

4.3.2 Contrastive CAD-Image Pre-training (CCIP) model:

  • CCIP is evaluated on image-based retrieval by matching a query CAD image to its corresponding program among sets of 10, 128, 1,024, or 2,048 examples.The comparison baseline is image-to-image search using a ResNet-18 with ImageNet weights.Source: §4.3.2 · p. 10

4.3.3 CAD Diffusion Prior (CDP) model:

  • CAD generation is assessed with coverage, minimum matching distance, and Jensen–Shannon divergence, alongside FID for alignment between generated CAD latents and test CAD-image latents.Comparisons include unconditional generation and four prior CAD-generation methods.Source: §4.3.3 · pp. 10–11

5 RESULTS

GenCAD’s conditional diffusion model performs strongly across CAD-generation metrics, while its retrieval and sequence-reconstruction evaluations show advantages over their reported baselines. Image-conditioned generation also achieves the lowest reported FID among the compared priors, with variation across samples for the same input.

5.1 CAD sequence reconstruction

  • CSR beats the DeepCAD transformer autoencoder on command accuracy, parameter accuracy, Chamfer distance, and invalid ratio, with greater accuracy at longer sequence lengths.The authors also report more accurate reconstruction of relatively complex shapes.Source: §5.1 · p. 11, Figure 5 · p. 11

5.2 Image-based CAD program retrieval

  • At 2048 CAD programs, retrieval is almost 61% accurate and over 15× more accurate than image-to-image search, though accuracy drops as collections grow.The model is reported as most accurate for small batches.Source: §5.2 · p. 12, Table 2 · p. 13

5.4 Image conditional CAD program generation

  • Conditional diffusion beats DeepCAD l-GAN, SkexGen, ContrastCAD, and unconditional diffusion on COV, MMD, and JSD; Brepgen leads MMD/JSD but trails COV.The authors describe GenCAD as balancing diversity, fidelity, and statistical distance.Source: Table 3 · p. 14, §5.4 · p. 14

5.3 Unconditional CAD program generation

  • The unconditional model beats DeepCAD l-GAN and SkexGen on COV and MMD, with performance comparable to ContrastCAD and Brepgen.The authors report that it generates complex CAD programs yielding realistic 3D shapes.Source: §5.3 · pp. 13–14, Table 3 · p. 14

5.4 Image conditional CAD program generation

  • Image-conditional diffusion has the lowest FID versus sketch-conditional diffusion, l-GAN, deterministic prior, and unconditional diffusion; multiple samples per image vary in parameter space.The authors associate the sketch model’s higher FID with its greater diversity.Source: §5.4 · p. 15, Figure 12 · p. 15, Figure 11 · p. 15

6 LIMITATIONS AND FUTURE WORK

The evaluation and current model have scope limits: the studied programs are simpler than industrial designs, validity is not guaranteed, and the images are mostly clean and isometric. The paper also demonstrates integration with commercial CAD software for user editing.

6 LIMITATIONS AND FUTURE WORK

  • The evaluated CAD programs are simpler than industrial design tasks, and the current vocabulary omits operations such as revolve, fillet/chamfer, and other sketch operations.The paper identifies extending the vocabulary as future work.Source: §6 · p. 15
  • Generation does not guarantee a valid CAD model; the authors propose CAD-verification feedback as a possible next step.They identify adding feedback from a geometry kernel as non-trivial.Source: §6 · p. 15
  • Inputs are mostly clean, isometric CAD images; noisy backgrounds and non-isometric real-world views remain future work.Source: §6 · p. 15
  • Commercial CAD integration demonstrates that generated shapes can be edited, including by changing a feature.The figure shows the generated program, its editable form in commercial software, and an edited feature.Source: §6 · p. 16, Figure 13 · p. 16

7 CONCLUSION

GenCAD generates CAD programs from image or sketch inputs and supports image-based CAD retrieval. The conclusion also identifies the study’s relatively simple CAD commands as a current limitation.

7 CONCLUSION

  • GenCAD generates CAD programs conditioned on images or sketches and retrieves CAD programs from large databases using images.The conclusion describes both tasks as challenges addressed by the model.Source: §7 · p. 16
  • The study’s dataset contains relatively simple CAD commands, and future work aims to create a more sophisticated, real-world CAD dataset.Source: §7 · p. 16

A CAD vocabulary

  • The vocabulary has three sketch tokens and one extrusion token; each token represents a CAD operation or command.Here, a CAD embedding is the parameterized representation of an operation or token.Source: A CAD vocabulary · p. 19

B GenCAD against image-to-mesh models

GenCAD produces editable CAD models with intricate features, whereas the compared image-to-mesh models produce outputs described as inaccurate and not useful for engineering tasks. The section also shows failed GenCAD samples and outlines the sampling pipeline.

B GenCAD against image-to-mesh models

  • Compared with image-to-mesh models, GenCAD creates actual CAD models with intricate features that expert engineers can edit and modify.The paper describes the mesh outputs as inaccurate and not useful for engineering tasks.Source: B GenCAD against image-to-mesh models · p. 19
  • Table 4 presents parameterized CAD commands as fixed-dimensional vectors, with blank squares denoting masked elements.Source: Table 4 · p. 20
  • GenCAD’s example output includes a design history and accurate geometry features, which the paper contrasts with image-to-mesh models not trained for solid modeling.Source: Figure 14 · p. 20
  • The illustrated failed samples include intersecting faces, irregular shapes, and disjoint shapes; the authors anticipate that most can be fixed by an expert designer.Source: Figure 15 · p. 21

C Pseudocode

  • The pseudocode labels CAD sequence reconstruction, contrastive CAD-image learning, and latent conditional diffusion as components of the sampling procedure.Source: C Pseudocode · p. 21

D Ablation studies

The ablations select a four-layer CSR architecture for command and parameter reconstruction and ResNet-18 for the CCIP sketch encoder, while noting trade-offs in CSR metrics.

D Ablation studies

  • The four-layer CSR architecture performs best in command and parameter accuracy, while the two-layer version is slightly better in chamfer distance and invalid ratio; the authors select four layers.Source: D Ablation studies · p. 21
  • ResNet-18 outperforms ResNet-10, ResNet-34, and ViT as the CCIP sketch encoder, so the authors select ResNet-18.The tested ViT uses six attention layers, 16 heads, and patch size 32.Source: D Ablation studies · p. 21
  • Table 5 concerns shape-encoding performance, while Table 6 reports image-encoder ablations and specifies repetition counts for nb values.The captions say arrows indicate metric direction and bold numbers mark the overall best performance.Source: Table 5 · p. 22, Table 6 · p. 22

E.1 CSR model

The implementation uses a four-layer transformer encoder and decoder for CSR, a ResNet-18 image encoder for CCIP, and a 500-timestep diffusion process for CDP.

E.1 CSR model

  • CSR uses four transformer encoder layers and four decoder layers, each with eight attention heads and a 512-dimensional feed-forward network.Training uses Adam, batch size 512, learning rate 1e-3, 2,000 warmup steps, 1,000 epochs, and gradient clipping at 1.0.Source: §E.1 · p. 22

E.2 CCIP model

  • CCIP pairs the pretrained CSR encoder as its CAD encoder with a ResNet-18 image encoder, which includes dropout between the first batch-normalization layer and ReLU in each block.The image encoder uses dropout 0.1; training uses Adam with weight decay for 500 epochs and batch size 256.Source: §E.2 · p. 22

E.3 CDP model

  • LDM uses 500 diffusion timesteps and 10 MLP-resnet blocks, with a 2,048-dimensional projection layer and 0.1 dropout.Training accumulates gradients every 2 steps, uses a fixed learning rate of 1e-5 and batch size 2,048, and runs for 1e6 timesteps.Source: §E.3 · p. 22

E.4 Evaluation metrics

The evaluation uses COV for generated-shape diversity, MMD for fidelity, and JSD for the statistical distance between generated and reference shape distributions.

E.4 Evaluation metrics

  • COV measures diversity as the fraction of reference shapes matched by at least one generated shape.Chamfer distance is used to match point clouds.Source: §E.4 · pp. 22–23
  • The shape evaluations uniformly sample 2,000 points per shape and use 7,000 generated shapes for each model.Source: §E.4 · p. 23
  • MMD measures fidelity by averaging nearest-neighbor Chamfer distances between shapes in the reference and generated sets.The nearest-neighbor distance is computed for each reference shape.Source: §E.4 · p. 23
  • JSD measures the statistical distance between generated and reference shape distributions using KL divergence.Source: §E.4 · p. 23
  • For direct benchmark comparison, the calculation uses 1,000 randomly sampled reference shapes and 3,000 generated shapes.Source: §E.4 · p. 23
Loading 2409.16294v2…