PaperlayerView exact revision on arXiv

Source-linked AI summary

CAD-Coder: An Open-Source Vision-Language Model for Computer-Aided Design Code Generation

Anna C. Doris, Md Ferdous Alam, Amin Heyrani Nobari, Faez Ahmed

arXiv:2505.14646v1cs.CVcs.AI

TL;DR

Existing AI-driven CAD methods have limits in editable operation coverage, generalization, and accuracy. CAD-Coder fine-tunes an open-source VLM to generate CadQuery code from images; it reports 100% valid syntax and improved geometric accuracy over baselines.

  • Problem

    Prior CAD generators often produce non-editable outputs or have limited generalization and operation coverage, while practical engineering needs editable, reusable CAD models.

    Source: §1 · p. 1

  • Method

    CAD-Coder fine-tunes an open-source vision-language model to generate editable CadQuery code conditioned on images.

    Source: §1 · p. 2

  • Results

    100% valid syntax rate, with CAD-Coder outperforming strong VLM baselines in code generation and improving geometric accuracy of generated solids.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The findings indicate the potential of fine-tuned VLMs to support engineering design workflows with editable CAD code generation.

    Source: §5 · p. 11

    CAD-Coder can struggle to infer precise dimensions and proportions from real-world images taken from varying perspectives, and generalization to unseen operations remains prompt-dependent.

    Source: §5 · p. 11

Abstract

from arXiv · show

Efficient creation of accurate and editable 3D CAD models is critical in engineering design, significantly impacting cost and time-to-market in product innovation. Current manual workflows remain highly time-consuming and demand extensive user expertise. While recent developments in AI-driven CAD generation show promise, existing models are limited by incomplete representations of CAD operations, inability to generalize to real-world images, and low output accuracy. This paper introduces CAD-Coder, an open-source Vision-Language Model (VLM) explicitly fine-tuned to generate editable CAD code (CadQuery Python) directly from visual input. Leveraging a novel dataset that we created--GenCAD-Code, consisting of over 163k CAD-model image and code pairs--CAD-Coder outperforms state-of-the-art VLM baselines such as GPT-4.5 and Qwen2.5-VL-72B, achieving a 100% valid syntax rate and the highest accuracy in 3D solid similarity. Notably, our VLM demonstrates some signs of generalizability, successfully generating CAD code from real-world images and executing CAD operations unseen during fine-tuning. The performance and adaptability of CAD-Coder highlights the potential of VLMs fine-tuned on code to streamline CAD workflows for engineers and designers. CAD-Coder is publicly available at: https://github.com/anniedoris/CAD-Coder.

1. INTRODUCTION

Prior CAD generators often lacked editable, complete representations or struggled beyond their training distributions. CAD-Coder fine-tunes an open-source VLM to generate CadQuery code from images, with strong reported accuracy and signs of generalization.

1. INTRODUCTION

  • Prior CAD generators often used incomplete operation sets or struggled with out-of-distribution inputs and CAD operations unseen during training.Many produced non-editable outputs, while trained-from-scratch models could fail on inputs outside their training distribution.Source: §1 · p. 1
  • CAD-Coder is an open-source, image-conditioned VLM fine-tuned to generate executable CadQuery Python code for editable CAD models.The model uses a LLaVA-derived architecture and is fine-tuned end-to-end.Source: §1 · pp. 2–3, Figure 1 · p. 2
  • 100% valid syntax rate, with CAD-Coder outperforming strong VLM baselines including GPT-4.5 and Qwen2.5-VL-72B in CAD code generation and improving geometric accuracy.Its generated solids closely match ground-truth solids.Source: §1 · pp. 2–3
  • Experiments report reasonable code generation from real-object photographs and use of CAD operations not included in fine-tuning.The paper describes these as signs of generalization, with performance on unseen operations dependent on prompting and model variant.Source: §1 · pp. 2–3
  • GenCAD-Code pairs rendered CAD images with CadQuery code for 163,671 CAD models.The dataset extends the GenCAD collection with Python files accompanying its rendered CAD images.Source: §1 · p. 3
  • The paper presents fine-tuned foundation models as a promising avenue for engineering design tools that generate editable CAD code.The authors frame this as a potential way to connect visual understanding with CAD programming.Source: §1 · p. 3

2. RELATED WORK

Related work spans conditional editable-CAD generation from scratch and with foundation models, using text, images, or both. The section contrasts limited CAD-program DSLs with code-based scripting libraries and reviews available datasets.

2. RELATED WORK

  • The review narrows its scope to conditional methods that generate editable CAD representations.It organizes prior approaches around model and training choices, conditioning, representations, and datasets.Source: §2 · p. 3

2.1 Editable CAD Generation: Models & Training

  • Most reviewed conditional editable-CAD models use transformers to predict CAD operations autoregressively, with architectures differing by pretraining and training-from-scratch choices.The models vary in whether they use pretrained LLMs or VLMs.Source: §2.1 · p. 3

2.1.1 Training from Scratch.

  • GenCAD [9] combines a CAD-command transformer, contrastively trained image encoder, and diffusion prior to generate CAD command sequences from images.It outperformed DeepCAD and SkexGen [7] on chamfer-distance-based solid-accuracy metrics.Source: §2.1.1 · p. 3

2.1.2 Leveraging Foundation Models.

  • Off-the-shelf foundation-model approaches use iterative VLM checking, multi-agent collaboration, or external CAD tools, while GPT-based code generation has also shown accuracy failures.CADCodeVerify uses VLM critique to refine GPT-4 code; a multi-agent approach [18] produced more complete, error-checked outputs than a zero-shot VLM, while GPT-4 and GPT-4V sometimes had near-zero IOU for spring models [12].Source: §2.1.2 · p. 3
  • Fine-tuning foundation models has improved CAD generation across code and DSL outputs, while pretrained code and image knowledge may support broader CAD operations and real-world image inputs.Examples include Blender-code fine-tuning [12], text-to-CadQuery fine-tuning, and the image-conditioned OpenECAD model.Source: §2.1.2 · pp. 4–5

2.2 Conditional CAD Generation

  • Conditional CAD generation uses text, images, or both; CAD-Coder takes image and text inputs, currently with uniform prompts.Prior work found image-plus-text conditioning increasingly beneficial for more complex solids, while text alone was also effective.Source: §2.2 · p. 5

2.3 Editable CAD Representations and Datasets

  • Parametric CAD programs preserve design history and allow modification, whereas many prior DSL representations are incomplete and require conversion scripts to produce solids.The reviewed DSLs cover restricted operation sets, and no complete DSL-based CAD representation is described.Source: §2.3 · p. 5
  • Established scripting libraries such as CadQuery offer editable, complete CAD representations; CadQuery and FreeCAD can run as standalone Python scripts.The paper uses “CAD code” for established-library scripting and “CAD program” for command-vocabulary or author-defined DSL representations.Source: §2.3 · p. 5

3. METHODS

The methods section constructs GenCAD-Code by converting GenCAD programs into CadQuery scripts, then trains CAD-Coder in two stages and evaluates it against image-conditioned CAD-code baselines using syntax and solid-similarity metrics.

3.1 Dataset Generation

  • GenCAD-Code contains 163,671 CadQuery scripts, each paired with five rendered images, with 147,289 training, 7,355 test, and 9,027 validation samples.The source programs are sequences of Sketch Line, Sketch Arc, Sketch Circle, and Extrude commands converted directly into corresponding CadQuery code.Source: §3.1 · p. 6
  • Scripts average 611 tokens, 99.9% are under 3,000 tokens, and the right-skewed distribution reflects an imbalance between simple and complex examples.The authors note that solid complexity generally corresponds to script token count and identify improving the representation of complex examples as future work.Source: §3.1 · p. 6, Figure 2 · p. 6
  • Direct command conversion can produce less concise code; rectangles use four sequential line commands rather than CadQuery’s more concise .rect() method.The authors identify post-processing to use native concise representations as future work.Source: §3.1 · p. 6

3.2 CAD-Coder Model Architecture and Training

  • CAD-Coder uses the LLaVA 1.5 architecture with Vicuna-13B-v1.5 and CLIP-ViT-L-336px, and first trains a two-layer MLP to align image features with LLM word embeddings.During this stage, the vision encoder and LLM remain frozen; training uses 595k image-caption samples from CC3M paired with randomly selected image-description questions.Source: §3.2 · p. 6

3.2.1 Stage 2: End-to-End GenCAD-Code Fine-tuning.

  • Stage 2 fine-tunes the MLP and all LLM weights on 147k GenCAD-Code examples while keeping the vision encoder frozen.Each example uses an unscaled isometric image, its CadQuery code, and a fixed code-generation prompt; fewer than 0.1% of training samples are filtered for exceeding 4,096 tokens.Source: §3.2.1 · pp. 6–7

3.3.1 Baselines.

  • Evaluation uses 100 randomly sampled test examples and compares CAD-Coder with closed- and open-source VLMs, plus LLaVA-v1.5-13B trained with generic VQA data.Baselines use default inference settings with adjustments to output-token limits and prompts; CAD-Coder is run at temperature 0, and the authors provide model weights and evaluation prompts.Source: §3.3 · p. 7, §3.3.1 · p. 7

3.3.2 Metrics.

  • Valid Syntax Rate measures the percentage of generated scripts that are syntactically valid and run as Python files without errors.The evaluation also reports IOUbest, the best-aligned intersection-over-union between generated and ground-truth solids among scripts that execute successfully.Source: §3.3.2 · p. 7
  • IOUbest normalizes translation and scale, aligns principal axes, and uses the best valid alignment to compare generated and ground-truth solid shapes.The normalization uses the matrix of inertia and centroid; the metric is computed only over scripts that execute successfully, excluding syntax errors.Source: §3.3.2 · p. 7

4. RESULTS

The evaluation measures valid CadQuery syntax and geometric accuracy against baselines, then tests generalization to real-object images and unseen CAD operations.

4.1 Training Details

  • Training used two one-epoch stages on 4 H100 GPUs, with max_model_length set to 4096 for longer CadQuery scripts.Stage 1 took 4.5 hours at learning rate 1e-3 and effective batch size 256; Stage 2 took 5.7 hours at 2e-5 and batch size 128.Source: §4.1 · p. 8

4.2 Comparison with Baselines

  • 100% valid syntax across 100 test samples and an IOUbest score of 0.675 put CAD-Coder ahead of the evaluated baselines, including Qwen2.5-VL-72B and GPT-4.5.The score is ∼60% above Qwen2.5-VL-72B; GPT-4.5 scores 0.151 lower. Table 2 reports both metrics on the GenCAD-Code test subset.Source: §4.2 · p. 8, Table 2 · p. 8
  • CAD-Coder better matched a complex solid than the baselines but still missed a rectangular feature; LLaVA-v1.5-13B scored zero on both VSR and IOUbest.The paper reports remaining difficulty with complex features and attributes LLaVA-v1.5-13B’s result to its apparent lack of CadQuery knowledge.Source: §4.2 · p. 8

4.3.1 Performance on Images of Real-World Objects.

  • On five photographed 3D-printed test objects, CAD-Coder generated models that broadly agreed with ground truth despite not being fine-tuned for real-image inputs.The models were not perfect matches; the paper reports that generated models from real images were weaker than those conditioned on rendered CAD images.Source: §4.3.1 · pp. 8–9
  • Real-image generations showed poorer dimensional accuracy and struggled to recover operations for multi-extrude solids.The authors link dimensional errors to training on gray, perfectly isometric views and propose varied material-rendered perspectives; they suggest real solid images in Stage 1 for more complex cases.Source: §4.3.1 · p. 9

4.3.2 Performance on Unseen CAD Operations.

  • CAD-Coder could not add fillets to a previously generated prism, even when prompted with the filleting command syntax.The dataset contained no fillet commands, and the model instead produced syntactically incorrect code that omitted the specified command.Source: §4.3.2 · p. 9
  • Only CAD-Coder-Qwen2.5-14B-LowLR correctly added fillets under the tested prompt, but its unseen-operation generalization was not robust.The prompt explicitly reminded the model of .fillet() usage; more abstract prompts did not elicit the same behavior. The variant used half the Stage 2 learning rate, 1e-5 instead of 2e-5.Source: §4.3.2 · p. 10, Figure 5 · p. 11
  • CAD-Coder-Qwen2.5-14B performed marginally worse than CAD-Coder on both metrics, while its 5 syntactically incorrect scripts were truncated at the 4096-token generation limit.Lowering the learning rate improved the filleting result but reduced IOUbest on the fine-tuning task, exposing a trade-off between fine-tuning performance and retaining pretrained knowledge.Source: §4.3.2 · pp. 10–11

5. LIMITATIONS, FUTURE WORK AND CONCLUSION

CAD-Coder shows potential for image-conditioned generation of editable CadQuery code, but its accuracy and generalization remain limited in specific ways. The authors identify perspective robustness, prompt-dependent unseen-operation performance, and stronger evaluation on real-world images as future-work priorities.

5. LIMITATIONS, FUTURE WORK AND CONCLUSION

  • The model can misinfer dimensions and proportions from images taken from varying perspectives, motivating multi-view or image-augmentation training.The authors call for further work on perspective and real-world image robustness.Source: §5 · p. 11
  • Generalization to unseen CAD operations remains heavily prompt-dependent, and the authors identify preserving pre-existing LLM knowledge during fine-tuning as a needed improvement.They also propose expanding the training dataset and exploring more model architectures.Source: §5 · p. 11
  • The authors propose an extensive test set pairing real-world images with CAD code to quantify real-photo-to-CAD-code performance more rigorously.They also suggest reasoning-based models or chain-of-thought training data as possible ways to improve generated-solid accuracy.Source: §5 · p. 11
  • CAD-Coder improves syntactic validity and geometric accuracy over image-conditioned code-generating baselines, supporting its potential to enhance engineering design workflows.The authors describe this as a foundation for future research into automated CAD modeling.Source: §5 · p. 11

APPENDIX A. OPTIMAL SOLID MODEL ALIGNMENT

The appendix formulates rigid-body alignment as finding a rotation, scale, and translation between two solids. For identical shapes, it describes an alignment solution and a normalization scheme for comparing shapes.

APPENDIX A. OPTIMAL SOLID MODEL ALIGNMENT

  • The alignment problem seeks a transformation that maps one solid geometry onto another, treating each solid as a set in three-dimensional space.The formulation concerns rigid bodies that may be rotated, scaled, and translated.Source: Appendix A · p. 13
  • For identical shapes, the formulation assumes a relative-volume-preserving bijection that corresponds points on the same parts of the two solids.The appendix notes that this ideal correspondence exists when the shapes are identical under an invertible affine transformation.Source: Appendix A · pp. 13–14
  • The alignment objective minimizes the distance between corresponding points by choosing a rotation, scale, and translation.The translation, scale, and rotation are represented by t, s, and R, respectively.Source: Appendix A · p. 13
  • The proposed normalization scheme guarantees perfect coincidence for identical shapes before their similarity is assessed.The appendix describes normalizing scale and translation, then aligning the solids’ principal axes of inertia.Source: Appendix A · p. 13
  • Principal-axis directions are ambiguous, so the procedure tests four valid rotations and selects the one with the best IOU.The text identifies eight possible directional alignments, four of which are valid rotations in SO(3).Source: Appendix A · p. 14

3. Change of Variables and Relative Volume Preserva-

This section develops the alignment objective using change of variables and relative volume preservation. Under the identical-shape assumption, the derivation yields a scale and rotation solution for aligning two solids.

3. Change of Variables and Relative Volume Preserva-

  • The mapping between identical solids is treated as a bijection that preserves their relative volume distribution.This property is used in deriving the alignment solution.Source: §3 · p. 14
  • Centering the solids by removing their centroids provides the translation normalization used in the change-of-variables derivation.The derivation then rewrites its objective using that variable change.Source: §3 · p. 15
  • The scale derivation assumes a Euclidean-norm objective; an L1 norm would yield a different scaling because rotation does not preserve L1 distance.The text identifies the Euclidean norm as the condition for the stated scale result.Source: §3 · p. 15
  • The optimal rotation is obtained by maximizing a trace objective, which the text identifies as an orthogonal Procrustes problem solvable by singular value decomposition.The derivation uses the SVD of S to determine the rotation.Source: §3 · p. 15
  • The resulting rotation aligns the solids’ principal axes of inertia, while testing four valid alignments is presented as an alternative to the continuous covariance-matrix and SVD approach.The four rotations address the directional ambiguity of inertia eigenvectors.Source: §3 · p. 15
Loading 2505.14646v1…