PaperlayerView exact revision on arXiv

Source-linked AI summary

CAD-Llama: Leveraging Large Language Models for Computer-Aided Design Parametric 3D Model Generation

Jiahao Li, Weijian Ma, Xueyang Li, Yunzhong Lou, Guichun Zhou, Xiangdong Zhou

arXiv:2505.04481v2cs.CV

TL;DR

Generating parametric CAD sequences with LLMs is challenging because CAD parameters differ from natural language and prior work has underexplored direct LLM generation. CAD-Llama addresses this with hierarchical SPCC representations, adaptive pretraining, and instruction tuning, outperforming prior CAD models and LLM baselines across evaluated tasks.

  • Problem

    Direct LLM generation of parametric CAD sequences remains underexplored and difficult because parameterized CAD sequences differ substantially from natural language.

  • Method

    CAD-Llama uses hierarchical annotations in Structured Parametric CAD Code, SPCC-adaptive pretraining, and instruction tuning to adapt LLMs for parametric CAD generation.

  • Results

    CAD-Llama outperformed recent parametric CAD models and LLM baselines across unconditional, conditional, and CAD-related downstream tasks.

  • Takeaways & Limitations

    The results support using structured hierarchical CAD descriptions and LLM adaptation for generating more accurate and complex parametric CAD models.

  • Takeaways & Limitations

    Transformer-based methods may predict corresponding commands yet struggle to accurately predict the parameters required for precise parameterized CAD models.

Abstract

from arXiv · show

Recently, Large Language Models (LLMs) have achieved significant success, prompting increased interest in expanding their generative capabilities beyond general text into domain-specific areas. This study investigates the generation of parametric sequences for computer-aided design (CAD) models using LLMs. This endeavor represents an initial step towards creating parametric 3D shapes with LLMs, as CAD model parameters directly correlate with shapes in three-dimensional space. Despite the formidable generative capacities of LLMs, this task remains challenging, as these models neither encounter parametric sequences during their pretraining phase nor possess direct awareness of 3D structures. To address this, we present CAD-Llama, a framework designed to enhance pretrained LLMs for generating parametric 3D CAD models. Specifically, we develop a hierarchical annotation pipeline and a code-like format to translate parametric 3D CAD command sequences into Structured Parametric CAD Code (SPCC), incorporating hierarchical semantic descriptions. Furthermore, we propose an adaptive pretraining approach utilizing SPCC, followed by an instruction tuning process aligned with CAD-specific guidelines. This methodology aims to equip LLMs with the spatial knowledge inherent in parametric sequences. Experimental results demonstrate that our framework significantly outperforms prior autoregressive methods and existing LLM baselines.

1. Introduction

The paper addresses the underexplored challenge of using LLMs to generate parametric CAD sequences, whose structured geometry differs from natural language. CAD-Llama combines hierarchical SPCC annotations with adaptive pretraining and instruction tuning, outperforming prior models and LLM baselines across CAD tasks.

  • Motivation: Parametric CAD sequence generation with LLMs remains underexplored despite LLMs’ strong generative and zero-shot capabilities.The task is motivated by the potential to apply LLM priors to CAD sequence generation and editing.
  • Challenges: Direct LLM generation is difficult because parameterized CAD sequences differ substantially from natural language, while prior methods often generalize weakly to complex models and instructions.Existing approaches commonly use encoder-decoder architectures trained on CAD datasets or generate only basic construction sequences.
  • Framework: CAD-Llama converts CAD design histories into Python-like Structured Parametric CAD Code with hierarchical descriptions of sketches, geometry, components, and their relationships.A visual language model annotates component-level three-dimensional geometry and two-dimensional sketches before producing higher-level semantic descriptions.
  • Training: The framework adapts open-source LLMs through SPCC-based adaptive pretraining and instruction tuning for varied CAD downstream tasks.These training stages are intended to impart CAD modeling capabilities and support applications including CAD sequence generation and editing.
  • Results: CAD-Llama outperforms recent parametric CAD generation models and open-source LLM baselines across unconditional, conditional, and CAD-related downstream tasks.The authors report more accurate and complex parametric CAD models, including generation under complex text instructions.

2. Related Work

Prior CAD representation work targets shape understanding and parametric modeling, while CAD-Llama organizes CAD sequences as structured code for LLM processing. Its framework combines SPCC synthesis with pretraining and instruction tuning.

  • CAD Representation Learning: Shape representations based on point clouds, meshes, voxels, or SDFs support CAD understanding but do not capture exact shape parameters needed for editing and reuse.The passage places these approaches before the emergence of large-scale parametric CAD datasets.
  • Framework Overview: The proposed framework separates SPCC data synthesis from SPCC-based pretraining and instruction tuning.The first part converts CAD sequences into SPCC representations; the second uses the resulting corpus to enhance model performance.

3. Method

CAD-Llama prepares parametric CAD data for LLMs by combining code-like representations with hierarchical geometric descriptions, then trains the model for CAD generation and downstream editing tasks.

  • Framework: CAD-Llama organizes its method around hierarchical annotation, SPCC data synthesis, adaptive pretraining, and CAD-specific instruction tuning.The framework section introduces annotation and SPCC synthesis before pretraining and instruction tuning.
  • Hierarchical Annotation Pipeline: The two-stage annotation pipeline first describes individual components and then produces global descriptions covering abstract structure, detailed geometry, and spatial relationships.The pipeline uses component-focused imagery in stage one and outline, original-model, and local-description inputs in stage two.
  • SPCC Data Synthesis: SPCC converts parametric CAD sequences into Python-like code and embeds local annotations with global descriptions into the resulting representation.Sketches are represented through loops and drawing operations, while component code and annotations are concatenated before global descriptions are added as a prefix.
  • SPCC Data Synthesis: The method standardizes parameter representation by recentering sketch coordinates to (0, 0) and discretizing angular parameters from 0 to 360 degrees.These choices are described as making scale information more intuitive and angular parameters discrete.
  • SPCC Data Synthesis: The training corpus includes both detailed and abstract-description SPCC examples, with special handling for single-component models.The final corpus is represented as DSPCC = {D̃, D˙}.
  • Instruction Dataset: CAD-Llama constructs instructional datasets for text-to-CAD, completion, captioning, addition, and deletion to support downstream CAD tasks.Addition and deletion examples use original and edited models together with task instructions.

4. Experiments

Experiments evaluate CAD-Llama across unconditional generation, text-to-CAD, downstream CAD tasks, and representation ablations. Results show strong generation quality and stability, while hierarchical descriptions and code-format representations provide the most effective combination.

  • Experimental Setup: The experiments evaluate unconditional generation, text-to-CAD, CAD-related downstream tasks, and alternative CAD representations.The study uses DeepCAD-derived data, task-specific metrics, and comparisons with parametric CAD models and LLM baselines.
  • Unconditional Generation: CAD-Llama achieves MMD 0.96, JSD 0.66, and SR 99.90 in unconditional generation, indicating distributional alignment and highly stable valid outputs.Its COV is comparable to HNC-CAD, while qualitative results show diverse outputs across complexity levels.
  • Text-to-CAD: CAD-Llama-INS surpasses CAD-Translator and Text2CAD in accuracy by approximately 14% and improves geometric-alignment metrics including MCD, MMD, and JSD.It also outperforms GPT, LLaMA3, and other listed baselines on the text-to-CAD task.
  • CAD-related Downstream Tasks: CAD-Llama-INS reaches an average score of 63.58%, exceeding GPT-4 by 15.7% and LLaMA3 and Mistral by approximately 30% across CAD-related tasks.The evaluation uses two-shot baseline prompting and covers multiple downstream tasks.
  • CAD-related Downstream Tasks: Structured annotation improves deletion and addition tasks, although GPT-4 outperforms CAD-Llama-INS on deletion while struggling with parameter-generating addition.The addition task requires generating CAD parameters, highlighting a distinction between module identification and parameter generation.
  • Ablation Studies: SPCC outperforms all other representation methods, followed by SPCS, while single-description methods underperform by approximately 30–40% in ACCcmd and ACCparam.The results attribute the advantage to hierarchical descriptions and further gains from representing CAD sequences in code format, including higher generation stability.

5. Conclusion

CAD-Llama leverages LLM generative priors to generate parametric CAD sequences, combining hierarchical visual-semantic annotation, supervised fine-tuning, and instruction tuning. Experiments report superiority over traditional autoregressive methods and prevailing LLM baselines.

  • CAD-Llama leverages LLM generative priors to generate parametric CAD sequences.
  • A hierarchical annotation pipeline infuses textual descriptions of visual semantics and 3D shape at multiple levels of each CAD model via VLMs.
  • Supervised fine-tuning provides general understanding and generation ability for parametric CAD models, while instruction tuning targets CAD editing and operation tasks.
  • Experimental results show superiority over traditional autoregressive methods and prevailing LLM baselines.

Supplementary Material

The supplementary material adds implementation, cost, annotation, and experimental details, while organizing the paper's remaining technical sections.

  • The supplementary material provides details on data selection and method design.
  • It includes GPU-resource and GPT-4o-token cost analyses alongside additional experiment results.
  • The remaining sections cover CAD code formatting, hierarchical annotation, and subsequent technical details.

B. Training Cost and GPT-4o Token Cost

The supplementary methods describe training costs, CAD-code formatting, component abstraction, two-stage visual annotation, and prompting choices for generating CAD descriptions.

  • Training Cost and GPT-4o Token Cost: SPCC-adaptive pretraining and instruction tuning use 4 A100 GPUs, requiring 70 million and 6 million tokens, respectively, for finetuning-data generation.
  • CAD Code Formatting: CAD commands are converted into Python-like pseudocode, with loops represented as Loop(), termination as # End of code, and drawing or extrusion commands as function calls.
  • Hierarchical Annotation: Consecutive identical sketch-extrude pairs exceeding three are grouped into one component, with equivalence allowing differences only in sketch-plane origins.
  • Hierarchical Annotation: The annotation pipeline uses component and outline image extractors across two stages to capture local geometry, sketches, and global spatial relationships.
  • Two Stage Prompting Methods: Prompts provide extrusion direction, length, and quantity information to improve component descriptions and reduce VLM hallucination.

E.2. Ablation Study Details in Main Experiment

The ablation study compares CAD representations for Text-to-CAD, varying code versus raw sequences and hierarchical versus single descriptions, while also evaluating In-Context Pretraining.

  • The main ablation study evaluates how different CAD representations affect Text-to-CAD performance.
  • The single description SD concatenates model-level information, a parts-description label, and the full CAD-model description.
  • In-Context Pretraining improves performance across all evaluated CAD-related tasks.

E.3. Ablation Studies on Pretraining Method

The ablation study evaluates In-Context Pretraining (ICP) for CAD-Llama-INS and reports improvements on downstream CAD editing tasks, with a marginal Text-to-CAD gain. ICP groups visually similar CAD documents during pretraining to help the model distinguish CAD structures.

  • Pretraining method: ICP groups related CAD documents using CLIP-image cosine similarity during pretraining.The method groups similar CADs within the same input context to encourage reasoning across document boundaries.
  • Ablation results: ICP improves downstream add and delete editing tasks by helping LLMs capture distinctions between CAD structures.The reported improvement supports more precise modifications in these editing tasks.
  • Ablation results: ICP contributes a marginal improvement to the Text-to-CAD task.
  • Cross-dataset evidence: CAD-Llama-INS trained only on DeepCAD demonstrates strong Text-to-CAD generalization on Fusion360.

E.4. Cross Dataset Generalization

The study tests CAD-Llama-INS on Fusion 360 after training and fine-tuning exclusively on DeepCAD. It uses the hierarchical annotation pipeline to generate descriptions for the new dataset and evaluates Text-to-CAD generation alongside qualitative results across several tasks.

  • Cross-dataset evaluation: CAD-Llama-INS is evaluated for Text-to-CAD generation on the Fusion 360 test set after training exclusively on DeepCAD.The Fusion 360 dataset also contains CAD construction sequences, enabling a cross-dataset evaluation.
  • Cross-dataset evaluation: The hierarchical annotation pipeline generates Fusion 360 descriptions that prompt CAD-Llama-INS to produce corresponding CAD outputs.
  • Qualitative evaluation: Qualitative results cover text-to-CAD generation, captioning, unconditional generation, and multitask construction-to-refinement workflows.The results are presented in Figures 10 and 13–24, with multitask examples in Figures 19–23.

E.6. Examples of Failure Cases

The failure-case analysis identifies parameter-generation errors and mismatches between text instructions and generated CAD command sequences. The surrounding examples illustrate complex CAD descriptions, SPCC representations, and captioning outputs used in the evaluation.

  • Failure cases: Failure cases include inaccurate parameter settings and misalignment between input text instructions and generated CAD command sequences.Figure 25 illustrates these two error types for Text-to-CAD generation.
  • CAD representations: The examples represent models through structured CAD descriptions paired with executable-looking command sequences.Examples include cylindrical bases with rectangular cutouts and tapered plates with slots and cylindrical shapes.
  • SPCC examples: SPCC examples are included to show how CAD-Llama-INS represents parametric CAD data.
  • Caption task: Caption-task examples assess understanding of raw CAD code and the geometric shapes it constructs.

CAD-Llama-INS

The qualitative results present CAD-Llama-INS outputs for Text-to-CAD, editing, multitask workflows, unconditional generation, and supplementary examples. Figure 25 also documents parameter errors and text-to-sequence mismatches as failure modes.

  • Text-to-CAD: Text-to-CAD examples cover varied generated structures, including cylindrical tubes, U-shaped blocks, concentric cylinders, and cross-shaped frameworks.
  • Supplementary results: Unconditional-generation results and supplementary Text-to-CAD outputs provide additional qualitative evidence for CAD-Llama-INS.
  • Multitask workflows: Multitask examples demonstrate workflows that combine initial Text-to-CAD construction with iterative additions or deletions.The supplementary figures present working examples across Text-to-CAD, Delete, and Add tasks.
  • Editing tasks: Editing examples use Add and Delete instructions to modify holes, cutouts, enclosures, and plates in existing CAD structures.
  • Failure cases: Figure 25 reports parameter-generation errors and mismatches between text instructions and generated CAD command sequences.
Loading 2505.04481v2…