PaperlayerView exact revision on arXiv

Source-linked AI summary

Text-to-CAD Generation Through Infusing Visual Feedback in Large Language Models

Ruiyu Wang, Yu Yuan, Shizhao Sun, Jiang Bian

arXiv:2501.19054v3cs.CVcs.LG

TL;DR

Text-to-CAD could streamline CAD creation, but sequence supervision alone does not capture how rendered objects are perceived, and different sequences can produce the same object. CADFusion alternates LLM-based sequence learning with visual feedback from rendered objects. It reports better performance than GPT-4o across all metrics and comparable or better performance than Text2CAD2, especially on visual-quality metrics.

  • Problem

    Text-to-CAD needs both sequence supervision for coherent parametric operations and visual signals that reflect rendered-object evaluation, particularly because rendering can map different sequences to the same object.

    Source: §1 · pp. 1–2

  • Method

    CADFusion uses an LLM backbone and alternates ground-truth sequence learning with DPO-based visual feedback from rendered objects, scored through an automated LVM pipeline.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    CADFusion outperforms GPT-4o across all metrics and achieves comparable or better performance than Text2CAD2 across all metrics, especially on visual-quality metrics.

    Source: §4.2 · p. 6

  • Takeaways & Limitations

    The authors conclude that alternating the stages preserves the contributions of both sequential and visual signals.

    Source: §1 · p. 2, §6 · p. 9

    The visual-feedback pipeline currently uses single-view images, and CADFusion struggles with very complex shapes requiring spatial and commonsense reasoning.

    Source: §5 · p. 8

Abstract

from arXiv · show

Creating Computer-Aided Design (CAD) models requires significant expertise and effort. Text-to-CAD, which converts textual descriptions into CAD parametric sequences, is crucial in streamlining this process. Recent studies have utilized ground-truth parametric sequences, known as sequential signals, as supervision to achieve this goal. However, CAD models are inherently multimodal, comprising parametric sequences and corresponding rendered visual objects. Besides,the rendering process from parametric sequences to visual objects is many-to-one. Therefore, both sequential and visual signals are critical for effective training. In this work, we introduce CADFusion, a framework that uses Large Language Models (LLMs) as the backbone and alternates between two training stages: the sequential learning (SL) stage and the visual feedback (VF) stage. In the SL stage, we train LLMs using ground-truth parametric sequences, enabling the generation of logically coherent parametric sequences. In the VF stage, we reward parametric sequences that render into visually preferred objects and penalize those that do not, allowing LLMs to learn how rendered visual objects are perceived and evaluated. These two stages alternate throughout the training, ensuring balanced learning and preserving benefits of both signals. Experiments demonstrate that CADFusion significantly improves performance, both qualitatively and quantitatively.

1. Introduction

Text-to-CAD could streamline a demanding CAD creation process, but effective training needs both sequence structure and rendered visual feedback. CADFusion alternates sequential learning and visual feedback using an LLM backbone.

1. Introduction

  • Text-to-CAD generates parametric sequences from textual descriptions, aiming to streamline CAD creation that otherwise demands expertise and numerous iterations.The paper also describes natural-language design as making creation more accessible to people without extensive training.Source: §1 · p. 1
  • Text-to-CAD has received limited attention; earlier attempts trained Transformer-based models from scratch using ground-truth parametric sequences as supervision.The paper calls this supervision the sequential signal.Source: §1 · p. 1
  • Sequence supervision teaches structure and operations, while visual feedback represents practical evaluation; sequence-only training may overlook valid sequences with the same visual outcome.CAD models combine parametric sequences and rendered objects, and rendering maps different sequences to identical objects.Source: §1 · pp. 1–2
  • CADFusion uses an LLM backbone and alternates between sequential learning on ground-truth parametric sequences and visual feedback from rendered objects.The alternation is intended to balance and preserve the contributions of both signals.Source: §1 · p. 2
  • DPO lets CADFusion optimize preferences over rendered objects despite non-differentiable rendering, while an LVM pipeline automates visual scoring.The automated scoring pipeline addresses the cost and labor of collecting reliable preference data.Source: §1 · p. 2
  • The paper contributes separate sequential- and visual-signal datasets and reports qualitative and quantitative experiments showing CADFusion’s superior ability.It presents the combined use of both signals as its training approach.Source: §1 · p. 2

2. Related Works

Related work covers varied CAD-generation inputs and output representations, alongside language-model training and preference optimization. CADFusion focuses on text input, Sketch-and-Extrude output, and combining sequential and visual signals.

CAD Generation

  • Prior CAD generation uses random noise, model-part edits, point clouds, or sketches.CADFusion instead focuses on textual descriptions as input.Source: §2 · pp. 2–3
  • Earlier Text-to-CAD methods use sequential signals alone, whereas CADFusion combines sequential and visual signals [Khan et al., 2024b; Li et al., 2024b].The cited prior work includes a transformer-based autoregressive network and an encoder-decoder framework with text-sequence alignment methods.Source: §2 · p. 3
  • CAD outputs include CSG, B-Rep, and Sketch-and-Extrude; CADFusion uses Sketch-and-Extrude to preserve design history and support intuitive editing.Sketch-and-Extrude builds 3D models by extruding 2D sketches made of lines, arcs, and circles.Source: §2 · p. 3

Large Language Models (LLMs)

  • Unlike work that applies supervised fine-tuning and reinforcement learning to one signal, CADFusion uses them for different signals and alternates the stages.The paper describes alternating the stages as a strategy to preserve both signals’ contributions.Source: §2 · p. 3

Reinforcement Learning with Human Feedback (RLHF)

  • CADFusion uses DPO to learn from visual signals despite non-differentiable rendering, and an LVM scoring pipeline to construct preferences without costly human annotation.The paper presents DPO as a simpler alternative to RLHF for this setting.Source: §2 · p. 3

3. Method

CADFusion trains LLMs with ground-truth parametric sequences and visual preferences, alternating the two stages to preserve both signals. Its visual-feedback stage uses LVM scores to guide sequence preferences.

3.1. Approach Overview

  • CADFusion alternates sequential learning, which teaches coherent parametric sequences, with visual feedback, which rewards sequences rendering into preferred objects and penalizes others.The alternation is intended to preserve the contributions of both signals.Source: §3.1 · p. 3, Figure 2 · p. 4

3.2. Sequential Learning Stage

  • Sequential learning fine-tunes an LLM on text-token CAD sequences with cross-entropy; human-refined captions and an LLM backbone distinguish CADFusion from prior approaches.The method adopts an existing text-based sequence format and uses training data paired with CAD sequences.Source: §3.2 · p. 4

3.3. Visual Feedback Stage

  • Visual feedback renders multiple generated sequences, scores their CAD images with LVMs on shape quality, component count, and distribution, then uses DPO to favor preferred sequences.DPO avoids requiring backpropagation through non-differentiable rendering.Source: §3.3 · pp. 4–5

3.4. Alternate Training

  • Because extended visual-feedback training can impair sequence formatting and prolonged sequential training can weaken visual naturalness, each training block applies visual feedback followed by sequential learning.Training begins with sequential learning to establish logically coherent sequence generation.Source: §3.4 · p. 5

4. Experiments

Experiments compare CADFusion with specialized and general baselines, then examine its visual feedback, alternating training, and annotation choices. CADFusion performs strongly in the main evaluation, while ablations show trade-offs among renderability, visual quality, and annotation approaches.

4.1. Setups

  • Experiments use 20k instruction–CAD-sequence pairs and a 90:5:5 train/validation/test split, compare with Khan et al. and 8-example few-shot GPT-4o, and evaluate sequence, geometry, validity, and visual correspondence.They also report average human rank across generations from different baselines.Source: §4.1 · pp. 5–6

4.2. Main Results

  • CADFusion outperforms GPT-4o across all metrics and is comparable to or better than Text2CAD2, with particular strengths in LVM score and Avg. Rank.Text2CAD2's COV, MMD, and JSD values are unavailable because of release and setup differences.Source: §4.2 · p. 6, Table 1 · p. 7
  • CADFusion provides the most precise response to input instructions and achieves the highest similarity to the ground truth, capturing complex shapes and interpreting numerical and qualitative descriptors accurately.Figure 5 compares the ground truth, CADFusion, GPT-4o, and Text2CAD; outputs that cannot be rendered are marked with a red cross.Source: §4.2 · p. 6, Figure 5 · p. 7

4.3. Ablation Studies

  • CADFusionSL improves invalidity ratio by 1.36% versus CADFusionSL-VFSL(5) but lowers LVM score; it also beats Text2CAD using 20k rather than 150k data.Table 2 reports the LVM scores and invalidity ratios across CADFusion variants.Source: §4.3 · pp. 7–8, Table 2 · p. 8
  • Visual-feedback-only training struggles with renderable sequences, while adding RPO yields lower visual quality than sequential learning alone.The authors report that visual-only training receives a low LVM rating and RPO's visual quality is lower than the SL-only setup.Source: §4.3 · p. 8
  • Across one, three, and five alternating rounds, LVM scores gradually increase while invalidity ratio remains stable.The variants are CADFusionSL-VFSL(*), with the parenthetical number indicating the number of alternating rounds.Source: §4.3 · p. 8, Table 2 · p. 8
  • LVM-scored preference pairs outperform human-scored pairs on LVM score and invalidity ratio, while removing human refinement worsens both metrics.The authors note that consistent human scoring is difficult and that LVM-scored data scales across more feedback rounds.Source: §4.3 · p. 8

5. Limitation

CADFusion's visual-feedback pipeline is limited to single-view images, and the model struggles with very complex shapes requiring spatial or commonsense reasoning.

5. Limitation

  • The LVM evaluator receives only a single view, and CADFusion struggles with letter- and word-like shapes requiring spatial or commonsense reasoning.The authors say modern LVMs' performance drops with multiple images and identify these complex shapes as a limitation.Source: §5 · p. 8

6. Conclusion

The paper presents CADFusion as a Text-to-CAD approach that incorporates visual feedback from rendered objects and alternates visual feedback with sequential learning. Experiments demonstrate its superiority and validate its design choices.

6. Conclusion

  • CADFusion incorporates rendered-object visual feedback into Text-to-CAD training and alternates it with sequential learning; experiments demonstrate its superiority and validate its design choices.Future work will improve preference-data construction and collect more CAD data with complex geometries.Source: §6 · p. 9

Impact Statement

The paper presents LLM-based Text-to-CAD generation with visual feedback as a potential route to more automated and efficient CAD design, with possible benefits for accessibility.

Impact Statement

  • LLM-based Text-to-CAD and visual feedback could make CAD design more automated, efficient, and accessible where skilled designers are scarce or rapid prototyping matters.The paper frames these benefits as potential impacts.Source: Impact Statement · p. 9

Ethics Statement

The paper describes ethical safeguards for crowd annotation and recommends a structured format for user prompts, with some description elements optional.

Ethics Statement

  • Crowd workers were treated fairly, respectfully, and safely; the study collected no personally identifiable information and used annotations only for CAD-generation research.Source: Ethics Statement · p. 9

A. User Guidelines for Prompting

B.1. Converting Raw Data into Strings

CADFusion serializes sketch-and-extrude CAD representations into design-ordered strings and pairs these with textual instructions created through captioning and human refinement.

B.1. Converting Raw Data into Strings

  • CADFusion represents each CAD instance as sketches and extrusions, with sketches organized into faces, loops, and geometric primitives.Source: §B.1 · p. 12
  • Extrusion strings encode the boolean operation, plane displacements, translation, rotation, scaling, and scaling center, while coordinate padding distinguishes values from special tokens.Source: §B.1 · pp. 12–13
  • The conversion iterates through sketch commands, coordinates, and extrusion entries so the string sequence follows the CAD model’s chronological design order.Lines, arcs, and circles use simplified coordinate encodings; an endpoint is inferred from the next curve or, at loop closure, the first curve.Source: §B.1 · pp. 12–13

B.2. Generating Textual Instructions

  • Textual instructions are generated from single-view CAD images with LVM captioning and then refined by annotators for clarity, accuracy, and figure correspondence.Source: §B.2 · p. 14

B.3. Dataset Construction

  • Dataset construction pairs each CAD representation with a textual instruction and a ground-truth string converted from the original CAD formatting.A separate captioning procedure renders a sequence and uses GPT-4o to describe geometric properties such as shape count, dimensions, and spatial arrangement.Source: §B.3 · p. 14, §B.2 · p. 14

C. Additional Training Detail

The appendix specifies sequential-learning and visual-feedback training details, as well as the evaluation procedures and baselines used in the experiments.

C.1. Sequential Learning

  • Sequential learning fine-tunes LLaMA-3-8b-Instruct for 40 epochs with LoRA rank 32 on four NVIDIA A6000-48GB SMX GPUs, mapping shape descriptions to CAD command sequences.Source: §C.1 · p. 14

C.2. Visual Feedback

  • Visual feedback scores rendered outputs from post-sequential-learning CADFusion on a 0–10 scale using visual descriptions and criteria covering shape correctness, component counts, and arrangement.The scoring prompt excludes texture, smoothness, and color.Source: §C.2 · p. 15

D.1. LVM Evaluation Setups

  • GPT-4o served as LVM evaluator, scoring whether a CAD instance corresponds to its 3D-figure description on a 10-point scale; chosen over LLaMA-ov to limit self-generation preference bias, it may still introduce bias.Source: §D.1 · p. 16, §D.2 · p. 16

D.2. Human Evaluation Setups

  • Human evaluation ranks four rendered outputs for correspondence to instructions, using six judges and approximately 50 unique samples.Source: §D.2 · p. 16

D.3. GPT Baselines

  • The GPT-4o baseline uses eight-shot prompting because the tested three-shot version had approximately 92% invalidity and its renderable outputs were barely recognizable relative to prompts.Source: §D.3 · p. 17

D.4. Additional Statements on Text2CAD Results

The additional comparison explains that Text2CAD’s results depend on prompt detail and clarifies how the reported comparisons were constructed.

D.4. Additional Statements on Text2CAD Results

  • The quantitative setups are not fully aligned because Text2CAD’s predictions use its expert prompts, whereas CADFusion and GPT-4o use the study’s prompts.Source: §D.4 · p. 17
  • Text2CAD performs well with expert-level prompts containing step-by-step sketching guidance, while the study’s prompts omit that level of detail.Source: §D.4 · p. 17
  • The main comparison reports Text2CAD with its expert prompts, while a separate table includes results using the study’s prompts and intermediate-level prompts.Source: §D.4 · p. 17, Table 3 · p. 18
  • The authors caution that detailed prompts with step-by-step instructions and point coordinates may be impractical in real-life scenarios.Source: §D.4 · p. 17
  • CADFusion using simplified prompts outperforms Text2CAD-expert, whose expert-level prompts are too long and specific for practical design use.Source: §D.4 · p. 17, Table 3 · p. 18

D.5. Additional Quantative Results

The additional sections report human-evaluation and dataset-size findings, qualitative generation behavior, observed failure modes, and a chamfer-distance correction. They also show how generation diversity varies designs while retaining their overall shape.

D.5. Additional Quantative Results

  • On the full 170k-sample dataset, added training data yielded only marginal gains in visual quality, and no w/o HA variant surpassed the human-annotated counterpart.At equal training-set size, the authors report that data quality outweighs scalability for model performance.Source: §D.5 · p. 18
  • Human judges preferred both CADFusion variants over GPT-4o and Text2CAD, ranking the Visual Feedback variant higher.The human-evaluation table states that lower ranks indicate higher preference.Source: §D.5 · p. 18, Table 5 · p. 18

D.6. Additional Qualitative Results

  • Qualitative examples span holes, frames, repeated identical objects, stars, and V-shapes, illustrating generation across varied instructions and shapes.The authors also report handling more complex shapes such as stars and V-shapes.Source: §D.6 · p. 18

D.7. Text to Multiple CAD Figures

  • With non-deterministic inference, CADFusion varies thickness, width, and cutout size while retaining overall shape, offering users more design choices.Figure 10 illustrates slight variations from a single prompt.Source: §D.7 · pp. 18–19, Figure 10 · p. 19

D.8. Failure Cases

  • Failures include unrenderable sequences for highly complex or word-shaped prompts, and rendered mismatches involving more than eight items or complex merges.The authors report difficulty mapping CAD shapes to letters and handling more than eight separate items.Source: §D.8 · p. 20

E. Change Log

  • The revised chamfer-distance results use accurate pairwise comparisons instead of distribution-level comparisons; the earlier measure was an upper bound that downgraded model performance.The authors say this update does not affect the work's integrity.Source: E. Change Log · p. 20
Loading 2501.19054v3…