PaperlayerView exact revision on arXiv

Source-linked AI summary

Text2CAD-Bench: A Benchmark for LLM-based Text-to-Parametric CAD Generation

Liang Wang, Heng Meng, Zekai Xiang, Jin Liu, Pingyi Zhou, Litao Chen, Yongqiang Tang

arXiv:2605.18430v1cs.LG

TL;DR

Existing text-to-CAD benchmarks underrepresent advanced geometry, diverse applications, and human-like design descriptions. Text2CAD-Bench addresses these gaps with 600 human-curated, four-level examples using geometric and procedural prompts, and finds that current models handle basic geometry better than advanced features while different capabilities diverge on L4.

  • Problem

    Existing text-to-CAD benchmarks focus on simple geometry, traditional mechanical parts, and structurally biased descriptions, limiting systematic evaluation.

  • Method

    Text2CAD-Bench evaluates 600 human-curated examples across geometric complexity levels L1–L3 and application-diverse L4 scenarios using dual-style prompts.

  • Results

    Current models perform reasonably on L1–L2 but degrade substantially on L3, while L4 results diverge across executability, geometric precision, and design-intent understanding.

  • Takeaways & Limitations

    Text2CAD-Bench establishes advanced CAD features as a significant capability frontier for future text-to-CAD research.

  • Takeaways & Limitations

    The benchmark covers only part of the CAD design space, uses single-turn evaluation without fine-tuning, and excludes multi-part assemblies with inter-part constraints.

Abstract

from arXiv · show

Text-to-CAD generation aims to create parametric CAD models from natural language, enabling rapid prototyping and intuitive design workflows. However, existing benchmarks focus on basic primitives and simple sketch-extrude sequences, lacking advanced features essential for real-world applications and covering only traditional mechanical parts. We introduce Text2CAD-Bench, the first benchmark systematically evaluating text-to-CAD across geometric complexity and application diversity. Our benchmark comprises 600 human-curated examples spanning four levels: L1-L2 cover fundamental geometry with standard features, L3 introduces complex topology and freeform surfaces, and L4 extends to real-world domains beyond mechanical parts. Each example pairs dual-style prompts -- geometric descriptions mimicking non-expert users, and procedural sequences aligned with expert-level conventions. Evaluating mainstream general LLMs and domain-specific models, we find that current models perform reasonably on basic geometry but degrade substantially on complex topology and advanced features. We release our benchmark to drive progress in text-to-CAD research.

1. Introduction

Text2CAD-Bench addresses limited geometric complexity, domain coverage, and linguistic diversity in existing text-to-CAD benchmarks. It evaluates 600 human-curated examples with dual-style prompts and finds substantial difficulty on advanced features and real-world scenarios.

  • Motivation: Existing benchmarks largely restrict geometry to simple sketch-extrude primitives, omit advanced features, and focus on traditional mechanical parts.These limitations reduce coverage of chamfers, fillets, sweeps, lofts, freeform surfaces, medical devices, consumer products, and architectural fabrication.
  • Motivation: Existing descriptions often mirror low-level command sequences, encouraging command translation rather than human-like shape understanding.The benchmark identifies both visual-geometric and procedural construction descriptions as important forms of design intent.
  • Benchmark: Text2CAD-Bench provides 600 human-curated parametric CAD examples organized across four progressively defined complexity levels.The benchmark is designed to evaluate text-to-CAD across geometric complexity and application diversity.
  • Benchmark: Each example pairs geometric descriptions resembling non-expert language with expert-aligned command sequences.The dual-style design enables comparison of input formats rather than evaluating only command-like descriptions.
  • Findings: Current systems perform reasonably on basic geometry but struggle significantly with advanced features and real-world scenarios.The benchmark positions L3 and L4 as challenging targets for future text-to-CAD research.

2. Related Work

Prior CAD-generation work spans sequence-based, boundary-representation, and code-based approaches, while existing datasets remain limited in representation, feature coverage, or linguistic alignment. Text2CAD-Bench instead emphasizes human-verified examples covering geometric complexity and application diversity.

  • Prior CAD Generation: Sequence-based and B-rep methods generate CAD representations but generally require task-specific models trained on specialized representations.This limits their ability to leverage pretrained language-model knowledge.
  • Prior CAD Generation: Text-to-3D methods produce meshes or implicit representations that lack the parametric structure and dimensional constraints required for engineering CAD.Text-driven CAD therefore distinguishes sequence-based and code-based output representations.
  • Code-Based Methods: Code-based methods use pretrained LLMs to generate executable CAD scripts, with prior work exploring multimodal inputs, fine-tuning, reasoning, and geometric rewards.Text2CAD-Bench adopts CadQuery because it supports LLM code generation, method chaining, advanced features, and executable validation.
  • Datasets and Benchmarks: Existing CAD datasets commonly lack construction sequences, advanced operations, linguistic diversity, or genuine alignment with human design intent.Text2CAD is described as text-annotated but template-generated from underlying command sequences.
  • Datasets and Benchmarks: Text2CAD-Bench follows quality-over-quantity benchmark principles by using 600 human-verified instances spanning geometric complexity and diverse application domains.Each instance undergoes quality assurance for ground truth, classification, and correspondence between descriptions and CAD.

3. Text2CAD-Bench

Text2CAD-Bench is built as a quality-controlled benchmark with unambiguous CAD ground truth, geometric complexity levels L1–L3, and application-diverse L4 scenarios. It uses dual description styles, CadQuery models, and complementary validity and geometric-fidelity metrics, while restricting L4 scoring to externally visible single-body designs.

  • Benchmark Design: The benchmark contains 600 examples with identical geometry represented by geometric and sequence descriptions and organized by geometric complexity or application diversity.L1–L3 form a geometric hierarchy, while L4 targets real-world domains rather than maximum geometric difficulty.
  • Description Styles: Geometric descriptions characterize appearance and spatial relationships, whereas sequence descriptions specify step-by-step construction operations.This duality evaluates robustness to input formats and supports comparison of description styles.
  • Construction Pipeline: The construction process extends coverage beyond sketch-extrude to chamfers, fillets, sweeps, lofts, shells, and complex patterns through human-AI CadQuery authoring and validation.Validated models must have fully specified parameters, valid watertight meshes, and complexity matching the target level.
  • Benchmark Levels: L1 covers primitives, L2 adds boolean and standard feature interactions, and L3 requires sophisticated path-based operations such as sweeps and lofts.Complex patterns and freeform surfaces are concentrated in L3, while basic finishing features occur across levels.
  • Real-World Applications: L4 includes 100 precisely specified examples across industrial, consumer, medical, architectural, and educational domains.Its purpose is application generalization, and its examples can be geometrically simpler or comparable to L3.
  • Real-World Applications: L4 is restricted to single solid bodies because eight-view exterior renderings cannot reliably judge internal structures, exact constraints, or inter-part relationships.This keeps VLM-based scoring tractable and interpretable.
  • Evaluation Metrics: Evaluation combines code-validity assessment with geometric fidelity using Chamfer Distance and IoU.Chamfer Distance uses bidirectional point-cloud distances, while IoU measures voxelized global shape agreement.

4. Experiments

Experiments evaluate general-purpose and CAD-specialized models across benchmark levels, prompt styles, and representations. Performance is strongest on simpler geometry, but complex operations expose substantial degradation and separate executability from geometric and application-level fidelity.

  • Overall Performance: Claude-4.5-Sonnet achieves the lowest L3-Geo CD at 70.13, surpassing GPT-5.2 at 93.46 despite trailing on simpler tasks.Across models, CD rises 1.3–2.1× and invalidity reaches approximately 70–90% from L1 to L3, with the sharpest decline between L2 and L3.
  • Prompt Style Comparison: Geometric prompts outperform sequence prompts on L1–L2, but sequence prompts become better for several models on L3.For GPT-5.2, geometric versus sequence CD is 44.31 versus 48.73 on L1, while L3 CD is 82.94 versus 93.46; advanced operations favor procedural descriptions.
  • General-purpose vs. Domain-specific Models: Text2CAD reaches invalidity as low as 2% yet produces the highest CD, 220, and near-zero IoU, showing that execution success can coexist with geometric inaccuracy.This IR–CD divergence motivates evaluating executability and geometric fidelity jointly rather than relying on execution rate alone.
  • L4 Evaluation: L4 results separate code executability, geometric similarity, and feature-level design understanding, with no single model dominating all three dimensions.Gemini3-Flash has the lowest IR at 17% but low feature scores, whereas MiniMax has IR of 81% and leads Q2, Q4, and Q5; GPT-5.2 and DeepSeek-V3.2 also trade off capabilities.
  • Ablation Studies: Command-sequence representations degrade performance relative to CadQuery, including Gemini3-Flash’s 2.2× L2-Seq CD degradation from 88.57 to 193.45.Invalidity also rises broadly, such as DeepSeek-V3.2 from 13.3% to 67.3% on L1-Geo and GLM 4.7 from 26.3% to 50.7% on L2-Geo.
  • Ablation Studies: Command-sequence comparisons are vulnerable to survivorship bias because CD and IoU are computed only on successfully executed samples.Higher invalidity can leave only easier surviving cases, inflating apparent quality; matched-subset or worst-case analyses would strengthen future evaluations.

5. Discussion & Conclusion

Text2CAD-Bench shows that text-to-CAD remains reliable mainly for basic geometry, while advanced features and real-world tasks expose substantial capability gaps. Results also show that output representation and prompt style materially affect performance, with broader capabilities remaining constrained by benchmark scope and evaluation setting.

  • Key conclusions: Text-to-CAD performs reasonably on L1-L2 but degrades substantially on L3 advanced features, leaving professional workflows unreliable.The conclusion identifies advanced CAD features as a significant capability frontier requiring further model or training-data advances.
  • Key conclusions: CadQuery representations substantially outperform command sequences in execution validity and geometric accuracy.The comparison attributes this advantage to CadQuery’s compatibility with pretrained code-generation capabilities.
  • Key conclusions: Geometric descriptions perform better on L1-L2, whereas sequence descriptions show advantages on L3 advanced features.The reported “global→local→detail→global” structure aligns with how language models process spatial information.
  • Key conclusions: L4 results separate code executability, geometric precision, and design-intent understanding as largely independent capabilities.Gemini3-Flash has the lowest L4 invalidity rate at 17%, MiniMax has the highest design-quality scores despite 81% invalidity, and GPT-5.2 leads L1-L3 precision but scores lowest on L4.
  • Scope and limitations: Text2CAD-Bench covers broader complexity and application domains than prior benchmarks but remains limited by 600 examples, single-turn evaluation, and single-part modeling.Future extensions include specialized domains, iterative execution feedback, multi-part assemblies, and additional CAD-script representations.

A.1. Dataset Statistics

Dataset statistics show a sharp increase in implementation complexity from L1 to L3, alongside increasingly divergent description lengths. These patterns support the benchmark’s intended complexity stratification and make prompt style more consequential for advanced geometry.

  • Complexity scaling: Average code length rises from 7.9 to 19.1 to 70.7 lines across L1-L3, an approximately 9× increase from basic to advanced levels.Average API calls also grow from 10.8 to 26.8, indicating more sophisticated operation sequences at L3.
  • Description scaling: At L3, geometric descriptions average 429.4 words versus 173.6 words for sequence descriptions.The divergence reflects the extensive spatial elaboration needed to describe swept surfaces and lofted profiles in natural language.
  • Description scaling: L1-L2 geometric and sequence descriptions have comparable lengths, but description style becomes more consequential as complexity reaches L3.Both styles lengthen with complexity, though geometric descriptions scale more dramatically.

B. Human Verification Details

Human verification used engineering-trained annotators who independently rated blinded multi-view renderings against original prompts. Human ratings largely preserved the VLM-based model ordering, supporting the use of GLM-4.6V as the L4 judge.

  • Annotators: Three engineering-background students independently judged correspondence between generated models and text from multi-view renderings.Each had introductory CAD coursework and more than six months of CAD-related research experience.
  • Scoring interface: Annotators saw identical eight-view renderings, the original prompt, and a 0–10 rubric while remaining blind to source models and other ratings.This protocol was used to reduce information about provenance and peer judgments.
  • Validation results: Human ratings largely preserved the relative ordering of the seven candidate models found in GLM-4.6V’s Q1-Q5 and Overall scores.No systematic same-family preference for GLM 4.7 was observed, supporting GLM-4.6V as the judge for the full L4 set.

C. Limitations

The authors acknowledge limitations in Text2CAD-Bench and identify future work as a direction for addressing them.

  • Text2CAD-Bench has acknowledged limitations that motivate directions for future work.

Benchmark Scope

Text2CAD-Bench covers diverse CAD challenges through four levels, but its scope remains bounded by limited domain, category, and sample coverage.

  • L4 spans industrial/mechanical, consumer products, medical devices, architectural elements, and educational models, but this domain selection is non-exhaustive.
  • A benchmark statistics table summarizes Text2CAD-Bench’s dataset scope.
  • The benchmark focuses primarily on solid modeling, excluding sheet metal, multi-part assemblies, and parametric families with complex constraints.
  • 600 examples prioritize quality and diversity but cannot exhaustively represent the full space of CAD tasks, leaving some configurations and edge cases underrepresented.

Representation Choice

Text2CAD-Bench exclusively uses CadQuery as its target representation, which limits direct comparison with systems targeting other CAD formats.

  • CadQuery is the exclusive target representation, so the benchmark does not directly compare systems using OpenSCAD, FreeCAD scripts, or proprietary formats.

Evaluation Protocol

The evaluation uses standardized CadQuery prompting with zero-shot, few-shot, and error-feedback settings, while human-in-the-loop construction supports benchmark quality and consistency.

  • Models are evaluated only in zero-shot and few-shot settings without fine-tuning, and interactive multi-turn refinement is not explored.
  • The prompting pipeline uses an initial generation request followed by a retry incorporating execution-error feedback when code fails.
  • Few-shot demonstrations come from held-out samples and cover advanced operations such as sweep, loft, shell, and complex boolean operations.
  • System prompts establish CadQuery expertise, enforce code-only output with a final result variable, and provide API guidance against common errors.
  • Five in-context examples demonstrate chamfering, sweeping, lofting, twist extrusion, and combined fillet/chamfer operations for L2–L3 tasks.
  • Human designers author descriptions before implementations, then iteratively refine CadQuery code with AI assistance while retaining oversight of correctness and difficulty.

G. Qualitative Results

Qualitative results show execution and geometric failures even on L1–L2, while advanced L3 operations produce widespread failures and L4 outputs are assessed through VLM scoring.

  • Even basic L1 tasks cause execution failures for multiple models, and successful execution does not guarantee geometric correctness.
  • Prompt style affects models differently on L1–L2: GPT fails on one L1 example with geometric prompts while Claude shows the opposite pattern.
  • Advanced L3 operations such as sweep, loft, and shell cause widespread execution failures, with only a few models producing partial outputs.
  • L4 visualizations show representative outputs across diverse application domains without ground-truth models because evaluation uses VLM scoring rather than geometric comparison.
  • Figure 3 compares generated L1–L2 models under geometric prompts on top and sequence prompts on bottom.
Loading 2605.18430v1…