Text-to-CAD Evaluation with CADTests

1 Introduction

• CADTESTS are executable software tests that verify whether generated CAD models satisfy a prompt’s geometric and topological requirements.The tests query models through CadQuery and can inspect prompt-relevant geometry and topology.Source: §1 · p. 2
• CADTESTS can account for valid design variations and identify subtle violations of prompt requirements.The paper contrasts this with geometric similarity to a single reference, which cannot capture the full space of valid designs.Source: §1 · pp. 1–2
• Test-based evaluation is reference-free, checks explicit requirements, returns interpretable failure messages, and is fast and deterministic.The paper contrasts this with learnable semantic evaluators such as CLIP or VLLM-based methods.Source: §1 · p. 2
• CADTESTBENCH is the first test-based Text-to-CAD benchmark, built through automated test synthesis and iterative refinement using mutation analysis.The benchmark pairs prompts with CADTEST suites.Source: §1 · pp. 2–3
• Recent Text-to-CAD methods are comprehensively evaluated on CADTESTBENCH, and test-based generation baselines outperform existing methods.The paper also reports that test-based evaluation aligns closely with human judgment in a mutation analysis and human study.Source: §1 · p. 3

Text-Conditioned CAD Model Generation

• Text-to-CAD methods span trained generative networks and training-free script generation, but the field lacks universally adopted benchmarks.CADTESTBENCH builds on CADPrompt and adds executable tests of prompt requirements.Source: §2 · p. 3

Text-to-CAD Evaluation

• Geometric similarity to one reference is ill-suited to ambiguous prompts, while visual scores do not explicitly verify precision-driven CAD requirements.Existing geometric metrics also apply only to executable generations, so reported aggregate scores use method-specific subsets.Source: §2 · p. 3

Evaluation Protocols for Code Generation

• Code-generation benchmarks use automated tests, but incomplete test suites can let incorrect programs pass; mutation testing probes test-suite fault detection.The paper adapts mutated CAD variants to construct tests while accommodating diverse valid geometries.Source: §2 · p. 3

Text-to-CAD Generation

• A B-rep represents a solid through faces, edges, and vertices, and is produced by executing a CAD program.Text-to-CAD methods synthesize programs represented as parametric tokens or executable scripts.Source: §3 · p. 4

Software Testing

• Property-based testing checks generated outputs against invariant properties rather than predefined input-output pairs.In this paper, CAD-specific property tests assess B-rep models, while mutation analysis measures test effectiveness and guides more discriminative tests.Source: §3 · p. 4

4 CADTESTS

• A CADTEST is a Boolean predicate on a B-rep that checks a specified geometric or structural constraint, and a suite evaluates a model by combining its tests.The B-rep’s queryable entities support atomic tests that can be composed into higher-level properties.Source: §4 · p. 4
• The synthesis pipeline uses an LLM planner to generate mutations of a reference CAD program and refine CADTESTS against them.The pipeline operates on a prompt, a reference program satisfying it, and the corresponding B-rep.Source: §4 · p. 4
• Mutants are modified CAD programs that deliberately violate one or more geometric requirements specified in the prompt.Each mutant is accompanied by a textual description of why it violates the design intent.Source: §4 · p. 4
• When a CAD program encounters a runtime error, its execution trace is returned to the planner as feedback for repair..Source: §4 · p. 5
• The planner generates initial tests from the prompt alone, with the number of tests determined by the verifiable properties it identifies.A structured CadQuery reference document guides test construction; tests are then refined using execution feedback.Source: §4 · p. 5
• Tests run on the reference model and its transformed valid variants, as well as mutants and their transformed variants.Scaling, rotation, and translation augment the sets to encourage tests that are invariant to pose and scale and to broaden mutation feedback.Source: §4 · p. 5
• The planner uses execution diagnostics to modify, add, or remove tests until they pass on valid models and detect each mutant, or the round limit is reached.Tests still failing on reference models after the limit are discarded.Source: §4 · pp. 5–6

5 The CADTESTBENCH

• CADTESTBENCH contains 5,937 tests across abstract and detailed prompts, averaging approximately 15 tests per sample.It is derived from CADPrompt's 200 CAD programs, with separate suites for each prompt type; detailed-prompt augmentations omit uniform scaling.Source: §5 · p. 6
• Evaluation reports Pass-Rate for samples passing every test, Requirement Score for satisfied requirement groups, category Accuracy, and Invalid-Ratio for runtime errors.Invalid generations count as failures in all metrics.Source: §5 · p. 6
• Table 1 compares LLM planners and two prompting strategies for generating tests on abstract prompts, reporting validity, soundness, mutation score, percentages, and counts.The strategies provide either 20 CAD-test demonstrations or a skill file that disambiguates test writing.Source: Table 1 · p. 6
• Table 2 compares refinement-loop scores for reference implementations, CAD mutants, and their augmented variants.The caption identifies the table as measuring refinement-loop effectiveness but supplies no score values.Source: Table 2 · p. 7
• Table 3 evaluates Text-to-CAD baselines across abstract and detailed partitions using pass rate, requirement score, and invalidity rate.The caption says the baselines use different LLM planners.Source: Table 3 · p. 7

• After four refinement rounds, mutation score exceeds 90% for both detailed and abstract prompts; only valid and sound tests are retained for CADTESTBENCH.Source: §6 · p. 7
• Solid-validity and geometric tests are least challenging, while topological, dimensional, volumetric, and spatial requirements are harder for current LLM planners.Source: Figure 3 · p. 8, §6 · p. 8
• All baselines perform better on abstract than detailed prompts; Text2CAD underperforms training-free baselines, and CADRille produced degenerate solutions and was excluded.Source: §6 · p. 8
• Claude planners significantly outperform GPT-based variants across methods, while ReAct-style CAD-kernel integration improves performance and lowers invalidity rates.Source: §6 · p. 8
• Multi-view visual feedback does not improve performance, and further exploration was limited because it significantly increases token usage.Source: §6 · p. 8
• CADTests+Log improves abstract-prompt pass rate by approximately 10% and requirement score by 3%, with consistent gains across all metrics.Source: §6 · p. 8
• CADTESTS-based evaluation aligns closer to human judgments than the other metrics considered, despite moderate inter-annotator agreement (Cohen’s κ=0.54).Source: §6 · pp. 8–9, Table 4 · p. 9
• Qualitative examples show tests checking star spacing, disregarding prompt-unspecified geometric variation, and detecting holes that break through the disk boundary.Source: §6 · p. 9, Figure 4 · p. 9

• CADTESTS verify whether generated CAD models satisfy prompt specifications, and CADTESTBENCH is presented as the first test-driven Text-to-CAD benchmark.Source: §7 · p. 9
• Iterative refinement against passing and mutation sets mitigates finite test-coverage limitations, but CADTEST verification may still diverge from human judgment.Source: §7 · p. 9
• The authors identify expanding test coverage for Text-to-CAD evaluation as an important direction for future work.Source: §7 · p. 9

A CADTESTBENCH Analysis

• 200 samples pair detailed and abstract prompts with 5,937 CADTESTS organized into prompt-requirement groups assigned by a GPT-4.1 classifier.Detailed prompts produce more tests because they specify more requirements.Source: A CADTESTBENCH Analysis · p. 13, Figure 5 · p. 14, Figure 7 · p. 15
• The six test categories cover solid and shell validity, topology, geometric types, dimensions and ratios, volume, and spatial arrangement.They include checks such as solid counts, face and edge counts, geometric elements, measurements, volume relationships, and feature placement.Source: A CADTESTBENCH Analysis · p. 13
• The benchmark’s category scheme also includes spatial-arrangement checks for feature placement, symmetry, and geometric relationships such as parallelism or perpendicularity.These checks can also distinguish through-holes from blind pockets.Source: A CADTESTBENCH Analysis · pp. 13–14
• Detailed prompts have a higher proportion of Dimensions and Ratios checks, reflecting their explicit dimensional requirements.The category distributions are compared across detailed and abstract partitions.Source: A CADTESTBENCH Analysis · p. 15
• CADTESTS use CADQuery API calls to analyze model geometry and topology, with val() the most frequent call because tests use it to access the geometry.The benchmark includes example test descriptions grouped into six categories and example CADTest implementations.Source: Figure 8 · p. 15, A CADTESTBENCH Analysis · p. 15, Figure 9 · p. 16

B CAD Mutants

• 1,275 CAD mutants were generated and filtered using IoU against the reference CAD program to ensure each measurably modified the shape.The mutants are B-Reps that violate one or more prompt requirements.Source: B CAD Mutants · p. 18
• The mutation pipeline produces a mutated CAD program and a description of why it fails the design intent.Mutants are modified CAD programs that deliberately violate one or more prompt requirements.Source: Figure 11 · p. 18, B CAD Mutants · p. 18
• The examples show mutated CAD B-Reps and omit their CAD programs and mutation descriptions for clarity.Source: Figure 12 · p. 18

C Text-to-CAD Methods and Baselines

• The evaluation groups methods into synthetic-data-trained methods, training-free approaches, and baselines that use CADTESTS during generation.Source: C Text-to-CAD Methods and Baselines · p. 19
• Text2CAD and CADRille are the finetuned methods considered, but CADRille produced degenerate solutions and was excluded from reported results.Text2CAD generates parametric token sequences from natural-language descriptions.Source: C Text-to-CAD Methods and Baselines · p. 19
• Training-free baselines include K-shot, Skilled, ReAct, ReAct + Vision, and CADCodeVerify, which use demonstrations, skill guidance, execution feedback, visual feedback, or validation questions.ReAct returns runtime errors to the planner, while ReAct + Vision adds four model renderings.Source: C Text-to-CAD Methods and Baselines · p. 19
• CADTest generates both a CAD program and tests, then iteratively refines them using execution feedback; CADTest + Log also exposes intermediate geometric and topological properties.The CADTest generation loop is illustrated with execution feedback to the planner.Source: C Text-to-CAD Methods and Baselines · p. 19

D Human Evaluation of CADTESTBENCH

• Two postgraduate researchers with computer-vision and CAD research experience judged whether generated models met prompt requirements across 125 prompt–model pairs.The pairs used detailed and abstract prompts and models sampled from the evaluated methods and baselines.Source: D Human Evaluation of CADTESTBENCH · pp. 19–20
• In the illustrated generation trajectory, execution logs reveal that a boolean union produced two separate solids, prompting the planner to make the parts overlap slightly.Source: Figure 13 · p. 20
• CADTESTBENCH’s requirement score aligns closely with human judgments, unlike the other metrics considered in the ROC-curve comparison.The study reports ROC curves for continuous metrics and raters.Source: Figure 14 · p. 20, D Human Evaluation of CADTESTBENCH · p. 20
• The qualitative-results figure reports pass rate, which requires all tests to pass, and requirement score, the fraction of sample requirements satisfied.It shows generated B-Reps and scores for Skilled, ReAct, and CADTest with GPT-5.2 and Claude 4.6 Sonnet planners.Source: Figure 15 · p. 21