PaperlayerView exact revision on arXiv

Source-linked AI summary

BenchCAD: A Comprehensive, Industry-Standard Benchmark for Programmatic CAD

Haozhe Zhang, Kaichen Liu, Miaomiao Chen, Lei Li, Shaojie Yang, Cheng Peng, Hanjie Chen

arXiv:2605.10865v2cs.AIcs.CVcs.SE

TL;DR

Industrial CAD generation requires models to infer structure, parameters, and operations, yet existing evaluations rarely test these capabilities together in realistic settings. BenchCAD evaluates them across verified industrial CAD programs and four task types. The benchmark finds that models often capture coarse geometry but remain unreliable at faithful parametric reasoning, while training gains do not remove held-out-family gaps.

  • Problem

    Industrial CAD generation requires executable parametric programs, but models are rarely evaluated on whether visual understanding, engineering parameters, and CAD operations work together in realistic industrial settings.

    Source: §1 · p. 2

  • Method

    BenchCAD provides 17,900 execution-verified CadQuery programs across 106 industrial families and evaluates image-to-code, visual and code QA, and instruction-guided editing.

    Source: §1 · pp. 2–3

  • Results

    Across 10+ frontier and CAD-specialized models, coarse geometry often outpaces faithful parametric understanding, and SFT/RL gains leave substantial out-of-distribution gaps.

    Source: §1 · p. 3, §6 · p. 9

  • Takeaways & Limitations

    BenchCAD offers a capability-decomposed way to measure industrial CAD reasoning beyond rendered shape similarity, including operations, parameters, constraints, and editable program structure.

    Source: §1 · p. 3, §6 · p. 9

    Voxel IoU for non-target preservation can miss small parametric shifts whose voxel footprint falls below its threshold.

    Source: A Limitations · p. 12

Abstract

from arXiv · show

Industrial Computer-Aided Design (CAD) code generation requires models to produce executable parametric programs from visual or textual inputs. Beyond recognizing the outer shape of a part, this task involves understanding its 3D structure, inferring engineering parameters, and choosing CAD operations that reflect how the part would be designed and manufactured. Despite the promise of Multimodal large language models (MLLMs) for this task, they are rarely evaluated on whether these capabilities jointly hold in realistic industrial CAD settings. We present BenchCAD, a unified benchmark for industrial CAD reasoning. BenchCAD contains 17,900 execution-verified CadQuery programs across 106 industrial part families, including bevel gears, compression springs, twist drills, and other reusable engineering designs. It evaluates models through visual question answering, code question answering, image-to-code generation, and instruction-guided code editing, enabling fine-grained analysis across perception, parametric abstraction, and executable program synthesis. Across 10+ frontier models, BenchCAD shows that current systems often recover coarse outer geometry but fail to produce faithful parametric CAD programs. Common failures include missing fine 3D structure, misinterpreting industrial design parameters, and replacing essential operations such as sweeps, lofts, and twist-extrudes with simpler sketch-and-extrude patterns. Fine-tuning and reinforcement learning improve in-distribution performance, but generalization to unseen part families remains limited. These results position BenchCAD as a benchmark for measuring and improving the industrial readiness of multimodal CAD automation.

1 Introduction

BenchCAD addresses the gap between coarse visual similarity and faithful, editable industrial CAD programs by evaluating perception, parametric reasoning, and executable synthesis together. Across frontier and CAD-specialized models, it finds persistent failures in local detail, operation choice, and edit fidelity, with out-of-distribution gaps remaining after training.

1 Introduction

  • CAD reasoning requires connecting visual evidence, 3D structure, symbolic operations, and executable program synthesis.Industrial families make local features and parameter relations consequential: a spring silhouette or gear-like outline alone does not specify its engineering design.Source: §1 · p. 2
  • Prior evaluations often rely on rendered-shape metrics, which can overlook differences in editability, operation choice, and engineering detail.Program editing and design QA have also been studied separately, leaving open how to evaluate CAD reasoning beyond shape fidelity.Source: §1 · p. 2
  • BenchCAD provides 17,900 execution-verified CadQuery programs spanning 106 industrial families and evaluates executable, editable, constraint-aware CAD reasoning.Its families include designs such as twist drills, bevel gears, compression springs, and brackets.Source: §1 · pp. 2–3
  • The benchmark covers image-to-code generation, visual and code-based design QA, and instruction-guided program editing.Image-to-code uses multi-view renders, while the QA tasks take visual or program inputs.Source: §1 · p. 3
  • Current models often recover global part families but miss local engineering details, choose simplified or incorrect operations, or fail to make localized edits faithfully.SFT and RL improve operation coverage and executable generation, but substantial gaps remain on held-out families requiring advanced operations and precise constraints.Source: §1 · p. 3

2 Related Work

Prior CAD generation and editing benchmarks cover only parts of industrial CAD reasoning, often emphasizing sketch-and-extrude programs or end-to-end shape matching. BenchCAD combines execution-verified parts, standards anchoring, broad operations, and capability-decomposed tasks.

2 Related Work

  • Earlier CAD code-generation corpora are dominated by sketch-and-extrude, while released corpora omit helical sweeps and parametric involute-gear construction.Prior evaluation also lacks family-level taxonomy and standard-table grounding, conflating perception, parametric abstraction, and code synthesis in a single IoU score.Source: §2 · p. 3
  • Closest prior CAD code/edit benchmarks each cover a slice, but none combines execution-verified parametric edit pairs, non-target preservation measurement, and capability-decomposed subtasks.The overview table summarizes benchmark positioning, but its caption alone does not specify comparative cell values.Source: §2 · p. 3, Table 1 · p. 3
  • BenchCAD combines execution verification at scale, standards anchoring, 49 CadQuery operations, and capability-decomposed tasks with matched image/code contrasts.Its task suite includes a verified parametric edit task; the pipeline begins with industry-standard designs and produces verified executable CadQuery code.Source: §2 · pp. 3–4, Figure 2 · p. 4

3 BenchCAD: Dataset and Tasks

BenchCAD constructs expert-authored, standard-anchored industrial part families and releases verified programs alongside QA and editing datasets. Its four tasks cover code generation, editing, and visual or symbolic design reasoning across a capability hierarchy.

3.1 Dataset

  • BenchCAD’s design principles combine expert-generated geometry, standard-table parameter anchoring, a family-level taxonomy, and broad coverage of CadQuery operations.Its 106 families include 52 standard-anchored families drawn from 47 ISO/DIN/EN/ASME/IEC codes, and include operations such as makeHelix, twistExtrude, and polarArray.Source: §3.1 · p. 4
  • The generation pipeline samples family, subfamily, and difficulty-tier parameters under standard constraints, then emits and sandbox-executes CadQuery programs.Programs that fail compilation, exceed the runtime budget, or produce degenerate volume are quarantined; surviving renders receive expert visual sign-off.Source: §3.1 · pp. 4–5
  • The release contains 17,900 verified code parts, 2,400 paired image/code numeric-QA items, and 748 curated edit pairs.Edit pairs cover dimensional, additive, subtractive, and multi-step categories.Source: §3.1 · p. 5

3.2 Tasks

  • The four tasks are VISION2CODE, EDIT CODE, CODE QA, and VISION QA, respectively targeting executable synthesis from images, instruction-following edits, program understanding, and reasoning from rendered views.VISION2CODE uses four canonical orthographic views; EDIT CODE asks for a minimally modified program matching the target solid.Source: §3.2 · p. 5
  • EDIT CODE measures headroom-normalized improvement over the original-to-target gap, while QA uses ±5% tolerance for ratios and exact match for integers.Vision2Code also reports execution, geometric, feature, and essential-operation metrics; edit metrics help attribute failures to parametric exactness, geometric closeness, or execution.Source: §3.2 · pp. 5–6
  • The QA hierarchy progresses from holistic recognition through CAD-operation understanding and industrial parametric abstraction to compositional spatial-code reasoning.Matched image/code questions help distinguish visual-recognition errors from CAD-operation understanding bottlenecks.Source: §3.2 · pp. 5–6

4 Experiments

The experiments compare frontier, open-weight, and CAD-specialist models using blind baselines and task-specific metrics. They also describe matched-compute SFT and RL training variants that differ in data composition.

4.1 Setup

  • Each task includes a blind baseline that preserves output format but removes informative input, measuring gains beyond dataset priors and metric floors.Examples include black views for image tasks, no code for Code QA, and unchanged programs for Code Edit.Source: §4.1 · p. 6

4.2 Models

  • The evaluation spans proprietary frontier MLLMs, open-source MLLMs or code LLMs, and CAD-specialist models.The evaluated groups include GPT, Claude, Gemini, Qwen3-VL, InternVL3, cadrille-RL, and CADEvolve v3.Source: §4.2 · p. 6

4.3 Evaluation

  • Vision2Code reports execution, voxel IoU, Chamfer and Hausdorff distances, Feature-F1, essential-operation recall, and a composite score.Edit accuracy is normalized to the original-to-target gap; QA accuracy uses ±5% tolerance for ratios and exact match for integers.Source: §4.3 · p. 6

4.4 Training

  • Training compares matched-compute SFT and RL settings for Qwen3-VL-2B that differ in training-data composition.SFT variants use iid, held-out-family OOD, or no BenchCAD data; GRPO-style RL starts from each SFT checkpoint, and the OOD-RL run also excludes held-out families.Source: §4.4 · pp. 6–7

5 Results

Across four tasks, BenchCAD exposes gaps between coarse geometric recognition and reliable operation, parameter, and editing reasoning. Results and error cases show where those gaps arise, while training offers partial transfer but leaves generalization to unseen families unresolved.

5 Results

  • QA-img consistently underperforms QA-code, while CodeEdit trails CodeGen across models, exposing persistent gaps in visual recognition and instruction-guided modification.The experiments separate results by modality, task type, and capability axis, and trace failures to shared reasoning bottlenecks.Source: Figure 4 · p. 7, §5 · p. 7

5.1 Overall Performance of 4 Tasks

  • The DeepCAD specialist lineage scores well on IoU but underperforms on nonextrude operations; gemini-3.1-pro-thinking reaches 0.318 total and thinking-mode behavior is non-monotonic.The unified leaderboard reports these VISION2CODE patterns in Table 10 and Table 18.Source: §5.1 · p. 7
  • 0.838 for the best Code QA models compared with 0.587 for Vision QA; Spatial / Code Reasoning remains weaker even with code access.The comparison indicates that explicit programs make geometric and parametric information easier to extract than rendered images.Source: §5.1 · pp. 7–8, Table 2 · p. 8
  • Simple API-level edits are nearly solved, but compositional edits remain difficult; target-render-only instructions drive every model to near-zero, while adding images barely helps overall.The textual instruction does the heavy lifting, and the visual signal is at best a clarifier.Source: §5.1 · p. 7, Figure 5 · p. 8

5.2 Error Analysis

  • A twisted bracket omits twist-extrusion, while a standardized spring uses a uniform helix instead of terminal pitch variation; layered attribution distinguishes L2 from L3 failures.The spring captures the part family and operation at L1–L2 but misses the standard-driven parameterization at L3.Source: §5.2 · p. 9
  • Code-generation errors include missed bolt threads and chamfers, incorrect extrusion planes, and geometrically similar shapes expressed in the wrong spatial frame.For the bearing retainer cap, 24-axis rotation-invariant IoU is substantially higher than single-axis IoU, partially diagnosing the frame mismatch.Source: §5.2 · pp. 8–9, Figure 6 · p. 9

5.3 BenchCAD as a Training Resource for Improving Model Capabilities

  • BenchCAD training improves in-distribution CodeGen to 0.7631 and broadens advanced-operation use, while transfer to 10 held-out mechanical families is non-trivial but the IID/OOD gap remains substantial.The reported operations include revolve, sweep, loft, and fillet.Source: §5.3 · p. 9

6 Conclusion

BenchCAD extends CAD evaluation beyond rendered shape similarity to operations, parameters, constraints, and editable program structure. The paper concludes that current models remain unreliable across these capabilities, while training yields only partial transfer to held-out families.

6 Conclusion

  • Across 10+ frontier and CAD-specialized models, systems often recover coarse geometry but remain unreliable at fine spatial grounding, operation selection, industrial parametric abstraction, and localized editing.These failures appear across paired image/code QA, operation-rich Vision2Code generation, and verified edit tasks.Source: §6 · p. 9
  • BenchCAD and its Croissant 1.0 metadata are released under CC-BY-4.0, with the evaluation harness under the MIT license, to support diagnostic progress toward industry-grade CAD automation.The conclusion also notes that robust industrial CAD generalization remains difficult.Source: §6 · pp. 9–10

A Limitations

BenchCAD's standard-parametric parts and evaluation measures have important boundaries: they do not represent all industrial designs, certify full standards compliance, or capture every form of metric error.

A Limitations

  • BenchCAD's expert-authored standard-parametric families do not capture the full diversity of proprietary industrial design, including hand-drawn tolerances, undocumented intent, and assembly information.49% of families are anchored to published standards, and verified Fusion360 Gallery and DeepCAD subsets are included for OOD comparison.Source: A Limitations · p. 12
  • Sampling within ISO 23509 parameter ranges and enforcing inter-parameter relations does not validate all mandated manufacturing tolerances or material specifications.Source: A Limitations · p. 12
  • Voxel IoU on the spatial complement of a targeted feature can miss small parametric shifts below its voxel threshold; the protocol also reports a complementary -AST diff metric.Source: A Limitations · p. 12

B Extended Related Work: CAD Code-Generation and Editing Benchmarks

The related-work comparison traces CAD code-generation systems and sets BenchCAD apart by its broader operation coverage and industry-family grounding. It also describes how its families relate to formal standards.

B Extended Related Work: CAD Code-Generation and Editing Benchmarks

C Full Comparison Against Prior CAD Code-Generation Work

  • Table 4 compares prior work by industrial-family and standard-code counts, operation breadth, advanced-operation support, editing and QA tasks, and evaluation-model counts.BenchCAD is described as a unified, capability-decomposed evaluation framework, distinct in purpose from the CadQuery-VLM corpus-and-model lineage.Source: C Full Comparison Against Prior CAD… · p. 12, Table 4 · p. 13

D Standard Codes Covered

E Dataset Generation Details

Dataset generation organizes designs into independently sampled subfamilies and difficulty tiers, then filters generated programs through execution checks and expert review.

Dataset generation details

  • Subfamilies capture mating, assembly, or construction variants, reuse their parent family's parameter schema, and use their own deterministic builder.Subfamilies are sampled and rendered independently, so one named family can contribute multiple subfamily-by-tier buckets.Source: E Dataset Generation Details · p. 13
  • Easy, medium, and hard tiers progressively expand parameter ranges and activate optional features, with sampling approximately uniform across tiers within each subfamily.Hard-tier examples include variable-pitch springs with closed-and-ground ends, exercising more advanced family operations.Source: E Dataset Generation Details · pp. 13–14
  • Programs that fail parsing or imports, raise runtime exceptions, exceed 30 seconds, or produce degenerate or inverted solids are quarantined before visual rendering and domain-expert sign-off.Only records passing every stage enter the release.Source: E Dataset Generation Details · p. 14

F Model Training

BenchCAD training improves operation use and executable CAD performance, but the reported outcomes distinguish broad supervised family coverage from reinforcement learning. Per-task results also expose uneven operation recall and reasoning trade-offs.

F Model Training

  • SFT compares an all-family IID mixture, a mixture holding out 10 mechanical families, and a no-BenchCAD baseline; RL starts from the matching SFT checkpoint and combines geometry and essential-operation rewards when available.The SFT mixtures use Qwen3-VL-2B; RL prompts mix BenchCAD, DeepCAD, and Fusion360, with held-out families excluded in OOD training.Source: F Model Training · p. 14
  • Applying RL to OOD-SFT further improves IID and held-out OOD performance, while IID-trained SFT obtains the strongest OOD score overall, indicating broad supervised family coverage remains a key generalization driver.The training comparison reports geometric fidelity, execution rate, essential-operation score, and final score.Source: F Model Training · p. 15

G Full Per-Model Per-Task Results

  • The full-results overview assigns Tables 7–12 to dataset composition, task-to-capability mapping, QA leaderboards, difficulty-stratified image-to-code results, and operation recall.Table 8 describes capability subsets, while matched image- and code-conditioned QA scores diagnose capability bottlenecks.Source: G Full Per-Model Per-Task Results · p. 15, Table 8 · p. 16, Table 9 · p. 16

H CAD Operational Blindspot: Per-Operation Recall

  • Vision2Code recall remains lower for industrial operations such as chamfer, revolve, threePointArc, shell, and counterbored holes than for common sketch-and-extrude operations.Thinking mode improves some planning-heavy operations but degrades common boolean and scaffolding operations, reducing macro recall and execution rate.Source: H CAD Operational Blindspot:… · p. 16

I Datasheet for BenchCAD

This appendix documents BenchCAD’s dataset contents and provenance, then specifies how its question-answering, code-generation, and editing evaluations are constructed and prompted. It also defines a diagnostic rotation-invariant IoU protocol for identifying wrong-plane outputs.

I Datasheet for BenchCAD

  • BenchCAD contains 17,900 verified parts, with separate QA and edit releases; instances pair executable CadQuery with STEP, four views, parameters, and an operation list.The parts are procedurally generated; verification includes sandbox execution and domain-expert visual sign-off.Source: I Datasheet for BenchCAD · p. 17

J Question Bank Construction

  • The numeric-QA bank uses per-family parameterized templates, deterministic answers, and review for ambiguity and visibility.It contains 2,400 question-answer pairs sampled across 17,900 records, evaluated under both image and code conditioning for 4,800 QA records.Source: J Question Bank Construction · pp. 18–19

K QA System Prompts

L.4 Edit-task System Prompt

  • Edit prompts ask models to preserve the original script and make minimal changes, using either explicit text instructions or visual target geometry.For image-only editing, the target render is the sole specification; for text-plus-image editing, the render supplements the instruction.Source: §L.4 · p. 22

M Rotation-Invariant IoU

  • Rotation-invariant IoU takes the best overlap across cube rotations after centering and scaling, and is diagnostic for geometrically similar outputs on the wrong workplane.A larger gain for XZ/YZ ground-truth parts than XY parts is reported as consistent with a workplane prior.Source: M Rotation-Invariant IoU · p. 23

N Edit Protocol

BenchCAD-Edit analyses failures across task types and model capability layers, alongside image-conditioning and scoring-protocol ablations. The results show that model generations shift the kinds of remaining errors, while image-only editing is substantially weaker than text-guided editing.

N Edit Protocol

  • Image-only editing reduces mean_norm by 0.40–0.85 pt at every task type, with dimensional edits near zero and T5 at zero.The target render supports feature-presence judgments but does not provide the exact dimensions supplied by text instructions.Source: N Edit Protocol · p. 24, Figure 10 · p. 28
  • Adding the original-part image leaves three of four models within ±2 pt of text-only on aggregate; o3 gains ∼7 pt, while gpt-5.3-thinking loses 7 pt on T1 and 20 pt on T5.Source: N Edit Protocol · pp. 24–25
  • Failure patterns shift across model generations: newer systems reduce L2 CAD-operation errors, while reasoning-tier models expose an L3–L4 ceiling on coordinated edits.The failure-layer distributions are normalized within each model, with absolute failure rates reported separately.Source: N Edit Protocol · p. 26, Figure 8 · p. 26, Figure 9 · p. 27

O Scoring-Protocol Ablations

  • Scoring ablations compare rotation-invariant with single-axis IoU and four-view with single-view input to isolate wrong-plane credit and removed depth or back-side cues.The provided passages specify what each comparison tests but give no ablation outcome values.Source: O Scoring-Protocol Ablations · p. 26, Table 17 · p. 27

P CadQuery Operation Coverage

  • CadQuery operation coverage is measured by extracting distinct method invocations under a shared static-analysis rule, but CADEvolve’s reported count is a strict lower bound from 46 public seed programs.Its expanded scripts are referenced in the paper but not included in the public release.Source: P CadQuery Operation Coverage · pp. 27, 29, Table 19 · p. 29

Q Case Study: motor_end_cap_bore_widen (Case 167)

Case 167 shows that a high-overlap edit can still miss the requested through-cut when the cut occurs before the boss is added. The appendix also provides reproduction instructions and bounds its ethical claims to impacts specific to BenchCAD.

Q Case Study: motor_end_cap_bore_widen (Case 167)

R Reproduction

  • The benchmark can be reproduced through unified Python 3.11 setup or by running each task independently with its own configuration.The full run uses uv run python run_all.py –config prod; smoke-test splits are included for each task.Source: R Reproduction · p. 31

S Ethical Considerations and Broader Impact

  • BenchCAD uses procedurally generated parts and public engineering standards, with no personal data, proprietary designs, or safety-sensitive specifications.The authors frame broader CAD-model risks as deployment-layer concerns, not impacts specific to this benchmark.Source: S Ethical Considerations and Broader… · p. 32

Related papers

Loading 2605.10865v2…