RealCADBench: Benchmarking Parametric CAD Modeling from Industrial Design Intents

1 INTRODUCTION

• Real industrial design inputs reveal different and often incomplete views of geometry, while existing benchmarks often underrepresent these settings and fail to separate execution, shape recovery, and product identity.A program can run while omitting mounting holes, and a plausible assembly render can still place the wrong components.Source: §1 · pp. 1–2
• RealCADBench contains 12,632 tasks across 19 factory-automation categories, four input modalities, and both Part and Assembly modeling.The reported slice has 1,745 Part tasks across four input regimes and the 25-task RCB-Assm25 assembly study; methods generate FreeCAD API Python for a shared runtime.Source: §1 · p. 2
• No standalone model leads all four metrics, and the highest regime-balanced composite comes from a different model than the leaders on the component metrics.On RCB-Assm25, Codex with GPT-5.5 improves executability and both IoU metrics over standalone GPT-5.5 but lowers the Judge by 6.98 percentage points.Source: §1 · p. 2
• The benchmark reports executability, Solid IoU, Surface IoU, and a rubric-based visual-semantic identity Judge, then compares frontier models and agents across realistic CAD tasks.The Judge assesses product identity, salient features, and assembly relations, while the quantitative and qualitative analyses identify recurring failure modes.Source: §1 · p. 3

2 RELATED WORK

• Geometry and construction-history corpora support study of CAD geometry, design sequences, command histories, and constrained sketches, while Text2CAD and CAD-Recode extend work to language-conditioned generation and executable code recovery.These resources do not evaluate executability, IoUs, and visual-semantic identity as distinct outcomes for real industrial design intents.Source: §2 · p. 3
• Recent benchmarks add mixed inputs, executable or editable outputs, and semantic, assembly, or engineering criteria, yet most still focus on synthetic or CAD-native tasks.RealCADBench evaluates real industrial design intents with four distinct measures.Source: §2 · p. 3
• Execution-time repair makes runtime feedback, tool use, and compute budget part of the evaluated method, so RealCADBench compares each agent with its paired standalone model on the same assembly tasks.Table 1 places this evaluation choice among selected CAD benchmarks across input, output, and system-level axes.Source: §2 · p. 3, Table 1 · p. 3

3.1 TASKS AND TRACKS

• The benchmark has a Multimodal Part Track and an Image-based Assembly Track, with a reported slice of 1,745 Part tasks across four input regimes and RCB-Assm25, a 25-task assembly study.Each task supplies an input and metadata; a method outputs FreeCAD API Python, and scoring uses the exported 3D model rather than source code or construction path.Source: §3.1 · pp. 3–4, Table 2 · p. 4, Figure 2 · p. 4

3.2 DATA AND REFERENCES

• Reference construction separates proposal, screening, expert verification, and evaluation; only about 17% of candidate models scoring at least 8 are retained as benchmark references.Visual and CAD assets are screened for corruption, ambiguity, duplicates, and usability, and the final tasks cover 19 factory-automation categories.Source: §3.2 · pp. 4–5, Table 3 · p. 5

3.3 EVALUATION PROTOCOL

• A valid submission exports a non-empty solid in millimetres; scoring combines execution success, aligned Solid and Surface IoUs, and a visual-semantic Judge that compares renders with input evidence.The Judge never sees the reference model; executability uses every assigned task, while each quality term uses only artifacts available to its evaluator.Source: §3.3 · p. 5

4.1 SETUP

• The nine standalone Part models and two agents are evaluated on the reported slice; agents are compared with their standalone models on the same assembly tasks as complete methods.Source: §4.1 · p. 6, §4 · p. 6

4.2 RQ1: DIFFERENT METRICS FAVOR DIFFERENT FRONTIER LARGE MODELS

• GPT-5.5 executes successfully on 93.11% of tasks but has conditional Surface IoU of 0.1393; Kimi K3 leads both IoUs while executing on 34.22%.Source: §4.2 · p. 6
• Gemini 3.1 Pro Preview has the highest regime-balanced PA (0.5507), GPT-5.5 leads executability and Judge, and Kimi K3 leads both IoU metrics; no model leads all four.Source: §4.2 · p. 6

4.3 RQ2: DIFFERENT INPUT REGIMES CHANGE THE DIFFICULTY ORDER

• Rendered-image inputs lead every component across six frontier-scale models, while the other three Part regimes change order across metrics.Source: §4.3 · pp. 6–7, Figure 3 · p. 7
• Assembly Solid IoU is 0.2290 versus 0.4071 on real picture parts; the studies are unmatched, while executability and Judge are similar.The comparison reflects two different populations, not a paired comparison.Source: §4.3 · p. 7

4.4 RQ3: FRONTIER MODELS AND AGENTS REVEAL RECURRING FAILURE MODES

• On RCB-Assm25, Codex + GPT-5.5 improves executability by 0.1600, Solid IoU by 0.0714, and Surface IoU by 0.0190 over GPT-5.5, but lowers Judge by 6.98 pp.Source: §4.4 · p. 7, Table 5 · p. 8
• Recurring failures include repaired execution alongside weak product identity, fine structures, or assembly placement.Source: §4.4 · pp. 7–8

• Different metrics favor different frontier models, and quality scores apply only to valid executions, so they should be read alongside the number of tasks entering each average.Source: §5 · p. 8
• The rendered-image regime is consistently easiest, but the other Part regimes change order across executability, IoUs, and Judge.Source: §5 · p. 9
• Agents are complete evaluated methods: tool use and iterative repair may improve execution or geometry without improving visual-semantic identity.Source: §5 · p. 9
• The Judge assesses product identity, salient features, and assembly relations against input evidence, complementing rather than duplicating IoUs.Source: §5 · p. 9
• Outputs can match overall shape yet omit fine structures, retain IoUs while missing product identity, or look plausible while assembly composition is wrong.Source: §5 · p. 9

• RealCADBench covers real industrial design intents, including text, drawings, product pictures, rendered images, and multi-part assemblies under a shared FreeCAD artifact contract.Source: §6 · p. 9
• Across the reported Part and Assembly studies, different systems lead different metrics, difficulty order shifts across input regimes, and recurring failures affect fine structures, identity, and placement.Source: §6 · p. 9
• Executability, IoUs, and visual-semantic identity should be reported separately rather than collapsed into one score or treated as if execution alone were sufficient.Source: §6 · p. 9
• The authors identify future work as releasing the full benchmark, reporting its full results, and comparing specialized AI-for-CAD algorithms with frontier models and general-purpose agents.Source: §6 · p. 9

• Non-rendered references are proposed, refined, screened, and reviewed by five professional CAD practitioners, with acceptance following cross-checks.About 17% of candidates scoring at least 8 are retained as references.Source: §A.1 · p. 12
• Rendered-image references and standardized input renders come from the same STEP asset, which must parse, tessellate, render, and export to non-empty STL.Source: §A.1 · p. 12
• RCB-Assm25 is a stratified 25-task study drawn from the 167-task Assembly Track, with six standalone models and two agents scored on the same tasks.Source: §A.1 · p. 12
• Visual-input screening removes corrupted, duplicate, ambiguous-identity, or inconsistent-version samples; assembly candidates also need multiple components with recoverable placement.Source: §A.1 · p. 12
• The public release includes the full benchmark, evaluation code, frozen prompts, and scoring-environment specifications, subject to source-license and access conditions.Source: §A.1 · p. 12

A.2 METRIC IMPLEMENTATION AND AGGREGATION DETAILS

• Executability averages every assigned task, while each quality metric averages only tasks with that evaluator's value; missing quality values are neither zero-filled nor inferred from executability.The aggregation specification records the corresponding counts, and component metrics are reported separately alongside Profile Average.Source: §A.2 · pp. 13–14
• The geometric alignment resolves PCA sign ambiguity by choosing the candidate with minimum one-way prediction-to-reference nearest-neighbor distance, then applies uniform scale and center translation once.The resulting aligned prediction is reused for both geometric metrics.Source: §A.2 · p. 14
• The shared voxel grid spans the joint bounding box with 2% padding and pitch set by its longest extent divided by R = 96; a sampled-surface histogram is the fallback for boundary occupancy.Filled occupancy includes mesh interiors, while boundary occupancy keeps cells intersecting the surface without dilation.Source: §A.2 · p. 14
• The frozen Judge uses separate Part and Assembly rubrics, compares exported-model renders with input evidence, and complements reference-based geometric IoUs.For text tasks, it uses the corresponding real picture evidence; each artifact receives one query under the frozen rubric.Source: §A.2 · pp. 14–15
• Profile Average equally weights executability, Solid IoU, Surface IoU, and normalized Judge, while the Part Track macro-average gives its four input regimes equal weight.The component columns remain the breakdown for interpreting the composite.Source: §A.2 · p. 14

A.3 ADDITIONAL AGGREGATE RESULTS

• Table 6 reports the exact values visualized in Figure 3.Source: §A.3 · p. 15
• Figure 4 shows system-level relationships among metrics, while Figure 8 shows paired agent differences on RCB-Assm25.Source: §A.3 · p. 15
• Appendix A.7 contains instance-level diagnostic traces for both outer loops.Source: §A.3 · p. 15

A.4 COMPLETE PART TRACK COMPONENT RESULTS

• Tables 7–10 expand the compact summary in Table 4.Source: §A.4 · p. 15
• Executability uses every assigned task, while Solid IoU, Surface IoU, and Judge use tasks with the corresponding evaluator outputs.This follows the reporting protocol described in Section 3.Source: §A.4 · p. 15
• Best and second-best values are highlighted only in the Exec. column.Source: §A.4 · p. 15

A.5 CROSS-DATASET CONSISTENCY CHECK FOR THE FROZEN JUDGE

• On six BenchCAD systems, Kimi K2.6 repeats rankings with 13/15 pairwise orderings and agrees with GPT-5.5 on 14/15, while Doubao’s near-identical runs agree less with GPT-5.5 (ρ = 0.60, τ = 0.47).Kimi K2.6 runs differ by 0.90 points on average (maximum 2.02); both select the same top-ranked system as GPT-5.5. This check uses fixed outputs and prompts, and does not use human ratings.Source: §A.5 · pp. 15, 17

A.6.1 BATCH VISIBILITY PASS

• Batch visibility labels are evidence metadata, not scoring routes: full, partial, and invisible depend on reconstruction observability and defining-feature sufficiency, not recognition or occlusion alone.The frozen adapter maps full/partial/invisible to observed/mixed/inferred; all labels leave subsequent scoring dimensions and weights unchanged.Source: §A.6.1 · pp. 17–18

A.6.2 PART JUDGE

• The Part Judge evaluates target-part geometry (P1) and functional design (P2), while mesh integrity is audited programmatically and assembly pose, contact, and relations remain out of scope.P1 weights primary form, defining features, detail fidelity and completeness, and surface refinement; P2 weights functional topology, interface design, and load/motion logic.Source: §A.6.2 · pp. 18–19

A.6.3 ASSEMBLY JUDGE

• The Assembly Judge independently scores component geometry (Q), assembly accuracy (F), and system design (D), assigning geometry, placement/connectivity, and system-function consequences to distinct dimensions.Its fixed aggregation does not add separate penalties for audit facts; a missing or fallback part is judged by its actual consequences in each dimension without repeating the same rationale.Source: §A.6.3 · pp. 20–21

A.7 INSTANCE-LEVEL AGENT TRACES ON RCB-ASSM25

• Table 12 presents a diagnostic sheet—not the official Table 5 Judge results—with real-picture inputs, two agent configurations’ renders and Q/F/D traces, runtime, and token counts.The sheet reproduces traces from a frozen RCB-Assm25 review workbook; the passage does not report comparative trace values.Source: §A.7 · p. 22
• Among 23 jointly scored assemblies, Codex had the higher total on 18; its mean total was 67.2 versus 56.1 for Claude Code.Each system exported 24 assemblies, with a different non-export case. Mean wall-clock time was 12.1 min for Codex and 31.6 min for Claude Code; mean token use was 24.5k and 47.1k, respectively. Layout F exceeded part geometry Q by mean gaps of 7.9 points for Codex and 13.8 for Claude Code.Source: §A.7 · p. 22
• Instance-level traces show that Q, F, and D need not move together, and are diagnostic rather than official Judge scores.Table 12 reports per-instance diagnostic tuples alongside renders, time, and token records for the real-picture assembly task.Source: §A.7 · p. 22, Table 12 · p. 23