PaperlayerView exact revision on arXiv

Source-linked AI summary

CADSmith: Multi-Agent CAD Generation with Programmatic Geometric Validation

Jesse Barkley, Rumi Loghmani, Amir Barati Farimani

arXiv:2603.26512v1cs.AI

TL;DR

Text-to-CAD systems struggle to guarantee both dimensional precision and global geometric correctness. CADSmith addresses this with multi-agent generation and nested refinement using OpenCASCADE measurements plus an independent vision-language Judge. On a 100-prompt benchmark, it reaches 100% execution success and reduces mean Chamfer Distance from 28.37 to 0.74 versus zero-shot generation.

  • Problem

    Existing text-to-CAD methods lack either geometric verification or feedback precise enough to resolve dimensional errors while preserving global shape coherence.

  • Method

    CADSmith uses five specialized agents, nested execution and geometric correction loops, OpenCASCADE measurements, an independent vision-language Judge, and API-documentation retrieval.

  • Results

    100% execution success and mean Chamfer Distance reduced from 28.37 to 0.74 versus the zero-shot baseline.

  • Takeaways & Limitations

    Closed-loop refinement with programmatic geometric feedback substantially improves the quality and reliability of LLM-generated CAD models.

  • Takeaways & Limitations

    Three fixed isometric views may miss small-scale errors such as gaps at joints or incomplete boolean unions.

Abstract

from arXiv · show

Existing methods for text-to-CAD generation either operate in a single pass with no geometric verification or rely on lossy visual feedback that cannot resolve dimensional errors. We present CADSmith, a multi-agent pipeline that generates CadQuery code from natural language. It then undergoes an iterative refinement process through two nested correction loops: an inner loop that resolves execution errors and an outer loop grounded in programmatic geometric validation. The outer loop combines exact measurements from the OpenCASCADE kernel (bounding box dimensions, volume, solid validity) with holistic visual assessment from an independent vision-language model Judge. This provides both the numerical precision and the high-level shape awareness needed to converge on the correct geometry. The system uses retrieval-augmented generation over API documentation rather than fine-tuning, maintaining a current database as the underlying CAD library evolves. We evaluate on a custom benchmark of 100 prompts in three difficulty tiers (T1 through T3) with three ablation configurations. Against a zero-shot baseline, CADSmith achieves a 100% execution rate (up from 95%), improves the median F1 score from 0.9707 to 0.9846, the median IoU from 0.8085 to 0.9629, and reduces the mean Chamfer Distance from 28.37 to 0.74, demonstrating that closed-loop refinement with programmatic geometric feedback substantially improves the quality and reliability of LLM-generated CAD models.

I. INTRODUCTION

CADSmith addresses the difficulty of generating CAD that is both dimensionally exact and geometrically valid by combining programmatic measurements with holistic visual feedback in nested refinement loops.

  • CAD outputs must be dimensionally exact, geometrically valid, and suitable for downstream manufacturing.
  • LLMs can generate plausible CAD code but routinely hallucinate dimensions and misuse APIs, requiring exact geometric feedback.
  • Kernel measurements expose dimensional discrepancies, while visual inspection detects malformed or globally incoherent shapes that numerical checks can miss.
  • Prior approaches use lossy visual feedback or single-pass generation, leaving dimensional and geometric errors unresolved.
  • CADSmith assigns planning, coding, execution, validation, and refinement to specialized agents connected by inner execution and outer geometric correction loops.

A. Contributions

CADSmith contributes a decomposed multi-agent architecture, dual-loop geometric validation, independent visual judging, and documentation-based retrieval instead of fine-tuning.

  • CADSmith decomposes text-to-CAD generation across Planner, Coder, Executor, Validator, and Refiner agents.
  • Its inner loop corrects execution errors, while its outer loop uses OpenCASCADE measurements to drive geometric correction.
  • An independent Claude Opus Judge combines three-view inspection with kernel metrics to reduce self-confirmation bias.
  • Retrieval-augmented generation over CadQuery API documentation avoids fine-tuning and keeps the system current as the library evolves.

A. Text-to-3D and Text-to-CAD Generation

Text-to-CAD research spans learned representations, multimodal generation, code synthesis, and iterative self-refinement, while CADSmith closes the loop with precise kernel and visual feedback.

  • DeepCAD and Text2CAD generate CAD through learned command sequences, whereas LLM-based methods increasingly generate structured geometric outputs from pretrained models.
  • OpenECAD and CAD-MLLM extend CAD generation to editable designs and multiple input modalities, respectively.
  • Query2CAD uses GPT-4, FreeCAD scripts, and BLIP-2 visual feedback, while Text-to-CadQuery fine-tunes on 170,000 description-code pairs and evaluates mesh similarity.
  • Self-Refine research identifies informative feedback as the critical bottleneck for iterative language-model correction.
  • CADSmith combines exact OpenCASCADE dimensional feedback with a vision-language Judge inside a closed refinement loop.

III. METHODS

CADSmith generates CadQuery models through five agents and two bounded correction loops, then evaluates them on a hand-validated 100-prompt benchmark spanning three complexity tiers.

  • Pipeline: Five specialized agents generate CAD models: Planner, Coder, Executor, Validator, and Refiner.
  • Correction loops: The inner loop retries execution-error correction up to three times, while the outer loop refines incorrect geometry up to five times.
  • Benchmark: The benchmark contains 100 natural-language prompts paired with handwritten CadQuery reference scripts across three difficulty tiers.
  • Benchmark: T1 contains 50 basic primitives, T2 contains 25 engineering parts, and T3 contains 25 complex multi-feature parts.
  • Benchmark: Reference scripts were manually written, executed for valid solids, and visually inspected before inclusion.

B. Planner Agent

The Planner converts natural-language CAD prompts into structured design specifications that guide downstream code generation. It extracts components, dimensions, constraints, and feature details while leaving CadQuery implementation to the Coder.

  • Planner role: The Planner outputs a JSON design plan containing components, target bounding-box dimensions, geometric constraints, and downstream notes.Its role is to decompose design intent into an unambiguous specification rather than generate code.
  • Benchmark scope: The benchmark spans T1 single primitives, T2 boolean combinations and hole patterns, and T3 multi-step constructions with workplane changes, sweeps, and complex feature interactions.These tiers define progressively more involved planning requirements.
  • Planner role: A T3 example plan can encode component geometry, explicit Z-ranges, a center bore, bolt-hole patterns, and a keyway slot.The example includes three cylindrical components, a 14 mm bore, six bolt holes on a 38 mm pitch circle, and a 5 mm keyway.
  • Planner-to-Coder handoff: The Coder receives the design plan and generates an executable CadQuery Python script using retrieval-augmented API documentation instead of fine-tuning.The API knowledge base covers 155 Workplane methods and 28 worked construction examples.
  • Planner-to-Coder handoff: The Error Refiner uses traceback-matched error-solution patterns in addition to API documentation when correcting failed CadQuery code.The error database covers 25 common CadQuery and OpenCASCADE failure modes, while retrieval uses deterministic keyword matching.

D. Executor

The Executor runs generated CadQuery scripts deterministically in isolation, exporting successful solids and extracting exact kernel measurements. Execution failures produce traceback information for an inner correction loop, while successful code proceeds to multimodal validation.

  • Execution: The Executor runs CadQuery scripts in an isolated Python subprocess with a 60-second timeout and makes no LLM calls.Successful scripts export STEP and STL files; failed scripts return the full traceback and error type.
  • Geometric outputs: Successful execution yields OpenCASCADE measurements of volume, bounding-box dimensions, center of mass, topology counts, and solid validity.The extracted counts include faces, edges, and vertices.
  • Execution correction: When execution fails, an Error Refiner receives the code, traceback, and retrieval context, then retries corrected code up to three times.This inner loop targets code-execution errors before geometric validation begins.
  • Validation: The Validator uses Claude Opus as an independent LLM-as-a-Judge, stronger than the Claude Sonnet generation model, to reduce self-evaluation confirmation bias.Validation is applied after code executes successfully.
  • Validation: The Judge combines the prompt, generated code, exact kernel measurements, and a three-view rendered image to produce a pass/fail decision with analytical feedback.The views cover overall 3D shape, top-face features, and front-profile structure.

F. Geometric Refinement

Geometric refinement forms an outer correction loop: the Refiner targets validator-identified discrepancies, incorporates prior attempts to avoid oscillation, and repeats execution until validation passes or five iterations are exhausted.

  • Outer correction loop: The Refiner receives the current code, exact Judge discrepancies, design plan, original prompt, and prior refinement history.At iteration three and beyond, it also receives explicit escalation guidance.
  • Outer correction loop: Corrected code returns to the Executor, and the outer loop continues until the Validator passes the geometry or five refinement iterations are reached.Each refinement therefore restarts execution and validation.

G. Evaluation Metrics

CADSmith evaluates generated geometry with Chamfer Distance, F1, and volumetric IoU in absolute millimeter space. Meshes are co-registered and aligned with ICP before metric computation, preserving dimensional accuracy alongside shape similarity.

  • Metrics: Chamfer Distance measures bidirectional squared nearest-neighbor distance between 10,000 sampled surface points, with lower values indicating closer alignment.The metric compares sampled surfaces in both directions.
  • Metrics: F1 Score is the harmonic mean of precision and recall at τ = 1.0 mm, classifying sampled points by nearest-neighbor distance.A point is correct when its nearest neighbor on the other surface lies within the threshold.
  • Metrics: Volumetric IoU compares voxelized occupancy grids at 1.0 mm resolution, with adaptive coarsening for parts exceeding 100 mm extent.The coarsening prevents memory exhaustion for larger parts.
  • Metric space: Absolute millimeter-space evaluation captures both shape and dimensional accuracy, unlike normalized coordinates that can score differently sized matching shapes identically.This choice matches the benchmark’s explicit millimeter dimensions.
  • Preprocessing: Before scoring, meshes are translated by bounding-box centers and aligned with Iterative Closest Point to resolve orientation mismatches.The alignment compensates for different construction frames.

H. Ablation Configurations

CADSmith is compared with a zero-shot one-pass baseline and a no-vision variant to isolate the effects of its refinement and visual validation components. Across 100 benchmark entries, the full pipeline achieves perfect execution and substantially lowers mean Chamfer Distance.

  • Ablation setup: The zero-shot baseline uses one Claude Sonnet call without planning, retrieval, refinement, validation, or vision.It receives only the raw prompt and must produce CadQuery code in one pass.
  • Ablation setup: The no-vision ablation retains the full agent pipeline but removes rendered images from the Judge’s input.The Judge still receives the prompt, generated code, and kernel metrics.
  • Ablation setup: The evaluation compares the full vision pipeline, a no-vision ablation, and a zero-shot baseline across all 100 benchmark entries.Metrics are computed in absolute millimeter space with ICP alignment.
  • Overall results: 100% execution success is achieved by the full pipeline, which also attains the highest scores across all reported metrics.The comparison is summarized in Table I.
  • Overall results: Mean Chamfer Distance falls from 28.37 for the zero-shot baseline to 0.74 for the full pipeline.This metric exposes catastrophic outlier failures that median metrics can obscure.
  • Convergence: 88 of 100 full-pipeline entries converge at iteration 0, with an average of 0.13 refinement iterations per entry.The remaining 12 entries require one or two additional iterations before validation accepts the geometry.

B. Per-Tier Breakdown

Performance declines with increasing geometric complexity, while vision is especially valuable for T3 parts. The tier results also show that the full pipeline can correct severe zero-shot geometry failures.

  • Tier performance: Full-pipeline performance degrades smoothly from T1 to T3 as geometric complexity increases.T1 and T2 achieve near-perfect F1 scores, whereas T3 parts average 0.886.
  • Validator views: Three rendered views combine isometric, high-angle rear, and front-profile perspectives for the Validator Judge.They expose overall shape, top-face features, and vertical structures while being cross-referenced with kernel metrics and the prompt.
  • Tier performance: T1 and T2 reach near-perfect F1 scores of 0.999 and 0.998, while T3 averages 0.886.The lower T3 result is consistent with multi-step construction and feature interactions.
  • Vision contribution: On T3, removing vision increases mean Chamfer Distance from 1.42 to 49.68 and decreases mean F1 from 0.85 to 0.74.The comparison concerns the full vision and no-vision configurations.
  • Vision contribution: On T1 and T2, the no-vision configuration performs comparably, whereas complex T3 geometry exposes more false convergence without visual evidence.Kernel metrics alone are reported as sufficient for the simpler tiers.
  • Largest improvements: The full pipeline improves T3 023 from F1 = 0.037 under zero-shot generation to F1 = 0.943.Other reported improvements include T1 021 from 0.168 to 0.995 and T2 005 from 0.095 to 0.867.

D. Failure Modes

CADSmith still produces structurally different solids and near-miss manufacturing failures despite strong aggregate validation results. These cases expose limits in construction fidelity and fixed-view vision assessment.

  • Structural divergence: F1 = 0.57 and F1 = 0.59 mark T3 cases where generated construction diverged from the reference, yielding valid but structurally different solids.These failures occurred in T3 016 and T3 024.
  • Near-miss geometry: F1 = 0.963 and IoU = 0.985 did not prevent small gaps between the quadcopter frame’s arms and central hub.The part converged on the first iteration and passed both metric and Judge assessment.
  • Near-miss geometry: The T3 019 gaps evaded both kernel metrics and the vision Judge, showing that the current validation stack can miss small-scale joint errors.The gaps were difficult to resolve at the rendered scale and viewing angles.
  • Validation limitations: Three fixed isometric views are not always sufficient to detect gaps at joints or incomplete boolean unions.Suggested remedies include adaptive view selection, higher-resolution joint crops, and additional views guided by Judge uncertainty.
Loading 2603.26512v1…