PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Improving CAD Generation Agents with Finite Element Analysis as Feedback

Guijin Son, Jehyun Park, Seyeon Park, Sunghee Ahn, Youngjae Yu

arXiv:2605.17448v2cs.GRcs.CL

TL;DR

Learned CAD generators rarely evaluate complete engineering artifacts, especially assembled designs with physical requirements. This paper introduces blueprint planning, 21-view inspection, and FEA feedback for assembled STEP generation, improving reconstruction metrics while strict-passing artifacts remain rare.

  • Problem

    Learned CAD systems typically evaluate isolated geometry, visual plausibility, or existing-part assembly constraints rather than complete engineering validity.

  • Method

    The paper generates assembled STEP artifacts from engineering briefs using structured blueprints, 21-view inspection, and finite-element feedback.

  • Results

    Blueprinting raises GPT-5.5/xhigh Box-IoU from 0.444 to 0.592 on S2O, while 21-view feedback raises Fusion360 IoU from 0.397 to 0.505.

  • Takeaways & Limitations

    The study moves CAD-agent evaluation toward assembled artifacts assessed jointly for geometric construction, selector metadata, and physical requirements.

  • Takeaways & Limitations

    Repeated feedback loops expose agents to target requirements and failed margins multiple times, reflecting engineering practice but resembling test leakage.

Abstract

from arXiv · show

Computer-aided design (CAD) is the backbone of modern industrial design, yet learned CAD generators still fall short of real engineering pipelines: they neither iterate like engineers nor evaluate what engineering requires. Prior work has treated CAD generation as two disjoint steps, part synthesis and assembly, where the former is graded by proximity to a gold reference and the latter, when handled at all, is reduced to a separate constraint solving step. In this work, we introduce a more industry-native task formulation that requires a model to produce a fully assembled multi-part STEP file from a free-form engineering brief, which is then validated via finite element analysis (FEA). FEA validation reveals that Codex (GPT-5.5) and Claude Code (Opus-4.7) agents do not produce a single strict-passing artifact in the main first-attempt sweep, with the best configuration meeting only about 20% of typed requirements on average. Moreover, we introduce two additional supervision signals, a novel text-only blueprint schema and a 21-view image renderer that aids the agent's visual inspection, that better align the generation loop with how engineers iterate in practice. On S2O and Fusion360, the same feedback tools improve geometric reconstruction, with GPT-5.5/xhigh rising from 0.444 to 0.592 Box-IoU on S2O and from 0.397 to 0.505 on Fusion360. Together these signals move CAD programs toward artifacts that are not only visually plausible but also checked against physical and structural requirements.

1 Introduction

The paper reframes CAD generation as producing assembled STEP artifacts evaluated against geometric and finite-element requirements, rather than only reference similarity or visual plausibility. It introduces blueprint, rich-view inspection, and FEA feedback tools, but shows that the task remains nearly unsolved for frontier models despite partial-credit improvements.

  • Motivation: Learned CAD systems can generate plausible geometry from natural language, but their dominant evaluations remain weakly coupled to engineering validity.Existing outputs are commonly graded by reference-shape distance or rendered visual plausibility.
  • Feedback tools: A 21-view rich-view image judge and CalculiX finite-element feedback are integrated into the CAD-agent revision loop.The image judge renders exterior views, close-ups, and additional calibrated views of the generated STEP.
  • Feedback tools: Blueprints constrain auditable parametric primitives, envelopes, interfaces, and dimensional or functional claims before CAD code is emitted.The blueprint exposes design commitments for validators and retry feedback.
  • Results: 400 first attempts from Codex and Claude Code produce 0 strict-passing artifacts, while one FEA-feedback round adds only 1 strict pass across another 400 revised submissions.Despite this difficulty, 21-view feedback raises GPT-5.5 mean requirement pass on Hephaestus-CCX from 19.4% to 29.3%.
  • Task formulation: The proposed task requires assembled STEP artifacts judged by geometric checks and finite-element analysis requirements.This formulation moves beyond reference matching and visual plausibility toward engineering-grounded validation.
  • Resources and study: Hephaestus-CCX is released as a 50-case benchmark paired with CalculiX evaluation kits and typed pass/fail requirement checkers.The paper also studies structured blueprints, rich-view inspection, and FEA retry feedback in production coding-agent harnesses.

2 Background and Problem Formulation

Prior CAD generation is chiefly evaluated by similarity to a single gold reference and treats part synthesis and assembly separately, diverging from industrial engineering practice. This work frames CAD generation around assembled designs whose structured requirements can be checked through finite element analysis.

  • 2.1 Prior Work on CAD Generation: Gold-reference CAD benchmarks primarily ask how closely generated geometry matches a curated reference solid.Common representations include command tokens, CadQuery, FreeCAD, and Blender scripts, but evaluation remains reference-based.
  • 2.1 Prior Work on CAD Generation: Prior work separates part synthesis from assembly, although industrial design concentrates on joints that mate, fit, clear neighboring components, and carry loads.Tolerance buildup across mates is itself an established engineering discipline.
  • 2.2 Problem Statement: From Geometric Matching to Engineering Validation: Industrial CAD design uses an iterative loop of dimensionally precise blueprint authoring, rendering, section cuts, load intuition, and revision.The paper identifies blueprint authoring as one issue preventing direct translation of this workflow into an LLM-driven loop.
  • 2.2 Problem Statement: From Geometric Matching to Engineering Validation: A high IoU score can still miss millimeter-scale bolt-pattern errors or structural buckling, while one gold reference rejects other engineering-valid solutions.The paper therefore contrasts geometric resemblance with satisfaction of physical constraints.
  • 2.2 Problem Statement: From Geometric Matching to Engineering Validation: 50 prompt-and-requirement pairs define the Hephaestus-CCX benchmark, comprising 20 single-part and 30 multi-part cases selected from 466 candidate briefs.The briefs draw on patents, supplier datasheets, engineering standards, industrial catalogs, and intercollegiate sources.
  • 2.2 Problem Statement: From Geometric Matching to Engineering Validation: FEA becomes benchmarkable only when each prompt includes structured pass/fail criteria that solver outputs can be checked against.The harness evaluates requirements through parametric checks without human interpretation.
  • 2.2 Problem Statement: From Geometric Matching to Engineering Validation: The target artifact is an ISO 10303 STEP AP242 file for exchanging 3D CAD models between systems, specifically supporting mechanical assemblies.The benchmark’s representative criteria combine geometry and CalculiX verdicts for requirements such as tube sizing, yielding, deflection, and rollover buckling.

3 Building a CAD-Agent Pipeline with Engineering Feedback

The pipeline assigns design decisions and iterative revision to an LLM agent while a deterministic controller executes tools, validates STEP artifacts, and routes compact engineering feedback. Optional blueprinting, 21-view inspection, and controller-level FEA align CAD generation with iterative engineering workflows.

  • Pipeline architecture: The LLM agent writes CadQuery, exports STEP, inspects feedback, and revises geometry and selector metadata, while a deterministic controller executes and validates the pipeline.The controller isolates attempts, runs deterministic checks, parses requirement verdicts, and returns concise reports for subsequent revisions.
  • Blueprint planning: Blueprinting converts the engineering brief into a design plan specifying requirements, materials, load paths, interfaces, selectors, and verification targets before CAD construction.Each part is decomposed into construction units drawn from a closed parametric grammar.
  • Visual inspection: The rich-view pass renders assembled STEP artifacts through 21 calibrated ParaView5 views, combining exterior coverage, feature close-ups, and x-ray views of internal mating and clearance.These fixed views provide static inspection context analogous to walking around an assembly, zooming into interfaces, and taking section cuts.
  • Visual inspection: The agent evaluates renders alongside deterministic measurements of dimensions, mating expectations, and hole positions, then records compact typed verdicts, issues, failure categories, claims, and retry advice.The report is designed to fit in a single agent context while covering external surfaces and internal mating faces.
  • FEA feedback: FEA runs outside the agent under controller control with the fixed Hephaestus-CCX CalculiX kit, making one solver evaluation correspond to one explicit feedback loop.Repeated loops expose requirements and failed margins as the agent revises toward satisfying known engineering constraints, matching iterative engineering practice.

4 Experimental setup

The experiments evaluate coding agents on one FEA-graded assembly benchmark and two geometric reconstruction benchmarks, using strict requirement-based and geometric similarity metrics. Models are tested through Codex and Claude Code under default harness configurations and multiple reasoning-effort settings.

  • Benchmarks: Three benchmarks comprise Hephaestus-CCX, sampled S2O, and sampled Fusion 360 Gallery Assembly Dataset.Hephaestus-CCX contains 50 cases, while S2O and Fusion 360 contain 133 and 225 cases, respectively.
  • Benchmarks: Hephaestus-CCX uses Strict pass and Mean req pass, measuring all-requirements success and average per-case requirement pass fraction.Strict pass counts cases where every typed requirement passes; Mean req pass averages the per-case requirement pass fraction.
  • Benchmarks: S2O and Fusion 360 are scored with Chamfer distance (CD, ↓), F-score at τ=1% (F1%, ↑), and bounding-box IoU (Box-IoU, ↑).These geometric benchmarks do not provide natural-language typed requirements.
  • Models: Codex runs GPT-5.5 and GPT-5.4, while Claude Code runs Opus-4.7 and Sonnet-4.6, without custom generation parameters.Each harness uses its default generation configuration, and experiments evaluate the highest and second-highest exposed reasoning settings.

5 Results

Results show that frontier coding agents rarely produce engineering-sound CAD under strict FEA grading, while blueprint and image-based feedback generally improve geometry, especially for GPT-5 models. Initial design quality and ability to repair designs using FEA feedback emerge as distinct capabilities.

  • 5.1 FEA-included evaluation: Strict FEA grading leaves Hephaestus-CCX nearly unsolved: only Codex GPT-5.5/high passes any single-part or multi-part artifact.All other evaluated models fail to build a strict-passing artifact in the reported Table 2 runs.
  • 5.1 FEA-included evaluation: 32.7% is Claude Code Opus-4.7/xhigh’s first-attempt single-part mean requirement pass rate, leading the GPT counterparts.Mean requirement pass provides a less binary view than strict pass/fail grading.
  • 5.1 FEA-included evaluation: Higher reasoning effort does not consistently improve performance, with high sometimes beating xhigh after retry and Opus-4.7/xhigh outperforming max in selected settings.The results suggest that test-time scaling depends on how agents use additional compute, not only on spending more compute.
  • 5.1 FEA-included evaluation: 18.8 percentage points is Codex GPT-5.4/high’s gain after one FEA-retry round, despite starting at 23.3% on single-part tasks.Claude Code Opus-4.7/xhigh starts stronger at 32.7% but slightly regresses after feedback.
  • 5.2 Blueprint and image-based feedback: About 0.09 is the average Box-IoU gain from blueprint and rich-view feedback across the two GPT rows on S2O and Fusion 360.Both tools improve geometry in these benchmarks, although improvement magnitude varies across tools, benchmarks, and models.

6 Test-time scaling

Test-time scaling is more effective when computation is spent in a closed-loop interaction with FEA feedback rather than only increasing reasoning effort within one attempt. Repeated, concrete feedback improves every configuration across retries and can repair structural failures that initially violate multiple checks.

  • Feedback versus reasoning effort: 13.4 points: FEA-guided revision raises the mean-requirement score across Table 2 cells, while expanded reasoning effort alone does not improve performance monotonically.The paper distinguishes computation within a single attempt from computation spent interacting with an external evaluator.
  • Repeated feedback: Every plotted configuration improves from the first attempt to the second, although higher reasoning effort does not consistently outperform lower-effort feedback runs.Codex GPT-5.5/high exceeds GPT-5.5/xhigh, while Claude Code Opus-4.7/xhigh exceeds Opus-4.7/max.
  • Repair mechanisms: Nine strict-passing artifacts from the longest GPT-5.5/high run reveal four repair types, including structural retuning that converts an underbraced failed column into one passing compression, bending, and deflection checks.The AISC steel column begins too slender and underbraced before becoming a four-chord braced structure.

7 Conclusion

This work shifts CAD-agent evaluation toward engineering validation by requiring assembled STEP artifacts from engineering briefs and assessing them with finite-element requirements. It contributes a benchmark and studies structured blueprints, rich-view inspection, and FEA feedback in production agent harnesses.

  • The study formulates CAD generation as producing assembled STEP artifacts from engineering briefs under finite-element requirements.
  • Hephaestus-CCX is introduced as a 50-case benchmark with CalculiX evaluation kits and typed pass/fail checkers.
  • The work studies structured blueprints, rich-view inspection, and FEA feedback inside production agent harnesses.

A Additional model baseline results

Additional OpenCode baselines on Hephaestus-CCX, evaluated without an FEA-retry round, remain in a low partial-credit regime. DeepSeek-V4-Pro reaches 28.6% mean requirement pass on single-part cases and 16.1% on multi-part cases, but complete engineering-valid artifacts remain rare.

  • Evaluation setup: OpenCode baseline rows on Hephaestus-CCX use no FEA-retry round.They are reported separately from the main Codex and Claude Code comparison.
  • Baseline performance: Open-model baselines satisfy some typed requirements but rarely produce complete engineering-valid artifacts.The OpenCode rows occupy the same low partial-credit regime as main production-agent first attempts.
  • Baseline performance: 28.6% mean requirement pass is achieved by DeepSeek-V4-Pro on single-part cases, versus 16.1% on multi-part cases.These results are comparable to the first-attempt range in Table 2.

B Full rich-view ablation results · C Sample S2O and Fusion 360 evaluation prompts · D First-shot versus repair performance

The rich-view ablation finds that additional visual context helps selectively rather than monotonically, while evaluation prompts are generated from target renderings and metadata. The repair analysis compares first-attempt quality with gains from one FEA-feedback round.

  • B Full rich-view ablation results: More image views help, but the 7-view budget can match or outperform 21 views for simpler or exterior-dominated objects.Additional close-up and x-ray views may add inspection burden without new geometric information.
  • B Full rich-view ablation results: The 21-view budget remains valuable when small, occluded, or internal features determine the geometry.The ablation indicates that optimal visual context depends on the target object.
  • C Sample S2O and Fusion 360 evaluation prompts: Evaluation prompts are generated from target renderings and structured metadata rather than authored directly by the researchers.Figures 4 and 5 provide representative target items, followed by their corresponding natural-language prompts.
  • C Sample S2O and Fusion 360 evaluation prompts: The S2O storage-bench prompt describes a single box-form carcass with a top-opening lid, squared corners, and enclosed internal storage.Its proportions are elongated, relatively narrow, and low enough for seating.
  • C Sample S2O and Fusion 360 evaluation prompts: The S2O kitchen-appliance prompt specifies stamped and folded sheet steel, slide rails or formed guides, hinges, and insulated heat-resistant internal construction.The description grounds the geometry in domestic cooking-appliance fabrication and use.
  • C Sample S2O and Fusion 360 evaluation prompts: The Fusion 360 robotic-chassis prompt describes a rigid open steel frame with cutout side plates, internal supports, and offset spoked wheels.These features indicate an electromechanical chassis and drivetrain-oriented assembly.
  • C Sample S2O and Fusion 360 evaluation prompts: The Fusion 360 electronics-board prompt describes a planar PCB assembly with edge-accessible connectors, headers, and surface-mounted components.The accompanying manufacturing description includes PCB fabrication, automated placement, reflow soldering, and through-hole connector insertion.
  • D First-shot versus repair performance: First-attempt mean requirement pass is compared with the percentage-point gain after one FEA-feedback repair round across Hephaestus-CCX configurations.Figure 6 plots each Table 2 row, with dotted least-squares fits shown within each panel.

E Strict-pass retry case studies … G.3 Source catalogs and per-catalog distribution

The paper presents strict-pass CAD retries that improve load paths, simplify geometry, repair hidden physical properties, or expose evaluator-required metadata, alongside a benchmark built from scrubbed, typed, self-contained engineering briefs. The benchmark comprises 466 cases from sixteen catalogs, with structured requirements, verification fields, tier labels, domain classifications, and a fixed 21-view render set.

  • E Strict-pass retry case studies: Nine artifacts became strict passes in the longest GPT-5.5/high-feedback run through retries spanning structural retuning, simplification, hidden-property repair, and checker-contract repair.The retained artifacts pair each earliest failing design with its selected passing retry and checker-derived fields.
  • E Strict-pass retry case studies: Structural retries replaced weak or inefficient load-bearing designs with braced, hollow, or cleaner structures that improved capacity, deflection, stress, fatigue, mass, or modal performance.Examples include a four-chord box column, a lighter stiff roll-protection system, and a hollow box-beam UGV arm.
  • E Strict-pass retry case studies: Checker-contract retries passed strict grading mainly by exposing required metric aliases, mass fields, selector bindings, or mesh-derived mass metadata rather than substantially changing shape.The ISO 10328 P5 pylon also trimmed visible mass, while AIJ and KOSEN designs were even more contract-dominated.
  • F Rich-view render set: The image-feedback judge uses a fixed ParaView 21-view render set scaled to each part’s bounding sphere, with close-ups and x-ray views varying zoom and body alpha.This provides the specified multi-view visual inspection setup.
  • G.1 Authoring methodology and scrub pass: Each brief was authored by catalog identification, inline brief drafting, typed requirement expansion, checker validation against reference geometry, and a final scrub pass.Retention required every requirement to parse, evaluate, and bind without skipping.
  • G.2 File schema: The fixed schema pairs a narrative prompt with structured geometry, materials, load cases, typed R1...Rn pass/fail criteria, verification scope, exclusions, evaluation coverage, and scrubbed provenance.Numeric limits are written inline, while standards are cited by name and their implied values are embedded in the brief.
  • G.3 Source catalogs and per-catalog distribution: The benchmark pool contains 466 cases—318 single-part and 148 multi-part briefs—from sixteen catalogs, while released specifications remove source identity.Catalog sources include patents, supplier datasheets, regional industrial catalogs, and A- and I-series collections.
  • G.3 Source catalogs and per-catalog distribution: Cases receive A+/A/B/C tiers based on source strictness and self-containment, with A+/A cases oversampled for the curated benchmark and every brief additionally classified by physical domain.Per-catalog counts appear in Table 9 and domain distributions in Figure 10.

G.4 Distribution by requirement type and per-brief count · G.5 Selection of the 50-case curated benchmark

The 466-brief pool contains 2,817 typed requirements spanning structural, buckling, vibration, thermal, dimensional, geometric, and material-compliance checks, while fluid and radiation requirements remain outside the CCX harness. The curated 50-case benchmark is stratified for domain, analysis-card, and difficulty coverage, including both single-part and assembly-interface cases.

  • G.4 Distribution by requirement type and per-brief count: 2,817 typed requirements span structural analysis, buckling, vibration, thermal, dimensional, geometric, and material-compliance checks across 466 briefs.Structural analysis is the dominant requirement type.
  • G.4 Distribution by requirement type and per-brief count: Single-part briefs average 5.7 requirements each, while multi-part briefs average 6.8.The medians are 6 and 7, respectively; both brief types range from 3–10 requirements.
  • G.4 Distribution by requirement type and per-brief count: Aerospace and ground-vehicle cases together account for over half the candidate pool, while all thirteen broad domain buckets include both single-part and multi-part cases.This distribution supports diversity in the curated benchmark despite catalog skew.
  • G.4 Distribution by requirement type and per-brief count: Fluid and radiation requirements are retained in the schema but are not evaluable by the CCX harness because they fall outside CalculiX’s scope.They are flagged through requires_non_fea_solver, with external CFD or transport-solver integration deferred to future work.
  • G.5 Selection of the 50-case curated benchmark: The curated benchmark contains 50 cases—20 single-part and 30 multi-part—sampled from the 466-case pool using domain, analysis-type, and difficulty stratification.Domain sampling required representation from every catalog with more than five candidate briefs and prioritized high-tier A+/A cases.
  • G.5 Selection of the 50-case curated benchmark: The 50 cases collectively exercise every *STATIC, *FREQUENCY, *BUCKLE, *DYNAMIC, and *HEAT TRANSFER card supported by the CalculiX backend.This ensures end-to-end analysis-card coverage within the curated set alone.
  • G.5 Selection of the 50-case curated benchmark: The candidate pool is largest in the intercollegiate and foundational A-series catalogs, while engineering standards and patents/datasheets supply most strict-spec briefs.Figure 13 compares the curated 50 cases against the full pool by catalog.
  • G.5 Selection of the 50-case curated benchmark: Difficulty stratification mixes monolithic single-part briefs with multi-part briefs that introduce assembly-interface checks.The assembly checks include weld DCR, mating clearance, and deformed envelope requirements, while single-part briefs mostly exercise R1–R5-type structural or buckling checks.

G.6 Sample briefs

Three sample briefs demonstrate the schema across aerospace and civil structural regimes, pairing engineering prompts with tagged pass/fail requirements. They span a monolithic jet-engine bracket, a multi-part satellite enclosure, and an anchored steel baseplate assembly.

  • Schema coverage: Three briefs span different physics regimes and pair excerpted prompts with tagged pass/fail requirements.The examples cover aerospace structural single-part and multi-part designs plus a civil structural multi-part assembly.
  • Brief A: GE jet engine bracket: 633 MPa is the binding von Mises stress limit for the GE jet-engine bracket across four load cases.The brief also requires displacement ≤ 1.0 mm, mass ≤ 1.0 kg, at least 40% mass reduction versus a bulk-block baseline, and a first-mode load factor ≥ 2.0 for LC3.
  • Brief B: NASA-HDBK-7005 random-vibration satellite enclosure: 220.8 MPa is the sine-peak-stress and quasi-static 30g stress limit for the NASA satellite enclosure, while shock SRS stress is limited to 221.4 MPa.The multi-part assembly additionally requires empty enclosure mass ≤ 800 g.

H Release, Reproducibility, Compute, and Impact · I Full blueprint for Brief 1

The paper releases an anonymized, reproducible CAD-agent benchmark and evaluates its engineering-grounded approach with explicit compute and responsible-use disclosures. It also specifies a schema-v4 blueprint for the Baja FSAE tubular space frame, including geometry, materials, supports, primitives, and acceptance claims.

  • H Release, Reproducibility, Compute, and Impact: Hephaestus-CCX is released as an anonymized supplemental zip containing benchmark briefs, requirement metadata, the evaluation harness, and execution scripts.Paper-authored assets and code use the MIT License, while third-party terms remain applicable.
  • H Release, Reproducibility, Compute, and Impact: The reproduction pipeline targets Python 3.12 and pins major dependencies including cadquery==2.7.0, openai==2.29.0, litellm==1.82.4, langgraph==1.1.3, and google-genai==1.68.0.The CCX grader is scripts/ccx_eval/grade_ccx.py, with configurable CCX, evaluation Python, and Gmsh settings.
  • H Release, Reproducibility, Compute, and Impact: 359,525.79 wall-clock seconds were used by the main long GPT-5.5/high feedback run, which used 8 jobs, a 2400-second model timeout, and a 900-second evaluation timeout.All reported orchestration, CAD execution, meshing, rendering, and FEA jobs ran on CPU workers, with aggregate metrics reported on fixed subsets rather than repeated-run estimates.
  • H Release, Reproducibility, Compute, and Impact: The approach evaluates generated artifacts against explicit physical and geometric requirements, while its main risks include over-trust, mechanically invalid load paths, and dual-use mechanical design automation.The stated benefit is helping researchers study failures before deployment in design workflows.
  • I Full blueprint for Brief 1: The schema-v4 blueprint for the Baja FSAE tubular space frame decomposes the full frame into eight named tube families, while presenting three representative parts.The families are main_hoop, front_hoop, lower_frame, side_impact, roof_bracing, front_nose, rear_bay, and lateral_bracing.
  • I Full blueprint for Brief 1: The blueprint defines brief:bajaframe in millimeters with x forward, y driver_right, and z up coordinates, using AISI 1018 DOM material with 370 MPa yield strength and a 1.5 safety factor.It also specifies an overall fit envelope of x [-460, 460], y [-50, 50], and z [0, 1100].
Loading 2605.17448v2…