PaperlayerView exact revision on arXiv

Source-linked AI summary

P3D-Bench: Benchmarking MLLMs for Parametric 3D Generation and Structural Reasoning

Yikang Yang, Zhanpeng Hu, Youtian Lin, Mengqi Zhou, Jingxi Xu, Feihu Zhang, Jiaheng Liu, Yao Yao

arXiv:2606.11152v2cs.CV

TL;DR

Existing benchmarks do not jointly assess executable parametric 3D generation and structural reasoning from text or images. P3D-Bench provides a unified evaluation across text, image, and assembly tasks, finding that models often achieve semantic alignment while remaining inaccurate in geometry and part structure.

  • Problem

    Existing benchmarks do not jointly evaluate executable parametric 3D generation, geometric precision, semantic alignment, and assembly structure.

  • Method

    P3D-Bench executes and renders generated programs across Text-to-3D, Image-to-3D, and Assembly-3D tasks, scoring validity, geometry, topology, semantics, and part structure.

  • Results

    J-Sem ≈0.8 versus J-Geo ≈0.35 for the strongest MLLM, while assemblies remain hardest and PartMatchF1 reaches only about 0.5.

  • Takeaways & Limitations

    P3D-Bench makes measurable the gap between plausible coarse shapes and reliable recovery of precise geometry and part structure.

  • Takeaways & Limitations

    The benchmark currently draws on two CAD sources and four output formats, with future work planned to broaden both and evaluate iterative coding agents.

Abstract

from arXiv · show

Multimodal large language models can write code to produce complex programs as well as use programs to do 3D modeling, which opens up a new avenue for 3D generation powered by their priors, world knowledge and reasoning. Yet existing benchmarks rarely evaluate 3D modeling through code. Such modeling demands more than runnable code: from a text or visual specification, a model must generate a parametric 3D program that is geometrically precise, semantically aligned and assembly-consistent. We introduce P3D-Bench, a benchmark for parametric 3D generation. Unlike a 3D mesh, a parametric 3D program exposes explicit dimensions, construction operations and part relations, revealing whether a model recovers a design's structure, not just its appearance. Under a unified protocol, P3D-Bench covers three task families (Text-to-3D, Image-to-3D and Assembly-3D) and scores each output for executability, geometric fidelity, topology, text-grounded constraints, multiview semantic alignment and part-level structure. We evaluate frontier MLLMs and text-only LLMs on 400 text cases, 400 image cases and 203 annotated assemblies, with domain-specific models as reference points. Our extensive evaluation yields three findings. First, assemblies are the hardest setting, where models still fail to compose multiple parts into a coherent structure. Second, models can often recover the global shape and semantic identity of the target object, yet fail to reproduce the precise parametric geometry specified by the input. Third, part-level modeling remains weak on assemblies, where models recover neither the geometry of each part nor the right number of parts. These results position P3D-Bench as a benchmark for evaluating precise parametric geometry and part-level structure in parametric 3D generation.

1 INTRODUCTION

P3D-BENCH introduces a unified benchmark for evaluating executable parametric 3D generation and structural reasoning from text and image specifications. Its evaluations show that models often produce plausible programs while failing to recover precise parametric geometry and coherent part-level structure, especially in full assemblies.

  • Motivation: P3D-BENCH addresses the lack of a benchmark evaluating executable parametric 3D generation as a whole, beyond code execution or spatial reasoning alone.Parametric programs expose editable dimensions, construction steps and part relations, unlike directly predicted meshes.
  • Benchmark design: The benchmark covers Text-to-3D, Image-to-3D and Assembly-3D, requiring increasingly complex inference from single parts to full assemblies with annotated part relations.Text-to-3D uses text for a single part; Image-to-3D uses one rendered view for a multi-part object; Assembly-3D adds assembly- and part-level annotations.
  • Findings: Full assemblies are substantially harder than single parts, while geometric alignment is harder than semantic alignment and the strongest MLLM reaches only about 0.5 PartMatchF1.Executing and rendering plausibly does not ensure recovery of correct parametric geometry or part-level structure.
  • Benchmark design: The unified protocol executes and renders outputs in JSON, OpenSCAD, CadQuery or Three.js, then scores validity, geometry, topology, judge assessment and part-level structure.The evaluation reports four dimensions: Geometry, Topology, Judge and Part.
  • Dataset: The benchmark includes 400 text, 400 image and 203 assembly cases with descriptive, parametric and part-level annotations.P3D-Dataset is built by filtering, annotating and verifying two CAD sources.

2 RELATED WORK

Prior 3D generation mainly targets visual geometry, whereas parametric 3D generation uses code-based representations with explicit structure. Existing benchmarks evaluate isolated capabilities, leaving no unified benchmark for executable parametric and structural correctness across general-purpose and domain-specific models.

  • Visual and parametric 3D generation: Visual 3D methods recover radiance fields or generate meshes from text or images, while parametric 3D generation targets a structured representation rather than appearance alone.The cited visual methods build on NeRF or neural signed distance fields; the supplied passage contrasts these outputs with parametric generation.
  • Visual and parametric 3D generation: Parametric 3D is increasingly produced by general-purpose models writing code, but no unified benchmark evaluates general-purpose LLMs/MLLMs and domain-specific models.This gap spans the model categories named in the passage.
  • Benchmarks related to parametric 3D generation: Existing benchmarks score only one aspect, such as executability, spatial reasoning, or visual quality of generated 3D shapes.The passage associates these aspects with Jimenez et al. (2024), Wang et al. (2026a) and Zhang et al. (2026b), or He et al. (2023) and Zhang et al. (2025), respectively.
  • Benchmarks related to parametric 3D generation: No existing benchmark scores executable code for both parametric and structural correctness.This is the central limitation identified for prior benchmarks related to parametric 3D generation.

3 P3D-BENCH: TASKS, DATASET AND EVALUATION

P3D-Bench defines parametric 3D generation as executable code-based reconstruction across text, image, and assembly conditions, using four target program formats. Its dataset construction balances filtered CAD cases and annotations, while evaluation separately measures validity, geometry, topology, semantic constraints, and assembly part structure.

  • Task formulation: P3D-Bench evaluates code-generated 3D outputs from text, image, or assembly-level and part-level conditions in minimal JSON, OpenSCAD, CadQuery, or Three.js.A deterministic operator compiles, executes, and renders each generated program for evaluation.
  • Dataset statistics: The final benchmark contains 400 Text-to-3D cases, 400 Image-to-3D cases, and 203 Assembly-3D cases.Assembly cases are further reduced by a deduplicated-part cap and MLLM verification.
  • Annotation pipeline: Annotations provide descriptive and parametric specifications for Text-to-3D, while image-based data receives part and assembly annotations to supply task-specific inputs absent from raw sources.Parametric specifications include dimensions, counts, offsets, and placements, whereas descriptive specifications emphasize shape, features, and function.
  • Dataset construction: The dataset combines Text2CAD’s 176,017 single-part programs with the Fusion 360 Gallery’s 8,251 multi-part assemblies, then filters cases for executability, interpretability, non-redundancy, and complexity balance.The filtering pipeline removes unevaluable or ambiguous records and samples across semantic categories and easy, medium, and hard tiers.
  • Evaluation metrics: Evaluation reports executable validity separately from aligned geometry and topology metrics, then adds MLLM Judge scores for semantic, geometric, and aesthetic correctness.Geometry uses CD, F-scores, normal consistency, and IoU; topology uses NoOE, InvN, and non-manifold measures, while Judge metrics include QA-S, QA-P, J-Sem, J-Geo, and J-Aes.
  • Assembly part metrics: Assembly-3D evaluates part-level shape and count by decomposing predictions, fidelity-checking reassembly, deduplicating and pose-aligning parts, and one-to-one matching them to ground truth.PartFS measures matched-part shape fidelity, while PartMatchF1 uses successful matches to compute part precision and recall.

4 EXPERIMENTS

Experiments show that general-purpose models outperform domain-specific baselines, while performance declines sharply from single-part tasks to assemblies. OpenSCAD is the strongest output format, but precise parameters, part-level geometry, and executable programs remain major weaknesses.

  • Model comparisons: GPT-5.5 and Gemini 3.1 Pro lead the evaluated models, followed by Claude Opus 4.6 and Kimi K2.6, while domain-specific models trail general-purpose models.The remaining GLM, DeepSeek, Qwen, MiMo, and Doubao models form a third tier, including on native tasks and formats.
  • Task difficulty: Assembly-3D is substantially harder than single-part tasks: weaker models drop markedly on assemblies, whereas the strongest models remain relatively high.Per-task bucket scores are averaged over output formats.
  • Output formats: OpenSCAD is the strongest output format because it is most balanced across the four buckets, while JSON is weakest on assembly tasks.The benchmark evaluates seven task–format combinations across minimal JSON, OpenSCAD, CadQuery, and Three.js.
  • Output formats: CadQuery and Three.js approach OpenSCAD on Geo and Judge but underperform for distinct reasons: invalid programs for CadQuery and weak topology and part matching for triangulated Three.js meshes.CadQuery failures reduce all metric scores, while Three.js does not reliably produce watertight meshes or clean per-part solids.
  • Judge bucket details: 0.84 vs. 0.90 averaged over the three shows QA-P below QA-S under parametric specifications, indicating that exact parameter recovery is harder than semantic matching.JSON is the exception because its textual specification is derived from the minimal JSON representation.
  • Part bucket details: ≈0.73 PartFS for GPT-5.5 contrasts with ≈0.80 J-Sem on Assembly-3D, showing stronger global semantic understanding than per-part geometric fidelity.PartFS is averaged over matched parts, while J-Sem is averaged over CadQuery and OpenSCAD.

5 CONCLUSION AND FUTURE WORK

P3D-BENCH evaluates parametric 3D generation and structural reasoning across text, image, and assembly tasks, revealing that plausible executable programs often remain geometrically imprecise and structurally incorrect. Future work will broaden data and formats and evaluate iterative coding agents alongside single-shot models.

  • Conclusion: P3D-BENCH unifies parametric 3D generation and structural reasoning evaluation across text, image, and assembly tasks.The benchmark evaluates multimodal LLMs, text-only LLMs, and domain-specific models.
  • Conclusion: Models often produce executable, plausible programs while remaining unreliable on precise parametric geometry and structural correctness.The reported weaknesses include precise dimensions, feature placement, topology, and part structure.
  • Future work: Future work will expand the benchmark beyond two CAD sources and four output formats with more diverse data and formats such as Blender and Unreal Engine.The planned additions will retain unified parametric and structural scoring.
  • Future work: Future evaluations will include coding agents such as Codex, Claude Code, and Gemini CLI that iteratively write, execute, and revise programs.These agents will be studied in addition to the single-shot models evaluated here.

Supplementary Material … A.1 SOURCE PREPROCESSING

The supplementary material provides implementation details and analyses supporting the benchmark, with Appendix A expanding dataset source preprocessing, filtering, annotation, and examples. Source preprocessing removes unevaluable Text2CAD records, ranks surviving candidates by modeling-operation complexity, and sends Fusion 360 assemblies directly into common filtering.

  • OVERVIEW: The appendix collects implementation details and supplementary analyses supporting the benchmark description.It covers dataset preprocessing, evaluation implementation, and additional analyses.
  • A DATASET PROCESSING DETAILS: Appendix A documents dataset source preprocessing, filtering, annotation implementation details, and annotation examples.These materials expand the dataset construction pipeline described in Section 3.2.
  • A DATASET PROCESSING DETAILS: The dataset construction pipeline comprises source preprocessing, filtering implementation, annotation implementation, and concrete outputs from two dataset tracks.The two tracks are Text2CAD and Fusion 360 Gallery.
  • A.1 SOURCE PREPROCESSING: Text2CAD removes records with missing programs, zero-depth extrusions, or empty shapes because they cannot be evaluated.This is the unevaluable-record removal stage.
  • A.1 SOURCE PREPROCESSING: Text2CAD ranks surviving candidates using sketch and extrude operation counts plus face count as heuristic geometric-complexity statistics.The score is derived from the modeling-operation sequence.
  • A.1 SOURCE PREPROCESSING: Text2CAD retains the top-ranked cases as candidates after geometric-complexity ranking.Ranking follows unevaluable-record removal.
  • A.1 SOURCE PREPROCESSING: Fusion 360 Gallery assemblies enter the common filtering pipeline directly because they already contain multi-part structure.They bypass Text2CAD-specific complexity preselection.

A.2 FILTERING IMPLEMENTATION … B EVALUATION DETAILS

The benchmark filters candidates through MLLM-based semantic and complexity review, removes near-duplicates, and constructs validated Text2CAD and Fusion 360 annotations. Evaluation details cover alignment, geometric and judge-based scoring, assembly part metrics, and bucket aggregation.

  • A.2 FILTERING IMPLEMENTATION: Gemini 3.1 Pro assigns semantic category, semantic confidence, and complexity tier from renders, geometric metadata, and few-shot examples.These assignments guide downstream complexity-balanced sampling.
  • A.2 FILTERING IMPLEMENTATION: DINOv2 feature matching over render embeddings removes visually similar candidates from Text2CAD samples and Fusion 360 assemblies.Embeddings use renders rasterized from OpenCascade geometry and the CLS token from the last hidden state.
  • A.4 ANNOTATION EXAMPLE OUTPUTS: Figure 12 illustrates Text2CAD descriptive and parametric specifications alongside Fusion 360 assembly-level captions and per-part descriptions.The examples show the two data sources and the annotation fields used by the Text-to-3D and Assembly-3D tasks.
  • A.3 ANNOTATION IMPLEMENTATION: Text2CAD specifications are generated from structured records of sketches, extrusions, and bounding boxes together with renders, then annotated by GPT-5.5.A static validator checks every number and feature against the geometric record; failing cases are audited and repaired against the render.
  • A.3 ANNOTATION IMPLEMENTATION: Claude Opus 4.6 labels unique Fusion 360 parts from STEP-derived geometry and renders, retaining assemblies with at most 20 deduplicated parts.The same MLLM produces assembly-level annotations from part annotations, the assembly render, and assembly information.
  • B EVALUATION DETAILS: Evaluation implementation covers mesh alignment and geometric metrics, MLLM Judge scoring, Assembly-3D part metrics, and bucket aggregation.These details are presented as orthogonal to dataset construction.

B.1 MESH ALIGNMENT AND GEOMETRY METRICS … C ADDITIONAL ANALYSES

The paper defines task-specific mesh alignment, geometry and topology metrics, then supplements them with MLLM-based semantic and visual judging. It also specifies assembly part scoring, bucket aggregation, and two supplementary diagnostics for shared-case comparison and decomposition fidelity.

  • B.1 MESH ALIGNMENT AND GEOMETRY METRICS: Predictions are compiled into meshes and aligned through normalization, translation, rotation, and bounded scale-position refinement to minimize bidirectional Chamfer Distance.Text-to-3D preserves explicit scale and applies only translation and rotation alignment.
  • B.1 MESH ALIGNMENT AND GEOMETRY METRICS: IoU uses CSG solids for single-part Text-to-3D and voxelized occupied volume for Image-to-3D and Assembly-3D outputs.CSG IoU requires fully manifold solids, whereas voxel IoU requires a closed surface.
  • B.2 MLLM JUDGE IMPLEMENTATION: The Judge evaluator is Gemini 3.1 Pro, using synthesized QA banks for Text-to-3D and multiview visual ratings for descriptive, image, and assembly tasks.QA-S contains four semantic questions per case, while QA-P contains eight parametric questions covering dimensions, counts, holes, arrays and placements.
  • B.2 MLLM JUDGE IMPLEMENTATION: The visual Judge rates multiview renders without source code on a 1–10 scale, using J-Sem for descriptive specifications and J-Sem, J-Geo, and J-Aes for image and assembly tasks.The three axes cover semantic similarity, geometric similarity and aesthetic quality.
  • B.3 ASSEMBLY-3D Part METRIC IMPLEMENTATION: Assembly per-part metrics decompose the predicted whole-assembly program with Claude Opus 4.6 and apply a fidelity gate before scoring parts.Cases are excluded only when both CD > τdec and IoUV < fdec, with τdec = 5×10−4 and fdec = 0.95.
  • B.4 BUCKET SCORE AGGREGATION DETAILS: Bucket scores normalize sub-metrics to [0, 1], invert lower-better InvN and NM, clip CD at a worst threshold of 0.01, and average applicable metrics equally.Headline tables additionally report executable validity, while cross-format Average columns average per-format bucket scores.
  • C ADDITIONAL ANALYSES: The additional analyses comprise a shared-case comparison and a decomposition-fidelity analysis supporting the main comparison.These diagnostics are grouped as supplementary analyses.

C.1 IMAGE-TO-3D VS. ASSEMBLY-3D COMPARISON · C.2 ASSEMBLY-3D DECOMPOSITION FIDELITY

On 203 shared cases, Assembly-3D generally lowers Image-to-3D performance, while decomposition fidelity is similar across models once predictions compile. Assembly-3D differences are driven primarily by executable-prediction rates rather than subsequent decomposition accuracy.

  • C.1 IMAGE-TO-3D VS. ASSEMBLY-3D COMPARISON: Assembly-3D adds part-level and assembly-level annotations, requiring one executable 3D output to express described parts and relations.The aligned comparison restricts Image-to-3D to 203 shared cases and compares shared Geo, Topo, and Judge buckets on CadQuery and OpenSCAD.
  • C.1 IMAGE-TO-3D VS. ASSEMBLY-3D COMPARISON: 7 of 8 models drop on the CadQuery/OpenSCAD average under Assembly-3D, with a mean change of −0.040.The average decrease is −0.057 on CadQuery and −0.023 on OpenSCAD.
  • C.1 IMAGE-TO-3D VS. ASSEMBLY-3D COMPARISON: −0.206 is the sharpest change, occurring for GLM 5V Turbo on CadQuery.Claude Opus 4.6 and Kimi K2.6 decrease by about 0.04 on the cross-format mean, while GPT-5.5 increases by +0.008.
  • C.1 IMAGE-TO-3D VS. ASSEMBLY-3D COMPARISON: Table 6 averages Geo, Topo, and Judge for each model, format, and task, excluding the assembly-specific Part bucket.It reports Assembly-3D minus Image-to-3D as ∆ on the 203 shared cases.
  • C.2 ASSEMBLY-3D DECOMPOSITION FIDELITY: Table 7 evaluates decomposition using raw Ddec, IoUVdec, Valid%, and fidelity-gate exclusions.Ddec and IoUVdec are averaged over the same valid, successfully decomposed cases; exclusions require Ddec > 5×10−4 AND IoUVdec < 0.95.
  • C.2 ASSEMBLY-3D DECOMPOSITION FIDELITY: Ddec remains in the 10−4–10−3 range even for low-Valid% models such as MiMo v2 Omni and Qwen3.6-Plus.Once predictions compile, decomposition reproduces union geometry tightly and at nearly the same level across models.
  • C.2 ASSEMBLY-3D DECOMPOSITION FIDELITY: Valid% ranges from 22% to 99% on CadQuery, whereas Ddec and IoUVdec hardly distinguish models.OpenSCAD is described as the more reliable ta...

D COST-QUALITY ANALYSIS

The cost-quality analysis compares task-specific P3D-BENCH scores with full-task execution costs, showing that GPT-5.5 leads all tasks but costs the most. Gemini 3.1 Pro closely approaches GPT-5.5 at substantially lower cost, especially on grounded tasks, while lower-cost models show a quality gap.

  • Cost-quality comparison: Figure 13 compares task-specific P3D-BENCH scores with the cost of running each full task.The task-separated view keeps text-only, image-grounded, and assembly-grounded workloads distinct because their token profiles and score buckets differ.
  • Cost-quality comparison: GPT-5.5 is the highest-scoring model on all three tasks and the most expensive evaluated model in every panel.The figure’s score is the arithmetic mean of post-executability quality buckets, excluding Valid.
  • Cost-quality comparison: Gemini 3.1 Pro trails GPT-5.5 by 0.008 on Image-to-3D and 0.022 on Assembly-3D, while costing about one quarter as much on both tasks.This close score-cost relationship is most pronounced on the grounded tasks.
  • Cost-quality comparison: Lower cost is accompanied by a clear quality gap across all three tasks.The supplied passage begins describing the lowest-cost Text-to-3D model but does not provide its remaining quantitative comparison.

E PER-TASK FULL-METRIC AGGREGATES · F QUALITATIVE VISUALIZATIONS

The paper reports full per-task metric aggregates across supported output formats and supplements them with qualitative visualizations. These analyses cover Text-to-3D, Image-to-3D, Assembly-3D, and representative outputs across specification and format combinations.

  • E PER-TASK FULL-METRIC AGGREGATES: Table 8 reports full Text-to-3D metrics for 400 cases across descriptive and parametric specifications.Descriptive metrics cover render-grounded QA and J-Sem; parametric metrics cover Geo, Topo, and Judge for JSON and OpenSCAD.
  • E PER-TASK FULL-METRIC AGGREGATES: Text-to-3D descriptive metrics evaluate render-grounded QA and J-Sem across JSON and OpenSCAD.
  • E PER-TASK FULL-METRIC AGGREGATES: Text-to-3D parametric metrics are separated into JSON and OpenSCAD evaluations.The parametric submetrics are Geo, Topo, and Judge.
  • E PER-TASK FULL-METRIC AGGREGATES: Table 9 details Image-to-3D Geo, Topo, and Judge submetrics across CadQuery, OpenSCAD, and Three.js.CADRILLE and CAD-CODER emit CadQuery only and are listed separately.
  • E PER-TASK FULL-METRIC AGGREGATES: Table 10 details Assembly-3D Geo, Topo, Judge, and Part submetrics across CadQuery and OpenSCAD.The Part bucket contains PartMatchF1 and PartFS, while PartMatchP and PartMatchR are separate diagnostics.
  • F QUALITATIVE VISUALIZATIONS: Figure 14 presents representative executable OpenSCAD outputs from six models across Text-to-3D, Image-to-3D, and Assembly-3D.Each row pairs the task input with model outputs annotated by per-case bucket scores.
  • F QUALITATIVE VISUALIZATIONS: Text-to-3D visualizations compare five fixed target parts across four specification-format combinations.The combinations cover descriptive and parametric specifications in JSON and OpenSCAD.
  • F QUALITATIVE VISUALIZATIONS: Image-to-3D visualizations use identical input cases across CadQuery, OpenSCAD, and Three.js to compare visible-view fidelity and global-geometry errors.
Loading 2606.11152v2…