Source-linked AI summary
OmniCAD: A Large-Scale Benchmark for 3D Spatial Reasoning in Robotics Assemblies
Mingjia Wang, Taiting Lu, Ziwei Dong, Sisong Bei, Jingying Zeng, Runze Liu, Kaiyuan Lin, Hongxing Pan, Kai Zhang, Yizheng Hou, Yangshoudu Zheng, Chenchen Guo, Weiyuan Meng, Shubin Lyu, Zhijun Zheng, Dexu Wang, Xinyu Bai, Shurui Qian, Zhangzixin, Mengyu Pan, Guoliang Shi, Ling Ma, Yifan Yang, Qi He, Yi-Chao Chen, Yincheng Jin, Sung-Liang Chen, Mahanth Gowda
TL;DR
Complex industrial assembly reasoning remains underexplored despite VLM progress in perception and spatial reasoning. OmniCAD introduces a large-scale benchmark spanning component poses, mating relations, and tool-augmented refinement, and finds that current LMMs struggle with accurate, complete, and physically valid reconstruction.
Problem
Complex industrial assemblies require precise spatial arrangement, connectivity, and interactions, but existing evidence about VLM ability to reason over them remains limited.
Method
OmniCAD benchmarks general-purpose LMMs on 25K real-world assemblies using part libraries, ground-truth assemblies, multi-view renderings, 6-DoF poses, mate graphs, and one-shot or agentic evaluation.
Results
Current LMMs struggle with repeated-part reuse, precise poses, complete mate relations, collision-free placement, and scaling to large component libraries.
Takeaways & Limitations
OmniCAD provides a benchmark and analysis for studying accurate, physically valid, and scalable 3D mechanical assembly reasoning.
Abstract
from arXiv · showhide
Recent vision-language models (VLMs) show strong capabilities in robotic perception and spatial reasoning, yet their ability to reason about complex mechanical assemblies remains underexplored. We introduce OmniCAD, a large-scale benchmark for assembly-aware 3D spatial reasoning across diverse industrial systems, including robotic mechanisms, automotive components, aerospace structures, and agricultural machinery. OmniCAD contains 25k mechanical assemblies, with an average of 12 parts per assembly and 21 types of mate relationships. Each assembly includes a human-verified ground-truth 3D model and renderings from 20 viewpoints. The benchmark evaluates three capabilities: (1) component-level 3D spatial reasoning, requiring prediction of part positions and orientations; (2) part-to-part relational reasoning, requiring identification of mating relationships and assembly constraints; and (3) tool-augmented agentic reasoning, where models iteratively select viewpoints, inspect visual evidence, and refine predictions. Experiments show that current VLMs struggle with industrial assembly reasoning, often producing inaccurate poses, invalid mating relationships, part interpenetration, and degraded performance as assembly complexity increases. We will open-source the benchmark, evaluation code, and tool interfaces to support research on accurate, physically valid, and scalable 3D assembly reasoning.
Introduction
OmniCAD addresses the underexplored challenge of reasoning about complex industrial mechanical assemblies, where precise component arrangement, connectivity, and interactions determine functional artifacts. It benchmarks component, relational, and tool-augmented assembly reasoning and documents substantial VLM limitations.
- Industrial assemblies are difficult because they contain geometrically complex or visually similar components, dense mating constraints, tight tolerances, and nontrivial insertion and rotational motions.
- Existing datasets commonly lack explicit mate graphs, standardized multi-view observations, or sufficient B-Rep geometry and mating-constraint supervision.
- Preliminary experiments found that general-purpose VLMs often misidentify components, produce intersecting parts, and struggle with assemblies containing many components and extensive geometric information.
- OmniCAD contains 25K real-world mechanical assemblies averaging 12 parts, with native editable CAD files and coverage across robotics, automotive systems, aerospace equipment, and medical devices.
- The benchmark evaluates component identification and 6-DoF pose estimation, mating-relation reasoning, and tool-augmented reasoning with adaptive viewpoints and iterative self-correction.
Related Work
Prior work spans 3D part assembly, mechanical mating constraints, and structured CAD datasets, but OmniCAD targets general-purpose LMMs reconstructing complex assemblies from multi-view references and deduplicated part libraries.
- Learning to Assemble: Learning-based assembly methods progressed from component retrieval and placement to relational pose prediction, repeated-part handling, generative assembly, hierarchy, order, and visual instruction guidance.
- Learning to Assemble: Mechanical CAD assembly research explicitly models connectivity and mating constraints through systems such as AutoMate, JoinABLe, and Mates2Motion.
- Learning to Assemble: Existing approaches largely use task-specific models focused on predefined components, pairwise joints, or category-level objects.
- Learning to Assemble: Object-level and structured CAD datasets model isolated parts but do not capture component instances, assembly poses, or inter-part relations.
Benchmark Construction
OmniCAD formulates assembly understanding as recovering a physically plausible assembly from a component library and multimodal evidence, using structured annotations for geometry, relations, and validity.
- The task requires identifying component instances, estimating spatial configurations, inferring assembly relationships, and iteratively refining predictions using diagnostic feedback.
- 3D pose reasoning estimates each component’s position and orientation in the assembly coordinate frame from visual and geometric evidence.
- Mate-graph reasoning reconstructs component connectivity and mate types, including concentric, coincident, parallel, and perpendicular relations.
- The annotation pipeline collects editable SolidWorks assemblies from open-access repositories and supplies evidence and ground truth for identification, pose, mate-graph, and physical-validity evaluation.
- The benchmark contains 25K assemblies, 300K component instances, 196K unique library parts, and 289K mate relations, supporting one-shot and iterative agentic reasoning.
Evaluation Metrics
OmniCAD evaluates assembly predictions across identification, pose, shape, mate-graph, physical-validity, efficiency, and generation metrics, with separate measures for repeated parts and agentic execution.
- PartID measures multiset matching accuracy, while ID-NR and ID-R measure exact inventory reconstruction without and with repeated library reuse.
- PosErr is mean Euclidean translation error in millimeters, while Pos@10mm and Rot@10° measure thresholded translation and geodesic rotation accuracy.
- CD measures symmetric per-component Chamfer distance, PA reports the fraction below 10 mm, and SCD measures assembly-level discrepancy over unions of transformed point clouds.
- Mate-graph quality uses PairF1 for mate-edge multiset F1, TypeAcc for mate-type correctness, and GSim for directed graph similarity after node alignment.
- C-Free measures predictions without detected geometric conflicts, while RT, Avg. Iter., and PR measure runtime, agent rounds, and parseable schema-valid API outputs.
Experiment and Findings
OmniCAD evaluates LMMs on one-shot and tool-augmented mechanical assembly generation, testing component identification, pose estimation, mating relations, and physical validity. Results show strong performance on non-repeated identification and coarse orientation, but persistent difficulty with repeated parts, precise placement, complete mate graphs, collision-free assemblies, and efficient generation.
- Component Identification: 95.1% ID-NR versus 26.4% ID-R under ColorView shows that repeated-instance assignment remains substantially harder than non-repeated component identification.Performance degrades sharply when a library part is reused.
- Component Placement: 27.9% best one-shot Pos@10 and 34.6% best agentic Pos@10 indicate that precise translation remains harder than orientation estimation.With τCD = 10 mm, the best PA increases only from 55.0% to 57.1%.
- Impact of Visual and Geometric Inputs: Mate-graph conditioning improves Pos@10 and Rot@10 for seven of nine models, but does not provide the relative transformations needed for precise placement.The mate graph specifies connectivity and reduces relational uncertainty while leaving relative translations and rotations unresolved.
- Agentic Visual and Conflict-Guided Reasoning: Conflict feedback raises the best C-Free to 43.8%, while more than half of predictions still contain conflicts, indicating mainly coarse correction.The best agentic Pos@10 and PA reach 34.6% and 57.1%.
- Generation Reliability and Efficiency: 72.3% peak one-shot API SR and 71.5% peak agentic API SR coexist with settings below 10%, making geometric quality dependent on generation coverage.Gemini 3.1 Pro reaches 55.0% PA with only 0.3% API SR, while Conflict+2DVis increases runtime from 49.7 s to 283.4 s.
Conclusion
OmniCAD benchmarks LMM reasoning over large-scale mechanical assemblies using structured part libraries, engineer-authored assemblies, multi-view renderings, 6-DoF poses, and mate graphs. Its evaluation exposes recurring failures in repeated-part reuse, pose accuracy, mating completeness, physical validity, and scalability, providing a foundation for more reliable assembly reasoning.
- OmniCAD comprises 25K real-world assemblies with deduplicated part libraries, engineer-authored ground-truth assemblies, multi-view renderings, 6-DoF poses, and explicit mate graphs.
- The benchmark evaluates one-shot and tool-augmented reasoning across component identification, pose estimation, mating inference, physical validity, and iterative refinement.
- Models show unreliable repeated-part reuse, inaccurate poses, incomplete mate graphs, severe interpenetration, and poor scalability with large component libraries.