PaperlayerView exact revision on arXiv

Source-linked AI summary

SIMART: Decomposing Monolithic Meshes into Sim-ready Articulated Assets via MLLM

Chuanrui Zhang, Minghan Qin, Yuang Wang, Baifeng Xie, Hang Li, Ziwei Wang

arXiv:2603.23386v1cs.CVcs.GRcs.RO

TL;DR

Static 3D generation largely produces unarticulated meshes, while existing articulated-object pipelines separate decomposition and kinematic inference and dense voxel tokenization is costly. SIMART unifies geometric understanding, part decomposition, and kinematic prediction with a Sparse 3D VQ-VAE, reducing token redundancy by 70%. It reports state-of-the-art articulation performance and downstream use in physics-based simulation, while open-world generalization remains limited by articulated-data scarcity and inconsistency.

  • Problem

    Existing 3D assets are mostly unarticulated, while prior articulated-object pipelines separate decomposition and joint inference and dense tokenization limits complex-object scalability.

  • Method

    SIMART is a unified MLLM framework that jointly performs part-level mesh decomposition and kinematic parameter prediction using Sparse 3D VQ-VAE geometric encoding.

  • Results

    SIMART reports state-of-the-art performance in part decomposition and joint parameter estimation across in-domain and AI-generated benchmarks.

  • Takeaways & Limitations

    The generated assets support physics-based simulation and VR/AR applications as functional, simulation-ready articulated assets.

  • Takeaways & Limitations

    Open-world generalization is limited by the scarcity and inconsistent quality of existing articulated datasets.

Abstract

from arXiv · show

High-quality articulated 3D assets are indispensable for embodied AI and physical simulation, yet 3D generation still focuses on static meshes, leaving a gap in "sim-ready" interactive objects. Most recent articulated object creation methods rely on multi-stage pipelines that accumulate errors across decoupled modules. Alternatively, unified MLLMs offer a single-stage path to joint static asset understanding and sim-ready asset generation. However dense voxel-based 3D tokenization yields long 3D token sequences and high memory overhead, limiting scalability to complex articulated objects. To address this, we propose SIMART, a unified MLLM framework that jointly performs part-level decomposition and kinematic prediction. By introducing a Sparse 3D VQ-VAE, SIMART reduces token counts by 70% vs. dense voxel tokens, enabling high-fidelity multi-part assemblies. SIMART achieves state-of-the-art performance on PartNet-Mobility and in-the-wild AIGC datasets, and enables physics-based robotic simulation.

1 Introduction

SIMART addresses the gap between static 3D mesh generation and simulation-ready articulated assets by unifying part decomposition, semantic grounding, and kinematic prediction in an MLLM framework. Its Sparse 3D VQ-VAE reduces token redundancy while supporting high-fidelity articulation modeling.

  • Motivation: Existing 3D assets are mostly unarticulated, while manual creation of simulation-ready assets is prohibitively labor-intensive.Physical properties and kinematic information are needed for physics-based animation and robotic interactive simulation.
  • Limitations of prior methods: Multi-stage articulated-object pipelines decouple part decomposition, joint inference, and assembly, allowing geometric or kinematic errors to accumulate.Existing decomposition methods may miss mechanically meaningful link boundaries and violate kinematic affordances.
  • Limitations of prior methods: Static 3D generators typically produce monolithic meshes without decomposition, kinematic metadata, or physical metadata.SIMART instead targets per-part geometry together with structured URDF specifications.
  • SIMART: SIMART jointly performs part-level mesh decomposition and precise kinematic parameter prediction within a unified multimodal architecture.The framework is designed to directly perceive and generate kinematically aware meshes with their underlying kinematic logic.
  • SIMART: 70% token redundancy reduction from the Sparse 3D VQ-VAE mitigates memory exhaustion and enables detailed articulation modeling of intricate assemblies.The representation selectively encodes occupied surface voxels rather than dense volumetric space.
  • Evaluation: SIMART-Bench supports evaluation of articulation accuracy, and experiments report state-of-the-art performance in part decomposition and joint parameter estimation.The benchmark consolidates assets from PartNet-Mobility and diverse generative sources with expert manual annotations.

2 Related Work

Prior articulated-asset methods use reconstruction, 2D-to-3D lifting, Gaussian representations, or MLLM-based image reasoning, but each has important supervision or geometric limitations. These approaches often lack integrated 3D understanding and generation or require observation-heavy inputs.

  • Reconstruction-based methods: Reconstruction-based methods can recover high-fidelity geometry but typically require multi-view supervision across multiple articulation stages.Such high-quality, multi-state visual inputs are difficult to obtain in the wild.
  • MLLM-based methods: MLLM-based image methods infer motion structures from rendered images but lack an integrated 3D geometric understanding and generation pathway.Their reliance on exclusively 2D visual inputs limits direct geometric processing.
  • Part understanding: 2D-to-3D lifting methods offer broad semantic coverage but often produce cross-view inconsistencies and blurry boundaries.These weaknesses hinder the structural rigor needed for precise kinematic joint estimation.
  • Gaussian-based methods: Gaussian-based architectures incorporate semantic descriptors and physical reasoning, yet remain observation-dependent.They require dense temporal sequences or predefined kinematic templates to anchor dynamics.

3 Approach

SIMART maps visual, geometric, and textual inputs into a unified MLLM process that produces decomposed meshes and simulation metadata. Sparse voxel encoding reduces empty-space tokens, after which generated part seeds are reconstructed, segmented, and paired with structured kinematic information.

  • Unified MLLM pipeline: The pipeline fuses sparse geometric tokens with visual and textual inputs in a unified MLLM to perform part grounding and joint parameter estimation.Its final output contains structured URDF metadata and decomposed segments.
  • Problem formulation: SIMART takes visual observations, raw geometry, and language instructions as multimodal inputs and generates a simulation-ready asset.The output combines part-segmented meshes with simulation metadata covering kinematic and dynamic properties.
  • Unified MLLM pipeline: Qwen3-VL serves as the MLLM backbone, supporting reasoning about physical attributes and potential kinematic structures from multimodal knowledge.The architecture is used for joint reasoning over vision, geometry, and text tokens.
  • Sparse 3D VQ-VAE: The Sparse 3D VQ-VAE maps a 64^3 voxel grid into a compact latent grid and assigns a specialized zero token to unoccupied voxels.Only occupied geometric regions proceed through vector quantization using the learned codebook.
  • Sparse 3D VQ-VAE: Approximately 70% fewer informative tokens result from bypassing empty space during sparse encoding.The reduction addresses sequence-length and memory bottlenecks in subsequent MLLM processing.
  • Sparse 3D VQ-VAE: Each occupied voxel is serialized with explicit coordinates and a discrete geometry index before symmetric 3D-UNet decoding reconstructs the geometry.This preserves structural topology while providing a compressed geometric representation for articulation reasoning.
  • Simulator-ready asset synthesis: Generated part-specific voxel tokens are decoded into sparse point clouds, mapped onto the input mesh through graph-based segmentation, and assigned coherent face labels.The original texture is preserved in the final decomposed mesh.
  • Simulator-ready asset synthesis: Structured URDF metadata specifies parent-child hierarchies, joint configurations, limits, density, and friction alongside the segmented sub-meshes.Together they form an asset capable of inertial modeling and physical interaction.

4 Experiments

SIMART is evaluated on articulated-object generation and part grounding using diverse training data, quantitative metrics, qualitative comparisons, and ablations. The experiments show strong structural fidelity, functional-part localization, and efficiency from sparse tokenization.

  • Experimental Setup: The training collection combines 39,600 objects with 20 rendered kinematic states per articulated model and two 960k-pair instruction datasets.SIMART-Bench combines in-domain PartNet-Mobility assets with out-of-distribution AIGC objects to test robustness beyond homogeneous categories.
  • Qualitative Comparison: Qualitative comparisons visualize two motion states per object to assess kinematic accuracy and geometric fidelity against generative baselines.SIMART produces more precise segmentation and structurally consistent meshes than simplified or misaligned baseline outputs.
  • Metrics: The evaluation measures joint classification, axis and origin errors, and geometric decomposition using Type Accuracy, Axis Error, Origin Error, IoU, and Chamfer Distance.The metrics cover both kinematic prediction and part-mesh reconstruction quality.
  • Results: SIMART achieves state-of-the-art performance across all reported metrics on in-domain and AI-generated benchmarks, outperforming the listed articulated-asset baselines.Existing methods often lack raw-mesh processing or show poor geometric alignment, whereas SIMART maintains stronger structural fidelity.
  • 3D Part Understanding: SIMART precisely identifies functional components such as lids and doors while preserving geometric consistency with ground-truth parts.On AI-generated items, it significantly outperforms Physx-Anything across both IoU and CD.
  • Ablation Studies: 70% token reduction from sparse representations mitigates dense-grid memory bottlenecks, while zero-token handling and visual features improve performance across evaluation metrics.Dense decoding scales with part count and can cause out-of-memory errors; the full model achieves the highest ablation performance.

5 APPLICATIONS

SIMART converts raw meshes into high-fidelity articulated assets for robotic simulation and supports interactive asset creation for VR/AR environments. Its outputs can be imported into simulators and functionalized through click-based workflows.

  • Robotic Simulation: SIMART integrates reconstructed geometry with predicted kinematic structures to produce assets directly importable into NVIDIA Isaac Sim for robotic manipulation testing.The framework also estimates real-world scales to support physical consistency.
  • Robotic Simulation: The simulation pipeline supports diverse training scenarios, interactive learning through dynamic feedback, and multimodal observation data for VLA-model benchmarking.These are presented as three advantages for embodied AI.
  • VR/AR Asset Creation: In VR/AR, users can click-select components while SAM3D supplies geometry and SIMART generates articulated digital twins with realistic kinematic constraints.The workflow transforms static virtual surroundings into interactive components.

6 Conclusion

The paper presents SIMART as a multimodal system for transforming static meshes into simulation-ready articulated assets while reducing sparse-token redundancy. It also introduces a benchmark for evaluating articulation across in-domain and out-of-distribution assets, but identifies dataset quality as a generalization boundary.

  • Conclusion: SIMART transforms static 3D meshes into functional simulation-ready assets through part decomposition and joint-parameter estimation.The framework decouples kinematic reasoning from geometric generation.
  • Conclusion: 70% token-redundancy reduction addresses memory exhaustion associated with dense volumetric representations.This is achieved with a Sparse 3D VQ-VAE.
  • Conclusion: SIMART-Bench evaluates articulation accuracy on both in-domain and out-of-distribution assets.The benchmark is intended to standardize evaluation across diverse object distributions.
  • Limitations: Scarce and inconsistently annotated articulated datasets remain a primary limitation for open-world generalization.Future work targets larger and more diverse datasets through automated articulation prediction and annotation support.

A Demo Video Details

The supplementary demo shows SIMART assets used in interactive robotic manipulation and illustrates diverse articulated behaviors across object categories. It combines a simulation environment with synthesized URDF assets and varied motion sequences.

  • Demo Video: The demo constructs a simulation environment and integrates synthesized URDF assets for interactive manipulation.The environment uses background from [15].
  • Demo Video: The video presents diverse kinematic motion sequences and articulated behaviors across multiple object categories.

B Sparse 3D VQ-VAE Implementation Details

SIMART’s Sparse 3D VQ-VAE compresses voxel representations while balancing reconstruction fidelity against MLLM sequence-length and memory constraints. Its sparse and zero-token design preserves structural detail for simulation-ready asset generation.

  • Sparse 3D VQ-VAE architecture: The model encodes a 64 × 64 × 64 voxel grid into a 16 × 16 × 16 latent grid, then aggregates neighboring tokens into an 8 × 8 × 8 grid with 64-dimensional features.This design reduces computational overhead while targeting high-fidelity reconstruction of manifold surfaces.
  • Training objective: Training combines binary cross-entropy reconstruction loss with a commitment loss to stabilize codebook learning.The VQ-VAE is additionally pre-trained on a 500k-object subset following the TRELLIS data distribution.
  • Reconstruction evaluation: Reconstruction quality is measured by comparing decoded occupancy grids against voxelized 64 × 64 × 64 input meshes using mean squared error and Chamfer Distance.Table 4 reports the ablation results with MSE and CD values scaled by 10^5.
  • Ablation findings: Zero-token integration improves sparse reconstruction, while a 16 × 8 × 8 latent resolution increases fidelity but doubles token sequence length and memory overhead.The comparison shows a direct fidelity–sequence-length trade-off.
  • Ablation findings: SIMART adopts the 8 × 8 × 8 latent grid and a 4,096-entry codebook because larger resolution or an 8,192-entry codebook provides insufficient benefit relative to resource costs.The selected representation preserves structural details needed for robot operation learning within MLLM sequence constraints.
  • Sparse representation mechanism: Force Sparse exploits naturally emerging codebook entries that represent unoccupied voxels, enabling a specialized zero-token mechanism for sparse encoding.Dense VQ-VAE training produced approximately two to four entries associated with empty-voxel distributions.

C MLLM Implementation Details

SIMART combines visual, sparse geometric, and textual inputs in a Qwen3-VL-based inference pipeline. It generates both part-specific geometry and structured simulation metadata for articulated asset construction.

  • Multimodal backbone and inputs: SIMART uses Qwen3-VL-8B as its multimodal backbone and processes system prompts, sparse voxel tokens, task questions, and visual inputs together.The mesh format is coordinate-aware, pairing quantized voxel coordinates with discrete Sparse 3D VQ-VAE features.
  • Task formulation: The kinematic reasoning task asks the model to describe real scale, separate functional parts, and specify their physical properties.A separate part-grounding template requests generation of a component from a semantic description.
  • Output constraints: A high-level system prompt constrains output formatting and physical logic for consistent real-to-sim transfer.The complete prompt is documented in Table 5.
  • Generated outputs: The framework decodes discrete voxel tokens into part-segmented meshes while concurrently producing a structured description of simulation metadata.The metadata representation specifies physical attributes for downstream robotics integration.

D Generation Asset Benchmark Build Pipeline

SIMART-Bench evaluates articulated asset generation on diverse AI-generated objects that lack functional articulation. Its benchmark construction combines automated segmentation with expert-guided kinematic annotation and presents generated assets for comparison.

  • Benchmark motivation: SIMART-Bench was constructed to address evaluation protocols dominated by PartNet-Mobility models and to test generalization on AI-generated raw meshes.The benchmark targets unstructured generated objects that lack functional articulation.
  • Evaluation materials: The benchmark materials include representative URDF-generation outputs and qualitative galleries comparing SIMART with the Particulate baseline.The figures display AI-generated objects and simulation-ready assets.
  • Automated segmentation: Ground-truth construction begins with automated P3SAM segmentation, which commonly over-segments a single object into 6 to 10 fragments.The resulting segments are subsequently refined before kinematic annotation.
  • Expert annotation: A specialized Web UI enables precise annotation of motion axes and joint positions, defining kinematic hierarchies and physical constraints for URDF generation.Expert-guided merging and annotation produce simulation-ready ground-truth metadata.
Loading 2603.23386v1…