Source-linked AI summary
MeshCoder: LLM-Powered Structured Mesh Code Generation from Point Clouds
Bingquan Dai, Li Ray Luo, Qihong Tang, Jie Wang, Xinyu Lian, Hao Xu, Minghan Qin, Xudong Xu, Bo Dai, Haoqian Wang, Zhaoyang Lyu, Jiangmiao Pang
TL;DR
Prior shape-to-code methods use limited DSLs and scarce paired datasets, making complex geometry and structure difficult to represent. MeshCoder uses expressive Blender Python APIs, paired object-code data, and a multimodal LLM to generate part-based scripts from point clouds. The paper reports stronger shape-to-code reconstruction, code-based editing, and improved LLM reasoning about 3D shapes.
Problem
Prior shape-to-code methods use DSLs limited to simple primitives, and large paired object-code datasets are scarce, while complex geometry and structure remain difficult to model.
Source: §1 · p. 2Method
MeshCoder develops expressive Blender Python APIs, constructs a large paired object-code dataset decomposed into semantic parts, and trains a multimodal LLM to generate Blender scripts from point clouds.
Source: §1 · p. 2Results
MeshCoder significantly outperforms prior work on shape-to-code reconstruction and improves LLM reasoning on 3D shape understanding tasks.
Source: §1 · p. 2Takeaways & Limitations
Structured executable code supports geometric and topological editing through code modifications and offers a flexible representation for 3D shape reconstruction and editing.
Source: §4.4 · p. 9, §6 · p. 10The method primarily targets human-made objects; its applicability to organic forms such as animals and humans remains underdeveloped.
Source: §5 · p. 9
Abstract
from arXiv · showhide
Reconstructing 3D objects into editable programs is pivotal for applications like reverse engineering and shape editing. However, existing methods often rely on limited domain-specific languages (DSLs) and small-scale datasets, restricting their ability to model complex geometries and structures. To address these challenges, we introduce MeshCoder, a novel framework that reconstructs complex 3D objects from point clouds into editable Blender Python scripts. We develop a comprehensive set of expressive Blender Python APIs capable of synthesizing intricate geometries. Leveraging these APIs, we construct a large-scale paired object-code dataset, where the code for each object is decomposed into distinct semantic parts. Subsequently, we train a multimodal large language model (LLM) that translates 3D point cloud into executable Blender Python scripts. Our approach not only achieves superior performance in shape-to-code reconstruction tasks but also facilitates intuitive geometric and topological editing through convenient code modifications. Furthermore, our code-based representation enhances the reasoning capabilities of LLMs in 3D shape understanding tasks. Together, these contributions establish MeshCoder as a powerful and flexible solution for programmatic 3D shape reconstruction and understanding. The project homepage is available at \href{https://daibingquan.github.io/MeshCoder}{this link}.
1 Introduction
MeshCoder targets limitations in prior shape-to-code work: simple DSLs struggle with intricate geometry, and paired object-code datasets are scarce. It combines expressive Blender Python APIs, a large paired dataset, and a multimodal LLM to generate part-based scripts from point clouds.
1 Introduction
- Its Blender Python APIs model intricate geometries using operations such as sweeping section curves, bridging curves, beveling, Boolean operations, and repetition.These capabilities are designed to address prior DSLs’ limits to simple primitives.Source: §1 · p. 2
- The dataset pipeline yields approximately 1 million objects across 41 categories, including objects with more than 100 parts, for training object-to-code inference.It synthesizes paired parts, predicts code for each part, and concatenates the parts’ code into object code.Source: §1 · p. 2
- A triplane-based tokenizer converts point clouds into fixed-length tokens that an LLM uses to generate executable Blender scripts for distinct semantic parts.The decoded code specifies the object’s name and part identities and names.Source: §1 · p. 2, Figure 2 · p. 3
- MeshCoder significantly outperforms prior work on shape-to-code reconstruction and also improves LLM reasoning on 3D shape understanding tasks.The paper also reports that code modifications support geometric and topological editing.Source: §1 · p. 2
- MeshCoder’s part-level code is presented as simplifying mesh editing and enhancing LLM understanding of 3D objects.The paper describes the scripts as structured and editable.Source: §1 · p. 3
2 Related Work
Prior shape-program methods use DSLs to represent 3D geometry, while part-based methods either assemble objects from parts or segment objects into parts. The related work identifies limitations in representing complex complete objects and connecting segmentation to executable code.
Shape programs
- Shape-program methods improved abstraction, learning efficiency, and scalability, but often struggle with complex part geometries, artist-grade quad meshes, and coherent complete multi-part objects.The passage also notes that CAD code-generation methods are limited to isolated components.Source: §2 · p. 3
Part-based Representation
- Part-based approaches include methods that assemble complete structures from predefined or learned parts and methods that segment objects into individual parts.The segmentation discussion includes zero-shot and low-shot approaches.Source: §2 · pp. 3–4
- Part-segmentation methods do not translate segmented parts into executable code representations, limiting their integration into code-driven design workflows.Source: §2 · p. 4
3 Methodology
MeshCoder builds object-to-code training data by coding parts individually, then trains a model to predict Blender scripts from point clouds. Its pipeline combines procedural part geometry, point-cloud tokenization, and semantic part-code assembly.
3 Methodology
- The methodology trains a part-to-code model on paired synthetic parts, applies it to object parts, concatenates their codes, and uses the resulting pairs to train an object-to-code model.The object’s parts are assigned semantic labels in the resulting code.Source: §3 · p. 4
3.1 Part Dataset
- The part dataset combines parameterized primitives and swept cross-sections with Bridge Loop, Boolean, and Array operations for constructing more varied geometries.Translation sweeps a 2D section along a 3D trajectory; Bridge Loop connects vertices of successive 2D shapes, while Boolean operations combine shapes and Arrays repeat them.Source: §3.1 · pp. 4–5
3.2 Part-to-code Inference Model
- The part-to-code model maps sampled mesh point clouds to code using a triplane shape tokenizer and a fine-tuned LLM.The tokenizer projects point features onto three planes, patchifies them, processes them with transformer blocks, and uses learnable tokens with cross-attention to produce fixed-length shape tokens for the LLM.Source: §3.2 · pp. 5–6, Figure 4 · p. 6
3.3 Assemble Parts to Objects
- To assemble object codes, each part is normalized to a cube, coded independently, transformed back to its original pose and scale, and concatenated with semantic information.Parts are ordered spatially from bottom to top, left to right, and front to back.Source: §3.3 · p. 6
3.4 Object-to-code Inference Model
- The object-to-code model uses the part-to-code model’s structure and initializes its weights from the trained part model; semantic information in target codes supports learning object structure.The object model is trained using the same method as the part-to-code model.Source: §3.4 · pp. 6–7
4 Experiment
MeshCoder consistently outperforms Shape2Prog and PLAD on reconstruction IoU and L2 CD. The experiments also describe the datasets and training setup, and show code-based geometric and topological editing and code-assisted shape understanding.
4.1.1 Synthetic Part Dataset
- The synthetic part dataset contains around 10 million point cloud–code pairs, generated from sampled API parameters and split 70%/15%/15% for training, validation, and testing.It includes primitive, Translation-based, Bridge Loop, Boolean-operation, and Array-based constructions.Source: §4.1.1 · p. 7
4.1.2 Object Dataset
- The object dataset contains 1 million object-code pairs across 41 common categories, generated with Infinigen Indoor and split into training, validation, and test sets.The split follows the Synthetic Part Dataset strategy.Source: §4.1.2 · p. 7
4.2 Implementation Details
- Training uses 20 epochs for part-to-code and 10 for object-to-code, initializing the latter from the former’s weights and selecting checkpoints by lowest CD loss.Both stages use a learning rate of 10^-4; batch sizes are 512 and 256, respectively.Source: §4.2 · pp. 7–8
4.3 Reconstruction Performance
- MeshCoder consistently outperforms Shape2Prog and PLAD in both IoU and L2 CD reconstruction metrics.The evaluation reports category-specific and overall results; the supplied table caption does not provide cell values.Source: §4.3 · p. 8, Table 1 · p. 8
4.4 Shape Editing
- Code edits support geometric changes such as reshaping a square tabletop into a circle and topological changes such as adjusting mesh resolution.Resolution parameters produce progressively higher resolution from left to right, allowing users to balance storage requirements and mesh quality.Source: §4.4 · pp. 8–9, Figure 7 · p. 9, Figure 8 · p. 9
4.5 Shape Undertanding
- Predicted object codes contain semantic information that can be input to GPT to assist its understanding of object structures.The paper reports experiments on shape understanding and illustrates an example pipeline in Figure 9.Source: §4.5 · p. 9
5 Limitations
MeshCoder primarily targets human-made objects; code-based representations for organic forms such as animals and humans remain underdeveloped.
5 Limitations
- MeshCoder primarily targets human-made objects, while applicability to organic forms such as animals and humans remains underdeveloped.The authors identify organic forms as a direction for future research.Source: §5 · p. 9
6 Conclusion
MeshCoder converts point clouds into editable Blender Python scripts for reconstructing complex 3D objects. Its structured code representation supports programmatic reconstruction and editing, while the approach also enhances LLM reasoning in 3D shape understanding.
6 Conclusion
- MeshCoder translates 3D point clouds into editable Blender Python scripts for detailed reconstruction of complex objects.The framework uses Blender Python APIs and a large-scale object-code dataset decomposed into semantic parts, with a multimodal LLM generating executable scripts.Source: §6 · p. 9
- MeshCoder also enhances LLM reasoning capabilities in 3D shape understanding.The conclusion reports this alongside superior performance in shape-to-code reconstruction tasks.Source: §6 · p. 9
- Representing shapes as structured code provides a flexible approach to programmatic 3D reconstruction and editing.The paper connects this representation to applications in reverse engineering, design, and analysis.Source: §6 · p. 10
A.1.1 The principles of Translation and Bridge Loop
Translation sweeps a 2D cross-section along a 3D trajectory, while Bridge Loop connects the vertices of two predefined 2D shapes. Bridge Loop permits differing shapes and orientations that Translation constrains.
A.1.1 The principles of Translation and Bridge Loop
- Translation sweeps a 2D cross-section along a 3D trajectory to generate a mesh, keeping the section perpendicular to the trajectory tangent and allowing only uniform scaling.The illustrated example uses a circle as the cross-section.Source: §A.1.1 · p. 13, Figure 10 · p. 13
- Bridge Loop constructs a mesh by connecting corresponding vertices of two predefined 2D shapes, which may differ in shape and relative orientation.The figure depicts the mesh wireframe formed by connecting the two shapes' vertices.Source: §A.1.1 · p. 13, Figure 10 · p. 13
- Unlike Translation, Bridge Loop does not require cross-sections to align with a trajectory tangent, enabling geometries Translation cannot produce.Source: §A.1.1 · p. 13
A.1.2 Part datasets
The part-dataset methods combine a basic shape code library with additional shape categories and parameterized synthesis. The paper illustrates code examples for several part-generation operations.
A.1.2 Part datasets
- The basic shape code library includes Fill Grid, Spoon, and Fork types, and the paper presents complete implementations for primitive and curve creation.The supplementary materials contain the full codebase.Source: Figure 11 · p. 13, §A.1.2 · p. 14
- For shapes difficult to represent with the Section 3.1 methods, the authors add Fill Grid, Spoon, and Fork categories.Fill Grid forms a surface from a closed 3D shape and extrudes it along its normal; Spoon and Fork use dedicated procedural functions inspired by Infinigen Indoor [4].Source: §A.1.2 · pp. 13–14, Figure 11 · p. 13
- Figures 12–16 show examples and complete code for Primitive, Translation, Boolean, Bridge Loop, and Array parts.The code shown corresponds to the first two objects in each figure.Source: §A.1.2 · p. 14, Figure 12 · p. 14, Figure 13 · p. 14
- Primitive parts are synthesized by sampling shape type, scale, orientation, and position, with the longest bounding-box edge constrained to [1, 2] and the part kept within [−1, 1] bounds.For each axis, the scale is set as 10^x after independently sampling x from [−2, 2]; other shape types use similar parameter sampling.Source: §A.1.2 · p. 14
A.1.3 Object datasets
Object-code assembly orders parts using a spatial heuristic, and dataset construction retains objects only when every part meets a reconstruction criterion. The resulting pairs are split by the original Infinigen dataset partitions.
A.1.3 Object datasets
- Parts are sorted bottom-to-top, left-to-right, and front-to-back using characteristic cells in a 32 × 32 × 32 grid.A part's characteristic cell is the occupied cell with smallest z, then smallest x, then smallest y; lexicographic sorting determines program order.Source: §A.1.3 · p. 15, Figure 17 · p. 16
- Only objects whose every part has predicted-point-cloud CD below 5 × 10^-3 are included, yielding 1 million pairs from 1.57 million Infinigen instances.Objects with any part failing the threshold are discarded.Source: §A.1.3 · p. 15
- The object dataset uses a 70%/15%/15% train, test, and validation split, and evaluation for MeshCoder and baselines uses the same complete-Infinigen test set.MeshCoder trains only on constructed pairs in the training portion, whereas baselines train on all objects in the original Infinigen training split.Source: §A.1.3 · p. 15
- The paper includes an example of converting part codes into a full object program and a figure detailing the shape tokenizer configuration.Source: Figure 17 · p. 16, Figure 18 · p. 16
A.2 Model architecture
MeshCoder encodes point-cloud shapes as a structured representation, then predicts Blender code that can be trained and evaluated for reconstruction, editing, and shape understanding.
A.2 Model architecture
- The tokenizer projects point clouds onto three planes, encodes patch tokens, and uses 128 learnable queries in 12 self- and cross-attention layers to produce 128 output tokens.An MLP maps the output tokens to a final representation of shape R128×2048.Source: §A.2 · p. 16
A.3 More training details
- The part-to-code model trains for 20 epochs, then initializes the object-to-code model, which trains on Infinigen Indoor for 10 epochs.Both use AdamW, a learning rate of 10^-4, and checkpoints selected by lowest CD loss; training uses 64 NVIDIA A100 GPUs.Source: §A.3 · p. 17
- Training augmentation applies random rotation and scaling, varies point-cloud size from 4096 to 16384, and adds Gaussian noise.MeshCoder is trained and evaluated on a unified dataset aggregating all object categories.Source: §A.3 · p. 17
A.4 Complete experiment result of Shape Reconstruction
- Reconstruction comparisons cover all object categories using CD and voxel IoU, with results summarized in Tables 2 and 3.Additional sofa, bathtub, and toilet examples show input point clouds, inferred code, and meshes generated by executing that code.Source: §A.4 · p. 17, Table 2 · p. 18, Figure 24 · p. 25
- The part-to-code model is evaluated on 10,000 test samples per category using CD and voxel IoU, with reported low CD and high IoU across categories.The quantitative results are reported in Table 4.Source: §A.4 · p. 17, Table 4 · p. 20
A.5 Complete experiment result of Shape Editing
- Editing examples change chair-leg and armrest thickness through scale parameters and alter a plate’s mesh resolution through its resolution parameter.The highlighted regions mark the changes.Source: §A.5 · p. 18, Figure 27 · p. 27, Figure 28 · p. 28
A.6 Complete experiment result of Shape Understanding
- GPT-4 answers geometry and structure questions using inferred code and its semantic comments, but cannot answer color questions because the code contains no color information.The examples use dishwasher, office-chair, and cell-shelf code.Source: §A.6 · p. 19, Figure 19 · p. 20, Figure 20 · p. 21