Source-linked AI summary
AutoPartGen: Autogressive 3D Part Generation and Discovery
Minghao Chen, Jianyuan Wang, Roman Shapovalov, Tom Monnier, Hyunyoung Jung, Dilin Wang, Rakesh Ranjan, Iro Laina, Andrea Vedaldi
TL;DR
Generating useful 3D assets requires parts rather than monolithic shells. AutoPartGen autoregressively generates parts from objects, images, or masks, and reports stronger part-generation performance than existing methods.
Problem
Many 3D generators treat objects as monolithic shells, while applications need compositional parts for finer-grained reasoning and manipulation.
Source: §1 · p. 2Method
AutoPartGen uses a compositional 3DShape2VecSet latent space and a stochastic autoregressive model that generates each part conditioned on prior parts and object, image, or mask evidence.
Source: §1 · p. 2Results
Takeaways & Limitations
Decomposable scene objects support flexible manipulation and editing of individual components.
Source: §4.5 · p. 9The method assumes objects are finite disjoint unions of closed regular surfaces smoothly embedded in 3D without self-intersections.
Source: §3 · p. 3, §3.3 · p. 5
Abstract
from arXiv · showhide
We introduce AutoPartGen, a model that generates objects composed of 3D parts in an autoregressive manner. This model can take as input an image of an object, 2D masks of the object's parts, or an existing 3D object, and generate a corresponding compositional 3D reconstruction. Our approach builds upon 3DShape2VecSet, a recent latent 3D representation with powerful geometric expressiveness. We observe that this latent space exhibits strong compositional properties, making it particularly well-suited for part-based generation tasks. Specifically, AutoPartGen generates object parts autoregressively, predicting one part at a time while conditioning on previously generated parts and additional inputs, such as 2D images, masks, or 3D objects. This process continues until the model decides that all parts have been generated, thus determining automatically the type and number of parts. The resulting parts can be seamlessly assembled into coherent objects or scenes without requiring additional optimization. We evaluate both the overall 3D generation capabilities and the part-level generation quality of AutoPartGen, demonstrating that it achieves state-of-the-art performance in 3D part generation.
1 Introduction
Applications need 3D parts for fine-grained reasoning and manipulation, but many generators treat objects as monolithic shells. AutoPartGen generates compositional objects part by part from 3D objects, images, or masks.
1 Introduction
- It handles object-to-parts, image-to-parts, and masks-to-parts, predicting semantic parts without annotations in the first two and allowing mask-guided partitioning in the third.Source: §1 · p. 2
- Autoregression jointly models parts and permits a variable count, addressing objects whose number of parts is unknown beforehand.Source: §1 · p. 2
- The 3DShape2VecSet latent representation is compositional: concatenated codes decode to the union of their corresponding surfaces [62] [30].Source: §1 · p. 2
- Each next part is generated from object-level evidence and previously generated parts, improving consistency over independently extracted parts [5] [58].A single stochastic denoising-diffusion model handles the 3D-object, image, and 2D-mask cases.Source: §1 · p. 2
- AutoPartGen is more accurate than PartGen and HoloPart, and more capable than HoloPart in automatic part discovery and reconstruction from images or shell 3D objects.It can optionally use 2D masks and does not require 3D annotation.Source: §1 · p. 2
2 Related Work
Related work frames image- or text-to-3D as challenging because the input contains less information than the 3D output and very large 3D datasets are unavailable. It surveys multi-view, latent, compositional-generation, and segmentation approaches.
2 Related Work
- Image- or text-to-3D is underdetermined because the input contains less information than the output, while 3D data at the scale used for image or video priors is unavailable.Source: §2 · pp. 2–3
Multi-view 3D Generation
Single-view Latent 3D Generation
Composable 3D Generation
3 Method
AutoPartGen learns to produce compositional 3D objects by generating parts sequentially in a latent shape space, supporting object-, image-, and mask-conditioned reconstruction.
3 Method
- The task is to infer a 3D part decomposition from an object, a single image, or an image plus masks, with mask-guided parts matching prescribed regions.Source: §3 · p. 4
3.1 Latent 3D shape space
- VecSet codes are compositional: concatenating two part codes decodes into an approximation of their surface union without retraining.Source: §3.1 · p. 4, §3.2 · p. 5, Figure 3 · p. 5
3.2 Latent 3D diffusion
- Latent diffusion learns a conditional flow velocity from noisy shape codes and evidence, enabling sampling of a shape conditioned on that evidence.Source: §3.2 · p. 5
3.3 Autoregressive 3D part generation
- Parts are sampled autoregressively from evidence and previous part codes, optionally anchored by an overall-object code; generation ends when masks are processed or an empty-shape [EoT] token appears.Source: §3.3 · pp. 5–6
4 Experiments
Experiments evaluate AutoPartGen across multiple input types, compare completion quality, ablate its autoregressive design and guidance, and demonstrate scene and city generation.
4 Experiments
- The experiments section introduces the implementation, evaluations, comparisons, ablations, and application demonstrations.It presents comparisons with state-of-the-art 3D completion methods and applications of AutoPartGen.Source: §4 · p. 7
4.1 Implementation Details
- The architecture builds on 3DShape2VecSet, uses a 24-layer, 2048-width DiT, and is trained on approximately 300K assets and 2M parts.Image features are encoded with DINOv2; training uses AdamW for 500K iterations on 256 NVIDIA H100 GPUs.Source: §4.1 · p. 7
4.2 Object, Image and Masks to 3D Parts Generation
- Tests cover image-to-parts, object-to-parts, and masks-to-parts generation, with accurate and consistent parts reported across these input types.The object-to-parts and masks-to-parts examples show regeneration from a 3D object and reconstructions guided by 2D masks; varying masks can produce decompositions at different granularities.Source: §4.2 · p. 8, Figure 5 · p. 8, Figure 6 · p. 8
4.3 Comparison to the State-of-the-Art
- AutoPartGen surpasses PartGen and HoloPart across IoU, F-score, and Chamfer Distance for both part completion and overall object reconstruction.The paper reports that HoloPart outperforms PartGen, likely because it receives more comprehensive input information.Source: §4.3 · p. 8
4.4 Ablation Study
- Removing autoregressive conditioning in the masks-to-parts setting causes generated parts to overlap and intersect.This is the only scenario where the condition can be removed, because external part guidance is needed to specify the next part and when generation stops.Source: §4.4 · p. 9
- Increasing image guidance aligns parts more closely with input masks, while stronger geometry guidance favors the learned prior for plausible structures and arrangements.The model learns a data-driven prior for part partitioning and ordering when image conditions are absent.Source: §4.4 · p. 9
4.5 Applications
- Given an isometric scene view, AutoPartGen generates decomposable objects such as chairs, clocks, plants, and tables for flexible component-level editing.The paper also presents a city-generation pipeline that assembles text-prompted tile images into coherent cityscapes.Source: §4.5 · p. 9
5 Conclusion
The conclusion presents AutoPartGen as a sequential model for coherent 3D part generation across several input types.
5 Conclusion
- AutoPartGen generates coherent 3D object parts sequentially from images, 2D masks, or 3D meshes, and the paper reports outperforming existing methods in part completion and coherence.The authors describe experiments across various tasks and applications as demonstrating potential for scalable and controllable 3D content creation.Source: §5 · p. 9
Supplementary Material
The supplementary material provides additional details and results, including implementation, comparisons, ablations, applications, failure cases, and limitations.
Supplementary Material
- Supplementary sections provide further implementation details on architectures, training and inference procedures, and evaluation protocols.The material also includes extended comparison with PartGen.Source: Supplementary Material · p. 14
- The supplement includes quantitative ablations of autoregressive training and guidance scales, along with additional qualitative examples for scene and city generation.Source: Supplementary Material · p. 14
- It also includes visualized failure cases and discussion of limitations and broader societal implications.Source: Supplementary Material · p. 14
A.1 Training
Training uses a 3D VAE and diffusion model, followed by inference procedures for different input settings.
A.1 Training
- The 3DShape2VecSet VAE is trained on approximately 1.7M 3D assets with an 8-layer encoder and a 16-layer decoder.Its training uses surface-normal, Eikonal, and KL-divergence losses, plus L1 and MSE losses over 24,576 points per shape.Source: §A.1 · p. 14
- The DiT diffusion model is pretrained for image-to-3D generation, then fine-tuned autoregressively on part data with image and geometry conditioning.Fine-tuning proceeds for approximately 300K steps, then continues at a longer token length for another 100K steps.Source: §A.1 · p. 14
A.2 Inference
- Inference uses 50 denoising steps; object-to-parts uses geometry guidance, while image- and masks-to-parts first reconstruct the overall object shape.With user-provided masks, the default guidance weights are image 7 and geometry 4; a visual comparison reports more accurate geometry and no overlapping parts relative to PartGen.Source: §A.2 · p. 14, Figure 9 · p. 15
A.3 Evaluation
Evaluation compares AutoPartGen with PartGen on GSO without user inputs and compares methods on PartObjaverse-Tiny when masks guide partitioning. A qualitative comparison reports sharper, more detailed meshes and less over-generation than PartGen.
A.3 Evaluation
- On GSO, AutoPartGen is compared with PartGen without user inputs; the mask-guided comparison uses PartObjaverse-Tiny and filters out objects with negligible-volume segments.The latter filtering follows [58].Source: §A.3 · p. 15
- With mask-guided partitioning, evaluation compares AutoPartGen with HoloPart and PartGen on PartObjaverse-Tiny.Objects containing segments that occupy only a small fraction of total volume are filtered out, as in [58].Source: §A.3 · p. 15
B Additional Comparison with PartGen
- AutoPartGen produces sharper, more detailed meshes than PartGen and avoids the over-generation issues seen in PartGen.The passage attributes PartGen’s over-generation to its lack of explicit modeling of the joint distribution of parts.Source: B Additional Comparison with PartGen · p. 15
C Ablation Study
The ablations report improvements from autoregressive generation and best performance at moderate guidance scales. The paper also demonstrates small-scene and city generation, and documents a failure involving identical parts.
C Ablation Study
- Autoregressive generation improves part-level completion and overall shape quality across all reported metrics.The autoregressive model achieves higher IoU and F-score and lower Chamfer Distance (CD); Table 2 compares the autoregressive ablation, with models trained for 200 epochs.Source: C Ablation Study · pp. 15–16, Table 2 · p. 16
- Moderate geometry/image guidance scales, such as 4/7, achieve peak IoU, F-score, and CD; excessively low or high scales degrade performance.Table 3 reports the three metrics for the part-completion task.Source: Table 3 · p. 16, C Ablation Study · p. 16
D 3D Scene Generation
- For small scenes, AutoPartGen decomposes generated scene shapes into components such as chairs and tables, enabling individual scene components to be edited.The input images come from a 2D text-to-image generator, and the figure describes scene generation with constituent elements.Source: D 3D Scene Generation · p. 16, Figure 10 · p. 17
E City Generation
- AutoPartGen integrates into a city-generation pipeline and generates cities including a medieval town and a solarpunk city.Further examples are presented in Figure 11.Source: E City Generation · p. 16, Figure 11 · p. 18
F Failure Cases
- When identical parts are present, the model may predict them together even when only one is masked.The authors conjecture that training-data grouping of identical parts may contribute; stronger image guidance can amplify the effect, and a later mask may then yield a nearly empty prediction.Source: F Failure Cases · p. 18, Figure 12 · p. 18
G Limitations and Boarder Impact
The model is limited to bounded scenes by its underlying VAE latent space, lacks explicit part-granularity control outside mask-guided generation, and may inherit training-data biases. The authors also note GPU demands and potential environmental impact when scaling or deploying the model.
Limitations
- The current model generates only bounded scenes because it inherits spatial constraints from the underlying VAE latent space.Unbounded world generation is identified as a future research direction.Source: G Limitations and Boarder Impact · p. 18
- Explicit control over part granularity is available only in masks-to-parts; image-to-parts and object-to-parts decompositions can vary.The authors suggest high-level granularity controls as a possible future improvement.Source: G Limitations and Boarder Impact · p. 19
Broader Impact
- Training-data biases may affect downstream tasks, while substantial GPU demands raise energy-consumption concerns when scaling or deploying the model.The authors recommend evaluating biases and auditing safeguards before practical deployment.Source: G Limitations and Boarder Impact · p. 19