PaperlayerView exact revision on arXiv

Source-linked AI summary

MegaParts: Scaling Part-Aware 3D Object Generation to 300 Parts via Token-Efficient Autoregressive Modeling

Manwen Liao, Xinyu Lian, Jian Mao, Kaixu Chen, Li Luo, Jinghao Yan, Wanshui Gan, Qiao Yu, Weitian Zhang, Chunhua Shen, Guang Chen, Bo Dai, Xudong Xu, Zhaoyang Lyu

arXiv:2608.14783v1cs.CVcs.GR

TL;DR

Part-aware 3D generation struggles to scale to structurally complex objects with hundreds of parts while preserving controllable composition and detailed geometry. MegaParts combines adaptive-length vector-quantized part tokens with long-context autoregressive modeling, generating objects with up to 300 parts while preserving geometric fidelity and structural coherence and outperforming prior autoregressive baselines.

  • Problem

    Existing part-aware 3D generation methods struggle to scale to structurally complex objects with hundreds of parts, limiting compositional control and detailed geometry generation.

  • Method

    MegaParts combines an adaptive-length vector-quantized part tokenizer with a long-context autoregressive model generating object boxes, part boxes, and geometry tokens as one structured sequence.

  • Results

    MegaParts generates objects with up to 300 parts while preserving geometric fidelity and structural coherence, outperforming prior autoregressive baselines and remaining competitive with diffusion-based methods.

  • Takeaways & Limitations

    Token-efficient autoregressive modeling supports larger-scale part-aware 3D generation while maintaining geometry fidelity and structural coherence.

  • Takeaways & Limitations

    Long-context autoregressive generation still incurs substantial training and inference costs, and the framework models geometry without textures, materials, semantic annotations, or physical relationships.

Abstract

from arXiv · show

Part-aware 3D object generation is essential for graphics applications such as controllable modeling, editing, and articulation, where objects are represented as coherent assemblies of semantic parts. However, existing part-aware generation methods, do not scale well to highly complex objects. As the number of parts increases, generating detailed geometry becomes prohibitively expensive in token length and memory. We introduce MegaParts, a scalable autoregressive 3D generation framework to address this challenge by combining structured sequence modeling with a token-efficient vector-quantized shape tokenizer. Our tokenizer learns discrete latent representations for part-level geometry by minimizing token usage subject to high-fidelity reconstruction, enabling adaptive-length tokenization based on geometric complexity. On top of this compact representation, we train a large language model to generate object bounding boxes, part bounding boxes, and part shape tokens within a unified structured sequence. Combined with efficient long-context training strategy, our token-efficient formulation scales to objects with up to 300 parts and sequence lengths up to 256k tokens. This substantially extends the scale of part-aware 3D generation while preserving compositional structure and enabling fine-grained part-level control. Our method achieves higher mesh quality than baseline autoregressive and diffusion models, showing that compressed discrete part tokens improve not only scalability but also the achievable fidelity of generated geometry. These results suggest that LLM native token-efficient autoregressive modeling is a compelling alternative to diffusion for large-scale part-aware 3D generation. The project page is available at https://expmaster.github.io/megaparts_webpage.

1 Introduction

MegaParts addresses the scalability limits of diffusion-based part-aware 3D generation with a token-efficient adaptive-length part tokenizer and a unified long-context autoregressive sequence model. It scales to objects with up to 300 parts and 256k tokens while producing higher-quality meshes than autoregressive and diffusion baselines.

  • Motivation: Diffusion-based part-aware pipelines struggle to generate structurally complex objects with hundreds of parts as token counts increase.These methods inherit scalability limitations from diffusion models, whose naive variants often struggle with increasing token numbers.
  • Method: MegaParts introduces a token-efficient vector-quantized tokenizer that adaptively represents part geometry with discrete latent sequences.The tokenizer minimizes token usage while maintaining high-fidelity reconstruction and concentrates geometric information in prefix tokens.
  • Method: The framework generates object bounding boxes, part bounding boxes, and part geometry as one unified structured sequence.Its long-context autoregressive model represents the object as a coherent assembly of localized parts.
  • Scalability: Up to 300 parts and 256k tokens are supported, substantially expanding the practical scale of part-aware 3D generation.This scaling is enabled by the adaptive-length part representation and long-context autoregressive framework.
  • Results: MegaParts generates higher-quality meshes than conventional autoregressive baselines and diffusion-based methods.The reported results indicate that adaptive-length discrete part representations improve both scalability and geometry fidelity.

2 Related Work

Prior 3D shape representations face limitations in adaptive capacity allocation, while part-aware methods improve suitability for complex multi-part shapes but remain constrained by decomposition assumptions or diffusion-based modeling. Autoregressive 3D generation is emerging, yet long-context modeling for highly complex, part-aware assets remains unexplored.

  • 3D Shape Representations: VecSet representations use shared latent-vector pools, whereas voxel-based representations improve locality but typically allocate capacity through fixed grids, hierarchies, or manually designed spatial rules.These schemes may assign similar representational budgets to geometrically simple and complex regions and do not naturally align with semantic decomposition.
  • Part-Aware Representations: Part-aware representations vary capacity across parts according to geometric complexity and separate global structure modeling from part-level representation for shapes with hundreds of parts.This design is motivated by the limitations of existing representation families in high-fidelity generation of complex shapes.
  • Part-Aware 3D Generation: Part-aware generation methods either condition geometry on prescribed decompositions or jointly predict part layouts and geometries without fixed decomposition inputs.The first stream receives semantic guidance from segmentation masks or part IDs but may miss internal structural details; the second is more expressive because it does not require a fixed partition.
  • Part-Aware 3D Generation: Existing part-aware generation methods are predominantly diffusion-based and model inter-part and intra-part relationships through designed global and local attention mechanisms.This applies to both decomposition-conditioned and native 3D generation streams.
  • Autoregressive 3D Generation: Autoregressive 3D generation has recently gained attention, including two-stage pipelines that generate coarse structured geometry autoregressively before diffusion-based fine-geometry recovery.Existing work includes generating coarse voxels and using a subsequent mesh-reconstruction stage.
  • Autoregressive 3D Generation: Long-context autoregressive modeling remains unexplored for highly complex, part-aware 3D assets requiring global coherence and fine-grained geometry across many detailed components.This gap motivates methods designed for assets with many parts while preserving structural and geometric detail.

3 Methodology

MegaParts combines a causal vector-quantized shape tokenizer with adaptive rate–distortion token allocation and a structured autoregressive representation that separates component layout from geometry generation. This design preserves essential geometry in early tokens while making global structure explicit and controllable at the part level.

  • Token-efficient shape representation: The causal VQ-VAE maps geometry to discrete latent sequences whose earlier tokens encode more essential content, enabling robust variable-length truncation.The encoder uses cross-attention and causal self-attention, while the decoder applies causal self-attention over quantized latents.
  • Token-efficient shape representation: Random prefix reconstruction during training concentrates geometric information into early tokens and supports variable-length tokenization without significant reconstruction degradation.Prefix lengths are sampled as L = {2^n} for n = 4 to 12; alternating encoder-only and decoder-only optimization phases improve training stability.
  • Adaptive component tokenization: Each semantic component is encoded independently in a normalized local coordinate frame, while its bounding box preserves global position and scale.Candidate prefixes are selected using a rate–distortion objective that balances reconstruction fidelity, bounding-box IoU, and token usage.
  • Structured global representation: The global representation factorizes generation into layout generation and autoregressive geometry generation, yielding an explicit sequence of component bounding boxes followed by component shape tokens.The full representation is Y = [P, C], where P contains component descriptors and C contains their discrete mesh-code prefixes.
  • Structured global representation: Deterministic spatial ordering of components and prefix-based local sequences provide stable structural dependencies while progressing from coarse geometry to finer details.This ordering combines a stable global layout with token-efficient local shape representations and supports part-level control.

4 Experimental Results

MegaParts is evaluated for reconstruction and generation on PartObjaverse-Tiny, including text-conditioned and part-bounding-box-conditioned tasks. Results also analyze metric limitations, qualitative image-conditioned comparisons, and the effects of staged VQ-VAE optimization and causal attention.

  • Evaluation Setup: The evaluation uses PartObjaverse-Tiny to assess both reconstruction and generative capabilities, with additional high-part-count results reported in the supplement.The training pipeline uses curated public and private data, while the benchmark evaluation uses the publicly available PartObjaverse-Tiny dataset.
  • Reconstruction: Reconstruction is evaluated with Chamfer Distance and Normal Consistency, using meshes normalized to [−1, 1]3 and 100,000 sampled points.Cube is the primary baseline, and MegaParts is additionally evaluated across multiple token-count settings for adaptive reconstruction.
  • Generation: Generative evaluation covers text-conditioned generation against SAR3D, Cube, TRELLIS-text, and ShapeLLM-Omni, plus part-bounding-box-conditioned generation against FullPart and XPart.The second task conditions synthesis on part bounding boxes and auxiliary inputs to evaluate structure-aware generation.
  • Generation: Text-conditioned generation uses FID and CLIP score on rendered normal maps, while part-bounding-box-conditioned generation reports part-level IoU, part-level Chamfer Distance, and BBox IoU.Quantitative results are summarized in Tables 2 and 3, with qualitative text-conditioned comparisons in Figure 5.
  • Metric Discussion: Part-level metrics can disproportionately penalize fused meshes because they measure adherence to prescribed part decomposition rather than only overall shape quality.Baseline methods often merge multiple semantic or structural parts into a single mesh, which is heavily penalized during normalization and metric computation.
  • Ablation Studies: Removing phased VQ-VAE optimization or causal attention causes clear reconstruction degradation, supporting staged optimization and causal decoding under adaptive token budgets.The ablation is conducted on PartObjaverse-Tiny; generation-model ablations are reported in Supplementary Material B.

5 Applications

MegaParts’ detailed part decomposition supports downstream digital content creation, including localized editing, recombination, secondary creation, and articulation assets with dense interactive structures.

  • Applications: Detailed part decomposition enables fine-grained secondary creation and articulation asset creation.The generated results provide structured part-level assets for downstream applications.
  • Applications: Explicit part-level structure lets artists edit specific regions, recombine parts across assets, and refine components without rebuilding the entire shape.The decomposition provides a localized manipulation prior for flexible digital content creation.
  • Applications: Separating small functional components, such as piano keys or control-panel buttons and switches, facilitates articulated assets with component-specific motion, interaction, or physical constraints.Fine-grained segmentation supports dense interactive structures beyond monolithic representations.

6 Conclusions and Discussions · A Implementation Details

MegaParts combines adaptive-length vector-quantized part tokens with long-context autoregressive modeling to generate structurally coherent objects with up to 300 parts. The framework achieves strong reconstruction and generation performance, while remaining limited by computational cost and its geometry-only scope.

  • 6 Conclusions and Discussions: MegaParts presents a scalable autoregressive framework for fine-grained part-aware 3D generation.Its core components are a token-efficient vector-quantized tokenizer and adaptive-length token selection.
  • 6 Conclusions and Discussions: The full VQ-VAE performs best across all tested token budgets and reconstruction settings.The ablation compares phased training and causal decoder attention using part-level and object-level reconstruction metrics.
  • 6 Conclusions and Discussions: Removing phased training degrades reconstruction quality, while removing causal decoder attention weakens the benefit of larger token budgets.These ablations indicate that both design choices contribute to reconstruction performance under varying token budgets.
  • 6 Conclusions and Discussions: Objects with up to 300 parts are generated while preserving geometric fidelity and structural coherence.The framework uses a long-context autoregressive model alongside compact part representations.
  • 6 Conclusions and Discussions: MegaParts outperforms prior autoregressive baselines and remains competitive with diffusion-based methods.These comparisons are reported alongside the method’s ability to preserve geometric fidelity and structural coherence.
  • 6 Conclusions and Discussions: Long-context autoregressive generation still incurs substantial training and inference cost for highly complex objects.This remains a limitation despite the method’s experimental results.
  • 6 Conclusions and Discussions: The current framework models geometry only, excluding textures, materials, semantic annotations, and physical or functional part relationships.Future directions include more efficient sequence modeling, hierarchical representations, and unified geometry-appearance modeling.

A.1 Implementation Details of Models · A.2 Data Collection and Processing · A.3 Training Recipes

The supplementary sections detail MegaParts’ adaptive tokenizer and structured LLM, its large-scale data processing pipeline, and a three-stage training recipe with long-context and part-count balancing. Together, these choices support robust geometry representation and generation across highly complex part-aware assets.

  • A.1 Implementation Details of Models: The pipeline combines an adaptive 3D VQVAE for part-level shape representation with a fine-tuned large language model for conditional generation.These are the two key components of the system.
  • A.1 Implementation Details of Models: The Qwen3-8B generator unifies object layout, part structure, and geometry through structural tokens, 65,536 discrete shape tokens, and serialized bounding-box fields.Planning tokens delimit object and part bounding boxes, while box values use three effective digits.
  • A.2 Data Collection and Processing: Nearly 10M part meshes train the VQ-VAE, while approximately 440K annotated part-aware assets train the generation model across multiple 3D sources.The curated datasets include PartVerse-XL, PartNext, PartNet, and Infinigen.
  • A.2 Data Collection and Processing: Meshes with more than 300 connected components are merged using bounding-box sizes and relative spatial positions before geometry repair.The preprocessing pipeline addresses scarce human-annotated part-aware data by decomposing object meshes into connected components.
  • A.2 Data Collection and Processing: Three sequential repair strategies address non-manifold and non-watertight decomposed parts: Blender’s 3D Print Toolbox, VolumeMaker, and Blender’s solidify modifier.The repairs target difficulties in SDF construction and geometry reconstruction.
  • A.2 Data Collection and Processing: A 1:1 mixture of object-level and part-level meshes trains the VQ-VAE to preserve detail and avoid holes across arbitrary geometric complexity.The object-level component is increased because too small a proportion caused detail loss and holes in reconstructed shapes.
  • A.3 Training Recipes: The generation model uses token alignment, short-context pretraining, and long-context fine-tuning, with frozen VQ-VAE codebook embeddings projected into Qwen’s hidden space.The shape-token codebook is initialized from the trained VQ-VAE, preserving its learned geometry priors; part-count balancing is also applied.

B Additional Ablation Studies … C More Evaluation Results

The ablations show that object-level bounding-box supervision improves adherence to user-specified global constraints, while part-ordering sensitivity is evaluated by comparing spatial and connectivity-based sequences.

  • B.1 Object-Level Control: 0.93 BBox IoU is achieved with object-box prediction supervision, up from 0.89 without it.The metric is computed between the user-provided object box and the generated mesh’s actual bounding box.
  • B.1 Object-Level Control: Object-level supervision strengthens adherence to user-specified global constraints.The comparison evaluates explicit object-box prediction supervision against directly prompting the box for controlled generation.
  • B.2 Sensitivity to Part Ordering: Part-ordering sensitivity is tested by comparing z–x–y ordering with connectivity-graph BFS ordering.The z–x–y ordering is the ordering used in the main paper.
  • B.2 Sensitivity to Part Ordering: For z–x–y ordering, part-box centers are quantized onto a 323 voxel grid.The quantized centers are sorted lexicographically.
  • B.2 Sensitivity to Part Ordering: Ties in z–x–y ordering are broken by bounding-box volume from large to small.This specifies the secondary sorting rule after lexicographic ordering of quantized part-box centers.
  • B.2 Sensitivity to Part Ordering: BFS ordering constructs a part-connectivity graph by voxelizing each component.The supplied passage ends while describing the graph-construction procedure.

C.1 High-Part-Count Evaluation

MegaParts is evaluated on held-out objects spanning four part-count ranges up to 300 parts, using text-to-3D and matched-layout bounding-box-conditioned protocols. It maintains text-generation quality and strong high-part-count inference success, while remaining limited by long sequence lengths under fixed GPU memory.

  • Evaluation Setup: Four mutually exclusive held-out test sets cover part-count ranges [1, 50), [50, 100), [100, 200), and [200, 300] across 400 objects.No test object appears in training; evaluation uses CLIP/FID for text-to-3D and Part CD, part IoU, BBox IoU, and success rate for bounding-box conditioning.
  • Text-to-3D Generation: 0.26–0.27 CLIP score is maintained across all part-count ranges, while FID increases from 46.3 to 73.2 but remains lower than every compared baseline.These results are reported for text-conditioned generation across the four held-out complexity ranges.
  • Bounding-Box-Conditioned Generation: The bounding-box experiment is a matched-layout comparison: all methods receive identical prescribed part boxes, but use different auxiliary modalities.MegaParts uses text, FullPart uses an image, and XPart uses a point cloud, so the comparison assesses geometry following the given boxes rather than modality-identical unrestricted generation.
  • Evaluation Limitations: A single NVIDIA A800 80 GB GPU limits inference primarily through sequence length rather than part count.Some objects with fewer than 300 parts require more tokens than some 300-part objects; additional GPU memory or distributed inference can extend feasible context length.

C.2 Inference Efficiency and Memory · C.3 Rate–Distortion Selection Cost

Inference is primarily bottlenecked by autoregressive decoding, while token length governs memory use for large objects. Rate–distortion selection is an offline, parallelized preprocessing step that adds no generation-time overhead.

  • C.2 Inference Efficiency and Memory: 42.5 tokens/s is achieved with vLLM, compared with 17.0 tokens/s for naive Hugging Face generation.The profile uses one NVIDIA A800 80 GB GPU and bfloat16 precision.
  • C.2 Inference Efficiency and Memory: Autoregressive generation is the primary inference bottleneck, whereas Table 9 separates it from VQ-VAE decoding and marching cubes.Shape extraction decodes each part sequentially before applying marching cubes.
  • C.2 Inference Efficiency and Memory: Mesh extraction adds overhead because parts are decoded sequentially and marching cubes runs at resolution 10243 for every part.This resolution is applied even when part geometry is simple.
  • C.2 Inference Efficiency and Memory: Extraction cost can be reduced by lowering the resolution, batching part decoding, or overlapping shape decoding with autoregressive generation.These optimizations target the sequential per-part decoding and high-resolution marching-cubes overhead.
  • C.2 Inference Efficiency and Memory: 71.24 GB is the peak GPU memory for one generated 300-part object with an 188k-token context, fitting on a single 80 GB GPU.Peak memory is determined mainly by token length, and part count is not strictly monotonic with sequence length.
  • C.3 Rate–Distortion Selection Cost: 1.8 seconds per part is the average cost of sweeping nine candidate prefix lengths with GPU parallelism.The candidates are defined by L∈{2^n} for n=4 through 12.
  • C.3 Rate–Distortion Selection Cost: Rate–distortion selection adds no generation-time overhead because the autoregressive model directly predicts each part’s number of shape tokens.It is used only offline to construct adaptive-length training sequences for the LLM.

C.4 Human Perceptual Evaluation · D Additional Visualization Results · D.1 Additional Emergent Generation Behaviors

The human perceptual evaluation measures text-prompt alignment and generated-geometry quality, with MegaParts receiving most preference votes. Additional visualizations show emergent plausible internal structures despite no explicit interior-structure training.

  • C.4 Human Perceptual Evaluation: The human perceptual study evaluates alignment between text prompts and generated geometry, as well as overall geometry quality.
  • C.4 Human Perceptual Evaluation: Generated samples are randomly shuffled and identities are hidden during the perceptual evaluation.
  • C.4 Human Perceptual Evaluation: The preference score is defined as the fraction of votes received by each method.
  • C.4 Human Perceptual Evaluation: 64% of votes favor our model for text alignment, exceeding the compared methods.
  • C.4 Human Perceptual Evaluation: 69% of votes favor our model for geometry quality, substantially exceeding the compared methods.
  • D Additional Visualization Results: Additional visualization results include representative examples of objects with plausible internal structures.The model occasionally produces these structures despite not being explicitly trained for interior-structure modeling.
  • D.1 Additional Emergent Generation Behaviors: MegaParts occasionally generates plausible internal structures without explicit interior-structure modeling training.

D.2 Additional Text to 3D Generation Results

This section presents additional visual results for text-to-3D generation, including comparisons with baselines and dishwashers featuring detailed interiors.

  • Text-to-3D generation: Figure 13 presents additional visual comparisons between the model and baselines for text-conditioned 3D generation.These results are described as additional visualizations of text-to-3D generation.
  • Generated examples: Figure 12 shows three dishwashers generated by the model with high-quality interior details.The figure specifically depicts three dishwasher examples.
Loading 2608.14783v1…