PaperlayerView exact revision on arXiv

Source-linked AI summary

Hunyuan3D-Buffalo 1.0: A Unified Multimodal Model for Scalable 3D Generation, Understanding, and Editing

Junliang Ye, Kenkun Liu, Guocun Wang, Yang Li, Yansong Qu, Chunshi Wang, Jingwei Xu, Yunhan Yang, Zibo Zhao, Jiachen Xu, Jiaao Yu, Lifu Wang, Zhihao Liang, Xin Huang, Zhuo Chen, Chunchao Guo

arXiv:2608.02711v3cs.CV

TL;DR

Unified 3D modeling is limited by scarce, geometrically consistent multimodal data, especially for editing. Hunyuan3D-Buffalo 1.0 addresses this with a unified framework and large multimodal corpus spanning understanding, generation, editing, and part generation, achieving state-of-the-art or leading performance across generation and editing benchmarks while showing strong understanding and part-generation abilities.

  • Problem

    Unified 3D modeling lacks large-scale, geometrically consistent multimodal data, leaving understanding, generation, and editing largely fragmented across separate systems.

  • Method

    Hunyuan3D-Buffalo 1.0 combines autoregressive understanding with diffusion-based 3D synthesis and an 87M-sample corpus spanning understanding, text-to-3D, editing, and part generation.

  • Results

    Hunyuan3D-Buffalo 1.0 achieves state-of-the-art or leading performance on text-to-3D generation and 3D editing benchmarks, with generation and understanding improving editing.

  • Takeaways & Limitations

    Unified 3D multimodal training supports capability transfer across tasks, particularly from generation and understanding to editing.

  • Takeaways & Limitations

    The Nano3D-v2 editing-data pipeline can introduce inconsistencies inside edit masks, degrading editing quality.

Abstract

from arXiv · show

Recent advances in image generation have demonstrated the potential of unified multimodal models that integrate understanding, generation, and editing. However, unified 3D modeling remains constrained by scarce multimodal data, particularly the lack of large-scale and geometrically consistent editing data. To address this limitation, we propose Hunyuan3D-Buffalo 1.0, a unified framework supporting 3D understanding, text-to-3D generation, instruction-guided 3D editing, and text-grounded part generation within a single architecture. To enable scalable training, we construct an 87M-scale 3D multimodal corpus, comprising 25M understanding samples, 50M text-to-3D pairs, and 12M editing pairs generated using Nano3D-v2. Architecturally, the framework combines Hunyuan3D-VLM for semantic, structural, and spatial understanding with Hunyuan3D DiT for high-fidelity 3D synthesis. The VLM provides multimodal semantic conditions for generation, while editing and part generation additionally condition the diffusion process on the source object representation to preserve its overall structure and unedited regions. Extensive experiments show that Hunyuan3D-Buffalo 1.0 achieves state-of-the-art or leading performance on text-to-3D generation and 3D editing benchmarks, while exhibiting strong understanding and part-generation capabilities. Our analysis further shows that both generation and understanding improve editing, demonstrating the effectiveness of unified 3D multimodal training. Project Page: https://tencent-hunyuan.github.io/Hunyuan3D-Buffalo1.0/

1 Introduction

Hunyuan3D-Buffalo 1.0 addresses the scarcity of large-scale, geometrically consistent 3D editing data with a unified framework for understanding, generation, editing, and part generation. It combines a large multimodal corpus and Hunyuan3D-VLM with diffusion-based synthesis, achieving leading benchmark performance and revealing cross-task synergies.

  • 1 Introduction: Large-scale and geometrically consistent 3D editing data remains scarce because 3D assets are difficult to collect, annotate, and edit while preserving identity, structure, and unedited regions.This bottleneck limits progress in 3D understanding, generation, and editing models [63] [64] [93] [40] [47] [92] [109] [12] [43] [104].
  • 1 Introduction: An 87M-sample 3D multimodal corpus combines 25M understanding samples, 50M text-to-3D pairs, and 12M editing pairs.Nano3D-v2 produces high-quality, geometrically consistent editing pairs at scale through agent-based data construction.
  • 1 Introduction: Hunyuan3D-VLM provides fine-grained semantic, structural, and spatial understanding for 3D grounding, part-level reasoning, and edit-aware understanding.It encodes geometric structure and appearance cues to support captioning, part-level question answering, edit-instruction synthesis, and edit-outcome reasoning.
  • 1 Introduction: The unified framework combines autoregressive modeling with diffusion-based 3D generation to support understanding, text-to-3D generation, 3D editing, and text-grounded part generation in one architecture.Hunyuan3D-VLM supplies multimodal reasoning that guides synthesis and editing through a unified conditional interface.
  • 1 Introduction: Hunyuan3D-Buffalo 1.0 achieves state-of-the-art or leading performance on text-to-3D generation and 3D editing benchmarks, while showing strong 3D understanding and part-generation abilities.Unified training further reveals that stronger generation and stronger understanding improve 3D editing.

2 Related Work

Related work spans optimization- and diffusion-based 3D generation, text-guided editing, structured part generation, and unified multimodal modeling. These paradigms address scalable synthesis, 3D-consistent modification, semantic decomposition, and joint image-text understanding and generation through distinct designs.

  • 3D Content Generation: 3D generation evolved from optimization-based distillation of pretrained 2D diffusion priors, exemplified by DreamFusion [62], toward approaches targeting greater efficiency and multi-view consistency, including 3DShape2VecSet [108] and TRELLIS [92].Early methods enabled text-to-3D generation without large-scale 3D supervision by distilling visual priors into 3D representations.
  • Text-Guided 3D Editing: Text-guided 3D editing modifies existing assets from natural-language instructions while preserving unedited regions and 3D consistency, using per-instance SDS or 2D diffusion optimization and, alternatively, edited-view fusion or reconstruction.The passage identifies both optimization-based methods and pipelines that edit rendered 2D views before producing edited 3D assets.
  • 3D Part Generation: 3D part generation seeks structured assets with distinct meshes for semantic components, progressing from fixed taxonomies [60] [95] and view-inconsistent 2D-segmentation pipelines toward coordinated multi-diffusion branches [35] [48] [49] [58] [97] [98] [117].Recent 3D-native approaches synthesize parts through coordinated diffusion paths, while the passage also introduces continuous feature fields for part segmentation.
  • Unified Multimodal Modeling: Unified image understanding and generation follows token-based autoregressive designs, such as Chameleon [67] [74] [82] [89], and more decoupled multimodal architectures, exemplified by MetaQuery [9] [10] [61] [75] [87] [88] [112].Chameleon converts images into discrete VQVAE visual tokens for joint text-image next-token modeling; MetaQuery represents a separate decoupled direction.

3 Data curation

The authors build a unified 3D data engine producing complementary understanding, text-to-3D, and editing corpora to address scarce, geometrically consistent multimodal training data. Automated pipelines generate these datasets while preserving semantic coverage, geometric fidelity, and consistency across edits and views.

  • Corpus overview: The data engine constructs three complementary corpora for 3D understanding, text-to-3D generation, and instruction-guided 3D editing.This design directly targets the scarcity of large-scale, high-quality, geometrically consistent data spanning all three capabilities.
  • 3D understanding corpus: The understanding corpus combines point-cloud dialogues with text and image–text data covering captioning, question answering, grounding, and editing-related tasks.The dialogues describe geometry, structure, and parts; answer spatial and attribute questions; localize regions; and connect editing operations with their geometric outcomes.
  • Text-to-3D corpus: The text-to-3D corpus uses a fully automated five-stage pipeline that synthesizes compositional prompts, generates and renders assets, captions them, scores geometry quality, and filters the results.Prompts are organized through a hierarchical taxonomy and assets are stored with canonical multi-view renders and sampled surface point clouds.
  • Editing corpus: Nano3D-v2 generates scalable, high-quality editing pairs by executing natural-language edits while preserving non-target geometry, identity, and multi-view consistency.Its quality control compares pre- and post-edit images, verifies unchanged regions outside detected differences, and rejects implausibly large edit masks.
  • Part-generation data: Part-level instruction data extends the unified model to language-guided part localization and decomposition rather than relying on specialized geometry modules or bounding-box supervision.Semantic decompositions support queries such as segmenting wheels or removing a handle, addressing the limited semantic information in raw mesh-level parts.

4 Method

Hunyuan3D-Buffalo 1.0 unifies 3D understanding, generation, grounding, and editing through a shared Hunyuan3D-VLM backbone connected to a Hunyuan3D DiT. Its multimodal architecture combines structure-and-appearance 3D representations, semantic conditioning, and source-object conditioning for structurally consistent editing and part generation.

  • Architecture: The unified pipeline connects Hunyuan3D-VLM to Hunyuan3D DiT through an MLP-Connector, using the VLM for multimodal reasoning and the DiT for 3D synthesis.The 3D-DiT is initialized from Hunyuan3D-2.1, while the connector aligns VLM hidden states with the DiT conditioning space.
  • 3D-aware Vision Language Model: Hunyuan3D-VLM encodes colored point clouds through geometric and RGB pathways, then compresses their latent tokens into 512-token sequences for efficient multimodal fusion.Geometric inputs include XYZ coordinates and surface normals, while RGB cues help distinguish visually different but geometrically similar parts.
  • 3D Editing and Part Generation: For editing and part generation, the diffusion process combines VLM semantic embeddings with the source object representation, which is concatenated with noisy latents in DiT self-attention.This gives the denoiser direct access to the original geometry during generation.
  • Training Pipeline: Training proceeds through four stages: 3D-VLM pre-training, text-to-3D pre-training, unified omni pre-training, and task-specific continued pre-training.The final stage branches into separate editing, text-to-3D, and part-generation paths, while generative stages use flow matching to predict the velocity field toward target 3D latents.
  • Training Pipeline: During omni pre-training, text-to-3D is balanced against editing plus part generation at a 1:1 sampling ratio, while continued training preserves generation through task-specific data mixing.Editing and part-generation paths retain half text-to-3D data, whereas the text-to-3D path uses only text-to-3D data.

5 Experiments

Experiments show that Hunyuan3D-Buffalo 1.0 provides strong unified 3D understanding, text-to-3D preference, instruction-guided editing, and open-vocabulary part generation. Its 3D-VLM conditioning and scalable text-to-3D training improve semantic accuracy, geometric quality, and localized structure-preserving edits.

  • 3D Understanding: Hunyuan3D-VLM achieves the best reported performance on UniPart-Bench part-level Q&A and object captioning, including 85.47 SBERT, 89.06 SimCSE, 49.95 BLEU-1, and 45.79 METEOR for part understanding.The benchmark evaluates complementary localized part reasoning and holistic object-level description.
  • 3D Understanding: Hunyuan3D-VLM obtains 0.864 IoU on pure box listing and performs strongly across multi-part grounding, single-part grounding, box-to-text generation, and part QA.These tasks jointly assess localization, region-conditioned description, and part-aware reasoning.
  • Text-to-3D Generation: Hunyuan3D-Buffalo 1.0 is preferred on text alignment, geometry quality, and overall preference at 55.2%, 57.1%, and 56.6%, respectively, versus Omni123 [101] at 17.5%, 21.0%, and 18.4%.The human evaluation used 100 diverse text prompts and four-way comparison groups; all reported margins exceed the 25% random-choice level.
  • Instruction-Guided 3D Editing: On Edit3D-Bench [86], the 3D-VLM version reduces average CD from 0.0684 to 0.0091 versus Omni123 [101], an 86.7% relative reduction, and outperforms prior methods across CD and F1.Against the CLIP-conditioned variant, 3D-VLM conditioning lowers average CD from 0.0158 to 0.0091 and raises average F1 from 0.6336 to 0.6515.
  • Instruction-Guided 3D Editing: Qualitative results show localized additions and removals that preserve the input shape’s overall structure, pose, fine-grained details, and unrelated regions.The method places added components at the correct semantic locations while maintaining unedited geometry.
  • Text-Grounded Part Generation: The method supports open-vocabulary, text-grounded part generation across diverse objects, extracting queried geometry with high fidelity to the input shape.This capability applies to structurally complex and geometrically simple objects, and multiple extracted parts can be combined.

6 Conclusion & Future Work

The unified multimodal 3D model jointly addresses understanding, generation, and editing, achieving state-of-the-art performance across all three tasks. Future work targets representation, data quality, texture editing, editing-data robustness, and architecture scaling.

  • Conclusion: The unified model achieves state-of-the-art performance in 3D understanding, generation, and editing, with significant gains over prior methods in generation and editing.This supports unified 3D modeling as a promising direction for handling multiple tasks within one framework.
  • Future Work: Future work should develop a scalable single-stage representation that delivers high-quality geometry without the multi-stage pipelines required by approaches such as TRELLIS [92].Multi-stage designs make high-quality unified editing difficult to scale.
  • Future Work: Improving captioning quality and scaling 3D data in both volume and quality are critical for reducing noisy text-to-3D training pairs and improving performance.Current captions from multimodal models such as Gemini remain ambiguous, while 3D data has not reached an ideal scale.
  • Future Work: End-to-end texture editing remains unexplored, requiring suitable data and potentially jointly modeled geometry and texture representations.A unified single-stage representation for high-quality geometry and texture remains an open question.
  • Future Work: More robust editing-data construction is needed because non-edited regions inside replacement masks can become inconsistent and degrade end-to-end editing quality.The Nano3D-v2-based pipeline preserves consistency outside masks but struggles with non-edited content inside them.
  • Future Work: Exploring Transfusion-style architectures could improve 3D generation by deeply fusing information across modalities beyond the current cascaded AR + DiT framework.Such architectures have shown effectiveness in image and video generation and are identified as a promising next step for 3D.

7 Author List

The paper credits 16 authors, identifies three project leaders and several core contributors, and lists contributors by major research area.

  • Authors: The author list comprises Junliang Ye, Kenkun Liu, Guocun Wang, Yang Li, Yansong Qu, Chunshi Wang, Jingwei Xu, Yunhan Yang, Zibo Zhao, Jiachen Xu, Jiaao Yu, Lifu Wang, Zhihao Liang, Xin Huang, Zhuo Chen, and Chunchao Guo.Junliang Ye, Kenkun Liu, Guocun Wang, Yang Li, Yansong Qu, and Chunshi Wang are marked as core contributors.
  • Author Roles: Yang Li, Zhuo Chen, and Chunchao Guo are identified as project leaders, while the asterisk marks core contributors.Yang Li, Zhuo Chen, and Chunchao Guo carry the project-leader designation; the source also marks core contributors with an asterisk.
  • Research Contributions: Contributors are assigned across 3D editing, text-to-3D, and 3D understanding, with Yang Li participating in all three areas.The listed 3D-editing contributors are Junliang Ye, Guocun Wang, Yansong Qu, Yang Li, Chunshi Wang, and Kenkun Liu; text-to-3D lists Kenkun Liu, Junliang Ye, and Yang Li; 3D understanding lists Guocun Wang, Junliang Ye, Kenkun Liu, and Yang Li.

A Additional Results

This section presents qualitative results for shape editing and text-to-3D generation. Together, the figures illustrate the model’s performance across both tasks.

  • Qualitative shape editing results are presented in Fig. 12.
  • The additional results cover both shape editing and text-to-3D generation qualitatively.
  • Qualitative text-to-3D results are presented in Fig. 13.
Loading 2608.02711v3…