Source-linked AI summary
Hunyuan3D-Omni: A Unified Framework for Controllable Generation of 3D Assets
Team Hunyuan3D, :, Bowen Zhang, Chunchao Guo, Haolin Liu, Hongyu Yan, Huiwen Shi, Jingwei Huang, Junlin Yu, Kunhong Li, Linus, Penghao Wang, Qingxiang Lin, Sicong Liu, Xianghui Yang, Yixuan Tang, Yunfei Zhao, Zeqiang Lai, Zhihao Liang, Zibo Zhao
TL;DR
Existing 3D generators primarily use image or text conditioning and lack fine-grained, cross-modal control over diverse geometric and pose inputs. Hunyuan3D-Omni builds a unified controllable generation framework on Hunyuan3D 2.1, and experiments show improved generation accuracy, geometry-aware transformations, and robustness for production workflows.
Problem
Most existing 3D generators lack unified support for complex conditions such as points, voxels, bounding boxes, and poses, limiting controllability and practical adoption.
Method
Hunyuan3D-Omni uses a lightweight unified control encoder to integrate point clouds, voxels, bounding boxes, and skeletons into one Hunyuan3D 2.1-based generative model.
Results
Additional controls improve generation accuracy, enable geometry-aware transformations, and increase robustness for production workflows.
Takeaways & Limitations
The framework provides fine-grained controllable 3D asset generation across geometry, aspect ratio, and pose-related inputs.
Abstract
from arXiv · showhide
Recent advances in 3D-native generative models have accelerated asset creation for games, film, and design. However, most methods still rely primarily on image or text conditioning and lack fine-grained, cross-modal controls, which limits controllability and practical adoption. To address this gap, we present Hunyuan3D-Omni, a unified framework for fine-grained, controllable 3D asset generation built on Hunyuan3D 2.1. In addition to images, Hunyuan3D-Omni accepts point clouds, voxels, bounding boxes, and skeletal pose priors as conditioning signals, enabling precise control over geometry, topology, and pose. Instead of separate heads for each modality, our model unifies all signals in a single cross-modal architecture. We train with a progressive, difficulty-aware sampling strategy that selects one control modality per example and biases sampling toward harder signals (e.g., skeletal pose) while downweighting easier ones (e.g., point clouds), encouraging robust multi-modal fusion and graceful handling of missing inputs. Experiments show that these additional controls improve generation accuracy, enable geometry-aware transformations, and increase robustness for production workflows.
1 Introduction
3D generation is becoming an important tool for digital content creation, but single-image reconstruction remains ambiguous. Hunyuan3D-Omni addresses this by adding multiple control signals within a unified framework for more accurate and customized generation.
- Growing 3D datasets are expected to support faster automated modeling workflows and dynamic interactions in digital content creation.
- Native 3D generation commonly combines a 3D VAE with a latent diffusion model to encode and generate 3D representations.The described VecSet pipeline compresses point clouds, decodes an SDF field, and extracts the visible model with iso-surface sampling.
- Single-image 3D reconstruction is ill-posed, creating uncertainty and ambiguity in complete geometry generation.Additional information can improve geometric fidelity and support customized outputs.
- Hunyuan3D-Omni unifies point clouds, voxels, bounding boxes, and skeletons as additional conditions for fine-grained controllable 3D generation.It builds on Hunyuan3D 2.1 and uses a unified control encoder alongside image features.
- These control signals are intended to improve geometric accuracy while enabling customized outputs beyond image-only generation.The framework targets distortions, flatness, missing details, and aspect-ratio discrepancies.
2 Related Work
Existing 3D generators provide fast image- or text-conditioned synthesis but generally lack broad, unified control over points, voxels, bounding boxes, and poses. Prior controllable methods address selected modalities, leaving cross-modal integration within one model insufficiently explored.
- 3D Native Generation: Native 3D methods based on VAE and latent diffusion architectures improve generation quality and speed compared with earlier optimization-based approaches.
- 3D Native Generation: Image- and text-based 3D generators generally lack complex conditioning signals such as points, bounding boxes, and voxels, limiting controllability and practical adoption.
- 3D Controllable Generation: 2D controllable diffusion methods show that structured conditions such as edges, depth, and pose can improve precision and practicality.
- 3D Controllable Generation: Prior controllable approaches adapt models to point clouds, bounding boxes, voxels, or pose, but mostly focus on single or limited conditions.
- 3D Controllable Generation: A unified framework integrating points, voxels, bounding boxes, and poses within one model remains an insufficiently explored challenge.
3 Method
Hunyuan3D-Omni extends Hunyuan3D 2.1 with a unified encoder for point clouds, voxels, bounding boxes, and skeletons, enabling controllable 3D generation. Its controls are represented and fused as point-cloud-like features before entering the DiT, while each modality targets different geometric or pose objectives.
- Hunyuan3D-Omni: The framework adds point clouds, voxels, bounding boxes, and skeletons to Hunyuan3D 2.1 through a unified control encoder.The architecture retains a DiT and VAE-based decoder while introducing additional geometric controls.
- Skeleton Condition: Skeleton inputs specify character pose using 3D bone-start coordinates, with body and hand bones represented as P_pose ∈ R^M×6.Training pairs use an image from one frame and the skeleton and mesh from another.
- Bounding Box Condition: Bounding-box inputs encode eight canonical-space vertices to control object aspect ratios and support edits to length, width, and height.Training perturbs rendered images or point clouds to create image–box misalignment.
- Point Cloud Condition: Point-cloud conditions use spatial coordinates at 512, 1024, or 2048 points, with random drops and noise perturbations simulating incomplete or sensor-derived inputs.The representation is P_c ∈ R^N_c×3, where N_c is the number of points.
- Unified Control Encoder: All controls are converted to point-cloud-like features, dimension-aligned, embedded, and distinguished by condition type before being aggregated with image features for the DiT.Voxel, point-cloud, and bounding-box channels are repeated to match dimensions, while skeletons use bone points.
4 Experiment
Hunyuan3D-Omni supports pose, bounding-box, point-cloud, and voxel controls that improve geometric alignment, scale, detail recovery, and controllable transformations.
- Experiment setup: The qualitative evaluation generates controlled results with Hunyuan3D-Omni before refining geometry using Hunyuan3D 2.5.Training uses randomly sampled control conditions with batch size 1 because the conditions have varying lengths.
- Skeleton condition: Skeleton conditioning produces detailed human meshes aligned with target poses across varied character-image styles without artifacts.The demonstrated poses include A pose, sky pose, and handsup pose.
- Bounding box condition: Bounding-box conditioning regulates output scale and aspect ratio while enabling plausible geometry changes and valid meshes when image-only generation fails.Lengthening a sofa adds supporting legs, and bounding-box cues can avoid thin, sheet-like outputs.
- Point cloud condition: Point-cloud conditioning improves alignment with real-world geometry and recovers occluded structure from complete, depth-derived, or scanned inputs.Even noisy or partial point clouds provide useful geometric cues compared with image-only generation.
- Voxel condition: Voxel conditioning improves scale alignment and recovers fine details such as shield surfaces, bird wings, and low-poly cup geometry.These controls help preserve object proportions and details relative to ground-truth geometry.
5 Conclusion
The paper presents Hunyuan3D-Omni as a unified framework for fine-grained, controllable 3D asset generation with multiple conditioning modalities.
- Conclusion: Hunyuan3D-Omni unifies point clouds, voxels, bounding boxes, and skeletons in one diffusion model through a lightweight control encoder.The framework builds on Hunyuan3D 2.1 and targets geometry distortion, style control, and controllable generation.
- Conclusion: Experiments show improved generation accuracy, geometry-aware transformations, and robustness for production workflows.
6 Contributors
The listed contributors are credited alphabetically by first name.
- Contributors: The authors are listed alphabetically by first name.
- Contributors: The contributor list includes Bowen Zhang, Chunchao Guo, Haolin Liu, Hongyu Yan, Huiwen Shi, Jingwei Huang, Junlin Yu, Kunhong Li, and Linus.
- Contributors: The contributor list also includes Penghao Wang, Qingxiang Lin, Sicong Liu, Xianghui Yang, Yixuan Tang, Yunfei Zhao, Zeqiang Lai, Zhihao Liang, and Zibo Zhao.