Source-linked AI summary
Know3D: Prompting 3D Generation with Knowledge from Vision-Language Models
Wenyue Chen, Wenjue Chen, Peng Li, Qinghe Wang, Xu Jia, Heliang Zheng, Rongfei Jia, Yuan Liu, Ronggang Wang
TL;DR
Single-view 3D generation struggles to infer unseen regions because observations are ambiguous and 3D data provide limited structural priors. Know3D transfers VLM knowledge through a multimodal diffusion bridge and intermediate hidden-state injection, achieving semantic control of backside generation and competitive semantic consistency on HY3D-Bench. The authors conclude that this converts stochastic back-view hallucination into a semantically controllable process, while noting dependence on the multimodal foundation model’s instruction understanding.
Problem
Single-view 3D generation lacks reliable global structural priors and produces unseen regions that can be stochastic, difficult to control, or geometrically implausible.
Method
Know3D uses a VLM-diffusion model to translate semantic knowledge into image-space structural priors, then injects intermediate hidden states into 3D generation.
Results
Know3D achieves competitive semantic consistency against state-of-the-art single-view 3D generation methods on HY3D-Bench and enables language-controllable backside generation.
Takeaways & Limitations
Know3D provides a semantically controllable process for reconstructing unobserved backside regions from textual instructions.
Takeaways & Limitations
Generated structural robustness remains dependent on whether the multimodal foundation model correctly understands the instructions.
Abstract
from arXiv · showhide
Recent advances in 3D generation have improved the fidelity and geometric details of synthesized 3D assets. However, due to the inherent ambiguity of single-view observations and the lack of robust global structural priors caused by limited 3D training data, the unseen regions generated by existing models are often stochastic and difficult to control, which may sometimes fail to align with user intentions or produce implausible geometries. In this paper, we propose Know3D, a novel framework that incorporates rich knowledge from multimodal large language models into 3D generative processes via latent hidden-state injection, enabling language-controllable generation of the back-view for 3D assets. We utilize a VLM-diffusion-based model, where the VLM is responsible for semantic understanding and guidance. The diffusion model acts as a bridge that transfers semantic knowledge from the VLM to the 3D generation model. In this way, we successfully bridge the gap between abstract textual instructions and the geometric reconstruction of unobserved regions, transforming the traditionally stochastic back-view hallucination into a semantically controllable process, demonstrating a promising direction for future 3D generation models.
1 Introduction
Single-view 3D generation is ambiguous, making unseen-region synthesis stochastic, semantically uncontrollable, and sometimes geometrically implausible. Know3D addresses this by transferring VLM knowledge through image-space structural priors into 3D generation.
- Motivation: Single-view observations make 3D reconstruction fundamentally ill-posed, while limited 3D data constrain models’ world knowledge and structural common sense.These constraints contribute to outputs that may deviate from user intentions or violate semantic commonsense constraints.
- Motivation: VLMs provide rich semantic knowledge and commonsense reasoning that may guide inference of unobserved object structures from visible evidence.
- Challenges: Direct autoregressive use of VLMs for 3D generation can underperform dedicated 3D models and disrupt pretrained semantic priors.
- Challenges: Directly feeding VLM representations into 3D networks is problematic because abstract features lack alignment with spatially structured features needed for shape synthesis.
- Know3D: Know3D uses a multimodal diffusion model as an intermediate bridge that translates VLM semantic knowledge into image-space structural priors for controllable, plausible 3D generation.The VLM provides semantic understanding and image-generation guidance, while the diffusion model generates unobserved parts.
- Results: Know3D achieves competitive semantic consistency on HY3D-Bench and enables language-controllable generation of unseen backside regions.
2 Related Works
Related work spans native 3D generation, text-to-3D distillation, multimodal 3D generation, and unified multimodal architectures. These approaches improve generation capabilities but retain limitations in controllability, geometric fidelity, or representation resolution.
- Single-view 3D generation: Native single-view 3D generation has developed around VecSet and Sparse Voxel latent paradigms, emphasizing global compression and local topological control, respectively.
- Text-to-3D generation: DreamFusion introduced score distillation from pretrained 2D diffusion models, followed by methods refining the distillation pipeline for improved 3D generation.
- Text-to-3D generation: Text-to-3D methods still lag image-guided image-to-3D approaches in controllability and fine-grained geometric accuracy.
- Multimodal 3D generation: Multimodal large language model approaches to 3D generation remain constrained by limited representation resolution and therefore do not achieve high-fidelity 3D content generation.
- Unified multimodal models: Unified multimodal models include autoregressive, diffusion, decoupled LLM-diffusion, and hybrid AR-diffusion architectures, while 3D generation develops later than multimodal modeling.
3 Method
Know3D fine-tunes a multimodal image-editing model to generate text-controlled back views, then injects intermediate diffusion hidden states as structural-semantic conditioning for complete 3D synthesis.
- Overview: Know3D takes a front-view image and back-view text description, generates a plausible backside image, and synthesizes a complete 3D representation.
- Semantic-aware back-view generation: The method fine-tunes Qwen-Image-Edit because its original model often misidentifies back viewpoints and alters the subject’s pose.
- Semantic-aware back-view generation: Training data pairs opposite rendered views of high-quality 3D assets and annotates salient back-view components with textual descriptions.
- Training strategy: A fixed 180° rotation prompt is combined with a component description sampled with probability 0.5, supporting both unconditional and semantically controlled back-view generation.
- Knowledge extraction and prompting: Qwen2.5-VL encodes image and prompt semantics, while VAE features and intermediate MMDiT hidden states provide structural-semantic conditioning during denoising.
- Knowledge extraction and prompting: Intermediate hidden states are concatenated and injected through a parallel cross-attention branch with zero-initialized residual scaling before two-stage 3D generation.The two stages generate coarse sparse structure and then recover high-fidelity fine geometry.
3D Geometry Generation
Know3D generates 3D geometry in two stages: a coarse sparse structure establishes a global topological prior, followed by high-fidelity geometry recovery conditioned on it.
- The generation equations apply separate structure and geometry decoders to noisy inputs, with geometry generation conditioned on V_ss.
- The first stage generates a coarse sparse structure V_ss to model the global topological prior.
- The second stage recovers high-fidelity fine geometry V_geo conditioned on the coarse structure V_ss.
- The diffusion process uses standard Gaussian noise z and timestep τ as inputs.
- Both stages are optimized with the Conditional Flow Matching objective.
4 Experiments
Experiments use TexVerse assets and rendered viewpoints to train semantic-aware front-back view generation and 3D generation, with specified viewpoint perturbations and Qwen-Image-Edit adaptation.
- Semantic-aware front-back view generation uses 5k high-quality 3D assets selected from TexVerse.
- For each asset, field of view is sampled from {35°, 50°, 85°, 105°, 135°} and elevation from [−15°, 45°].
- The training renders 12 uniformly spaced azimuthal views over 360° per mesh, forming 6 front-back pairs.
- 3D generation training uses 60k TexVerse meshes with perturbation-free and perturbed view sets.
- Qwen-Image-Edit-2511 is fine-tuned with LoRA, while its parameters are frozen for 3D generation to preserve visual priors.
Training Details.
Training details describe semantic-consistency evaluation against ULIP and Uni3D, alongside an ablation protocol focused on sparse voxel generation and geometric metrics.
- ULIP and Uni3D measure semantic consistency between images and generated meshes.
- The ablation study trains only the first stage, which generates sparse voxels.
- Ablation performance is evaluated using IoU and Chamfer Distance (CD).
- The experiments compare Know3D with current state-of-the-art methods and analyze its semantic controllability.
- Generation quality is evaluated against both single-view and multi-view 3D generation methods.
Comparison with Baselines.
Know3D achieves competitive semantic alignment against single-image baselines and outperforms Hunyuan3D-2mv in semantic scores, while qualitative results show more plausible and controllable unseen geometry.
- Know3D achieves competitive ULIP and Uni3D scores against several state-of-the-art single-image-to-3D methods.
- Know3D outperforms Hunyuan3D-2mv in ULIP and Uni3D scores.
- Hunyuan3D-2mv generates distorted and implausible geometries from novel viewpoints, whereas Know3D maintains consistent structures.
- Know3D’s back-view content can be semantically controlled through natural-language instructions.
- Unconstrained Know3D back-view completions already produce geometrically plausible and semantically consistent structures aligned with the front view.
Back-View Semantic Controllability.
Know3D uses semantic knowledge to improve the plausibility of unseen 3D regions and enables text-guided control of back-side content while preserving front-view consistency.
- Know3D leverages semantic knowledge from pretrained multimodal models to improve the structural plausibility of unseen 3D parts.
- Text prompts modify unseen back-side content while preserving geometric consistency with the original front view.
4.3 Ablation Study
The ablations examine denoising timestep and feature representation choices for prompting 3D generation. Intermediate MMDiT hidden states at t = 0.25 provide the strongest reported guidance.
- The ablation varies MMDiT feature-extraction timestep and compares hidden states, VAE features, and DINOv3 features for guiding 3D generation.
- t = 0.25 achieves the best overall performance, with the highest IoU and lowest Chamfer Distance.
- At t = 0.25, MMDiT hidden states outperform alternative feature representations on both reported metrics.
Different Feature Representations to Prompt.
Intermediate MMDiT hidden states provide more effective semantic and structural guidance than VAE or DINOv3 representations. The comparison links these differences to the quality of generated back-view geometry.
- MMDiT hidden states extracted at t = 0.25 consistently outperform other feature representations on both metrics.
- VAE features perform worst because pixel-level reconstruction preserves low-level appearance while discarding higher-level semantic and structural cues.
- Intermediate MMDiT hidden states produce reasonable results, whereas VAE and DINOv3 can preserve or fit erroneous back-view information.
5 Limitation and Conclusion
Know3D frames semantic controllability as a solution to ambiguity in single-view 3D synthesis by using intermediate multimodal diffusion states as structural-semantic priors. Its robustness remains dependent on the underlying multimodal foundation model's instruction understanding.
- Know3D addresses ambiguity and limited semantic control in single-view 3D synthesis by prompting generation with intermediate MMDiT structural-semantic priors.
- The framework makes back-view hallucination semantically controllable by connecting textual instructions with geometric reconstruction of unobserved regions.
- When the multimodal foundation model does not fully understand instructions, 3D generation can still be misled toward incorrect shapes.