Source-linked AI summary
ImageDream: Image-Prompt Multi-view Diffusion for 3D Generation
Peng Wang, Yichun Shi
TL;DR
Image-conditioned 3D generation must preserve detailed image-specific appearance while maintaining consistent geometry across views. ImageDream addresses this with canonical camera coordination and multi-level image-prompt control, and evaluations report superior quality over state-of-the-art image-conditioned methods, with limitations under overly strict image constraints.
Problem
Image-conditioned 3D generation is difficult because image-specific textures, shapes, and fur details can cause novel-view generation to deviate from trained distributions.
Method
ImageDream combines canonical camera coordination with hierarchical image-prompt controllers integrated into a multi-view diffusion model.
Results
ImageDream surpasses existing state-of-the-art zero-shot single-image 3D generators in geometry and texture quality, with ImageDream-P particularly favored in a double-blind user study.
Takeaways & Limitations
ImageDream enables image-prompt 3D generation with improved geometric correctness and supports leveraging image generation models for image-text alignment.
Takeaways & Limitations
The pixel-controller model can struggle with small facial details under overly stringent image constraints and may require better controller balance or larger models.
Abstract
from arXiv · showhide
We introduce "ImageDream," an innovative image-prompt, multi-view diffusion model for 3D object generation. ImageDream stands out for its ability to produce 3D models of higher quality compared to existing state-of-the-art, image-conditioned methods. Our approach utilizes a canonical camera coordination for the objects in images, improving visual geometry accuracy. The model is designed with various levels of control at each block inside the diffusion model based on the input image, where global control shapes the overall object layout and local control fine-tunes the image details. The effectiveness of ImageDream is demonstrated through extensive evaluations using a standard prompt list. For more information, visit our project page at https://Image-Dream.github.io.
1. Introduction
ImageDream targets the geometric and textural difficulties of image-conditioned 3D generation with canonical camera coordination and hierarchical image-prompt control. It reports stronger geometry and texture quality than prior state-of-the-art methods in evaluations.
- Images provide richer visual cues than text for conveying textures, colors, spatial relationships, and object details in 3D generation.
- Image-conditioned generation remains challenging because lighting variation, shape, self-occlusion, and limited 3D data can produce inconsistent views and blurry or incomplete models.
- Canonical camera coordination makes the default rendered view a centered front view, simplifying the mapping from input-image variations to 3D.
- The multi-level image-prompt controller provides hierarchical guidance from the image input to diffusion-model blocks, streamlining information transfer.
- ImageDream generates more geometrically correct objects and surpasses state-of-the-art methods such as Magic123 in geometry and texture quality.
2. Related Works
Related work spans direct 3D generative models, diffusion-prior lifting into 3D representations, and image-based novel-view synthesis. These approaches advance generation but face generalization, geometric, and consistency challenges.
- Deep generative 3D methods progressed from reconstruction with multi-view images through GANs to diffusion-based frameworks.
- Recent direct 3D diffusion models use triplanes or feature grids but often target specific object categories and remain limited in generalizability.
- Score distillation sampling lifts 2D diffusion priors into 3D generation, often optimizing representations such as NeRF.
- Image-based novel-view synthesis includes latent-space, epipolar-transformer, view-consistency, and unprojected-feature-grid approaches for reconstructing views from single images.
- Parallel methods incorporate normal diffusion outputs or second-stage fine-tuning for texture modeling, remaining distinct from ImageDream.
3. Methodology
ImageDream extends MVDream’s multi-view diffusion and score-distillation pipeline with canonical camera coordination and image-prompt controllers. Its local-and-pixel control strategy preserves image appearance while supporting consistent multi-view fusion, with score-distillation refinements addressing optimization issues.
- MVDream Pipeline: MVDream trains a multi-view diffusion network for four orthogonal, consistent views, then uses MV-SDS to optimize a detailed 3D NeRF.The diffusion model is trained jointly on rendered 3D data and LAION5B text-to-image data, and its loss operates on noisy multi-view latents.
- Canonical Camera: Canonical camera coordination maps the default camera to the centered front view, simplifying correspondence between input-image variations and 3D geometry.ImageDream trains on random-view input images while regressing toward canonical multi-view images using MVDream’s camera embeddings.
- Multi-level Controllers: ImageDream adds image conditioning through multi-level controllers that inject adapted global, local, and pixel features into the multi-view diffusion network.The global controller uses image cross-attention, while local and pixel controllers provide more detailed structural and appearance information.
- Multi-level Controllers: Pixel conditioning concatenates the input image with four generated views for joint 3D self-attention, while the input latent remains unnoised and receives a zero camera embedding.This design helps retain input appearance and multi-view consistency during subsequent 3D model fusion.
- Multi-level Controllers: The final controller combines local and pixel control because pixel features are considered significantly beneficial for rapidly training robust multi-view diffusion.The global controller is omitted from the combined design because it was judged to add limited information.
- Image-Prompt Score Distillation: Image-prompt score distillation modifies NeRF optimization within the MVDream framework, including background alignment to reduce conflicts between rendered backgrounds and diffusion conditioning.The method acknowledges that mismatched input-image camera parameters can make the resulting 3D shape unpredictable.
4. Experiments
ImageDream is evaluated on diverse image prompts against several state-of-the-art baselines using user studies and quantitative image-quality analyses. Results favor ImageDream, while exposing trade-offs from stringent image constraints and limitations of current metrics.
- Experimental Setup: The evaluation uses 39 curated prompts spanning objects with relatively complex geometries and appearances, generated outside the Objaverse distribution.Images were generated with SDXL, background-removed, and re-centered before testing.
- Experimental Setup: ImageDream is compared with Zero123-XL, Magic123, and SyncDreamer using geometry quality and similarity to the image prompt.Geometry quality measures plausible shape and minimal artifacts, while image-prompt similarity measures resemblance to the input image.
- Qualitative Evaluation: In a double-blind user study, ImageDream-P and ImageDream-G significantly outperform other baselines, with ImageDream-P receiving the strongest preference.SyncDreamer was omitted from the figure because its NeuS results received 0% preference.
- Qualitative Evaluation: Qualitative comparisons show ImageDream producing more accurate geometry than single-view diffusion baselines, which struggle to represent complete object structure.For example, Magic123 and Zero123 fail to accurately represent the span of a horse body in the cited case.
- Numerical Evaluation: ImageDream maintains high image quality after diffusion and 3D fusion, while its local controller achieves better post-fusion image CLIP scores through richer image features.The pixel-controller model performs best on image CLIP scores during both stages, whereas other scores vary across models.
- Limitations: Image-quality scores do not fully capture multi-view consistency or geometric correctness, so more comprehensive evaluation metrics remain necessary.Zero123-XL can achieve high IS despite poorer consistency.
- Limitations: The pixel controller can struggle with small facial details when image constraints are overly stringent, whereas global control may recover them from the text prompt.The authors suggest better estimation of image properties or improved controller balancing as possible remedies.
5. Conclusion
ImageDream combines multi-view diffusion with canonical camera coordination and multi-level image-prompt controllers to address geometric inaccuracies in 3D generation. The authors identify increased training-time image-prompt randomness as a future direction for reducing texture blurriness.
- 5. Conclusion: ImageDream uses multi-view diffusion, canonical camera coordination, and multi-level image-prompt controllers to improve control and address geometric inaccuracies.The conclusion frames these components as the model’s central design choices.
- 5. Conclusion: Increasing randomness in image-prompts during training is proposed as a future improvement for reducing texture blurriness in generated models.The conclusion presents this as a future direction rather than a demonstrated result.