Source-linked AI summary
SyncDreamer: Generating Multiview-consistent Images from a Single-view Image
Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjie Liu, Taku Komura, Wenping Wang
TL;DR
Single-view images provide limited 3D information, while generated novel views can lack geometric and appearance consistency. SyncDreamer uses synchronized multiview diffusion with shared cross-view attention to generate consistent images, improving reconstruction quality and generalizing across input styles. Its main scope limitation is the smaller fine-tuning batch size imposed by GPU memory constraints.
Problem
Generating multiview-consistent images from a single-view object image remains difficult because the image contains limited 3D information.
Method
SyncDreamer models the joint distribution of multiview images with synchronized noise predictors that share information through attention during each denoising step.
Results
SyncDreamer generates strongly consistent multiview images, improves reconstruction quality over baseline methods, and generalizes to varied input styles.
Takeaways & Limitations
The generated views can support 3D reconstruction with vanilla NeRF or NeuS without special losses and can produce multiple reasonable objects from one image.
Takeaways & Limitations
GPU memory constraints limited SyncDreamer's fine-tuning batch size to 192, compared with 1536 for Zero123.
Abstract
from arXiv · showhide
In this paper, we present a novel diffusion model called that generates multiview-consistent images from a single-view image. Using pretrained large-scale 2D diffusion models, recent work Zero123 demonstrates the ability to generate plausible novel views from a single-view image of an object. However, maintaining consistency in geometry and colors for the generated images remains a challenge. To address this issue, we propose a synchronized multiview diffusion model that models the joint probability distribution of multiview images, enabling the generation of multiview-consistent images in a single reverse process. SyncDreamer synchronizes the intermediate states of all the generated images at every step of the reverse process through a 3D-aware feature attention mechanism that correlates the corresponding features across different views. Experiments show that SyncDreamer generates images with high consistency across different views, thus making it well-suited for various 3D generation tasks such as novel-view-synthesis, text-to-3D, and image-to-3D.
1 INTRODUCTION
Single-view images contain limited 3D information, making multiview-consistent generation difficult. SyncDreamer addresses this by synchronizing multiview diffusion and demonstrates consistent views, improved reconstruction, and broad input-style support.
- Limited 3D information makes generating multiview-consistent images from a single-view object image challenging.
- Maintaining consistency across views is the central challenge when 2D diffusion models generate images for 3D reconstruction.
- SyncDreamer models the joint distribution of multiview images using synchronized noise predictors that share information through attention during every denoising step.
- Consistent geometry and appearance allow vanilla NeRF or NeuS reconstruction without special losses, while preserving multiple plausible reconstructions from one input.
- SyncDreamer generates more consistent images and reconstructs better shapes than baseline methods on the Google Scanned Object dataset.
- The method supports cartoons, sketches, ink paintings, and oil paintings for consistent-view generation and 3D reconstruction.
2 RELATED WORK
Prior single-view reconstruction methods include diffusion-based multiview generation, distillation, regression, retrieval, and NeRF-GAN approaches. These methods face challenges including insufficient 3D data, limited category generalization, and multiview inconsistency.
- Distillation methods use high-quality 2D diffusion models for 3D tasks, including text-to-3D shape generation.
- Other methods directly apply 2D diffusion models to generate multiview images for 3D reconstruction, but maintaining consistency across views remains challenging.
- Earlier single-view reconstruction methods based on regression or retrieval have difficulty generalizing to new categories.
- NeRF-GAN methods achieve strong results for specific categories but fail to generalize reliably to arbitrary objects.
3 METHOD
SyncDreamer extends diffusion to jointly generate multiple views, synchronizing their denoising states through shared multiview features. Its 3D-aware attention links corresponding features across views while retaining a Zero123-initialized UNet backbone.
- Multiview diffusion: The model jointly generates N target views by learning a multiview diffusion distribution conditioned on the input view.Each view has a synchronized noise predictor, and the reverse process denoises all views while exchanging information across their states.
- Multiview diffusion: The forward process independently adds noise to every target view, while the reverse process predicts each view’s noise from all current multiview states.The shared formulation preserves per-view noise prediction while conditioning denoising on the complete set of target-view states.
- 3D-aware feature attention: The shared UNet receives the input view, current noisy target view, and viewpoint difference, then applies depth-wise attention before denoising.The backbone is initialized from pretrained Zero123 weights to support novel-view generation from a single image.
- 3D-aware feature attention: 3D-aware feature attention constructs a spatial volume from noisy target views and samples a view-frustum volume for the current target view.Features are extracted from each target view, concatenated into a spatial feature volume, and processed with a 3D CNN before depth-wise attention in the UNet.
- 3D-aware feature attention: The shared spatial volume imposes a global same-object constraint, while depth-only attention imposes a local epipolar-line constraint.These design choices respectively connect all target views globally and correlate features along the relevant depth dimension.
4 EXPERIMENTS
Experiments evaluate SyncDreamer on novel-view synthesis, single-view reconstruction, generalization, and ablations. The results indicate improved multiview consistency and reconstruction quality, while design choices affect generalization and geometry.
- Consistent novel-view synthesis: SyncDreamer generates images that are semantically consistent with the input while preserving multiview color and geometry consistency.The same input can also produce different plausible instances with different random seeds.
- Single-view reconstruction: SyncDreamer achieves the best reconstruction quality among the compared methods, with smooth surfaces and detailed geometry.The comparison includes Point-E, Shap-E, Zero123, Magic123, and One-2-3-45.
- Generalization ability: SyncDreamer produces reasonable 3D geometry and multiview-consistent images from sketches, cartoons, and traditional Chinese ink paintings.These inputs differ substantially from real-world images in lighting and shadow effects.
- Ablation studies: Removing 3D-aware feature attention leaves the model unable to produce strongly consistent images, supporting the module’s necessity.This ablation corresponds to fine-tuning Zero123 on fixed viewpoints.
- Ablation studies: Initializing from Stable Diffusion yields worse generalization than initializing from Zero123.The authors associate Zero123 initialization with learned 3D priors and note batch size as important for learning 3D priors from Objaverse.
- Ablation studies: Jointly training the UNet tends to predict thin-plate objects, especially for 2D hand drawings.The authors speculate that overfitting to thin-plate objects and fixed training viewpoints causes this behavior.
5 CONCLUSION
SyncDreamer uses synchronized multiview diffusion with a Zero123 backbone and volume condition module to generate consistent views from one image. Experiments report improved consistency, reconstruction quality, and generalization across input styles.
- 5 CONCLUSION: SyncDreamer models the joint probability distribution of multiview images through synchronized multiview diffusion.Its architecture uses Zero123 as the backbone and a volume condition module to model cross-view dependency.
- 5 CONCLUSION: SyncDreamer generates multiview images with strong consistency and improves reconstruction quality compared with baseline methods.The conclusion also reports generalization to various input styles.
A.1 IMPLEMENTATION DETAILS
The implementation trains SyncDreamer on Objaverse with 16 target viewpoints and uses fixed spatial and view-frustum volume resolutions.
- A.1 IMPLEMENTATION DETAILS: The spatial volume is sized 32^3, while the view-frustum volume is sized 32 × 32 × 48 with 48 depth planes.The resolutions are chosen to match the 32×32 latent feature map of 256×256 Stable Diffusion images and diagonal viewing directions.
A.2 TEXT-TO-IMAGE-TO-3D
SyncDreamer can combine text-to-image generation with multiview reconstruction, allowing users to select a desirable generated image before producing a 3D model.
- SyncDreamer supports generating 3D models from text when combined with text-to-image models such as Stable Diffusion or Imagen.Examples are shown in Fig. 8.
- SyncDreamer’s text-to-3D workflow provides more flexibility than existing text-to-3D distillation methods.
- SyncDreamer outputs are not always desirable because diffusion randomness can produce low-quality failure instances.Figure 9 illustrates this generation-quality limitation.
- Users can generate multiple images from text and select a desirable one as SyncDreamer’s input for 3D reconstruction.
A.3 LIMITATIONS AND FUTURE WORKS
SyncDreamer has fixed viewpoint and perspective assumptions, can produce implausible or blurry outputs, and may generate less detailed textures than Zero123.
- SyncDreamer generates fixed viewpoints, limiting applications that require other viewpoints.NeuS can render additional viewpoints, but the results are somewhat blurry.
- The generated images are not always plausible, so multiple random-seed instances may be needed to select a desirable reconstruction.The paper attributes this issue to the stochastic diffusion process.
- Generation quality is sensitive to foreground object size because it changes the input camera’s perspective pattern relative to the fixed training setup.
- The current implementation assumes perspective inputs, so orthogonally projected drawings can cause unnatural distortion in reconstructed geometry.The paper suggests using orthogonal projection during volume construction to alleviate this issue.
- Generated textures can be less detailed than Zero123 because enforcing consistency across all views may favor larger, less detailed texture blocks.
A.4 DISCUSSION ON DEPTH-WISE ATTENTION LAYERS
Depth-wise attention layers are important to SyncDreamer’s multiview image quality: removing them produces degenerated images with undesirable shape distortions.
- Removing depth-wise attention layers produces degenerated images with undesirable shape distortions.The alternative model flattens the view-frustum feature volume into a 2D feature map and applies 2D convolutions.
- Depth-wise attention layers improve the quality of generated multiview-consistent images.
A.5 GENERATING IMAGE ON OTHER VIEWPOINTS
SyncDreamer can be adapted to other fixed viewpoint arrangements and its outputs can support arbitrary-view rendering through NeuS, though rendering quality and reconstruction quality depend on view count and representation.
- Generating images on other viewpoints: A SyncDreamer model trained on 16 evenly azimuth-spaced viewpoints with 0° elevation generates images for that viewpoint arrangement.The azimuths span [0°, 360°].
- Regenerating novel views: SyncDreamer can regenerate novel views from one generated image, producing plausible images that reasonably differ from the original input view.Figure 12 compares original generations with regenerated views.
- NeuS novel-view rendering: NeuS trained on SyncDreamer outputs can render images from arbitrary viewpoints, but its renderings are blurrier than the generated images.The paper attributes this limitation to training NeuS with only 16 generated images.
- Effect of view count: Reducing generated views from 16 to 8 does not significantly affect overall NeuS reconstruction quality, whereas using 4 views sharply degrades surface and novel-view quality.This suggests an 8-view SyncDreamer variant could improve efficiency without compromising reconstruction quality.
- NeuS implementation: Hash-grid NeuS takes about 3 minutes, less than half the 10-minute runtime of vanilla MLP NeuS, but usually produces noisier surfaces.Additional normal-smoothness losses are added for the hash-grid method.
- Generation randomness: Generation randomness can change PSNR, SSIM, LPIPS, Chamfer Distance, and Volume IOU across instances from the same input image.Table 3 reports minimum, maximum, and average values from four generated instances.
- Attention mechanism comparison: SyncDreamer’s depth-wise attention is closely related to epipolar attention, but both require a tensor of size H × W × D when geometry is unknown.The paper notes that this volume-based representation has comparable GPU-memory cost to epipolar-line attention.
- Multiview diffusion formulation: The proposed multiview diffusion formulation is visualized through a diagram deriving its forward and reverse processes.