PaperlayerView exact revision on arXiv

Source-linked AI summary

MVDiffusion++: A Dense High-resolution Multi-view Diffusion Model for Single or Sparse-view 3D Object Reconstruction

Shitao Tang, Jiacheng Chen, Dilin Wang, Chengzhou Tang, Fuyang Zhang, Yuchen Fan, Vikas Chandra, Yasutaka Furukawa, Rakesh Ranjan

arXiv:2402.12712v3cs.CV

TL;DR

Conventional reconstruction uses many images and precise camera parameters, while this paper targets object reconstruction from one or a few unposed images. MVDiffusion++ generates dense, high-resolution views using pose-free attention and view dropout; it reports state-of-the-art results across novel view synthesis and single- and sparse-view reconstruction.

  • Problem

    Conventional 3D reconstruction takes hundreds of images and estimates precise camera parameters, while the paper targets reconstruction from one or a few images without poses.

    Source: §1 · pp. 1–2

  • Method

    MVDiffusion++ generates dense, high-resolution views from one or a few unposed images using pose-free multi-view diffusion and view dropout.

    Source: §1 · p. 2, §4.1 · p. 5, §4.2 · p. 8

  • Results

    MVDiffusion++ achieves state-of-the-art results in novel view synthesis and single- and sparse-view reconstruction, improving sparse-view PSNR by 8.19 over LEAP [12].

    Source: §1 · p. 3

  • Takeaways & Limitations

    The approach enables users to achieve high-quality 3D models from just a few object snapshots.

    Source: §6.2 · p. 15

    The method struggles with thin structures and can generate implausible images for views occluded in the inputs, shortcomings the authors attribute predominantly to limited training data.

    Source: §7 · p. 15

Abstract

from arXiv · show

This paper presents a neural architecture MVDiffusion++ for 3D object reconstruction that synthesizes dense and high-resolution views of an object given one or a few images without camera poses. MVDiffusion++ achieves superior flexibility and scalability with two surprisingly simple ideas: 1) A ``pose-free architecture'' where standard self-attention among 2D latent features learns 3D consistency across an arbitrary number of conditional and generation views without explicitly using camera pose information; and 2) A ``view dropout strategy'' that discards a substantial number of output views during training, which reduces the training-time memory footprint and enables dense and high-resolution view synthesis at test time. We use the Objaverse for training and the Google Scanned Objects for evaluation with standard novel view synthesis and 3D reconstruction metrics, where MVDiffusion++ significantly outperforms the current state of the arts. We also demonstrate a text-to-3D application example by combining MVDiffusion++ with a text-to-image generative model. The project page is at https://mvdiffusion-plusplus.github.io.

1 Introduction

Conventional 3D reconstruction relies on hundreds of images and precise camera parameters, unlike human vision’s ability to form coherent 3D representations from a few views. MVDiffusion++ generates dense, high-resolution views from unposed images using pose-free attention and view dropout, and reports state-of-the-art results across three reconstruction tasks.

1 Introduction

  • Conventional 3D reconstruction takes hundreds of images, estimates precise camera parameters, and reconstructs high-fidelity geometry at sub-millimeter accuracy.The introduction contrasts this workflow with human vision’s ability to form coherent 3D representations from a few images, including obscured parts.Source: §1 · pp. 1–2
  • MVDiffusion++ generates dense (32), high-resolution (512×512) object images from single or sparse unposed inputs.The method targets reconstruction without camera poses, which are difficult to estimate when visual overlap is minimal or absent.Source: §1 · p. 2
  • Self-attention among 2D features learns 3D consistency without camera poses, while view dropout randomly excludes generation views during training.The authors say view dropout enables high-resolution training and that the model generates dense views simultaneously at test time.Source: §1 · p. 3
  • MVDiffusion++ achieves state-of-the-art performance in novel view synthesis, single-view reconstruction, and sparse-view reconstruction.On Google Scanned Objects, single-view reconstruction reaches 0.6973 IoU and 0.0165 Chamfer distance; sparse-view novel view synthesis improves PSNR by 8.19 over LEAP [12].Source: §1 · p. 3

2 Related work

Prior multi-view generation methods face constraints in image correspondences, output density and resolution, or computational cost. The related work describes MVDiffusion++ as generating dense, high-resolution views from arbitrary image counts and using diffusion-generated views for sparse-view reconstruction.

2 Related work

  • Prior multi-view methods tend to produce sparse, low-resolution images because their attention mechanisms are computationally intensive; MVDiffusion++ targets dense, high-resolution views from arbitrary image counts.MVDiffusion [33] is constrained to one-to-one image correspondences, while several approaches use 3D self-attention.Source: §2 · p. 3
  • Single-view reconstruction methods include triplane prediction, cost-volume-based multi-view generation, and image generation followed by conventional 3D recovery.The cited approaches face resolution limits from 3D volumes or substantial GPU memory and self-attention demands.Source: §2 · p. 3
  • Sparse-view reconstruction typically uses two to ten images, where minimal overlap makes camera-pose estimation difficult for conventional reconstruction methods.FvOR jointly optimizes poses and shapes, while LEAP [12] and PF-LRM [38] use pose-free approaches but lack generative priors and have low-resolution outputs.Source: §2 · p. 4
  • For sparse-view reconstruction, the paper describes generating high-resolution multi-view images with diffusion and recovering a mesh with NeuS.The related-work passage also describes view dropout as sampling a limited number of views per training iteration to support variable-number, high-resolution generation.Source: §2 · p. 4

3 Preliminary: Multi-view latent diffusion models

MVDiffusion is a multi-view latent diffusion model that generates images when pixel-wise correspondences are available. Its panorama setup denoises multiple latent images together, using correspondence-aware attention across overlapping views alongside standard UNet modules.

3 Preliminary: Multi-view latent diffusion models

  • MVDiffusion generates multiple images from text or an image when pixel-wise correspondences are available across views.The paper presents it as the foundation for the proposed approach.Source: §3 · p. 4
  • For an eight-view panorama, eight latent diffusion models denoise their latent images simultaneously through a UNet feature pyramid.A CNN initializes each image’s features from its noisy latent at the first block.Source: §3 · p. 4
  • Correspondence-aware attention enforces consistency between neighboring views with visual overlap, alongside self-attention, conditioning cross-attention, and CNN layers.At test time, a DDPM sampler updates all noisy latents using the predicted noise.Source: §3 · p. 4
  • The training objective sums squared noise-prediction errors across the views.The cited equation defines the loss over the latent images and Gaussian noise terms.Source: §3 · p. 4
  • The MVDiffusion++ target comprises 32 views arranged across eight azimuths and four elevation levels, with view dropout training on the remaining views.The figure caption distinguishes dropped views from those the model is trained to denoise.Source: Figure 2 · p. 5

4 MVDiffusion++

MVDiffusion++ targets flexible, high-resolution 3D reconstruction from unposed single or sparse views. It combines pose-free multi-view diffusion with view dropout to learn consistency across views and scale generation to dense outputs.

4 MVDiffusion++

  • MVDiffusion++ uses self-attention over 2D latent features to learn 3D consistency without camera poses or image projection formulas.View dropout addresses the training scalability challenge of applying global attention across 32 generation and 10 condition views.Source: §4 · p. 5, §4.3 · p. 8

4.1 Task: Input condition images and output target images

  • The task generates 32 images at 512×512 resolution on a uniform spherical grid, conditioned on up to 10 input images without camera poses.Input and output views use a 60° horizontal and vertical field of view; the output grid has eight azimuth and four elevation angles.Source: §4.1 · p. 5

4.2 Pose-free multi-view conditional diffusion model

  • The condition and generation branches share an architecture and weights, while global self-attention learns consistency across views and cross-attention injects condition-image CLIP embeddings.The denoising UNet predicts noise for a DDPM sampler; the model is trained first with ϵ-prediction and then v-prediction.Source: §4.2 · pp. 6–8

4.3 View dropout training strategy

  • View dropout randomly removes 24 of 32 generation views per training iteration, reducing memory use while retaining full 32-view generation at test time.The full 32 generation and 10 condition views yield more than 130k UNet tokens, making global self-attention infeasible during training.Source: §4.3 · p. 8

5 Remaining system details

The system details describe training and test data preparation, mesh extraction, mask-aware VAE fine-tuning, and a three-stage training procedure.

5.1 Training data preparation

  • Training uses 180k Objaverse models selected from 800k for aesthetic scores of at least 5, with randomized condition-view elevation and camera distance.Objects are centered and scaled so their longest dimension spans [−1, 1].Source: §5.1 · p. 8

5.2 Testing data preparation

  • Single-view tests use 30 GSO objects and eight fixed views; sparse-view tests use 32 targets and vary the number of inputs across 1, 2, 4, 8, and 10.Single-view inputs are selected by visual plausibility; sparse-view inputs are rendered at random azimuth and elevation angles.Source: §5.2 · pp. 8–9

5.3 Mesh extraction from generated images

  • After generating 32 images, grid-based NeuS reconstructs a mesh using foreground masks and additional normal supervision from an Omnidata estimator.The reported reconstruction time is around three minutes on a single Nvidia 2080 Ti.Source: §5.3 · p. 9

5.4 Mask-aware VAE pre-fine-tuning

  • PSNR rises from 36.6 to 41.2 after fine-tuning the mask-aware VAE on object images.The fine-tuning uses approximately 3 million rendered Objaverse RGBA images for 60,000 iterations.Source: §5.4 · p. 9

5.5 Three-stage training strategy

  • Training proceeds from single-view ϵ-prediction to single-view v-prediction, then to v-prediction with both single- and sparse-view conditioning.In the final stage, half the samples use single-view conditioning and half use sparse-view conditioning with 2–10 inputs.Source: §5.5 · p. 10

6 Experiments

The experiments evaluate single-view and sparse-view reconstruction and a text-to-3D application. MVDiffusion++ outperforms competing methods in the reported single-view and sparse-view comparisons, with qualitative examples illustrating its generated images and reconstructed meshes.

6 Experiments

  • Training used batch size 1024 on 128 Nvidia H100 GPUs for about a week; sampling 32 views took 181 seconds.At test time, the model uses a 75-step DDPM sampler.Source: §6 · p. 10

6.1 Single-view object modeling

  • MVDiffusion++ consistently outperforms competing methods in single-view evaluations, though the comparison with Wonder3D is not completely fair because it assumes orthographic projections.The experiments use perspective images for Wonder3D.Source: §6.1 · p. 12
  • For single-view object modeling, our 512×512 renderings show clock numbers clearly (row 3), while Wonder3D and SyncDreamer’s 256×256 images show blurry numbers, with ours offering higher fidelity and richer details.Source: Figure 5 · p. 11, §6.1 · p. 12
  • MVDiffusion++ reconstructs a plausible, detailed turtle mesh where Wonder3D and Open-LRM fail to recognize the turtle and show artifacts.The meshes are exported from 32 generated images at 512×512 resolution.Source: Figure 6 · p. 12, §6.1 · p. 13

6.2 Sparse-view object modeling

  • MVDiffusion++ generates higher-quality images than LEAP and reconstructs meshes that outperform NeuS in most settings, despite NeuS receiving ground-truth camera poses.The authors attribute the image-quality difference to image priors inherited from pre-trained latent diffusion models.Source: §6.2 · p. 15

6.3 Text-to-3D application

  • The text-to-3D examples combine a text-to-image model that supplies an input image with MVDiffusion++ reconstruction, and include failure examples.The authors report consistent performance with minimal errors on GSO despite training solely on Objaverse.Source: Figure 8 · p. 14, §6.3 · p. 15

7 Limitations and future challenges

MVDiffusion++ reconstructs objects from an arbitrary number of images and produces dense, consistent views. Its reported limitations include failures on thin structures and implausible images for views occluded in the inputs.

7 Limitations and future challenges

  • MVDiffusion++ processes an arbitrary number of conditional images to produce dense, consistent views from fixed perspectives, supporting single-view and sparse-view reconstruction.The paper reports new-standard performance for both reconstruction settings.Source: §7 · p. 15
  • The method struggles with thin structures such as cables and can generate implausible images for views occluded in the inputs, which the authors attribute mainly to limited training data.The authors propose expanding the framework to videos as future work.Source: §7 · p. 15

Related papers

Loading 2402.12712v3…