PaperlayerView exact revision on arXiv

Source-linked AI summary

SparseFusion: Distilling View-conditioned Diffusion for 3D Reconstruction

Zhizhuo Zhou, Shubham Tulsiani

arXiv:2212.00792v3cs.CVcs.GR

TL;DR

Sparse-view reconstruction must infer plausible unseen content while preserving consistency across views, but existing approaches typically trade off probabilistic image generation against a consistent underlying 3D. SparseFusion distills a view-conditioned latent diffusion model into an instance-specific 3D representation, and on 51 CO3D categories it outperforms existing methods across distortion and perception metrics for sparse-view novel-view synthesis.

  • Problem

    Sparse-view reconstruction must handle uncertain unseen regions and large viewpoint changes while producing a consistent underlying 3D representation.

  • Method

    SparseFusion uses geometry-informed view-conditioned latent diffusion and distills its predicted image distributions into an instance-specific neural 3D representation through likelihood-based mode seeking.

  • Results

    SparseFusion outperforms existing methods across distortion and perception metrics for sparse-view novel-view synthesis on 51 CO3D categories.

  • Takeaways & Limitations

    The method recovers 3D-consistent representations with plausible, realistic outputs in unobserved regions from as few as 2 input images.

  • Takeaways & Limitations

    The approach relies on known relative camera poses and requires computationally expensive instance-specific neural-field optimization.

Abstract

from arXiv · show

We propose SparseFusion, a sparse view 3D reconstruction approach that unifies recent advances in neural rendering and probabilistic image generation. Existing approaches typically build on neural rendering with re-projected features but fail to generate unseen regions or handle uncertainty under large viewpoint changes. Alternate methods treat this as a (probabilistic) 2D synthesis task, and while they can generate plausible 2D images, they do not infer a consistent underlying 3D. However, we find that this trade-off between 3D consistency and probabilistic image generation does not need to exist. In fact, we show that geometric consistency and generative inference can be complementary in a mode-seeking behavior. By distilling a 3D consistent scene representation from a view-conditioned latent diffusion model, we are able to recover a plausible 3D representation whose renderings are both accurate and realistic. We evaluate our approach across 51 categories in the CO3D dataset and show that it outperforms existing methods, in both distortion and perception metrics, for sparse-view novel view synthesis.

1. Introduction

SparseFusion addresses sparse-view 3D reconstruction by combining probabilistic novel-view generation with a consistent 3D representation. It distills view-conditioned diffusion predictions into a likely 3D mode for accurate and realistic renderings.

  • Sparse-view methods can produce blurry predictions under large viewpoint changes and cannot plausibly model uncertain unseen regions.Independent pixelwise or pointwise predictions cannot represent alternatives such as an unobserved teddy bear nose being red or black.
  • Probabilistic novel-view synthesis generates detailed plausible images but independently sampled views may be inconsistent with one underlying 3D object.Different query views can assign conflicting appearances to an unobserved region.
  • SparseFusion models possible query-view images conditioned on context views and geometry-informed features, then distills them into an instance-specific 3D representation.The diffusion model approximately provides likelihoods for generated views, enabling optimization of the 3D representation through its renderings.
  • Diffusion-based likelihood optimization performs mode-seeking inference, producing accurate and realistic renderings while recovering a 3D-consistent object representation.The approach is demonstrated on more than 50 real-world CO3D categories using as few as 2 input images.

2. Related Work

Related work spans instance-specific reconstruction, single-view 3D prediction, generalizable novel-view synthesis, and diffusion-based generation. SparseFusion combines geometry-aware view synthesis with diffusion-guided extraction of a consistent 3D representation.

  • Instance-specific Reconstruction from Multiple Views: Classical and neural multi-view reconstruction methods recover 3D from multiple views but generally require many input views, limiting real-world practicality.Some later methods reduce the required views, but the supplied passage does not describe their full capabilities.
  • Single-view 3D Reconstruction: Single-view 3D methods predict volumetric, mesh-based, or neural implicit representations, but one image fundamentally limits recoverable detail.These methods also do not prioritize novel-view synthesis as their goal.
  • Generalizable View Synthesis from Fewer Views: Feature re-projection methods provide 3D-consistent views but regress toward the mean and fail to produce probabilistic variation for unseen content.This motivates combining re-projected geometric features with a generative model.
  • Diffusion Models: Diffusion models have been used for probabilistic novel-view synthesis, while SparseFusion uses geometry-aware conditioning and diffusion distillation to extract 3D modes.The approach is inspired by optimizing 3D scenes with text-conditioned diffusion models.
  • Concurrent Works: Concurrent approaches differ in whether they infer a 3D representation or operate in 3D diffusion space, whereas SparseFusion uses 2D diffusion to optimize a 3D representation.The supplied comparison identifies 3DiM as lacking a 3D representation and RenderDiffusion as learning diffusion directly in 3D space.

3. Background: Denoising Diffusion

Denoising diffusion models learn to remove time-dependent noise from data, with training objectives connected to a variational log-likelihood bound. Conditioning extends this framework to model distributions given additional inputs.

  • Denoising diffusion training adds time-dependent noise to a data point and trains a network to predict that noise from the noisy point.The objective optimizes a variational lower bound on the observed-data log-likelihood.
  • The schedule parameter ᾱ_t controls the learning schedule, while weights w_t depend on that schedule and are often simplified to 1.These weights determine how diffusion timesteps contribute to the objective.
  • The noise-prediction objective can also be interpreted as a reconstruction error for recovering the original input from a noisy version.This interpretation connects denoising optimization to reconstructing clean data.
  • Conditional diffusion models infer distributions p(x|y) by providing the conditioning variable y as an additional input to the noise-prediction network.SparseFusion applies this idea to view-conditioned image generation.

4. Approach

SparseFusion combines geometry-guided probabilistic view synthesis with diffusion distillation to infer a consistent 3D representation from sparse views. It uses neural mode seeking to preserve plausible, detailed renderings while addressing uncertainty and inconsistency in independently sampled novel views.

  • Problem: Given typically 2–3 posed images, the method targets a 3D representation that synthesizes novel views while capturing geometric structure.Sparse observations leave object regions and geometry uncertain, making direct predictions blurry.
  • Geometry-guided probabilistic view synthesis: The geometry-guided view-synthesis model extracts query-view features with EFT and conditions a latent diffusion model on them to represent plausible novel-view images.EFT produces geometry-aware features from epipolar lines in the context images; the diffusion model operates in latent space.
  • Geometry-guided probabilistic view synthesis: VLDM sampling produces multiple plausible predictions for unseen regions, but these independently sampled views lack a single consistent underlying 3D interpretation.The probabilistic model captures uncertainty and improves image quality, while its outputs alone do not constitute a 3D representation.
  • Extracting 3D modes via diffusion distillation: Diffusion distillation optimizes an instance-specific neural scene representation so renderings from sampled cameras have high likelihood under the conditional diffusion distribution.This reframes reconstruction as neural mode seeking rather than minimizing distance to sampled images.
  • Extracting 3D modes via diffusion distillation: The final objective uses pixel-space and perceptual reconstruction losses with multi-step denoising, reaching up to 50 denoising steps.The method reconstructs denoised outputs from noisy rendered-image latents and ignores their indirect dependence on the neural representation when computing parameter gradients.

5. Experiments

SparseFusion is evaluated on CO3Dv2 across sparse-view settings, where it combines probabilistic view synthesis with diffusion distillation to produce plausible, geometrically consistent renderings. It performs especially strongly under sparse inputs and larger viewpoint changes, while ablations support mode-seeking distillation and multi-step denoising.

  • Experimental Setup: SparseFusion is evaluated on CO3Dv2 across 51 diverse object categories, with comparisons against prior methods and analyses of diffusion distillation and mode seeking.The core subset uses 10 categories, while the all-category evaluation covers all 51 categories.
  • Core Subset: 2-view: SparseFusion outperforms all other methods in LPIPS and loses PSNR in only 3 categories, while achieving higher PSNR in 7 categories.The qualitative results show that it combines perceptually reasonable views with geometric consistency.
  • Core Subset: Varying Views: SparseFusion outperforms current state-of-the-art methods in PSNR, SSIM, and LPIPS for 2-, 3-, and 6-view reconstruction.Feature re-projection methods improve as more views are added, and EFT exceeds SparseFusion in PSNR for the 3-view and 6-view settings, but SparseFusion remains better in LPIPS.
  • All Categories: 2-views: Across all 51 categories, EFT leads in PSNR, while SparseFusion achieves better SSIM and LPIPS than EFT and substantially outperforms NerFormer and ViewFormer.The evaluation uses two random input views and reports averaged metrics.
  • Performance Binned by Viewpoint Changes: Across larger viewpoint changes, feature re-projection methods decline quickly, whereas SparseFusion remains more robust and relatively better.For small viewpoint changes, SparseFusion performs better in LPIPS and competitively in PSNR against EFT.
  • Additional Analysis: Optimizing Instant NGP with diffusion distillation improves all metrics, unlike fitting VLDM predictions directly, whose perceptual quality decreases under mean-seeking optimization.Fitting EFT renderings produces no significant change because EFT predicts consistent mean outputs.
  • Ablating Distillation Objective: Pixel-space loss with perceptual loss improves SSIM and LPIPS, and multi-step denoising gives the best perceptual results despite smoother, less realistic textures from single-step denoising.PSNR remains relatively similar across the distillation variants.

6. Discussion

SparseFusion infers 3D neural representations from sparse views and produces plausible, realistic outputs in unobserved regions. The approach still depends on known relative camera poses and computationally expensive instance-specific neural-field optimization.

  • Discussion: SparseFusion predicts 3D-consistent representations with plausible and realistic outputs, including in unobserved regions.The paper presents this as an approach for inferring 3D neural representations from sparse-view observations.
  • Discussion: Known relative camera poses remain a key limitation, because recovering them generally is still challenging.The authors note recent promising advances but retain this reliance as a limitation.
  • Discussion: Instance-specific neural-field optimization is computationally expensive.The discussion identifies this requirement as an additional challenge.

7. Ethics and Broader Impact

SparseFusion’s broader-impact concerns center on computational cost and bias inherited from view-conditioned latent diffusion models trained on multi-view data. These concerns become more consequential when adapting the method from static objects to humans or animals.

  • Computational Impact: SparseFusion is more computationally expensive than existing novel view synthesis methods, creating hardware limitations and potentially increasing carbon emissions.The paper explicitly connects the computational burden to downstream-task constraints.
  • Bias and Generalization: View-conditioned latent diffusion models may learn harmful biases from their multi-view training data and propagate them into reconstructed 3D scenes.The authors state that the current static-object use case does not present ethical concerns, but broader adaptations require examination.

B. Implementation Details

The paper provides detailed implementation and training details for all components of SparseFusion.

  • Implementation Details: Detailed implementation and training details are provided for all components of SparseFusion.

B.1. Epipolar Feature Transformer

The Epipolar Feature Transformer aggregates geometry-aware features sampled along query rays across input views and depths, producing both query-view colors and pixel-aligned features.

  • EFT gathers features along input-image epipolar lines and aggregates them through transformers for sparse-view processing.
  • Given input images with known poses and a query pose, EFT outputs a query-view RGB image and feature grid.
  • For each query ray, EFT projects 20 points at linearly spaced depths and uses their multiview projections as inputs.
  • Three transformers aggregate epipolar features to predict an RGB pixel color and a 256-dimension feature.
  • The transformer inputs combine sampled features with ray, depth, and connecting-ray encodings, while weighted averaging aggregates sequence elements.

B.2. View-conditioned Diffusion Model

The view-conditioned diffusion model is a latent diffusion system conditioned on EFT's pixel-aligned features to model novel-view images.

  • The model conditions a latent diffusion process on a pixel-aligned feature grid produced for the query view.
  • It uses the Stable Diffusion VAE with frozen v1-3 weights, encoding 256 × 256 RGB images into 32 × 32 × 4 latents.
  • The denoising network is a 400M-parameter UNet with four down-sampling blocks, a middle block, and four up-sampling blocks.
  • Instead of text conditioning and cross attention, the UNet concatenates EFT features with image latents for the 32 × 32 ray patch centers.
  • Training jointly optimizes UNet and EFT weights over randomly selected sets of 2–5 input views using diffusion and pixel-wise reconstruction losses.

B.3. Diffusion Distillation

Diffusion distillation optimizes an instance-specific Instant NGP scene so its renderings align with both the feed-forward color prediction and diffusion-refined images across sampled cameras.

  • The method optimizes an Instant NGP neural scene representation using a view-conditioned diffusion model and an instance-specific camera distribution.
  • Instant NGP uses scene bounds of 4, hashgrid resolution 8,192, and a three-layer MLP with 64-dimensional hidden states.
  • The camera distribution samples viewpoints on a 3D circle around the input cameras, jittering angles before aiming cameras at the look-at point.
  • For a rendered image, the procedure encodes it, adds noise at a sampled timestep, and applies multi-step denoising to obtain a predicted clean image.
  • Distillation runs for 3,000 steps, first matching Instant NGP renderings to EFT colors and then matching them to detached diffusion predictions.
Loading 2212.00792v3…