Source-linked AI summary
SPAR3D: Stable Point-Aware Reconstruction of 3D Objects from Single Images
Zixuan Huang, Mark Boss, Aaryaman Vasishta, James M. Rehg, Varun Jampani
TL;DR
Single-image 3D reconstruction must balance visible-surface fidelity with uncertain occluded geometry, while existing regression and generative approaches expose complementary weaknesses. SPAR3D combines point diffusion and image-conditioned meshing, achieving strong benchmark performance with 0.7-second inference and interactive point-cloud editing. Its material and illumination estimates remain limited by the ambiguity of unsupervised inverse rendering from a single image.
Problem
Single-image reconstruction must model uncertain occluded surfaces without sacrificing the speed and visible-surface alignment of regression methods or the efficiency of generative methods.
Method
SPAR3D first uses point diffusion to sample a sparse point cloud, then combines it with the input image to reconstruct a detailed mesh.
Results
SPAR3D significantly outperforms previous state-of-the-art methods across evaluated datasets, with 0.7 seconds per-object inference among the fastest reconstruction speeds.
Takeaways & Limitations
Sparse point clouds provide a lightweight intermediate representation that supports probabilistic reconstruction, high fidelity, and human editing in the loop.
Takeaways & Limitations
Material and illumination decomposition can be suboptimal because unsupervised inverse rendering from a single image is inherently ambiguous.
Abstract
from arXiv · showhide
We study the problem of single-image 3D object reconstruction. Recent works have diverged into two directions: regression-based modeling and generative modeling. Regression methods efficiently infer visible surfaces, but struggle with occluded regions. Generative methods handle uncertain regions better by modeling distributions, but are computationally expensive and the generation is often misaligned with visible surfaces. In this paper, we present SPAR3D, a novel two-stage approach aiming to take the best of both directions. The first stage of SPAR3D generates sparse 3D point clouds using a lightweight point diffusion model, which has a fast sampling speed. The second stage uses both the sampled point cloud and the input image to create highly detailed meshes. Our two-stage design enables probabilistic modeling of the ill-posed single-image 3D task while maintaining high computational efficiency and great output fidelity. Using point clouds as an intermediate representation further allows for interactive user edits. Evaluated on diverse datasets, SPAR3D demonstrates superior performance over previous state-of-the-art methods, at an inference speed of 0.7 seconds. Project page with code and model: https://spar3d.github.io
1. Introduction
SPAR3D addresses the trade-off between fast, visible-surface-aligned regression and probabilistic but costly, misaligned generation through a two-stage point-cloud reconstruction design.
- Monocular 3D reconstruction is an ill-posed inverse problem because visible surfaces are observable, while occluded surfaces require a strong 3D prior.
- Regression models align well with visible surfaces and are fast, but their bijective image-to-3D assumption produces ambiguous learning targets and poor occluded surfaces and textures.
- Diffusion models represent uncertainty without predicting statistical means, but high-resolution iterative sampling is inefficient and often misaligned with input-visible surfaces.
- SPAR3D samples sparse point clouds with diffusion, then transforms them into detailed meshes, assigning uncertainty modeling to the fast low-resolution stage.
- Point clouds provide a lightweight bridge between stages and support topology-free local edits before meshing, producing meshes tailored to user requirements.
- SPAR3D reports superiority over prior state-of-the-art methods across varied data sources while keeping total inference below 0.7 seconds.
2. Related Work
Related work spans feedforward regression, diffusion-based generation, and optimization-based reconstruction, each with distinct strengths and limitations; SPAR3D is presented as a point-cloud-mediated alternative.
- Feedforward methods have progressed from geometry prediction on small datasets to large-scale training, improving generalization and alignment with observed surfaces.
- SPAR3D conditions point diffusion on the input image to generate a sparse point cloud, which is then used with image features for high-resolution triplane reconstruction.
- Diffusion-based 3D generation models image-conditioned asset distributions, avoiding over-smoothed results but suffering from input misalignment and computational inefficiency.
- Optimization-based single-view methods use 2D generative priors without large-scale annotation, but optimization is inefficient and prone to local minima without a strong explicit 3D prior.
3. Method
SPAR3D reconstructs single-image 3D objects through point sampling followed by detailed meshing, combining diffusion-based uncertainty modeling with image-aligned regression.
- Point Sampling and Meshing: SPAR3D uses a point diffusion model to sample sparse point clouds, then transforms them into highly detailed meshes.The point sampling stage models uncertainty at low resolution, while the meshing stage uses local image features for detail.
- Point Sampling Stage: The point cloud contains 512 points with six channels encoding XYZ coordinates and RGB values, conditioned on the input image.The point diffusion model generates p0 ∈ R^n×6, with n set to 512.
- Point Sampling Stage: The diffusion process adds Gaussian noise and trains a denoiser to recover it, using DDIM sampling and classifier-free guidance during inference.Image condition tokens provide conditioning, while classifier-free guidance improves sampling fidelity.
- Meshing Stage: The meshing model uses a triplane transformer conditioned on the image and point cloud to estimate geometry, texture, lighting, metallic values, and roughness.Geometry and materials are rendered differentiably during training, enabling rendering-loss supervision.
- Surface Estimation: Geometry is extracted from triplane-predicted density using differentiable Marching Tetrahedron, with predicted vertex offsets and normals for smoother surfaces.The additional attributes reduce artifacts introduced by Marching Tetrahedron.
- Rendering and Training: The rendering loss combines image L2, LPIPS perceptual distance, and opacity-mask L2, alongside mesh and shading regularization.The differentiable renderer uses PBR materials, geometry, a Disney shader, Monte Carlo integration, and visibility testing for shadows.
- Interactive Editing: Sparse point clouds support interactive editing because users can modify unseen surfaces without directly managing mesh topology.Edited point clouds are fed into the meshing stage to produce meshes tailored to user expectations.
4. Experiments
SPAR3D is evaluated on standardized object datasets and against efficient regression, multiview-diffusion, and generative baselines. Results report strong quantitative and qualitative reconstruction, generalization, editing, and ablation performance, with 0.7-second inference.
- 4.1. Evaluation: Evaluation uses GSO and OmniObject3D, each with around 250 objects rendered under varied viewpoints, environments, and focal lengths.Predictions are aligned to ground truth using rotation search followed by ICP before metric computation.
- 4.2. Main Results: SPAR3D outperforms regression and generative baselines across most metrics on both datasets, while achieving 0.7 seconds per reconstructed object.It is slightly worse than the strongest baseline on SSIM, which the authors associate with Monte Carlo noise from the shader.
- 4.2. Main Results: Qualitative comparisons show better visible-surface alignment and more detailed occluded geometry and textures than prior regression, multiview-diffusion, and pure-generative methods.The compared methods respectively exhibit oversmoothed backsides, view-inconsistency artifacts, or hallucinated details that do not follow the input image.
- 4.3. Editing Results: Explicit point clouds enable interactive edits such as adding object parts or improving generated details before remeshing.Examples include adding a mug handle or elephant tail and modifying points to correct local mesh imperfections.
- 4.4. Ablation and Analysis: Removing the point sampling stage causes a significant performance drop, supporting its contribution to the two-stage design.A conflicting-cue experiment also shows image-guided visible surfaces and point-cloud-guided backside surfaces.
5. Conclusion
SPAR3D reconstructs single-view 3D objects through point diffusion followed by detailed mesh reconstruction. It combines distribution learning with fast inference and supports user editing.
- SPAR3D first generates a sparse point cloud via point diffusion, then reconstructs a highly detailed mesh from the point cloud and image.
- The two-stage design combines the advantages of regression-based and generative modeling for single-view 3D reconstruction.
- SPAR3D significantly outperforms previous state-of-the-art methods on standard benchmarks and in-the-wild images while maintaining fast inference.
- The method is intended to support scalable generation of high-quality 3D content.
A. Limitations
SPAR3D has limitations involving artifacts in generated point clouds and occasional inaccuracies in material decomposition.
- Generated point clouds occasionally contain artifacts such as small surface spikes or detached parts.The authors report that these imperfections can typically be remedied through SPAR3D’s editing capabilities.
- More principled denoiser designs or diffusion samplers could improve the method’s utility and robustness.
- Material decompositions can sometimes be suboptimal despite being learned during training.
- The material limitation reflects ambiguity in single-image inverse rendering, particularly for unsupervised learning.The authors suggest semi-supervised learning as a possible path toward more plausible material estimates.
B. Additional Illustrations of our Architecture
Additional illustrations describe the denoiser and meshing architectures, including their inputs, latent processing, and outputs.
- Point Cloud Denoiser Architecture: The point cloud denoiser takes a noisy point cloud and image as input and produces a denoised point cloud.
- Point Cloud Denoiser Architecture: The image and noisy point cloud are encoded into latent vectors, concatenated, processed by transformer blocks, and decoded as the denoised point cloud.
- Meshing Model Architecture: The meshing model takes a point cloud and image as input and produces a textured mesh and environment map.
- Meshing Model Architecture: The meshing architecture encodes the inputs as latent vectors, conditions a triplane transformer, and queries the triplane with MLPs.
C. Decomposition Results
SPAR3D’s decomposition and relighting results are compared with SF3D, a fully regressive method.
- Compared with SF3D, SPAR3D often estimates albedo with fewer baked-in lighting artifacts.The authors report that this improves relighting quality under different illumination conditions.
D. Additional In-the-wild Results
SPAR3D achieves high reconstruction quality on in-the-wild images from 3D-Arena, supporting its generalization ability.
- SPAR3D achieves high reconstruction quality on 3D-Arena in-the-wild images, further validating its strong generalization ability.