Source-linked AI summary
$PC^2$: Projection-Conditioned Point Cloud Diffusion for Single-Image 3D Reconstruction
Luke Melas-Kyriazi, Christian Rupprecht, Andrea Vedaldi
TL;DR
Single-image 3D reconstruction is challenging because one view leaves shape ambiguous. PC2 uses image- and pose-conditioned diffusion over point clouds, with projection conditioning for shape and color. It reports competitive synthetic results and qualitative reconstructions on challenging real-world images.
Problem
Reconstructing an object’s 3D structure from a single 2D view is a long-standing, highly ill-posed problem.
Source: §1 · p. 1Method
PC2 iteratively denoises a point cloud conditioned on image features projected onto its partially denoised points from the given camera pose.
Source: §1 · p. 2, §3.4 · p. 5Results
Takeaways & Limitations
The probabilistic method can generate multiple plausible 3D point clouds consistent with one input image.
Source: §1 · p. 2Training requires point-cloud ground truth, which can be noisy; the authors report relative robustness to noisy data in Co3D.
Source: §4 · p. 8
Abstract
from arXiv · showhide
Reconstructing the 3D shape of an object from a single RGB image is a long-standing and highly challenging problem in computer vision. In this paper, we propose a novel method for single-image 3D reconstruction which generates a sparse point cloud via a conditional denoising diffusion process. Our method takes as input a single RGB image along with its camera pose and gradually denoises a set of 3D points, whose positions are initially sampled randomly from a three-dimensional Gaussian distribution, into the shape of an object. The key to our method is a geometrically-consistent conditioning process which we call projection conditioning: at each step in the diffusion process, we project local image features onto the partially-denoised point cloud from the given camera pose. This projection conditioning process enables us to generate high-resolution sparse geometries that are well-aligned with the input image, and can additionally be used to predict point colors after shape reconstruction. Moreover, due to the probabilistic nature of the diffusion process, our method is naturally capable of generating multiple different shapes consistent with a single input image. In contrast to prior work, our approach not only performs well on synthetic benchmarks, but also gives large qualitative improvements on complex real-world data.
1. Introduction
The paper uses projection-conditioned diffusion to reconstruct point-cloud shapes and colors from a single image and camera pose. It reports competitive synthetic-benchmark performance and high-quality qualitative results on real-world images.
1. Introduction
- Single-view reconstruction is ill-posed and requires prior knowledge of plausible object shapes and appearances.The paper contrasts this with reconstructing shape from multiple views using multiple-view geometry.Source: §1 · pp. 1–2
- Prior deep-learning methods can predict 3D volumes, but many produce low-resolution geometries, while some implicit representations cannot generate a distribution of possible shapes from one image.The paper motivates diffusion as a way to generate shapes for unseen object regions.Source: §1 · p. 2
- At each diffusion step, PC2 projects image features onto partially denoised points, conditioning point-cloud shape reconstruction on the image and viewpoint.The same projection conditioning is used to predict point colors after shape reconstruction.Source: §1 · p. 2
- The point-cloud representation, projection-conditioned diffusion, and projection-based color prediction are linked; without point clouds, the approach would be limited to coarse voxel-based shape approximation.The authors argue that unstructured point clouds suit diffusion’s flexible nature.Source: §1 · p. 2
- PC2 can generate multiple plausible point clouds consistent with one image, then filter them by input-mask fit to address single-view ambiguity.The authors describe this filtering as an automated way to benefit from reconstruction diversity.Source: §1 · p. 2
- PC2 performs competitively on the synthetic ShapeNet benchmark and produces high-quality qualitative reconstructions across multiple categories in real-world Co3D.The authors frame this as extending beyond synthetic data.Source: §1 · p. 2
2. Related work
Prior single-view reconstruction methods include fixed-grid decoders, implicit representations, and diffusion models for point clouds. The paper distinguishes PC2 by applying point-cloud diffusion to image-conditioned reconstruction and using projection conditioning for shape and color.
2. Related work
- Fixed-grid decoders include 3D-R2N2, LSM, Pix2Vox, Pix2Vox++ [55], and LegoFormer.These methods encode images with 2D networks and decode fixed, regular 3D representations such as voxel grids.Source: §2 · pp. 2–3
- Figure 2 depicts image-feature projection during point-cloud denoising and reuse of projection for coloring; Figure 3 shows intermediate denoising and novel-view renders on Co3D [34].Figure 3’s caption describes challenging real-world viewpoints.Source: Figure 2 · p. 3, Figure 3 · p. 4
- Single- and few-view NeRF methods condition radiance fields on reference-image features, but strong single-view results remain elusive.The paper identifies NeRF-WCE and PixelNeRF as especially relevant.Source: §2 · p. 3
- In a two-category ShapeNet motivating experiment, projection conditioning reconstructs geometries more effectively than global conditioning, as measured by F-scores.The table caption states that higher F-scores are better but supplies no cell values here.Source: Table 1 · p. 3
- Earlier point-cloud diffusion methods address unconditional generation or completion and train only on synthetic data, whereas PC2 addresses reconstruction from real-world images.The cited prior methods are,, and.Source: §2 · p. 3
3. Method
The method uses point-cloud diffusion conditioned on an image and camera view, with projection-based local conditioning and silhouette-based filtering. It also uses the same projection conditioning to predict point colors.
3.1. Diffusion Models
- Diffusion models add Gaussian noise to target samples over scheduled steps, then learn a reverse process that iteratively denoises from noise toward the target distribution.For sufficiently small sampling steps, the reverse conditional is approximated by an isotropic Gaussian with fixed small covariance, so the network predicts its mean via noise prediction.Source: §3.1 · pp. 4–5
3.2. Point Cloud Diffusion Models
- The point-cloud model treats N points as a 3N-dimensional object, predicts added noise, and iteratively denoises Gaussian-sampled points into a shape.At inference, it samples a point cloud from a 3N-dimensional Gaussian and runs the reverse process.Source: §3.2 · p. 5
3.3. Conditional Point Cloud Diffusion Models
- A global image embedding provides only weak geometric consistency and can yield plausible shapes that do not match the input view, motivating locally conditioned PC2.The paper relates this weakness to prior findings that global-feature methods can generate generic objects from coarse image categories.Source: §3.3 · p. 5
3.4. PC2: Projection-Conditional Diffusion Models
- On ShapeNet-R2N2, PC2 performs similarly to prior work without filtering and outperforms it with PC2-FA or PC2-FM; it rasterizes image features onto points during denoising.The same projection conditioning supports a separate single-step point-color network, which can learn different colors for an object's front and backside.Source: §3.4 · p. 5, Table 2 · p. 6
3.5. PC2-FM and PC2-FA: Filtering for 3D Reconstruction
- Silhouette-based filtering selects among multiple reconstructions using either object-mask IoU or agreement among predicted masks; filtering improves results but remains below the oracle.PC2-FM's gain is 0.02 F-score from one sample to two, with diminishing returns as more samples are filtered.Source: Table 3 · p. 6, §3.5 · p. 6, Table 4 · p. 6
4. Experiments
Experiments find that filtering improves PC2's synthetic benchmark performance, while qualitative results show detailed reconstructions on synthetic and real-world objects. Its probabilistic sampling supports diverse shapes and filtering, though training requires point-cloud ground truth.
Datasets and Evaluation
- Evaluation uses the 13-category ShapeNet-R2N2 benchmark and F-score, while Co3D experiments use point clouds generated from image sequences with COLMAP.The Co3D qualitative baseline is NeRF-WCE because the paper reports no prior single-view Co3D reconstruction work.Source: §4 · p. 7
Quantitative Results
- On ShapeNet-R2N2, PC2 performs better on fine-detail categories such as rifle and airplane, and its reconstructions show high detail on synthetic examples.The paper notes that all compared models produce adequate results on the simple, synthetic dataset.Source: Figure 6 · p. 7, §4 · p. 8
- With filtering, PC2-FM consistently outperforms prior work, improving on the state of the art for most ShapeNet-R2N2 categories and on average.Without filtering, PC2 performs on par with prior work.Source: §4 · p. 8
Qualitative Results
- On Co3D, PC2 produces realistic shapes from novel viewpoints, with finer detail than prior methods and less-blurry renders than deterministic NeRF-WCE.The comparison includes hydrants; NeRF-WCE's deterministic nature is associated with blurry novel-view outputs.Source: Figure 4 · p. 7, §4 · p. 8
Diversity of Generations
- For ambiguous chair images, different random seeds yield meaningful shape variations that remain consistent with the reference view.The examples show eight generations for each of three highly ambiguous images.Source: §4 · p. 8
Filtering Analysis
- Mask-free filtering improves results over single-sample performance, mask-based filtering improves them further, and both remain below the oracle upper bound.The oracle selects the best sample per image by F-score.Source: §4 · p. 8
- Using two images for filtering significantly improves results, and additional images bring further gains with diminishing returns.The passage does not report specific score values.Source: §4 · p. 8
Limitations
- Training requires point-cloud ground truth, which can be noisy; Co3D hydrant clouds include holes in regions unobserved in the source video.The authors report relative robustness to this kind of noise.Source: §4 · p. 8
5. Conclusions
The conclusion presents PC2 as a diffusion method for single-view 3D reconstruction, reporting synthetic benchmark gains and detailed qualitative results on challenging real-world images. It frames scaling to larger datasets and models as future work.
Conclusion
- PC2 iteratively reconstructs shapes by projecting image features onto a partially denoised point cloud, and experiments demonstrate geometry and point-color reconstruction.The conclusion reports quantitative outperformance on synthetic benchmarks and detailed qualitative reconstructions from challenging real-world images.Source: §5 · p. 8
Future Work
- The authors identify scaling the method to larger datasets and models as future work.They describe the work as a step toward similar scaling of diffusion models for 3D reconstruction.Source: §5 · p. 8
Ethics
- The paper reports that ShapeNet and Co3D are used compatibly with their terms and that the images contain no personal data.ShapeNet models are used compatibly with its Data Analysis Permission.Source: §5 · pp. 8–9
6. Implementation Details
The implementation uses a point-voxel network to process partially denoised clouds and rasterizes points to attach image features and mask cues during projection conditioning.
Point-Voxel Model
- The Point-Voxel model combines an independent per-point MLP branch with a voxel-grid branch that processes a 128^3 grid using a 3D U-Net.The U-Net's downsampling and upsampling structure captures global and local shape information.Source: §6 · p. 12
- At each diffusion step, the model concatenates a timestep embedding with point features at the input to each layer.The point-voxel model processes the partially denoised point cloud.Source: §6 · p. 12
Projection Features
- Projection conditioning rasterizes the cloud from the camera view and attaches pixel image features, the object-mask value, and a vector toward the nearest mask pixel to each rasterized point.Points not rasterized to a pixel receive zero vectors for these quantities.Source: §6 · p. 12
7. Additional Qualitative Examples
Supplementary examples show reconstructions across Co3D and ShapeNet categories, including best-ranked, random, and worst-ranked ShapeNet outputs.
Co3D Examples
- Figures 7–9 show Co3D reconstructions across hydrants, teddy bears, glasses, remotes, motorcycles, hairdryers, plants, and donuts.These are additional qualitative examples.Source: §7 · p. 12
ShapeNet Examples
- Figure 10 presents the highest-F-score ShapeNet reconstructions selected for each category.Source: §7 · p. 12
- Figures 11 and 12 show, respectively, the lowest-F-score selections and random ShapeNet reconstructions across categories.Source: §7 · p. 12
8. Additional Ablations
Additional sofa-category ShapeNet ablations compare removing mask-distance supervision with replacing rasterization-based projection by naive projection.
8. Additional Ablations
- 0.081 F-score points (40% relative decrease) from naive projection, versus 0.019 F-score points (9% relative decrease) from removing the mask distance function.Both ablations use only the sofa category of ShapeNet; the naive-projection point clouds were qualitatively significantly worse, while the mask-distance variant looked similar.Source: §8 · p. 12
- Removing the 2D mask distance function caused a small F-score reduction on the sofa subset.The reduction was 0.019 points, a relative decrease of 9%.Source: §8 · p. 12
- Projecting all points, including occluded points, instead of using rasterization-based projection substantially worsened qualitative results.The change reduced F-score by 0.081 points, a relative decrease of 40%.Source: §8 · p. 12
9. Analysis of Failure Cases
Failure analysis finds that ambiguous object categories can lead to incorrect reconstructions, while other models also struggle on some challenging examples.
9. Analysis of Failure Cases
- Ambiguous categories can lead the model to confuse objects that resemble rectangular prisms from certain views, such as chairs with boxes or couches with boxes.The analyzed ShapeNet-R2N2 model is image-conditional but not class-conditional.Source: §9 · p. 12
- Other models also struggle to a similar degree on some challenging examples, including watercraft.The cited examples are from the watercraft category in the final row of the figure.Source: §9 · p. 12
- The Co3D hydrant and teddy-bear examples show diffusion intermediates, final colored point clouds, and novel-view predictions; six further categories are also shown.The additional categories are glasses, remotes, motorcycles, hairdryers, plants, and donuts.Source: Figure 7 · p. 13, Figure 8 · p. 14, Figure 9 · p. 15
- The ShapeNet-R2N2 examples cover successes, failures, and random outputs across all 13 classes.Each set places the input image at the left; the figure's final row on the right is empty because the examples cover all 13 classes.Source: Figure 10 · p. 16, Figure 11 · p. 17, Figure 12 · p. 18