PaperlayerView exact revision on arXiv

Source-linked AI summary

3D-LMNet: Latent Embedding Matching for Accurate and Diverse 3D Point Cloud Reconstruction from a Single Image

Priyanka Mandikal, K L Navaneet, Mayank Agarwal, R. Venkatesh Babu

arXiv:1807.07796v2cs.CV

TL;DR

Single-view 3D reconstruction is ambiguous because occluded images can admit multiple valid shapes. 3D-LMNet learns a 3D point-cloud prior, maps images into its latent space, and uses probabilistic training for diverse outputs; evaluations report more accurate reconstructions than state-of-the-art methods on synthetic and real datasets.

  • Problem

    Single-view reconstruction must infer hidden 3D structure from ambiguous, self-occluded images, where multiple plausible reconstructions may exist.

  • Method

    3D-LMNet trains a point-cloud auto-encoder, maps 2D images to its learned latent space, and uses probabilistic training with a view-specific loss to generate multiple plausible outputs.

  • Results

    Quantitative and qualitative evaluations on synthetic and real datasets report more accurate and realistic point clouds than current state-of-the-art reconstruction methods.

  • Takeaways & Limitations

    The approach produces accurate single-view reconstructions while modeling view-specific uncertainty through multiple plausible 3D generations.

Abstract

from arXiv · show

3D reconstruction from single view images is an ill-posed problem. Inferring the hidden regions from self-occluded images is both challenging and ambiguous. We propose a two-pronged approach to address these issues. To better incorporate the data prior and generate meaningful reconstructions, we propose 3D-LMNet, a latent embedding matching approach for 3D reconstruction. We first train a 3D point cloud auto-encoder and then learn a mapping from the 2D image to the corresponding learnt embedding. To tackle the issue of uncertainty in the reconstruction, we predict multiple reconstructions that are consistent with the input view. This is achieved by learning a probablistic latent space with a novel view-specific diversity loss. Thorough quantitative and qualitative analysis is performed to highlight the significance of the proposed approach. We outperform state-of-the-art approaches on the task of single-view 3D reconstruction on both real and synthetic datasets while generating multiple plausible reconstructions, demonstrating the generalizability and utility of our approach.

1 Introduction

Single-view 3D reconstruction must resolve ambiguity in occluded regions while preserving accurate shape inference. 3D-LMNet addresses this through latent-embedding matching, multiple plausible outputs, and evaluation on real data.

  • Motivation: Single-view reconstruction is challenging because occluded images can support multiple plausible 3D shapes rather than one deterministic answer.An ideal system should produce multiple solutions under uncertainty but deterministic outputs when the image is sufficiently informative.
  • Representation: Point clouds are selected over voxel grids because they encode surface information more richly and require less computation than 3D-CNN voxel representations.Voxel occupancy grids are sparse in useful surface information, while point-cloud architectures are described as compute-friendly for shape generation and reconstruction.
  • Approach: 3D-LMNet uses latent-embedding matching to learn how 2D images map into a learned 3D point-cloud representation.The approach emphasizes learning a strong 3D prior before transferring knowledge from the 3D domain to image-based reconstruction.
  • Approach: The method generates multiple plausible 3D shapes from one image and evaluates whether these predictions remain diverse for ambiguous views.This contribution directly targets the problem of ambiguous ground truths in single-view reconstruction.
  • Evaluation: 3D-LMNet is evaluated on real data and reported to significantly outperform state-of-the-art reconstruction methods for single-view reconstruction.The introduction presents this evaluation as evidence of the approach’s generalizability.

Related Work

Prior work largely predicts deterministic reconstructions or directly maps images to 3D outputs, while 3D-LMNet first learns a point-cloud latent space and then maps images into it. Its probabilistic variant models view-specific uncertainty to generate multiple plausible shapes.

  • Related Work: Earlier approaches learned joint voxel-image embeddings or directly predicted point clouds from 2D images, whereas 3D-LMNet emphasizes learning a 3D point-cloud latent space first.The proposed setup separates 3D representation learning from 2D-to-3D knowledge transfer.
  • Generating Multiple Plausible Outputs: Because multiple correct reconstructions may fit one image, the probabilistic variant samples a latent Gaussian and uses a view-specific diversity loss to regulate variation.The loss suppresses uncertainty for unambiguous views while allowing exploration for ambiguous views.
  • 3D-LMNet: The pipeline trains a point-cloud auto-encoder, then an image encoder that maps 2D images to the learned latent space.The decoder converts the predicted embedding back into a point cloud during inference.
  • Latent Matching: Latent-vector matching improves over direct reconstruction-loss training, with L1 and L2 latent losses evaluated as alternatives.The authors report that matching latent vectors provides substantial improvement over optimizing reconstruction loss directly.
  • Generating Multiple Plausible Outputs: The model can generate diverse predictions by varying the latent sampling variable during inference, without using pose information at test time.The joint objective combines latent matching and diversity losses, weighted by λ.

4 Experiments

Experiments evaluate latent matching against direct reconstruction and prior methods on synthetic and real data, including probabilistic generation of multiple plausible outputs.

  • 4.2 Comparison with other methods on ShapeNet and Pix3D: Qualitatively, 3D-LMNet captures overall shape and finer details with more uniformly distributed points than PSGN and the baseline on ShapeNet.The figure also includes failure cases, and the methods predict different point counts: 1024 for PSGN versus 2048 for the baseline and 3D-LMNet.
  • 4.1 Empirical Evaluation on ShapeNet: Chamfer and EMD results show latent matching variants outperform the direct-training baseline, with latent-space losses producing lower reconstruction errors.The L1 formulation performs best among the tested variants, while more accurate latent matching corresponds to lower reconstruction errors.
  • 4.2 Comparison with other methods on ShapeNet and Pix3D: 3D-LMNet achieves state-of-the-art ShapeNet performance, outperforming PSGN in 8 of 13 categories for Chamfer and 10 of 13 for EMD.It also uses half PSGN’s trainable parameters while predicting point clouds at double the resolution.
  • 4.2 Comparison with other methods on ShapeNet and Pix3D: On real-world Pix3D, 3D-LMNet outperforms PSGN and the baseline by a large margin in both Chamfer and EMD.The authors report that it generalizes well, whereas the comparison methods struggle to generate meaningful reconstructions.
  • 4.3 Generating multiple plausible outputs: On ambiguous chair views, probabilistic latent matching produces semantically different reconstructions consistent with the input, while epsilon has minimal influence on unambiguous views.Variant II also slightly improves Chamfer from 6.48 to 6.45 and EMD from 8.1 to 8.04 relative to Variant I.

5 Conclusion

The paper concludes that rich 3D point-cloud latent representations and probabilistic image encoding improve single-view reconstruction accuracy, realism, and output diversity.

  • 5 Conclusion: 3D-LMNet learns a rich latent representation, maps images into it, and uses probabilistic training to generate multiple plausible 3D shapes.The conclusion covers both latent-space learning and ambiguity-aware prediction.
  • 5 Conclusion: Evaluations on synthetic and real datasets report more accurate and realistic point clouds than current state-of-the-art reconstruction methods.The conclusion cites both quantitative and qualitative evaluation.

1 Training Dataset Details

The networks are trained on synthetic ShapeNet models using a standardized split, rendered views, cropped images, and sampled ground-truth point clouds.

  • 1 Training Dataset Details: Training uses ShapeNet’s 80%-20% train/test split across 13 categories for comparability with previous work.The split is the one provided by the referenced prior setup.
  • 1 Training Dataset Details: Each ShapeNet model provides 24 pre-rendered azimuth views, cropped to 128 × 128 before network input.
  • 1 Training Dataset Details: Ground-truth point clouds are generated by uniformly sampling 16,384 points from the models.

2 Network Architectures

The architecture details cover point-cloud, image, and decoder components, with 3D-LMNet substantially smaller than PSGN in trainable parameters.

  • 2 Network Architectures: 3D-LMNet contains 22.7M parameters, compared with nearly 42.9M for PSGN.The parameter comparison is reported alongside the architecture specifications.
  • 2 Network Architectures: The architecture documentation includes separate point-cloud encoder, image encoder, and decoder specifications.These are provided in Tables 1, 2, and 3, respectively.
  • 2 Network Architectures: Category-wise metrics are reported for different latent matching variants on ShapeNet, with all metrics scaled by 100.The results are organized in Table 4.

3 Quantitative Comparison of 3D-LMNet Variants on ShapeNet

On ShapeNet, latent-matching variants outperform direct Chamfer training, with 3D-LMNet-L1 strongest across nearly all reported category-wise metrics.

  • 3D-LMNet-L1 and L2 significantly outperform 3D-LMNet-Chamfer on ShapeNet validation metrics.The comparison uses category-wise Chamfer and EMD errors.
  • 3D-LMNet-L1 achieves better Chamfer scores in every category than the direct-Chamfer variant.
  • 3D-LMNet-L1 achieves better EMD scores in all but one category.

4 Reconstructions on ShapeNet

The ShapeNet validation evaluation presents qualitative single-view reconstruction comparisons against state-of-the-art and baseline methods using randomly selected samples.

  • Figures 1 and 2 compare 3D-LMNet reconstructions with state-of-the-art and baseline methods.
  • The qualitative evaluation targets single-view reconstruction on the ShapeNet validation set.
  • The displayed ShapeNet examples are randomly selected from the validation evaluation.

5 Reconstructions on Pix3D

The Pix3D section combines qualitative real-world reconstruction comparisons with examples showing variation from probabilistic latent matching.

  • Figure 3 presents qualitative single-view reconstruction comparisons on the real-world Pix3D dataset.
  • The Pix3D comparison uses randomly selected samples.
  • Probabilistic latent matching produces chair reconstructions varying in legs, handles, and backs.

7 Auto-Encoder Results

The auto-encoder produces reconstructions resembling ground-truth point clouds, while latent interpolation yields smooth valid models and probabilistic matching generates varied views.

  • Auto-encoder reconstructions closely resemble ground-truth point clouds in appearance and point spread.
  • Latent interpolation produces smooth intermediate reconstructions that form valid models, including across categories.
  • ShapeNet qualitative results show 3D-LMNet points distributed more evenly than baseline car reconstructions.The passage links the more even distribution to lower EMD error metrics.
  • Additional qualitative figures cover ShapeNet and Pix3D reconstruction examples.
  • Probabilistic latent matching generates multiple reconstructions by sampling ε for ambiguous input views.Each reconstruction is shown from two viewing angles to assess correspondence with the input image.
  • The auto-encoder reconstruction and interpolation figures visualize reconstruction quality and latent-space behavior.
Loading 1807.07796v2…