PaperlayerView exact revision on arXiv

Source-linked AI summary

Pix2Vox++: Multi-scale Context-aware 3D Object Reconstruction from Single and Multiple Images

Haozhe Xie, Hongxun Yao, Shengping Zhang, Shangchen Zhou, Wenxiu Sun

arXiv:2006.12250v2cs.CV

TL;DR

Single- and multi-view 3D reconstruction needs methods that avoid order sensitivity, memory loss, and sequential processing costs in RNN-based approaches. Pix2Vox++ generates per-image coarse volumes in parallel, fuses them with multi-scale context awareness, and refines the result; it reports favorable accuracy and efficiency, while its object-centered coordinates and dense voxel representations constrain generalization and scalability.

  • Problem

    RNN-based reconstruction can produce order-inconsistent results, forget early-view features, and require sequential processing.

  • Method

    Pix2Vox++ uses an encoder-decoder to generate coarse volumes in parallel, multi-scale context-aware fusion to combine them, and a refiner for correction.

  • Results

    Pix2Vox++ performs favorably against state-of-the-art methods in accuracy and efficiency on ShapeNet, Pix3D, and Things3D.

  • Takeaways & Limitations

    The framework provides a unified approach for single-view and multi-view 3D object reconstruction and introduces the Things3D naturalistic-image benchmark.

  • Takeaways & Limitations

    Object-centered coordinates may encourage memorization and poor generalization, while dense high-resolution voxel volumes sharply increase memory requirements.

Abstract

from arXiv · show

Recovering the 3D shape of an object from single or multiple images with deep neural networks has been attracting increasing attention in the past few years. Mainstream works (e.g. 3D-R2N2) use recurrent neural networks (RNNs) to sequentially fuse feature maps of input images. However, RNN-based approaches are unable to produce consistent reconstruction results when given the same input images with different orders. Moreover, RNNs may forget important features from early input images due to long-term memory loss. To address these issues, we propose a novel framework for single-view and multi-view 3D object reconstruction, named Pix2Vox++. By using a well-designed encoder-decoder, it generates a coarse 3D volume from each input image. A multi-scale context-aware fusion module is then introduced to adaptively select high-quality reconstructions for different parts from all coarse 3D volumes to obtain a fused 3D volume. To further correct the wrongly recovered parts in the fused 3D volume, a refiner is adopted to generate the final output. Experimental results on the ShapeNet, Pix3D, and Things3D benchmarks show that Pix2Vox++ performs favorably against state-of-the-art methods in terms of both accuracy and efficiency.

1 Introduction

Pix2Vox++ addresses order variance, memory loss, and sequential-processing costs in RNN-based multi-view reconstruction with parallel coarse-volume generation and multi-scale context-aware fusion. Evaluations and extensions report favorable accuracy and efficiency, including fewer parameters and faster inference.

  • Motivation: RNN-based methods can vary with input order, forget early-view features, and process images sequentially without parallelization.These limitations affect consistency, information retention, and computation time.
  • Framework: Pix2Vox++ generates coarse 3D volumes from input images in parallel using shared encoder and decoder weights.Parallel processing removes the effect of input-image order and accelerates computation.
  • Framework: Its multi-scale context-aware fusion module selects high-quality reconstructions for different object parts from multiple coarse volumes.The selected parts are combined into a fused 3D volume without long-term memory loss.
  • Framework: A refiner corrects wrongly recovered parts of the fused volume to produce the final reconstruction.The refiner operates after fusion in the proposed four-module framework.
  • Evaluation: Pix2Vox++ outperforms competitive methods in accuracy and efficiency across ShapeNet, Pix3D, and Things3D evaluations.The paper also introduces Things3D as a large-scale naturalistic multi-view reconstruction dataset.
  • Extensions: 25% fewer parameters and 5% faster inference are reported for the improved method versus the preliminary version.The extension replaces VGG with ResNet as the backbone network.

2 Related Work

Prior 3D reconstruction methods span single-view and multi-view settings, but each faces limitations involving ambiguity, viewpoint coverage, matching, or camera information.

  • Single-view 3D Reconstruction: Single-view 3D reconstruction is inherently ambiguous because one partial observation can correspond to infinitely many possible 3D models.
  • Multi-view 3D Reconstruction: Traditional multi-view methods recover 3D structure through feature matching or reprojection-error minimization across RGB images.
  • Multi-view 3D Reconstruction: Large viewpoint separation makes multi-view matching difficult and can produce incomplete shapes when occluded or hollowed-out surfaces are not scanned.
  • The paper compares Pix2Vox++/F and Pix2Vox++/A architectures for low-resolution reconstruction, with EDLoss and RLoss defined in Equation 3.
  • Multi-view 3D Reconstruction: Obtaining extrinsic camera parameters is not always feasible, particularly when viewpoints are separated by a large margin.

3 The Proposed Method: Pix2Vox++

Pix2Vox++ reconstructs voxelized 3D shapes by processing views in parallel, adaptively fusing coarse volumes with multi-scale context, and refining the fused result.

  • Overview: Pix2Vox++ encodes each input image, decodes its features into a coarse 3D volume, fuses volumes, and refines the fused reconstruction.
  • Overview: The framework supports single or multiple RGB images and represents object shapes as 3D voxel grids with empty or occupied cells.
  • Framework variants: Pix2Vox++/F uses fewer parameters and lower computational complexity, whereas Pix2Vox++/A uses more parameters for more accurate shapes at higher computational cost.
  • Encoder and Decoder: The encoder computes image features for shape recovery, while the decoder transforms 2D feature maps into 3D volumes.
  • Multi-scale Context-aware Fusion: The fusion module generates score maps from coarse volumes and context, then combines them by weighted summation while preserving voxel spatial information.
  • Refiner: The refiner is a residual 3D encoder-decoder with U-net connections that corrects wrongly recovered parts while preserving local structure.
  • Loss Function: The training loss is the mean voxel-wise binary cross entropy between reconstructed occupancy and ground truth, with smaller values indicating closer predictions.

4 The Proposed Dataset: Things3D

Things3D is introduced to address the limited scale and realism of existing 3D reconstruction datasets by providing naturalistic multi-view images paired with diverse CAD models.

  • Existing datasets trade off scale and realism: ShapeNet lacks naturalistic backgrounds, while Pix3D contains only 395 models and 10,069 images.
  • Things3D contains 1.68M images of 280K objects, including 21K unique objects, collected from over 39K indoor scenarios.
  • The dataset increases object diversity by replacing SUNCG scene models with randomly selected ShapeNet models from the same categories.
  • Each model is rendered across a diverse set of naturalistic scenes, pairing sample images with corresponding CAD models.
  • Image generation uses tracked cameras and lights, excludes views with more than 12.5% occlusion, and produces 256 × 256 images.

5 Experiments

Experiments evaluate Pix2Vox++ on ShapeNet, Pix3D, and Things3D using datasets that span synthetic CAD renderings, aligned real images, and naturalistic scenes.

  • The evaluation compares proposed methods with state-of-the-art methods across ShapeNet, Pix3D, and Things3D.
  • ShapeNet experiments use 44K models across 13 major categories, with 24 rendered views per model and uniformly colored backgrounds.
  • Pix3D provides aligned real-world images and CAD models; testing uses 2,894 untruncated and unoccluded chair images.
  • Things3D contributes 1.68M images of 280K objects for evaluation in naturalistic indoor scenarios.

5.2 Metrics

The experiments measure voxel reconstruction with thresholded IoU and surface reconstruction with F-Score, using category-wise comparisons on ShapeNet.

  • IoU measures similarity between predicted and ground-truth voxel occupancy after binarizing probabilities at a fixed threshold of 0.3.
  • Higher IoU values indicate better reconstruction results.
  • F-Score is used as an additional reconstruction metric based on precision and recall at a distance threshold d.
  • For voxel methods, marching cubes generates an object surface, from which 8,192 points are sampled to compute F-Score against the ground truth.
  • Table 1 reports mean IoU per category for single-view ShapeNet reconstruction at 32^3 resolution.
  • Table 2 reports mean F-Score@1% per category, with voxel-method points sampled from marching-cubes meshes.

5.3 Implementation Details

Pix2Vox++ is trained on 224 × 224 RGB inputs with batch size 64 and produces 32^3 voxelized reconstructions using Adam optimization.

  • Training uses batch size 64, 224 × 224 RGB images, and output voxelized reconstructions of size 32^3.
  • Both Pix2Vox++/F and Pix2Vox++/A are optimized with Adam using β1 = 0.9, β2 = 0.999, and an initial learning rate of 0.001.
  • The learning rate is halved after 150 epochs.
  • Networks excluding multi-scale context-aware fusion are first trained with single-view images.

5.4 Evaluation on the ShapeNet Dataset

On ShapeNet, Pix2Vox++ achieves strong single-view, multi-view, and higher-resolution reconstruction results, with qualitative improvements and favorable comparisons against established methods.

  • Single-view 3D Object Reconstruction: Both Pix2Vox++/F and Pix2Vox++/A outperform all competitive methods in single-view ShapeNet reconstruction on IoU and F-Score@1%.Qualitative results also show more visually compelling 3D shapes.
  • Multi-view 3D Object Reconstruction: Both Pix2Vox++/F and Pix2Vox++/A consistently outperform 3D-R2N2 and AttSets across all evaluated numbers of views.The methods also recover better details than the compared approaches.
  • Evaluation Setup: The ShapeNet evaluations compare voxel and alternative reconstruction methods using IoU and F-Score@1% across single-view and multi-view settings.The reported voxel outputs include 32^3, 64^3, and 128^3 resolutions.
  • Higher-Resolution 3D Object Reconstruction: At 64^3 resolution, both variants outperform Matryoshka Networks and OGN, while at 128^3 Pix2Vox++/A outperforms both and Pix2Vox++/F is comparable.The methods recover better details qualitatively at 128^3 resolution.

5.5 Evaluation on the Pix3D Dataset

Pix2Vox++ is evaluated on real-world Pix3D chair images using models trained on ShapeNet-Chairs and Things3D-Chairs, with Things3D-based training producing better results.

  • Training Data: Models trained on Things3D-Chairs achieve better Pix3D results than models trained on ShapeNet-Chairs.The comparison uses mean IoU and F-Score@1% for the chair category.
  • Training Data: Pix2Vox++/A trained on Things3D-Chairs-RfC achieves the best Pix3D results among the reported training variants.Things3D-Chairs-RfC uses chairs rendered in naturalistic scenes with camera poses estimated from Pix3D.

5.6 Evaluation on the Things3D Dataset

On Things3D naturalistic images, Pix2Vox++ variants outperform competing methods in single-view reconstruction, while multi-view evaluation uses samples with sufficient rendered views.

  • Single-view Reconstruction: Both Pix2Vox++/F and Pix2Vox++/A outperform all competitive methods on Things3D single-view IoU and F-Score@1%.Pix2Vox++/A shows the best qualitative ability to recover shapes from a single natural-scene image.
  • Multi-view Reconstruction: The multi-view natural-scene experiment restricts testing to samples with at least eight rendering images so the same test set can be used across view counts.The passage describes different object occlusions across scenes as producing different available view counts.
  • Single-view Reconstruction: The Things3D single-view evaluation reports mean IoU and mean F-Score@1% per category, with the best number highlighted for each category.The experiments use the same input images for all methods during testing.

6 Analysis and Discussion

The analyses examine backbone choice, refinement, coordinate systems, fusion strategies, generalization, and efficiency, identifying accuracy gains alongside resolution-dependent resource costs.

  • Backbone Analysis: ResNet50 provides the best accuracy and efficiency among the evaluated Pix2Vox++/A backbones.Pretrained encoders perform slightly better than non-pretrained encoders.
  • Refiner Analysis: Adding the refiner increases single-view ShapeNet IoU from 0.658 to 0.670 for Pix2Vox++/A.Removing the refiner causes considerable degradation in reconstruction accuracy.
  • Coordinate and Fusion Analysis: LSM significantly outperforms Pix2Vox++ with more than one view, while replacing LSM’s recurrent fusion with multi-scale context-aware fusion improves LSM for all view counts.LSM uses camera parameters as an additional input.
  • Fusion Analysis: Multi-scale context-aware fusion outperforms average and context-aware fusion for high-resolution reconstruction, and replacing it with average fusion degrades results.The proposed fusion preserves details by concatenating feature maps at different scales.
  • Space and Time Complexity: Both Pix2Vox++ variants are approximately seven times faster than 3D-R2N2 for single-view 32^3 forward inference.For multi-view reconstruction, they also outperform 3D-R2N2 and AttSets in inference and training time.

7 Conclusion

Pix2Vox++ unifies single-view and multi-view 3D reconstruction while introducing the Things3D dataset. Across ShapeNet, Pix3D, and Things3D, it performs favorably against state-of-the-art methods and is computationally efficient.

  • Pix2Vox++ provides a unified framework for single-view and multi-view 3D reconstruction.
  • Things3D contains 1.68M images of 280K objects collected from over 39K indoor scenes.
  • Evaluation on ShapeNet, Pix3D, and Things3D shows favorable performance against state-of-the-art methods for both single-view and multi-view reconstruction.
  • About seven times faster than 3D-R2N2 in single-view inference time, the proposed methods are computationally efficient.
Loading 2006.12250v2…