PaperlayerView exact revision on arXiv

Source-linked AI summary

Pix2Vox: Context-aware 3D Reconstruction from Single and Multi-view Images

Haozhe Xie, Hongxun Yao, Xiaoshuai Sun, Shangchen Zhou, Shengping Zhang

arXiv:1901.11153v2cs.CV

TL;DR

RNN-based 3D reconstruction can produce order-dependent results, lose long-term information, and process views sequentially. Pix2Vox instead reconstructs coarse volumes per image, fuses their high-quality parts contextually, and refines the result; it outperforms state-of-the-art methods on ShapeNet and Pix3D and is 24 times faster than 3D-R2N2 in backward inference time. The method remains limited by low reconstruction resolution.

  • Problem

    RNN-based multi-view reconstruction can be permutation-variant, lose long-term information, and require time-consuming sequential processing.

  • Method

    Pix2Vox generates a coarse 3D volume from each image in parallel, adaptively fuses high-quality parts across volumes, and refines the fused volume.

  • Results

    Pix2Vox outperforms state-of-the-art methods on ShapeNet and Pix3D in accuracy and efficiency, with 24 times faster backward inference than 3D-R2N2.

  • Takeaways & Limitations

    Context-aware fusion also improves an RNN-based MVS system by about 2% IoU on the ShapeNet testing set.

  • Takeaways & Limitations

    The reconstruction results remain at low resolution, which the authors identify as a target for future improvement.

Abstract

from arXiv · show

Recovering the 3D representation of an object from single-view or multi-view RGB images by deep neural networks has attracted increasing attention in the past few years. Several mainstream works (e.g., 3D-R2N2) use recurrent neural networks (RNNs) to fuse multiple feature maps extracted from input images sequentially. However, when given the same set of input images with different orders, RNN-based approaches are unable to produce consistent reconstruction results. Moreover, due to long-term memory loss, RNNs cannot fully exploit input images to refine reconstruction results. To solve these problems, we propose a novel framework for single-view and multi-view 3D reconstruction, named Pix2Vox. By using a well-designed encoder-decoder, it generates a coarse 3D volume from each input image. Then, a context-aware fusion module is introduced to adaptively select high-quality reconstructions for each part (e.g., table legs) from different coarse 3D volumes to obtain a fused 3D volume. Finally, a refiner further refines the fused 3D volume to generate the final output. Experimental results on the ShapeNet and Pix3D benchmarks indicate that the proposed Pix2Vox outperforms state-of-the-arts by a large margin. Furthermore, the proposed method is 24 times faster than 3D-R2N2 in terms of backward inference time. The experiments on ShapeNet unseen 3D categories have shown the superior generalization abilities of our method.

1. Introduction

Pix2Vox addresses order inconsistency, memory loss, and sequential computation in RNN-based multi-view reconstruction with parallel coarse-volume processing and context-aware fusion. It reports stronger accuracy and efficiency than state-of-the-art methods on ShapeNet and Pix3D.

  • Motivation: RNN-based reconstruction is permutation-variant, memory-limited, and time-consuming because images are processed sequentially.Different image orders can produce inconsistent shapes, while long-term memory loss limits multi-view refinement.
  • Approach: Pix2Vox uses an encoder, decoder, context-aware fusion module, and refiner for single-view and multi-view 3D reconstruction.The encoder and decoder produce coarse volumes in parallel, fusion combines high-quality parts, and the refiner corrects recovered errors.
  • Scope: Pix2Vox is presented as a unified framework for both single-view and multi-view reconstruction across synthetic and real-world images.The framework includes two versions, Pix2Vox-F and Pix2Vox-A, designed to balance accuracy and model size.
  • Approach: Context-aware fusion adaptively selects high-quality parts from different coarse volumes in parallel to produce a fused whole-object reconstruction.The method exploits context across multiple views rather than sequentially retaining information in an RNN.
  • Results: ShapeNet and Pix3D experiments show that Pix2Vox outperforms state-of-the-art methods in both accuracy and efficiency.Additional experiments report strong generalization when reconstructing unseen 3D objects.

2. Related Work

Prior work addresses single-view and multi-view reconstruction with specialized shape cues, volumetric representations, point clouds, feature pooling, and recurrent fusion. These approaches face assumptions, computational costs, distortion, permutation variance, or loss of useful features.

  • Single-view 3D Reconstruction: Single-view 3D reconstruction is theoretically ill-posed, and traditional ShapeFromX methods rely on strong presumptions and expertise in natural images.ShapeFromX may use silhouettes, shading, or texture as the image cue.
  • Single-view 3D Reconstruction: 3D-VAE-GAN requires class labels, while MarrNet estimates depth, surface normals, and silhouettes but can produce severe distortion.These methods illustrate constraints in prior single-view reconstruction pipelines.
  • Single-view 3D Reconstruction: OGN and O-CNN use octrees for higher-resolution volumetric objects, but octree representations are complex and consume more computational resources.PSGN and 3D-LMNet instead generate point clouds from single-view images.
  • Multi-view 3D Reconstruction: 3D-R2N2 and LSM use RNNs for single- or multi-view reconstruction, but their fusion is time-consuming and permutation-variant.3DensiNet uses max pooling, which may ignore other valuable features useful for reconstruction.

3. The Method

Pix2Vox reconstructs 3D voxel volumes through parallel encoder-decoder processing, context-aware fusion of part-level reconstructions, and refinement. Its variants trade computational complexity for reconstruction accuracy, while the fusion module preserves spatial information and selects high-quality parts across views.

  • Overview: Pix2Vox encodes each input image and decodes its feature map into a corresponding coarse 3D volume, supporting single- and multi-view reconstruction.The encoder extracts image features, and the decoder transforms each 2D feature map into a voxelized 3D volume.
  • Overview: Parallel coarse-volume generation removes dependence on input-image order, while the context-aware fusion module combines information from multiple views.The fusion module adaptively selects high-quality reconstructions for each object part before producing a fused volume.
  • Architecture variants: Pix2Vox-F uses fewer parameters and lower computational complexity, whereas Pix2Vox-A uses more parameters to construct more accurate 3D shapes at higher computational cost.The two variants differ in encoder design and feature-vector size, with Pix2Vox-F producing 2048-dimensional vectors and Pix2Vox-A producing 16384-dimensional vectors.
  • Context-aware fusion: Context-aware fusion generates normalized score maps and forms each fused voxel by weighted summation of corresponding coarse voxels.The scoring network operates on context derived from decoder features, with shared weights across views.
  • Refiner: The refiner uses a residual 3D encoder-decoder with U-net skip connections to correct wrongly recovered parts while preserving local structure.Its decoder reconstructs the refined volume from encoded fused-volume features.
  • Loss function: The network is trained with the mean voxel-wise binary cross-entropy between reconstructed occupancy and ground-truth occupancy.The loss decreases as the prediction becomes closer to the ground truth.

4. Experiments

Pix2Vox-F and Pix2Vox-A are evaluated on ShapeNet synthetic images using IoU after thresholding predicted occupancy probabilities. The experiments include single-view and multi-view reconstruction comparisons.

  • Datasets and Metrics: The evaluation uses 43,783 ShapeNet models across 13 major categories and binarizes predicted occupancy probabilities at a fixed threshold of 0.3.IoU measures similarity between the predicted voxel grid and ground truth.
  • Datasets and Metrics: IoU compares predicted and ground-truth voxel occupancy, with higher values indicating better reconstruction.The indicator function selects predicted occupied voxels above the voxelization threshold.
  • Datasets and Metrics: Figure 6 presents single-view reconstructions on the left and multi-view reconstructions on the right, with GT denoting ground truth.DRC is trained and tested separately for each category.

4.2. Implementation Details

The implementation trains Pix2Vox-F and Pix2Vox-A on RGB inputs using a staged optimization procedure and compares them with existing methods on ShapeNet. Pix2Vox-A improves IoU over 3D-R2N2 in both single-view and multi-view settings.

  • Implementation Details: The models use 224 × 224 RGB inputs and produce voxelized reconstructions of size 32^3.Training uses Adam with an initial learning rate of 0.001 and staged single-view then multi-view training.
  • Synthetic Reconstruction Results: Pix2Vox-A increases single-view IoU over 3D-R2N2 by 18%.The single-view results significantly outperform other compared methods.
  • Synthetic Reconstruction Results: Pix2Vox-A achieves 13% higher IoU than 3D-R2N2 for multi-view reconstruction.It consistently outperforms 3D-R2N2 across all tested numbers of views.
  • Evaluation Setup: Table 3 reports single-view Pix3D reconstruction using Intersection-over-Union (IoU), with the best number highlighted in bold.The table concerns real-world images rather than ShapeNet synthetic reconstruction.
  • Qualitative Results: Both Pix2Vox variants recover thin object parts such as lamps and table legs in ShapeNet examples.Pix2Vox-A also produces better results than Pix2Vox-F in the qualitative comparison.

4.4. Reconstruction of Real-world Images

The real-world evaluation tests single-view reconstruction on Pix3D using chair images and finds that Pix2Vox-A outperforms competing approaches without pose estimation. Qualitative results indicate improved handling of real-world scenarios.

  • Dataset: Pix3D evaluation uses 2,894 untruncated and unoccluded chair images.The images follow the dataset settings used by the Pix3D work.
  • Data Preparation: Training data are generated from ShapeNet CAD models and augmented with random color and light jittering.The RenderForCNN pipeline generates 60 images for each CAD model before preprocessing.
  • Qualitative Results: Figure 7 shows single-view Pix3D reconstructions alongside ground-truth 3D objects.The qualitative analysis reports that the proposed methods are more effective in real-world scenarios.
  • Quantitative Results: Pix2Vox-A outperforms competing approaches on the Pix3D testing set without estimating object pose.The comparison uses mean IoU reported for single-view reconstruction.

4.5. Reconstruction of Unseen Objects

The unseen-object experiment evaluates models trained on 13 ShapeNet categories against objects from 44 remaining categories. Pix2Vox-F and Pix2Vox-A generalize better than 3D-R2N2 under this setting.

  • Experimental Setting: The models are tested on objects from 44 ShapeNetCore categories excluded from training.All pretrained models are trained on 13 major categories and have not seen the tested categories or their labels.
  • Qualitative Results: Figure 8 presents reconstructions of unseen ShapeNet objects from five-view images, with GT representing ground truth.The figure provides qualitative examples for the generalization experiment.
  • Results: Pix2Vox-A reaches reconstruction IoU 0.227 on unseen objects, compared with 0.120 for 3D-R2N2.Pix2Vox-F reaches 0.209 in the same experiment.
  • Results: 3D-R2N2 can hardly recover unseen-object shapes, whereas Pix2Vox-F and Pix2Vox-A show satisfactory generalization abilities.This qualitative conclusion accompanies the reported IoU values.

4.6. Ablation Study

The ablations show that context-aware fusion outperforms average and RNN-based fusion, while the refiner substantially improves single-view accuracy but contributes less as views increase.

  • Context-aware fusion: Context-aware fusion performs better than average fusion at selecting high-quality reconstructions for each part.
  • Context-aware fusion: Both average fusion and context-aware fusion consistently outperform RNN-based fusion across all tested numbers of views.
  • Refiner: The refiner increases single-view ShapeNet IoU from 0.636 without refinement to 0.661 with Pix2Vox-A.
  • Refiner: As the number of views increases, the refiner's effect on reconstruction becomes weaker.
  • Overall ablation findings: Both context-aware fusion and the refiner contribute to performance improvements over previous state-of-the-art methods.

4.7. Space and Time Complexity

Pix2Vox reduces model size and accelerates inference compared with 3D-R2N2, especially in backward inference for Pix2Vox-F.

  • 80% fewer parameters are used by Pix2Vox-F than by 3D-R2N2.
  • Both Pix2Vox variants are about 8 times faster than 3D-R2N2 in single-view forward inference.

4.8. Discussion

Pix2Vox selects high-quality reconstruction parts using context-aware score maps and improves reconstruction in an MVS system, while its output resolution remains low.

  • Score maps identify high-quality parts across coarse volumes, allowing bad reconstructions to be eliminated during fusion.For a table reconstructed from three views, lower scores identify low-quality table-top regions.
  • Replacing recurrent fusion with context-aware fusion increases LSM's ShapeNet testing IoU by about 2%.
  • The reconstruction results remain low resolution despite outperforming state-of-the-art methods.The authors propose introducing GANs to improve reconstruction resolution in future work.

5. Conclusion and Future Works

Pix2Vox is a unified and efficient framework for single-view and multi-view 3D reconstruction that outperforms state-of-the-art methods, while future work targets resolution and RGB-D inputs.

  • Pix2Vox unifies single-view and multi-view 3D reconstruction by fusing multiple decoder-produced coarse volumes.
  • Quantitative and qualitative evaluations on ShapeNet and Pix3D show large-margin improvements over state-of-the-art methods.
  • Pix2Vox is 24 times faster than 3D-R2N2 in backward inference time.
  • Future work will improve reconstructed-object resolution and extend Pix2Vox to RGB-D images.
Loading 1901.11153v2…