PaperlayerView exact revision on arXiv

Source-linked AI summary

Hierarchical Surface Prediction for 3D Object Reconstruction

Christian Häne, Shubham Tulsiani, Jitendra Malik

arXiv:1704.00710v2cs.CV

TL;DR

Coarse voxel grids limit CNN-based 3D reconstruction because high-resolution volumetric prediction is computationally expensive and poorly focuses learning on surfaces. HSP uses a three-label voxel block octree to refine surface regions hierarchically while reconstructing complete grids, and it reports more accurate results than low-resolution baselines across multiple input settings.

  • Problem

    CNN-based voxel predictors generally produce coarse grids because high-resolution volumetric prediction is computationally infeasible and increasingly dominated by non-surface voxels.

  • Method

    HSP predicts a three-label voxel block octree that recursively refines boundary regions while using coarse representations for interior and exterior space.

  • Results

    HSP outperforms low-resolution baselines across categories and evaluation metrics at 256^3 resolution, while producing more detailed surfaces.

  • Takeaways & Limitations

    High-resolution prediction is feasible from color images, depth images, and partial low-resolution voxel grids using little input data.

Abstract

from arXiv · show

Recently, Convolutional Neural Networks have shown promising results for 3D geometry prediction. They can make predictions from very little input data such as a single color image. A major limitation of such approaches is that they only predict a coarse resolution voxel grid, which does not capture the surface of the objects well. We propose a general framework, called hierarchical surface prediction (HSP), which facilitates prediction of high resolution voxel grids. The main insight is that it is sufficient to predict high resolution voxels around the predicted surfaces. The exterior and interior of the objects can be represented with coarse resolution voxels. Our approach is not dependent on a specific input type. We show results for geometry prediction from color images, depth images and shape completion from partial voxel grids. Our analysis shows that our high resolution predictions are more accurate than low resolution predictions.

I. INTRODUCTION

HSP addresses the computational and representational limits of coarse voxel prediction by predicting high-resolution geometry selectively around object surfaces. It achieves more accurate and visually detailed reconstructions across several input types.

  • CNN voxel predictors commonly use coarse grids because computational cost grows cubically with resolution.At higher resolutions, non-surface voxels increasingly dominate and contribute largely uninformative gradients.
  • The framework hierarchically generates surfaces from 16^3 to 256^3 resolution while passing feature channels between levels.The feature channels support both the current-level output and predictions at the next level.
  • HSP predicts high-resolution voxels near surfaces while representing most interior and exterior regions coarsely.Its three labels—free space, boundary, and occupied space—identify nodes requiring finer prediction.
  • HSP reduces computational cost enough to predict 256 × 256 × 256 voxel grids.
  • Quantitative evaluation finds HSP more accurate than upsampled low-resolution baselines, with qualitatively more detailed and higher-quality surfaces.

III. FORMULATION

The formulation begins with a conventional encoder-decoder voxel predictor and extends it toward hierarchical surface prediction. It addresses the high-resolution bottleneck by using three labels and an octree-based representation.

  • III. FORMULATION: The formulation first describes a coarse-resolution baseline before introducing the voxel block octree for prediction.
  • A. Voxel Prediction: The baseline encoder maps an input image, depth image, or partial voxel grid to a 128-dimensional shape code.
  • A. Voxel Prediction: An up-convolutional decoder transforms the shape code into a predicted voxel grid.
  • A. Voxel Prediction: The voxel prediction formulation uses volumetric up-convolutions and three labels: free space, boundary, and occupied space.
  • A. Voxel Prediction: Thresholding predicted occupancies or applying marching cubes at an iso value can convert voxel predictions into geometry.
  • A. Voxel Prediction: High-resolution uniform prediction is impractical because surface area grows quadratically while volume grows cubically with edge division.

B. Voxel Block Octree

The voxel block octree hierarchically partitions space into voxel blocks and refines only regions marked as boundary. Coarser predictions are upsampled elsewhere to reconstruct a complete high-resolution model.

  • The voxel block octree is an octree-based data structure predicted with an up-convolutional decoder.
  • An octree partitions 3D space recursively, with internal nodes subdividing a cube into up to eight child octants.
  • Each tree node contains a voxel block, and non-final levels classify its voxels as occupied, boundary, or free space.
  • Boundary labels signal that descendants contain both free and occupied space, so only those nodes require higher-resolution prediction.
  • Unpredicted high-resolution voxels are upsampled from the closest predicted resolution, preserving fine resolution near the surface.
  • The method can extract intermediate-resolution models by treating the boundary label as occupied space.

C. Network Architecture

HSP predicts voxel-block octrees through a hierarchical encoder–decoder that crops and upsamples features for child octants. It selectively refines regions likely to contain surfaces, while overlap improves smoothness and only a small fraction reaches the highest resolution.

  • Network Architecture: The encoder maps the input to a shape code, while the decoder predicts a voxel-block octree through hierarchical feature blocks.Inputs can be color images, depth images, or partial low-resolution voxel grids; the shape code has 128 dimensions.
  • Overlap: Padding creates overlap between neighboring predictions, producing smoother outputs and reducing visible grid patterns.Without overlap, predictions are less smooth and the individual voxel blocks are more apparent.
  • Feature Cropping: Feature cropping extracts the region centered on a child octant, reducing processing to information relevant for that child.Neighboring octants can yield overlapping extracted features.
  • Upsampling and Output Generation: Upsampling converts each cropped feature region into a higher-resolution feature block, which the output network maps to the child voxel block.The hierarchy repeats feature cropping, upsampling, and output generation across levels.
  • Adaptive Refinement: Child nodes are added when the corresponding octant has sufficient boundary evidence, allowing refinement near predicted surfaces.The maximum boundary response in the octant is compared with a threshold γ.
  • Adaptive Refinement: About 5 to 10% of voxels are predicted at high resolution for aeroplanes, chairs, and cars, reducing the number of voxels requiring fine prediction.The remaining regions are represented at coarser levels of the hierarchy.

D. Efficient Training with Subsampling

Training traverses only a subsampled set of hierarchical child nodes because full-tree evaluation is slow and early boundary predictions are unreliable. The traversal probability can be fixed or increased during training, with gradients accumulated across the minibatch.

  • Motivation: Full-tree training is slow, taking approximately 0.7 seconds for a single forward pass, and early boundary predictions are inaccurate.These issues motivate subsampling child nodes during training.
  • Subsampled Traversal: During depth-first traversal, a child node is visited with probability ρ whenever its octant contains a predicted boundary.The schedule for ρ may remain fixed or gradually increase from a low initial value.
  • Optimization: Gradients from traversing the subsampled trees are summed, and one gradient step is taken after all examples in the minibatch are processed.The model uses cross-entropy loss.

IV. EXPERIMENTS

Experiments evaluate HSP on ShapeNet-Core using aeroplane, chair, and car categories, with high-resolution voxelization and multiple input settings. The study compares HSP with hard and soft low-resolution baselines and includes RGB, depth, and partial-volume applications.

  • Dataset and Setup: The evaluation uses ShapeNet-Core CAD models from three categories: aeroplanes, chairs, and cars.The system is implemented in Torch and optimized with Adam using minibatches of four.
  • Ground-Truth Voxelization: Ground-truth geometry is voxelized at 256^3 resolution using a multiscale procedure that fills interiors from a 32^3 representation.The procedure fixes high-resolution voxels inside eroded low-resolution occupied regions as occupied and voxels inside low-resolution free space as free.
  • Ground-Truth Voxelization: The hierarchy labels boundary voxels at each level and uses free or occupied labels elsewhere, with the coarsest resolution set to 16^3.These labels provide supervision for the remaining pyramid levels.
  • Baselines: LR Hard labels a low-resolution voxel occupied when it contains at least one high-resolution occupied voxel, whereas LR Soft assigns the ratio of free to occupied voxels.Both baselines use uniform low-resolution prediction, generally at 32^3.
  • Input Data: RGB and depth inputs are rendered from CAD models at random viewpoints, with uniformly sampled azimuth and elevation ranges and randomized RGB lighting.Ten images are rendered for each CAD model.
  • Input Data: Partial voxel-grid completion is simulated by randomly zeroing half of an LR Soft ground-truth volume and predicting the complete high-resolution geometry.This setting represents multiple depth maps obtained from one side of an object.

D. Quantitative Evaluation

The quantitative evaluation measures RGB-based reconstruction with IoU and symmetric Chamfer Distance across aeroplanes, chairs, and cars. HSP matches low-resolution performance at 64^3 and outperforms the baselines at 256^3, where uniform prediction cannot scale similarly.

  • Evaluation Setup: RGB-image reconstruction is evaluated using Intersection over Union and symmetric Chamfer Distance on category-specific train, validation, and test splits.The splits assign models with probabilities 70%, 10%, and 20%, respectively.
  • Coarse Resolution: At 64^3 resolution, HSP performs similarly to LR Hard; on chair validation data, LR Hard reaches 43.47% IoU versus 43.12% for HSP.Both methods are trained and evaluated at the same coarse resolution.
  • Qualitative Results: Qualitative comparisons for chairs, aeroplanes, and cars show HSP alongside LR Soft and LR Hard reconstructions from color images.The examples are selected to illustrate within-category shape variability.
  • High Resolution: At 256^3 resolution, HSP outperforms the low-resolution baselines across all categories and evaluation metrics.Thresholds are selected per category and metric using the validation set before testing.

E. Qualitative Evaluation

HSP produces high-quality, detailed surfaces across three categories and generalizes from synthetic training data to external car images. The paper concludes that high-resolution predictions are more accurate and qualitatively superior to low-resolution baselines.

  • E. Qualitative Evaluation: HSP generates high-quality surfaces across aeroplane, car, and chair geometry predictions from single color images.The comparison includes LR Soft and LR Hard baselines.
  • E. Qualitative Evaluation: LR Hard produces visually pleasant chair reconstructions but lacks detailed results for aeroplanes and cars.
  • E. Qualitative Evaluation: LR Soft often misses thin structures because it uses fractional labels.
  • E. Qualitative Evaluation: A car-prediction network trained only on synthetic data generalizes to two white-background car images downloaded from the Internet.The paper also trains a network to complete a whole high-resolution car from a partial low-resolution voxel grid.
  • E. Qualitative Evaluation: The quantitative evaluation finds high-resolution predictions more accurate than low-resolution baselines, while qualitative results show superior predicted surfaces.

APPENDIX A. DETAILED CNN ARCHITECTURES

The appendix defines CNN architecture notation and identifies convolution and leaky ReLU components.

  • APPENDIX A. DETAILED CNN ARCHITECTURES: Conv denotes convolution in the network architecture descriptions.
  • APPENDIX A. DETAILED CNN ARCHITECTURES: LR denotes leaky ReLU with a negative slope of 0.2.

R ReLU

The architecture notation specifies dropout, kernel sizes, strides, output channels, and output dimensions used in the CNN descriptions.

  • R ReLU: DO denotes dropout with probability 0.5.
  • R ReLU: kW, kH, and kD denote kernel sizes in the three dimensions.
  • R ReLU: sW, sH, and sD denote strides in the three dimensions.
  • R ReLU: oC denotes the number of output feature channels.
  • R ReLU: oW, oH, and oD denote output sizes in the three dimensions.
  • R ReLU: The proposed HSP decoder uses modules for bottleneck expansion, level transitions, outputs, and the final level output.

APPENDIX B. QUALITATIVE RESULTS

The appendix presents randomly selected validation reconstructions across aeroplanes, cars, and chairs, alongside the architecture tables and figure-based comparisons of HSP with low-resolution baselines.

  • APPENDIX B. QUALITATIVE RESULTS: Randomly selected validation examples are shown every twentieth element for aeroplanes, cars, and chairs.Aeroplanes use Figures 12–13, cars Figures 14–16, and chairs Figures 17–18.
  • APPENDIX B. QUALITATIVE RESULTS: The appendix includes tables for the color/depth encoder, baseline decoder, HSP decoder, upsampling, output, and final-level output modules.
  • APPENDIX B. QUALITATIVE RESULTS: Figures 12 and 13 compare input images, HSP, LR Soft, and LR Hard for aeroplanes.
  • APPENDIX B. QUALITATIVE RESULTS: Figures 14–16 compare input images, HSP, LR Soft, and LR Hard for cars.
  • APPENDIX B. QUALITATIVE RESULTS: Figures 17 and 18 compare input images, HSP, LR Soft, and LR Hard for chairs.
Loading 1704.00710v2…