PaperlayerView exact revision on arXiv

Source-linked AI summary

Octree Generating Networks: Efficient Convolutional Architectures for High-resolution 3D Outputs

Maxim Tatarchenko, Alexey Dosovitskiy, Thomas Brox

arXiv:1703.09438v3cs.CV

TL;DR

High-resolution 3D decoders on regular voxel grids become infeasible because computation and memory scale cubically. This paper introduces an octree-based convolutional decoder that matches dense decoders’ accuracy while scaling better in memory and runtime to higher resolutions.

  • Problem

    Regular-grid volumetric decoders become infeasible for high-resolution 3D outputs because their computational and memory requirements scale cubically.

  • Method

    The octree generating network predicts both adaptive octree structure and cell values, processing only regions containing fine details at high-resolution stages.

  • Results

    OGNs achieve the same accuracy as dense voxel-grid decoders while using less memory and runtime, including single-pass generation at 512^3-voxel resolution.

  • Takeaways & Limitations

    Octree decoding makes end-to-end deep learning for high-resolution 3D shape and scene tasks feasible within improved memory and runtime scaling.

  • Takeaways & Limitations

    The paper focuses on generating 3D shapes and does not implement an octree-based convolutional encoder.

Abstract

from arXiv · show

We present a deep convolutional decoder architecture that can generate volumetric 3D outputs in a compute- and memory-efficient manner by using an octree representation. The network learns to predict both the structure of the octree, and the occupancy values of individual cells. This makes it a particularly valuable technique for generating 3D shapes. In contrast to standard decoders acting on regular voxel grids, the architecture does not have cubic complexity. This allows representing much higher resolution outputs with a limited memory budget. We demonstrate this in several application domains, including 3D convolutional autoencoders, generation of objects and whole scenes from high-level representations, and shape from a single image.

1. Introduction

The paper introduces Octree Generating Networks, convolutional decoders that replace dense grids with octrees to generate high-resolution volumetric shapes efficiently. OGNs preserve dense-decoder accuracy while reducing memory use and increasing speed at high resolutions.

  • Motivation: Regular-grid volumetric decoders have cubic computational and memory requirements, making high-resolution 3D generation difficult.Up-convolutional decoders progressively increase resolution on regular grids, and their volumetric generalization inherits cubic scaling.
  • Motivation: Octrees reduce computation and memory by representing uniform neighboring voxels with large adaptive cells while retaining fine details through smaller cells.This structure is especially useful for binary occupancy maps and multi-class three-dimensional labeling.
  • Method: OGN is a convolutional decoder that gradually refines coarse structure and replaces dense grids with octrees during decoding.The network predicts large uniform output regions at early decoding stages using learned filters and up-sampling.
  • Evaluation: OGNs are evaluated against standard dense networks on shape auto-encoding, high-level shape generation, and single-image 3D reconstruction.The paper focuses on shapes represented as binary occupancy maps.
  • Results: 512^3 voxels can be generated in a single forward pass, while matching conventional dense decoders’ accuracy with substantially lower memory use and faster high-resolution inference.The OGN implementation is publicly available.

2. Related work

Prior deep-learning methods commonly generate volumetric data on voxel grids, but their computational limitations constrain output resolution. OGN instead predicts where high-resolution information is needed and focuses computation there, drawing on sparse representations while differing from related octree methods that assume test-time structure.

  • Voxel-grid methods: Voxel-grid convolutional networks support many 3D applications, but their output resolution is fundamentally limited.These applications include object reconstruction, probabilistic shape generation, semantic segmentation, and shape deformation.
  • Alternative representations: Computational limitations of voxel grids motivated alternative volumetric representations, including multi-view fusion and two-dimensional geometry images.The cited multi-view fusion approach is not end-to-end trainable and does not handle strong self-occlusion.
  • Sparse representations: OGN is inspired by sparse convolutional networks that store sparse non-trivial features instead of dense feature maps for efficient shape analysis.Geodesic CNNs are also cited as an alternative explored for extracting local features in non-Euclidean domains, although such representations are not directly applicable to generating 3D outputs.
  • OGN: OGN predicts which output regions contain high-resolution information and restricts extensive calculations to those regions.This formulation is described as solving the inverse problem of identifying where high-resolution output information is needed.
  • Octree methods: Riegler et al. also operate on octrees rather than voxel grids, but their method assumes the octree structure is known at test time.The paper distinguishes OGN from this work in implementation and application range; the cited assumption applies, for example, to semantic segmentation.

3. Octrees

Octrees represent 3D grids with adaptive cell sizes by recursively partitioning space into eight octants, enabling lossless memory reduction relative to regular voxel grids. Their hash-table representation supports constant-time access, while comparing differently structured octrees requires resolving values across levels.

  • Octree representation: Octrees use adaptive cell sizes and recursively partition a root cell into eight octants, enabling lossless memory reduction versus regular voxel grids.A voxel-grid function can be converted into an octree through recursive subdivision.
  • Octree representation: Storing octree cell-value pairs in a hash table enables constant-time element access.The octree is represented as a set of pairs, although the passage truncates their full definition.
  • Octree comparison: When comparing differently structured octrees, a cell’s value is uniquely determined if the corresponding signal is stored at the same or an earlier level.If the corresponding value is stored at a later level, the cell is subdivided and its whole-cell value is undefined.

4. Octree Generating Networks

The Octree Generating Network is an end-to-end trainable convolutional decoder that predicts both octree structure and cell signals through sparse, hash-table-based octree blocks. Its blocks classify cells as empty, filled, or mixed and propagate unresolved features across levels using training and testing regimes suited to whether the tree structure is available.

  • Architecture: OGN decodes an octree by predicting which cells subdivide and the signal value of each cell, using a dense-grid block followed by hash-table-based octree blocks.The framework concentrates on binary occupancy values but can be extended to arbitrary signals.
  • Octree blocks: Each octree block predicts one level by classifying cells as “empty,” “filled,” or “mixed” with a three-way softmax.“Empty” and “filled” provide signal values, while “mixed” indicates that the value remains undetermined and requires further processing.
  • Training objective: The final objective sums cross-entropy losses computed across all octree levels from predictions of empty, filled, and mixed cell states.Losses are averaged over the cells predicted at each level and implemented in the custom OGNLoss layer.
  • Feature propagation: OGN processes sparse feature maps as hash tables and propagates mixed-cell features to subsequent blocks, adding neighboring cells when later convolutions require them.The custom OGNConv layer supports strided convolutions and up-convolutions with arbitrary filter sizes.
  • Propagation regimes: When the test-time tree structure is unavailable, OGN is trained with Prop-known and then fine-tuned with Prop-pred; alternative combinations performed worse.When the structure is available at test time, the network uses Prop-known for both training and testing.

5. Experiments

The experiments show that OGNs retain competitive reconstruction quality while substantially reducing memory and computation, especially at high resolutions. They also demonstrate effective 3D shape generation, interpolation, scene reconstruction, and single-view reconstruction across multiple datasets and resolutions.

  • Efficiency: OGNs match dense voxel networks at low resolutions but become substantially faster and more memory-efficient as output resolution increases.At 512^3 voxel resolution, OGN uses almost two orders of magnitude less memory and runs 20 times faster.
  • Efficiency: Training at 256^3 resolution takes approximately 5 days for OGN versus almost a month estimated for the dense counterpart.A dense 512^3 network, even if it fit in memory, would take many months to train.
  • Shape reconstruction: With predicted octree structure, OGN performance shows no significant difference from the dense baseline, while known tree structure outperforms dense decoding.Known structure provides additional shape information that simplifies the decoder’s learning problem.
  • Scene generation: Scene reconstruction benefits from higher output resolutions because 64^3 and 128^3 are inadequate for details, while even 256^3 can struggle with fine-grained structure.The results support using OGNs for high-resolution scene reconstruction.
  • Single-view reconstruction: For single-view 3D reconstruction on ShapeNet-all, both OGN and the dense baseline compare favorably with Choy et al. for most object classes.The experiment used the same dataset and 32^3 resolution as Choy et al. for a fair comparison.

6. Conclusions · Appendix

The paper concludes that its octree-based convolutional decoder generates high-resolution 3D outputs efficiently, matching low-resolution dense-grid accuracy while scaling better in memory and runtime. It also enables previously infeasible 3D learning tasks and can extend beyond binary occupancy to richer octree-attached outputs.

  • 6. Conclusions: The proposed decoder generates high-resolution 3D outputs represented as octrees.The architecture is convolutional and supports flexible layer configurations.
  • 6. Conclusions: The architecture provides the same accuracy as dense voxel grids at low resolution.This establishes comparable low-resolution reconstruction accuracy with dense voxel representations.
  • 6. Conclusions: At higher resolutions, the octree architecture scales better than dense voxel grids in memory and runtime.Its improved scaling is attributed to both memory and runtime efficiency.
  • 6. Conclusions: The architecture enables end-to-end deep learning for tasks previously considered unfeasible.The conclusion specifically identifies 3D shape learning as a beneficiary of this capability.
  • 6. Conclusions: 3D object and scene reconstruction are highlighted as tasks likely to benefit from the architecture.These applications involve learning tasks over 3D shapes.
  • 6. Conclusions: The framework can be extended from binary occupancy maps to multidimensional outputs attached to the octree structure.Examples include textured shapes and signed distance functions, enabling a wider range of applications.

A. Computational efficiency

The octree-based decoder scales substantially better than dense voxel decoders in memory consumption and computation time. In the evaluated scene, dense networks scale almost cubically from 64^3 resolution, whereas OGN scales better than quadratically, though octree scaling depends on the data.

  • Overall comparison: OGN networks scale much better than dense counterparts in both memory consumption and computation time on the BlendSwap “houses” scene.The measurements compare practical architectures and use the “houses” scene from the BlendSwap dataset.
  • Experimental setup: Slim OGN and dense decoder networks were designed to fit on a GPU at every tested resolution, including 512^3.The slim networks used one channel per convolutional layer and a 64-unit fully connected encoder layer; measurements appear in Figures 9 and 10.
  • Experimental setup: Memory scaling was quantified after subtracting caffe’s constant 190 MB GPU reservation from all measurements.The networks used batch size 1 and measured forward and backward passes.
  • Scaling behavior: Dense networks scale almost cubically in runtime and memory consumption starting from 64^3 voxel resolution.The comparison uses increasing output resolution as the basis for assessing scaling.
  • Scaling behavior: For this example, OGN scales better than quadratically, while octree scaling generally depends on the applied data.The reported scaling behavior is specific to the particular example and is not claimed as universal.

B. Train/test modes · C. Feature propagation

The paper evaluates how Prop-known and Prop-pred should be paired between training and testing, finding that matching modes gives the best results. It also explains how OGNProp automatically determines which neighboring features must be propagated from the architecture and filter configuration.

  • B. Train/test modes: When the output structure is unknown at test time, networks are trained to convergence with Prop-known and then fine-tuned with Prop-pred.Without fine-tuning, performance decreases, especially with larger convolutional filters, because training did not expose the network to erroneous propagations.
  • B. Train/test modes: Matching the training and testing propagation modes produces the best reconstruction results.Prop-known is preferred when the output structure is known at test time, while Prop-pred fine-tuning is needed when it is unknown.
  • B. Train/test modes: When the output structure is known at test time, training and testing with Prop-known is best, whereas additional Prop-pred fine-tuning slightly hurts performance.These findings correspond to lines 1 and 3 in Table 7.
  • C. Feature propagation: With 23 up-convolutions of stride 2, no neighboring features need to be propagated in the described configuration.The one-dimensional illustration shows a single predicted mixed input cell and the output cells affected by it.
  • C. Feature propagation: The number of features propagated by an OGNProp layer depends on convolutional filter sizes in all subsequent blocks.The required propagation is determined from the network architecture before training begins.
  • C. Feature propagation: Using 4^3 filters and stride 2 followed by 3^3 filters and stride 1 requires propagating neighboring cells in addition to the predicted mixed cell.With this configuration, two output cells are affected by four input cells.

D. 3D shape from high-level information: additional experiments · D.1. MPI-FAUST

The MPI-FAUST experiments showcase OGNs at higher resolutions on high-resolution human-body scans. These models contain finer details than ShapeNet cars and therefore benefit from higher resolution, although the networks cannot generalize to new samples because of limited training data.

  • D. 3D shape from high-level information: additional experiments: OGNs were trained to fit the MPI-FAUST dataset as an additional higher-resolution experiment.The experiment was designed to showcase the benefit of using higher resolutions.
  • D.1. MPI-FAUST: MPI-FAUST contains 300 high-resolution scans of human bodies.The scans cover 10 different people and 30 different poses.
  • D.1. MPI-FAUST: The dataset covers 10 different people.These people are represented across the MPI-FAUST human-body scans.
  • D.1. MPI-FAUST: The scans represent 30 different poses.MPI-FAUST provides these poses across its human-body examples.
  • D. 3D shape from high-level information: additional experiments: Because of the low amount of training data, the trained networks cannot generalize to new samples.The passage states the same limitation for the BlendSwap experiments.
  • D.1. MPI-FAUST: Figure 12 and Table 8 provide qualitative and quantitative MPI-FAUST results, respectively.The passage identifies Figure 12 as qualitative and Table 8 as quantitative evidence.
  • D.1. MPI-FAUST: MPI-FAUST human models include finer details than ShapeNet cars.The finer details are given as the reason these human models benefit from higher resolution.
  • D.1. MPI-FAUST: The human models therefore benefit from higher resolution.This benefit follows from their finer details compared with cars from ShapeNet.

D.2. Fitting reduced ShapeNet-cars · E. Shift invariance

The paper tests whether higher-resolution OGN models can avoid performance degradation when trained on reduced ShapeNet-Cars, and finds that higher resolution improves detail. It also evaluates shift robustness, finding OGN comparable to a dense decoder despite octree structure changes under shifts.

  • D.2. Fitting reduced ShapeNet-cars: Higher resolution was beneficial on the reduced dataset, with models producing more fine details.Quantitative results across resolutions and comparisons with the full dataset were reported in Table 9; Figure 13 provided visual support.
  • D.2. Fitting reduced ShapeNet-cars: These results support the hypothesis that the higher-resolution performance drop is not caused by resolution itself.The conclusion follows from the reduced-dataset experiment, where higher resolution improved performance and detail.
  • E. Shift invariance: Unlike voxel-grid convolutions, octree convolutions are not perfectly shift invariant because a one-pixel shift can substantially alter octree structure.The study compared OGN and dense decoders using fully convolutional autoencoders trained on 64^3 models with lowest feature-map resolution 4^3.
  • E. Shift invariance: The shift experiment trained both autoencoders on non-shifted ShapeNet-Cars and tested them on models shifted by varying voxel counts along the z-axis.With a lowest feature-map resolution of 4^3, the networks were expected to be invariant to shifts of 16 voxels.
  • E. Shift invariance: There was no significant performance difference between OGN and the dense network on shifted data.Although OGN is not shift invariant by design, the results indicate robust performance under the tested shifts.
  • E. Shift invariance: Different training-model octree structures may act as an implicit regularizer, allowing the network to learn shapes while remaining robust to exact structure.This explanation was proposed for OGN’s comparable performance with the dense decoder.

F. Network architectures … F.3. Single-image 3D reconstruction

The paper specifies the exact network architectures used in its evaluations, covering OGN autoencoders, high-level 3D shape generation, and single-image 3D reconstruction. These designs vary with representation resolution, dataset, and encoder-decoder configuration.

  • F. Network architectures: The paper provides exact network architectures for its experimental evaluations.This architecture description spans the network-design sections covered here.
  • F.1. Autoencoders: OGN autoencoders use OGN-specific architectures, while dense baselines retain the same layer configurations with usual convolutions and predict only after the final layer.All networks were trained with batch size 16.
  • F.2. 3D shape from high-level information: Shapenet-cars OGN decoders receive encoded representations whose final encoder-layer size matches the corresponding decoder input size.The encoders consist of three fully-connected layers.
  • F.2. 3D shape from high-level information: FAUST and BlendSwap 256^3 output octrees use four levels, replacing one octree block with an additional deconvolution-convolution pair in the dense block.This differs from the five-level configuration in Table 13.
  • F.2. 3D shape from high-level information: The BlendSwap 512^3 decoder adds one octree block with 32 output channels.This is an additional configuration change for the higher-resolution decoder.
  • F.3. Single-image 3D reconstruction: Single-image 3D reconstruction reuses the decoder architectures from Table 13 and uses the convolutional encoder specified in Table 11.Table 11 presents the convolutional encoder for this experiment, while Table 13 presents the decoder architectures.
  • F.1. Autoencoders: Table 12 documents the OGN autoencoder architectures and defines OGNConv as up-convolution and OGNConv* as convolution.Layer annotations such as →lk indicate the octree level predicted by a classifier attached to that layer.
Loading 1703.09438v3…