PaperlayerView exact revision on arXiv

Source-linked AI summary

OctNet: Learning Deep 3D Representations at High Resolutions

Gernot Riegler, Ali Osman Ulusoy, Andreas Geiger

arXiv:1611.05009v4cs.CV

TL;DR

Existing 3D networks are constrained by the cubic memory and computation costs of dense voxel grids, limiting high-resolution learning. OctNet uses unbalanced octrees with pooled leaf features to exploit sparsity, and experiments show that higher resolution is especially valuable for orientation estimation and point-cloud labeling.

  • Problem

    Dense 3D networks are typically limited to coarse resolutions because computational and memory requirements grow cubically, restricting detailed high-resolution learning.

  • Method

    OctNet hierarchically partitions sparse 3D space with unbalanced octrees and stores pooled feature representations at leaf nodes for direct network operations.

  • Results

    Across classification, orientation estimation, and point-cloud labeling, OctNet enables higher resolutions, matches equivalent dense networks at lower resolutions, and accelerates processing at 128^3 and above.

  • Takeaways & Limitations

    High resolution provides limited additional benefit for ModelNet10 classification but matters for 3D orientation estimation and semantic point-cloud labeling.

Abstract

from arXiv · show

We present OctNet, a representation for deep learning with sparse 3D data. In contrast to existing models, our representation enables 3D convolutional networks which are both deep and high resolution. Towards this goal, we exploit the sparsity in the input data to hierarchically partition the space using a set of unbalanced octrees where each leaf node stores a pooled feature representation. This allows to focus memory allocation and computation to the relevant dense regions and enables deeper networks without compromising resolution. We demonstrate the utility of our OctNet representation by analyzing the impact of resolution on several 3D tasks including 3D object classification, orientation estimation and point cloud labeling.

1. Introduction

OctNet addresses the cubic cost of dense 3D voxel grids by exploiting sparse 3D structure, enabling deep networks at higher resolutions. It uses unbalanced octrees to focus computation and memory on relevant regions while maintaining performance across several 3D tasks.

  • Dense 3D networks are typically limited to roughly 30^3 voxels because memory and computation grow cubically with resolution.
  • 3D data is often sparse, and intermediate activations in a voxelized-shape classifier concentrate near object boundaries.
  • OctNet hierarchically partitions 3D space with unbalanced octrees whose variable-size leaves store pooled feature activations.
  • OctNet defines convolution, pooling, and unpooling directly on the tree structure, reducing memory and computation for high-resolution inputs.
  • Across classification, orientation estimation, and semantic point-cloud segmentation, OctNet supports higher resolutions, matches dense low-resolution performance, and speeds processing at 128^3 and above.

2. Related Work

Prior 3D models largely use dense, coarse voxel grids because of computational and memory limits. Sparse approaches reduce some costs but remain constrained in depth, operations, resolution, or compatibility with convolutional networks.

  • Dense Models: Most dense 3D methods process regular voxel grids and therefore operate at coarse resolutions, typically around 30^3 voxels.
  • Dense Models: High-resolution outputs in prior dense methods can require inefficient sliding-window techniques with limited receptive fields.
  • Dense Models: Naively increasing resolution reduces network depth and expressiveness, whereas OctNet supports deep architectures at substantially higher resolutions.
  • Sparse Models: Sparse convolutions can reduce convolution counts but may leave memory requirements unchanged, restricting prior networks to shallow models of up to three layers.
  • Sparse Models: Other sparse approaches are limited by small convolution sizes, indexing overhead, rising operations, or resolutions no higher than 80^3 voxels.
  • Sparse Models: Field probing reduces memory and computation but cannot stack, convolve, or pool its probing layers, limiting the use of convolutional-network computation.

3. Octree Networks

OctNet uses a hybrid grid-octree representation to allocate features adaptively while retaining efficient access and direct implementations of convolutional-network operations. Shallow octrees are encoded compactly, and convolution, mapping, pooling, and unpooling operate on the structure without dense tensor conversion.

  • Hybrid Grid-Octree Data Structure: Octrees recursively subdivide cells containing relevant information, using small cells for dense regions and large cells for empty regions.
  • Hybrid Grid-Octree Data Structure: The hybrid structure places shallow octrees on a regular grid, limiting tree depth to avoid the traversal costs of a single deep octree.A shallow octree of depth 3 can represent an empty region with one vector instead of 512 finest-resolution voxel vectors.
  • Hybrid Grid-Octree Data Structure: Bit strings encode shallow-octree structure compactly, with 73 bits representing a complete depth-3 tree and bit counting enabling efficient feature-array offsets.Increasing depth causes exponential growth in structure bits and increases cell traversal time.
  • Network Operations: OctNet maps between grid-octrees and tensors by assigning each tensor location the smallest containing octree cell and pooling covered tensor voxels in the reverse direction.A single octree voxel can cover up to 512 tensor elements, so reverse mapping requires pooling.
  • Network Operations: Convolution is implemented directly on the hybrid structure: constant-valued voxel interiors are evaluated once, while boundaries incorporate neighboring cells.The direct implementation avoids costly conversion to and from dense 3D tensors.
  • Network Operations: Pooling combines eight neighboring shallow octrees into one, whereas unpooling spawns eight shallow octrees from depth-zero nodes and increases the size of other voxels.These operations change the grid-octree structure rather than merely applying dense-grid arithmetic.

4. Experimental Evaluation

OctNet experiments evaluate how resolution affects 3D classification, orientation estimation, and point-cloud labeling while comparing sparse and dense networks. Higher resolution is especially valuable for orientation estimation and semantic segmentation, whereas classification shows diminishing returns beyond 32^3 voxels.

  • Experimental setup: The evaluation isolates input resolution across 3D classification, orientation estimation, and semantic segmentation tasks.The experiments use simple architectures to separate resolution effects from orthogonal improvements such as augmentation or ensemble learning.
  • Experimental setup: OctNet uses a standalone C++/CUDA implementation of its grid-octree structure and network layers, evaluated with Torch3.
  • 3D classification: At 256^3 resolution, OctNet fits a batch of 32 samples in 12GB GPU memory, while the dense model fits only resolutions ≤64^3.OctNet also runs faster than dense counterparts above 64^3, but is slightly slower at lower resolutions because of grid-octree overhead.
  • 3D classification: OctNet performs on par with its dense equivalent and both outperform the shallower VoxNet architecture, indicating the importance of network depth.
  • 3D classification: Classification accuracy improves at lower resolutions but shows diminishing returns beyond 32^3 voxels, with higher resolution helping some classes while leaving others ambiguous.Bathtubs become more distinguishable from similar classes at higher resolution, whereas dresser versus night stand remains ambiguous.
  • 3D orientation estimation: Orientation estimation benefits from fine details: performance steadily increases for OctNet architectures, while constant-capacity networks level off beyond 128^3 input resolution.At higher resolutions, predictions cluster around the true pose; the dense representation is not feasible at 128^3 or beyond.
  • 3D semantic segmentation: Increasing input resolution is essential for state-of-the-art semantic point-cloud labeling because finer details vanish at coarser resolutions.

5. Conclusion and Future Work

OctNet makes high-resolution 3D deep learning tractable and shows that resolution matters differently across tasks. The authors identify orientation estimation and point-cloud labeling as benefiting from higher input or output resolution.

  • OctNet makes deep learning with high-resolution 3D inputs tractable.
  • Low-resolution networks are sufficient for ModelNet10 classification.
  • High input and output resolution matters for 3D orientation estimation and 3D point-cloud labeling.
  • The authors expect OctNet to support improvements as research moves toward high-resolution, large-scale 3D data.
  • They identify multi-view 3D reconstruction as a promising application requiring high-resolution voxelized shapes.

A.1. Introduction

The supplemental material documents OctNet’s data structure, efficient operations, additional experiments, and network architectures. It also explains the data index and convolution implementation for the grid-octree structure.

  • The supplement details operations on the hybrid grid-octree data structure.
  • Section A.2 explains the data index for efficiently accessing data in the grid-octree structure.
  • Section A.3 provides additional insight into efficient convolution on octrees.
  • Section A.4 presents further quantitative and qualitative results.
  • Section A.8 specifies the network architectures used in the experiments.

A.2. Data Index

The data index maps bit-string positions in breadth-first octrees to compact leaf-data offsets. Its counting procedure excludes split-node storage and can be verified with a concrete quadtree example.

  • Leaf data are stored in a compact array aligned breadth-first, while only leaf nodes carry associated data.
  • The offset calculation counts nodes up to a voxel and subtracts split nodes before it because split nodes store no data.
  • A voxel is uniquely identified by its bit index within the octree’s bit-string representation.
  • In the quadtree example, bit index 51 maps to data index 13 through counts of preceding nodes, split nodes, and sibling position.
  • The quadtree illustration uses a bit string and marks split nodes and leaves with associated data.

A.3. Efficient Convolution

OctNet accelerates convolution on large octree cells by exploiting constant interior values and handling changes primarily near voxel surfaces. The resulting computation substantially reduces multiplications relative to naive evaluation.

  • A naive convolution on an 8^3 cell with a 3^3 kernel requires 13,824 multiplications.
  • The efficient implementation evaluates the constant cell interior once and separately processes corners, edges, and faces.
  • 3,203 multiplications represent 23.17% of the naive implementation’s required multiplications.

A.4. Additional Results

This section presents additional quantitative and qualitative results for 3D shape classification, orientation estimation, and semantic 3D point labeling.

  • Additional experiments cover 3D shape classification, 3D orientation estimation, and semantic 3D point labeling.

A.5. 3D Classification

The ModelNet40 experiments examine OctNet’s memory advantage across resolutions and occupancies, showing that sparsity-aware storage is beneficial even at low resolutions.

  • Even at 8^3 resolution, OctNet’s data structure uses less memory than dense representation up to 50% occupancy.
  • As voxel resolution increases, occupancy decreases because the data becomes sparser.
  • OctNet exploits input sparsity to achieve a significant reduction in memory consumption.

A.6. 3D Orientation Estimation

Additional orientation experiments show that OctNet handles 512^3 inputs, with performance increasing as input resolution rises. The section also describes the evaluated architectures and qualitative labeling results.

  • A.6. 3D Orientation Estimation: 512^3 input resolution demonstrates that OctNet can handle resolutions larger than 256^3 for 3D orientation estimation.
  • A.6. 3D Orientation Estimation: Orientation-estimation performance increases with increasing input resolution.
  • A.6. 3D Orientation Estimation: The additional experiments include Biwi Kinect head-pose estimation using annotated 3D rotations from depth-camera sequences.
  • A.8. Network Architecture Details: The architecture details define 3 × 3 convolutional layers, max-pooling operations, and ReLU activations for the evaluated networks.
  • A.8. Network Architecture Details: Classification and orientation estimation use architecture classes with either resolution-dependent parameter counts or fixed parameter counts.
  • A.8. Network Architecture Details: Semantic point labeling uses a U-Net architecture whose concatenation layer combines decoder and encoder outputs to preserve details.
Loading 1611.05009v4…