PaperlayerView exact revision on arXiv

Source-linked AI summary

4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks

Christopher Choy, JunYoung Gwak, Silvio Savarese

arXiv:1904.08755v4cs.CVcs.AI

TL;DR

The paper studies perception of 3D-video sequences composed of temporal 3D scans. It develops generalized sparse convolutions and 4D spatio-temporal networks, which outperform 2D networks and can be more robust to noise.

  • Problem

    The paper addresses perception of 3D-video sequences, including depth-camera videos and LIDAR scans.

  • Method

    The paper combines sparse tensors, generalized sparse convolutions, an auto-differentiation library, 4D convolutional networks, hybrid kernels, and a 7D trilateral-stationary conditional random field.

  • Results

    3D convolutional neural networks alone outperform 2D networks, while 4D perception is more robust to noise.

  • Takeaways & Limitations

    Generalized sparse convolutions support high-dimensional spatio-temporal perception, with 4D networks showing robustness to noise.

  • Takeaways & Limitations

    Temporal averaging introduces noise because the input point-cloud coordinates are noisy.

Abstract

from arXiv · show

In many robotics and VR/AR applications, 3D-videos are readily-available sources of input (a continuous sequence of depth images, or LIDAR scans). However, those 3D-videos are processed frame-by-frame either through 2D convnets or 3D perception algorithms. In this work, we propose 4-dimensional convolutional neural networks for spatio-temporal perception that can directly process such 3D-videos using high-dimensional convolutions. For this, we adopt sparse tensors and propose the generalized sparse convolution that encompasses all discrete convolutions. To implement the generalized sparse convolution, we create an open-source auto-differentiation library for sparse tensors that provides extensive functions for high-dimensional convolutional neural networks. We create 4D spatio-temporal convolutional neural networks using the library and validate them on various 3D semantic segmentation benchmarks and proposed 4D datasets for 3D-video perception. To overcome challenges in the 4D space, we propose the hybrid kernel, a special case of the generalized sparse convolution, and the trilateral-stationary conditional random field that enforces spatio-temporal consistency in the 7D space-time-chroma space. Experimentally, we show that convolutional neural networks with only generalized 3D sparse convolutions can outperform 2D or 2D-3D hybrid methods by a large margin. Also, we show that on 3D-videos, 4D spatio-temporal convolutional neural networks are robust to noise, outperform 3D convolutional neural networks and are faster than the 3D counterpart in some cases.

1. Introduction

The paper targets 3D-video perception with sparse, high-dimensional convolutions rather than frame-by-frame processing. It introduces generalized sparse convolutions, hybrid kernels, and a 7D conditional random field, reporting strong 3D and 4D results.

  • 3D-videos provide temporal sequences of 3D scans from depth cameras, LIDAR, or repeated MRI scans for robotics and AR/VR perception.
  • Existing 3D-video perception faces heterogeneous data representations, weaker or comparable 3D CNN performance, and limited open-source support for large-scale 3D data.
  • Sparse tensors and generalized sparse convolutions provide a unified framework for high-dimensional perception and an open-source autodifferentiation library.
  • 625 weights in a 4D tesseract with kernel size 5 illustrate the dimensionality cost that motivates non-hypercubic custom kernels.
  • The proposed 7D trilateral space-time-chroma conditional random field enforces spatio-temporal consistency through differentiable recurrent layers trained end-to-end.
  • 67.9% mIoU on ScanNet exceeded the best peer-reviewed work by 19% mIoU at submission, while 4D networks were more noise-robust and sometimes more efficient than 3D counterparts.

2. Related Work

Prior work addresses 3D perception with dense, surface, point-based, continuous, or limited 4D approaches. This paper instead uses homogeneous high-dimensional convolutions over sparse tensors, with arbitrary kernel shapes encompassing standard discrete convolutions.

  • 3D-convolutional neural networks: Prior 3D CNNs use dense grids or octrees, but dense representations consume memory and compute because most 3D scan space is empty.
  • Neural networks without 3D convolutions: Continuous-kernel methods avoid discrete grids but incur expensive KD-tree neighbor searches and sensitivity to uneven point-cloud distributions.
  • Neural networks without 3D convolutions: Surface methods apply 2D convolutions to thin observable surfaces, while PointNet-based methods use point features but can require cropping that limits receptive fields.
  • 4D perception: Earlier 4D methods include deformable balloon models, Markov random fields, and a 3D-UNet with a 1D autoencoder rather than 4D convolutions.
  • 4D perception: The proposed networks use homogeneous sparse representations and convolutions across 4D space-time and 7D space-time-chroma space instead of recurrent or shallow temporal components.
  • Generalized Sparse Convolution: The generalized sparse convolution supports arbitrary input and output coordinates and kernel offsets, including sparse submanifold, dense, strided, dilated, and hypercubic convolutions.

4. Minkowski Engine

The Minkowski Engine provides sparse-tensor data processing, generalized convolution, pooling, and standard neural-network operations for high-dimensional networks.

  • Library: The paper introduces an open-source auto-differentiation library implementing generalized sparse convolution and essential sparse-tensor functions.The library covers forward-pass operations and supports high-dimensional convolutional neural networks.
  • Sparse Tensor Quantization: Sparse-tensor quantization converts input coordinates, features, and optional labels into unique sparse coordinates and associated features.Coordinate collisions with multiple labels are marked with IGNORE_LABEL during GPU processing.
  • Generalized Sparse Convolution: Generalized sparse convolution uses explicit input-output kernel maps and arbitrary kernel offsets to compute sparse outputs.The same framework supports arbitrary output coordinates and transposed convolution by reversing input and output roles.
  • Max Pooling: Sparse max pooling reduces variable numbers of input features mapped to each output coordinate.Unlike dense tensors, sparse tensors can provide different numbers of inputs for different outputs, requiring explicit mappings and custom reduction.
  • Global / Average Pooling, Sum Pooling: Average and global pooling use sparse matrix multiplication, while sum pooling omits normalization to preserve density information.Average pooling divides pooled features by the number of mapped inputs; sum pooling does not.
  • Neural-Network Functions: Coordinate-free functions such as ReLU and batch normalization can operate directly on sparse feature matrices.Batch normalization treats each row of the feature matrix as a feature vector.

5. Minkowski Convolutional Neural Networks

Minkowski networks treat time as an additional spatial dimension for 4D perception, while specialized kernels and architectures address high-dimensional cost and structure. The hybrid kernel improves efficiency and accuracy over a tesseract kernel.

  • 4D Spatio-Temporal Networks: 4D spatio-temporal networks process 3D-video by treating time as an extra spatial dimension in high-dimensional convolutions.The approach introduces computational and consistency challenges specific to high-dimensional perception.
  • Tesseract Kernel and Hybrid Kernel: Non-conventional kernel shapes reduce high-dimensional memory and computation while improving performance.The generalized sparse convolution permits arbitrary kernel offsets and supports these kernel designs.
  • Tesseract Kernel and Hybrid Kernel: The hybrid kernel combines a spatial cubic kernel with a temporal cross-shaped kernel connecting the same spatial point across time.This design captures spatial geometry while limiting temporal connections.
  • Tesseract Kernel and Hybrid Kernel: The hybrid kernel outperforms the tesseract kernel in both speed and accuracy.The comparison concerns the proposed non-hypercubic kernel and the conventional 4D tesseract kernel.
  • Residual Minkowski Networks: Generalized sparse convolutions support high-dimensional residual and U-shaped architectures with arbitrary strides and kernel shapes.The residual design follows ResNet structure, while the U-shaped variant adds strided sparse convolutions, transpose convolutions, and skip connections.

6. Trilateral Stationary-CRF

The paper introduces a trilateral-stationary CRF in 7D space-time-chroma space to enforce spatio-temporal consistency. Variational inference converts its updates into differentiable 7D generalized sparse-convolution layers trained jointly with the 4D network.

  • The CRF models 3D space, 1D time, and 3D chromatic space, using color separation to prevent information from leaking across boundaries.
  • The pairwise potential is stationary in the 7D coordinate space, while camera extrinsics place spatial coordinates in the world system across time.
  • Variational inference uses a mean-field approximation because directly maximizing the CRF distribution is intractable.
  • The fixed-point update becomes a 7D generalized sparse convolution because stationary pairwise potentials encode neighborhood edges through N^7(x_i).
  • The recurrent implementation initializes unary logits, repeatedly applies sparse convolution and softmax updates, and returns the final distribution.
  • The unary and pairwise potentials are learned end-to-end using one 4D and one 7D Minkowski network.

7. Experiments

Experiments evaluate Minkowski networks on standard 3D indoor benchmarks and constructed 4D video datasets, including noisy inputs and runtime comparisons. The reported results show strong 3D segmentation performance, improved 4D robustness, and occasional speed gains over 3D processing.

  • Datasets and evaluation: Experiments use ScanNet and S3DIS for 3D segmentation, and Varcity and Synthia sequences for 4D spatio-temporal evaluation.Synthia sequences are formed from stereo RGB-D images back-projected into 3D.
  • Implementation and training: The implementation uses C++/CUDA with PyTorch, parallel data preparation, augmentation, and on-the-fly quantization.
  • Datasets and evaluation: Evaluation reports mean Intersection over Union and mean Accuracy, with voxel predictions propagated to points from the nearest voxel center.
  • 3D results: 3D benchmark results reached +19% mIoU on ScanNet and +7% on Stanford versus the best published works by the CVPR deadline.A later 2cm-voxel ScanNet training run achieved 72.1% mIoU.
  • 4D results: 4D networks were more robust to simulated noise, while adding less than 6.4% parameters over the 3D network and 6e-3% for TS-CRF.Noise included elastic distortion, Gaussian noise, and chromatic shifts.
  • 4D results: 4D processing could provide small speed gains over 3D processing in batch mode, particularly for large voxel sizes.Runtime was measured while varying voxel size and sequence length, including 4D networks with TS-CRF.

8. Conclusion

The paper introduces generalized sparse convolutions and an auto-differentiation library, then uses them for 4D spatio-temporal perception. Experiments show advantages for 3D-only networks and noise robustness in 4D perception.

  • The paper proposes generalized sparse convolution and an auto-differentiation library for sparse tensors.
  • The proposed tools support 4D convolutional neural networks for spatio-temporal perception.
  • Experiments show that 3D convolutional neural networks alone can outperform 2D networks.
  • Experiments show that 4D perception can be more robust to noise.

The supplementary material for 4D Spatio-Temporal ConvNets:

The supplementary material accompanies the paper on 4D spatio-temporal convolutional networks.

  • The supplementary material concerns 4D spatio-temporal convolutional networks.
  • The paper is titled Minkowski Convolutional Neural Networks.
  • The supplied section identifies supplementary material for the paper.

1. Minkowski Engine

The Minkowski Engine implements sparse tensor operations through quantization, hashing, coordinate management, and generalized sparse convolution support.

  • Sparse quantization finds unique voxels and assigns labels associated with them.Cells containing different labels receive IGNORE_LABEL, so the network excludes them from cross-entropy loss during training.
  • GPU sparse tensor quantization converts coordinates to quantized coordinates, hashes them, sorts entries, removes duplicate key-label pairs, and reduces labels by key.
  • The implementation uses a modified FNV64-1A hash function for int32 coordinates.
  • Generalized sparse convolution first defines output coordinates, reusing input coordinates except for strided operations that create new coordinates.Transposed strided convolutions preserve coordinate order by using the same coordinates and input stride.
  • Strided coordinate creation scales the input stride, maps coordinates to the resulting grid, and retains unique coordinates.

2. Derivation of the Trilateral Stationary-Conditional Random Field Fixed Point Update Equation

The TS-CRF fixed-point update is derived by replacing intractable optimization with variational mean-field inference. The approximation uses factorized distributions and divergence minimization.

  • TS-CRF optimization is intractable, so the method uses variational inference to approximate the optimal distribution.The approximation minimizes divergence D(Q∥P), described as the coding loss when using Q with P.
  • The derivation adopts a mean-field approximation because it has a closed-form solution.
  • The approximation factorizes into per-variable distributions Q_i(x_i).
  • The final result is expressed as a fixed-point update equation.

3. Experiments and Analysis

The experiments report semantic-segmentation evaluations across indoor, outdoor, synthetic, and reconstructed 4D datasets, with visualizations and per-class IoU analyses. RueMongue results saturate around 66% mIoU because the dataset is small.

  • RueMongue 2014 results saturate around 66% mIoU because the dataset is quite small for deep learning.
  • The paper provides per-class IoU numbers and qualitative visualizations of predictions, ground truth labels, and input point clouds.
  • The evaluation includes semantic-segmentation result tables for ScanNet, Stanford 3D Indoor Spaces, RueMongue, and 4D Synthia.
  • ScanNet benchmark entries note that test labels are not publicly available and identify methods using additional 2D images.
  • The visual analyses include noisy 4D Synthia predictions and MinkNet predictions on the ScanNet validation set.
Loading 1904.08755v4…