PaperlayerView exact revision on arXiv

Source-linked AI summary

MeteorNet: Deep Learning on Dynamic 3D Point Cloud Sequences

Xingyu Liu, Mengyuan Yan, Jeannette Bohg

arXiv:1910.09165v2cs.CVcs.LGcs.RO

TL;DR

Dynamic 3D point-cloud sequences are difficult to learn because they are irregular, extend beyond short frame pairs, and grid-based processing introduces quantization and convolution inefficiencies. MeteorNet directly processes these sequences by aggregating spatiotemporal neighborhoods with two motion-aware grouping strategies. It outperforms grid-based and prior short-sequence baselines across recognition tasks, including state-of-the-art semantic segmentation on Synthia.

  • Problem

    Dynamic 3D point-cloud sequences require learning from irregular spatiotemporal data beyond two frames, while grid-based approaches introduce quantization error and inefficient fixed-kernel convolution.

  • Method

    MeteorNet uses stacked Meteor modules to aggregate spatiotemporal neighborhoods into per-point features, with direct grouping and chained-flow grouping for object motion.

  • Results

    MeteorNet outperforms grid-based and single-frame methods across activity recognition, semantic segmentation, and scene-flow estimation, achieving state-of-the-art results on Synthia.

  • Takeaways & Limitations

    MeteorNet provides a point-based deep-learning architecture for dynamic raw point-cloud sequences across multiple 3D recognition tasks.

  • Takeaways & Limitations

    Chained-flow grouping can accumulate errors when point clouds are sparse and initial flow estimates are inaccurate, potentially misaligning neighborhoods across time.

Abstract

from arXiv · show

Understanding dynamic 3D environment is crucial for robotic agents and many other applications. We propose a novel neural network architecture called $MeteorNet$ for learning representations for dynamic 3D point cloud sequences. Different from previous work that adopts a grid-based representation and applies 3D or 4D convolutions, our network directly processes point clouds. We propose two ways to construct spatiotemporal neighborhoods for each point in the point cloud sequence. Information from these neighborhoods is aggregated to learn features per point. We benchmark our network on a variety of 3D recognition tasks including action recognition, semantic segmentation and scene flow estimation. MeteorNet shows stronger performance than previous grid-based methods while achieving state-of-the-art performance on Synthia. MeteorNet also outperforms previous baseline methods that are able to process at most two consecutive point clouds. To the best of our knowledge, this is the first work on deep learning for dynamic raw point cloud sequences.

1. Introduction

MeteorNet addresses the difficulty of learning from dynamic raw 3D point-cloud sequences without grid quantization. Its Meteor modules aggregate spatiotemporal neighborhoods into hierarchical per-point features for several recognition tasks.

  • Motivation: Grid-based dynamic-scene methods introduce inevitable quantization error and inefficient fixed-kernel convolution, especially for precise localization.Sparse convolution can require special engineering efforts to address the inefficiency.
  • Method: MeteorNet directly processes dynamic point-cloud sequences and aggregates information from spatiotemporal neighborhoods to learn features for each point.The Meteor module can be stacked so successive modules hierarchically aggregate information from larger neighborhoods.
  • Method: Two neighborhood constructions address object motion: direct grouping expands grouping radius over time, while chained-flow grouping tracks motion using offline-estimated scene flow.The paper experimentally compares the two approaches.
  • Results: MeteorNet supports classification, semantic segmentation, and scene-flow estimation, achieving state-of-the-art semantic segmentation on Synthia and leading performance on several other benchmarks.It also outperforms previous methods on KITTI-derived semantic segmentation and leads on FlyingThings3D, KITTI scene flow, and MSRAction3D.

2. Related Work

Prior work on 3D learning uses grid-based convolutions, point-based architectures, or hybrids, while video methods model temporal relationships through recurrent, flow-based, convolutional, or non-local designs. MeteorNet belongs to the point-based category and applies it to point-cloud sequences without quantizing raw points into occupancy grids.

  • Deep learning for RGB videos: Deep-learning approaches for RGB videos model temporal relationships with recurrent networks, optical-flow branches, 3D CNNs, or non-local and correspondence-based operations.These categories provide inspiration for MeteorNet's treatment of 3D videos.
  • Grid-based 3D deep learning: MeteorNet directly processes point clouds instead of quantizing raw points into occupancy grids.This distinguishes it from the grid-based dynamic-sequence methods described above.
  • Deep learning on 3D point clouds: Point-based methods derive global features with shared networks and symmetric max pooling, while PointNet++ extracts local neighborhood features hierarchically.These methods form the main point-based lineage relevant to MeteorNet.
  • Deep learning on 3D point clouds: Hybrid methods combine point and grid representations, such as voxel-local PointNets followed by 3D convolution or interpolation between points and grids.MeteorNet instead lies in the point-based category and focuses on point-cloud sequences.

3. Deep Learning on 3D Point Cloud Sequences

MeteorNet learns representations for dynamic 3D point cloud sequences by aggregating information from spatiotemporal neighborhoods while respecting unordered points within frames and ordered frames across time. Its Meteor modules support multiple neighborhood constructions, fusion designs, downstream tasks, and a universal-approximation foundation.

  • Design requirements: MeteorNet treats intra-frame points as unordered, inter-frame points as time-ordered, and neighborhoods as jointly spatial-temporal.These properties ensure that point ordering within a frame does not affect outputs while timestamps distinguish different frames.
  • Meteor module: The Meteor module aggregates features from local spatiotemporal neighborhoods to produce updated features for every point, and modules can be stacked hierarchically.Inputs may include sensor attributes or features from a previous Meteor module.
  • Neighborhood construction: Direct grouping expands spatial radius with temporal separation, whereas chained-flow grouping follows estimated motion trajectories and can use a smaller radius.Chained-flow grouping is also described as potentially more computationally efficient than direct grouping.
  • Downstream applications: Meteor features support sequence-level classification, per-point semantic segmentation, and per-point scene-flow estimation.The resulting architectures are named MeteorNet-cls, MeteorNet-seg, and MeteorNet-flow, respectively.
  • Architecture design: MeteorNet uses early fusion for classification and segmentation, while late fusion first learns per-frame features and is used for scene-flow estimation.Early fusion mixes frames at the first layer; late fusion mixes them after several per-frame feature-learning layers.
  • Theoretical foundation: The authors provide a theoretical foundation by showing that Meteor modules have universal approximation ability for continuous functions on point cloud sequences.The foundation extends the universal-approximation perspective associated with PointNet to sequences.

4. Grids versus Point Clouds: A Toy Example

A toy moving-particle dataset tests whether grid-based dynamic-scene networks can distinguish motion across different distance scales. MeteorNet learns the task accurately, whereas the grid-based alternatives retain significant errors because their fixed grid scale cannot accommodate both short- and long-range motion.

  • The toy dataset classifies four-frame particle sequences into static, slow, medium, and fast motion categories.Particle displacement ranges from zero to approximately 0.1, 1, or 10 units per step for the non-static categories.
  • The experiment compares shallow FaF, MinkNet, and MeteorNet models using grid occupancy for the grid methods and raw 3D points for MeteorNet.All architectures are limited to three neuron layers and convolution kernels no larger than 3.
  • MeteorNet perfectly learns the toy dataset, while the grid-based methods cannot avoid significant errors regardless of the chosen grid size.The reported results list training and validation accuracy, with N/A indicating insufficient resources.
  • A grid matched to slow motion misses medium and fast displacement, whereas a grid matched to fast motion makes static and slow particles appear similar.Longer motion requires huge convolution receptive fields at a fine grid scale, while coarse grids collapse short-range motion.
  • The experiment presents MeteorNet as grid-size agnostic, allowing it to learn short- and long-range motion simultaneously.

5. Experiments

Experiments evaluate MeteorNet across action recognition, semantic segmentation, and scene flow estimation, comparing multi-frame point-based processing with prior baselines. Across tasks, longer sequences and motion-aware grouping generally improve recognition or flow estimation, while chained-flow grouping is especially effective for movable objects and overall scene flow.

  • Experimental Setup: MeteorNet is evaluated on action recognition, semantic segmentation, and scene flow estimation against grid-based, single-frame, and prior point-cloud baselines.The experiments use MSRAction3D, Synthia, KITTI, FlyingThings3D, and KITTI scene flow data.
  • 5.1. Classification: MeteorNet-cls significantly outperforms all baselines on MSRAction3D, and accuracy benefits from longer point cloud sequences.The dataset contains 567 Kinect depth-map sequences spanning 20 action categories.
  • 5.2. Semantic Segmentation: Multi-frame MeteorNet-seg outperforms single-frame PointNet++, while point-based methods achieve higher IoU than sparse-convolution methods on Synthia.Increasing the pooling bottleneck is more effective than enlarging the non-bottleneck layer for multi-frame input, but adding frames without enlarging the model can hurt performance.
  • 5.2. Semantic Segmentation: Chained-flow grouping achieves the best performance on movable Synthia objects such as Car and Pedestrian, and KITTI accuracy for Pedestrian/Cyclist increases with more input frames.MeteorNet has similar KITTI Car accuracy to PointNet++ but significantly improves on Pedestrian/Cyclist.
  • 5.3. Scene Flow Estimation: MeteorNet-flow improves scene-flow estimation with more frames; direct grouping is better for small displacements, whereas chained-flow grouping has better overall performance.On KITTI, three- and four-frame MeteorNet-flow outperform the baselines; chained flow gives lower mean error, while direct grouping gives lower standard deviation.
  • 5.3. Scene Flow Estimation: Chained-flow grouping supplies evidence near predicted corresponding positions, helping estimate flow direction and magnitude as input-frame count increases.The scene-flow experiments train on FlyingThings3D and fine-tune on KITTI because dense real-world per-point scene-flow labels are expensive.

6. Discussion

MeteorNet generalizes several point-cloud architectures depending on the number of input frames and module instantiation. Direct grouping has rapidly growing neighborhoods, whereas chained-flow grouping is more efficient but can accumulate errors when point clouds are sparse or flow estimates are inaccurate.

  • Relation to Other Architectures: With one input frame, MeteorNet can reduce to PointNet++ or DGCNN, and with two frames it can reduce to FlowNet3D.The reduction depends on the instantiation of the h function for the one-frame case.
  • Direct Grouping vs. Chained-flow Grouping: Direct grouping can require O(T^4) points in a point’s spatiotemporal neighborhood as sequence length T grows.Its spatial neighborhood radius is described as growing linearly with temporal distance.
  • Direct Grouping vs. Chained-flow Grouping: Chained-flow grouping is computationally efficient but risks accumulated neighborhood errors when point clouds are sparse and initial flow estimates are inaccurate.The authors leave the effect of initial scene-flow error on final performance for future work.

7. Conclusion

MeteorNet directly consumes dynamic 3D point cloud sequences and is evaluated across several recognition tasks. The paper reports advantages over grid-based and single-frame methods, along with visualized example results and additional supporting analyses.

  • 7. Conclusion: MeteorNet directly consumes dynamic 3D point cloud sequences and supports activity recognition, semantic segmentation, and scene flow estimation.The conclusion describes the architecture as a novel deep neural network and reports visualizations of example results.
  • 7. Conclusion: The supplementary document adds action-recognition accuracy, scene-flow outlier-ratio, architecture, runtime, proof, and qualitative-result analyses.It also gives a rationale for the MeteorNet name.

B. MSRAction3D Per-class Accuracy

Using 24-frame sequences improves recognition especially for actions requiring temporal information, while performance saturates near the average duration of a complete action.

  • B. MSRAction3D Per-class Accuracy: Classification accuracy saturates at 24 frames, corresponding to 1.6 seconds at MSRAction3D's 15fps frame rate.This duration is close to the average length of a complete action.
  • B. MSRAction3D Per-class Accuracy: Temporal input significantly improves categories whose actions are difficult to distinguish from single frames.“Forward punch,” “horizontal arm wave,” and “draw x” show large improvements because they involve similar forward arm stretches.
  • B. MSRAction3D Per-class Accuracy: Categories with highly discriminative single-frame postures show slight or negative accuracy gains from temporal input.Examples include “pick up & throw” and “golf waving.”
  • B. MSRAction3D Per-class Accuracy: The per-class results support that the Meteor module captures dynamic content in point cloud sequences.

D.1. Meteor Module Architecture

Meteor modules aggregate spatiotemporal neighborhood information through two variants: one preserves point correspondence, while the other is designed for settings where correspondence is unnecessary.

  • D.1. Meteor Module Architecture: The Meteor module updates each point's feature vector from its local spatiotemporal neighborhood.
  • D.1. Meteor Module Architecture: Meteor-rel processes paired points using their features and 4D position difference, then applies a shared MLP and element-wise max pooling.It is intended for applications where point correspondence matters, such as scene flow estimation.
  • D.1. Meteor Module Architecture: Meteor-ind aggregates neighborhood information without requiring point correspondence, supporting applications such as semantic segmentation.
  • D.1. Meteor Module Architecture: Both Meteor-ind and Meteor-rel outputs can be downsampled using farthest-point sampling, analogous to pooling in CNNs.

D.3. MeteorNet-seg Architecture

MeteorNet-seg uses early fusion and Meteor-ind modules to produce per-point predictions over point cloud sequences, with downsampling and feature propagation in the segmentation architecture.

  • D.3. MeteorNet-seg Architecture: MeteorNet-seg takes a point cloud sequence as input and produces a classification score for every point in the sequence.
  • D.3. MeteorNet-seg Architecture: The segmentation network uses four Meteor-ind modules and mixes points from different frames at the first layer through early fusion.
  • D.3. MeteorNet-seg Architecture: The point cloud is downsampled and then upsampled to its original point resolution through feature-propagation layers.Skip connections reuse local features from early network stages.
  • D.3. MeteorNet-seg Architecture: The output has 12 channels, matching the number of classes in the Synthia dataset.

E. Model Run Time Analysis

Runtime analysis compares MeteorNet-seg with direct and chained-flow grouping across batch sizes and frame counts, showing additional overhead for chained-flow grouping.

  • E. Model Run Time Analysis: Runtime was measured on Synthia using 8,192 points per frame, a GTX 1080 Ti GPU, an Intel Core i7 CPU, and TensorFlow 1.9.0.The study searched over batch size and number of frames.
  • E. Model Run Time Analysis: Chained-flow grouping introduces additional computational overhead compared with direct grouping.
  • E. Model Run Time Analysis: 8.0 seq/s versus 4.1 seq/s is the runtime for 2 frames and batch size 1 with direct versus chained-flow grouping.
  • E. Model Run Time Analysis: 4.3 seq/s versus 2.8 seq/s is the runtime for 4 frames and batch size 1 with direct versus chained-flow grouping.

F. Proof of Theorem

Theorem 2 establishes that continuous functions on fixed-length point-cloud sequences can be approximated using continuous per-point transformations and max pooling. The supplementary material also reports qualitative segmentation and scene-flow results.

  • Proof strategy: PointNet’s supporting lemma uses a continuous transformation followed by element-wise vector max pooling to approximate continuous set functions.MAX returns the element-wise maximum of a set of vectors.
  • Proof strategy: The proof maps a point-cloud sequence indexed by time into a single point-cloud space, enabling application of PointNet’s universal approximation result.The construction uses a time-dependent mapping and establishes continuity for the mapping and its inverse.
  • Theorem 2: Theorem 2 states that any continuous function on length-T point-cloud sequences can be approximated arbitrarily closely by continuous functions and max pooling.The sequence distance is defined as the maximum Hausdorff distance across corresponding frames.
  • Qualitative results: MeteorNet-seg can accurately segment most objects in additional Synthia test-set examples.Figure 15 presents RGB input, ground truth, and predictions for two examples.
  • Qualitative results: MeteorNet-flow can accurately estimate flow for moving objects in additional KITTI scene-flow examples.Figure 14 colors points by frame and shows translated earlier-frame points in black; perfect estimation would make green and black shapes overlap.
Loading 1910.09165v2…