PaperlayerView exact revision on arXiv

Source-linked AI summary

FlowNet3D: Learning Scene Flow in 3D Point Clouds

Xingyu Liu, Charles R. Qi, Leonidas J. Guibas

arXiv:1806.01411v3cs.CVcs.LG

TL;DR

Existing scene flow methods mostly use stereo or RGB-D images, leaving direct point-cloud estimation relatively limited. FlowNet3D learns scene flow end-to-end with hierarchical point-cloud features, flow embeddings, and trainable feature propagation, and generalizes from synthetic training to real LiDAR scans. It achieves competitive or better results against baselines and prior art while supporting scan registration and motion segmentation applications.

  • Problem

    Most previous methods estimate scene flow from stereo or RGB-D images, while direct point-cloud estimation remains limited.

  • Method

    FlowNet3D is an end-to-end deep network that learns hierarchical point-cloud features, flow embeddings, and trainable set-feature propagation.

  • Results

    63% relative error reduction from in 3D end-point error is reported on KITTI, alongside lower 3D outlier ratios and stronger results than ICP baselines.

  • Takeaways & Limitations

    Synthetic-only training generalizes robustly to real LiDAR scans, and the estimated flow supports scan registration and motion segmentation.

  • Takeaways & Limitations

    The KITTI evaluation uses only 200 annotated frames, while a larger mentioned dataset is not publicly available.

Abstract

from arXiv · show

Many applications in robotics and human-computer interaction can benefit from understanding 3D motion of points in a dynamic environment, widely noted as scene flow. While most previous methods focus on stereo and RGB-D images as input, few try to estimate scene flow directly from point clouds. In this work, we propose a novel deep neural network named $FlowNet3D$ that learns scene flow from point clouds in an end-to-end fashion. Our network simultaneously learns deep hierarchical features of point clouds and flow embeddings that represent point motions, supported by two newly proposed learning layers for point sets. We evaluate the network on both challenging synthetic data from FlyingThings3D and real Lidar scans from KITTI. Trained on synthetic data only, our network successfully generalizes to real scans, outperforming various baselines and showing competitive results to the prior art. We also demonstrate two applications of our scene flow output (scan registration and motion segmentation) to show its potential wide use cases.

1. Introduction

FlowNet3D addresses scene flow estimation directly from point clouds, rather than relying on stereo or RGB-D representations. It introduces trainable point-set layers and reports improved performance on real LiDAR scans, including when trained only on synthetic data.

  • Scene flow is the 3D motion field of points and provides cues for segmentation, action recognition, camera pose estimation, and other 3D vision tasks.
  • Most previous methods estimate optical flow and disparity from stereo or RGB-D images instead of directly optimizing 3D scene flow.
  • FlowNet3D estimates scene flow end-to-end from two consecutive point clouds using deep hierarchical features and motion-representing flow embeddings.
  • The flow embedding layer correlates two point clouds, while the set upconv layer learns to propagate features between point sets.
  • KITTI evaluation shows greatly improved performance over traditional non-deep-learning methods, with robust real-scan estimation after synthetic-only training.

2. Related Work

Prior scene flow methods largely derive motion from stereo or RGB-D images, while point-cloud approaches use hand-crafted, assumption-heavy pipelines. FlowNet3D instead learns scene flow directly from point clouds without hard correspondences or rigidity assumptions.

  • Stereo-based scene flow methods combine optical flow and depth estimation, often using variational regularization or rigidity assumptions.
  • RGB-D approaches generalize 2D flow methods to 3D and exploit geometric cues from depth images.
  • Earlier point-cloud methods use hand-crafted descriptors, logistic classifiers, factor graphs, or EM pipelines with correspondence and rigidity assumptions.
  • FlowNet3D is an end-to-end deep-learning solution that learns features without hard correspondences or assumptions on rigidity.

3. Problem Definition

The task takes two potentially unmatched point clouds from consecutive frames and predicts the translational motion of every sampled point in the first cloud. The architecture uses point-set processing layers to learn features and motion relations.

  • The inputs are point sets P and Q sampled from a dynamic 3D scene at two consecutive time frames.
  • Because of object motion and viewpoint changes, the two point clouds may differ in size and lack point correspondences.
  • For a point moving from x_i to x′_i, its translational motion vector is d_i = x′_i − x_i.
  • The goal is to recover scene flow D for every sampled point in the first point cloud, given P and Q.
  • The model includes set convolution, flow embedding, and set upconvolution layers for feature learning, point mixing, and flow refinement.

4. FlowNet3D Architecture

FlowNet3D estimates scene flow from two point-cloud frames using hierarchical point features, learned cross-cloud flow embeddings, and informed feature propagation. Its architecture combines set convolution, flow embedding, and set upconv layers to predict motion for every point in the first frame.

  • FlowNet3D contains point feature learning, point mixture, and flow refinement modules built from set conv, flow embedding, and set upconv layers.
  • Hierarchical Point Cloud Feature Learning: Set conv layers learn hierarchical features from irregular, orderless point clouds through spatially local and translation-invariant processing.The layer samples regions, applies a nonlinear function to point features and relative coordinates, and aggregates with element-wise max pooling.
  • Point Mixture with Flow Embedding Layer: The flow embedding layer correlates two point clouds by aggregating neighboring candidate displacements and produces a motion embedding for each first-frame point.It uses both point features and spatial relationships to combine potential flow vectors when explicit correspondences are unavailable.
  • Point Mixture with Flow Embedding Layer: Additional set conv layers mix flow embeddings to promote spatial smoothness and resolve ambiguous motions requiring larger receptive fields.
  • Flow Refinement with Set Upconv Layer: Set upconv layers learnably propagate and refine intermediate flow embeddings onto target locations instead of relying on direct 3D interpolation.They aggregate neighboring source features at specified target coordinates and can propagate or summarize features between point clouds.
  • FlowNet3D Architecture: The final architecture uses four set conv layers, one flow embedding layer, four corresponding set upconv layers, skip connections, and a linear flow regression layer.The regression layer outputs a 3D scene-flow vector for each original point.

5. Training and Inference wtih FlowNet3D

FlowNet3D is trained with supervised scene-flow targets using smooth L1 loss and cycle-consistency regularization, then uses random re-sampling during inference to reduce prediction noise. The model trained on FlyingThings3D generalizes well to real LiDAR scans.

  • Training: FlowNet3D is trained with ground-truth scene-flow supervision, despite dense supervision being difficult to acquire for real data.
  • Architecture: The architecture specification ends with a linear layer, which therefore omits ReLU and batch normalization.
  • Training: The training objective combines smooth L1 loss for scene-flow supervision with cycle-consistency regularization.
  • Training: The network predicts scene flow D from consecutive point clouds P and Q using the parameterized function F(P, Q; Θ).
  • Training: Cycle consistency evaluates backward flow from the shifted point cloud against the reverse of the forward flow.
  • Inference: Randomly re-sampling point clouds across multiple inference runs and averaging pointwise flow vectors reduces down-sampling noise and yields a slight performance gain.

6. Experiments

Experiments validate FlowNet3D on synthetic FlyingThings3D and real KITTI LiDAR scans, including design ablations, generalization, and downstream applications.

  • FlyingThings3D: FlyingThings3D provides a large-scale synthetic setting for training, testing, and validating FlowNet3D design choices.The prepared data use 20,000 training examples and 2,000 test examples converted from rendered depth and optical flow.
  • FlyingThings3D: FlowNet3D is evaluated with 3D endpoint error and flow accuracy at 5% and 10% thresholds.EPE is the average L2 flow error, while ACC measures the fraction of estimates below each threshold.
  • FlyingThings3D: FlowNet3D achieves lower EPE and higher accuracy than the evaluated baselines on FlyingThings3D.The comparison includes image-based FlowNet-C and ICP, which struggle with occlusions, clutter, or independently moving objects.
  • FlyingThings3D: The ablations favor max pooling, the proposed flow-embedding formulation, and the set upconv layer, with the latter reducing flow error by 20%.Ten-time inference resampling and cycle-consistency regularization further improve performance.
  • KITTI generalization: On KITTI, FlowNet3D reduces 3D endpoint error by 63% relative to and outperforms image-based and ICP baselines without ground points.The model is trained only on FlyingThings3D and evaluates 150 KITTI frames with available point clouds.
  • Applications: Predicted scene flow captures motion on KITTI vehicles and pedestrians and supports scan registration and motion segmentation applications.A demonstrated registration case succeeds with dense scene flow where ICP fails to align the chair.

7. Conclusion

The paper presents FlowNet3D for direct end-to-end scene-flow estimation from point clouds and validates it on synthetic and real data. It also demonstrates applications using the estimated flow.

  • Conclusion: FlowNet3D estimates scene flow directly from 3D point clouds in an end-to-end deep network.The architecture introduces flow embedding for motion encoding and set upconv for trainable set-feature propagation.
  • Conclusion: Experiments on synthetic data and real LiDAR point clouds show competitive or better results against baselines and prior art.The conclusion also reports two example applications using the estimated scene flow.

Supplementary

The supplementary document expands experimental, baseline, KITTI-preparation, application, runtime, and visualization details.

  • Supplementary contents: Supplementary material adds details on FlyingThings3D experiments, baseline architectures, KITTI LiDAR preparation, and the two scene-flow applications.It also reports model size, runtime, and additional visualizations of data and learned features.

B. Details on FlyingThings 3D Experiments (Sec. 6.1)

The FlyingThings3D point-cloud experiments reconstruct 3D scene flow from rendered depth and apply preprocessing and masking choices for training and evaluation.

  • Data preparation: Point clouds are constructed by lifting FlyingThings3D depth maps into 3D, without using RGB images.The reconstruction uses the virtual camera intrinsics, including focal lengths and principal-point coordinates.
  • Data preparation: The experiments retain points whose Z value exceeds 35 to limit depth explosion and emphasize foreground motion.This threshold is fixed across experiments.
  • Training and evaluation: Training ignores scene-flow loss for points masked as disappearing or emerging from viewpoint changes and occlusions, while testing still evaluates them.The test-time procedure lacks masks.

C. Details on Baseline Architectures (Sec. 6.1)

The baselines compare alternative ways to combine two point clouds, using image-based, input-level, global-feature, or intermediate-point representations. Their architectures differ in how they encode cross-cloud relations and refine point features.

  • FlowNet-C: FlowNet-C adapts an RGB-image optical-flow CNN to depth or RGB-D inputs represented as XYZ coordinate maps.RGB-D inputs use six channels: RGB values plus XYZ maps.
  • EM-baseline: The EM-baseline mixes two point clouds at the input level and marks each point’s source with a two-dimensional one-hot feature.The source indicators are (1, 0) for the first set and (0, 1) for the second.
  • EM-baseline: EM-baseline feature propagation upsamples subsampled point features through 3D interpolation.Its set-convolution layers use radius-based neighborhoods, multilayer perceptrons, and specified downsampling rates.
  • LM-baseline: The LM-baseline mixes the two clouds at the global-feature level, making detailed local cross-cloud relations difficult to recover.It computes global features separately, concatenates them, processes them with fully connected layers, and tiles the result.
  • DM-baseline: The DM-baseline concatenates fixed-neighborhood feature distances and XYZ displacements, then uses fully connected layers and 3D interpolation for refinement.Unlike FlowNet3D, it does not use a flow embedding layer or trainable set upconv refinement.

D. Details on KITTI Data Preparation (Sec. 6.2)

KITTI preparation removes ground points, evaluates chunked large scenes, and adapts FlowNet3D for partial-scan registration and motion segmentation. The model is compact but requires practical preprocessing and inference procedures.

  • Ground removal: Ground removal supports fair comparison with prior methods and addresses weak generalization to KITTI ground motion when training uses synthetic data.The ground is a large, flat geometry with little motion cue.
  • Ground removal: 94.02% and 97.60% accuracy were achieved by RANSAC and GroundSegNet, respectively, for KITTI ground segmentation.Their runtimes were 43ms and 57ms per frame; main-paper evaluation used annotated ground points.
  • Large-scene inference: Large KITTI scenes are split into overlapping 5m×5m chunks aligned with the XY axes, with neighboring chunks offset by 2.5m plus Gaussian noise.The same chunk positions are used for both frames.
  • Large-scene inference: Flow estimates from same-location frame chunks are averaged for points appearing in multiple chunks.This produces the final scene-flow output for the full scene.
  • Partial-scan registration: Partial ModelNet40 scans omit object regions across views, making registration and warping challenging.The dataset was created by virtually scanning CAD models with a rotated camera around each object.
  • Partial-scan registration: FlowNet3D is fine-tuned to predict warping flow for partial scans and iteratively regresses a residual flow when displacements are large.The final flow is the first prediction plus the residual prediction.
  • Motion segmentation: Motion segmentation clusters six-dimensional point-flow vectors formed by concatenating coordinates with scaled flow.Clusters use a minimum size and an upper distance bound.
  • Model size and runtime: FlowNet3D has a 15MB model size, smaller than most deep convolutional neural networks.Runtime is evaluated across input point-cloud and batch sizes in Table 8.

H. More Visualizations

Visualizations show FlowNet3D handling complex synthetic scenes, learning feature similarities robust to partial scans, and encoding displacement-sensitive flow embeddings. They also expose failures when visibility changes substantially.

  • Scene-flow results: The FlyingThings3D visualizations present whole scenes from above and zoomed object views with consistent XYZ coordinates.Figures 10–12 show scene-flow results for three test examples.
  • Scene-flow results: The network estimates motion well for an occluded gray box and sword, but struggles when most of a wheel disappears between frames.The non-visible wheel motion is inaccurate in that failure case.
  • Feature visualization: A heatmap in Figure 13 visualizes correlations between one frame-1 point and all frame-2 points, with more red indicating greater similarity.The learned local features are described as robust to scan partiality.
  • Flow-embedding visualization: Figure 14 varies feature similarity and displacement vectors to show which inputs activate one flow-embedding neuron.The displacement vectors lie in a [−5, 5] × [−5, 5] × [−5, 5] cube, and similarity is defined by one minus cosine distance.
  • Flow-embedding visualization: As similarity increases from -1 to 1, one neuron’s activation becomes significantly larger and appears associated with positive-Z flow.This interpretation comes from the visualization of a model using cosine distance.
Loading 1806.01411v3…