PaperlayerView exact revision on arXiv

Source-linked AI summary

PointRNN: Point Recurrent Neural Network for Moving Point Cloud Processing

Hehe Fan, Yi Yang

arXiv:1910.08287v2cs.CV

TL;DR

Dynamic point clouds are difficult for conventional recurrent models because their unordered structure prevents direct pointwise state operations and can be damaged by vectorization. PointRNN uses coordinate-based local correlation, with PointGRU and PointLSTM variants, to process point-cloud sequences. These models produce correct moving-point predictions on synthetic and real-world datasets, while the correlation design remains a stated area for improvement.

  • Problem

    Unordered point clouds make conventional vector-based recurrent processing difficult because flattening can damage local structure and pointwise correspondence.

  • Method

    PointRNN aggregates current point features with previous point states through spatiotemporally-local correlation based on point coordinates, with PointGRU and PointLSTM variants.

  • Results

    PointRNN, PointGRU, and PointLSTM produce correct moving-point predictions on synthetic and real-world datasets, demonstrating point-cloud sequence modeling.

  • Takeaways & Limitations

    The proposed recurrent units provide components for point-cloud sequence processing and have potential for temporal applications such as 3D action recognition and sequential scene segmentation.

  • Takeaways & Limitations

    The authors identify more effective spatiotemporally-local correlation methods as future work for replacing the current point-rnn operation.

Abstract

from arXiv · show

In this paper, we introduce a Point Recurrent Neural Network (PointRNN) for moving point cloud processing. At each time step, PointRNN takes point coordinates $\boldsymbol{P} \in \mathbb{R}^{n \times 3}$ and point features $\boldsymbol{X} \in \mathbb{R}^{n \times d}$ as input ($n$ and $d$ denote the number of points and the number of feature channels, respectively). The state of PointRNN is composed of point coordinates $\boldsymbol{P}$ and point states $\boldsymbol{S} \in \mathbb{R}^{n \times d'}$ ($d'$ denotes the number of state channels). Similarly, the output of PointRNN is composed of $\boldsymbol{P}$ and new point features $\boldsymbol{Y} \in \mathbb{R}^{n \times d''}$ ($d''$ denotes the number of new feature channels). Since point clouds are orderless, point features and states from two time steps can not be directly operated. Therefore, a point-based spatiotemporally-local correlation is adopted to aggregate point features and states according to point coordinates. We further propose two variants of PointRNN, i.e., Point Gated Recurrent Unit (PointGRU) and Point Long Short-Term Memory (PointLSTM). We apply PointRNN, PointGRU and PointLSTM to moving point cloud prediction, which aims to predict the future trajectories of points in a set given their history movements. Experimental results show that PointRNN, PointGRU and PointLSTM are able to produce correct predictions on both synthetic and real-world datasets, demonstrating their ability to model point cloud sequences. The code has been released at \url{https://github.com/hehefan/PointRNN}.

1. Introduction

PointRNN adapts recurrent processing to unordered, dynamic 3D point clouds by retaining coordinates and aggregating states through local spatiotemporal correlations. Its PointGRU and PointLSTM variants support moving-point prediction, producing correct predictions on synthetic and real-world datasets.

  • Motivation: Point clouds are unordered 3D point sets, so flattening them into vectors can damage local structure and make processing sensitive to point permutations.PointRNN instead represents coordinates P together with per-point features X, states S, and outputs Y.
  • PointRNN: PointRNN extends RNN state updates from vectors to coordinate-linked point features and states, preserving each point's spatial identity.Its state and output include point coordinates because features and states depend on those coordinates.
  • PointRNN: PointRNN aggregates current features with previous states by finding neighbors across consecutive point clouds and pooling locally processed feature, state, and displacement information.This spatiotemporally-local correlation replaces direct concatenation, which cannot be applied across unordered point sets.
  • Variants: PointGRU and PointLSTM combine the point-based correlation with gated recurrent architectures to address the exploding and vanishing gradient problems inherited from RNN.The models extend PointRNN for sequence processing while retaining point-cloud structure.
  • Application: PointRNN, PointGRU, and PointLSTM are applied to moving point cloud prediction and produce correct predictions on synthetic and real-world datasets.The task predicts future point trajectories from historical movements and supports robot and self-driving-car planning.

2. Related Work

Prior work largely addressed static point cloud understanding or spatiotemporal modeling with structured tensors, while conventional recurrent models risk damaging point-cloud spatial structure. The paper's related-work context motivates a point-native recurrent alternative.

  • Static point cloud understanding: Static point cloud research has covered classification, part segmentation, scene semantic segmentation, reconstruction, and object detection, often processing point sets without regular voxel grids.These methods address static understanding rather than temporal point-cloud sequences.
  • RNN variants for spatiotemporal modeling: Spatiotemporal RNN variants such as ConvLSTM and ST-LSTM modify recurrent architectures to retain spatial local structure in tensor-valued sequences.ConvLSTM replaces fully connected layers with convolution, while ST-LSTM builds on ConvLSTM.

3. PointRNN

PointRNN extends recurrent processing to unordered point clouds by preserving point coordinates and aggregating current features with prior states through local spatiotemporal correspondence.

  • 3. PointRNN: For each current point, PointRNN searches neighbors in the previous cloud and combines query features, neighbor states, and relative displacements.The resulting neighbor representations pass through a shared fully connected layer and are pooled into one representation.
  • 3. PointRNN: PointRNN directly processes point coordinates and features, retaining spatial structure instead of flattening the point cloud into a permutation-sensitive vector.Its state and output also include point coordinates because point states and features depend on point correspondence.
  • 3. PointRNN: The point-rnn operation aggregates past and current information according to point coordinates, producing a state for each current point.By default, PointRNN uses the new point state as its output.
  • 3. PointRNN: PointRNN supports k-nearest-neighbor and radius-based ball-query neighborhood construction for matching points across time.Ball query first selects points within a radius and then samples k neighbors.
  • 3. PointRNN: Because PointRNN is a recurrent architecture, it may still encounter exploding and vanishing gradients, motivating more specialized gated variants.The paper identifies more effective spatiotemporally-local correlation functions as future work.

4. PointGRU and PointLSTM

PointGRU and PointLSTM replace vector-based recurrent concatenations with point-rnn operations, preserving coordinate-based correspondence while adding gated memory mechanisms.

  • 4. PointGRU and PointLSTM: PointGRU and PointLSTM replace the concatenation operations in GRU and LSTM with spatiotemporally-local point-rnn correlations.These variants are designed to address the exploding and vanishing gradient problems inherited by PointRNN from vanilla RNNs.
  • 4. PointGRU and PointLSTM: PointGRU uses update and reset gates, with an additional step that weights and permutes the previous state according to current point coordinates.This alignment enables the Hadamard product between the previous state and the current reset gate.
  • 4. PointGRU and PointLSTM: PointLSTM uses input, forget, and output gates with cell and hidden states, plus coordinate-based alignment of the previous cell state.Its additional alignment step weights and permutes C_t−1 according to the current input points P_t.

5. Moving Point Cloud Prediction

The paper applies PointRNN, PointGRU, and PointLSTM to predict future point trajectories from moving point-cloud histories. It uses hierarchical sequence-to-sequence models, free-running training, and permutation-invariant Chamfer and Earth Mover’s losses.

  • PointRNN, PointGRU, and PointLSTM are applied to predict future trajectories from the history of a moving point cloud.The task targets moving point-cloud prediction rather than direct pointwise sequence modeling.
  • Basic model: The basic sequence-to-sequence model encodes the input sequence, initializes a predicting recurrent unit with the final state, and predicts displacements added to the current coordinates.The architecture can stack recurrent units along the prediction direction.
  • Advanced model: The advanced model reduces the cost of processing high-resolution point sets through sampling and grouping, then propagates subsampled features back to the original points.Its hierarchical design uses recurrent layers for aggregation and a fully connected layer to regress predicted displacements.
  • Training: Free-running training uses the network’s predictions as subsequent inputs because teacher forcing caused displacement predictions to collapse toward zero.The authors therefore select free-running training instead of supplying ground-truth inputs during prediction.
  • Loss functions: Chamfer Distance and Earth Mover’s Distance provide order-invariant losses for comparing predicted and ground-truth point sets.The combined objective is L(P, P′) = αL_CD(P, P′) + βL_EMD(P, P′), with α, β ≥ 0.

6. Experiments

Experiments evaluate PointRNN, PointGRU, and PointLSTM on synthetic moving MNIST and real-world Argoverse and nuScenes prediction tasks. The proposed models outperform comparison methods while advanced architectures reduce computation and prediction error.

  • 6.1. Moving MNIST Point Cloud: PointRNN, PointGRU, and PointLSTM outperform other methods on moving MNIST; advanced PointLSTM obtains CD 1.16 for one-digit prediction.This is lower than ConvLSTM by 56.93, CubicLSTM by 8.35, PointNet++ · LSTM by 174.1, and PointCNN · ConvLSTM by 14.21.
  • 6.1. Moving MNIST Point Cloud: The advanced architecture reduces computation and prediction error compared with the basic architecture.For one-digit prediction, PointGRU decreases FLOPs from 14.84 billion to 2.00 billion; for two-digit PointRNN, CD decreases from 16.08 to 11.44.
  • 6.1. Moving MNIST Point Cloud: Stacking more hierarchical layers improves accuracy, with advanced PointGRU CD decreasing from 3.72 with one layer to 1.18 with three layers on one-digit prediction.The reported CDs for one, two, and three layers are 3.72, 1.48, and 1.18.
  • 6.1. Moving MNIST Point Cloud: Point-based models consume less computation than voxel-based models because their FLOPs depend on input points rather than the entire space.For two-digit prediction, basic PointLSTM uses 49.55 billion FLOPs versus 345.26 billion for voxel-based ConvLSTM.
  • 6.2. Argoverse and nuScenes: PointRNN, PointGRU, and PointLSTM outperform PointNet++ · LSTM and PointCNN · ConvLSTM on Argoverse and nuScenes, achieving lower prediction error at most time steps.On Argoverse, advanced PointRNN reaches CD 0.2789, lower than PointNet++ · LSTM by 0.1037 and PointCNN · ConvLSTM by 0.0668.
  • 6.2. Argoverse and nuScenes: The proposed models correctly predict backward-moving vehicles and produce reasonable scene flow on real-world point-cloud sequences.Visualization examples report correct vehicle-motion predictions and suggest unsupervised 3D scene-flow learning.

7. Conclusion

The paper proposes recurrent units for point-cloud sequence processing and evaluates them on moving point-cloud prediction. The units model sequences successfully, while more effective spatiotemporally-local correlations remain future work.

  • 7. Conclusion: PointRNN, PointGRU, and PointLSTM demonstrate the ability to model point-cloud sequences through moving point-cloud prediction experiments.The proposed units have potential for 3D action recognition and sequential scene semantic segmentation.
  • 7. Conclusion: More effective spatiotemporally-local correlation methods could improve PointRNN, PointGRU, and PointLSTM.

1. PointGRU and PointLSTM

PointGRU and PointLSTM adapt gated recurrent architectures to unordered point clouds. They replace vector-based recurrent aggregation with PointRNN’s coordinate-aware local correlation while retaining GRU and LSTM state mechanisms.

  • Gated recurrent mechanisms: GRU uses update and reset gates to control retained and forgotten information, while LSTM uses input, output, and forget gates around a cell-state accumulator.These gate mechanisms address the exploding and vanishing gradient problems of standard RNNs.
  • Outputs: PointGRU outputs its state by default, whereas PointLSTM outputs its hidden state by default.The PointGRU and PointLSTM diagram specifies Y_t as S_t and H_t, respectively.
  • PointGRU: PointGRU replaces GRU’s concatenation operations with PointRNN’s spatiotemporally-local correlation to handle unordered point clouds.It additionally weights and permutes the previous state according to the current input points.
  • PointLSTM: PointLSTM replaces LSTM’s recurrent function with point-rnn and adds coordinate-based weighting and permutation of the previous cell state.Its state contains the point coordinates, hidden state, and cell state.

2. More Experiments

Additional experiments examine neighborhood radius, pooling, and qualitative predictions on moving MNIST, Argoverse, and nuScenes. The results show sensitivity to ball-query radii, an advantage for max pooling, and correct qualitative motion predictions.

  • Radius analysis: Ball-query radius substantially affects prediction accuracy; PointRNN CD on Argoverse is 0.2875 under setting 1 versus 0.3001 under setting 2.The results suggest that carefully selecting search radiuses can reduce prediction error.
  • Pooling analysis: Max pooling performs better than average pooling on moving MNIST for PointRNN, PointGRU, and PointLSTM.
  • Visualization: The advanced models produce clearer moving-MNIST digits than the basic models for PointRNN, PointGRU, and PointLSTM.The visualization covers one- and two-digit sequences.
  • Visualization: On Argoverse and nuScenes, the proposed models correctly predict backward motion of vehicles, including cases where a vehicle leaves the visualization field.
Loading 1910.08287v2…