PaperlayerView exact revision on arXiv

Source-linked AI summary

PSTNet: Point Spatio-Temporal Convolution on Point Cloud Sequences

Hehe Fan, Xin Yu, Yuhang Ding, Yi Yang, Mohan Kankanhalli

arXiv:2205.13713v1cs.CV

TL;DR

Raw point cloud sequences are difficult to model because their spatial structure is irregular and points are inconsistent across frames, unlike grid-based videos. The paper introduces PST convolutions that separate spatial and temporal modeling, incorporates them into hierarchical PSTNet architectures, and reports improved performance on 3D action recognition and 4D semantic segmentation. The method still relies on large-scale labeled datasets for training.

  • Problem

    Raw point cloud sequences are spatially irregular, unordered, and inconsistent across frames, so grid-based video convolutions cannot be directly applied to them.

  • Method

    PST convolution separates spatial convolution for 3D point structure from temporal convolution for sequence dynamics, and PSTNet applies these operations hierarchically with transposed convolutions for point-level prediction.

  • Results

    PSTNet improves 3D action recognition and 4D semantic segmentation performance across the evaluated datasets.

  • Takeaways & Limitations

    The proposed convolutions provide a unified approach for modeling raw point cloud sequences in both sequence-level classification and point-level prediction tasks.

  • Takeaways & Limitations

    PSTNet relies on large-scale labeled datasets, motivating integration of few-shot learning to reduce dependence on human-annotated data.

Abstract

from arXiv · show

Point cloud sequences are irregular and unordered in the spatial dimension while exhibiting regularities and order in the temporal dimension. Therefore, existing grid based convolutions for conventional video processing cannot be directly applied to spatio-temporal modeling of raw point cloud sequences. In this paper, we propose a point spatio-temporal (PST) convolution to achieve informative representations of point cloud sequences. The proposed PST convolution first disentangles space and time in point cloud sequences. Then, a spatial convolution is employed to capture the local structure of points in the 3D space, and a temporal convolution is used to model the dynamics of the spatial regions along the time dimension. Furthermore, we incorporate the proposed PST convolution into a deep network, namely PSTNet, to extract features of point cloud sequences in a hierarchical manner. Extensive experiments on widely-used 3D action recognition and 4D semantic segmentation datasets demonstrate the effectiveness of PSTNet to model point cloud sequences.

1 INTRODUCTION

PSTNet addresses the difficulty of modeling raw point cloud sequences, whose spatial structure is irregular and whose points are inconsistent across frames. It separates spatial and temporal modeling with PST convolutions and builds a hierarchical network for recognition and prediction tasks.

  • Dynamic point clouds are spatially irregular, unordered, and inconsistent across frames, limiting direct use of grid-based video convolutions.
  • Voxelizing point clouds enables 4D convolution but introduces high computation and quantization errors that can restrict precise scene-geometry applications.
  • PST convolution directly processes raw point cloud sequences by separating spatial structure modeling from temporal dynamics modeling.
  • A point tube preserves spatio-temporal local structure despite points emerging inconsistently across frames.
  • PSTNet hierarchically extracts features, while its transposed convolution supports point-level prediction by interpolating temporal dynamics and spatial features.
  • The method improves 3D action recognition and 4D semantic segmentation across four datasets.

2 RELATED WORK

Prior work models dynamic point clouds with voxel grids, recurrent or point-based methods, transformers, and learned spatio-temporal representations. These approaches span recognition, forecasting, reconstruction, and super-resolution while using different strategies for handling temporal structure.

  • Learning Representations on Grid based Videos: Video representation methods model spatial and temporal dependencies using spatial and optical-flow streams, recurrent networks, pooling, and 3D convolutions.
  • Static Point Cloud Processing: Static point cloud research covers classification, segmentation, reconstruction, and detection, often manipulating point sets without voxelization.
  • Dynamic Point Cloud Modeling: Grid-based methods convert point clouds into bird’s-eye-view voxels or 4D occupancy grids before applying 3D or 4D convolutions.
  • Dynamic Point Cloud Modeling: PointRNN and MeteorNet process raw point cloud sequences with point-based recurrent or PointNet++-derived architectures.
  • Dynamic Point Cloud Modeling: 3DV integrates 3D motion into compact voxels, whereas P4Transformer uses a transformer to avoid point tracking.
  • Dynamic Point Cloud Modeling: Other methods learn temporal and spatial vector fields or temporally coherent feature spaces for reconstruction and point-based super-resolution.

3 PROPOSED POINT SPATIO-TEMPORAL CONVOLUTIONAL NETWORK

PST convolution addresses irregular, unordered spatial structure in point cloud sequences by separating spatial modeling from temporal modeling. It organizes local spatio-temporal neighborhoods with point tubes and supports hierarchical feature extraction through subsampling.

  • Hierarchical Network: Frame and point subsampling make PSTNet temporally and spatially hierarchical, allowing local tube information to be merged into global movement representations.The supplied passages describe convolutional processing and hierarchical receptive-field growth through subsampling.
  • Decomposing Space and Time: PST convolution separates spatial and temporal information to reduce the impact of spatial irregularity on temporal modeling.Its spatial and temporal kernels are decomposed, with spatial modeling performed before temporal modeling to avoid requiring point tracking.
  • Decomposing Space and Time: The spatial convolution captures local 3D point structure, while the temporal convolution models local dynamics across the sequence.The spatial kernel is generated as a function of continuous point displacements, enabling convolution over irregular neighborhoods.
  • Point Tubes: For the illustrated input, temporal anchors are the 1st, 3rd, and 5th frames, with 2 spatial anchors sampled per anchor frame.The example uses L = 5 frames, N = 8 points per frame, temporal kernel size l = 3, stride st = 2, padding p = 1, and spatial subsampling rate ss = 4.
  • Point Tubes: Point tubes preserve local spatio-temporal structure despite irregular point distributions across frames.The network applies spatial encoding around each anchor and temporal encoding over the resulting spatial features.
  • Point Tubes: PST convolution constructs point tubes by selecting temporal anchor frames, sampling spatial anchor points, and transferring them to neighboring frames.Anchor-frame selection uses temporal kernel size, stride, and padding; spatial anchors are sampled with farthest point sampling.

4 EXPERIMENTS

Experiments evaluate PSTNet on 3D action recognition and 4D semantic segmentation, with ablations examining temporal clip length, temporal kernel size, and spatial search radius. PSTNet outperforms compared methods across reported tasks while reducing runtime and parameter count in supported comparisons.

  • 3D action recognition: PSTNet significantly outperforms state-of-the-art methods on MSR-Action3D action recognition.From 16 to 24 frames, PSTNet improves accuracy by 1.30%, whereas MeteorNet improves by 0.29%.
  • 3D action recognition: PSTNet outperforms all compared approaches on NTU RGB+D 60 and NTU RGB+D 120 evaluation settings.On NTU RGB+D 120 cross-setup evaluation, PSTNet surpasses 3DV-PointNet++ by 4.6%.
  • Computational efficiency: PSTNet reduces average recognition running time per NTU RGB+D 60 video by about 2 seconds compared with 3DV-PointNet++.The comparison uses one Nvidia RTX 2080Ti GPU.
  • 4D semantic segmentation: PSTNet outperforms the state of the art on Synthia 4D semantic segmentation and uses 0.11M fewer parameters than MeteorNet.The parameter reduction is reported as 6% relative to MeteorNet.
  • Ablation study: Increasing clip length benefits action recognition because short clips may miss key frames.The experiments evaluate clips with fixed frame counts, and sequence-level predictions average clip-level probabilities.
  • Ablation study: Temporal kernel size balances motion modeling and fast-motion capture: sizes above 1 improve motion-dependent actions, while sizes above 3 reduce accuracy.For 4D semantic segmentation, PSTNet with l = 3 improves mIoU by 1.45% over l = 1.
  • Ablation study: Spatial search radius controls the modeled spatial-structure range, with overly small or large radii reducing effectiveness.Small radii miss sufficient structure information, whereas large radii reduce the discriminativeness of local spatial structure.

5 CONCLUSION

PSTNet models point cloud sequences with PST and transposed PST convolutions, addressing both representation learning and point-level dense prediction. Experiments demonstrate improved performance on 3D action recognition and 4D semantic segmentation.

  • PSTNet uses PST convolution to learn informative representations from raw point cloud sequences and PST transposed convolution for point-level dense prediction.The transposed version interpolates temporal dynamics and spatial features.
  • PSTNet significantly improves 3D action recognition and 4D semantic segmentation by effectively modeling point cloud sequences.
  • The PST transposed convolution generates original point features by temporal transposed convolution, temporal interpolation, and spatial interpolation.In the example, three input frames are expanded to five frames using temporal kernel size 3 and stride 2.

B PSTNET ARCHITECTURES

PSTNet uses hierarchical architectures tailored to sequence-level action recognition and point-level 4D semantic segmentation. The action-recognition network progressively reduces spatial and temporal resolution, while the segmentation network combines convolutional and transposed-convolutional layers.

  • 3D action recognition: The 3D action-recognition architecture contains six PST convolutions followed by spatial and temporal pooling and a fully connected classifier.Spatial subsampling halves resolution in selected layers, and temporal stride 2 halves temporal resolution in two layers.
  • 4D semantic segmentation: The 4D semantic-segmentation architecture contains four PST convolutional layers and four PST transposed-convolutional layers.Each PST convolution layer uses spatial subsampling rate 4, and segmentation operates on clips of three frames.

C IMPLEMENTATION DETAILS

Implementation and experiments use fixed neighborhood sampling, standard optimization settings, and dataset-specific configurations. Results show that temporal modeling helps distinguish similar actions and yields more separable features than MeteorNet.

  • Implementation: PSTNet fixes the number of sampled spatial neighbors to 9 for action recognition and 32 for 4D semantic segmentation.When too few neighbors exist, some neighbors are randomly repeated.
  • Implementation: Models are trained for 35 epochs with SGD, starting at learning rate 0.01 and decaying by 0.1 at epochs 10 and 20.
  • Feature visualization: PSTNet features are more compact and discriminative than MeteorNet features when projected into two dimensions with t-SNE.The visualization assigns the same color to sequences from the same action.

F SYNTHIA 4D SEMANTIC SEGMENTATION RESULT DETAILS

On Synthia 4D, PSTNet achieves the best accuracy for five of twelve semantic categories. Additional ablations show that disentangling space and time and retaining both displacement and sharing kernels improve action-recognition accuracy.

  • Synthia 4D result: PSTNet achieves the best accuracy for five of the twelve categories in the Synthia 4D dataset.The result is reported in the per-class semantic-segmentation results.
  • Disentangling ablation: The disentangling PST convolution achieves better MSR-Action3D accuracy than a non-decomposing convolution.The comparison is conducted across different clip lengths.
  • Kernel ablation: The spatial convolution uses a displacement transform kernel and a sharing kernel to capture local structure while increasing point-feature dimension.The displacement kernel models spatial displacements, while the sharing kernel expands feature dimension.
  • Kernel ablation: Using only points in the final layer causes accuracy to decrease dramatically.
  • Kernel ablation: Removing the sharing kernel treats neighborhood points equally and ignores their positions, reducing accuracy because spatial structure is not properly captured.

I COMPUTATIONAL EFFICIENCY AND MEMORY USAGE

PSTNet is evaluated for computational efficiency, memory usage, irregular frame sampling, and scene flow estimation. It is faster than MeteorNet, while hierarchical design limits running-time growth and PSTNet remains effective under irregular sampling and scene-flow evaluation.

  • Computational efficiency: 41.5% faster than MeteorNet on MSR-Action3D, attributed to PSTNet's convolution-based operations being more efficient than MeteorNet's MLPs.The comparison uses clips of length 16 on one Nvidia Quadro RTX 6000 GPU.
  • Depth scaling: Deeper PSTNet models substantially increase parameter count, but running time does not increase significantly because the network is spatio-temporally hierarchical.The reported study varies the number of PSTNet layers and measures both running time and parameters.
  • Spatial-operation ablation: Replacing the lightweight spatial convolution with PointConv slightly improves accuracy but significantly increases parameters and running time.The efficiency difference is attributed to PointConv's MLP-based point processing.
  • Irregular frame sampling: PSTNet achieves the best accuracy when eight of 24 frames are randomly removed and missing frames are interpolated by replication or ICP.The experiment evaluates robustness to irregularly sampled frame sequences on MSR-Action3D.
  • Scene flow estimation: PSTNet achieves promising accuracy on scene flow estimation for the KITTI scene flow dataset after adapting temporal anchor-frame selection and spatial anchoring.The task estimates a flow vector for every point in the last frame.

L MOTION RECOGNITION ON THE SYNTHETIC MOVING MNIST POINT CLOUD DATASET

The synthetic Moving MNIST Point Cloud dataset tests motion recognition when digit appearance and motion are independent and point identities can be tracked. PSTNet performs similarly with and without tracking, indicating limited reliance on point IDs in this simulated setting.

  • Dataset construction: The dataset contains 16-frame sequences of moving handwritten digits, with 128 sampled points per digit and point order maintained across frames.The benchmark uses the original MNIST training/test split and includes 144 designed motions.
  • Motion design: The benchmark varies nine initial locations, eight velocities, and two digit-distortion types to construct motion examples.Distortion scales digits horizontally or vertically over time.
  • Recognition challenge: Motion recognition is challenging because motion and digit appearance are independent, requiring recognition without interference from appearance.The setup separates the motion signal from the digit's visual identity.
  • Tracking variant: Anchors and neighboring points can be selected in the first frame and propagated across later frames to track corresponding local regions.This tracking-based variant explicitly exploits point correlation.
  • Results: Both original and tracking-based PSTNet achieve promising accuracy, with similar accuracy in the simulated setting.The result indicates that PSTNet does not heavily rely on point IDs or tracking.

M VISUALIZATION OF THE OUTPUT OF EACH PST CONVOLUTION LAYER IN PSTNET

Layer visualizations show a progression from appearance-sensitive processing to motion-sensitive representations. Early PSTConv1 lacks temporal modeling, whereas deeper PST convolutions focus mainly on moving areas.

  • PSTConv1: PSTConv1 focuses on performer contours because its temporal kernel size is l = 1 and therefore does not capture temporal correlation.The visualization uses activation magnitude, with larger outputs indicating higher activation.
  • Deeper PST convolutions: PSTConv2a through PSTConv4 mainly activate moving areas while modeling the spatio-temporal structure of point cloud sequences.These deeper layers are associated with action-relevant dynamics rather than only appearance.
  • Interpretation: The visualization results support that PST convolution captures dynamics in point cloud sequences.The network's layer outputs are visualized after squeezing output vectors to scalars with the l1 norm.
  • Segmentation examples: PSTNet accurately segments most objects in the shown Synthia 4D examples.The figures compare inputs, ground truth, and PSTNet predictions.

O LIMITATION

PSTNet relies on large-scale labeled datasets for training. The paper identifies reducing this dependence through methods such as few-shot learning as a potential improvement.

  • Data dependence: PSTNet relies on large-scale labeled datasets for training, like most deep neural networks.The limitation concerns dependence on human-annotated data.
  • Potential improvement: Few-shot learning is suggested as a potential way to reduce reliance on human-annotated data for point cloud sequence modeling.This is presented as a possible improvement rather than an evaluated component of PSTNet.
Loading 2205.13713v1…