PaperlayerView exact revision on arXiv

Source-linked AI summary

Receding Moving Object Segmentation in 3D LiDAR Data Using Sparse 4D Convolutions

Benedikt Mersch, Xieyuanli Chen, Ignacio Vizzo, Lucas Nunes, Jens Behley, Cyrill Stachniss

arXiv:2206.04129v1cs.ROcs.CV

TL;DR

Autonomous vehicles need to distinguish currently moving objects from static ones in LiDAR scans for navigation in dynamic environments. This paper processes scan sequences with sparse 4D convolutions and receding-horizon Bayesian updates; the authors report more accurate segmentation than existing methods and generalization to unseen environments.

  • Problem

    The paper addresses how to identify LiDAR points belonging to currently moving objects rather than static objects, a distinction relevant to safe navigation in dynamic environments.

    Source: §I · p. 1

  • Method

    Sparse 4D convolutions process sequences of LiDAR scans, while a receding horizon and binary Bayes filter incorporate new observations and fuse predictions.

    Source: §I · p. 2, B. Sparse 4D Convolutions · p. 3

  • Results

    The authors report more accurate moving-object segmentation than existing methods, generalization to unseen environments without domain adaptation, and improved results from integrating new observations.

    Source: §I · p. 2

  • Takeaways & Limitations

    The receding-horizon strategy with a binary Bayes filter extends the time horizon for segmentation and increases robustness to false positive and false negative predictions.

    Source: §V · p. 8

Abstract

from arXiv · show

A key challenge for autonomous vehicles is to navigate in unseen dynamic environments. Separating moving objects from static ones is essential for navigation, pose estimation, and understanding how other traffic participants are likely to move in the near future. In this work, we tackle the problem of distinguishing 3D LiDAR points that belong to currently moving objects, like walking pedestrians or driving cars, from points that are obtained from non-moving objects, like walls but also parked cars. Our approach takes a sequence of observed LiDAR scans and turns them into a voxelized sparse 4D point cloud. We apply computationally efficient sparse 4D convolutions to jointly extract spatial and temporal features and predict moving object confidence scores for all points in the sequence. We develop a receding horizon strategy that allows us to predict moving objects online and to refine predictions on the go based on new observations. We use a binary Bayes filter to recursively integrate new predictions of a scan resulting in more robust estimation. We evaluate our approach on the SemanticKITTI moving object segmentation challenge and show more accurate predictions than existing methods. Since our approach only operates on the geometric information of point clouds over time, it generalizes well to new, unseen environments, which we evaluate on the Apollo dataset.

I. INTRODUCTION

The paper addresses online identification of currently moving objects in LiDAR scans, a task relevant to safe navigation and other autonomous-vehicle capabilities. It proposes sparse 4D processing with receding-horizon updates and claims improved segmentation, generalization, and robustness from integrating new observations.

I. INTRODUCTION

  • Online moving-object segmentation distinguishes points on objects moving now from static points, a task relevant to collision-free planning and safe navigation.The target includes moving cars or bicyclists and static objects such as parked cars, buildings, or trees.Source: §I · p. 1
  • The approach jointly extracts spatio-temporal features with sparse 4D convolutions and predicts moving-object confidence scores for points across a short sequence of scans.A receding window admits new scans, while a binary Bayes filter fuses predictions to increase robustness to false predictions.Source: §I · p. 2
  • The authors claim more accurate segmentation than existing methods, generalization to unseen environments without domain adaptation, and improved results from integrating online observations.The authors state that their evaluation supports these claims.Source: §I · p. 2

II. RELATED WORK

Earlier approaches identify objects that have moved during mapping or use temporal information from projected representations, tracking, or point-cloud methods. This paper instead targets currently moving objects online, jointly processing LiDAR sequences in sparse 4D space and refining predictions as new scans arrive.

II. RELATED WORK

  • Mapping approaches often identify objects that moved over the mapping process, whereas this method targets objects actually moving within a limited time horizon.Prior approaches include map cleaning, mapping-based removal, and offline occupancy-based candidate detection followed by clustering and tracking.Source: §II · p. 2
  • Scene-flow methods can classify moving points before estimating separate static and moving flows, but many use only two consecutive frames.That time horizon may be too short to classify slowly moving objects.Source: §II · p. 2
  • Projection-based methods use range or bird’s-eye-view images to reduce the burden of sequence processing, but can lose information or require back-projection and kNN clustering.The paper contrasts these approaches with direct prediction in 4D space without post-processing.Source: §II · p. 2
  • The method jointly applies sparse 4D convolutions to LiDAR sequences and uses receding-horizon updates with Bayesian filtering to refine predictions online.Figure 2 depicts scan alignment, aggregation into a sparse 4D point cloud, feature extraction, and confidence prediction.Source: §II · p. 2, Figure 2 · p. 3

III. OUR APPROACH

The method aligns past LiDAR scans to the current viewpoint, adds time as a fourth coordinate, and processes the resulting sparse 4D representation. It assumes access to relative pose transformations between scans.

III. OUR APPROACH

  • Past scans are transformed to the current scan’s viewpoint, aggregated into a 4D point cloud, and processed to predict moving confidence scores for points across the sequence.When a new scan arrives, the prediction window shifts and later predictions can be fused for the same scan with a binary Bayes filter.Source: §III · p. 3

A. Input Representation

  • The method assumes estimated relative pose transformations between scans and uses them to align past point clouds locally.The authors state that the odometry source is not important and that local consistency is sufficient.Source: A. Input Representation · p. 3
  • After alignment, points are represented with spatial coordinates and time, then quantized into a sparse voxel grid with fixed spatial and temporal resolutions.The sparse tensor stores only occupied voxels, reducing memory use and enabling efficient spatio-temporal CNN processing.Source: A. Input Representation · p. 3

B. Sparse 4D Convolutions

Sparse 4D convolutions jointly extract spatio-temporal features from the occupancy grid and produce moving-object confidence scores for points. The network uses occupancy-only inputs and a modified sparse U-Net architecture.

B. Sparse 4D Convolutions

  • Sparse 4D convolutions operate on the sparse occupancy grid to extract spatio-temporal features and predict a moving-object confidence score for each point.The authors use the Minkowski engine, whose sparse convolutions define kernel maps between input and output coordinates.Source: B. Sparse 4D Convolutions · p. 3
  • The modified MinkUNet14 uses strided sparse convolutions for downsampling, transpose convolutions for upsampling, and UNet skip connections to retain fine-grained predictions.The model has 1.8 M parameters; its final sparse convolution and softmax produce confidence scores between 0 and 1.Source: B. Sparse 4D Convolutions · p. 4
  • Occupied voxels receive a constant feature of 0.5, so the input is an occupancy grid rather than RGB or intensity features.The authors state that this avoids needing to estimate coordinate or intensity distributions for standardizing inputs in new environments.Source: B. Sparse 4D Convolutions · p. 4

C. Receding Horizon Strategy

The receding-horizon strategy updates a fixed-length LiDAR window as each scan arrives, enabling online re-estimation from new observations. A binary Bayes filter recursively combines predictions for each point to refine moving-object estimates across observations.

C. Receding Horizon Strategy

  • Each new LiDAR scan enters a first-in, first-out window as the oldest scan is discarded, allowing moving objects to be re-estimated from new observations.The strategy increases the time horizon used for prediction.Source: C. Receding Horizon Strategy · p. 4

D. Binary Bayes Filter

  • The shifted window re-estimates the previous N−1 scans, and recursive fusion combines predictions from different times to help detect slowly moving objects.The paper states that fusion also reduces false positives and negatives arising from occlusions or noisy measurements.Source: D. Binary Bayes Filter · p. 4
  • The filter fuses predictions for scans shared by successive input sequences; points with final confidence above 0.5 are classified as moving.The receding-horizon strategy shifts the window to incorporate the next observation, and the filter combines multiple confidence scores.Source: D. Binary Bayes Filter · pp. 4–5, Figure 3 · p. 4
  • For each point, the filter updates recursive log-odds only at time steps when that point is observed, adding its new prediction and subtracting the prior log-odds.The prior p0 controls how much a predicted moving point in a single scan influences the final prediction.Source: D. Binary Bayes Filter · p. 4

IV. EXPERIMENTAL EVALUATION

The evaluation is intended to assess moving-object segmentation accuracy, generalization to unseen environments, and the effect of integrating new observations online. The authors state that their experiments support all three claims.

IV. EXPERIMENTAL EVALUATION

  • The authors claim their method segments moving objects in LiDAR data more accurately than existing methods.The paper presents the experiments as evidence for this comparison.Source: §IV · p. 5
  • The authors claim their approach generalizes to unseen environments without additional domain adaptation techniques.The experiments are described as supporting this generalization claim.Source: §IV · p. 5
  • The authors claim that integrating new observations online improves the results.The experiments are described as supporting this claim.Source: §IV · p. 5

A. Experimental Setup

The experiments train models on SemanticKITTI and use its validation and test sequences for evaluation. The setup also specifies a moving-class IoU metric and an additional Apollo evaluation for cross-environment generalization.

A. Experimental Setup

  • Models are trained on SemanticKITTI sequences 00–07 and 09–10, validated on sequence 08, and tested on sequences 11–21.Unless otherwise specified, inputs contain 10 scans at 0.1-second resolution, voxelized at 0.1 m.Source: A. Experimental Setup · p. 5
  • The standard intersection-over-union metric is reported for the moving class, using true-positive, false-positive, and false-negative point classifications.The setup defines IoUMOS as TP/(TP + FP + FN).Source: A. Experimental Setup · p. 5
  • Table I reports performance on the SemanticKITTI moving-object segmentation benchmark and identifies baseline results as taken from [6].The caption does not provide cell values.Source: Table I · p. 5
  • Cross-environment generalization is evaluated on Apollo-ColumbiaParkMapData sequences 2 and 3 without domain adaptation.The selected Apollo frames are annotated the same way as SemanticKITTI; both datasets use Velodyne HDL-64E scanners mounted at different heights on different cars and recorded in different environments.Source: A. Experimental Setup · p. 5

B. Moving Object Segmentation Performance

On SemanticKITTI, the method outperforms the compared moving-object segmentation baselines. On Apollo, it achieves the highest reported IoU_MOS without domain adaptation or retraining; the authors hypothesize that its geometry-based features support this generalization.

B. Moving Object Segmentation Performance

  • 65.2% IoU_MOS outperforms all compared baselines on the SemanticKITTI hidden test set, including LMNet+AutoMOS+Extra.The comparison follows the LMNet setup and uses SemanticKITTI poses estimated with an online SLAM system.Source: B. Moving Object Segmentation… · p. 5

C. Generalization Capabilities

  • 73.1% IoU_MOS is the highest reported Apollo result, achieved without domain adaptation or retraining.The best baseline reaches 65.9% IoU_MOS with domain-adaptation steps.Source: C. Generalization Capabilities · pp. 5–6, Table II · p. 6
  • The authors hypothesize that sparse 4D occupancy features support generalization by avoiding sensor-specific inputs and semantic-class distributions that can differ across environments.They also suggest direct 4D processing makes the network less prone to overfitting to a particular sensor location than range-image methods.Source: C. Generalization Capabilities · p. 6

D. Receding Horizon Strategy and Fusion

The section examines how scan count, temporal resolution, and the Bayesian prior affect receding-horizon predictions. A longer effective time horizon and an intermediate prior perform best in the reported SemanticKITTI validation experiments.

D. Receding Horizon Strategy and Fusion

  • A prior between 0.1 and 0.3 works best on the SemanticKITTI validation sequence, with the best reported setup using N=10 scans and p0=0.25.The Fig. 4 comparison covers IoU_MOS across moving-object priors, time horizons, and temporal resolutions.Source: D. Receding Horizon Strategy and Fusion · p. 6, Figure 4 · p. 6
  • A small prior such as p0=0.01 fuses moving predictions aggressively, increasing true positives but also false positives; a large prior such as p0=0.99 requires predictions to agree.The prior controls how new predicted log-odds contribute relative to the initially expected log-odds.Source: D. Receding Horizon Strategy and Fusion · p. 6
  • With five input scans, temporal resolutions of 0.2 s or 0.3 s outperform 0.1 s, and the longer time horizon improves segmentation of slowly moving objects.The compared five-scan setups span 0.4 s at 0.1 s spacing, 0.8 s at 0.2 s spacing, and 1.2 s at 0.3 s spacing.Source: D. Receding Horizon Strategy and Fusion · p. 6
  • Combining more scans with receding-horizon fusion generally improves segmentation by using predictions over a larger time horizon.The compared models use N=2, 5, or 10 scans, and Fig. 4 marks the past scans used for prediction at time t.Source: D. Receding Horizon Strategy and Fusion · p. 6, Figure 4 · p. 6

E. Ablation Study

The ablation study finds gains from receding-horizon fusion and longer temporal horizons, while the qualitative results show direct 3D predictions without geometric post-processing. The examples also illustrate how predictions follow recent motion rather than semantic object identity.

E. Ablation Study

  • 77.2% IoU_MOS is achieved by the best ablation model, which uses N=10 scans at 0.1 s resolution and fuses predictions.Across models, receding-horizon fusion improves IoU_MOS by up to 5.4 percentage points; for model [A], Bayesian fusion reduces false negatives by 8.2% and false positives by 18.9%.Source: E. Ablation Study · p. 7, Table III · p. 7
  • Larger temporal resolutions improve results over five scans at 0.1 s spacing, while untransformed point clouds yield 39.9% IoU_MOS with Bayesian fusion.A single-output-scan model cannot fuse additional predictions, and the two-scan model performs worse; the authors attribute the untransformed-input reduction to needing to infer both sensor ego-motion and object motion.Source: E. Ablation Study · p. 7

F. Qualitative Results

  • Against LMNet with kNN post-processing, the sparse voxel approach predicts directly in 3D without boundary effects in the illustrated car scene.The LMNet example retains artifacts, including bleeding false-positive labels behind the moving car.Source: F. Qualitative Results · p. 7, Figure 5 · p. 7
  • When a vehicle stops to yield, the method changes its prediction from moving to static, while still classifying a bicyclist as moving.The vehicle is labeled as a false negative against SemanticKITTI ground truth because those labels reflect whether an object moved during data collection, not only recent movement.Source: F. Qualitative Results · p. 7, Figure 6 · p. 7

G. Runtime

The conclusion reports improved SemanticKITTI performance and generalization to unseen data, while the runtime subsection quantifies inference and filtering costs.

G. Runtime

  • 0.078 s for 10 scans and 0.047 s for 5 scans; filtering adds 0.008 s and 0.004 s, respectively, on an NVIDIA RTX A5000.The network timings use an unoptimized Python implementation; filter timings are for fusing predictions.Source: G. Runtime · p. 7

V. CONCLUSION

  • The method improves performance on the SemanticKITTI moving object segmentation benchmark and generalizes well to unseen data.Source: §V · p. 8
  • The method predicts moving objects for all input scans; its receding-horizon strategy and binary Bayes filter extend the time horizon and increase robustness to false positives and negatives.Source: §V · p. 8
  • Odometry estimation and moving object segmentation are currently performed separately, and joint optimization is proposed as future work.Source: §V · p. 8

Related papers

Loading 2206.04129v1…