PaperlayerView exact revision on arXiv

Source-linked AI summary

Moving Object Segmentation in 3D LiDAR Data: A Learning-based Approach Exploiting Sequential Data

Xieyuanli Chen, Shijie Li, Benedikt Mersch, Louis Wiesmann, Jürgen Gall, Jens Behley, Cyrill Stachniss

arXiv:2105.08971v2cs.RO

TL;DR

The paper addresses moving-object segmentation in 3D LiDAR data, where distinguishing moving objects from static or merely movable ones matters for autonomous systems. It combines sequential range images, residuals, and CNNs for online segmentation, reports improved urban-scene performance over existing methods, and introduces a benchmark.

  • Problem

    3D LiDAR systems need to distinguish actually moving objects from static or merely movable objects, while few benchmarks support LiDAR-based moving-object segmentation.

  • Method

    The approach projects LiDAR scans into range images, computes residual images from past scans, and feeds them with the current range image to a CNN for binary moving/static segmentation.

  • Results

    The method runs faster than the typical 10 Hz LiDAR frame rate and outperforms several state-of-the-art moving-object segmentation approaches in reported experiments.

  • Takeaways & Limitations

    The released SemanticKITTI-based benchmark and source code support transparent comparison and further research on LiDAR-based moving-object segmentation.

Abstract

from arXiv · show

The ability to detect and segment moving objects in a scene is essential for building consistent maps, making future state predictions, avoiding collisions, and planning. In this paper, we address the problem of moving object segmentation from 3D LiDAR scans. We propose a novel approach that pushes the current state of the art in LiDAR-only moving object segmentation forward to provide relevant information for autonomous robots and other vehicles. Instead of segmenting the point cloud semantically, i.e., predicting the semantic classes such as vehicles, pedestrians, roads, etc., our approach accurately segments the scene into moving and static objects, i.e., also distinguishing between moving cars vs. parked cars. Our proposed approach exploits sequential range images from a rotating 3D LiDAR sensor as an intermediate representation combined with a convolutional neural network and runs faster than the frame rate of the sensor. We compare our approach to several other state-of-the-art methods showing superior segmentation quality in urban environments. Additionally, we created a new benchmark for LiDAR-based moving object segmentation based on SemanticKITTI. We published it to allow other researchers to compare their approaches transparently and we furthermore published our code.

I. INTRODUCTION

The paper targets online 3D LiDAR moving-object segmentation that distinguishes actually moving objects from static or merely movable ones. It combines sequential range-image information with CNNs and reports faster-than-sensor-rate operation and improved segmentation over existing methods.

  • Moving-object segmentation supports safe autonomous navigation, future-state prediction, collision avoidance, planning, pose estimation, registration, and SLAM.
  • The task separates actually moving objects, such as driving cars, from static or non-moving objects, including buildings and parked cars.
  • The method uses CNNs with range images and residual images from past 3D LiDAR scans to label each current range measurement as moving or not.
  • The network exploits temporal information, is trained with binary moving/static labels derived from SemanticKITTI, and runs faster than the typical 10 Hz sensor frame rate.
  • The authors report more accurate moving-object segmentation than multiple existing methods and release a SemanticKITTI-based benchmark and source code.

II. RELATED WORK

Prior LiDAR approaches include map-based cleaning, semantic or potentially movable-object segmentation, motion estimation, and class-agnostic methods. This work operates online without a pre-built map, using sequential range scans and residuals to target moving-object segmentation with real-time capability.

  • Map-based LiDAR approaches often run offline, rely on prebuilt maps, and may require time-consuming ray casting or accurately aligned poses.
  • Related work also includes methods predicting potentially movable objects, estimating scene flow, and exploiting sequential scans for motion-aware classification or segmentation.
  • Map-free semantic segmentation methods generally identify potentially movable classes but do not distinguish actually moving objects from static members of the same class.
  • The proposed method uses range-projection segmentation networks with residuals between current and previous frames, operates online, and does not require a pre-built map.
  • The pipeline projects scans into range images, computes sequential residuals, concatenates them with range information, and feeds them to a segmentation CNN.

A. Range Image Representation

The method projects 3D LiDAR points into spherical and image coordinates to form a range-image representation. Each pixel stores range, spatial coordinates, and remission, enabling extra channels without changing existing network architectures.

  • Each LiDAR point p = (x, y, z) is mapped from R3 to R2 through spherical coordinates and then to image coordinates.
  • The representation uses image height h, width w, vertical field of view f, and point range r = ||p_i||_2 to determine pixel placement and stored measurements.
  • Pixels store range, x, y, z coordinates, and remission, allowing additional information to be added as channels.
  • Because the representation is compatible with existing range-image networks without architectural changes, the method is tested with three segmentation networks.

B. Sequence Information

The approach uses the current LiDAR scan together with recent scans from SLAM or odometry history. Estimated relative transformations align past scans to the current viewpoint so their sequential information can be used for segmentation.

  • Online segmentation uses only the current and recent LiDAR scans, enabling potential use in odometry estimation and dynamic-object removal from maps.
  • The current scan is denoted S_0, while a history of N previous scans contains homogeneous-coordinate points.
  • The method assumes N consecutive relative transformations estimated by SLAM or sensor-based odometry are available between scanning poses.
  • Using the estimated poses, points from a previous scan are transformed into the coordinate frame of another scan before sequential comparison.

C. Residual Images

The method converts sequential LiDAR scans into residual images that encode temporal differences after ego-motion compensation and reprojection. These residuals complement spatial range information, providing a cue for distinguishing moving objects from static background despite ambiguity and weak signals for slow motion.

  • C. Residual Images: Residual images compare the current LiDAR frame with transformed earlier frames to encode temporal differences.The pipeline compensates ego-motion, reprojects past scans into the current range-image view, and computes normalized range differences.
  • C. Residual Images: Each residual is computed only at valid corresponding pixels, while invalid pixels receive zero.The comparison uses the current range and the corresponding range from the transformed scan at the same image pixel.
  • C. Residual Images: Moving objects produce larger displacement-related residuals than static background after scans are aligned to a common viewpoint.Figure 3 illustrates continuous residual discrepancies caused by a moving car.
  • C. Residual Images: Residual patterns are ambiguous because one moving object can generate two large patterns, while slowly moving objects may produce weak patterns.Consequently, residual images alone provide a cue but do not yield strong moving-object segmentation performance directly.
  • C. Residual Images: The method concatenates residual images with the current range image, combining spatial information with temporal information as extra channels.The fused representation supplies coordinates, range, remission, and residual values to the segmentation network.

D. Range Projection-based Segmentation CNNs

The approach reuses established range-projection LiDAR segmentation CNNs rather than designing a new architecture. It feeds them fused range and residual images, retrains them for binary moving-object segmentation, and applies post-processing for projection artifacts.

  • D. Range Projection-based Segmentation CNNs: The method evaluates SalsaNext, RangeNet++, and MINet as existing range-projection CNNs for moving-object segmentation.SalsaNext and RangeNet++ use encoder-decoder architectures, while MINet uses a lightweight multipath architecture.
  • D. Range Projection-based Segmentation CNNs: A fast GPU-based k-Nearest-Neighbor search removes artifacts produced by range projection after segmentation.The post-processing preserves the real-time orientation of the selected range-projection methods.
  • D. Range Projection-based Segmentation CNNs: The networks receive fused range images with residual information instead of a newly designed network architecture.The networks are retrained with the proposed inputs while retaining their original loss functions.
  • D. Range Projection-based Segmentation CNNs: Using residual images produces a large moving-object-segmentation improvement across the evaluated segmentation networks.The comparison is conducted on the paper’s proposed MOS benchmark.

E. Moving Object Segmentation Benchmark

The paper introduces a LiDAR-only moving-object-segmentation benchmark built from SemanticKITTI to address the limited availability of such evaluation resources. It preserves the original odometry split while reorganizing labels into moving and non-moving classes and evaluates with IoU.

  • E. Moving Object Segmentation Benchmark: The benchmark addresses the shortage of datasets and benchmarks for 3D LiDAR-based moving-object segmentation.It is constructed as a new benchmark task using SemanticKITTI.
  • E. Moving Object Segmentation Benchmark: The benchmark uses SemanticKITTI’s original odometry split, training on sequences 00–10 and testing on sequences 11–21.SemanticKITTI contains 28 semantic classes and distinguishes moving from non-moving vehicles and humans.
  • E. Moving Object Segmentation Benchmark: The paper evaluates moving-object segmentation with the Jaccard Index, or intersection-over-union (IoU), over moving objects.IoU is defined using true positives, false positives, and false negatives for the moving class.
  • E. Moving Object Segmentation Benchmark: The experiments test whether LiDAR-only segmentation runs faster than 10 Hz, benefits from residual images, and outperforms several state-of-the-art networks.All methods are evaluated on the proposed MOS benchmark using SemanticKITTI odometry information.

A. Ablation Study on Input and Architecture

The ablations compare direct binary training, paired-frame inputs, and residual-image inputs across three range-projection networks. Temporal inputs improve MOS, with residuals achieving the best performance, while residual count and pose noise reveal diminishing returns and sensitivity to inaccurate poses.

  • A. Ablation Study on Input and Architecture: Three networks are tested with direct binary labels, two-frame inputs, and the proposed residual-image inputs.The evaluated networks are RangeNet++, SalsaNext, and MINet.
  • A. Ablation Study on Input and Architecture: Single-frame training limits separation of moving and static objects from the same semantic class, and MINet fails in this setup.The networks may instead learn scene heuristics, such as cars on roads being moving and cars in parking lots being static.
  • A. Ablation Study on Input and Architecture: Adding a second frame improves MOS because the networks gain temporal information, enabling MINet to predict moving objects in this setting.This is the second experimental setup in the ablation.
  • A. Ablation Study on Input and Architecture: Residual-image inputs achieve the best MOS performance, because they provide direct temporal differences rather than requiring the network to extract them from full range views.Direct residual access also reduces the time needed to extract temporal information from redundant successive frames.
  • A. Ablation Study on Input and Architecture: N = 1 residual image gives the largest MOS improvement, while additional images help with diminishing returns for N > 8.A separate pose-noise ablation shows performance drops with noisy poses, but noise above 20 units may be ignored by the network.

B. MOS Performance and Comparisons

The study evaluates LiDAR-based MOS against adapted semantic-segmentation and scene-flow baselines, testing temporal residual inputs, semantic information, and multiple residual-image histories. The N = 8 residual-image setup with semantics achieves the strongest reported comparison, while qualitative results show improved detection of slowly moving objects.

  • Comparisons: The evaluation compares the proposed method with adapted semantic-segmentation, scene-flow, and semantic-enhanced point-cloud methods on the MOS benchmark.Baselines include SalsaNext variants, FlowNet3D-based SceneFlow, and other point-cloud segmentation methods.
  • Proposed setups: The two proposed setups use SalsaNext with either N = 1 residual image or N = 8 residual images plus semantic information.The N = 8 configuration is selected as the best validation setup.
  • Results: N = 1 already outperforms most baselines but remains worse than KPConv, whose dense multiple-point-cloud processing prevents real-time performance.This comparison highlights a trade-off between segmentation performance and computational cost.
  • Results: The N = 8 residual-image method with semantic information outperforms all other compared methods.The supplied passage states the overall comparison but does not provide the table’s numerical metric values.
  • Qualitative results: In a challenging intersection scene, the method distinguishes moving and static points even when some moving objects travel slowly, while other methods fail to detect them.The qualitative comparison includes many moving and static objects simultaneously.

C. Applications

The authors apply MOS predictions as masks before LiDAR odometry and evaluate the resulting pipeline against SuMa and SuMa++. The masked approach improves odometry on KITTI training and test data and is slightly better than SuMa++.

  • 1) Odometry/SLAM:: MOS predictions are used as preprocessing masks to remove moving-object points before an off-the-shelf SLAM pipeline.The experiment compares SuMa, SuMa++, and SuMa+MOS on the KITTI odometry benchmark.
  • 1) Odometry/SLAM:: Applying the MOS mask improves odometry results on both KITTI training and test data and makes them slightly better than SuMa++.No additional pipeline modifications are reported beyond applying the MOS predictions as a mask.

2) 3D Mapping:

The mapping application uses MOS predictions to clean aggregated LiDAR maps by masking moving objects. The approach also remains faster than the frame rate of a typical rotating LiDAR sensor.

  • 2) 3D Mapping:: Applying MOS predictions as masks removes moving-object points from aggregated LiDAR scans and produces cleaner maps.The raw and cleaned aggregated point-cloud maps are compared directly, without further optimization.
  • 2) 3D Mapping:: In the range-projection visualization, red pixels represent moving objects.The figure presents qualitative MOS results in the range-image representation.
  • 2) 3D Mapping:: In the point-cloud visualization, range is encoded from purple for near points to yellow for far points, while red points indicate the moving class.Panel (a) shows the raw cloud, panel (b) ground truth, and panels (c,d) predictions.
  • Runtime: Using SalsaNext, the complete LiDAR odometry pipeline takes approximately 51 ms per scan, corresponding to 20 Hz and exceeding a typical 10 Hz LiDAR frame rate.The timing includes odometry estimation and residual-image generation, with network inference kept nearly unchanged.
  • Conclusion: The paper concludes that range projections and sequential information support online MOS while directly improving SLAM and mapping systems.The experiments report good MOS performance and outperformance of several state-of-the-art approaches.
Loading 2105.08971v2…