PaperlayerView exact revision on arXiv

Source-linked AI summary

SegMap: 3D Segment Mapping using Data-Driven Descriptors

Renaud Dubé, Andrei Cramariuc, Daniel Dugas, Juan Nieto, Roland Siegwart, Cesar Cadena

arXiv:1804.09557v2cs.ROcs.CV

TL;DR

SegMap addresses the need for compact, robust 3D LiDAR map representations that support more than localization alone. It partitions point clouds into segments and learns descriptors usable for localization, reconstruction, and semantic extraction. The resulting representation improves localization performance while retaining reconstruction, semantic, and compression capabilities demonstrated in real-world scenarios.

  • Problem

    Existing 3D LiDAR SLAM methods lack a compact representation that supports localization together with reconstruction and semantic information for real-time multi-robot use.

  • Method

    SegMap partitions 3D point clouds into segments and learns compact data-driven descriptors that can be retrieved for localization, decoded for reconstruction, and classified semantically.

  • Results

    SegMap outperforms hand-crafted features and the evaluated autoencoder baseline while achieving 91% average point correspondences in reconstructed segments and 89% training and 85% validation semantic accuracy.

  • Takeaways & Limitations

    The shared compact representation supports large-scale 3D LiDAR SLAM with low communication bandwidth and can provide reconstructed maps for navigation and situational awareness.

Abstract

from arXiv · show

When performing localization and mapping, working at the level of structure can be advantageous in terms of robustness to environmental changes and differences in illumination. This paper presents SegMap: a map representation solution to the localization and mapping problem based on the extraction of segments in 3D point clouds. In addition to facilitating the computationally intensive task of processing 3D point clouds, working at the level of segments addresses the data compression requirements of real-time single- and multi-robot systems. While current methods extract descriptors for the single task of localization, SegMap leverages a data-driven descriptor in order to extract meaningful features that can also be used for reconstructing a dense 3D map of the environment and for extracting semantic information. This is particularly interesting for navigation tasks and for providing visual feedback to end-users such as robot operators, for example in search and rescue scenarios. These capabilities are demonstrated in multiple urban driving and search and rescue experiments. Our method leads to an increase of area under the ROC curve of 28.3% over current state of the art using eigenvalue based features. We also obtain very similar reconstruction capabilities to a model specifically trained for this task. The SegMap implementation will be made available open-source along with easy to run demonstrations at www.github.com/ethz-asl/segmap. A video demonstration is available at https://youtu.be/CMk4w4eRobg.

I. INTRODUCTION

SegMap represents 3D LiDAR environments as compact, descriptive segments for localization and mapping. Its data-driven descriptors also support reconstruction and semantic extraction, enabling capabilities demonstrated in multi-robot urban and disaster scenarios.

  • SegMap partitions 3D point clouds into descriptive segments that provide compact environmental representations for localization and mapping.Segment-based features support efficient processing and data association.
  • Segment-based representations reduce computational, memory, and bandwidth requirements for real-time single- and multi-robot applications.The approach is designed for large-scale and long-term use where communication resources are limited.
  • The approach addresses limitations of hand-crafted segment features, including weak generalization across environments and limited access to 3D structure.The proposed descriptor is data-driven rather than hand-crafted.
  • The data-driven segment descriptor improves retrieval under viewpoint variation while supporting environment reconstruction and semantic extraction.Descriptors can be decoded into 3D reconstructions and classified semantically.
  • SegMap was evaluated with real-world multi-robot automotive and disaster-scenario datasets.The paper demonstrates the approach in urban driving and search-and-rescue settings.

II. RELATED WORK

Prior work established learning-based descriptors for 3D point-cloud tasks, but real-time outdoor localization with reusable representations remained limited. SegMap combines segment-based localization with reconstruction and semantic extraction in a unified representation.

  • CNNs have become a leading approach for learning-based 3D descriptors across detection, semantic segmentation, and 3D object generation.These applications motivate using learned features for point-cloud localization.
  • 3DMatch demonstrated viewpoint-robust keypoint retrieval indoors, but its real-time applicability to large-scale outdoor environments was unclear.The cited work had not been optimized for real-time operation and provided no timings.
  • SegMap combines data-driven segment localization with map reconstruction and semantic extraction while targeting real-time operation.Its unified representation is designed for single- and multi-robot systems.
  • Autoencoder-inspired architectures offer simultaneous compression and feature extraction, but optimal performance on both tasks is not guaranteed.The paper notes that reconstruction and viewpoint robustness can have conflicting goals.
  • SegMap extracts local segments incrementally, stores compact descriptors with centroids, and verifies candidate matches through geometric consistency.The pipeline retains the latest complete observation for the global map.
  • The compressed representation can be reconstructed and semantically interpreted, including for distinguishing static from dynamic objects.This semantic information can improve localization robustness to environmental changes.

IV. THE SegMap DESCRIPTOR

SegMap uses a learned descriptor for 3D segment point clouds that supports localization, map reconstruction, and semantic extraction. Its architecture converts voxelized segments into compact features through a 3D convolutional network.

  • The descriptor is designed as a unified representation for segment retrieval, 3D map reconstruction, and semantic extraction.The descriptor training addresses both retrieval and reconstruction, with semantic extraction performed afterward.
  • A. Descriptor extractor architecture: Each input segment is represented as a 32 × 32 × 16 binary voxel grid processed by three 3D convolutional layers, pooling layers, and two fully connected layers.The dimension was selected empirically to balance descriptiveness and network size.
  • A. Descriptor extractor architecture: The descriptor is obtained from the activations of the extractor’s final fully connected layer.The original segment scale is also supplied to the first fully connected layer to improve robustness to varying aspect ratios.

B. Segment alignment and scaling

SegMap aligns segments before voxelization and trains a shared descriptor with classification and reconstruction objectives. The descriptor is later used for retrieval, while decoding reconstructs segment geometry.

  • Alignment: Segments are aligned using gravity-aligned z axes and 2D PCA before descriptor extraction.
  • Scaling: Input segments are voxelized on a fixed grid, with individually enlarged voxels for oversized dimensions to reduce aliasing.
  • Joint training: The combined training loss adds classification and reconstruction losses as L = Lc + αLr.
  • Descriptor training: The descriptor uses classification classes based on segment or object observations, then removes the classification layer for deployment and k-NN retrieval.
  • Reconstruction: A decoder is trained alongside the descriptor, while decoding can run later because only descriptor extraction is required in real time.
  • Reconstruction: The reconstruction loss uses weighted binary cross entropy because training voxel grids are only 3% occupied, with γ = 0.9 yielding good results.

D. Knowledge transfer for semantic extraction

SegMap transfers the descriptor’s learned representation to a separate semantic classifier. The approach supports three semantic classes and can reject potentially dynamic objects during localization.

  • Knowledge transfer: A semantic extraction network is trained on top of the descriptor while the descriptor weights remain frozen.
  • Semantic classes: The classifier distinguishes vehicles, buildings, and other objects.
  • Localization use: The semantic labels can improve localization robustness and reduce map size by rejecting segments associated with potentially dynamic objects.
  • Reconstruction: Reconstructions in the illustrated KITTI examples remain recognizable despite compressing segments to 64 values, with voxelization more visible for buildings.

V. EXPERIMENTS

The experiments validate SegMap’s descriptor for localization, reconstruction, and semantic extraction, followed by large-scale demonstrations. The implementation uses real-time CNN inference with GPU acceleration and fast descriptor retrieval.

  • Experimental scope: The evaluation covers localization, reconstruction, semantic extraction, and two large-scale experiments.
  • Experimental setup: Experiments run on an Intel i7-6700K with an Nvidia GeForce GTX 980 Ti, using TensorFlow-based real-time CNN models.
  • Experimental setup: Descriptor retrieval uses libnabo for fast k-NN search in low-dimensional space, and the optimization backend uses iSAM2.

B. Baselines

SegMap is evaluated against eigenvalue-based features and a reconstruction-only autoencoder using KITTI-derived training and correspondence data. The protocol includes augmentation for viewpoint and occlusion variation.

  • Baselines: The baselines are Eigen, using eigenvalue-based features, and AE, using the same architecture trained only with reconstruction loss.
  • Data: KITTI sequences 05 and 06 generate training data, while sequence 00 is reserved for descriptor evaluation.
  • Data: Region growing produces 3300, 1750, and 810 segments from sequences 00, 05, and 06, respectively, averaging 12 observations per segment.
  • Augmentation: Training augmentation rotates segments and removes points on one side of random slicing planes to simulate viewpoint changes and occlusion.
  • Ground-truth generation: Ground-truth correspondences use GPS visits, retain segment pairs within 3.0 m centroid distance, and are split between training augmentation and validation.
  • Training: The classification and reconstruction losses are monitored while jointly training the descriptor and auxiliary networks.

D. Training the models

SegMap trains its descriptor and reconstruction decoder on segments from KITTI drives 05 and 06, then evaluates retrieval against eigenvalue-based and autoencoder descriptors. The descriptor is designed to support rapidly convergent segment matching for localization.

  • D. Training the models: The descriptor extractor and reconstruction decoder are trained using all segments extracted from KITTI drives 05 and 06.Training lasts three to four hours on a GPU.
  • D. Training the models: Validation classification accuracy converges to 41% for N = 2500 classes, reflecting the difficulty of distinguishing similar segments such as flat walls.The reported training accuracy is 43%.
  • D. Training the models: Retrieval performance is evaluated using 45M labelled descriptor pairs from KITTI sequence 00, with positives from ground-truth correspondences and negatives sampled beyond 20 m.Each positive sample is paired with one thousand randomly sampled negative segment pairs.
  • D. Training the models: As segments grow, k-NN retrieval aims to reduce the number of neighbours needed to identify the corresponding target segment and avoid false localizations.The descriptor should converge quickly toward the target segment descriptor.
  • D. Training the models: SegMap provides the best retrieval performance at every stage of segment growth, while the autoencoder performs worst early and improves later.This early retrieval behavior supports closing challenging loops.

F. Reconstruction performance

SegMap reconstructs recognizable 3D maps from compressed descriptors while retaining strong retrieval performance. Its 64-dimensional descriptor achieves high reconstruction correspondence and supports semantic classification, with performance affected by voxelization for larger segments.

  • F. Reconstruction performance: SegMap reconstructs recognizable 3D maps from descriptors, including large-scale buildings fused from multiple sensors.Voxelization error mostly affects larger segments downscaled to fit the voxel grid.
  • F. Reconstruction performance: The SegMap descriptor offers one order of magnitude better retrieval performance for over 40% of the segment-growing process.Figure 7 plots segment completeness against the number of target-map neighbours required to include the correct segment.
  • F. Reconstruction performance: 91% average point correspondence is achieved by the 64-dimensional SegMap descriptor, only slightly below the reconstruction-focused AE baseline.Correspondence counts points within one voxel distance between original and reconstructed segments.
  • F. Reconstruction performance: SegMap is presented as an all-rounder because its reconstruction performance is close to the AE baseline while its retrieval performance is higher.The comparison uses descriptor sizes reported in Table I.
  • F. Reconstruction performance: Semantic extraction reaches 85% validation accuracy from 20k labelled segment observations, illustrating additional use of the compressed representation.The training accuracy is 89%, and the authors state that semantic extraction is illustrative rather than a comparison objective.

1) Multi-robot SLAM in urban scenario:

In an urban multi-robot experiment, SegMap uses semantic filtering and descriptor associations to connect robot trajectories and build a shared map in real time. The system operates at 10.5 Hz while reducing transmitted map data relative to raw point clouds.

  • 1) Multi-robot SLAM in urban scenario:: Semantic descriptors reject segments classified as vehicles during a five-sequence playback of KITTI sequence 00 lasting 114 seconds.The experiment simulates a multi-robot setup on one computer.
  • 1) Multi-robot SLAM in urban scenario:: 113 global associations link all robot trajectories and create a common representation.ICP would refine the transformation by only 0.13 ± 0.06 m on average, near the voxelization resolution.
  • 1) Multi-robot SLAM in urban scenario:: Localization and map reconstruction run at an average frequency of 10.5 Hz, with segment description taking 28.4 ms per local cloud.Segment description accounts for 30% of the computing share, or 1.6 ms per segment.
  • 1) Multi-robot SLAM in urban scenario:: The final KITTI sequence 00 map contains 1341 segments, including 284 classified as vehicles.The bandwidth comparison estimates descriptor transmission from point and linking-data sizes while ignoring transfer overheads.
  • 1) Multi-robot SLAM in urban scenario:: The compressed mapping approach is intended to support mapping much larger environments.This conclusion follows the urban experiment’s map-size and bandwidth comparison.

2) Multi-robot SLAM in disaster environments:

SegMap is evaluated in two disaster-environment missions using UGVs at a powerplant and foundry. Despite using planar segments unlike those in training, the system performs multiple real-time localizations and reconstructs consistent maps.

  • 2) Multi-robot SLAM in disaster environments:: Two UGV missions collect data at the Gustav Knepper powerplant and Phoenix-West foundry using motor encoders, an IMU, and a rotating 2D LiDAR.The powerplant utility building measures 100 m by 25 m, while the foundry section measures 100 m by 40 m.
  • 2) Multi-robot SLAM in disaster environments:: An incremental smoothness-based region-growing algorithm extracts plane-like segments for both disaster-environment experiments.The resulting reconstructions are shown in Figure 9 and summarized in Table II.
  • 2) Multi-robot SLAM in disaster environments:: Multiple localizations are made in real time, enabling consistent map reconstruction in both experiments despite the planar segments differing from the training segments.The same descriptor approach is applied across the two environments.
  • 2) Multi-robot SLAM in disaster environments:: SegMap represents each segment with a distinctive, low-dimensional learning-based descriptor and identifies data associations through descriptor retrieval and matching.The conclusion describes this representation as the basis for the demonstrated localization and mapping behavior.
  • 2) Multi-robot SLAM in disaster environments:: The approach combines global localization, environment reconstruction, and semantic extraction while maintaining a high compression rate.The authors connect this combination to large-scale 3D-LiDAR SLAM with low communication bandwidth.
Loading 1804.09557v2…