PaperlayerView exact revision on arXiv

Source-linked AI summary

SegMap: Segment-based mapping and localization using data-driven descriptors

Renaud Dubé, Andrei Cramariuc, Daniel Dugas, Hannes Sommer, Marcin Dymczyk, Juan Nieto, Roland Siegwart, Cesar Cadena

arXiv:1909.12837v1cs.ROcs.CV

TL;DR

Global localization in changing, dynamic environments is difficult because local features may be insufficiently discriminative and global descriptors can be coarse. SegMap represents 3D LiDAR maps with extracted segments and one compact learned descriptor for localization, reconstruction, and semantics. Across urban-driving and search-and-rescue experiments, the approach improves retrieval and localization while reducing odometry drift, with stated limitations in repetitive or featureless environments.

  • Problem

    Global localization in prior maps remains challenging in unstructured, dynamic environments, where local features are not sufficiently discriminative and global scene descriptors provide coarse information.

  • Method

    SegMap partitions 3D LiDAR point clouds into segments and uses a compact learned descriptor across global localization, map reconstruction, and semantic extraction.

  • Results

    SegMap improves localization performance through a data-driven 3D segment descriptor and is evaluated on real-world multi-robot automotive and disaster-scenario datasets.

  • Takeaways & Limitations

    SegMap supports large-scale LiDAR SLAM and 3D reconstruction with low communication bandwidth, while reconstructed maps can support navigation and situational awareness.

  • Takeaways & Limitations

    SegMap is limited to surrounding geometry, making repetitive man-made or featureless environments challenging because of perceptual aliasing or unreliable segment extraction.

Abstract

from arXiv · show

Precisely estimating a robot's pose in a prior, global map is a fundamental capability for mobile robotics, e.g. autonomous driving or exploration in disaster zones. This task, however, remains challenging in unstructured, dynamic environments, where local features are not discriminative enough and global scene descriptors only provide coarse information. We therefore present SegMap: a map representation solution for localization and mapping based on the extraction of segments in 3D point clouds. Working at the level of segments offers increased invariance to view-point and local structural changes, and facilitates real-time processing of large-scale 3D data. SegMap exploits a single compact data-driven descriptor for performing multiple tasks: global localization, 3D dense map reconstruction, and semantic information extraction. The performance of SegMap is evaluated in multiple urban driving and search and rescue experiments. We show that the learned SegMap descriptor has superior segment retrieval capabilities, compared to state-of-the-art handcrafted descriptors. In consequence, we achieve a higher localization accuracy and a 6% increase in recall over state-of-the-art. These segment-based localizations allow us to reduce the open-loop odometry drift by up to 50%. SegMap is open-source available along with easy to run demonstrations.

1 Introduction

SegMap addresses the difficulty of prior-free global localization in changing, dynamic environments by representing 3D LiDAR maps with descriptive segments and compact learned features. The approach supports localization, reconstruction, and semantic extraction from shared representations.

  • Visual place-recognition systems struggle with seasonal, weather, and day-night changes, while LiDAR SLAM systems generally lack prior-free global localization.
  • Local 3D features can be insufficiently discriminative and non-repeatable under environmental change, while matching them also carries substantial computational cost.
  • SegMap partitions 3D point clouds into descriptive segments and combines global-descriptor efficiency with local-feature matching to provide real-time 6-DoF localization.
  • A learned segment descriptor supports localization, 3D environment reconstruction, and semantic information extraction from the same compact representation.
  • The paper evaluates SegMap on real-world, multi-robot automotive and disaster-scenario datasets and compares its localization and training methods with prior approaches.
  • SegMap is released as open-source software with demonstrations, and its compact features are designed for computational, memory, and bandwidth constraints.

2 RELATED WORK

Prior 3D localization methods use local features, global descriptors, or geometric objects, each with limitations in discriminativeness, viewpoint robustness, assumptions, or scalability. Learning-based 3D methods broaden descriptor capability but leave outdoor real-time applicability and compression-reconstruction trade-offs relevant.

  • 3D point-cloud place recognition commonly follows local-feature, global-descriptor, or plane/object-based approaches.
  • Local keypoint methods establish geometric correspondences but often lack descriptive power and can be computationally expensive when many features are processed.
  • Global descriptors summarize scans with histograms or grid-based shape properties, but their performance can be sensitive to viewpoint variation.
  • Plane- and object-based methods address structural matching, yet a general solution should avoid assuming simplistic primitives or requiring a rich object library.
  • CNN-based descriptors have shown strong results for 3D tasks, although the applicability of some approaches to real-time, large-scale outdoor environments remains unclear.
  • Autoencoder architectures combine compression and feature extraction, but reconstruction and viewpoint-robust retrieval can have conflicting objectives.

3 The SegMap approach

SegMap builds a unified 3D LiDAR representation by extracting segments, encoding them with compact learned descriptors, matching them for localization, and reusing them for reconstruction and semantics. The pipeline is designed for transferable multi-robot mapping under constrained communication.

  • SegMap comprises segment extraction, description, localization, map reconstruction, and semantic extraction modules forming one transferable representation.
  • Segment extraction: Point clouds are accumulated in a dynamic voxel grid, then incrementally segmented using Euclidean-distance or curvature-based region-growing methods.
  • Description: Each segment is encoded into a compact descriptor, and the global map retains the descriptor from the latest and most complete observation.
  • Localization: Localization retrieves candidate segment correspondences with k-NN in feature space and verifies geometric consistency using segment centroids before estimating a 6-DoF transformation.
  • Reconstruction: The autoencoder-like descriptor can reconstruct an approximate map and transmit its 3D information over wireless networks with limited bandwidth.
  • Semantics: The descriptor contains semantic information that can distinguish vehicles, buildings, and other classes, including static versus dynamic objects for localization.

4 The SegMap Descriptor

SegMap is a data-driven descriptor for 3D segment point clouds that supports localization, reconstruction, and semantic extraction. Its encoder compresses voxelized segments into descriptors, while jointly trained retrieval and reconstruction objectives support these capabilities.

  • 4.1 Descriptor extractor architecture: SegMap uses a 32 × 32 × 16 binary voxel grid and a CNN with 3D convolutions, max pooling, and fully connected layers to produce descriptors.The segment’s original scale is also passed to the first fully connected layer to improve robustness to voxelization at different aspect ratios.
  • 4.3 Training the SegMap descriptor: The descriptor is trained with a combined loss that merges softmax cross entropy for retrieval and reconstruction loss for map reconstruction.The relative contribution of the two losses is controlled by α.
  • 4.3 Training the SegMap descriptor: Training organizes observations into classes representing segments or object and environment parts, then removes the classification layer at deployment for k-NN retrieval.The resulting descriptor is taken from the activations of the preceding fully connected layer.
  • 4.3 Training the SegMap descriptor: A decoder network reconstructs voxelized segments from the compact descriptor, while decoding may be performed offline even though descriptor extraction runs in real time.The decoder has one fully connected and three deconvolutional layers with a sigmoid output.
  • 4.4 Semantic extraction: A semantic network trained on frozen descriptors distinguishes vehicles, buildings, and other objects, enabling dynamic-segment rejection for more robust localization and smaller maps.The semantic network is appended to the descriptor rather than changing the descriptor extractor itself.
  • 4.5 SegMini: SegMini halves convolutional filter counts and dense-layer sizes, providing 2x GPU and 6x CPU speedups without much retrieval-performance loss.It retains the SegMap architecture while targeting resource-limited platforms.

5 EXPERIMENTS

The experiments evaluate SegMap’s descriptor and complete system using real-world KITTI and multi-scenario datasets. Training data are generated from segmented LiDAR sequences, augmented for viewpoint, occlusion, and noise robustness, and paired through GPS-based correspondences.

  • 5 EXPERIMENTS: The evaluation covers localization, reconstruction, semantic extraction, and complete SegMap experiments in multiple real-world settings.The implementation uses real-time TensorFlow CNNs, fast low-dimensional k-NN retrieval, and an iSAM2 optimization backend.
  • 5.1 Experimental setup: SegMap training and testing use KITTI sequences 05 and 06, while sequence 00 is reserved for descriptor validation and sequences 00 and 08 support end-to-end experiments.Segments are extracted with incremental Euclidean distance-based region growing.
  • 5.1 Experimental setup: Data augmentation simulates viewpoint changes through rotations, occlusion by removing points on one side of a slicing plane, and noise by removing up to 10% of points.The occlusion simulation removes no more than 50% of a segment’s points.
  • 5.1 Experimental setup: GPS readings identify ground-truth segment correspondences between repeated visits, retaining pairs whose centroids are within 3.0 m and whose convex-hull overlap satisfies the correspondence condition.The procedure generates 150, 260, and 320 correspondences from sequences 00, 05, and 06 respectively.

5.3 Training the models

The models combine classification and reconstruction objectives, and retrieval is evaluated against handcrafted and learned descriptor variants using ROC and segment-growth analyses. SegMap provides strong retrieval performance, particularly for complete segments, while the evaluation notes limitations of ROC for k-NN retrieval.

  • 5.3 Training the models: Training combines classification and scaled reconstruction losses for the descriptor extractor and associated networks.The reconstruction loss shown in Figure 6 is already scaled by α.
  • 5.3 Training the models: 41% of validation samples were correctly assigned to one of N = 2500 classes.The reported accuracy reflects the large number of classes and visually similar training samples.
  • 5.4 Descriptor retrieval performance: The retrieval study compares SegMap with eigenvalue-based features and networks trained primarily for classification, reconstruction, or triplet loss.The evaluation uses 45M labeled descriptor pairs from KITTI sequence 00, with ground-truth correspondences and negatives sampled beyond 20 m.
  • 5.4 Descriptor retrieval performance: ROC is not optimal for this retrieval task because localization uses per-query k-NN ordering rather than one global distance threshold.The practical objective is to reduce the number of neighbours needed to retrieve the correct segment as it grows.
  • 5.4 Descriptor retrieval performance: Over most of the segment-growing process, SegMap achieves one order of magnitude better retrieval performance than the handcrafted baseline.Figure 8 measures the median number of target-map neighbours needed to retrieve the correct segment; lower is better.
  • 5.4 Descriptor retrieval performance: SegMap performs best for very complete segments while maintaining comparable performance across very partial observations.The autoencoder is weaker early but improves later, whereas triplet-loss performance can suffer for very similar segments.

5.5 Reconstruction performance

SegMap reconstructs 3D segments from the compact features used for localization, producing recognizable reconstructions and strong point correspondence while balancing retrieval and reconstruction.

  • 5.5 Reconstruction performance: SegMap reconstructs 3D maps using a decoding CNN applied to the descriptor features.Examples include objects from KITTI sequence 00 and larger buildings reconstructed by fusing multiple sensors.
  • 5.5 Reconstruction performance: The reconstruction metric counts original and reconstructed points corresponding within one voxel distance.This tolerance evaluates shape preservation without requiring every individual point to be reconstructed.
  • 5.5 Reconstruction performance: 91% of points correspond on average for the 64-dimensional SegMap descriptor, only slightly below the autoencoder baseline.The comparison uses different descriptor sizes and measures correspondence between original and reconstructed segments.
  • 5.5 Reconstruction performance: The reconstructions remain recognizable despite the high compression ratio.Voxelization quantization error mainly affects larger segments that are downscaled to fit the voxel grid.

5.6 Semantic extraction performance

The supplied passages describe the global-localization evaluation on KITTI 00, where SegMap retrieves more accurate poses and localizes more scans than LocNet. They do not provide semantic-extraction performance details.

  • 5.6 Semantic extraction performance: The evaluation passages supplied for this section do not report semantic-extraction accuracy or other semantic-performance results.The available evidence here concerns global localization and pose retrieval rather than semantic labeling.
  • 5.6 Semantic extraction performance: SegMap is evaluated against LocNet on 4541 scans from the KITTI 00 odometry sequence.The comparison uses the first 3000 scans to create a map and the last 1350 scans for localization.
  • 5.6 Semantic extraction performance: SegMap successfully localizes about 6% more scans than LocNet and returns more accurate localized poses.Only 65% of query positions lie within 50 m of the target map, limiting the maximum possible saturation.

5.8 A complete mapping and localization system

SegMap is combined with LOAM to form a real-time LiDAR-only mapping and localization pipeline. On KITTI sequences 00 and 08, global localizations improve long-path trajectory accuracy and reduce drift by up to two times.

  • 5.8 A complete mapping and localization system: SegMap is combined with LOAM to evaluate global localization alongside LiDAR odometry and mapping.The resulting system uses segment-based maps and adds global-localization constraints to correct drifting odometry.
  • 5.8 A complete mapping and localization system: The complete pipeline is real-time, LiDAR-only, and produces segment-based maps with loop closures.Global-localization results are added as graph constraints to correct odometric drift.
  • 5.8 A complete mapping and localization system: 64 neighbours and at least 7 geometrically consistent correspondences are required to output a localization.The system uses a local map with a 50 m radius around the robot, with parameters chosen empirically.
  • 5.8 A complete mapping and localization system: Up to 2 times less trajectory drift is obtained on longer KITTI paths when SegMap is combined with LOAM.Both translation and rotation errors improve, while shorter paths improve marginally or remain unchanged.

5.9 Multi-robot experiments

SegMap was evaluated in urban-driving and search-and-rescue multi-robot experiments, using segment descriptors for real-time localization, map reconstruction, and semantic filtering. The experiments produced global associations and consistent reconstructions while maintaining compact processing and communication requirements.

  • Experiments: Three large-scale multi-robot experiments evaluated SegMap in one urban-driving and two search-and-rescue scenarios.The same model was trained on KITTI sequences 05 and 06 and used in indoor and outdoor settings.
  • KITTI urban-driving experiment: Vehicle-classified segments were rejected from retrieval in the KITTI multi-robot setup.Five simultaneous playback sequences simulated the multi-robot experiment for 114 seconds.
  • KITTI urban-driving experiment: 113 global associations linked all robot trajectories into a common representation.ICP refinement averaged 0.13 ± 0.06 m but would require retaining and transmitting original point clouds.
  • KITTI urban-driving experiment: 10.5 Hz average localization and map-reconstruction frequency was achieved, with segment description taking 30% of runtime and 28.4 ms per local cloud.The reconstructed target map was generated from descriptors.
  • KITTI urban-driving experiment: The final KITTI sequence 00 map contained 1341 segments, including 284 classified as vehicles.The bandwidth estimate counted point coordinates and descriptor-to-trajectory links while ignoring transfer overhead.
  • Search-and-rescue experiments: Multiple real-time localizations enabled consistent reconstructions at both the Gustav Knepper powerplant and Phoenix-West foundry despite differing segment characteristics from training.The experiments used plane-like segments extracted by incremental smoothness-based region growing.

6 DISCUSSION AND FUTURE WORK

The discussion identifies geometry-only perception, repetitive structures, featureless environments, and segmentation variability as limitations, while proposing multimodal and temporal extensions. Future work must balance richer information against real-time computational costs.

  • Limitations: Geometry-only observation can cause perceptual aliasing in repetitive man-made environments, affecting descriptors and geometric consistency verification.The authors suggest detecting aliasing and tightening geometric-verification constraints.
  • Limitations: Featureless environments such as flat fields or straight corridors hinder segment extraction and can let the map drift until a distinct loop closure is reached.The text notes that LiDAR-based odometry also struggles to maintain accurate pose estimates in these settings.
  • Future work: The two segmentation algorithms have different strengths: Euclidean-distance segmentation suits outdoor scenarios, whereas curvature-based segmentation better fits indoor scenarios.Running both strategies in parallel is proposed to compensate for their shortcomings across environments.
  • Future work: Adding camera color and semantic information could improve descriptors for segments with little distinctive geometry and strengthen segmentation robustness.The authors flag additional computational expense from processing and combining more modalities under real-time constraints.
  • Future work: Incremental descriptor updates and recurrent architectures are proposed to make description more efficient and use temporal segment evolution.The present work performs segment description discretely with a feed-forward network.

7 CONCLUSION

SegMap represents 3D-sensor environments as segments with compact learned descriptors, enabling descriptor-based associations for localization and mapping. The conclusion reports stronger descriptive performance than evaluated alternatives, drift correction, lightweight deployment, reconstruction, and semantic extraction.

  • Conclusion: SegMap decomposes robot surroundings into segments represented by distinctive, low-dimensional learning-based descriptors.Descriptor retrieval and matching identify data associations through repeatable segment features.
  • Conclusion: SegMap's descriptive power outperforms handcrafted features and the evaluated data-driven baseline solutions.The experiments also indicate competitive localization performance compared with LocNet.
  • Conclusion: Reconstructions of the Gustav Knepper powerplant and Phoenix-West foundry illustrate the mapping experiments.The point clouds are colored by height and estimated robot trajectories are shown as colored lines.
  • Conclusion: SegMap localization helps correct open-loop odometry drift when combined with LOAM, while SegMini supports deployment on platforms with limited computational power.SegMini is introduced as a lightweight version of the SegMap descriptor.
  • Conclusion: The descriptor supports global localization, 3D environment-map reconstruction, and semantic-information extraction.These capabilities are presented as demonstrated functions of the SegMap descriptor.
Loading 1909.12837v1…