Source-linked AI summary
SqueezeSeg: Convolutional Neural Nets with Recurrent CRF for Real-Time Road-Object Segmentation from 3D LiDAR Point Cloud
Bichen Wu, Alvin Wan, Xiangyu Yue, Kurt Keutzer
TL;DR
Road-object segmentation from 3D LiDAR requires accurate, real-time categorization of cars, pedestrians, cyclists, and other obstacles. SqueezeSeg transforms point clouds for CNN processing, refines predictions with a recurrent CRF, and achieves fast, stable inference while synthetic training data improves real-world validation accuracy.
Problem
Accurate, real-time, robust segmentation must categorize and locate road-objects in 3D LiDAR point clouds despite limitations of hand-crafted, multi-stage pipelines.
Method
SqueezeSeg uses a spherical-projection CNN to produce point-wise labels, refines them with a recurrent CRF, and clusters points for instance-level labels.
Results
8.7 ms per frame with SqueezeSeg without CRF, while adding synthesized GTA data significantly increases accuracy on real-world validation data.
Takeaways & Limitations
SqueezeSeg provides accurate, fast, stable LiDAR road-object segmentation suitable for autonomous driving applications.
Abstract
from arXiv · showhide
In this paper, we address semantic segmentation of road-objects from 3D LiDAR point clouds. In particular, we wish to detect and categorize instances of interest, such as cars, pedestrians and cyclists. We formulate this problem as a point- wise classification problem, and propose an end-to-end pipeline called SqueezeSeg based on convolutional neural networks (CNN): the CNN takes a transformed LiDAR point cloud as input and directly outputs a point-wise label map, which is then refined by a conditional random field (CRF) implemented as a recurrent layer. Instance-level labels are then obtained by conventional clustering algorithms. Our CNN model is trained on LiDAR point clouds from the KITTI dataset, and our point-wise segmentation labels are derived from 3D bounding boxes from KITTI. To obtain extra training data, we built a LiDAR simulator into Grand Theft Auto V (GTA-V), a popular video game, to synthesize large amounts of realistic training data. Our experiments show that SqueezeSeg achieves high accuracy with astonishingly fast and stable runtime (8.7 ms per frame), highly desirable for autonomous driving applications. Furthermore, additionally training on synthesized data boosts validation accuracy on real-world data. Our source code and synthesized data will be open-sourced.
I. INTRODUCTION
SqueezeSeg targets real-time road-object segmentation from 3D LiDAR while avoiding the hand-crafted, multi-stage limitations of prior pipelines. It combines a CNN with a recurrent CRF and uses a spherical LiDAR representation to produce fast, stable predictions.
- 3D LiDAR segmentation isolates and categorizes road-objects such as cars, pedestrians, and cyclists.
- Prior pipelines rely on hand-crafted features, multi-stage processing, and iterative algorithms that can compound errors, fail to generalize, or require substantial computation.
- SqueezeSeg uses an end-to-end CNN to output point-wise labels, refines them with a CRF, and obtains instance labels through conventional clustering.
- Spherical projection converts sparse 3D point clouds into dense 2D grid representations suitable for CNN processing.
- SqueezeSeg achieves high accuracy with fast, stable inference, and simulated noise-injected data further boosts validation accuracy on real-world data.
II. RELATED WORK
Prior LiDAR segmentation spans hand-designed geometric pipelines and deep models operating on projected or voxelized data. CNN–CRF methods from image segmentation provide a foundation for combining learned predictions with refinement.
- Earlier LiDAR methods include mesh-based segmentation and iterative algorithms such as RANSAC and GP-INSAC for ground removal.
- Deep LiDAR approaches represent point clouds in two dimensions through projections or in three dimensions through voxelization and engineered features.
- Image-segmentation research established fully convolutional networks for pixel-wise prediction and CRFs for refining segmentation details.
- Mean-field CRF inference was reformulated as a recurrent neural-network module, enabling CNN–CRF integration for segmentation.
D. Data Collection through Simulation
The paper addresses sparse, irregular LiDAR data and difficult annotation by projecting point clouds onto a spherical grid. This produces a compact, image-like tensor while retaining structural channel information for CNN input.
- Point-wise and pixel-wise annotations are difficult to obtain, motivating synthetic datasets such as those generated from Grand Theft Auto.
- Naively discretizing sparse, irregular LiDAR points is inefficient and wastes computation.
- Spherical projection maps each LiDAR point to a dense, grid-based representation that resembles an ordinary image.
- The input uses 64 vertical channels, 512 front-view grids, and five point features including coordinates, intensity, and range.
- Each projected channel reflects structural information in the camera-view image.
B. Network structure
SqueezeSeg uses a lightweight SqueezeNet-derived CNN to process a 64 × 512 × 5 LiDAR tensor. It downsamples semantic features, restores resolution with skip-connected transposed convolutions, and reduces computation with fire modules.
- The network takes a 64 × 512 × 5 tensor and adapts SqueezeNet layers for LiDAR feature extraction.
- Because the input is much narrower in height than width, intermediate feature maps are downsampled only along width.
- Transposed convolutions restore full-resolution point-wise predictions, while skip-connections combine upsampled and lower-level feature maps.
- A softmax convolution produces the probability map, which is then refined by a recurrent CRF layer.
- FireModules and fireDeconvs replace standard convolution and deconvolution layers to reduce model parameters and computation.
C. Conditional Random Field
SqueezeSeg uses a CRF to refine blurry CNN label boundaries by combining CNN predictions with pairwise consistency among similar points, implemented through recurrent mean-field inference.
- CNN down-sampling can blur label boundaries, motivating CRF refinement in SqueezeSeg.The paper links this loss of detail to max-pooling and notes that SqueezeSeg exhibits the same phenomenon.
- The CRF energy combines unary terms from CNN-predicted probabilities with binary penalties for assigning different labels to similar points.The unary term uses −log P(ci), while the binary term penalizes differing labels according to Gaussian-kernel similarities.
- Gaussian kernels model point similarity using angular position and Cartesian coordinates, with intensity and RGB available as additional features.The first kernel uses angular and Cartesian coordinates, whereas the second uses angular positions only.
- Exact CRF energy minimization is intractable, so SqueezeSeg uses approximate mean-field inference reformulated as an RNN.The recurrent formulation allows the CRF module to be trained end-to-end with the CNN.
- The recurrent CRF performs local message passing by filtering probability maps with small 3 × 5 Gaussian-kernel neighborhoods.The kernels decay rapidly with distance, allowing the implementation to restrict interactions to a local region.
D. Data collection
The paper combines KITTI-derived point labels with a GTA-V LiDAR simulator to expand training data, while modeling KITTI noise in synthetic samples and restricting synthetic-data evaluation to cars.
- KITTI point-wise labels are generated by assigning object categories to points inside annotated 3D bounding boxes.This process produced 10,848 labeled images from the KITTI raw dataset.
- A GTA-V simulator mounts a virtual LiDAR scanner on an autonomously driven in-game car and collects corresponding point clouds and game-screen images.The virtual scanner and camera share a position, supporting consistency checks and reuse for sensor-fusion research.
- Ray casting simulates each emitted laser ray and records hit coordinates, object classes, instance IDs, and object bounding boxes.These outputs provide labels and metadata useful for object and instance-wise segmentation.
- The simulator produced 8,585 synthesized samples, roughly doubling the training set, with synthetic noise augmented using KITTI's radial-coordinate noise distribution.Noise is added by setting all feature values to zero with a coordinate-dependent probability.
- Because GTA-V provides unrealistic pedestrian models and lacks a separate cyclist category, synthesized-data training and evaluation focus on the car class.The simulator is also limited in producing realistic pedestrian and cyclist labels.
A. Evaluation metrics
The evaluation measures class-level and instance-level segmentation using precision, recall, and IoU, with IoU as the primary accuracy metric. Instance predictions are matched to ground-truth instances before computing these scores.
- Class-level segmentation compares predicted and ground-truth labels, using IoU as the primary accuracy metric.
- Instance-level evaluation first matches each predicted instance to a ground-truth instance using the largest IoU.Ground-truth instances are sorted by point count before matching.
- Instance-level precision, recall, and IoU are computed for each object class from matched predicted and ground-truth point sets.
B. Experimental Setup
Experiments use converted KITTI LiDAR data with sequence-disjoint training and validation splits. Because reliable KITTI bounding boxes cover only front-view scans, evaluation is restricted to a 90° forward-facing field of view.
- The converted KITTI dataset is split into 8,057 training frames and 2,791 validation frames.
- Training and validation frames come from separate sequences to avoid temporal correlation between the splits.
- The experiments limit the horizontal field of view to the forward-facing 90° because KITTI provides reliable 3D bounding boxes only there.
C. Experimental Results
SqueezeSeg attains high segmentation accuracy and fast runtime, with CRF effects varying by category. Combining KITTI and GTA-simulated data improves real-world car segmentation, while visualizations show accurate predictions including objects omitted from ground truth.
- Segmentation accuracy: Car recalls exceed 90% at both class and instance levels, while pedestrian and cyclist performance is lower because these objects are fewer, smaller, and more detailed.
- Segmentation accuracy: Adding the recurrent CRF significantly improves car IoU, mainly by filtering misclassified border points, but slightly worsens pedestrian and cyclist segmentation.The authors suggest insufficient CRF hyperparameter tuning may explain the latter effect.
- Runtime: 8.7 ms per frame is the runtime without CRF, compared with 13.5 ms per frame with CRF on a TITAN X GPU.Both models have very small runtime standard deviations.
- Simulated data: Combining KITTI training data with GTA-simulated data significantly increases car segmentation accuracy beyond the baseline summarized in Table I.The simulator-based evaluation considers only cars because its pedestrian and cyclist labels are not sufficiently realistic.
- Qualitative results: SqueezeSeg predictions are usually nearly identical to ground truth and also accurately segment objects assigned to KITTI’s “Don’t Care” category.The omitted objects may be obscured or too small for the benchmark labels.
V. CONCLUSIONS
SqueezeSeg is presented as an accurate, fast, stable, end-to-end LiDAR road-object segmentation approach. The paper also reports improved real-world validation performance when synthesized data is added to training.
- SqueezeSeg uses learned convolutional filters instead of hand-crafted features and reduces segmentation to a single stage.The design is intended to avoid propagated errors and leverage object context.
- The model achieves high accuracy with faster-than-real-time inference and low runtime variance for autonomous-driving applications.
- Adding synthesized data to training significantly boosts performance when validating on real-world data.