SqueezeSeg: Convolutional Neural Nets with Recurrent CRF for Real-Time Road-Object Segmentation from 3D LiDAR Point Cloud

• 3D LiDAR segmentation isolates and categorizes road-objects such as cars, pedestrians, and cyclists.Source: §I · p. 1
• Prior pipelines rely on hand-crafted features, multi-stage processing, and iterative algorithms that can compound errors, fail to generalize, or require substantial computation.Source: §I · p. 1
• SqueezeSeg uses an end-to-end CNN to output point-wise labels, refines them with a CRF, and obtains instance labels through conventional clustering.Source: §I · p. 1
• Spherical projection converts sparse 3D point clouds into dense 2D grid representations suitable for CNN processing.Source: §I · p. 2
• SqueezeSeg achieves high accuracy with fast, stable inference, and simulated noise-injected data further boosts validation accuracy on real-world data.Source: §I · p. 2

• Earlier LiDAR methods include mesh-based segmentation and iterative algorithms such as RANSAC and GP-INSAC for ground removal.Source: A. Semantic segmentation for 3D LiDAR… · p. 2
• Deep LiDAR approaches represent point clouds in two dimensions through projections or in three dimensions through voxelization and engineered features.Source: B. CNN for 3D point clouds · p. 2
• Image-segmentation research established fully convolutional networks for pixel-wise prediction and CRFs for refining segmentation details.Source: C. Semantic Segmentation for Images · p. 2
• Mean-field CRF inference was reformulated as a recurrent neural-network module, enabling CNN–CRF integration for segmentation.Source: C. Semantic Segmentation for Images · p. 2

• Point-wise and pixel-wise annotations are difficult to obtain, motivating synthetic datasets such as those generated from Grand Theft Auto.Source: D. Data Collection through Simulation · p. 2
• Naively discretizing sparse, irregular LiDAR points is inefficient and wastes computation.Source: A. Point Cloud Transformation · p. 2
• Spherical projection maps each LiDAR point to a dense, grid-based representation that resembles an ordinary image.Source: A. Point Cloud Transformation · p. 2
• The input uses 64 vertical channels, 512 front-view grids, and five point features including coordinates, intensity, and range.Source: A. Point Cloud Transformation · p. 2
• Each projected channel reflects structural information in the camera-view image.Source: Figure 2 · p. 3

• The network takes a 64 × 512 × 5 tensor and adapts SqueezeNet layers for LiDAR feature extraction.Source: B. Network structure · p. 3
• Because the input is much narrower in height than width, intermediate feature maps are downsampled only along width.Source: B. Network structure · p. 3
• Transposed convolutions restore full-resolution point-wise predictions, while skip-connections combine upsampled and lower-level feature maps.Source: B. Network structure · p. 3
• A softmax convolution produces the probability map, which is then refined by a recurrent CRF layer.Source: B. Network structure · p. 3
• FireModules and fireDeconvs replace standard convolution and deconvolution layers to reduce model parameters and computation.Source: B. Network structure · p. 3

• CNN down-sampling can blur label boundaries, motivating CRF refinement in SqueezeSeg.The paper links this loss of detail to max-pooling and notes that SqueezeSeg exhibits the same phenomenon.Source: C. Conditional Random Field · p. 3
• The CRF energy combines unary terms from CNN-predicted probabilities with binary penalties for assigning different labels to similar points.The unary term uses −log P(ci), while the binary term penalizes differing labels according to Gaussian-kernel similarities.Source: C. Conditional Random Field · p. 3
• Gaussian kernels model point similarity using angular position and Cartesian coordinates, with intensity and RGB available as additional features.The first kernel uses angular and Cartesian coordinates, whereas the second uses angular positions only.Source: C. Conditional Random Field · p. 3
• Exact CRF energy minimization is intractable, so SqueezeSeg uses approximate mean-field inference reformulated as an RNN.The recurrent formulation allows the CRF module to be trained end-to-end with the CNN.Source: C. Conditional Random Field · p. 3
• The recurrent CRF performs local message passing by filtering probability maps with small 3 × 5 Gaussian-kernel neighborhoods.The kernels decay rapidly with distance, allowing the implementation to restrict interactions to a local region.Source: C. Conditional Random Field · p. 4

• KITTI point-wise labels are generated by assigning object categories to points inside annotated 3D bounding boxes.This process produced 10,848 labeled images from the KITTI raw dataset.Source: D. Data collection · p. 4
• A GTA-V simulator mounts a virtual LiDAR scanner on an autonomously driven in-game car and collects corresponding point clouds and game-screen images.The virtual scanner and camera share a position, supporting consistency checks and reuse for sensor-fusion research.Source: D. Data collection · pp. 4–5
• Ray casting simulates each emitted laser ray and records hit coordinates, object classes, instance IDs, and object bounding boxes.These outputs provide labels and metadata useful for object and instance-wise segmentation.Source: D. Data collection · p. 5
• The simulator produced 8,585 synthesized samples, roughly doubling the training set, with synthetic noise augmented using KITTI's radial-coordinate noise distribution.Noise is added by setting all feature values to zero with a coordinate-dependent probability.Source: D. Data collection · p. 5
• Because GTA-V provides unrealistic pedestrian models and lacks a separate cyclist category, synthesized-data training and evaluation focus on the car class.The simulator is also limited in producing realistic pedestrian and cyclist labels.Source: D. Data collection · p. 5

• Class-level segmentation compares predicted and ground-truth labels, using IoU as the primary accuracy metric.Source: A. Evaluation metrics · p. 5
• Instance-level evaluation first matches each predicted instance to a ground-truth instance using the largest IoU.Ground-truth instances are sorted by point count before matching.Source: A. Evaluation metrics · p. 5
• Instance-level precision, recall, and IoU are computed for each object class from matched predicted and ground-truth point sets.Source: A. Evaluation metrics · p. 5

• The converted KITTI dataset is split into 8,057 training frames and 2,791 validation frames.Source: B. Experimental Setup · p. 5
• Training and validation frames come from separate sequences to avoid temporal correlation between the splits.Source: B. Experimental Setup · pp. 5–6
• The experiments limit the horizontal field of view to the forward-facing 90° because KITTI provides reliable 3D bounding boxes only there.Source: B. Experimental Setup · p. 6

Segmentation accuracy

• Car recalls exceed 90% at both class and instance levels, while pedestrian and cyclist performance is lower because these objects are fewer, smaller, and more detailed.Source: C. Experimental Results · p. 6
• Adding the recurrent CRF significantly improves car IoU, mainly by filtering misclassified border points, but slightly worsens pedestrian and cyclist segmentation.The authors suggest insufficient CRF hyperparameter tuning may explain the latter effect.Source: C. Experimental Results · p. 6

Runtime

• 8.7 ms per frame is the runtime without CRF, compared with 13.5 ms per frame with CRF on a TITAN X GPU.Both models have very small runtime standard deviations.Source: C. Experimental Results · p. 6

Simulated data

• Combining KITTI training data with GTA-simulated data significantly increases car segmentation accuracy beyond the baseline summarized in Table I.The simulator-based evaluation considers only cars because its pedestrian and cyclist labels are not sufficiently realistic.Source: C. Experimental Results · pp. 6–7

Qualitative results

• SqueezeSeg predictions are usually nearly identical to ground truth and also accurately segment objects assigned to KITTI’s “Don’t Care” category.The omitted objects may be obscured or too small for the benchmark labels.Source: Figure 8 · p. 6, C. Experimental Results · p. 7

• SqueezeSeg uses learned convolutional filters instead of hand-crafted features and reduces segmentation to a single stage.The design is intended to avoid propagated errors and leverage object context.Source: §V · p. 7
• The model achieves high accuracy with faster-than-real-time inference and low runtime variance for autonomous-driving applications.Source: §V · p. 7
• Adding synthesized data to training significantly boosts performance when validating on real-world data.Source: §V · p. 7