PaperlayerView exact revision on arXiv

Source-linked AI summary

Range Conditioned Dilated Convolutions for Scale Invariant 3D Object Detection

Alex Bewley, Pei Sun, Thomas Mensink, Dragomir Anguelov, Cristian Sminchisescu

arXiv:2005.09927v3cs.CVcs.LGcs.RO

TL;DR

Range-image LiDAR detection offers compact processing but must address scale variation and occlusion. This paper introduces range-conditioned dilation with soft range gating and a second-stage box-refinement network. On the Waymo Open Dataset, the authors report that their model significantly outperforms other methods on both public leaderboard difficulty levels.

  • Problem

    Range-image detectors must handle scale variation and occlusion while using a representation that avoids long-range sparsity.

    Source: §1 · p. 2

  • Method

    The paper combines range-conditioned dilation and soft range gating in a range-image detector with a second-stage RCNN for box refinement.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    The range-based RCD model significantly outperforms other methods at both difficulty levels on the public Waymo Open Dataset benchmark.

    Source: §5.3 · p. 8

  • Takeaways & Limitations

    The authors report that the method sets a new state-of-the-art for vehicle detection at long distances on Waymo Open Dataset.

    Source: §6 · p. 9

    The authors note that KITTI's small size can lead to overfitting and poor generalization for range-image methods.

    Source: E Comparison to LaserNet on KITTI · p. 14

Abstract

from arXiv · show

This paper presents a novel 3D object detection framework that processes LiDAR data directly on its native representation: range images. Benefiting from the compactness of range images, 2D convolutions can efficiently process dense LiDAR data of a scene. To overcome scale sensitivity in this perspective view, a novel range-conditioned dilation (RCD) layer is proposed to dynamically adjust a continuous dilation rate as a function of the measured range. Furthermore, localized soft range gating combined with a 3D box-refinement stage improves robustness in occluded areas, and produces overall more accurate bounding box predictions. On the public large-scale Waymo Open Dataset, our method sets a new baseline for range-based 3D detection, outperforming multiview and voxel-based methods over all ranges with unparalleled performance at long range detection.

1 Introduction

The paper motivates range images as a compact native representation for efficient LiDAR detection, while identifying scale variation and occlusion as challenges. It proposes range-conditioned dilation, soft range gating, and a second-stage box-refinement network.

1 Introduction

  • Voxel methods incur memory and computation demands, BEV projections lose information through quantization, and point-based methods face long-range sparsity and costly neighborhood operations.These approaches frame the paper's motivation for a different LiDAR detection representation.Source: §1 · pp. 1–2
  • Range images project Cartesian points onto unique spherical-image pixels, encoding range in pixel values and inclination and azimuth in row and column indices.Their compactness supports efficient 2D convolutions, and they do not suffer from long-range sparsity.Source: §1 · p. 2
  • The proposed detector uses range-conditioned dilation to adjust receptive-field scale with measured range, with soft range gating and second-stage local box refinement addressing scale variation and occlusion.The paper presents the method as an efficient two-stage range-image detector.Source: §1 · p. 2
  • The authors report especially strong long-range performance, where voxel- and sparse-point-based approaches suffer from point sparsity.They describe the work as the first combination of a range-image network with an RCNN second stage for 3D object detection.Source: §1 · p. 2

2 Related Work

The related-work section reviews rasterized, voxel, point-based, and range-image approaches to 3D LiDAR detection, then situates range-conditioned dilation among adaptive and dilated convolution methods.

2.1 3D LiDAR Detection

  • Voxel and BEV approaches process discretized point clouds with 3D or 2D networks, while point-based methods extract point features and regress 3D boxes.The paper benchmarks against PV-RCNN on Waymo Open Dataset.Source: §2.1 · p. 2
  • Range images are compact and avoid sparsity-related issues, but require more training data, sensor information for high-quality generation, and methods to handle scale variance and occlusion.LaserNet is identified as the primary representative range-image detector.Source: §2.1 · pp. 2–3

2.2 Adaptive and Dilated Convolutions

  • Prior adaptive-convolution methods use learned dilation rates, depth weighting, or dynamic filters, while ASPP combines a fixed set of dilation rates.The discussion describes these approaches as alternatives or context for the proposed method.Source: §2.2 · p. 3
  • Unlike fixed-rate combinations, the proposed sampler uses range observations to rescale dilation and reuse kernel weights across scales, with a continuous pattern that adapts during training.The paper relates this approach to distance-based input filtering extended to 3D point clouds represented as range images.Source: §2.2 · p. 3

3 Range Condition to Learn Scale Invariant Features

The proposed range-conditioned dilation block uses range-image measurements to adapt a convolution's spatial sampling scale, and can be placed wherever a range image is available.

3.1 Range Conditioned Dilated Convolution

  • The RCD block replaces discrete kernel lookups with sparse, local, continuous spatial sampling conditioned on input range, adapting the convolution's spatial extent.The paper describes it as a range-conditioned form of deformable convolution.Source: §3.1 · p. 3
  • A learnable relative sampling pattern is shared across pixels, then transformed using each pixel's range and the pixel coordinates.The pattern is initialized as a uniform grid with mean (0, 0); λ is a learnable nominal receptive-field width.Source: §3.1 · p. 4
  • The sampler gathers N spatial samples per pixel, and bilinear sampling provides spatial derivatives that pass loss gradients to λ and the sampling pattern.The resulting sampled tensor contains N features per pixel.Source: §3.1 · p. 4
  • The dilation multiplier varies with distance and nominal object size λ, with its angular units converted to pixels using the LiDAR angular resolution.Figure 3 depicts the rate as a function of distance and λ.Source: Figure 3 · p. 4, §3.1 · p. 4
  • Samples beyond vertical image boundaries are clamped, while horizontal samples wrap around to use the range image's 360° view.These rules handle sampling coordinates that extend outside the input image.Source: §3.1 · p. 4
  • Soft range gating applies Gaussian weights to sampled features, reducing the importance of distant points likely to belong to a different object from the convolution center.The gate's learnable variance γ controls its length scale and is initialized to 1 meter.Source: §3.1 · p. 4
  • Pointwise convolutions split features between the spatial sampler and pass-through path, then concatenate their outputs and complete the convolution.The sampler input projection reduces feature dimensionality and computation; in the RCD experiments, C = 3.Source: §3.1 · pp. 4–5
  • The RCD block sits in the first-stage RPN, which proposes boxes for points; high-scoring proposals proceed to a second RCNN stage for refinement.The figure shows range-image input of size 64 × 2650 × 8.Source: Figure 4 · p. 5

4 Two Stage Detection Network

The detector places RCD blocks in an RPN stage and refines high-scoring proposals with an RCNN stage. The RPN and RCNN are trained jointly.

4.1 First stage: RCD-RPN

  • The RCD-RPN uses a fully convolutional deep layer aggregation architecture and predicts per-point classification scores and 7D boxes.It also predicts whether each point’s top and left neighboring pixels belong to the same object.Source: §4.1 · p. 5
  • The RPN uses focal loss for its three classification scores and bin loss for 3D box regression.Focal loss addresses class imbalance and focuses on difficult boxes; bin loss discretizes center distances and heading before regression within each bin.Source: §4.1 · p. 5

4.2 Second stage: 3D RCNN

  • The RCNN refines initial proposals using pooled point features and predicts final box parameters and classification scores.The two-stage design is described as improving box accuracy and mitigating occlusion.Source: §4.2 · p. 6
  • The second-stage box loss uses residual loss for centers and dimensions and bin loss for heading.The RCNN also averages classification loss over selected proposals, with M = 100 during training.Source: §4.2 · p. 6

4.3 Joint Training:

  • Both the RPN and RCNN networks are trained jointly with a final objective.Source: §4.3 · p. 6

5 Experiments

The experiments evaluate the range-image detector on KITTI and Waymo, comparing it with range-image, voxel, and other 3D detection methods. On Waymo, the paper reports strong results across difficulty levels and ranges, and examines the effects of RCD, SRG, and the two-stage refinement network.

5 Experiments

  • The evaluation uses Waymo validation and test sets, with validation metrics including 3D and BEV AP and APH for vehicles.The Waymo data described here comprises 1,000 sequences and has a 75-meter effective annotation radius.Source: §5 · p. 6

5.1 Baseline Methods

  • The Waymo comparisons include a range-image detector, voxel and point-based methods, and multiview fusion; LaserNet is the only baseline described as directly using range imagery.The authors also compare the full RCD model with a two-stage baseline using fixed dilation and no SRG.Source: §5.1 · p. 7

5.2 Implementation details

  • The RCD block uses 23K FLOPs per pixel, compared with 262K for an equivalent 2D convolution, and the RPN takes 74 ms per frame versus 301 ms for the PV-RCNN RPN implementation.Validation runs use 350K iterations; the Waymo test submission model uses 1 million iterations.Source: §5.2 · p. 7

5.3 Discussion of Results

  • 57.2 AP: the RPN alone improves on the direct range-image comparison, LaserNet.The authors attribute the gain mainly to RCD and SRG handling scale variance and occlusion in range images.Source: §5.3 · p. 7, Table 2 · p. 7
  • The range-based RCD model significantly outperforms other methods on the Waymo Open Dataset test benchmark at both difficulty levels.Level 1 includes objects with at least five points; Level 2 includes objects with as few as one point.Source: Table 3 · p. 8, §5.3 · p. 8
  • The method shows its greatest improvements over voxel-based methods at long range, while PV-RCNN is strongest at close range.The paper links distant-object voxel sparsity to the long-range results and notes that RCD can use range-image context for objects with few points.Source: §5.3 · p. 8
  • RCD with SRG achieves the best single-stage ablation performance, while ASPP performs worst; the 3D RCNN can recover from an underperforming ASPP-based RPN.RCD reuses the same filter weights across a continuum of dilation rates, unlike ASPP's combination of fixed rates.Source: §5.3 · p. 8, Table 4 · p. 8

6 Conclusions

The paper introduces range-conditioned dilation for LiDAR range images and reports a strong Waymo benchmark result across ranges. It also identifies other 2D range-image sensors and a hybrid detector as future research directions.

6 Conclusions

  • RCD dynamically adjusts the dilation rate for LiDAR range images to support scale-invariant 3D object detection.Source: §6 · p. 8
  • The RCD-based method is reported as the top-performing range-image-based detector across ranges on the Waymo Open Dataset BEV benchmark, with competitive results in other tests.The paper specifically reports a new state of the art for long-distance vehicle detection.Source: §6 · pp. 8–9
  • Future work includes applying RCD to structured-light or time-of-flight cameras and combining it with PV-RCNN.The paper notes that these other sensors also capture 2D range images.Source: §6 · p. 9

A Further Implementation Details

The network input follows the Waymo range-image format, with rows and columns corresponding to LiDAR beam inclinations and laser-shot azimuth. Each pixel includes range and other sensor or geometric channels.

A.1 Input Range Image

  • Rows encode LiDAR beam inclinations, while columns encode laser-shot azimuth.Source: §A.1 · p. 12
  • The range-image pixel channels include range, intensity, elongation, inclination, azimuth, and Cartesian x, y, z coordinates.Source: §A.1 · p. 12
  • The x, y, and z channels give each point's Cartesian coordinates in the vehicle frame.Source: §A.1 · p. 12

A.2 RCD-RPN Backbone Network

The RCD-RPN uses a multiresolution 2D convolutional backbone, with RCD blocks available at selected stages and range information downsampled to condition dilation. The RPN comparison reports further AP and APH gains when soft range gating is added.

A.2 RCD-RPN Backbone Network

  • The RCD-RPN backbone is a 2D convolutional network inspired by deep layer aggregation and LaserNet, with ResNet bottleneck units and aggregation modules.Aggregation modules combine feature maps at different resolutions by upsampling the lower-resolution map with transposed convolutions.Source: §A.2 · p. 12
  • RCD blocks can replace convolutional blocks at Res1, Res2, or Res3, and max-pooling coarsens the range image used to condition dilation.The stated intent of the coarse downsampling is to favor distant objects, which occupy fewer pixels.Source: §A.2 · p. 12
  • The RPN backbone uses ResNet and aggregation blocks with specified bottleneck counts, downsampling strides, and output channels across its stages.Figure 5 gives counts of 5, 7, 9, and 4 bottlenecks in the Res1, Res2/Res2a, Res3/Res3a, and aggregation blocks, respectively; the output channels are 64, 128, and 256 by block number.Source: Figure 5 · p. 12
  • Adding soft range gating to the RCD RPN improves AP and APH beyond using RCD alone.Table 5 compares the first-stage RPN with standard convolution at fixed dilation rates and reports measurements on the WOD validation set after 350k training iterations.Source: Table 5 · p. 13

B Learnable Sampling Pattern Analysis

The learned sampling pattern concentrates samples near the center, while the nominal width grows during training. Across the fixed-dilation comparisons, RCD gives the best overall performance, whereas larger fixed rates and ASPP have drawbacks.

B Learnable Sampling Pattern Analysis

  • Training shifts the learned kernel samples toward greater concentration near the center, with outer samples spreading little or slightly outward.The authors describe this pattern as similar to the fovea in the human eye.Source: B Learnable Sampling Pattern Analysis · p. 13
  • The learnable nominal width starts at 1 m, generally increases during training, and settles between 2 and 3 m, comparable to average vehicle dimensions.The nominal width is initialized to 1 m in all experiments.Source: B Learnable Sampling Pattern Analysis · p. 13
  • Adding a soft range gate improves training stability, speeds convergence, and slightly improves performance.The comparison includes a standard 7 × 7 convolution with 64 channels to match the RCD output.Source: B Learnable Sampling Pattern Analysis · p. 13

C Comparison to Fixed Dilation Rates

  • RCD models with dilation adjusted for near and far ranges give the best overall performance in the fixed-dilation comparison.Rates 3 and 5 slightly improve on standard convolution, while rate 7 drops sharply; ASPP with various dilations significantly underperforms standard dilated convolutions.Source: C Comparison to Fixed Dilation Rates · p. 14
  • Fixed dilation rates of 3 and 5 slightly improve over rate 1, but rate 7 sharply reduces performance.The proposed possible explanations are insufficient sampling of small distant objects or increased padding given the range image's 64-pixel height.Source: C Comparison to Fixed Dilation Rates · p. 14
  • ASPP with fixed large dilation rates underperforms standard dilated convolutions, and its parameter search shows low, sporadic RPN performance and early peaks suggesting overfitting and unstable training.The reported ASPP search used 350K training iterations; peak@iteration refers to the checkpoint with the highest second-stage RCNN performance on WOD validation.Source: C Comparison to Fixed Dilation Rates · p. 14, Table 6 · p. 14

D Qualitative Results

Qualitative WOD examples show both an occlusion-shadow proposal removed by RCNN refinement and an RCNN background-classification error. The KITTI comparison is framed as a small-dataset reference, and the appendix describes reconstructing range images from point clouds.

D Qualitative Results

  • On WOD validation sequences, Figure 10 shows an RPN proposal in a vehicle’s occlusion shadow correctly removed by second-stage RCNN refinement, while a failure case shows RCNN mistakenly classifying a proposal as background.Source: D Qualitative Results · p. 14

E Comparison to LaserNet on KITTI

  • KITTI serves as an indication of comparison with LaserNet in the small-dataset regime, where the authors also observe overfitting and poor generalization for range-image methods.The paper notes that KITTI's small size has been shown to cause these issues for range-image methods.Source: E Comparison to LaserNet on KITTI · p. 14

E.1 Range Images from Point Clouds

  • Direct Cartesian-to-spherical conversion produces poor-quality KITTI range images because its 64 Velodyne lasers do not share a common origin, so the authors reconstruct denser, more compact images.The reconstruction begins by estimating each laser's inclination and height through Hough voting.Source: §E.1 · p. 14
  • For KITTI point clouds, the reconstruction estimates each point's laser ID and azimuth, then creates a 64 × 2048 range image and retains the closest point when observations collide at a pixel.The pixel value encodes range, and laser intensity is added as another channel.Source: §E.1 · p. 15

E.2 KITTI Results

The KITTI test comparison uses a fixed training schedule and only the forward-labeled sector of each range image. The section frames the LaserNet comparison as indicative rather than directly comparable, and includes qualitative WOD detections.

E.2 KITTI Results

  • KITTI test performance is reported after 100K iterations with flipping and horizontal pixel shifting, using only the 90° forward-labeled portion of each range image.Source: §E.2 · p. 15
  • The KITTI table compares the method with LaserNet and includes an RCD model pretrained on WOD then fine-tuned on KITTI for 100K iterations.The authors say these numbers are not intended for direct comparison with other published works, but highlight the importance of large and diverse training datasets for range-image detectors.Source: §E.2 · p. 15
  • The WOD validation visualization presents four scenes with RPN outputs, RCNN outputs, and ground-truth boxes in separate columns.Predictions are filtered with a minimum confidence of 0.5.Source: Figure 10 · p. 16

Related papers

Loading 2005.09927v3…