PaperlayerView exact revision on arXiv

Source-linked AI summary

RoarNet: A Robust 3D Object Detection based on RegiOn Approximation Refinement

Kiwoo Shin, Youngwook Paul Kwon, Masayoshi Tomizuka

arXiv:1811.03818v1cs.CV

TL;DR

3D object detection from images and Lidar remains challenging, especially when camera and Lidar sensors are not synchronized. RoarNet combines monocular geometric proposal generation with recursive point-cloud refinement, and it outperforms publicly available state-of-the-art methods on KITTI, including under asynchrony.

  • Problem

    3D object detection from 2D images and 3D point clouds is less explored, while synchronization failures between camera and Lidar can significantly degrade performance.

  • Method

    RoarNet uses RoarNet 2D to generate multiple geometrically feasible 3D candidates, then RoarNet 3D recursively refines point-cloud regions into final detections.

  • Results

    RoarNet outperforms publicly available state-of-the-art methods on KITTI in synchronized and nonsynchronized camera–Lidar settings.

  • Takeaways & Limitations

    At 0.8m sensor asynchrony, RoarNet retains 72.5% recall versus 67.5% for F-PointNet and 65% for AVOD (FPN).

Abstract

from arXiv · show

We present RoarNet, a new approach for 3D object detection from a 2D image and 3D Lidar point clouds. Based on two-stage object detection framework with PointNet as our backbone network, we suggest several novel ideas to improve 3D object detection performance. The first part of our method, RoarNet_2D, estimates the 3D poses of objects from a monocular image, which approximates where to examine further, and derives multiple candidates that are geometrically feasible. This step significantly narrows down feasible 3D regions, which otherwise requires demanding processing of 3D point clouds in a huge search space. Then the second part, RoarNet_3D, takes the candidate regions and conducts in-depth inferences to conclude final poses in a recursive manner. Inspired by PointNet, RoarNet_3D processes 3D point clouds directly without any loss of data, leading to precise detection. We evaluate our method in KITTI, a 3D object detection benchmark. Our result shows that RoarNet has superior performance to state-of-the-art methods that are publicly available. Remarkably, RoarNet also outperforms state-of-the-art methods even in settings where Lidar and camera are not time synchronized, which is practically important for actual driving environments. RoarNet is implemented in Tensorflow and publicly available with pre-trained models.

I. INTRODUCTION

RoarNet addresses 3D object detection from 2D images and 3D point clouds, emphasizing robustness to camera–Lidar synchronization problems. Its two-stage design uses monocular image estimates to narrow 3D search regions before recursively refining point-cloud-based detections.

  • 3D object detection predicts object bounding boxes from 2D images and 3D point clouds, supporting geometric understanding for mobile robots and autonomous vehicles.
  • Sensor-fusion systems can lose information when projecting point clouds into lower-dimensional representations and require high-quality camera–Lidar synchronization.Performance degrades significantly when synchronization breaks down.
  • RoarNet combines RoarNet 2D and RoarNet 3D to improve detection performance and reduce problems caused by sensor synchronization.
  • RoarNet 2D estimates monocular 3D poses and generates multiple geometrically feasible candidates, narrowing the regions examined by RoarNet 3D.
  • RoarNet 3D samples point clouds from candidate regions, predicts objectness and relative locations recursively, and then regresses final 3D bounding boxes.The final regression includes location, rotation, and size.
  • On KITTI, RoarNet outperforms publicly available state-of-the-art methods in synchronized and nonsynchronized camera–Lidar settings.The implementation is publicly available with pretrained models.

II. RELATED WORK

Prior 3D detection methods use voxel, image-like, or sensor-fusion representations for point clouds, while RoarNet targets strong KITTI performance and broader synchronization robustness.

  • Existing point-cloud methods commonly use 3D voxel grids or other representations to process unstructured Lidar data.The related-work passage introduces three popular point-cloud representations, though the supplied excerpt is incomplete.
  • Image-and-point-cloud sensor-fusion methods such as F-PointNet and AVOD achieve state-of-the-art performance on the KITTI leaderboard.
  • RoarNet is reported to outperform these methods on standard KITTI 3D detection and to provide better robustness in a more general setting.

III. DESIGNING A ROARNET DETECTOR

RoarNet is a sequential two-stage detector that progressively reduces the 3D search space, assigning region proposal, objectness, localization, and box-regression tasks to successive steps.

  • RoarNet sequentially refines the search space so each network performs a simpler task, improving training and prediction efficiency.
  • From image detections and monocular pose estimates, geometric agreement search generates multiple standing-cylinder region proposals in 3D space.Multiple proposals account for prediction error in bounding boxes and poses.
  • RoarNet 3D processes points sampled from each proposal to predict relative object location and objectness, retaining high-objectness proposals for the next step.
  • The final stage predicts 3D bounding-box location, rotation, and size, and repeating this stage more than once improves detection performance.
  • RoarNet 2D supplies 3D region proposals, while RoarNet 3D predicts bounding boxes from point clouds.

P ROJ)

RoarNet 2D converts monocular image detections into scattered, geometrically feasible 3D proposals, while a unified network improves the efficiency of this initialization.

  • RoarNet 2D architecture: A limitation of the prior monocular method is that pose inference and inverse projection require separate computation for each detected vehicle.For k objects, the excerpt states that the network computation is repeated k times.
  • RoarNet 2D architecture: RoarNet 2D unifies 2D detection with width, height, length, and heading inference in one forward calculation for k objects.
  • Spatial scattering: Because monocular pose estimation has limited accuracy, spatial scattering creates multiple candidates between extreme size-based poses to increase recall.The extreme cases use physical sizes 1 − s and 1 + s times the regressed dimensions.
  • Spatial scattering: Geometric constraints and spatial scattering narrow feasible 3D regions into a few linear regions distributed along camera projection rays and more broadly for distant objects.

B. RoarNet 3D

RoarNet 3D processes point clouds with a simplified PointNet-inspired backbone and uses separate proposal and box-regression networks to refine 3D detections.

  • Architecture: RoarNet 3D predicts 3D bounding boxes from point clouds using a simplified PointNet-inspired backbone with max-pooling for global features.The backbone omits PointNet’s T-Net.
  • Architecture: RoarNet 3D contains an RPN and BRN with the same structure but different output sizes.
  • Prediction outputs: Location predictions use three coordinates relative to each proposal center and are bounded by specified maximum offsets.
  • Prediction outputs: Rotation uses a hybrid classification-regression output over equally divided bins spanning [0, pi).
  • Prediction outputs: Size prediction uses 4*NC coordinates in a hybrid classification-regression formulation, with NC clusters obtained by K-Means.
  • Prediction outputs: Objectness predicts the probability that each region proposal contains an object and bounds it with a sigmoid in [0.0, 1.0).

C. Training and prediction

RoarNet trains its networks with multi-task classification and regression losses, samples point clouds from proposals, and applies non-maximal suppression at test time.

  • Training: RPN and BRN training optimizes multi-task losses for objectness, location, rotation classification and regression, and size classification and regression.
  • Training: Classification losses use cross-entropy, while location, rotation-regression, and size-regression losses use Huber loss.The objectness indicator activates location loss for positive proposals.
  • Training and prediction: Point clouds are down-sampled at 0.1m resolution, with 256 points sampled per proposal for training and 512 for prediction.
  • Training: Each network is trained with batch size 512 for 500k iterations using learning rates of 5e-3 initially and 5e-4 thereafter.
  • Prediction: Testing applies bird’s-eye-view non-maximal suppression with threshold 0.05 to remove overlapping predictions.

IV. EXPERIMENTS

Experiments use KITTI car detection under the moderate difficulty setting, while the supplied passages identify the dataset and evaluation setup but provide no Table II scores.

  • Dataset: Experiments use KITTI, a 3D detection benchmark providing synchronized images, LiDAR point clouds, and annotations for cars, pedestrians, and cyclists.The study focuses on cars because that class has the most training examples.
  • Evaluation: Evaluation focuses on the moderate difficulty level, a standard performance setting, with 3D detection evaluated at a 0.7 IoU threshold.
  • Evaluation: Table II reports publicly available KITTI test-set 3D detection performance at a 0.7 3D IoU threshold.The supplied table passage contains the caption but no performance values.
  • Evaluation: The training set is split into 3,717 training frames and 3,769 validation frames, with splits drawn from different video clips.

A. Comparison of the 3D object detection performance

RoarNet is evaluated on KITTI in synchronized and deliberately asynchronized sensor settings. It achieves state-of-the-art performance in synchronized evaluation and retains higher recall than AVOD (FPN) and F-PointNet under sensor translation.

  • Experiment settings: RoarNet is evaluated with synchronized sensors and with simulated sensor asynchrony created by translating point clouds and regenerating ground-truth labels.Translations are constrained to 0.8m along x and y and 0.2m along z.
  • Synchronized evaluation: RoarNet shows state-of-the-art 3D object detection performance on KITTI at both easy and moderate evaluation levels.The synchronized comparison uses publicly available 3D object detection methods.
  • Asynchronized evaluation: The asynchronized comparison selects AVOD (FPN) and F-PointNet (v1) as state-of-the-art representatives of sensor-fusion and direct-point-cloud methods.The passage identifies AVOD (FPN) as the best sensor-fusion method and F-PointNet (v1) as the best direct-point-cloud method.
  • Asynchronized evaluation: At 0.8m sensor asynchrony, RoarNet reaches 72.5% recall, versus 67.5% for F-PointNet and 65% for AVOD (FPN).All three methods have 82.5% recall when sensors are synchronized.

B. Region proposals analysis

RoarNet trades proposal coverage against proposal count through spatial scattering, then refines the search space using an objectness threshold. The selected settings prioritize recall while limiting proposals per ground-truth object.

  • Spatial scattering parameter: With s = 0, only 26.3% of objects are captured in region proposals, despite higher confidence in monocular pose estimation.Smaller s indicates higher confidence in the monocular pose estimate.
  • Spatial scattering parameter: Increasing s captures more objects but linearly increases the number of region proposals, creating a detection-pipeline bottleneck.The implementation uses s = 0.5 to aim for high recall.
  • Objectness threshold: An objectness threshold of 0.25 yields 83.2% recall with fewer than two region proposals per ground-truth object.RoarNet 3D further refines the search space at this stage.

C. Network design analysis

The network-design analysis compares a single-stage detector with recursive refinement and a task-specialized final design. Recursive refinement improves performance, while specialization produces the strongest reported recall and mAP.

  • Architectural comparison: The comparison evaluates three network architectural designs, with all methods trained and evaluated on the same train/validation split for the car class.The architectures are shown in Figure 8.
  • Single-stage design: The single-stage detector in Figure 8(a) predicts the 3D bounding box and objectness in one step, achieving 67.5% recall and 54.3% mAP.This design is inspired by YOLO detectors.
  • Recursive refinement: Using each step’s predicted location as the next step’s region proposals raises performance to 59.9%, an increase of 5.6% from the single-stage design.The modification requires no further training step.
  • Task-specialized design: The final RoarNet 3D design specializes each detection step and removes redundant predictions, reaching 82.5% recall and 74.02% mAP.This design was motivated by the recursive-refinement result.

V. CONCLUSION

RoarNet recursively refines 3D search regions from monocular pose estimates and Lidar point clouds. It reports superior KITTI performance, including when camera and Lidar are not time synchronized.

  • Conclusion: RoarNet estimates 3D poses from a monocular image and generates multiple nearby geometrically feasible candidates.These candidates initialize the subsequent 3D refinement process.
  • Conclusion: A two-stage detector recursively refines the search space from 3D point clouds to make training and prediction efficient.The method combines monocular initialization with further 3D point-cloud refinement.
  • Conclusion: RoarNet shows superior performance to state-of-the-art methods on KITTI, including when Lidar and camera are not time synchronized.The paper identifies this setting as practically important for future video-frame-based detection research.
Loading 1811.03818v1…