PaperlayerView exact revision on arXiv

Source-linked AI summary

Joint 3D Proposal Generation and Object Detection from View Aggregation

Jason Ku, Melissa Mozifian, Jungwook Lee, Ali Harakeh, Steven Waslander

arXiv:1712.02294v4cs.CV

TL;DR

AVOD tackles the difficulty of 3D detection from sparse, distance-degraded data by combining LIDAR and RGB features in a high-resolution multimodal architecture. It generates high-recall proposals and refines them into oriented 3D detections, achieving state-of-the-art KITTI performance while running in real time with low memory overhead.

  • Problem

    3D detection remains substantially behind 2D detection because it requires depth and oriented-box estimation from sparse, distance-degraded input.

  • Method

    AVOD extracts high-resolution features from LIDAR and RGB inputs, fuses them in an RPN, then predicts box dimensions, orientation, and categories in a second stage.

  • Results

    AVOD achieves state-of-the-art KITTI detection performance across localization, orientation estimation, and classification while running in real time with low memory overhead.

  • Takeaways & Limitations

    The architecture is presented as a suitable candidate for autonomous-vehicle deployment and generalizes to new scenes and more extreme weather and lighting conditions.

Abstract

from arXiv · show

We present AVOD, an Aggregate View Object Detection network for autonomous driving scenarios. The proposed neural network architecture uses LIDAR point clouds and RGB images to generate features that are shared by two subnetworks: a region proposal network (RPN) and a second stage detector network. The proposed RPN uses a novel architecture capable of performing multimodal feature fusion on high resolution feature maps to generate reliable 3D object proposals for multiple object classes in road scenes. Using these proposals, the second stage detection network performs accurate oriented 3D bounding box regression and category classification to predict the extents, orientation, and classification of objects in 3D space. Our proposed architecture is shown to produce state of the art results on the KITTI 3D object detection benchmark while running in real time with a low memory footprint, making it a suitable candidate for deployment on autonomous vehicles. Code is at: https://github.com/kujason/avod

I. INTRODUCTION

AVOD addresses the persistent gap between 2D and 3D detection by combining high-resolution multimodal features with proposal and detection networks tailored to road scenes.

  • Motivation: Missed instances during proposal generation cannot be recovered later, making high proposal recall essential.Later stages can apply more complex processing to a smaller search space, but only to retained proposals.
  • Contributions: AVOD introduces high-resolution feature maps from LIDAR point clouds and RGB images to localize small object classes.The feature extractor is inspired by feature pyramid networks and is designed for small classes in the scene.
  • Contributions: Its multimodal fusion RPN uses these features to produce high-recall proposals for small classes.The architecture diagram separates feature extractors, the RPN, and the second-stage detector.
  • Contributions: 1 × 1 RPN convolutions and a fixed lookup table of 3D anchor projections support high speed and low memory usage while maintaining detection performance.The authors also report real-time operation and integration into an autonomous-driving stack.

II. RELATED WORK

Prior 3D detectors generate proposals from handcrafted features, single-shot networks, or monocular cues, while AVOD builds on 3D RPNs with multimodal high-resolution fusion.

  • Hand Crafted Features For Proposal Generation: Handcrafted approaches score 3D sliding windows using geometric features, then pass the top proposals to a modified Fast-RCNN.3DOP and Mono3D are examples using stereo point clouds and monocular images.
  • Proposal Free Single Shot Detectors: Proposal-free methods directly predict dense 3D boxes from projected point clouds or voxel grids, but AVOD uses an RPN-based two-stage design.The cited methods include VeloFCN, 3D-FCN, and VoxelNet.
  • Proposal Free Single Shot Detectors: VoxelNet is reported as 3× slower than AVOD, which provides better results on the car and pedestrian classes.This comparison concerns computational speed and reported detection results.
  • Monocular-Based Detectors: Monocular-based approaches use 2D detections or prior 3D knowledge, but the cited text reports poorer 3D detection than point-cloud methods.Deep MANTA additionally requires a database of 3D vehicle models, limiting generalization to classes without such models.
  • 3D Region Proposal Networks: MV3D extends Faster R-CNN to 3D with BEV anchors and multiview feature fusion, but its downsampled BEV maps can undersupply information for small objects.AVOD instead fuses full-resolution image and BEV feature crops in its RPN to target higher-recall proposals for smaller classes.

III. THE AVOD ARCHITECTURE

AVOD extracts features from BEV and RGB views, uses them to generate non-oriented proposals, and refines each proposal through detection and classification heads.

  • Architecture: Feature extractors generate feature maps from both the BEV map and the RGB image.These two views provide the inputs to the proposal stage.
  • Architecture: The RPN generates non-oriented region proposals from both feature maps.The proposals reduce the search space before second-stage processing.
  • Architecture: The detection network refines dimensions, estimates orientation, and classifies each proposal.The pipeline therefore separates proposal generation from final 3D box and category prediction.

A. Generating Feature Maps from Point Clouds and Images

AVOD represents the LIDAR point cloud as a six-channel BEV map built from a 0.1-meter voxel grid within the camera field of view.

  • BEV Representation: The BEV representation uses a voxel grid with 0.1 meter resolution and crops points to [−40, 40] × [0, 70] meters.The crop is selected to contain points within the camera’s field of view.
  • BEV Representation: Five BEV channels encode maximum point heights across equal vertical slices, while the sixth stores per-cell point density.The density channel uses a clipped logarithmic function of the number of points in each cell.

B. The Feature Extractor

AVOD uses paired, view-specific feature extractors that encode and then decode RGB and BEV inputs to recover high-resolution feature maps. The decoder fuses multiscale features while restoring the input resolution.

  • Architecture: Two identical feature extractors process the two input views, with each comprising an encoder and a decoder.The encoder is modeled after VGG-16, with fewer channels and truncation at conv-4.
  • Encoder: Each extractor accepts an M × N × D image or BEV map and produces a feature map with reduced spatial resolution.The resulting representation has high representational power but is 8× lower in resolution than the input.
  • Motivation: An 8× reduction can make small pedestrians occupy less than one output pixel in the BEV feature map.A typical pedestrian spans 8×6 pixels at 0.1-meter BEV resolution before downsampling.
  • Decoder: Inspired by FPNs, the bottom-up decoder learns to upsample encoder features so small objects remain localized in high-resolution maps.The decoder restores spatial detail through multistage feature fusion.

C. Multimodal Fusion Region Proposal Network

The multimodal RPN uses 3D anchors to sample aligned crops from BEV and image feature maps, fuses the views, and predicts object proposals. Learned dimensionality reduction makes this high-volume processing practical in memory and computation.

  • Anchor generation: 3D anchors are arranged on a 0.5-meter BEV grid, with height and dimensions determined from the sensor and training data.Anchors use centroid and axis-aligned dimensions as their parameterization.
  • Multiview crops: Each 3D anchor is projected into the BEV and image feature maps, whose regions are cropped and bilinearly resized to 3 × 3 feature vectors.The operation produces equal-length crops from both views for multimodal fusion.
  • Dimensionality reduction: A learned 1 × 1 convolution reduces feature dimensionality before crops are extracted for large numbers of anchors.This reduces crop memory overhead by approximately ˜D× and enables processing tens of thousands of anchors with only a few megabytes of additional memory.
  • Proposal prediction: The RPN fuses the two view-specific crops by element-wise mean and predicts axis-aligned proposal boxes together with objectness scores.Separate fully connected branches perform box regression and object/background classification.

D. Second Stage Detection Network

The second-stage detector refines projected proposal crops from both views and jointly predicts box geometry, orientation, and class. AVOD uses a compact four-corner encoding with explicit orientation regression to represent oriented 3D boxes.

  • D. Second Stage Detection Network: AVOD encodes each 3D box with four corners and two height values instead of an unconstrained eight-corner representation.The design preserves physical box constraints because top corners need not be independently regressed from the bottom corners.
  • D. Second Stage Detection Network: The high-resolution feature extractor architecture uses encoder-to-decoder propagation with learned upsampling, concatenation, and convolutional mixing.The final decoder layer produces a full-resolution feature map.
  • D. Second Stage Detection Network: The proposed encoding reduces the box representation from a 24-dimensional vector to a 10-dimensional one.Corner offsets are matched between proposal and ground-truth boxes using the closest BEV corners.
  • D. Second Stage Detection Network: Explicit orientation-vector regression addresses failures of inferring orientation solely from box extents, including ambiguity up to ±π radians.It also avoids assuming that an object's orientation always follows the longer box side.
  • D. Second Stage Detection Network: The detector projects proposals into both input views, resizes their crops to 7×7, fuses them by element-wise mean, and predicts regression, orientation, and category outputs.Three fully connected layers process each fused proposal crop, with multitask losses for geometry, orientation, and classification.

E. Training

AVOD jointly trains its RPN and detection networks end to end using separate class configurations and mini-batches containing sampled regions of interest. Optimization uses ADAM with a decayed learning rate over 120K iterations.

  • E. Training: Two networks are trained: one for cars and one jointly for pedestrians and cyclists.The RPN and detection networks are optimized together in an end-to-end fashion.
  • E. Training: Training uses mini-batches containing one image with 512 and 1024 ROIs, respectively.The passage reports these ROI counts for the jointly trained proposal and detection stages.
  • E. Training: The model is trained for 120K iterations with ADAM, starting at a 0.0001 learning rate.The learning rate decays exponentially every 30K iterations by a factor of 0.8.

IV. EXPERIMENTS AND RESULTS

AVOD is evaluated on KITTI for 3D proposal recall, 3D detection, qualitative outputs, and computational efficiency. The architecture achieves strong detection and proposal performance while operating with low runtime and memory requirements.

  • Evaluation Setup: AVOD is evaluated on the three KITTI object classes using training/validation splits and easy, moderate, and hard difficulty levels.The evaluation covers proposal generation and object detection, with separate AVOD variants using VGG-like and Feature Pyramid extractors.
  • 3D Proposal Recall: 86% 3D recall is achieved for cars with 10 proposals per frame, compared with 73.87% for 3DOP and 65.74% for Mono3D.At 1024 proposals, AVOD also exceeds the competing methods by more than 20% recall for pedestrians and cyclists.
  • 3D Object Detection: 2.09% AP and 4.09% AP gains over MV3D are reported for cars at moderate and hard difficulty, respectively.AVOD also improves AHS over MV3D by 30.36% at moderate difficulty and 28.42% at hard difficulty.
  • 3D Object Detection: The Feature Pyramid version outperforms all compared methods on KITTI car 3D detection, including a 4.19% margin over F-PointNet for hard instances.The encoder-only version is reported as twice as fast as the next fastest method, F-PointNet.
  • Runtime and Memory Requirements: The final Feature Pyramid fusion architecture uses 38.073 million parameters, 231.263 billion FLOPs per frame, and processes frames in 0.1 seconds on a TITAN Xp GPU.The reported timing includes 20 ms preprocessing and 80 ms inference, and is 1.7× faster than F-PointNet.
  • Runtime and Memory Requirements: AVOD requires only 2 gigabytes of GPU memory at inference time, supporting its proposed use in autonomous-vehicle deployment.Qualitative results visualize proposal outputs, final 3D detections, and image-space projections for cars, pedestrians, and cyclists.

A. Ablation Studies:

The ablations examine proposal inputs, box encoding, and feature extraction. Multimodal and high-resolution features particularly improve proposal recall and detection AP for smaller object classes.

  • Ablation Protocol: Table III evaluates hyperparameter variants at moderate difficulty using AP, AHS, parameter count, and FLOP count.Car evaluation uses a 0.7 3D IoU threshold, while pedestrian and cyclist evaluation uses 0.5.
  • RPN Input Variations: For cars, image features and the high-resolution extractor do not improve proposal recall over BEV-only input.The paper attributes this to cars occupying larger regions of the BEV map, which provide sufficient low-resolution features for proposal generation.
  • RPN Input Variations: Fusing image and BEV features at the RPN stage increases pedestrian and cyclist AP by 6.9% and 9.4%, respectively, over BEV-only features.At 1024 proposals, the corresponding recall increases are 10.1% and 8.6%, rising to about 10.5% and 10.8% with the high-resolution extractor.
  • Bounding Box Encoding: The study compares axis-aligned, four-corner, and eight-corner bounding-box encodings to assess their effects on detection performance.The compared encodings differ in whether orientation is separately estimated or inferred from the regressed orientation vector.
  • Feature Extractor: The high-resolution Feature Pyramid extractor increases AP by 19.3% for pedestrians and 8.1% for cyclists, but only 0.3% for cars.The gains are reported relative to the VGG-based feature extractor from MV3D and require only a minor increase in computational requirements.

V. CONCLUSION

AVOD combines high-resolution multimodal feature extraction, fusion-based proposals, and explicit orientation regression for 3D autonomous-driving detection. On KITTI, it outperforms prior methods across localization, orientation estimation, and classification while running in real time with low memory overhead.

  • AVOD uses a high-resolution feature extractor with multimodal fusion RPN architecture to produce accurate region proposals for small classes in road scenes.The architecture processes LIDAR point clouds and RGB images and explicitly regresses orientation vectors to resolve bounding-box orientation ambiguity.
  • AVOD outperforms the state of the art on KITTI 3D localization, orientation estimation, and category classification tasks.
  • AVOD runs in real time with low memory overhead.
Loading 1712.02294v4…