PaperlayerView exact revision on arXiv

Source-linked AI summary

AutoShape: Real-Time Shape-Aware Monocular 3D Object Detection

Zongdai Liu, Dingfu Zhou, Feixiang Lu, Jin Fang, Liangjun Zhang

arXiv:2108.11127v1cs.CV

TL;DR

Monocular 3D detection often ignores object shape while facing difficult depth estimation and computational trade-offs. AutoShape learns shape-aware 2D/3D keypoint correspondences, automatically fits deformable models for annotations, and incorporates geometric constraints into detection. On KITTI, it reports state-of-the-art accuracy with real-time performance, including 25 fps.

  • Problem

    Monocular 3D detection commonly models objects as rotated cuboids while ignoring geometric shape, and single-image depth estimation remains challenging.

  • Method

    AutoShape learns meaningful 2D image keypoints and corresponding local 3D coordinates, uses their geometric constraints for detection, and automatically fits deformable 3D models to generate annotations.

  • Results

    The framework achieves state-of-the-art detection accuracy with real-time performance on KITTI, reported at 25 fps.

  • Takeaways & Limitations

    Shape-aware 2D/3D constraints provide a real-time monocular 3D detection framework that the authors report as suitable for autonomous-driving perception.

  • Takeaways & Limitations

    The model cannot generate accurate keypoint annotations for Pedestrian and Cyclist because CAD models are unavailable.

Abstract

from arXiv · show

Existing deep learning-based approaches for monocular 3D object detection in autonomous driving often model the object as a rotated 3D cuboid while the object's geometric shape has been ignored. In this work, we propose an approach for incorporating the shape-aware 2D/3D constraints into the 3D detection framework. Specifically, we employ the deep neural network to learn distinguished 2D keypoints in the 2D image domain and regress their corresponding 3D coordinates in the local 3D object coordinate first. Then the 2D/3D geometric constraints are built by these correspondences for each object to boost the detection performance. For generating the ground truth of 2D/3D keypoints, an automatic model-fitting approach has been proposed by fitting the deformed 3D object model and the object mask in the 2D image. The proposed framework has been verified on the public KITTI dataset and the experimental results demonstrate that by using additional geometrical constraints the detection performance has been significantly improved as compared to the baseline method. More importantly, the proposed framework achieves state-of-the-art performance with real time. Data and code will be available at https://github.com/zongdai/AutoShape

1. Introduction

Monocular 3D detection is attractive for autonomous driving but must address depth ambiguity and the loss of detailed object shape in compact representations. AutoShape learns meaningful shape-aware 2D/3D keypoints, generates their annotations automatically, and uses them as geometric constraints for real-time detection.

  • Motivation: Monocular cameras avoid LiDAR cost and stereo calibration issues, but accurate depth estimation from a single image remains difficult.Depth-map and pseudo-LiDAR pipelines also impose heavy computation burdens.
  • Motivation: 20∼30 fps direct-regression methods are efficient, yet center-point representations ignore object shape and become ambiguous under occlusion.A projected center may fall on another object’s surface, making localization unreliable.
  • Approach: AutoShape learns meaningful surface keypoints and uses their 2D/3D correspondences with object attributes to solve the 3D bounding box through linear equations.The framework is trained end-to-end.
  • Approach: The method automatically fits a deformable 3D shape to visual observations to generate ground-truth 2D/3D keypoint pairs.The authors state that the source code and dataset will be made public.
  • Results: 25 fps real-time performance and state-of-the-art KITTI results are reported for the proposed framework.The framework is described as suitable for integration into an autonomous-driving perception module.

2. Related Work

Image-based 3D detection methods commonly use depth maps, direct regression, or CAD models. CAD-guided approaches introduce shape information through keypoints, dense matching, and automated shape annotation, but annotation quality and required supervision remain concerns.

  • Overview: Image-based 3D object detection is motivated by inexpensive cameras, while stereo methods can suffer from calibration issues.The related work categorizes approaches into depth-map-based, direct regression-based, and CAD model-based methods.
  • Depth-map-based methods: Depth-map-based methods estimate depth before transforming it into point clouds or using it directly to enhance 3D detection.The supplied passage describes both point-cloud and direct depth-map uses.
  • CAD model-based methods: CAD keypoint methods predefine 3D model points, predict corresponding image points, and solve object pose with a 2D/3D pose solver.Deep MANTA and ApolloCar3D are cited as examples.
  • Shape annotation: Shape information is often omitted from cuboid representations, while manual shape labeling is difficult and its annotation quality is not guaranteed.CAD-guided annotation methods seek denser shape supervision from additional geometric or rendering-based information.

3. Problem Definition

The problem is to estimate an object’s 3D pose and dimensions from an image while connecting object-coordinate geometry to projected image points. Learning-based detectors often regress cuboid parameters without explicit geometric constraints, motivating their incorporation into the formulation.

  • Problem formulation: A 6D pose is represented by rotation R and translation T mapping points from the object coordinate system to the camera coordinate system.The transformation defines the object pose relative to the camera.
  • Geometric formulation: Object-coordinate 3D points are transformed into camera coordinates and projected to image points through the camera model.The supplied formulation introduces the 3D point, transformed point, and image point relationships.
  • Geometric formulation: Geometric constraints between corresponding 3D object points and 2D image points can theoretically recover object pose.This establishes the geometric basis for using 2D/3D correspondences in detection.
  • Problem formulation: Image-based 3D detection estimates an object’s location, orientation, and dimensions in camera coordinates, commonly representing it as a rotated 3D bounding box.The parameterization uses orientation r, location t, and dimensions d.
  • Learning-based detection: Deep learning detectors commonly regress pose parameters directly without imposing additional geometric constraints.The paper therefore explicitly incorporates geometric constraints into learning-based 3D detection.

4. Proposed Method

The framework uses CAD-derived 2D/3D keypoint correspondences and additional confidence weighting to impose geometric constraints on monocular 3D detection. A multi-branch network regresses keypoints and object attributes, after which the object pose and 3D bounding box are recovered.

  • Point-wise 2D-3D Constraints: The locally flat road assumption reduces object orientation to a single yaw angle around the Y-axis.
  • Network: The network uses eight branches for center detection, offsets, 2D and 3D keypoints, confidence, orientation, dimensions, and 3D detection score.The regressed information is combined to recover the object’s 3D bounding box in camera coordinates.
  • Point-wise 2D-3D Constraints: 2n constraints from n keypoints are weighted by learned confidence scores to reduce the influence of severely occluded keypoints.The weights determine each constraint’s importance during pose calculation.
  • Point-wise 2D-3D Constraints: An automatic annotation pipeline is introduced because manually labeling 2D/3D keypoint ground truth is costly and tedious.The pipeline optimizes 2D and 3D reprojection errors.
  • Network: 2D keypoints are regressed as ordered offsets from the object center, while normalized local 3D keypoints keep regression values in a relatively small range.The network also regresses object dimensions relative to class-specific mean dimensions.
  • Loss Function: The multi-task loss combines center, offset, 2D keypoint, 3D keypoint, orientation, dimension, confidence, and 3D bounding-box IoU losses.The 3D IoU confidence is supervised by the overlap between predicted and ground-truth boxes, and multiplies the classification score for final confidence.

5. 3D Shape Auto-Labeling

The paper automatically fits a deformable vehicle shape and full 6-DoF pose to image masks, 3D boxes, and sparse LiDAR observations. The resulting optimization generates shape-aware 2D/3D keypoint annotations for training.

  • 5.1. Deformable Vehicle Template: A deformable vehicle template represents varied vehicle geometries using a mean shape and PCA principal components.The shape parameters are optimized rather than selecting from a small fixed CAD-model set.
  • 5.2. 3D Shape Optimization: The fitting objective combines 2D instance-mask and 3D point constraints to estimate shape coefficients and the object pose.The 3D annotation computes PCA coefficients and a 6-DoF pose from visual observations, including the mask, 3D bounding box, and LiDAR points.
  • 5.2. 3D Shape Optimization: The optimization iteratively updates shape and pose by minimizing the 2D and 3D losses with gradient descent.The pipeline initializes translation and yaw from the vehicle annotations, initializes pitch, roll, and PCA coefficients at zero, then back-propagates the losses.
  • 5.2. 3D Shape Optimization: During optimization, the rendered mask increasingly covers the target mask while model vertices align with the point cloud and the shape changes from mean shape to notchback.This progression is illustrated from step 0 through step 200.
  • 5.2. 3D Shape Optimization: Including pitch and roll improves shape annotation when vehicles are observed on non-flat roads.Using yaw alone can produce serious model misalignment in such settings.

6. Experimental Results

Experiments on KITTI evaluate AutoShape’s automatic shape labeling, shape-aware detection, qualitative predictions, and keypoint-count trade-offs. The method improves detection over baseline geometric constraints while retaining real-time inference.

  • The automatic labeling pipeline generates 2D keypoints and local 3D locations supervised during training and verifies them using 2D instance-segmentation and 3D-box mean AP.
  • 25 FPS is achieved with the 16-keypoint configuration on an NVIDIA V100 GPU.
  • 48 keypoints achieve four first-place results in six KITTI test tasks under AP|R40.
  • +5.74%, +2.72%, 1.44%, +7.22%, +3.88%, and +1.12% improvements are reported over baseline geometric-constraint methods across AP3D and APBEV difficulty categories.
  • 6.5. Ablation Studies: Performance improves significantly from 0 to 16 extra keypoints, while denser configurations increase overlap and GPU memory, making 16 a practical compromise.
  • 6.5. Ablation Studies: Using both 2D mask loss and 3D point loss improves labeling accuracy to 86.35 and 76.92, producing better 3D detection performance.

7. Conclusion

The paper presents a real-time monocular 3D detector that uses learned shape-aware 2D/3D keypoint constraints and an automatic annotation pipeline. Experiments report state-of-the-art detection accuracy with real-time performance.

  • The framework explicitly uses geometric constraints between 3D keypoints and their 2D image projections for monocular 3D detection.
  • An automatic annotation pipeline generates shape-aware 2D/3D keypoint correspondences for each object.
  • Experiments show state-of-the-art detection accuracy with real-time performance.

8. Supplemental Material

Supplemental experiments examine confidence-weighted keypoint regression and multi-class detection. They show benefits from confidence integration while identifying CAD-model availability and object rigidity as scope boundaries.

  • 8.1. Ablation Study for Keypoints Confidence Regression: Keypoint confidence weighting improves 3D detection performance independently of the number of keypoints.
  • 8.1. Ablation Study for Keypoints Confidence Regression: High-confidence keypoints usually correspond to ground-contact points and distinguished shape-border points that contribute more to pose estimation.
  • 8.2. Multi-classes Detection: AutoShape cannot generate keypoint annotations for pedestrians and cyclists because CAD models are unavailable.
  • 8.2. Multi-classes Detection: Cyclists perform better than pedestrians because cyclists are more rigid, whereas pedestrian keypoint locations vary with pose.
Loading 2108.11127v1…