PaperlayerView exact revision on arXiv

Source-linked AI summary

CubeSLAM: Monocular 3D Object SLAM

Shichao Yang, Sebastian Scherer

arXiv:1806.00557v2cs.ROcs.CV

TL;DR

CubeSLAM addresses monocular 3D object detection and SLAM without prior object models, in both static and dynamic environments. It generates cuboids from 2D boxes and vanishing points, then couples cameras, objects, and points in bundle adjustment. Across the reported evaluations, the method improves object detection and camera-pose estimation, with a relative depth error of 0.8% on sequence 56.

  • Problem

    Monocular methods often separate 3D object detection from SLAM and rely on prior object models, limiting general-environment applications.

  • Method

    The method generates cuboid proposals from 2D boxes and vanishing points, scores them with image cues, and jointly optimizes cameras, objects, and points through bundle adjustment.

  • Results

    The method achieves the best reported 3D object detection accuracy on SUN RGBD subset data and camera pose estimation on KITTI odometry, with 0.8% relative depth error on sequence 56.

  • Takeaways & Limitations

    Object detection and geometric SLAM benefit each other within one unified framework, including through object-based geometric and scale constraints for camera pose estimation.

Abstract

from arXiv · show

We present a method for single image 3D cuboid object detection and multi-view object SLAM in both static and dynamic environments, and demonstrate that the two parts can improve each other. Firstly for single image object detection, we generate high-quality cuboid proposals from 2D bounding boxes and vanishing points sampling. The proposals are further scored and selected based on the alignment with image edges. Secondly, multi-view bundle adjustment with new object measurements is proposed to jointly optimize poses of cameras, objects and points. Objects can provide long-range geometric and scale constraints to improve camera pose estimation and reduce monocular drift. Instead of treating dynamic regions as outliers, we utilize object representation and motion model constraints to improve the camera pose estimation. The 3D detection experiments on SUN RGBD and KITTI show better accuracy and robustness over existing approaches. On the public TUM, KITTI odometry and our own collected datasets, our SLAM method achieves the state-of-the-art monocular camera pose estimation and at the same time, improves the 3D object detection accuracy.

I. INTRODUCTION

CubeSLAM unifies monocular 3D cuboid detection with object-level SLAM without requiring prior object models, covering static and dynamic environments. Its coupled formulation lets object detection and SLAM provide complementary geometric information.

  • Motivation: Monocular 3D object detection and SLAM are important for autonomous driving and augmented reality, but existing monocular methods commonly treat them separately and depend on prior object models.Monocular cameras are attractive for their low cost and small size, while prior models limit application to general environments.
  • Coupled object-SLAM framework: Object measurements provide geometric and scale constraints for camera pose estimation, while SLAM poses initialize and refine single-view 3D object detection.Objects can also initialize depths for points that are difficult to triangulate.
  • Dynamic environments: Dynamic regions are modeled rather than discarded, with camera and object trajectories jointly optimized using dynamic point observations and motion-model constraints.This extends the system to environments containing moving vehicles and pedestrians, where static-scene assumptions may not hold.
  • Reported contributions: The paper reports improved 3D object detection, object-level SLAM pose estimation, and mutual benefits between detection and SLAM across indoor and outdoor datasets.The introduction identifies better pose estimation on datasets including KITTI and results showing that object detection and SLAM benefit each other.

C. Dynamic environment SLAM

For dynamic environments, CubeSLAM jointly estimates camera poses and moving-object trajectories instead of treating dynamic features as outliers. Its object representation and motion assumptions support coupled optimization, while cuboid geometry is parameterized through vanishing points for general objects.

  • C. Dynamic environment SLAM: Most dynamic-environment SLAM methods reject features from moving regions and rely on static backgrounds for camera pose estimation.Some later systems optimize dynamic-object trajectories after solving static SLAM, but dynamic objects are not generally integrated into the initial pose problem.
  • 1) Principles:: The method represents general 3D objects as cuboids with position, rotation, and dimensions, then uses vanishing points to reduce regression parameters rather than relying on predicted dimensions.The cuboid has 9 DoF: 3 for position, 3 for rotation, and 3 for dimensions.
  • 1) Principles:: Cuboid corners are derived from projected vanishing points and observed image geometry across common configurations, enabling efficient proposal construction.The corner-construction procedure uses intersections of lines defined by vanishing points and known or estimated image corners.

3) Get 3D box pose from 2D corners:

The method recovers 3D cuboid pose from projected corners using PnP for arbitrary objects and direct ground-plane back-projection for ground objects. It samples rotations and scores cuboid proposals using image-edge alignment and shape constraints.

  • Arbitrary pose objects: Arbitrary-pose cuboids use four adjacent projected corners with PnP to estimate pose and dimensions up to monocular scale.Each corner supplies two constraints, so four corners constrain the 9-DoF pose except scale.
  • Ground objects: Ground objects simplify pose recovery by assuming zero roll and pitch, back-projecting ground corners, and analytically computing vertical corners.The camera height determines scale in this projection process.
  • Cuboid proposal generation: Three vanishing points and one top 2D corner determine the remaining 2D cuboid corners through line intersections before 3D box computation.This converts box estimation into estimating vanishing points and one corner.
  • Proposal sampling: The method manually samples rotation parameters and ranks proposals rather than directly predicting vanishing points with a deep network.For ground objects, known or SLAM-estimated camera roll and pitch reduce the sampling space; experiments consider ground objects.
  • Proposal scoring: Proposal costs align cuboids with image edges while penalizing implausible skew, and the selected proposal is the best-scoring candidate.The approach works best for boxy objects with clear edges but also performs decently on bicycles and toilets.

1) Distance error φdist(O, I):

Proposal scoring combines image-edge distance and vanishing-point angle alignment, with a shape term discouraging highly skewed cuboids. The resulting object SLAM backend jointly optimizes cameras, objects, and points.

  • Distance error φdist(O, I): Distance scoring samples visible cuboid edges against a Canny-edge distance transform, normalized by the 2D box diagonal.Ten evenly spaced points are sampled on each visible cuboid edge, similar to Chamfer distance.
  • Angle alignment error φangle(O, I): Angle scoring associates long line segments with vanishing points and measures whether their slopes align with those points.Outermost line segments associated with each vanishing point define the alignment comparison.
  • Shape error φshape(O): A shape cost penalizes cuboids with large length-to-width skew because similar 2D corners can correspond to different 3D cuboids.Stricter priors, including estimated or fixed object dimensions, may also be applied.
  • Object SLAM: The modified bundle adjustment jointly optimizes camera poses, cuboid objects, and feature points using camera-object, camera-point, and object-point measurement errors.The nonlinear least-squares problem can be solved with Gauss-Newton or Levenberg-Marquardt methods.
  • Object representation: Cuboid objects are represented by 6-DoF pose and 3D dimensions, while KITTI can provide dimensions instead of optimizing them.Camera poses use SE(3), and points use R^3.

B. Measurement Errors

Camera-object measurements use either detected 3D cuboids or projected 2D bounding boxes, with robust weighting and optimization alongside points. The 2D formulation is less uncertain but loses 3D information.

  • 3D measurements: The 3D measurement compares a detected cuboid pose and dimensions with the landmark object after transforming it into the camera frame.The resulting SE(3)-based error has nine dimensions, and Huber robust costs are applied to measurement errors.
  • Ambiguity: Without prior object models, image-based cuboid detection cannot distinguish an object’s front from its back.The method tests equivalent height-axis rotations and selects the one with the smallest error.
  • 2D measurements: The 2D measurement projects all eight cuboid corners, forms a bounding rectangle from coordinate minima and maxima, and compares it with the detected box.Its center and size form a four-dimensional rectangle error.
  • Measurement trade-off: 2D box measurements are less uncertain than 3D measurements but lose information because multiple 3D cuboids can share one projected rectangle.Additional observations are therefore needed to fully constrain camera poses and cuboids.
  • Measurement weighting: Bundle adjustment models measurement errors among cameras, objects, and points, with covariance weighting that gives greater weight to confident and nearby objects.On KITTI, the weight depends on detection probability and distance, with a 70 m truncation distance.

2) Object-point measurement:

Object-point measurements constrain cuboid geometry and standard SLAM reprojection, while feature-based association addresses occlusion and dynamic-object handling challenges.

  • Object-point measurement: Associated points are encouraged to lie inside their assigned 3D cuboid rather than exactly on its surfaces.The cuboid-frame point error uses a max operator to enforce containment.
  • Object-point measurement: Standard 3D point reprojection error remains part of the feature-based SLAM measurement model.The observed pixel coordinate is used as the reprojection target.
  • Data association: Object association is difficult under severe occlusion, repeated objects, and dynamic regions that cannot simply be treated as static landmarks.These conditions limit box-overlap and conventional tracking approaches.
  • Data association: Feature points are assigned to objects when observed inside their 2D boxes across at least two frames and within 1m of the cuboid center.The association combines feature matching with geometric proximity.
  • Data association: Overlapping object regions remain unassociated when correspondence is ambiguous, and a moving car is excluded when no feature point supports it.The figure marks non-object points green and object-associated points by object color.

V. DYNAMIC SLAM

Dynamic object SLAM jointly estimates camera poses, object states, and object-anchored points under rigidity and physically feasible motion assumptions.

  • Dynamic representation: Dynamic object estimation introduces per-frame object poses and object-anchored dynamic points whose local positions remain fixed under rigidity.World positions change with the moving object and are therefore not directly suitable as static SLAM variables.
  • Dynamic factor graph: The dynamic factor graph adds motion, camera-object, and point-camera-object measurement factors to constrain camera poses with moving elements.Static SLAM components are extended with dynamic objects, points, and motion velocity.
  • Object motion model: Object motion is represented with a restricted nonholonomic wheel model using linear velocity and steering angle.For approximately planar vehicle motion, roll, pitch, and z translation are fixed, leaving x, y, and heading yaw.
  • Dynamic measurements: Dynamic-point reprojection transforms the point through the object pose and camera pose before comparing it with the observed pixel.The formulation treats the point position as local to its associated object.

C. Dynamic data association

Dynamic association uses direct image tracking and motion-aware triangulation, while object tracking handles cases where feature tracking becomes unreliable.

  • C. Dynamic data association: Static-environment association is unsuitable for dynamic scenes because object motion makes feature prediction and epipolar checking inaccurate.The difficulty is especially pronounced in monocular dynamic cases.
  • C. Dynamic data association: Dynamic feature points are tracked with 2D KLT optical flow, then triangulated while accounting for object movement.This avoids requiring the 3D point position during pixel tracking.
  • C. Dynamic data association: Motion-aware triangulation treats M2∆T as a modified camera pose that compensates for object movement before solving the two-view problem by SVD.The object transformation relates the point positions across the two frames.
  • C. Dynamic data association: When KLT fails under large pixel displacement, the system tracks 2D object bounding boxes and matches them to current detections.This addresses approaches such as vehicles moving toward the camera.
  • Implementation: The implementation uses YOLO indoors and MS-CNN outdoors, both running in real time on a GPU.The stated probability thresholds are 0.25 for indoor YOLO and 0.5 for outdoor MS-CNN.
  • Implementation: Cuboid proposals sample object yaw and, when necessary, camera roll and pitch; multi-view SLAM supplies camera pose estimates.The approach is designed to operate without large training data and can run in real time with detection and edge extraction.
  • SLAM pipeline: The SLAM pipeline detects and associates cuboids in new keyframes, optimizes them with cameras and points, and can reconstruct or ignore dynamic objects.Cuboids also initialize depths for points that are difficult to triangulate.
  • Optimization: Object and pixel-space costs require hand-tuned weights so different measurement types contribute roughly equally.The paper notes that cuboid detection uncertainty is difficult to analyze.

C. Dynamic object

Experiments evaluate proposal recall, 3D detection, and monocular SLAM, showing strong proposal coverage and improved KITTI pose accuracy when object constraints are combined with ground scaling.

  • Dynamic object: Dynamic motion constraints are applied only over the last 5 seconds because constant velocity may fail during acceleration and deceleration.Ground-truth velocity analysis found approximately stable velocity over that interval.
  • Experiments: Detection is evaluated on SUN RGBD and KITTI using 3D IoU and average precision, with a 25% IoU threshold because SUN RGBD lacks depth data.The datasets provide ground-truth 3D bounding-box annotations.
  • Proposal Recall: 90% 3D proposal recall is achieved with around 50 proposals on SUN RGBD when 2D box IoU is 0.6.Recall increases with the 2D association-IoU threshold.
  • Proposal Recall: On KITTI, the approach is reported to obtain higher proposal recall with fewer proposals, while its recall is limited by missed 2D detections.The comparison includes proposal quality before and after scoring and selection.
  • 3D Object Detection: On SUN RGBD, the method detects about ten times more cuboids than the compared approaches and reports 27% AP on the whole dataset.The comparison controls 2D detector effects for IoU and notes that other methods detect around 200 cuboids.
  • 3D Object Detection: On KITTI, the method performs similarly to SubCNN and Mono3d, while Deep3D is reported as the best-performing approach in the comparison.SubCNN’s false positives lower its AP despite generating many detections.

VIII. EXPERIMENTS - OBJECT SLAM

Experiments across indoor, chair, and KITTI datasets show that object measurements can support camera pose estimation while multi-view optimization improves 3D object detection. The method also handles challenging low-texture and large-rotation settings, with scale and depth benefits from object landmarks.

  • TUM fr3 cabinet: On TUM fr3 cabinet, 3D object IoU improved from 0.46 to 0.64 after SLAM optimization, with 0.17m absolute camera pose error.The dataset has few point features, and only the object was used as the SLAM landmark.
  • TUM fr3 cabinet: Existing monocular SLAM algorithms failed on the low-texture cabinet dataset, whereas object-only SLAM produced a map cube matching the ground-truth point cloud.The 3D object-camera measurement provided more constraints than a 2D measurement.
  • ICL living room: On ICL living room, the system detected sofas, chairs, and potplants without prior object models, while improving object accuracy at some cost to camera pose accuracy.The passage attributes the camera-pose trade-off to imperfect object measurements.
  • Collected chair datasets: On collected chair datasets, cuboids fit associated 3D points tightly after optimization, and cuboid detection provided depth initialization when point-based monocular SLAM failed.The second dataset included large camera rotation, where DSO performed poorly and Mono ORB SLAM failed to initialize.
  • KITTI dataset: KITTI experiments used 2D object-camera measurements for bundle adjustment and evaluated both annotated short sequences and a long odometry benchmark.The implementation also used a prior car size to maintain long-term scale consistency when objects were observed infrequently.

1) KITTI raw sequence:

KITTI raw-sequence experiments show that data association and bundle adjustment generally improve object accuracy, while object geometry reduces monocular scale drift. Dynamic-object optimization uses motion constraints to refine distant detections and estimate trajectories and velocity, but sparse objects and acceleration remain limitations.

  • KITTI raw sequence: Object accuracy increased after data association and bundle adjustment in most KITTI raw sequences, although local position drift sometimes reduced object IoU.Object geometry also provided constraints that reduced monocular scale drift in camera pose estimation.
  • KITTI odometry benchmark: Object SLAM performed worse in Seq 02, 06, and 10 when few objects were visible over long distances, causing large scale drift.The method combined ground-height scaling with object constraints to address this condition.
  • KITTI odometry benchmark: 1.78% translation error was achieved by combining ground scaling with objects on the KITTI odometry benchmark.Ground scaling alone achieved 3.39% translation error, while the combined method achieved state-of-the-art monocular SLAM accuracy.
  • Qualitative results: Multi-view dynamic-object bundle adjustment improved distant car localization because motion-model constraints compensated for ill-constrained single-image depth.Distant cars observed mainly from their back faces could have large 3D distance errors despite good-looking 2D cuboid detections.
  • Qualitative results: The dynamic SLAM system jointly estimated camera and object trajectories instead of treating moving objects as outliers.The resulting object trajectories were smooth due to motion-model constraints.
  • Dynamic velocity estimation: Using only a monocular camera, the proposed algorithm roughly estimated a moving object’s absolute velocity under a piecewise constant velocity model.The model was considered reasonable because the measured ground-truth velocity changed over time.

2) Quantitative results:

The system improves monocular camera pose estimation and 3D object localization across static and dynamic settings, while maintaining modest static-environment optimization overhead.

  • The method improves camera pose estimation on most dynamic sequences, especially when objects are observed and tracked across many consecutive frames.Longer observation tracks improve estimates of object velocity and dynamic-point positions, strengthening their effect on camera pose estimation.
  • 0.8% relative depth error is achieved for 3D object localization on sequence 56.The method outperforms other monocular methods on most evaluated sequences using the relative object depth error metric.
  • Adding objects increases static-environment bundle-adjustment optimization by 7%.The reported increase is attributed to the small number of objects in local-map optimization.
  • Dynamic cases increase computation by a factor of two because of additional dynamic-point variables and measurements.
  • The unified framework couples object detection and geometric SLAM through cuboid proposals, object-level measurements, and tightly coupled optimization.Objects provide long-range geometric and scale constraints, while SLAM supplies camera-pose initialization for detecting and refining 3D objects.
  • The system achieves the best reported 3D object-detection accuracy on SUN RGBD subset data and camera pose estimation on KITTI odometry datasets.
Loading 1806.00557v2…