PaperlayerView exact revision on arXiv

Source-linked AI summary

QuadricSLAM: Dual Quadrics from Object Detections as Landmarks in Object-oriented SLAM

Lachlan Nicholson, Michael Milford, Niko Sünderhauf

arXiv:1804.04011v3cs.RO

TL;DR

SLAM maps often lack immediate object-level semantics, motivating object-oriented landmarks that support richer robot understanding. QuadricSLAM jointly estimates camera poses and constrained dual quadrics from odometry and bounding-box detections using a perspective-camera factor graph with a geometric error robust to partial visibility. Its experiments report substantial improvements in trajectory and landmark estimates, while emphasizing object-centric maps as the main benefit.

  • Problem

    SLAM maps typically represent geometry without immediate object-level semantic information, limiting their object-oriented character.

  • Method

    QuadricSLAM jointly estimates camera poses and constrained dual-quadric object landmarks from odometry and detector bounding boxes in a factor-graph formulation using a partial-visibility-robust geometric error.

  • Results

    65.2% improvement on trajectory error and 70.4%, 26.7%, and 30.6% improvement on landmark position, shape, and quality were reported for quadric landmarks.

  • Takeaways & Limitations

    The main benefit of quadric-based landmarks is estimating maps that contain objects as distinct elements while providing information for correcting odometry errors.

Abstract

from arXiv · show

In this paper, we use 2D object detections from multiple views to simultaneously estimate a 3D quadric surface for each object and localize the camera position. We derive a SLAM formulation that uses dual quadrics as 3D landmark representations, exploiting their ability to compactly represent the size, position and orientation of an object, and show how 2D object detections can directly constrain the quadric parameters via a novel geometric error formulation. We develop a sensor model for object detectors that addresses the challenge of partially visible objects, and demonstrate how to jointly estimate the camera pose and constrained dual quadric parameters in factor graph based SLAM with a general perspective camera.

I. INTRODUCTION

QuadricSLAM addresses the lack of immediate object-level semantics in conventional SLAM by using objects as compact, object-oriented landmarks. It jointly estimates camera poses and constrained dual quadrics from odometry and object detections.

  • Motivation: Conventional SLAM maps typically encode geometric information without immediate object-level semantic information.Semantically enriched maps are motivated by richer robot understanding and interaction.
  • Motivation: Quadrics compactly represent an object's size, position, and orientation while remaining efficient within projective geometry.They can also anchor more detailed 3D reconstructions when needed.
  • Contributions: QuadricSLAM parametrizes object landmarks as constrained dual quadrics and uses object detections to directly constrain their parameters.The formulation jointly estimates dual quadrics and robot poses in a factor graph with solved data association.
  • Evaluation: The paper evaluates the approach on 250 indoor trajectories in high-fidelity simulation and on real-world TUM RGB-D sequences.The evaluation examines how object detections and dual-quadric parametrization affect SLAM.
  • Contributions: The method estimates camera pose and quadric parameters directly from bounding boxes using a geometric error robust to partially visible objects.This avoids the ellipse-fitting preprocessing required by earlier approaches.

A. Maps and Landmark Representations in SLAM

The paper positions dual quadrics as object-level alternatives to geometry-only SLAM landmarks. Compared with prior quadric methods, it performs full perspective-camera SLAM directly from detector bounding boxes and uses a geometric representation robust to partial visibility.

  • Maps and Landmark Representations in SLAM: Point-, line-, plane-, and dense-reconstruction SLAM maps generally provide geometric but not immediate semantic information.Object-oriented alternatives place recognizable objects at the center of the map.
  • Maps and Landmark Representations in SLAM: Unlike methods that add semantics after reconstruction or require known CAD models, this approach uses detected objects as landmarks encoded by quadrics.It uses YOLOv3 detections rather than a-priori known object instances.
  • Prior Quadric Methods: Earlier quadric methods were limited either to orthographic cameras, ellipse-fitting preprocessing, or landmark mapping with known camera poses.These restrictions limited their direct use in full robotic SLAM.
  • Prior Quadric Methods: QuadricSLAM works with perspective cameras and estimates camera pose, landmark pose, and shape parameters simultaneously.Its formulation therefore performs full simultaneous localization and mapping rather than mapping alone.
  • Robust Quadric SLAM: The method replaces a prior algebraic error with a geometric error that remains well-defined when object landmarks are only partially visible.It also constrains landmarks to ellipsoids, initializes them correctly, and evaluates them in simulation and real image sequences.
  • Dual Quadrics: A projected dual quadric produces a dual conic through C* = P Q* P^T, linking 3D landmark geometry to image-space observations.Here P = K[R|t] contains the camera's intrinsic and extrinsic parameters.

B. Constrained Dual Quadric Parametrization

The paper restricts dual quadrics to closed ellipsoidal or spherical surfaces and parameterizes them compactly by rotation, translation, and three semi-axis lengths. It then models detector bounding boxes as predictions from camera pose and quadric parameters.

  • Constrained Dual Quadric Parametrization: The constrained representation ensures that each object landmark is an ellipsoid or sphere rather than a non-closed quadric surface.This restriction selects closed surfaces as meaningful object representations.
  • Constrained Dual Quadric Parametrization: A homogeneous transformation places an origin-centered ellipsoid using arbitrary rotation and translation.The transformation separates the canonical ellipsoid from its pose in 3D space.
  • Constrained Dual Quadric Parametrization: The constrained dual quadric is represented by a 9-vector containing three rotation angles, three centroid-translation coordinates, and three semi-axis lengths.The full dual quadric is reconstructed from these parameters.
  • Object Detection Sensor Model: The object-detector sensor model predicts an axis-aligned bounding box from camera pose x_i and quadric parameters q_j.The observed box is represented by xmin, ymin, xmax, and ymax pixel coordinates.
  • Object Detection Sensor Model: The sensor model supports a geometric error between predicted and observed detections, forming a crucial component of the SLAM system.The detector therefore acts as a sensor that constrains object landmarks through image-space observations.

B. Deriving the Object Detection Sensor Model β

The sensor model predicts an object’s visible image bounding box by projecting its quadric into the image and accounting for conic–image-border intersections. This avoids errors caused by truncating boxes when conic extrema lie outside the image.

  • Projection: Projecting the estimated quadric with the camera pose produces a dual conic, whose primal form is obtained by taking the adjugate.The camera projection uses the camera pose and quadric parameters before converting the projected dual conic into primal form.
  • Partial visibility: A naive enclosing-box model can introduce significant errors when conic extrema lie outside the image boundaries.Simply truncating the conic’s enclosing box to the image dimensions does not correctly represent partial visibility.
  • Visible conic bounding box: The correct prediction is the minimal axis-aligned rectangle enclosing all of the conic contained within the image dimensions.This sensor output is denoted BBox(C).
  • BBox(C) algorithm: BBox(C) finds conic extrema and border intersections, removes non-real or out-of-image points, then returns the remaining coordinate minima and maxima.The procedure considers four extrema and up to eight image-boundary intersections.

V. SLAM WITH DUAL QUADRIC LANDMARK REPRESENTATIONS

The SLAM formulation jointly estimates robot poses and dual-quadric landmarks from odometry and associated object detections. A factor graph and MAP objective combine motion and landmark likelihoods into a nonlinear least-squares problem.

  • Odometry factors: The problem uses odometry between successive poses, with unknown robot poses linked by a nonlinear motion model and Gaussian error.Wheel odometry and visual odometry are both possible sources of the measurements.
  • Landmark observations: Object detections provide bounding boxes associated with physical landmarks, assuming data association is already solved.Each detection is indexed by the observing pose and the object identity.
  • Factor graph: The conditional distribution over poses and landmarks given odometry and detections is represented as a factor graph with odometry and landmark factors.The factorization provides the probabilistic structure for the landmark SLAM problem.
  • MAP estimation: MAP estimation seeks the pose and dual-quadric configuration that maximizes the joint probability of the observations.The MAP solution is the mode of the joint probability distribution.
  • Optimization: The joint MAP objective becomes nonlinear least squares after taking the negative log and factoring the joint probability.Odometry terms use Gaussian motion assumptions, and the odometry difference is computed in SE(3).
  • Landmark factors: The landmark likelihood is Gaussian around the sensor-model prediction, with covariance representing spatial uncertainty in image-space detections.Under a uniform quadric prior, maximizing the posterior is equivalent to maximizing this likelihood.
  • Optimization: Sparse nonlinear least-squares solvers such as Levenberg–Marquardt and Gauss–Newton can efficiently solve typical problems with thousands of variables.This efficiency comes from exploiting the sparse factorization structure.

C. The Geometric Error Term

The landmark factor uses a geometric pixel-coordinate error between observed and predicted bounding boxes. Unlike the prior algebraic error, it remains well-defined for partially visible objects and avoids shrinking the estimated quadric.

  • Geometric formulation: The geometric error compares observed and predicted bounding-box pixel coordinates through the landmark covariance model.Because both quantities are vectors of image coordinates, the resulting error has direct geometric meaning.
  • Partial visibility: For partially visible objects, the geometric formulation remains well-defined where the algebraic formulation fails and shrinks the estimated quadric.Truncated bounding-box observations are the specific case that invalidates the algebraic error.
  • Uncertainty: The geometric error also supports future propagation of detector spatial uncertainty through the covariance matrices Λ_ij.The paper identifies this as a future integration of detector uncertainty into SLAM.

D. Variable Initialization

Initialization uses odometry for initial camera poses and multi-view detections for quadric landmarks. A linear SVD estimate is then constrained to an ellipsoid and parameterized by shape, rotation, and translation.

  • Initial estimates: Robot poses are initialized from raw odometry, whereas dual-quadric landmark initialization requires additional processing.All variable parameters must be initialized for incremental solvers.
  • Alternative initialization: An alternative initialization fits each landmark’s defining equation by least squares using the vectorized general dual quadric.The general dual-quadric vector is distinct from the constrained parametrized quadric vector.
  • Linear constraints: Bounding-box lines are back-projected into planes using the initial camera matrix, producing linear constraints on each dual quadric.The initial camera poses determine the projection matrices used for these plane constructions.
  • SVD initialization: Collecting constraints across views yields A_j q̂_j = 0, whose least-squares solution is obtained from the SVD’s last right-singular vector.The matrix contains coefficients from planes associated with observations of the landmark.
  • Ellipsoid constraint: The unconstrained SVD solution is a generic quadric, so the estimated landmarks are constrained to ellipsoids.This converts the generic surface estimate into the constrained landmark representation used by the method.
  • Parameterization: Quadric parameters are initialized by extracting shape, rotation, and translation from the primal or dual quadric representation.Rotation comes from eigenvectors of the upper-left submatrix, while translation is recovered from the homogeneous last column.

VI. EXPERIMENTS AND EVALUATION

The paper evaluates quadric landmarks on real-world TUM RGB-D sequences and separately tests landmark quality in simulation with ground-truth object shapes and positions.

  • Quadric landmarks are evaluated on sequences from the publicly available TUM RGB-D dataset.The evaluation compares localization performance against state-of-the-art techniques.

A. TUM RGB-D Experiments

Real-world experiments evaluate camera localization from two visual-odometry sources, compare trajectory error with ORB-SLAM2, and assess object-centric map quality under realistic detection challenges.

  • A. TUM RGB-D Experiments: Four TUM RGB-D sequences evaluate camera trajectories under image noise, blur, lighting changes, occlusions, and moving objects.The sequences are fr1 desk, fr1 desk2, fr2 desk, and fr3 office.
  • A. TUM RGB-D Experiments: QuadricSLAM uses Fovis or loop-closure-disabled ORB-SLAM2 for visual odometry, with YOLOv3 detections and manually provided object associations.The factor graph connects robot poses and dual quadrics through odometry and bounding-box factors.
  • A. TUM RGB-D Experiments: Depth information supplies consistent odometry scale but does not yet support estimating quadric parameters.Bounding-box covariance is approximated from width and height variability across detections.
  • B. TUM RGB-D Results: QuadricSLAM improves trajectory quality over Fovis visual odometry but performs slightly worse than ORB-SLAM2 on the TUM sequences.The reported performance loss is attributed to constant noise estimates for non-Gaussian bounding-box measurements and object occlusions.
  • B. TUM RGB-D Results: Rejecting objects with high bounding-box variance improves trajectory error over ORB-SLAM2 on some sequences.Using width or height standard-deviation thresholds above 50 pixels and 30 pixels yields errors of 0.0239 meters on fr1 desk2 and 0.0087 meters on fr3 office.
  • B. TUM RGB-D Results: Projected quadric surfaces align with object boundaries and provide consistent class labels and object shapes across successive frames.Figure 3 displays projected quadrics only when their associated objects are visible; occluded objects are not displayed.
  • B. TUM RGB-D Results: Some SVD-initialized quadric landmarks appear behind the camera, particularly for objects with limited viewing angles.Such objects are observed only a few times and typically have little impact on the resulting trajectory.

C. Simulated Experiments

Simulated experiments use UnrealCV scenes with ground-truth trajectories, detections, and 3D object boxes to evaluate localization and quadric landmark accuracy under controlled noise.

  • C. Simulated Experiments: The synthetic dataset contains 10 UnrealCV scenes and 50 trajectories with ground-truth camera trajectories, 2D detections, and 3D object bounding boxes.Data association between 3D objects and 2D detections is provided by the simulator.
  • C. Simulated Experiments: Simulation adds roughly 5% translation error and 15% rotation error to camera motion and adds 4 pixels of detection variance.Factor-graph covariance matrices are set to the introduced odometry and bounding-box noise.
  • C. Simulated Experiments: Localization is evaluated with Absolute Trajectory Error against noisy odometry, while landmark estimates are compared with ground-truth object boxes using three metrics.The first metric measures translation error between the estimated quadric and the ground-truth object centroid.

D. Simulated Results

Simulated experiments show that quadric landmarks improve camera trajectories and landmark maps, while occlusion and limited viewing angles remain sources of estimation error. The paper positions dual quadrics as compact, object-based landmarks for semantically meaningful SLAM maps.

  • 65.2% improvement in trajectory error accompanies 70.4%, 26.7% and 30.6% improvements in landmark position, shape and quality.The correcting effect comes from reobserving landmarks between frames, which helps mitigate accumulated odometry errors.
  • Quadric landmarks provide accurate high-level information about object shape and position while improving the estimated robot trajectory and map.
  • Occlusion shrinks estimated landmark surfaces, whereas limited viewing angles can overestimate landmark shapes.
  • The standard algebraic error improves camera trajectory and landmark quality by only 0.6% and 1.5%, while worsening landmark position and shape by 2.6% and 2.0%.Partial object visibility, especially for large objects, causes this behavior.
  • Quadric-based landmarks primarily enable maps containing objects as distinct elements, while also providing information that corrects odometry errors.
  • The method jointly estimates camera trajectory and object parameters using closed-surface dual quadrics, a detector sensor model and a geometric error robust to partial observations.
  • The evaluation combines trajectory and landmark-quality experiments in real-world data and simulation, with future work targeting detector uncertainty, initialization constraints and depth-based rejection.
Loading 1804.04011v3…