PaperlayerView exact revision on arXiv

Source-linked AI summary

VDO-SLAM: A Visual Dynamic Object-aware SLAM System

Jun Zhang, Mina Henein, Robert Mahony, Viorela Ila

arXiv:2005.11052v3cs.RO

TL;DR

Dynamic environments challenge SLAM systems that assume predominantly static scenes, while robot planning and obstacle avoidance require estimates of moving-object motion. VDO-SLAM combines semantic information, feature tracking, and unified estimation to jointly localise the robot, map static and dynamic structure, and track rigid objects without prior object geometry. On real indoor and outdoor datasets, it reports robust, accurate object-motion estimation and state-of-the-art performance.

  • Problem

    Most SLAM systems assume predominantly static environments, while optical-flow and scene-flow methods do not represent full rigid-object SE(3) motion needed for dynamic-scene understanding.

  • Method

    VDO-SLAM uses image-based semantic information in a unified feature-based framework to estimate robot pose, static and dynamic structure, and rigid-object motion without prior object geometry.

  • Results

    VDO-SLAM delivers robust and accurate object-motion estimation with state-of-the-art performance on real indoor and challenging outdoor datasets.

  • Takeaways & Limitations

    The system produces a spatiotemporal map, full SE(3) object motions, and linear velocity estimates for dynamic-scene robotics.

Abstract

from arXiv · show

Combining Simultaneous Localisation and Mapping (SLAM) estimation and dynamic scene modelling can highly benefit robot autonomy in dynamic environments. Robot path planning and obstacle avoidance tasks rely on accurate estimations of the motion of dynamic objects in the scene. This paper presents VDO-SLAM, a robust visual dynamic object-aware SLAM system that exploits semantic information to enable accurate motion estimation and tracking of dynamic rigid objects in the scene without any prior knowledge of the objects' shape or geometric models. The proposed approach identifies and tracks the dynamic objects and the static structure in the environment and integrates this information into a unified SLAM framework. This results in highly accurate estimates of the robot's trajectory and the full SE(3) motion of the objects as well as a spatiotemporal map of the environment. The system is able to extract linear velocity estimates from objects' SE(3) motion providing an important functionality for navigation in complex dynamic environments. We demonstrate the performance of the proposed system on a number of real indoor and outdoor datasets and the results show consistent and substantial improvements over the state-of-the-art algorithms. An open-source version of the source code is available.

I. INTRODUCTION

VDO-SLAM addresses the static-world assumption in SLAM by jointly estimating robot pose, static structure, dynamic object structure, and rigid-object motion. It uses semantic information and feature-based tracking to produce a unified dynamic-scene representation without prior object models.

  • Motivation: Conventional dynamic-SLAM methods either discard moving-object measurements or track objects separately, limiting maps or increasing dependence on camera-pose estimation.The paper argues that dynamic information can benefit SLAM and other robotics tasks when properly modelled.
  • Motivation: Existing optical-flow and scene-flow methods estimate pointwise translation but do not capture the full SE(3) motion of rigid objects.VDO-SLAM instead exploits the collective behaviour of points belonging to individual objects.
  • Motivation: Object-centric map representations commonly require prior 3D object models or explicit object-pose information, restricting applicability and complicating the front end.The paper identifies a need to exploit detection and segmentation without requiring object geometry.
  • Approach: VDO-SLAM unifies robot localisation, static and dynamic mapping, and rigid-object motion tracking in a feature-based stereo/RGB-D system using image-based semantic information.The formulation estimates robot poses, static and dynamic 3D points, and object motions together.
  • Approach: The system introduces denser object-feature representations, rigid-object motion-smoothing factors, and tracking robust to indirect occlusions from failed semantic segmentation.These design choices extend the authors’ previous work for real-world dynamic scenes.
  • Results: VDO-SLAM estimates full SE(3) object motion, extracts object velocity, and reports motion results comparable to camera-pose accuracy while outperforming state-of-the-art methods by an order of magnitude in urban driving scenarios.The authors demonstrate a complete system on real indoor and outdoor datasets.

III. METHODOLOGY

The methodology models camera, object, and point states in a common geometric framework, using projection and optical-flow correspondences for estimation. It then applies tracking and mapping components to refine poses, motions, and a consistent static-dynamic map.

  • Optimisation and mapping: The tracking and mapping components estimate camera pose and object motion, then refine them through factor-graph optimisation while building a globally consistent map of static and dynamic structure.The pipeline combines motion estimation with spatiotemporal map construction.
  • State representation: The system represents robot and object poses as SE(3) transformations and models 3D points in a global reference frame across time.Camera and object poses are defined at each time step, while points are associated with image observations.
  • Projection model: A camera projection function maps homogeneous 3D points to image pixels using the camera intrinsics matrix.The image frame and corresponding pixel locations are defined for each observed point.
  • Feature correspondence: Camera and object motion produce optical flow, which the system uses to establish correspondences between consecutive frames.The flow is defined as the displacement of a pixel between successive images.

3) Object and 3D Point Motions:

The paper represents rigid-object motion as a homogeneous SE(3) transformation relating corresponding object points across consecutive time steps. This enables model-free estimation of object pose change directly from points on the object.

  • The object motion between times k −1 and k is described by a homogeneous transformation in SE(3).
  • For rigid objects, the same object-frame point persists across time, allowing corresponding 3D points to be related by a pose transformation.
  • Equation (6) expresses rigid-object pose change from points residing on the object without including the object 3D pose as an estimation variable.
  • The transformation 0 k−1Hk represents object point motion in the global reference frame and is also called the object pose change or object motion.

B. Camera Pose and Object Motion Estimation

Camera pose is estimated from static 3D-to-2D correspondences by minimizing image-plane reprojection error. The pose uses an SE(3) Lie-algebra parameterization and is optimized with a robust Levenberg–Marquardt procedure.

  • Camera pose 0Xk is estimated by minimizing reprojection error between static 3D points and their 2D image correspondences.
  • The camera pose is parameterized using an se(3) Lie-algebra element xk.
  • The image-plane cost uses a Huber function and reprojection-error covariance for robust estimation from visible static background correspondences.
  • The optimized camera pose is recovered by exponentiating the estimated Lie-algebra pose and solving with Levenberg–Marquardt.

2) Object Motion Estimation:

Object motion is estimated from dynamic-point reprojection error using visible 3D–2D correspondences. The procedure combines robust optimization with optical-flow refinement to address difficult tracking conditions.

  • Object motion is estimated by minimizing a reprojection-error cost over visible dynamic 3D–2D point correspondences.
  • Camera pose and object motion depend on reliable image correspondences, which are difficult under occlusions, large relative motions, and long camera-object distances.
  • The system jointly refines optical flow and motion estimation to improve tracking of points on moving objects.
  • The motion optimization uses a Lie-algebra parameterization of SE(3) and minimizes the associated cost function.

C. Graph Optimisation

The dynamic SLAM problem is formulated as a factor-graph optimization over camera poses, static and dynamic points, and object motions. It jointly integrates point, odometry, dynamic-motion, and smooth-motion information using nonlinear least squares.

  • The factor graph jointly refines camera poses and object motions while building a globally consistent map of static and dynamic structure.
  • The optimization integrates 3D point measurements, visual odometry, dynamic-point motion, and smooth object-motion observations.
  • In the graph, camera poses are connected to static points, dynamic points, object pose changes, and measurement or motion factors.
  • All points on a detected rigid object share the same SE(3) pose transformation between consecutive time steps.
  • Smooth-motion factors penalize abrupt changes between consecutive object motions, motivated by vehicle motion and physical constraints.
  • The variables include camera poses, 3D points, and object motions parameterized through SE(3) and its se(3) Lie algebra.
  • The nonlinear least-squares problem is solved using the Levenberg–Marquardt method and incorporates measurement, odometry, motion, and smooth-motion covariances.

IV. SYSTEM

VDO-SLAM is an object-aware dynamic SLAM system that estimates camera and object motions alongside static and dynamic scene structure. It accepts stereo or RGB-D inputs and can also operate with RGB-only images using estimated depth.

  • IV. SYSTEM: VDO-SLAM jointly estimates camera motion, object motion, and static and dynamic environmental structure.The system comprises image pre-processing, tracking, and mapping components.
  • IV. SYSTEM: Stereo or RGB-D images provide the system input, with stereo images converted into RGB-D data through stereo depth estimation.
  • IV. SYSTEM: An RGB-only version obtains depth through single-image depth estimation while formulating estimation using RGB-D data.

A. Pre-processing

The pre-processing and tracking pipeline separates static background from potentially movable objects and supports persistent tracking through instance segmentation, dense optical flow, and motion-based classification. It combines static-feature camera tracking with object-level correspondence and dynamicity decisions.

  • A. Pre-processing: Instance-level segmentation and dense optical flow address static-background separation and long-term dynamic-object tracking.Segmentation identifies individual object instances, while dense flow increases tracked points on moving objects.
  • A. Pre-processing: The tracking component separately estimates camera ego-motion and dynamic-object motion.
  • A. Pre-processing: Camera pose tracking uses sparse corner features on static background regions, retaining inliers and adding features when tracks fall below 1200.
  • A. Pre-processing: Camera pose estimation compares propagated and P3P-RANSAC motion models using reprojection-error inlier counts.
  • A. Pre-processing: Objects are classified as static or dynamic from scene-flow magnitudes, using a 0.12 point threshold and a 30% dynamic-point proportion threshold.The system conservatively permits static objects to be modeled as dynamic with zero motion.
  • A. Pre-processing: Object labels are propagated from correspondence labels, with a new tracking label assigned when an object’s most frequent previous-frame label is 0.
  • A. Pre-processing: The system processes input images into instance segmentation, dense optical flow, tracked static and dynamic features, refined poses and motions, and an updated local map.

4) Object Motion Estimation:

Object motion estimation samples and tracks points within object masks, while mapping maintains local and global representations refined through batch optimization. The evaluation covers camera motion, object motion, velocity, and tracking on indoor and outdoor datasets.

  • 4) Object Motion Estimation:: Object masks are sampled every third point to provide trackable correspondences, with new points added when tracked inliers decline.
  • 4) Object Motion Estimation:: Global and local maps are maintained and updated through batch optimization.
  • 4) Object Motion Estimation:: The local map uses a 20-frame sliding window to refine camera poses and static structure before updating the global map.Dynamic structure is not locally optimized unless a hard constraint such as constant object motion is assumed.
  • 4) Object Motion Estimation:: The global map stores camera poses, object motions, and inlier structure, with temporally tracked points incorporated into a factor graph.
  • 4) Object Motion Estimation:: Evaluation measures camera motion, object motion, velocity, and object tracking on Oxford Multimotion and KITTI Tracking datasets.Each sequence is run five times and median values are reported because of nondeterministic processing.
  • 4) Object Motion Estimation:: The open-source implementation includes demonstration YAML files and instructions for both evaluation datasets.
  • 4) Object Motion Estimation:: The system uses Mask R-CNN for instance segmentation and PWC-Net for dense optical flow without fine-tuning Mask R-CNN.

B. Error Metrics

The evaluation defines pose-change and speed errors for camera and object motion and reports qualitative and comparative results on Oxford and KITTI data. VDO-SLAM improves several Oxford estimates over MVO, while performance varies with object rotation and depth estimation.

  • B. Error Metrics: Pose-change error evaluates estimated SE(3) camera or object motion against ground truth using a one-frame interval.
  • B. Error Metrics: Object speed error compares estimated and ground-truth linear velocity magnitudes after averaging measurements over object points.
  • B. Error Metrics: The Oxford evaluation uses 500 frames of swinging boxes observed by a moving stereo or RGB-D camera with Vicon ground truth.
  • B. Error Metrics: 35% better camera-pose accuracy and 15% and 40% better swinging-box motion accuracy are reported against MVO on Oxford sequences.
  • B. Error Metrics: Dense optical flow performs worse for self-rotating objects, while the method achieves more than twice the improvement over ClusterVO for other object-motion estimates.
  • B. Error Metrics: Oxford trajectories show camera and box-center paths alongside static-background and object-body feature tracks.
  • B. Error Metrics: KITTI evaluation excludes static-only scenes and nonrigid pedestrians, and assigns zero ground-truth rotation to unavailable axes.
  • B. Error Metrics: Figure 6 compares translation error on a logarithmic axis and rotation error on a linear axis against CubeSLAM.

1) Camera Pose and Object Motion:

VDO-SLAM evaluates camera-pose estimation, dynamic-object motion estimation, tracking, and joint graph optimization on KITTI sequences. It achieves competitive camera-pose accuracy, consistently lower object-motion errors than comparison methods, and improved estimates when motion and optical flow are optimized jointly.

  • Camera Pose and Object Motion: VDO-SLAM achieves competitive camera-pose accuracy against DynaSLAM II, with slightly lower rotational errors but higher translational errors.
  • Camera Pose and Object Motion: Both VDO-SLAM variants consistently obtain lower object-motion errors than CubeSLAM across the evaluated sequences.For RGB-D, translation errors are 0.1–0.3 meters and rotation errors are 0.2–1.5 degrees; the learning-based monocular version has rotation errors of 0.4–3.1 degrees.
  • Object Tracking and Velocity: VDO-SLAM tracks most dynamic objects for more than 80% of their occurrence, while estimated object speeds remain consistently close to ground truth.
  • Robust Tracking of Points: Joint optimization of motion and optical flow improves camera-pose and object-motion estimation over motion-only optimization.The joint process refines optical flow together with motion estimation before graph optimization.
  • Robust Tracking of Points: About 10% improvement in camera errors and 25% improvement in object errors, in both translation and rotation, were observed over nine KITTI sequences.

2) Robustness against Non-direct Occlusion:

The system maintains object tracking and speed estimation despite segmentation failures, and global graph refinement improves object-motion estimates. Its experiments also expose computational and long-term scalability considerations.

  • Robustness against Non-direct Occlusion: VDO-SLAM continuously tracks a van and estimates its speed despite segmentation failure in 33 of 80 frames.The average speed error across the sequence is 2.64 km/h, with higher errors during partial direct occlusions and increased object distance.
  • System Output: The system outputs camera poses, static structure, and dynamic tracks for detected moving objects in a spatiotemporal map.
  • Global Refinement on Object Motion: Global refinement makes estimated object speeds smoother and significantly improves them in the latter half of the sequence.
  • Global Refinement on Object Motion: Graph optimization reduces object-motion errors by up to 39% in translation and 55% in rotation across KITTI sequences.Translation errors increase slightly in Seq.18 and Seq.20, where alternating acceleration and deceleration violate the imposed smooth-motion constraint.
  • Computational and Long-Term Considerations: Long-term dynamic SLAM remains constrained by computational complexity, motivating history summarization or deletion of old dynamic-object map points.
Loading 2005.11052v3…