Source-linked AI summary
DynaSLAM II: Tightly-Coupled Multi-Object Tracking and SLAM
Berta Bescos, Carlos Campos, Juan D. Tardós, José Neira
TL;DR
Static-scene assumptions limit visual SLAM in populated environments that require explicit motion information. DynaSLAM II tightly integrates multi-object tracking with stereo and RGB-D SLAM through joint bundle adjustment, and reports state-of-the-art accuracy at real-time performance while noting feature-based limits for bounding boxes and low-texture objects.
Problem
Static-scene assumptions limit visual SLAM in populated environments where explicit motion information is needed for decision-making and scene understanding.
Method
DynaSLAM II combines instance semantic segmentation and ORB features with tightly coupled bundle adjustment for camera, static-scene, and dynamic-object estimation.
Results
DynaSLAM II achieves state-of-the-art accuracy at real-time performance, while tracking dynamic objects provides benefits for camera tracking.
Takeaways & Limitations
Joint optimization makes camera and surrounding-object trajectory estimates mutually beneficial and supports scene understanding across applications.
Takeaways & Limitations
The feature-based core limits accurate 3D bounding-box discovery and tracking of low-texture objects.
Abstract
from arXiv · showhide
The assumption of scene rigidity is common in visual SLAM algorithms. However, it limits their applicability in populated real-world environments. Furthermore, most scenarios including autonomous driving, multi-robot collaboration and augmented/virtual reality, require explicit motion information of the surroundings to help with decision making and scene understanding. We present in this paper DynaSLAM II, a visual SLAM system for stereo and RGB-D configurations that tightly integrates the multi-object tracking capability. DynaSLAM II makes use of instance semantic segmentation and of ORB features to track dynamic objects. The structure of the static scene and of the dynamic objects is optimized jointly with the trajectories of both the camera and the moving agents within a novel bundle adjustment proposal. The 3D bounding boxes of the objects are also estimated and loosely optimized within a fixed temporal window. We demonstrate that tracking dynamic objects does not only provide rich clues for scene understanding but is also beneficial for camera tracking. The project code will be released upon acceptance.
I. INTRODUCTION
Visual SLAM traditionally assumes static scenes, limiting use in populated environments where explicit motion information supports navigation, decision-making, and scene understanding. DynaSLAM II addresses this by jointly estimating camera motion, static structure, and moving-object trajectories.
- Most visual SLAM approaches assume static scenes and reject dynamic regions as outliers, limiting their use in populated environments.
- Explicit motion information is required in autonomous driving, multi-robot collaboration, and AR/VR for decision-making and scene understanding.The paper highlights collision avoidance and interaction between virtual objects and moving real-world instances.
- DynaSLAM II is a stereo and RGB-D SLAM system that simultaneously estimates camera poses, the map, and trajectories of moving objects.
- Its bundle adjustment tightly optimizes scene structure, camera poses, and object trajectories in a local temporal window.Object bounding boxes are optimized separately to estimate object dimensions and 6 DoF poses without a particular use case.
- The system combines instance semantic information with sparse visual features to track dynamic objects while retaining real-time operation.
II. RELATED WORK
Prior work either tracks dynamic objects separately from SLAM or tightly combines selected motion and reconstruction components with scope or modeling constraints. DynaSLAM II uses a tightly coupled, feature-based formulation that combines cameras, objects, and structure points without hard-coded priors.
- Traditional multi-object tracking is performed separately from SLAM, making tracking accuracy highly dependent on camera pose estimation.Camera-pose failures are especially problematic when reliable static structure is unavailable.
- Joint camera-and-object estimation has been extended to RGB-D systems that reconstruct static indoor scenes and moving objects using instance segmentation.
- Existing feature-based approaches impose different limitations, including synthetic-only evaluation, car-specific models, object priors, or visibility requirements.
- DynaSLAM II uses tightly coupled bundle adjustment with measurements linking cameras, objects, and structure points.It integrates instance semantic priors with sparse image features while emphasizing computational complexity and parameter count.
- The system independently handles dynamic-object trajectories and object-structure or bounding-box representations to retain benefits from both modeling directions.
B. Objects Data Association
DynaSLAM II associates semantic instances and ORB features across frames, initializes object tracks with geometric information, and refines their poses through reprojection-error minimization. Its object-aware formulation enables joint optimization of camera poses, object poses, and 3D points.
- The system detects dynamic instances through pixel-wise segmentation and ORB features, then assigns corresponding key points to newly created objects.Objects are created for dynamic classes with many nearby new key points.
- Object instances are matched using dynamic feature correspondences after static and dynamic features are associated with previous-frame and map features.
- An object track is initialized from the 3D points’ center of mass with identity rotation, while later poses use a constant-velocity prediction.
- The object pose estimate is refined by minimizing reprojection error, which projects 3D points into rectified stereo or RGB-D image coordinates.
- Dynamic-object reprojection modeling uses an object pose and object-reference 3D points to explain observed image measurements.
- The formulation jointly optimizes camera and moving-object poses together with the positions of their 3D points.
C. Object-Centric Representation
DynaSLAM II represents dynamic objects through shared object references rather than independently repeated point clouds, reducing the optimization parameter count and supporting real-time operation.
- Object-Centric Representation: Reducing parameters is important because tracking moving objects adds substantial complexity on top of SLAM while real-time performance must be maintained.The conventional representation becomes prohibitive as the number of cameras, objects, and object points grows.
- Object-Centric Representation: Independent dynamic point tracking requires N = 6Nc + Nc × No × 3Nop parameters, compared with N′ = 6Nc + Nc × 6No + No × 3Nop using object representations.Object-centric modeling makes 3D object points unique and models object poses over time.
- Object-Centric Representation: Object-centric modeling brings great savings in the number of utilized parameters for tracking dynamic objects.The parameter compression ratio is illustrated for 10 objects in Fig. 3.
D. Bundle Adjustment with Objects
DynaSLAM II extends bundle adjustment to jointly optimize camera, static-map, and dynamic-object variables, while modeling object motion with velocity constraints and exploiting sparse structure.
- Bundle Adjustment with Objects: The bundle adjustment jointly refines camera poses, static map points, dynamic object points, and object poses through reprojection-error terms.The factor-graph formulation connects cameras, objects, and structure points in a local optimizable window.
- Bundle Adjustment with Objects: Weak camera or object tracking triggers key-frame insertion, after which local BA optimizes the relevant camera, object, velocity, point, and map variables.When both camera and object tracking are weak, the system jointly optimizes camera poses, map structure, object poses, velocities, and points.
- Bundle Adjustment with Objects: A constant-velocity assumption imposes smooth object trajectories by constraining consecutive observations with linear and angular velocities.The formulation uses the exponential map for SO(3) to construct the motion transformation.
- Bundle Adjustment with Objects: Object velocities are coupled to object poses and corresponding 3D points through an additional error term between consecutive observations.The pose transformation ΔT_i,i+1^k represents the object motion over the interval between observations.
- Bundle Adjustment with Objects: The BA problem uses a robust Huber cost and covariance matrices that reflect keypoint scale or uncertainty from the observation time interval.Longer intervals receive greater uncertainty under the constant-velocity assumption.
- Bundle Adjustment with Objects: The Hessian sparsity pattern reflects distinct connections for static map points and dynamic object points, while point variables dominate matrix size.Schur-complement computation is affected by the numbers of static and dynamic points.
E. Bounding Boxes
DynaSLAM II separates dynamic-object trajectory estimation from bounding-box estimation, enabling early viewpoint-independent tracking while refining object dimensions and pose over multiple views.
- Bounding Boxes: The Hessian representation is evaluated for 5 key frames, 1 object with 10 object points, and 10 static map points.The figure provides a concrete configuration for inspecting the optimization structure.
- Bounding Boxes: Trajectory estimation and bounding-box estimation are decoupled so dynamic objects can be tracked from their first visible frame independently of camera-object viewpoint.The trajectory uses a reference point defined from the object points observed initially.
- Bounding Boxes: Bounding boxes are initialized from perpendicular planes fitted to object points, with a class-related prior for an unobservable direction when only one plane is found.A RANSAC procedure selects the box whose image projection has the largest IoU with the CNN 2D bounding box.
- Bounding Boxes: Bounding-box dimensions and pose are refined by minimizing image-projection distance to the CNN 2D prediction within a temporal window.Refinement begins only after an object has at least three observing key frames because fewer views are insufficient for observability.
IV. EXPERIMENTS
The experiments evaluate DynaSLAM II in two blocks: camera-motion estimation with object tracking and multi-object tracking performance.
- IV. EXPERIMENTS: The experimental evaluation assesses how tracking objects affects camera-motion estimation and separately analyzes multi-object tracking performance.These evaluations correspond to Subsections IV-A and IV-B.
A. Visual Odometry
DynaSLAM II is evaluated for visual odometry on KITTI stereo sequences and compared with ORB-SLAM2, DynaSLAM, and joint SLAM-tracking systems. It achieves lower translational relative error than VDO-SLAM across all sequences, while VDO-SLAM usually has lower rotational pose error.
- Datasets: The visual odometry experiments use KITTI tracking and raw datasets containing stereo urban and road scenes with vehicles, pedestrians, and GPS data.The tracking sequences provide grayscale and RGB stereo data recorded from a car perspective.
- Comparisons: Tables I and II compare DynaSLAM II with ORB-SLAM2, DynaSLAM, and systems that jointly track dynamic objects within SLAM.ORB-SLAM2 is the underlying static-scene system, while DynaSLAM detects dynamic features but ignores and does not track them.
- Results: DynaSLAM II achieves better performance than ORB-SLAM2 and DynaSLAM across many sequences containing moving or parked dynamic objects.The comparison covers both circulating objects and representative but stationary objects such as parked cars.
- Results: DynaSLAM II has lower translational relative error (RPEt) than VDO-SLAM in all sequences, whereas VDO-SLAM usually has lower rotational pose error.The reported errors use different metrics supplied by the compared systems; ClusterVO is quite similar to DynaSLAM II in most sequences.
B. Multi-Object Tracking
The multi-object tracking evaluation uses KITTI annotations and compares object detection and trajectory estimation with established tracking criteria. DynaSLAM II handles truncation and occlusion with a minor precision loss, but usually discovers fewer bounding boxes.
- Evaluation setup: KITTI tracking provides manually annotated trajectories and 3D bounding boxes from LiDAR point clouds for evaluating dynamic-object tracking.The evaluation uses expensive manual annotations as ground truth.
- Metrics: CLEAR MOT metrics assess per-frame tracking characteristics, but do not account for the quality of tracked object trajectories.The paper therefore recommends reporting MOTP together with common trajectory error metrics in SLAM tracking.
- Detection evaluation: Table III evaluates object detections on KITTI using the 3D object detection benchmark, with Easy, Moderate, and Hard categories based on height, occlusion, and truncation.MOTP is multiple object tracking precision, computed from prediction precision over true positives using a cost function.
- Results: DynaSLAM II handles object truncation and occlusion with a minor precision loss, but usually discovers fewer bounding boxes than comparison systems.The authors associate the reduced number of detections with the challenges of the feature-based approach compared with dense stereo matching.
- Trajectory evaluation: Trajectory evaluation selects 12 long KITTI sequences with visible, sufficiently large objects and matches estimated boxes using at least 25% overlap.The selected objects have neither occlusion nor truncation and are at least 40 pixels high.
C. Timing Analysis
DynaSLAM II’s runtime depends strongly on the number of tracked objects. It runs at 12 fps with two simultaneous objects and approximately 10 fps with up to 20, while being reported as the only real-time joint SLAM-tracking system in the KITTI comparison.
- Runtime dependence: DynaSLAM II’s computational timing varies with the number of objects tracked simultaneously.The timing analysis reports average runtimes for the system’s different building blocks.
- Runtime results: 12 fps is achieved on KITTI tracking sequence 0003, which contains at most two objects at a time.The reported rate refers to the system excluding semantic-segmentation CNN time.
- Runtime results: Approximately 10 fps is achieved on sequence 0020 with up to 20 objects, despite a slight performance compromise.The authors still characterize this performance as real time.
- System comparison: DynaSLAM II is reported as the only system providing a real-time joint SLAM and multi-object tracking solution on the KITTI dataset.The timing comparison appears in the final rows of Table V.
V. CONCLUSIONS AND FUTURE WORK
DynaSLAM II tightly couples camera, map, and dynamic-object estimation through object-level measurements while decoupling trajectories from bounding-box estimation. The feature-based design limits accurate boxes and tracking for low-texture objects, motivating denser and monocular extensions.
- Conclusions: The system introduces object-level measurements between cameras, objects, and 3D map points so camera and surrounding-object trajectories can be jointly optimized.The coupled estimates are intended to make self-motion and surrounding-motion estimation mutually beneficial.
- Conclusions: DynaSLAM II decouples dynamic-object trajectory estimation from bounding-box estimation and makes no assumptions about object motion, pose, or model.Bounding boxes are optimized separately from the object trajectories.
- Conclusions: The experiments report state-of-the-art accuracy at real-time performance, supporting the framework’s suitability for many applications.This is the paper’s stated overall experimental conclusion.
- Future work: The feature-based core limits accurate 3D bounding-box discovery and tracking of low-texture objects.The authors identify dense visual information as a way to push these limits forward.
- Future work: Future work includes multi-object tracking and SLAM with only a monocular camera, where dynamic tracking could provide clues about map scale.The paper describes monocular operation as a more challenging task.