PaperlayerView exact revision on arXiv

Source-linked AI summary

Stereo Vision-based Semantic 3D Object and Ego-motion Tracking for Autonomous Driving

Peiliang Li, Tong Qin, Shaojie Shen

arXiv:1807.02062v3cs.CV

TL;DR

The paper addresses the challenge of jointly tracking camera ego-motion and 3D objects consistently in dynamic autonomous-driving scenes. It combines lightweight 2D-and-viewpoint-based semantic inference with object-aware camera tracking and dynamic bundle adjustment, and reports accurate camera and object localization across diverse scenarios.

  • Problem

    Existing 3D detection requires extensive 3D labeling and produces frame-independent results, while dynamic driving requires continuous object and camera-state estimation.

  • Method

    The method combines 2D detection, discrete viewpoint classification, semantic 3D measurement, object-aware camera tracking, temporal feature correspondences, and dynamic bundle adjustment.

  • Results

    Quantitative comparisons with state-of-the-art approaches show accuracy for both camera estimation and object localization across diverse scenarios.

  • Takeaways & Limitations

    The integrated framework provides continuous object-state and dynamic sparse-feature tracking with instance accuracy and temporal consistency.

  • Takeaways & Limitations

    The current system tracks the camera and objects successively and plans a fully integrated optimization framework for future work.

Abstract

from arXiv · show

We propose a stereo vision-based approach for tracking the camera ego-motion and 3D semantic objects in dynamic autonomous driving scenarios. Instead of directly regressing the 3D bounding box using end-to-end approaches, we propose to use the easy-to-labeled 2D detection and discrete viewpoint classification together with a light-weight semantic inference method to obtain rough 3D object measurements. Based on the object-aware-aided camera pose tracking which is robust in dynamic environments, in combination with our novel dynamic object bundle adjustment (BA) approach to fuse temporal sparse feature correspondences and the semantic 3D measurement model, we obtain 3D object pose, velocity and anchored dynamic point cloud estimation with instance accuracy and temporal consistency. The performance of our proposed method is demonstrated in diverse scenarios. Both the ego-motion estimation and object localization are compared with the state-of-of-the-art solutions.

1 Introduction

The paper targets inconsistent 3D object localization and camera ego-motion estimation in dynamic driving scenes by combining lightweight semantic inference with temporal geometric tracking.

  • The system addresses 3D object detection's heavy labeling requirements and frame-independent results that lack continuous temporal consistency.The authors motivate a lighter alternative for autonomous-driving perception.
  • Object-aware camera tracking remains robust to dynamic objects, while temporal constraints support tracking in extremely truncated cases.The system also estimates consistent movement and orientation for tracked objects.
  • Dynamic object bundle adjustment tightly couples semantic and feature measurements to track object states and recover sparse dynamic structure with instance accuracy and temporal consistency.The framework uses temporal feature correlations alongside semantic inference.
  • The paper demonstrates practicability across diverse scenarios using a semantic tracking system that combines stereo matching, dynamic feature recovery, and trajectory estimation.

2 Related Work

The related work covers semantic SLAM and learning-based 3D object detection, including methods that use object semantics, data association, and shape or depth priors.

  • This work reviews both semantic SLAM and image-based 3D detection as contexts for its integrated perception solution.
  • Prior semantic SLAM methods use object size or probabilistic data association to improve localization and correct visual-odometry drift.

3D Object Detection

Image-based 3D object detection methods infer pose from learned shape, voxel, depth, or ground-plane cues, while this paper emphasizes integrating semantic priors with precise temporal geometry.

  • Existing approaches infer 3D object pose using dense shape, wireframe, voxel visibility, stereo depth, or ground-plane and segmentation cues.
  • The system architecture is organized around semantic tracking modules rather than direct end-to-end 3D box regression.
  • Purely relying on an instance 2D box limits performance when objects are truncated.
  • The proposed solution combines instance semantic priors with precise spatial-temporal feature correspondences for continuous ego-camera and object-state estimation.

3 Overview

The semantic tracking system uses 2D detection and viewpoint classification, guided stereo-temporal feature matching, and tightly coupled state optimization.

  • The first module detects objects and classifies discrete viewpoints to infer rough poses from 2D box edges and 3D box vertices.
  • The second module projects inferred boxes into images and uses their contours and occlusion masks to guide robust stereo and temporal feature associations.
  • The third module integrates the resulting measurements for semantic object tracking and optimization.

4 Viewpoint Classification and 3D Box Inference

The method combines 2D detection with discrete viewpoint classification to infer rough 3D object pose from box-edge and vertex constraints. This lightweight estimate supports subsequent semantic measurement and optimization while remaining approximate under tightness, dimension-prior, and viewpoint assumptions.

  • 4.1 Viewpoint Classification: Sixteen horizontal–vertical viewpoint combinations associate 2D box edges with 3D box vertices, forming the semantic measurement model for box inference.The classifier uses eight horizontal and two vertical viewpoints.
  • 4.1 Viewpoint Classification: Viewpoint classification is reported as easy to train and accurate for small and extremely occluded objects, while being more robust across diverse scenarios than direct 3D regression.These properties motivate using the classification output as the semantic prior for inference.
  • 4.2 3D Box Inference Based on Viewpoint: The four projection constraints solve the 4 DoF object pose in closed form when the object dimensions prior is known.The constraints use normalized image-plane box edges, projected vertices, and the classified orientation.
  • 4.2 3D Box Inference Based on Viewpoint: The approach replaces direct 3D regression with 2D detection, viewpoint classification, and straightforward closed-form calculation.The inferred pose is used to generate object contours and occlusion masks and to initialize later MAP estimation.
  • 4.2 3D Box Inference Based on Viewpoint: The inferred pose is approximate because it depends on tight 2D boxes and dimension priors, and the box-fit assumption can fail for some top-view cases.The authors state that the assumption is reasonable for almost horizontal or slightly looking-down autonomous-driving viewpoints.

5 Feature Extraction and Matching

The system projects inferred 3D boxes into stereo images to create visibility-aware object contours and masks, then uses constrained stereo and temporal matching to extract corresponding features.

  • 5 Feature Extraction and Matching: Projected 3D boxes generate object contours and visibility masks that distinguish visible, occluded, and background regions.Occluded areas are masked using 2D overlap and 3D depth relations.
  • 5 Feature Extraction and Matching: Stereo matching searches along epipolar lines within the depth range predicted by the inferred object pose, narrowing correspondence search for robustness.The predicted depth range limits matching to a small search interval.
  • 5 Feature Extraction and Matching: Temporal object association uses weighted 2D box center distance and shape similarity after camera-rotation compensation, followed by ORB feature matching.Objects below the maximum similarity threshold are treated as lost.

6 Ego-motion and Object Tracking

The method jointly tracks camera ego-motion and semantic 3D objects by combining stereo feature geometry, semantic measurements, motion priors, and dynamic bundle adjustment. Point-cloud alignment further refines object poses for consistent estimation.

  • 6.1 Ego-motion Tracking: Camera ego-motion is estimated from static background features through maximum-likelihood bundle adjustment.The camera pose and background point cloud are estimated conditionally on the first state.
  • 6.2 Semantic Object Tracking: Object states use semantic measurements, class-specific dimension priors, rigid-body feature anchoring, and temporally correlated observations.Different objects can be tracked independently once the camera pose is known.
  • 6 Ego-motion and Object Tracking: Dynamic object bundle adjustment fuses feature reprojection, dimension-prior, motion-model, and semantic bounding-box residuals for temporally consistent tracking.The formulation converts MAP estimation into nonlinear optimization using Mahalanobis-weighted residuals.
  • 6.2 Semantic Object Tracking: Viewpoint classification links 2D box edges to 3D box vertices, enabling semantic reprojection residuals that connect object pose and dimensions.Only the left image is used for 2D detection to meet real-time requirements.
  • 6.3 Motion Model: The vehicle kinematics model continuously tracks velocity and orientation, while pedestrians use a constant-velocity model for smoothness.The vehicle model parameterizes wheelbase length through object dimensions.
  • Point Cloud Alignment: 3D boxes are aligned to recovered stereo point clouds to reduce pose bias caused by object-size differences, yielding consistent and accurate static and dynamic object poses.The alignment minimizes distances between anchored 3D points and corresponding box surfaces.

7 Experimental Results

The system is evaluated across long trajectories, diverse datasets and dynamic scenarios, with quantitative comparisons covering camera pose and object localization. Results show robust performance in dynamic environments, while object localization remains slightly weaker than 3DOP at the stricter 0.5 IoU threshold.

  • Qualitative Results: On a roughly 700 m close-loop trajectory, the system tracks static and dynamic cars, including one car followed over 200 meters.A continuously truncated car remains trackable through temporal feature constraints and dynamic point-cloud alignment.
  • Qualitative Results Over Diverse Scenarios: Qualitative results span concentrated-car, crossroads and dynamic-road scenarios on both KITTI and Cityscapes datasets.The system does not solve orientation for pedestrians.
  • Quantitative Evaluation: The evaluation compares camera and object poses with isolated state-of-the-art baselines because no integrated academic solution covers both tasks.ORB-SLAM2 is used for camera estimation and 3DOP for stereo 3D object detection.
  • Camera Pose Evaluation: Our system maintains camera-estimation accuracy in high-dynamic KITTI sequences where ORB-SLAM2 shows non-trivial errors from outliers.In less dynamic sequences, both methods show almost the same accuracy.
  • Object Localization Evaluation: Average position errors are 5.9%, 6.1% and 6.3% for easy, moderate and hard objects, respectively.The average error for half of the true-positive objects is below 5%.
  • Object Localization Evaluation: The integrated method achieves more accurate bird’s eye view and 3D box results than 3DOP at the 0.25 IoU threshold, but is slightly worse for 3D boxes at 0.5 IoU.The authors attribute the stricter-threshold gap to unregressed object size.

8 Conclusions and Future work

The paper concludes that tightly coupling semantic, feature and motion information enables robust camera-pose estimation and continuous tracking of static and dynamic objects. Future work targets denser temporal correlations and joint optimization of camera and object tracking.

  • Conclusions: The system tightly couples instance semantics, sparse feature measurements and a kinematic motion model to estimate camera pose and continuously track object states and dynamic sparse features.Demonstrations across diverse scenarios and datasets support its practicability, while quantitative comparisons report accuracy for camera estimation and object localization.
  • Future Work: Future work will exploit dense visual information and replace successive camera and object tracking with a fully integrated optimization framework.The intended integration would allow camera and dynamic-object estimates to benefit from each other.
Loading 1807.02062v3…