PaperlayerView exact revision on arXiv

Source-linked AI summary

No Blind Spots: Full-Surround Multi-Object Tracking for Autonomous Vehicles using Cameras & LiDARs

Akshay Rangesh, Mohan M. Trivedi

arXiv:1802.08755v4cs.CV

TL;DR

Autonomous-vehicle reasoning needs accurate, long, noise-robust tracks, while prior fusion approaches remain limited to vision-sensor fields of view. M3OT extends MDP-based tracking to real-world coordinates and fuses proposals across calibrated cameras and optional range sensors. Tests on full-surround highway data support tracking across camera configurations and long sequences.

  • Problem

    Autonomous-vehicle tasks such as obstacle avoidance and path planning need accurate, long, noise-robust tracks, while prior fusion approaches remain limited to vision sensors’ fields of view.

    Source: §I · p. 1

  • Method

    M3OT extends MDP-based tracking to real-world coordinates and fuses object proposals across calibrated cameras, with optional range sensors and variable camera configurations.

    Source: §I · p. 2, §III · p. 3, §VI · p. 11

  • Results

    More than 70% of targets are mostly tracked in sequences about 3 minutes long, with strong scores across camera configurations and metrics.

    Source: A. Experimenting with Number of Cameras · p. 9

  • Takeaways & Limitations

    Highway-data results support M3OT’s modularity and long-duration tracking, and the framework can be used to evaluate camera setups.

    Source: §VI · p. 11

Abstract

from arXiv · show

Online multi-object tracking (MOT) is extremely important for high-level spatial reasoning and path planning for autonomous and highly-automated vehicles. In this paper, we present a modular framework for tracking multiple objects (vehicles), capable of accepting object proposals from different sensor modalities (vision and range) and a variable number of sensors, to produce continuous object tracks. This work is a generalization of the MDP framework for MOT, with some key extensions - First, we track objects across multiple cameras and across different sensor modalities. This is done by fusing object proposals across sensors accurately and efficiently. Second, the objects of interest (targets) are tracked directly in the real world. This is a departure from traditional techniques where objects are simply tracked in the image plane. Doing so allows the tracks to be readily used by an autonomous agent for navigation and related tasks. To verify the effectiveness of our approach, we test it on real world highway data collected from a heavily sensorized testbed capable of capturing full-surround information. We demonstrate that our framework is well-suited to track objects through entire maneuvers around the ego-vehicle, some of which take more than a few minutes to complete. We also leverage the modularity of our approach by comparing the effects of including/excluding different sensors, changing the total number of sensors, and the quality of object proposals on the final tracking result.

I. INTRODUCTION

The introduction motivates full-surround tracking in real-world coordinates and presents M3OT as a modular, multi-sensor extension of online MOT. The related research surveys 2D tracking-by-detection and 3D methods using vision, range sensing, or sensor fusion.

I. INTRODUCTION

  • Prior fusion approaches lack full-surround vision solutions and remain limited to the field of view of their vision sensors.Stereo and single-camera methods also face field-of-view or localization limits; dense-point-cloud approaches depend on representations that do not scale well to LiDAR with fewer scan layers.Source: §I · pp. 1–2
  • The paper extends online 2D tracking-by-detection, where detections are linked to targets through learned similarity, using the MDP framework proposed in.The framework models each object's lifetime with an MDP and treats target birth, death, appearance, and disappearance as state transitions.Source: §I · p. 2, §II · p. 2
  • M3OT extends an MDP formulation to real-world 3D tracking, fuses proposals across calibrated cameras, and can include range sensors in modular camera configurations.The framework is evaluated using naturalistic highway data with full-surround vision and range modalities.Source: §I · p. 2

II. RELATED RESEARCH

  • Related 2D MOT work frames data association as linking detections to targets, using global optimization for batch methods and probabilistic or deterministic approaches online.Some methods learn similarity functions from training data, as does the MDP-based approach extended here.Source: §II · p. 2
  • Related 3D MOT studies use stereo-derived stixels, voxels, or grids, dense LiDAR point clouds, and sensor-fusion schemes with different tracking and association methods.The surveyed approaches include Kalman and particle filters, greedy association, and Hungarian matching.Source: §II · pp. 2–3

III. FUSION OF OBJECT PROPOSALS

M3OT fuses proposals from calibrated camera arrays and optional LiDAR before tracking, associating detections across views and modalities online. It produces tracks in individual sensor frames and the global coordinate frame.

III. FUSION OF OBJECT PROPOSALS

  • Early fusion combines proposals from camera arrays and optional LiDAR at each time step before the MDP tracking stage.The framework supports calibrated cameras with varying degrees of FoV overlap.Source: §III · p. 3, Figure 2 · p. 4
  • The global coordinate frame is centered at the ego-vehicle’s rear-axle midpoint, with sensor-to-global mappings based on calibration.LiDAR mappings use coordinate transforms; camera back-projection uses calibration matrices.Source: §III · p. 4
  • For vision-only proposals, inverse perspective mapping projects each detection’s bottom center into the global frame.The paper notes that single-camera projection is ambiguous because of depth ambiguity.Source: §III · p. 4, Figure 3 · p. 5
  • With LiDAR, points inside a detection box are flattened and RANSAC-fitted to two vehicle sides to estimate the vehicle center and dimensions.The method fits one side with a 0.3 inlier ratio, then the other with 0.4.Source: §III · p. 5, Figure 3 · p. 5
  • Vision-only fusion sorts proposals by detector confidence and groups nearby global projections using a 1 m distance threshold.Each group’s projection is replaced by the mean of its members’ projections.Source: §III · p. 5
  • For LiDAR fusion, back-projected point sets inside proposal boxes determine whether detections match, and their union can improve the fused proposal’s projection.The figure marks points common to both detections in green and the remaining points in red.Source: §III · p. 5, Figure 4 · p. 5

IV. M3OT FRAMEWORK

Fused object proposals are passed to the MDP framework, which the paper extends to account for multiple sensors and modalities.

IV. M3OT FRAMEWORK

  • The M3OT framework feeds fused object proposals into the MDP tracking stage.The MDP formulation introduced in is a building block of M3OT.Source: §IV · p. 5
  • Extending the MDP framework to handle multiple sensors and modalities is described as a non-trivial task.The section explains the modifications made for these objectives.Source: §IV · p. 5
  • The paper treats the MDP framework introduced in as a crucial component of M3OT’s multi-sensor extensions.Source: §IV · p. 5

A. Markov Decision Process

The framework retains an MDP structure for target lifetimes while adapting its policies to multi-camera tracking. Per-view classifiers and appearance cues govern target transitions, with global-position gating helping constrain tracking decisions.

A. Markov Decision Process

  • The MDP partitions each target’s lifetime into Active, Tracked, Lost, and terminal Inactive states, with transitions representing detection, visibility loss, recovery, and termination.States encode target information such as appearance, location, size, and history.Source: A. Markov Decision Process · p. 6
  • Seven deterministic actions define transitions between state subspaces, while the reward function is learned from ground-truth trajectories through inverse reinforcement learning.The MDP retains the original states, actions, and transition function while changing its reward function.Source: A. Markov Decision Process · p. 6

B. Policy

  • For Active targets, separate per-camera SVMs classify detections as Tracked or Inactive using normalized image coordinates, box dimensions, and detection score.A positive reward from any fused proposal can initialize a tracker; false positives may still enter Tracked and are handled in later states.Source: B. Policy · p. 6
  • The appearance model uses image-patch templates collected across camera views, providing view-specific appearance history for decisions when a target is Lost.Templates in a new view can be initialized from proposals fused with detections at the tracked location in an adjacent view.Source: B. Policy · pp. 6–7
  • Tracked targets remain tracked when optical-flow and bounding-box-overlap cues pass thresholds in a camera view and their global position passes a gate; otherwise they become Lost.The gate checks whether the current global position falls within a window around the target’s last known global position.Source: B. Policy · p. 7
  • Data association adds lateral and longitudinal global-coordinate displacements to the retained feature representation, while dense optical flow and median forward-backward error assess template-tracking stability.Matches above the forward-backward error threshold are filtered before predicting a new target box.Source: B. Policy · pp. 7–8

C. Multi-Object Tracking with MDPs

The online tracker processes tracked targets before lost targets, then initializes tracks from proposals not already covered. Lost-target associations use classifier scores and a Hungarian assignment.

C. Multi-Object Tracking with MDPs

  • Each frame first advances tracked targets according to the learned policy, then compares lost targets with uncovered detections for data association.Detections overlapping tracked-target boxes are suppressed to reduce ambiguity, and tracked targets have priority over lost targets.Source: C. Multi-Object Tracking with MDPs · pp. 8–9
  • After association, each uncovered proposal initializes a new target MDP, and a positive policy action moves it to Tracked with templates initialized in observed camera views.Proposals not accepted for tracking are transferred to Inactive.Source: C. Multi-Object Tracking with MDPs · p. 9
  • The tracker uses binary-classifier similarity scores in the Hungarian algorithm to assign detections to lost targets, returning linked targets to Tracked and leaving unmatched ones Lost.Pairwise similarity is computed only for detections not covered by tracked targets.Source: C. Multi-Object Tracking with MDPs · p. 9

V. EXPERIMENTAL ANALYSIS

The evaluation uses full-surround camera coverage and long multi-camera/LiDAR sequences, with one sequence for training and the remaining sequences for testing. Tracking is assessed with 3D MOT metrics and global-coordinate visualizations.

V. EXPERIMENTAL ANALYSIS

  • The testbed uses eight outward-facing RGB cameras with full-surround coverage and overlap between adjacent views, alongside full-surround Radar and LiDAR fields of view.The experiments also consider subsets of the vision sensors to examine modularity.Source: §V · p. 9
  • Four annotated sequences, each 3–4 minutes long, provide multi-camera video and LiDAR data; one sequence trains the tracker and the other three form the test set.Each vehicle has the same track ID across all camera views, and results are reported over the entire test set.Source: §V · p. 9
  • Evaluation uses MOTA, MOTP, Mostly Track, Mostly Lost, and ID Switches, supplemented by top-down track visualizations in coordinates centered on the ego-vehicle.Mostly Track counts trajectories covered for at least 80%, while Mostly Lost counts those covered for less than 20%.Source: §V · p. 9

A. Experimenting with Number of Cameras

Across camera configurations, the tracker mostly tracks more than 70% of targets in roughly three-minute test sequences while mostly losing only a few. Reported results are consistent within each configuration’s field of view, including configurations without camera overlap.

A. Experimenting with Number of Cameras

  • Track results remain consistent within the field of view across camera configurations, even when the cameras have no overlap.The tested configurations use 2, 3, 4, 6, and 8 cameras; each is scored only on ground-truth tracks visible in that configuration.Source: A. Experimenting with Number of Cameras · p. 9
  • More than 70% of targets are mostly tracked, while only a few are mostly lost, in test sequences about three minutes long.The paper reports this as evidence of long-term target tracking; configurations with overlapping fields of view perform especially well on MT and ML.Source: A. Experimenting with Number of Cameras · p. 9

B. Effect of Projection Scheme

The section compares two projection schemes and reports more stable, less fragmented tracks with LiDAR-based projection than with IPM.

B. Effect of Projection Scheme

C. Effect of Fusion Scheme

LiDAR point-cloud fusion is more reliable than distance-based fusion, while the latter can fail to associate proposals and shorten tracks; its drawbacks worsen with IPM projection.

C. Effect of Fusion Scheme

  • LiDAR point-cloud fusion tracks objects longer across camera views and more accurately than distance-based fusion.Source: C. Effect of Fusion Scheme · p. 10
  • Distance-based fusion can fail to associate proposals, leaving no templates for new camera views and cutting tracks short when targets leave the current view.Source: C. Effect of Fusion Scheme · p. 10
  • Distance-fusion drawbacks are exacerbated with IPM projection, which produces a large MOTA drop for a purely vision-based system.The text notes that this drop is expected without LiDAR sensors.Source: C. Effect of Fusion Scheme · pp. 10–11
  • Without LiDAR, half the targets are still tracked for most of their lifetime, while a quarter are mostly lost.Source: C. Effect of Fusion Scheme · p. 11

D. Effect of using Different Vehicle Detectors

Tracking results are relatively unaffected by which of three vehicle detectors supplies proposals, despite their suboptimal performance and differing ROC curves.

D. Effect of using Different Vehicle Detectors

E. Effect of Global Position based Features

Adding features {φ13, φ14} for Lost-state data association improves all reported metrics by helping reject proposals far from a target’s last known location.

E. Effect of Global Position based Features

VI. CONCLUDING REMARKS

The M3OT framework extends tracking-by-detection to real-world full-surround tracking and supports varied camera configurations, with or without LiDAR. Highway tests support its modularity and ability to track objects for long durations.

VI. CONCLUDING REMARKS

  • M3OT extends 2D tracking-by-detection to track objects in the real world using camera and LiDAR data.Its modular design accommodates camera configurations with varying fields of view, with or without LiDAR.Source: §VI · p. 11
  • An efficient, fast early-fusion scheme handles object proposals from different sensors within a calibrated camera array.The framework is tested on naturalistic full-surround vision and LiDAR highway data.Source: §VI · p. 11
  • Highway tests support the framework’s modularity and its ability to track objects for long durations.The tests examine different camera setups, fusion schemes, and 2D-to-3D projection schemes qualitatively and quantitatively.Source: §VI · p. 11
  • The authors propose using M3OT to assess camera setups and make modifications for optimum coverage from vision and range sensors.The framework’s modularity supports tests of different camera setups.Source: §VI · p. 11
Loading 1802.08755v4…