Source-linked AI summary
Mono-Camera 3D Multi-Object Tracking Using Deep Learning Detections and PMBM Filtering
Samuel Scheidegger, Joachim Benjaminsson, Emil Rosenberg, Amrit Krishnan, Karl Granstrom
TL;DR
Monocular cameras lack direct distance measurements, limiting tracking to the image plane despite the need for world-coordinate vehicle information. The paper combines a distance-estimating deep detector with PMBM filtering to produce 3D multi-object tracks from camera images, achieving accurate tracking on KITTI with efficient runtime.
Problem
Monocular cameras observe objects in 2D and lack direct distance information, while automated driving requires vehicle tracking in world coordinates.
Method
A deep neural network estimates 2D detections and object distance, and a PMBM filter processes detection sequences into multi-object estimates.
Results
The algorithm achieves accurate 3D tracking on KITTI, handles overlapping objects through data association, and runs at an average total runtime of 52 ms.
Takeaways & Limitations
A CNN followed by PMBM filtering can track multiple objects in 3D using a single camera image stream.
Abstract
from arXiv · showhide
Monocular cameras are one of the most commonly used sensors in the automotive industry for autonomous vehicles. One major drawback using a monocular camera is that it only makes observations in the two dimensional image plane and can not directly measure the distance to objects. In this paper, we aim at filling this gap by developing a multi-object tracking algorithm that takes an image as input and produces trajectories of detected objects in a world coordinate system. We solve this by using a deep neural network trained to detect and estimate the distance to objects from a single input image. The detections from a sequence of images are fed in to a state-of-the art Poisson multi-Bernoulli mixture tracking filter. The combination of the learned detector and the PMBM filter results in an algorithm that achieves 3D tracking using only mono-camera images as input. The performance of the algorithm is evaluated both in 3D world coordinates, and 2D image coordinates, using the publicly available KITTI object tracking dataset. The algorithm shows the ability to accurately track objects, correctly handle data associations, even when there is a big overlap of the objects in the image, and is one of the top performing algorithms on the KITTI object tracking benchmark. Furthermore, the algorithm is efficient, running on average close to 20 frames per second.
I. INTRODUCTION
The paper addresses multi-vehicle 3D tracking from monocular images, overcoming image-plane-only observations by combining learned distance-aware detections with PMBM filtering.
- Environment perception must model static surroundings and dynamic vehicles for automated driving.
- Monocular cameras lose distance information, so image-plane tracking cannot provide the world-coordinate relative pose needed for decision making and control.
- Prior monocular-camera tracking was restricted to the image plane, whereas this paper outputs multi-vehicle estimates in world coordinates.
- The paper evaluates the proposed algorithm on the publicly available KITTI tracking dataset.
- The method combines a deep neural network that estimates 2D boxes and camera distance with a PMBM filter that processes detections.
II. PROBLEM FORMULATION AND ALGORITHM OVERVIEW
The algorithm converts forward-camera image sequences into estimates of a time-varying set of vehicle states by pairing distance-aware CNN detections with recursive PMBM tracking.
- Each 3D vehicle state contains world position, velocity, and image bounding-box width and height.
- The task is to estimate the number and states of nearby vehicles from a sequence of forward-looking camera images.
- The detection module processes each image into a set of CNN-based object detections.
- The tracking module uses prediction, update, and extraction to recursively estimate a multi-object set density.
- The PMBM prediction uses a Markovian motion model for persistent objects together with a Poisson point-process birth model.
- The measurement model represents noisy detections, detection uncertainty, and PPP clutter during the Bayes update.
III. OBJECT DETECTION
The detector predicts object classes, image bounding boxes, and camera distances from each image, enabling the monocular camera to supply range-bearing information for 3D tracking.
- Each detection contains a 2D bounding box and the distance from the camera to the detected object.
- Bounding-box geometry and estimated distance allow the camera to be transformed into a range/bearing sensor for world-coordinate tracking.
- The network uses a feature extractor followed by three parallel output headers for classification, bounding-box regression, and distance estimation.
- The detector’s outputs comprise classification scores, bounding boxes, and distances.
- The feature extractor is pretrained on ImageNet, while the full network is fine-tuned with annotated KITTI object labels.
IV. OBJECT TRACKING
The tracking stage applies a PMBM filter to successive detection sets, estimating the multi-object density and extracting object-set trajectories.
- A PMBM filter associates objects across consecutive frames and filters neural-network detections.
- Both vehicles and image detections are modeled as random finite sets within the tracking formulation.
- The tracking module processes detection-set sequences to estimate the true set of objects at each time step.
A. RFS background
Random finite sets model uncertain collections of objects and measurements. PPPs represent independent, Poisson-cardinality sets, while Bernoulli and multi-Bernoulli mixtures represent object existence, state uncertainty, and alternative association hypotheses.
- RFS fundamentals: A PPP models a set whose cardinality is Poisson distributed and whose elements are independent and identically distributed.It is parameterized by an intensity function with a Poisson rate and spatial distribution.
- RFS fundamentals: PPPs model object births, undetected objects, and clutter measurements in the tracking system.
- RFS fundamentals: A Bernoulli RFS contains either no element or one element, with probability r and state density f(x) when present.
- RFS fundamentals: Bernoulli RFSs jointly represent an object's existence probability and uncertainty in its state.
- Multi-object representations: An MB is a disjoint union of independent Bernoulli RFSs, whereas an MBM is a weighted sum of MB densities representing different data-association sequences.
B. Standard models
The standard models describe Gaussian motion and measurement assumptions while explicitly representing uncertainty about whether measurements come from objects, clutter, or new objects. Data associations are constrained by independent-object and point-object assumptions.
- Model assumptions: The standard measurement and motion models are presented under Gaussian assumptions.
- Measurement model: At each time step, object-generated measurements, an index set, and clutter measurements form the measurement model; clutter follows a PPP with uniform spatial distribution.
- Measurement model: An object is correctly detected with probability pD, producing either no measurement or one measurement with likelihood pDφz(x).
- Measurement model: The point-object assumption assigns zero likelihood when multiple measurements are associated with one object.
- Data association: Each data association assigns measurements to clutter, new objects, or existing objects and partitions the combined measurement-object indices into non-empty disjoint cells.
- Data association: Each association cell contains at most one object index and one measurement index, because independent-object and point-object violations have zero likelihood.
- Data association: Unknown measurement origins require the update to handle uncertainty about both clutter and which object generated each measurement.
2) Standard dynamic model:
The dynamic model lets existing detected and undetected objects survive, evolve independently under a Gaussian constant-velocity transition, and allows new objects to enter through a Poisson birth process.
- Standard dynamic model: Existing detected and undetected objects survive from k to k + 1 with probability pS.
- Standard dynamic model: Objects evolve independently according to a Gaussian transition density using a constant-velocity motion model.
- Standard dynamic model: New objects appear independently through a Poisson point process with birth intensity Db_k+1(x).
C. PMBM filter
The PMBM filter represents undetected objects with a PPP and previously detected objects with an MBM. Its conjugate-prior structure lets prediction and update propagate the multi-object density through a finite set of parameters.
- PMBM representation: The PMBM density combines a PPP for currently existing but undetected objects with an MBM for objects detected at least once.
- PMBM representation: The object set is partitioned into detected objects Xd and undetected objects Xu.
- PMBM parameters: The PPP component is parameterized by the undetected-object intensity Du(·).
- PMBM parameters: Each MBM component corresponds to a global data-association hypothesis and has weight wj.
- PMBM parameters: Within each MBM component, Bernoulli densities represent potentially detected objects through existence probabilities and state PDFs.
- Filter operation: As an MOT conjugate prior, PMBM prediction and update preserve the PMBM form for the standard point-object models.
- Gaussian implementation: A non-normalized Gaussian-mixture birth intensity yields a Gaussian-mixture undetected intensity and Gaussian Bernoulli densities, with UKF prediction and update used for Gaussian parameters.
2) Update:
The PMBM update incorporates a measurement set under a standard measurement model and retains a PMBM-form updated density. Gaussian-mixture components can be updated using an unscented Kalman filter.
- A prior PMBM density is updated using a measurement set and the standard measurement model.
- The updated density remains a PMBM density after incorporating the measurements.
- Gaussian-mixture and Gaussian-density updates are computed using an unscented Kalman filter update.
3) Extraction:
The extraction procedure selects the most probable hypothesis, then returns sufficiently probable Bernoulli components as object estimates using their expected states.
- The hypothesis with the highest probability is selected to extract object estimates.
- Object estimates are extracted from the corresponding multi-Bernoulli component.
- Bernoulli components whose existence probability exceeds threshold τ contribute their expected object state to the estimate set.
V. EXPERIMENTAL RESULTS
The method is evaluated on KITTI in both 2D and 3D, with qualitative tracking across varied vehicle motions and quantitative results showing robust associations, improved F1 over raw detections, efficient runtime, and strong benchmark performance.
- Dataset and evaluation: KITTI provides 21 training sequences and 29 testing sequences collected from sensors mounted on a moving car.
- Dataset and evaluation: 3D evaluation uses Euclidean distance within 3 m, while 2D evaluation uses image-plane bounding-box IOU of at least 50%.
- Qualitative results: The tracker handles vehicles moving with or against the ego-vehicle and vehicles making sharp turns, including large bounding-box overlaps.
- Validation results: The MOT algorithm achieves higher F1 than raw CNN detections by trading lower precision for higher recall.
- Runtime: 52 ms per frame comprises 38 ms for detection and 14 ms for tracking on the reported hardware.
- KITTI benchmark: The method ranked 3rd in MOTA on the published KITTI benchmark, with faster runtime and fewer identity switches than two higher-MOTA methods.
- Conclusion: The paper concludes that CNN detections followed by PMBM filtering can track multiple objects in 3D from a single camera image.