Source-linked AI summary
Beyond Pixels: Leveraging Geometry and Shape Cues for Online Multi-Object Tracking
Sarthak Sharma, Junaid Ahmed Ansari, J. Krishna Murthy, K. Madhava Krishna
TL;DR
Accurate multi-object tracking supports urban-scene tasks, while the paper targets a simple, fast, accurate, and robust solution. It designs monocular geometry, shape, and pose-based pairwise costs and reports state-of-the-art KITTI performance, including with two-frame Hungarian association.
Problem
Accurate multi-object tracking is important for urban-scene tasks including navigation, planning, localization, and traffic behavior analysis.
Method
The paper designs pairwise association costs from road-scene geometry, monocular 3D cues, object shape and pose, motion, and appearance information.
Results
The proposed costs achieve state-of-the-art KITTI performance, exceeding 90% tracking accuracy on training data and 84% on the online server, while improving across detectors and association frameworks.
Takeaways & Limitations
Monocular 3D, pose, and shape cues provide simple pairwise costs for tracking-by-detection and improve tracking in challenging road scenes.
Abstract
from arXiv · showhide
This paper introduces geometry and object shape and pose costs for multi-object tracking in urban driving scenarios. Using images from a monocular camera alone, we devise pairwise costs for object tracks, based on several 3D cues such as object pose, shape, and motion. The proposed costs are agnostic to the data association method and can be incorporated into any optimization framework to output the pairwise data associations. These costs are easy to implement, can be computed in real-time, and complement each other to account for possible errors in a tracking-by-detection framework. We perform an extensive analysis of the designed costs and empirically demonstrate consistent improvement over the state-of-the-art under varying conditions that employ a range of object detectors, exhibit a variety in camera and object motions, and, more importantly, are not reliant on the choice of the association framework. We also show that, by using the simplest of associations frameworks (two-frame Hungarian assignment), we surpass the state-of-the-art in multi-object-tracking on road scenes. More qualitative and quantitative results can be found at the following URL: https://junaidcs032.github.io/Geometry_ObjectShape_MOT/.
I. INTRODUCTION
The paper targets fast, accurate online MOT in urban road scenes, where data association is difficult and existing costs often miss intuitive 3D information. It introduces geometry-, appearance-, shape-, and pose-based pairwise costs and reports strong performance across association frameworks and detectors.
- The approach is designed to be intuitive, easy to compute, and robust across varied camera and target motions in urban road scenes.
- Data association is the hardest tracking-by-detection step because missing, spurious, repeated detections, occlusions, and interactions confound matching.
- The proposed costs exploit road-scene geometry to capture 3D and appearance cues, including novel 3D shape and pose similarity.
- Projected 3D boxes create gated search areas that reduce candidate regions and pairwise costs, while 3D volume overlap improves association with Hungarian matching.
- The method achieves state-of-the-art KITTI performance with simple two-frame Hungarian association and remains effective across detectors and association frameworks.
A. Global Tracking
The tracking formulation assigns framewise detections to target trajectories using pairwise costs, with online, near-online, and global association frameworks solving the resulting matching problem. Existing frameworks commonly rely on handcrafted costs that do not exploit readily available road-scene 3D information.
- A. Global Tracking: Global trackers assume detections from all frames are available and commonly formulate association as a min-cost network flow problem.
- A. Global Tracking: Online trackers use only current and past detections, often solving bipartite matching with the Hungarian algorithm.
- A. Global Tracking: Common handcrafted costs are sophisticated yet often fail to capture 3D information readily available in road scenes.
- A. Global Tracking: Each trajectory contains bounding boxes from different frames belonging to one target, while the formulation assumes one trajectory per target and unassigned spurious detections.
- A. Global Tracking: All association frameworks rely on pairwise costs to define affinity between detection pairs before optimizing trajectory assignments.
IV. GEOMETRY AND OBJECT SHAPE COSTS
The paper designs efficient, intuitive pairwise costs for urban MOT by exploiting road-scene geometry and combining complementary monocular 3D cues with appearance. A novel shape-and-pose cost also helps reject false detections without large computational overhead.
- Road-scene geometry provides 3D cues that can be converted into efficient pairwise costs for urban driving scenarios.
- The proposed cost set complements standard location, motion, overlap, and appearance terms with information from monocular 3D localization.
- A novel 3D shape-and-pose cost improves association and helps discard false detections without large computational overhead.
A. System Setup
The system uses a monocular, vehicle-mounted camera and combines feature-based odometry with single-image estimates of object geometry. Propagated 3D boxes are compared with later detections through 2D projections and 3D volume intersections.
- A. System Setup: The setup assumes a monocular camera mounted on a moving car and targets moving on the road plane.
- A. System Setup: A CNN localizes object keypoints in 2D, supporting estimation of 3D shape and pose from a single image.
- A. System Setup: The 3D-2D cost compares later-frame detections with projections of uncertainty-aware propagated boxes, whereas the 3D-3D cost compares their 3D box intersections.
- A. System Setup: Object shape is represented as a mean category shape plus a linear combination of deformation basis vectors.
- A. System Setup: The shape vector uses deformation coefficients, typically with B = 5, while pose is represented by an axis-angle vector and dimensions are also estimated.
B. 3D-2D Cost
The 3D-2D cost backprojects detections onto the road plane, transports the resulting 3D location across estimated camera motion, and compares the projected target region with a candidate detection. Uncertainty from localization, detector confidence, and target dimensions shapes the comparison and reduces unnecessary pair evaluations.
- 3D backprojection: The method backprojects a detection’s bottom-center image point through the road plane to estimate its 3D location and depth.The estimate uses the camera height, intrinsic matrix, and the road-plane assumption.
- Uncertainty modeling: Uncertainty in the 3D location is estimated by linearizing backprojection and modeling detector confidence as an isotropic 2D Gaussian.The uncertainty region is then expanded anisotropically using estimated target dimensions.
- Pairwise comparison: The estimated 3D target region is transported using camera motion, projected into the next image, and compared with a candidate detection.Feature-based odometry supplies rough frame-to-frame motion, while projection and rigid-body transformation define the expected image region.
- Pairwise comparison: The cost measures weighted overlap between the expected 2D target region and the candidate detection in the next frame.It is evaluated only for selected candidate detections, significantly reducing pairwise comparisons.
C. 3D-3D cost
The 3D-3D cost replaces image-space overlap with overlap between reconstructed 3D target volumes, addressing confounding cases caused by 2D projection. Road-scene geometry makes this comparison faster by reducing it to overlap on the ground-plane dimensions.
- 3D overlap: The 3D-3D cost measures overlap in 3D rather than image space to mitigate confounding cases caused by 2D overlap.Candidate detections are backprojected through the road plane and compared with the transformed 3D volume from the earlier frame.
- Road-plane simplification: Because objects lie on the road plane and have nearly constant heights, evaluating overlap in the XZ plane is sufficient.Similar overlap in the Y direction enables this simplification.
- Descriptor cue: The weighted hourglass descriptor captures dissimilarity between detections by combining features from the network’s activation maps.The descriptor is shown as a weighted feature combination.
D. Appearance Cost
The appearance cost uses a stacked-hourglass network descriptor built from weighted activation maps to compare detections. Similarity is computed by the L2 distance between descriptors extracted from the image patches inside their bounding boxes.
- Descriptor construction: A stacked-hourglass CNN provides discriminative features and keypoint evidence for each detection.The paper combines activation maps from the network’s output layers into a detection descriptor.
- Appearance comparison: The appearance cost uses the L2 norm between descriptors computed from the image patches inside two detection bounding boxes.The resulting distance is normalized by ηapp.
E. Shape and Pose Cost
The shape-and-pose cost compares detections using single-image estimates of 3D object shape and pose. Shape is represented by deformation coefficients and pose by an axis-angle vector, within an overall cost whose weights are cross-validated.
- Cost design: The shape-and-pose cost uses single-image shape and pose estimates produced by the referenced reconstruction pipeline.The paper presents this as a novel cost for comparing target hypotheses.
- Shape representation: Shape is parameterized by deformation coefficients Λ, whose values represent different 3D object-instance shapes.Varying coefficients can encode distinctions such as SUV-like versus sedan-like appearance.
- Pose representation: Pose is parameterized as an axis-angle vector ω for each detection.The shape and pose terms use normalization constants ηs and ηp.
- Cost integration: The overall pairwise cost is a weighted linear combination of the proposed costs, with weights selected by four-fold cross-validation on the training set.The experiments evaluate the framework on KITTI Tracking sequences using cross-validation splits.
- Evaluation: The framework reports a substantial performance boost over the state of the art using bipartite matching with the Hungarian algorithm.The evaluation covers challenging urban-driving sequences and uses a simple association framework.
B. Evaluation Metrics
The evaluation uses CLEAR MOT metrics on KITTI Tracking, comparing the proposed two-frame approach with published competitors and reporting both training and online test-set results.
- MOTA measures tracking accuracy, while MOTP measures object localization precision under the CLEAR MOT evaluation.
- The framework is evaluated with RRC and SubCNN detections after confidence thresholding and non-maxima suppression.
- The proposed approach is compared against current competitors using KITTI Tracking benchmark results and online evaluation.
- 91.4% MOTA and 89.84% MOTP are achieved on the KITTI training sequences using appearance, 3D, pose, and shape cues.The passage also notes ID switches and fragmentations, which are typical of online and two-frame greedy trackers.
- 6% is the reported margin over the next-best competitor on the KITTI test set, with 84.24% MOTA and 85.73% MOTP.The same evaluation reports 73.23% mostly tracked and 2.77% mostly lost.
F. Ablation Study
Ablation and qualitative analyses examine how individual cues and detector choices affect tracking, with combined monocular 3D, shape, and pose cues helping in challenging scenes.
- Ablation Study: Combining all proposed cues is crucial for overall performance across the RRC and SubCNN detectors.The analysis reports improvements in MOTA, ID switches, and fragmentations, including with the underperforming SubCNN detector.
- Shape and Pose Cues: Shape and pose cues are most discriminative at intersections and roundabouts, where vehicle poses and viewpoints vary heterogeneously.They are less discriminatory during typical lane driving, where vehicles align with the lane direction.
- Qualitative Results: The qualitative results show tracking across viewpoint variation, clutter, and changing relative motion between camera and target.
- Evaluation Tables: The study reports training-set results through Table I and test-set results through Table II, while Table III compares cue combinations and detectors.
- Qualitative Results: Cars are tracked through side occlusions, varying depths and poses, overtaking, and cases where only a very small visible portion remains.The examples include intersection traffic and cars tracked nearly to the horizon.
H. Summary of Results
Single-view monocular 3D cues, together with pose and shape cues, improve tracking performance on KITTI and support tracking through challenging occlusions, viewpoints, and relative motions. The gains are reported with both quantitative improvements and qualitative robustness, while the demonstrated tracker uses simple Hungarian association.
- Over 90% tracking accuracy is reported on the KITTI training evaluation, with significant improvements over current state-of-the-art methods.
- Over 84% tracking accuracy is reported on the KITTI online evaluation server, exceeding published approaches.
- 3D cues consistently improve appearance-only tracking across object detectors and reduce identity switches and fragmentations.The ablation results show repeatability across detector baselines.
- Pose and shape association cues improve baseline performance on a challenging subsequence.
- Monocular 3D cues with pose and shape cues achieve the best reported tracking performance on popular object-tracking training datasets.The paper presents these cues as a central contribution to multi-object tracking from single-view geometry.
- Qualitative results show accurate tracking under substantial occlusion, minimal visible object area, distant objects, viewpoint variation, clutter, and changing relative motion.The reported qualitative sequences include cars near the horizon and vehicles with only a very small visible portion.