PaperlayerView exact revision on arXiv

Source-linked AI summary

Robust Multi-Modality Multi-Object Tracking

Wenwei Zhang, Hui Zhou, Shuyang Sun, Zhe Wang, Jianping Shi, Chen Change Loy

arXiv:1909.03850v1cs.CV

TL;DR

Autonomous-driving MOT needs reliable perception despite sensor failures and accurate association across modalities, but existing systems either depend tightly on one sensor or underuse fused information. mmMOT independently extracts modality features, learns cross-modality adjacency end-to-end, and introduces deep point-cloud representations into association. On KITTI, it achieves state-of-the-art online results, with the same model remaining competitive under single-modality failure conditions.

  • Problem

    Existing multi-sensor MOT methods can lack reliability through tight dependence on one input source or lose accuracy by fusing sensor results only in post processing.

  • Method

    mmMOT independently extracts features from each sensor, robustly fuses available modalities, jointly optimizes feature extractors and adjacency estimation end-to-end, and uses deep point-cloud representations for association.

  • Results

    State-of-the-art online performance is reported on KITTI tracking, while the same model's single-modality sensor-failure results are only 0.28% worse.

  • Takeaways & Limitations

    The framework supports multi-modality tracking while retaining operation from individual sensors, including under sensor malfunctioning.

  • Takeaways & Limitations

    Failure cases remain under long range, poor illumination, occlusion, and partial observation, including detector-induced false negatives and identity switches.

Abstract

from arXiv · show

Multi-sensor perception is crucial to ensure the reliability and accuracy in autonomous driving system, while multi-object tracking (MOT) improves that by tracing sequential movement of dynamic objects. Most current approaches for multi-sensor multi-object tracking are either lack of reliability by tightly relying on a single input source (e.g., center camera), or not accurate enough by fusing the results from multiple sensors in post processing without fully exploiting the inherent information. In this study, we design a generic sensor-agnostic multi-modality MOT framework (mmMOT), where each modality (i.e., sensors) is capable of performing its role independently to preserve reliability, and further improving its accuracy through a novel multi-modality fusion module. Our mmMOT can be trained in an end-to-end manner, enables joint optimization for the base feature extractor of each modality and an adjacency estimator for cross modality. Our mmMOT also makes the first attempt to encode deep representation of point cloud in data association process in MOT. We conduct extensive experiments to evaluate the effectiveness of the proposed framework on the challenging KITTI benchmark and report state-of-the-art performance. Code and models are available at https://github.com/ZwwWayne/mmMOT.

1. Introduction

mmMOT targets both reliability and accuracy in autonomous-driving MOT by allowing independent sensor operation and learning attention-guided multi-modality fusion. It reports state-of-the-art KITTI results while remaining competitive under single-modality sensor failure.

  • Motivation: mmMOT addresses reliability and accuracy by combining independent multi-sensor clues with sequential multi-object tracking information.Independent clues help prevent failures, while tracking and multi-sensor fusion reinforce perception.
  • Motivation: Camera-dependent MOT methods can fail when the camera is unavailable, while hand-crafted feature fusion limits accuracy.Existing approaches either bind association to image representations or rely on hand-crafted features with limited expression ability.
  • Framework: mmMOT extracts features independently from each sensor, fuses them, and uses an adjacency estimator capable of inference from each modality.The framework is designed to remain operational when one sensor is unavailable.
  • Contributions: The framework jointly optimizes multi-modality feature extractors and a cross-modality adjacency estimator through end-to-end training.It also introduces deep point-cloud representation into MOT data association.
  • Results: On KITTI tracking, mmMOT achieves state-of-the-art online results using image and point cloud, while single-modality results are only 0.28% worse.The same model is evaluated under sensor-failure conditions without additional modifications.

2. Related Work

Prior MOT work commonly uses tracking-by-detection and diverse association methods, but many systems are not end-to-end or are tightly centered on camera data. mmMOT addresses these gaps through adjacency-matrix learning and independently extracted image and point-cloud features.

  • Multi-Object Tracking Framework: Tracking-by-detection systems detect objects first and link them into trajectories through data association.Prior association strategies include min-cost flow, Markov decision processes, partial filtering, Hungarian assignment, and graph cut.
  • Multi-Object Tracking Framework: Many existing association methods are not trained end-to-end, leaving parameters such as cost weights heuristic.End-to-end min-cost-flow approaches exist, but the cited frameworks are not designed for cross-modality.
  • Deep Representation of Point Cloud: Adjacency matrix learning is used to support end-to-end cross-modality inference, while deep point-cloud features are introduced into data association.The framework extends feature learning beyond traditional point-cloud distance, grid, and hand-crafted representations.
  • Correlation Features: Image-centric methods mainly use deep image-patch features, with hand-crafted geometric, shape, bounding-box, or temporal features as auxiliaries.These approaches remain dependent on camera input.
  • Correlation Features: mmMOT extracts deep image and point-cloud features independently, treats sensors as decoupled inputs, and applies attention-guided fusion.This design is contrasted with camera-dependent prior methods.

3. Multi-Modality Multi-Object Tracking

mmMOT follows tracking-by-detection and combines object detection, modality-specific feature extraction, adjacency estimation, and min-cost-flow optimization. Its fusion module supports both combined modalities and operation with a single remaining sensor.

  • Framework Overview: The framework contains an object detector, feature extractor, adjacency estimator, and min-cost-flow optimizer.An arbitrary detector localizes objects of interest; PointPillar is used in the described implementation.
  • Framework Overview: mmMOT extracts modality-specific features independently before fusing them for adjacency estimation.This preserves the ability to track using each sensor separately while exploiting multi-sensor information.
  • Framework Overview: The robust fusion module can fuse multiple modalities or pass through original features from a single modality.This supports tracking when sensor malfunction leaves only one input modality available.

𝜃"𝜃"'()*Linear Programming𝑌'()*

mmMOT extracts modality-specific features, fuses them robustly, and predicts detection associations for min-cost-flow tracking. Its design preserves single-modality operation while supporting joint multi-sensor inference.

  • Framework: mmMOT uses feature extractors, a fusion module, an adjacency estimator, and a min-cost-flow optimizer for tracking-by-detection.The optimizer finds the optimal flow solution from predicted scores.
  • Problem Formulation: Detections in consecutive frames are represented through binary variables for true positives, links, new trajectories, and trajectory ends.These variables are flattened into Y, which encodes network-flow edge states.
  • Single Modality Feature Extractor: Image and point-cloud features are extracted independently, with deep point-cloud representations produced from 3D coordinates using a modified PointNet.The point-cloud extractor uses a global feature branch and average pooling for tracking.
  • Robust Multi-Modality Fusion Module: The robust fusion module outputs both original single modalities and a fused modality, allowing the adjacency estimator to process each modality independently or together.The outputs are concatenated along the batch dimension into a tensor of size (S + 1) × D × (N + M).
  • Robust Multi-Modality Fusion Module: Fusion modules concatenate features, add reprojected features, or adaptively weight sensors with attention.The additive design makes fused correlation features more similar to single-sensor features, supporting modality-agnostic adjacency estimation.
  • Deep Adjacency Matrix Learning: Correlation features are processed by point-wise convolution to predict adjacency, while linear constraints enforce valid trajectory-flow relationships.Each modality receives supervision for the same adjacency target, enabling joint optimization of modality extractors and the affinity estimator.

4. Experiments

Experiments on KITTI evaluate mmMOT’s modality choices, robust fusion, ranking and correlation mechanisms, benchmark performance, and failure cases. The results show that point-cloud features and robust attention-guided fusion improve tracking while retaining single-modality operation, with remaining errors linked to detection and challenging visual conditions.

  • Experimental Setup: The KITTI evaluation uses 10 training sequences and 11 validation sequences, with 3975 and 3945 frames respectively.
  • Experimental Setup: 3D and 2D box annotations with persistent vehicle IDs provide ground-truth associations, while detections are matched when IoU exceeds 0.5.
  • Modality Ablation: Point-cloud methods outperform the image baseline, and frustum-based point clouds remain competitive with bounding-box point clouds.
  • Fusion Ablation: MOTA 77.54 is achieved by ensembling image and point-cloud models, while attention-based robust fusion module C surpasses this ensemble and all single-sensor baselines.
  • Further Analysis: The ranking mechanism improves MOTA by at least 0.2, while adding softmax output improves MOTA by about 2.5.
  • Further Analysis: Absolute-subtraction correlation improves MOTA by 1, and additive softmax ranking improves it by another 1 while reducing ID switches to 13.
  • KITTI Results: Under both multimodal and simulated sensor-failure conditions, mmMOT surpasses published online methods on KITTI MOTA; modality fusion also yields 184 fewer ID switches than MOTBeyondPixels.
  • Failure Case Analysis: Failures arise from distant-object detector errors, illumination, occlusion, and partial observation, with temporal association or multimodal detection suggested as future remedies.

5. Conclusion

mmMOT is an end-to-end multi-object tracking framework designed to preserve independent sensor reliability while exploiting multi-modality information. It learns cross-modality inference and introduces deep LiDAR point-cloud representations into data association.

  • mmMOT uses a deep end-to-end network to avoid single-sensor instability while retaining multi-modality effectiveness.
  • Adjacency matrix learning enables the framework to infer from arbitrary modalities simultaneously.
  • mmMOT introduces deep LiDAR point-cloud representations into the data association problem.
  • The framework is presented as important for safe autonomous driving and robustness against sensor malfunctioning.

A1. Model Details

The model details combine multi-level VGG-Net image features with a PointNet-based point-cloud representation and a detector-driven frustum design. The best KITTI model uses VGG-16 with Batch Normalization and RRC-Net detections.

  • The skip pooling layer aggregates multi-level VGG-Net outputs using global average pooling and channel re-scaling before concatenation.Outputs from max-pooling layers have 64, 128, 256, and 512 channels, then are re-scaled to 128 channels.
  • The KITTI best model uses ImageNet-1k-pretrained VGG-16 with Batch Normalization as its image feature-extractor backbone.
  • RRC-Net provides the 2D detections, and point-cloud data within each detection frustum is used for fusion.The passage states that RRC-Net has higher recall and precision than PointPillar for detection.

A2. Failure Analysis

The failure analysis examines identity switches across occlusion, distance, and bounding-box size. Fusion improves robustness under difficult occlusion and distance conditions, while sparse or noisy point-cloud evidence remains challenging.

  • The analysis measures identity switches by occlusion condition, distance from the ego car, and bounding-box size.False negatives and false positives are attributed mainly to the detector rather than the association process.
  • Fusion makes the tracker more robust under difficult occlusion and distance conditions.
  • Most identity switches occur with occlusion, which makes objects harder to recognize or distinguish from partial observations.
  • Point-cloud-only tracking suffers more occlusion-related errors because frustum points become noisy, while small or distant objects provide insufficient point representation.Image patches can still be interpolated to 224 × 224 pixels for these objects.
  • Occlusion levels 0 through 3 represent objects that are not, moderately, highly, or extremely occluded and truncated in the image.
Loading 1909.03850v1…