Source-linked AI summary
FAMNet: Joint Learning of Feature, Affinity and Multi-dimensional Assignment for Online Multiple Object Tracking
Peng Chu, Haibin Ling
TL;DR
Tracking-by-detection MOT typically separates feature extraction, affinity estimation, and assignment, complicating design and parameter tuning. FAMNet unifies these stages in a differentiable network trained from assignment ground truth, supplements it with single-object tracking and target management, and reports promising performance across four benchmarks.
Problem
Tracking-by-detection MOT separates feature extraction, affinity estimation, and association, producing complex designs and extensive parameter tuning.
Method
FAMNet jointly learns features, higher-order affinity, and multi-dimensional assignment in a differentiable network, with single-object tracking and target management integrated into the tracking system.
Results
FAMNet achieves promising performance compared with published methods on MOT2015, MOT2017, KITTI-Car, and UA-DETRAC.
Takeaways & Limitations
The proposed system provides an end-to-end MOT architecture that learns features and higher-order affinity directly from assignment ground truth while incorporating tracking predictions and detector refinement.
Takeaways & Limitations
The online architecture uses K = 2, making three frames the minimum temporal span for motion-affinity calculation under a constant-velocity assumption.
Abstract
from arXiv · showhide
Data association-based multiple object tracking (MOT) involves multiple separated modules processed or optimized differently, which results in complex method design and requires non-trivial tuning of parameters. In this paper, we present an end-to-end model, named FAMNet, where Feature extraction, Affinity estimation and Multi-dimensional assignment are refined in a single network. All layers in FAMNet are designed differentiable thus can be optimized jointly to learn the discriminative features and higher-order affinity model for robust MOT, which is supervised by the loss directly from the assignment ground truth. We also integrate single object tracking technique and a dedicated target management scheme into the FAMNet-based tracking system to further recover false negatives and inhibit noisy target candidates generated by the external detector. The proposed method is evaluated on a diverse set of benchmarks including MOT2015, MOT2017, KITTI-Car and UA-DETRAC, and achieves promising performance on all of them in comparison with state-of-the-arts.
1. Introduction
FAMNet addresses the fragmented design of tracking-by-detection MOT by jointly refining feature representation, affinity estimation, and multi-dimensional assignment in one differentiable network. The tracking system further combines single-object tracking and target management, and is evaluated across pedestrian and vehicle benchmarks.
- Motivation: Tracking-by-detection separates feature extraction, affinity evaluation, and association, creating complex designs and extensive parameter tuning.These components are processed or optimized differently across tracking systems.
- Motivation: Prior unified architectures combine feature extraction and affinity learning, but retain association outside the network and require manually fabricated affinity samples.Such architectures do not directly train the full association process from assignment ground truth.
- FAMNet: FAMNet jointly refines feature representation, affinity modeling, and multi-dimensional assignment in a single differentiable deep network.The affinity sub-network estimates higher-order affinity, while the MDA sub-network obtains globally optimized assignments.
- FAMNet: The affinity sub-network fuses higher-order appearance and motion information, and the MDA sub-network uses a differentiable modified rank-1 tensor approximation power iteration.These designs adapt higher-order association to the deep-learning architecture.
- Tracking system: The system integrates single-object tracking and a dedicated CNN-based target management scheme to recover false negatives and suppress noisy detector candidates.Detections and tracking predictions are merged and selected through MDA, while bounding boxes are refined by the dedicated CNN.
- Evaluation: FAMNet is evaluated on MOT2015, MOT2017, KITTI-Car, and UA-DETRAC, with promising performance compared with published methods.The cited contribution passage reports evaluation across pedestrian and vehicle tracking challenge benchmarks.
2. Related Work
Prior MOT research commonly formulates data association using pairwise or higher-order affinity and optimization algorithms, while newer deep methods learn discriminative association cues. End-to-end approaches increasingly integrate parts of this process, but the related work positions FAMNet within this progression toward unified tracking.
- Traditional association: Traditional MOT methods solve data association with Hungarian, network-flow, or multiple-hypotheses algorithms over affinity estimation schemes.These approaches include methods that incorporate higher-order motion models through iterative assignment optimization.
- Higher-order association: Higher-order affinity supplies global and discriminative information unavailable to pairwise association, motivating multi-dimensional assignment formulations.Rank-1 tensor approximation has been used with power iteration to seek an assignment solution.
- Deep affinity learning: Deep MOT methods use neural networks for discriminative affinity estimation, including Re-ID features, long-term connections, and recurrent higher-order modeling.Several methods train these network components separately using affinity supervision fabricated from ground-truth associations.
- End-to-end MOT: Early end-to-end MOT work used recurrent networks to estimate candidate states or integrate motion affinity with bipartite association.The cited approaches demonstrate increasingly unified tracking architectures with differing treatment of explicit data association.
3. Overview
The FAMNet tracking pipeline formulates MOT as multi-dimensional assignment over candidate trajectories and processes frame batches through feature, affinity, and assignment components. Training compares assignments directly with ground truth, while tracking combines network assignments with single-object predictions under a three-frame online setting.
- Problem formulation: Each hypothesis trajectory is composed of candidates across K + 1 frames, with candidate status represented by variables such as image-frame center coordinates.The input contains K + 1 target-candidate sets, one for each frame.
- Problem formulation: MOT is formulated as finding a multi-dimensional association that maximizes overall affinity subject to assignment constraints.The association variables indicate which hypothesis trajectories are selected as true.
- Problem formulation: The multi-dimensional assignment tensor is decomposed into local assignments between candidates in adjacent frames and optimized through their tensor representation.The R1TA framework rewrites local assignment matrices as vectors and reshapes the affinity tensor for tensor products.
- Tracking pipeline: For each association batch, external detections and K + 1 image frames generate hypotheses whose candidate patches are processed by FAMNet.The network extracts candidate features, estimates affinities for hypothesis trajectories, and computes local assignments.
- Training and tracking: Assignment ground truth is directly compared with network output, allowing the loss to back-propagate through the feature and affinity sub-networks.During tracking, output assignments and single-object tracking predictions update target trajectories through target management.
- Architecture and setting: K = 2 is used so three frames provide the minimum temporal span for motion affinity under constant velocity while balancing computation cost and association depth.Association batches overlap by two frames.
4. FAMNet
FAMNet combines feature extraction, higher-order affinity estimation, and multi-dimensional assignment in a differentiable tracking network. Its affinity module aligns candidate features around anchor candidates, while differentiable assignment layers estimate globally consistent associations.
- Affinity Sub-Network: The affinity sub-network outputs an affinity tensor for hypothesis trajectories by combining candidate features and trajectory structure.The final trajectory affinity sums pair-wise and long-term affinity components.
- Affinity Sub-Network: The feature sub-network extracts spatially aligned candidate features from all K + 1 frames, using the middle frame as the anchor frame.For K = 2, the anchor frame is k = 1.
- Affinity Sub-Network: Candidate features are centered on the anchor candidate and concatenated across channels so their spatial arrangement encodes motion information.Spatial masks encode candidate bounding-box locations relative to the anchor before CNN-based long-term affinity estimation.
- R1TA Power Iteration Layer: FAMNet uses R1TA power iteration to approximate optimal multi-dimensional assignments from the affinity tensor.The global optimum is generally NP-hard, so the network uses a sub-optimal iterative approximation.
- ℓ1 Normalization Layer: The assignment design separates tensor power iteration from alternating row/column ℓ1 normalization to reduce operation depth and potential gradient vanishing.This adaptation may not retain the convergence property of the original method.
5. Experiment
Experiments evaluate FAMNet across pedestrian and vehicle MOT benchmarks using standard tracking metrics, benchmark comparisons, and module ablations. The method reports strong performance across datasets, while ablations examine initialization, target management, and SOT integration.
- Evaluation setup: Experiments cover MOT2015, MOT2017, KITTI-Car, and UA-DETRAC, spanning pedestrian and vehicle tracking scenarios.The datasets use referred detections from real detectors.
- Evaluation setup: Performance is evaluated with MOTP, MOTA, MT, and ML, with MOTA combining false positives, false negatives, and identity switches.These metrics capture localization precision, tracking accuracy, and target coverage.
- Benchmark results: FAMNet achieves clearly state-of-the-art performance on MOT2015 and surpasses RNN-based end-to-end online methods and the hand-crafted R1TA method in most metrics.The paper attributes the comparison advantage to discriminative higher-order affinity and the adapted optimization method.
- Benchmark results: FAMNet surpasses the hand-crafted R1TA method on KITTI-Car and achieves state-of-the-art performance among published works on UA-DETRAC.Motion affinity is reported as more important for KITTI because targets and the camera move faster and more regularly.
- Ablation study: Ablations compare no training, training from scratch, fine-tuning, CNNBBE removal, and SOT removal; limited MOT training may overfit feature and affinity sub-networks.The dedicated CNNBBE is reported to help prevent false positives from merging into tracking results, while SOT predictions introduce some false positives.
6. Conclusion
FAMNet jointly learns features and higher-order affinity end to end from ground-truth trajectories, while SOT predictions and dedicated target management further support tracking robustness. Experiments across four MOT datasets show the effectiveness of the approach.
- Conclusion: FAMNet learns features and high-order affinity jointly in an end-to-end architecture directly from ground-truth trajectories.The conclusion presents this joint learning design as the paper’s central architecture.
- Conclusion: The tracking system incorporates SOT predictions and dedicated target management to further boost tracking robustness.These components are integrated during tracking alongside the FAMNet architecture.
- Conclusion: Experiments on MOT2015, MOT2017, KITTI-Car, and UA-DETRAC show the effectiveness of the proposed approach.The evaluation spans pedestrian and vehicle tracking datasets.