Source-linked AI summary
Exploit the Connectivity: Multi-Object Tracking with TrackletNet
Gaoang Wang, Yizhou Wang, Haotian Zhang, Renshu Gu, Jenq-Neng Hwang
TL;DR
MOT remains difficult under unreliable detections, occlusion, and fast camera motion, while combining appearance and temporal information is not well addressed. TNT uses a tracklet-based graph with epipolar-constrained generation and multi-scale TrackletNet connectivity measurement, then reports superior performance on MOT16 and MOT17. The paper further discusses occlusion handling but identifies fast camera motion in 2D tracking as a remaining challenge.
Problem
MOT is challenging under unreliable detection, occlusion, and fast camera motion, and existing approaches have not adequately combined appearance and temporal features.
Method
TNT constructs a graph whose vertices are tracklets, generates them using appearance, IOU, and epipolar constraints, and measures pairwise connectivity with a multi-scale TrackletNet.
Results
TNT outperforms other state-of-the-art methods on the MOT16 and MOT17 benchmarks.
Takeaways & Limitations
The tracklet-graph framework is reported to support robust tracking across benchmark datasets and occlusion scenarios.
Takeaways & Limitations
Fast camera motion remains a challenge for TNT's 2D tracking, motivating future conversion to 3D tracking with visual odometry.
Abstract
from arXiv · showhide
Multi-object tracking (MOT) is an important and practical task related to both surveillance systems and moving camera applications, such as autonomous driving and robotic vision. However, due to unreliable detection, occlusion and fast camera motion, tracked targets can be easily lost, which makes MOT very challenging. Most recent works treat tracking as a re-identification (Re-ID) task, but how to combine appearance and temporal features is still not well addressed. In this paper, we propose an innovative and effective tracking method called TrackletNet Tracker (TNT) that combines temporal and appearance information together as a unified framework. First, we define a graph model which treats each tracklet as a vertex. The tracklets are generated by appearance similarity with CNN features and intersection-over-union (IOU) with epipolar constraints to compensate camera movement between adjacent frames. Then, for every pair of two tracklets, the similarity is measured by our designed multi-scale TrackletNet. Afterwards, the tracklets are clustered into groups which represent individual object IDs. Our proposed TNT has the ability to handle most of the challenges in MOT, and achieve promising results on MOT16 and MOT17 benchmark datasets compared with other state-of-the-art methods.
1. Introduction
TNT addresses difficult multi-object tracking by representing tracklets in a graph and combining appearance, temporal, and motion-compensation cues. It measures tracklet connectivity with a multi-scale TrackletNet and reports stronger benchmark performance on MOT16 and MOT17.
- Motivation: MOT is challenging because noisy detections and occlusions can cause targets to be lost during long-term tracking.The paper frames tracking-by-detection as an association problem given detection results.
- Framework: TNT represents each tracklet as a graph vertex and groups connected tracklets into clusters corresponding to object identities.A tracklet is a consecutive sequence of detections that may form when an object's full trajectory is fragmented.
- Tracklet Generation: Tracklets are generated using appearance similarity and IOU, with epipolar geometry compensating camera movement during adjacent-frame association.The epipolar constraint addresses shifts in detection positions caused by moving or revolving cameras.
- Connectivity Measurement: A multi-scale TrackletNet measures connectivity between tracklet pairs by combining trajectory and appearance information.The network supplies edge similarities for the graph model.
- Results: TNT outperforms state-of-the-art methods on both MOT16 and MOT17 benchmarks.The paper also reports that the method can be applied to different scenarios.
2. Related Work
Prior MOT research includes graph-based, recurrent, feature-fusion, and end-to-end tracking approaches. Their reported trade-offs concern temporal utilization, long-range relations, feature availability, error accumulation, and joint detection-tracking.
- Graph Model based Tracking: Graph-based tracking associates detections across frames by minimizing total graph cost, but detection-based graphs can underuse temporal information and become computationally expensive.Tracklet-based graphs better use short-trajectory information, while requiring careful handling of generation errors.
- Tracking by RNN: RNN-based trackers support online prediction, but relations between faraway detections weaken as recurrent propagation continues.The passage reports degraded long-run performance and sensitivity to unreliable detections without direct connections.
- Tracking by Feature Fusion: Feature-fusion methods commonly combine appearance features, such as CNN embeddings or handcrafted descriptors, with temporal features from bounding-box motion.The supplied passage identifies appearance and temporal features as the two common feature types.
- End-to-End Tracking: End-to-end trackers jointly train detection and tracking from raw video sequences, reducing accumulated detection-to-tracking errors.The passage also describes temporal information improving detection and detections feeding back into tracking.
3. Tracklet Graph Model
The tracklet graph uses short trajectories as vertices and connectivity-weighted edges to represent object identities. Tracking is then formulated as clustering tracklets into disjoint object-specific sub-graphs.
- Vertex Set: Tracklets, rather than individual detections, serve as graph vertices, reducing the number of entities and retaining short-term trajectory information.Each tracklet is a consecutive sequence of detections from an object, potentially created when unreliable detections divide a full trajectory.
- Edge Set: Edges connect tracklets that are sufficiently close in time, while distant tracklet pairs are excluded because their relationships cannot be reliably measured.The temporal proximity condition is controlled by δt.
- Edge Set: Each edge carries a connectivity measure representing the similarity between its two tracklets, with TrackletNet providing the measurement.Connectivity is set to zero when two tracklets overlap temporally because they must belong to distinct objects.
- Feasible Solutions: Object trajectories are obtained by partitioning the graph into disjoint connected sub-graphs, with each sub-graph representing one object ID.Feasible partitions require connected sub-graphs and finite costs on edges internal to each sub-graph.
- Objective Function: The clustering objective minimizes total edge cost by assigning tracklet pairs to the same or different sub-graphs.The clustering label is +1 for pairs in one sub-graph and −1 otherwise; greedy assign, merge, split, switch, and break operations optimize the APX-hard problem.
4. Proposed TrackletNet Tracker
Tracklet generation combines adjacent-frame IOU and appearance similarity, while epipolar geometry compensates for camera motion. Multi-scale TrackletNet then fuses temporal and appearance information to measure tracklet similarity.
- 4.1. Tracklet Generation with Epipolar Constraints: Adjacent detections are associated using IOU and appearance similarity with a high threshold to reduce mis-association during tracklet generation.This initial association remains vulnerable to fast camera motion.
- 4.1. Tracklet Generation with Epipolar Constraints: Figure 2 illustrates how epipolar-constrained prediction avoids an incorrect high-IOU association under camera motion.The predicted red box in the next frame overlaps the correct yellow detection rather than the alternative candidate.
- 4.1. Tracklet Generation with Epipolar Constraints: Epipolar geometry predicts the next-frame bounding box by aligning it with corresponding epipolar lines while constraining its size under slow-target-motion assumptions.The predicted box is then compared with next-frame detections using IOU.
- 4.2. Multi-Scale TrackletNet: TrackletNet measures connectivity by combining temporal and appearance features from two tracklets within a fixed time window.Each detection representation contains four bounding-box parameters and an embedded appearance feature; the two tracklets are concatenated across the window.
- 4.2. Multi-Scale TrackletNet: The network uses three temporal convolution layers with 1 × 3, 1 × 5, 1 × 9, and 1 × 13 kernels, followed by pooling and fully connected layers.Different kernel sizes capture feature changes at different temporal scales, while max pooling downsamples the time domain.
- 4.2. Multi-Scale TrackletNet: TrackletNet outputs a similarity between zero and one while using binary masks to distinguish missing detections from abrupt tracklet changes.The architecture performs convolution only along time and integrates Re-ID, temporal, and spatial dependencies.
5. Experiments
Experiments evaluate TNT on MOT16 and MOT17, including quantitative benchmark performance, cross-scenario tracking, occlusion handling, epipolar tracklet generation, and robustness to appearance noise.
- Experimental Setup: MOT16 and MOT17 provide the training and evaluation benchmarks, with MOT17 additionally offering Faster-RCNN and SDP detections alongside DPM.MOT16 contains seven training and seven testing sequences; MOT17 uses the same video sequences with more accurate ground truth.
- Quantitative Results: IDF1 is much higher than competing state-of-the-art methods, while TNT ranks among the top methods on other reported metrics.The paper uses IDF1 and MOTA as major reliability factors, emphasizing IDF1’s trajectory-level evaluation.
- Qualitative Results: Without fine-tuning, TNT achieves promising tracking results in 3D pose estimation and UAV applications.Figure 5 illustrates transfer to campus pose estimation and UAV tracking scenarios.
- Occlusion Handling: TNT handles long partial and full occlusions by forming separate tracklets, reconnecting them through connectivity, and interpolating missing detections after ID confirmation.Examples include recovery after full occlusion, identity consistency while crossing pedestrians, and tracking with a moving camera.
- Epipolar Tracklet Generation: Epipolar geometry substantially reduces FNR, especially on MOT17-13, while FDR remains quite small during tracklet generation.The evaluation uses MOT17-10 and MOT17-13 with Faster-RCNN detections because both sequences contain large camera motion.
- Robustness to Appearance Features: TNT remains comparatively robust to disturbed appearance features because TrackletNet measures temporal continuity rather than feature distance alone.The baseline degrades as Gaussian noise increases, whereas TNT’s tracking performance changes little.
6. Conclusion and Future Work
TNT is presented as a graph-based multi-object tracker combining epipolar-geometry tracklet generation with multi-scale TrackletNet connectivity measurement. It outperforms state-of-the-art methods on MOT16 and MOT17, while fast camera motion remains a challenge for its 2D tracking.
- TNT outperforms other state-of-the-art methods on the MOT16 and MOT17 benchmark datasets.
- The paper discusses TNT's qualitative results across different scenarios and applications, including robustness to occlusions.
- Fast camera motion remains a challenge for TNT's 2D tracking.