Source-linked AI summary
Chained-Tracker: Chaining Paired Attentive Regression Results for End-to-End Joint Multiple-Object Detection and Tracking
Jinlong Peng, Changan Wang, Fangbin Wan, Yang Wu, Yabiao Wang, Ying Tai, Chengjie Wang, Jilin Li, Feiyue Huang, Yanwei Fu
TL;DR
MOT methods often separate detection, feature extraction, and data association, or integrate only some of these subtasks. CTracker unifies all three in an end-to-end online model using attentive paired-box regression across chained adjacent-frame nodes. The model is reported to achieve state-of-the-art tracking performance on MOT16 and MOT17 while remaining simple and fast.
Problem
Existing MOT solutions separate key subtasks or integrate them only partially, while adding re-identification and attention can increase model complexity and computational cost.
Method
CTracker jointly detects, extracts features, and associates targets by attentively regressing paired boxes from adjacent frames and chaining overlapping nodes.
Results
CTracker achieves state-of-the-art tracking performance on MOT16 and MOT17 tracking result lists with private detection.
Takeaways & Limitations
The chained structure and joint attention module make CTracker a simple end-to-end approach that uses temporal information to facilitate data association.
Takeaways & Limitations
CTracker omits patch-level ReID features to preserve simplicity and efficiency, resulting in lower IDF1 and higher IDS than several methods.
Abstract
from arXiv · showhide
Existing Multiple-Object Tracking (MOT) methods either follow the tracking-by-detection paradigm to conduct object detection, feature extraction and data association separately, or have two of the three subtasks integrated to form a partially end-to-end solution. Going beyond these sub-optimal frameworks, we propose a simple online model named Chained-Tracker (CTracker), which naturally integrates all the three subtasks into an end-to-end solution (the first as far as we know). It chains paired bounding boxes regression results estimated from overlapping nodes, of which each node covers two adjacent frames. The paired regression is made attentive by object-attention (brought by a detection module) and identity-attention (ensured by an ID verification module). The two major novelties: chained structure and paired attentive regression, make CTracker simple, fast and effective, setting new MOTA records on MOT16 and MOT17 challenge datasets (67.6 and 66.6, respectively), without relying on any extra training data. The source code of CTracker can be found at: github.com/pjl1995/CTracker.
1 Introduction
CTracker addresses limitations of fragmented and increasingly complex MOT systems with an end-to-end online model that jointly detects, extracts features, and associates targets. It uses adjacent-frame paired regression, joint attention, and chained nodes, achieving state-of-the-art tracking performance on MOT16 and MOT17.
- MOT remains challenging in crowded scenes because of occlusions, overlapping trajectories, and difficult backgrounds.
- Existing tracking-by-detection methods separate object detection, feature extraction, and data association, limiting global optimization and increasing computation cost.
- Re-identification and attention can improve MOT, but their use in existing solutions increases model complexity and computational cost.
- CTracker unifies object detection, feature extraction, and data association in a single end-to-end online model.
- CTracker takes adjacent frame pairs as input, jointly regresses paired boxes for shared targets, and chains overlapping nodes through common-frame boxes using IoU matching.
- A joint attention module uses object-classification confidence maps and ID-verification predictions to focus paired-box regression on foreground regions and corresponding targets.
- CTracker achieves state-of-the-art performance on MOT16 and MOT17 tracking result lists with private detection.
2 Related Work
Prior MOT work includes isolated detection and tracking components, partially integrated models, and increasingly complex attention-based association methods. CTracker differs by unifying detection, feature extraction, and association in one integrated model.
- Detection-based methods such as POI, EDMT, and Tracktor use detection-oriented designs, while their detection and tracking models remain independent.
- Partially End-to-end MOT Methods: RetinaTrack combines detection and feature extraction but still uses greedy bipartite matching for data association.
- Partially End-to-end MOT Methods: Other methods jointly model appearance, affinities, feature extraction, or assignment, but integrate only parts of the full MOT pipeline.
- Partially End-to-end MOT Methods: TrackNet jointly detects and tracks with frame tubes, but does not model links among tubes and is designed and tested only for rigid-object vehicle tracking.
- Attention-based MOT Methods: Attention-based MOT methods address occlusion and interaction with spatial, temporal, or appearance attention, but use complex attention models.
3 Methodology
CTracker processes adjacent-frame pairs, regresses paired boxes, and chains overlapping node outputs into trajectories. Its single network jointly performs detection, feature extraction, and data association, while attention modules and feature reuse support the pipeline.
- Problem Settings: MOT outputs bounding boxes for all objects across the frames where they appear, together with their identities.
- Problem Settings: CTracker uses a single network to simultaneously perform object detection, feature extraction, and data association.The model is described as converting data association into a pair-wise object detection problem.
- Chained-Tracker Pipeline: Each chain node takes two adjacent frames and regresses paired bounding boxes for targets appearing in both frames.The first node is (F1, F2), and the final node uses a copied version of frame N as hypothetical frame N + 1.
- Chained-Tracker Pipeline: Adjacent nodes are chained by IoU matching boxes on their shared frame, with the Kuhn-Munkres algorithm assigning matches sequentially across the video.Matched boxes extend existing tracklets, while unmatched boxes start new tracklets with new identities.
- Chained-Tracker Pipeline: The chaining operation is not fully parameterized, so a differentiable replacement for IoU-based chaining would be required for pure end-to-end trainability.The authors nevertheless describe CTracker as end-to-end because the regression model performs the major work without feedback from chaining.
- Network architecture: CTracker concatenates features from adjacent frames and applies object-classification and ID-verification attention maps to guide paired-box regression.The network uses tied-weight backbone branches, multi-scale features, and parallel classification and ID-verification branches.
- Network architecture: Memory sharing reuses extracted frame features across successive nodes and reduces almost half of the overall computation and time cost.For the final node, frame N is copied as hypothetical frame N + 1 and its features are reused.
4 Experiment
Experiments evaluate CTracker on MOT16 and MOT17 using standard MOT metrics, ablations, runtime analysis, and benchmark comparisons. The results show gains from joint attention and strong online tracking performance, with a deliberate trade-off between simplicity, speed, and identity accuracy.
- Experimental Setup: Experiments use MOT16 and MOT17, which share image sequences but differ in detection inputs and ground-truth labels.MOT17 includes DPM, Faster R-CNN, and SDP detections, whereas MOT16 includes DPM detection.
- Experimental Setup: MOTA is the primary metric among CLEAR MOT measures, alongside MOTP, FN, FP, IDS, MT, ML, and IDF1.IDF1 evaluates trajectory identity accuracy.
- Ablation Study: MOTA rises from 64.4 to 66.0 and IDF1 from 51.6 to 55.7 when object attention is added to the baseline.MOTP also increases from 78.2 to 78.8, indicating more accurate predicted boxes in this ablation.
- Ablation Study: Adding an independent ID verification branch slightly worsens performance because it lacks bounding-box information for reliable identification.The authors state that reliable identification needs good bounding boxes.
- Ablation Study: Adding ID attention improves IDF1 by 1.7 and MOTA by 0.6 over object attention, while MOTP decreases from 78.8 to 78.2.The ablation attributes the trade-off to greater emphasis on data association and slightly lower regression precision.
- Benchmark Evaluation: On private detection, CTracker significantly outperforms existing online MOT methods in MOTA on both MOT16 and MOT17.On MOT16, its MOTA is 0.6 below offline KDNT and 1.5 above online POI, while using only MOT16 training data.
- Benchmark Evaluation: Using CTracker detections raises Tracktor’s MOTA from 53.5 to 54.4 and IDF1 from 52.3 to 56.1 on public detection.The comparison is reported for Tracktor+CTdet in Table 4.
5 Conclusion
CTracker integrates joint detection and tracking through paired attentive bounding-box regression and chaining across adjacent frame nodes.
- CTracker regresses paired bounding boxes for the same target across two adjacent frames.
- A joint attention module combines detection-driven object attention with ID-verification-injected identity attention.
- Overlapping nodes are chained using IoU information in their common frame to generate tracking trajectories.
- Experiments on widely used MOT benchmarks demonstrate CTracker's effectiveness and efficiency.
Chained-Tracker: Chaining Paired Attentive Regression Results for End-to-End Joint Multiple-Object Detection and Tracking (Supplementary Material)
The supplementary material identifies the paper's authors, including Jinlong Peng, Changan Wang, and Fangbin Wan.
- Jinlong Peng and Changan Wang are listed among the paper's authors.
- Fangbin Wan and Yang Wu are listed among the paper's authors.
- Yabiao Wang, Ying Tai, Chengjie Wang, Jilin Li, Feiyue Huang, and Yanwei Fu are also listed.
1 Overview
The supplementary material covers the CTracker architecture, training augmentation, Chained-Anchors settings, experiments, qualitative comparisons, and appearance features.
- It provides the detailed design of the CTracker network architecture.
- It documents data augmentation in training and the Chained-Anchors setting.
- It reports detailed experiments and qualitative comparisons with POI and Tracktor.
- It includes an experiment adding appearance features to CTracker.
2 Details of Network Architecture
CTracker builds multi-scale features from adjacent frames, applies parallel classification and ID-verification branches, and uses their combined attention to guide paired-box regression.
- Backbone: ResNet50 and FPN produce multi-scale feature representations at five levels, denoted P2 through P6.
- Backbone: Figure 1 presents the detailed architecture of the CTracker backbone.
- Prediction head: Features from two adjacent frames are combined at each scale for subsequent prediction.
- Prediction head: Parallel object-classification and ID-verification branches predict confidence after task-specific convolutional feature learning.
- Prediction head: Figure 2 presents the detailed architecture of the CTracker prediction head and its multi-scale inputs.
- Attention and regression: The two predictions are multiplied into a single-channel joint attention map that guides paired-box regression through element-wise products.
3 Details of Implementation
The implementation samples nearby frame pairs, applies shared augmentations, and configures Chained-Anchors from dataset-specific box clusters. Detailed MOT16 and MOT17 tracking results are reported in Tables 1 and 2.
- Training data and augmentation: Training pairs use frames separated by 1–3 positions, with frame order reversed randomly with 50% probability.The procedure targets objects with different motion speeds.
- Training data and augmentation: Each frame pair receives shared photometric, crop, expansion, flipping, and resizing augmentations to reduce over-fitting.Expansion uses a random factor in, followed by resizing to a square patch.
- Evaluation: Detailed CTracker tracking results on the MOT16 and MOT17 test datasets are presented in Tables 1 and 2.
- Anchor configuration: Chained-Anchor scales are selected using k-means clustering over dataset ground-truth boxes, with scales {38, 86, 112, 156, 328} assigned to P2–P6.All Chained-Anchors use the same ratio of 2.9.
4 Detailed Experiment Results
The experiments report detailed MOT16 and MOT17 results and qualitative comparisons with POI and Tracktor. CTracker avoids long-term drift in one case and handles an occlusion-induced identity switch in another.
- Quantitative results: Detailed CTracker tracking results are reported for the MOT16 and MOT17 test datasets.The results appear in Tables 1 and 2.
- Comparison with POI: CTracker shows no long-term cross-frame tracking drift in the displayed MOT16-03 trajectories, unlike POI in the comparison.The compared POI trajectories contain several marked drift cases.
- Comparison with Tracktor: Under the same detection, Tracktor produces an occlusion-related identity switch, whereas CTracker correctly associates the two pedestrian trajectories.The comparison attributes the correct result to accurate box-pair association in CTracker.
- Design trade-off: CTracker prioritizes short-term tracking and abandons patch-level ReID features, trading some trajectory integrity for substantially improved trajectory accuracy.
5 Appearance Feature Experiment
An appearance-feature experiment extends node chaining beyond IoU affinity by adding appearance similarity from feature-map representations.
- Appearance features: The appearance experiment adds appearance similarity to IoU affinity in the node chaining module.The added appearance features are 256-dimensional vectors extracted from the feature map before the described layer.