Source-linked AI summary
TubeTK: Adopting Tubes to Track Multi-Object in a One-Step Training Model
Bo Pang, Yizhuo Li, Yifan Zhang, Muchen Li, Cewu Lu
TL;DR
MOT traditionally relies on a two-step tracking-by-detection pipeline whose separate spatial and temporal processing limits integration, occlusion handling, and independence from detection results. TubeTK addresses this by directly regressing spatial-temporal bounding-tubes with a 3D CNN in an end-to-end, one-step framework. The model achieves state-of-the-art performance on mainstream MOT benchmarks and shows robustness to occlusions and noisy tube linking without Re-ID.
Problem
Tracking-by-detection separates detection from association, leaving MOT dependent on external detections and insufficiently integrating spatial-temporal information, especially under occlusion.
Method
TubeTK uses a 3D CNN to directly regress bounding-tubes that encode targets’ spatial locations, temporal positions, and motion trails, then links them with IoU-based post-processing.
Results
TubeTK achieves new state-of-the-art performance on mainstream MOT benchmarks and remains effective under severe occlusion without ancillary technologies such as Re-ID.
Takeaways & Limitations
One-step end-to-end TubeTK offers a simpler MOT paradigm independent of ready-made detection results while retaining robust tracking through occlusions.
Abstract
from arXiv · showhide
Multi-object tracking is a fundamental vision problem that has been studied for a long time. As deep learning brings excellent performances to object detection algorithms, Tracking by Detection (TBD) has become the mainstream tracking framework. Despite the success of TBD, this two-step method is too complicated to train in an end-to-end manner and induces many challenges as well, such as insufficient exploration of video spatial-temporal information, vulnerability when facing object occlusion, and excessive reliance on detection results. To address these challenges, we propose a concise end-to-end model TubeTK which only needs one step training by introducing the ``bounding-tube" to indicate temporal-spatial locations of objects in a short video clip. TubeTK provides a novel direction of multi-object tracking, and we demonstrate its potential to solve the above challenges without bells and whistles. We analyze the performance of TubeTK on several MOT benchmarks and provide empirical evidence to show that TubeTK has the ability to overcome occlusions to some extent without any ancillary technologies like Re-ID. Compared with other methods that adopt private detection results, our one-stage end-to-end model achieves state-of-the-art performances even if it adopts no ready-made detection results. We hope that the proposed TubeTK model can serve as a simple but strong alternative for video-based MOT task. The code and models are available at https://github.com/BoPang1996/TubeTK.
1. Introduction
TubeTK replaces tracking-by-detection with one-step, end-to-end MOT by regressing bounding-tubes that jointly encode spatial location and motion over time. This design reduces dependence on external detections and improves robustness to occlusions while achieving strong benchmark performance.
- Motivation: Tracking-by-detection separates detection and association, making MOT overly dependent on detection quality and incomplete in its use of spatial-temporal information.Detectors mainly model spatial features while trackers capture temporal features, leaving video information insufficiently integrated.
- Contributions: One-step training makes MOT a stand-alone task without restrictions from detection models, while TubeTK is reported to outperform TBD-based counterparts.The authors position this as a simpler alternative to two-step tracking frameworks and ancillary technologies such as Re-ID.
- TubeTK: TubeTK represents each target with a bounding-tube that captures spatial location, temporal position, and motion trail in a unified space-time structure.The Btube extends a 2D box into 2D spatial and 1D temporal dimensions, using 15 defining points.
- TubeTK: A 3D CNN extracts spatial-temporal features simultaneously and directly regresses Btubes, avoiding handcrafted tracking cues and external detection results.An FPN supports targets at different scales, and simple IoU-based post-processing links Btubes into tracks.
- Contributions: TubeTK achieves better performance than TBD-based methods on MOT15, MOT16, and MOT17 without ready-made detection results.The authors also report smoother Btube-based tracks, with fewer false negatives and identity switches than tracks based on pre-generated image-level boxes.
2. Related Work
Prior tube-based approaches still depend on external detections and construct tubes as a second-stage operation. TubeTK instead directly regresses Btubes in a one-step tracking framework.
- Tracking-by-detection: Tracking-by-detection methods commonly use external object detectors and associate detected boxes across frames.Association has been formulated through graph-based methods involving maximum-flow or minimum-cost objectives.
- TubeTK: TubeTK introduces a one-step end-to-end MOT paradigm intended to model spatial-temporal features and alleviate issues caused by dense crowds and occlusions.Its pipeline defines Btubes, predicts them from videos, trains the Btube representation, and links outputs into tracks.
- Tube-based methods: Earlier tube-based methods combine detection results into tube proposals or use single-object tracking to capture trajectories.These approaches apply tubes to video detection or trajectory capture rather than directly regressing them for end-to-end MOT.
3. The Proposed Tracking Model
TubeTK reframes multi-object tracking as one-step, end-to-end regression of bounding-tubes that jointly encode objects’ spatial positions, temporal positions, and motion. A 3D convolutional network predicts Btubes from short clips, trains with tube-aware losses, and links overlapping predictions using spatial agreement and motion direction.
- 3.1. From Bounding-Box to Bounding-Tube: Whole tracks are split into overlapping short Btubes because linear interpolation captures complex motion more accurately over short clips.The method dynamically varies Btube length: longer for monotonous motion and shorter when trajectories change sharply.
- 3.1. From Bounding-Box to Bounding-Tube: TubeTK defines a Btube as a 3D extension of a bounding box that represents spatial location, temporal position, and motion trajectory.Each Btube is identified by 15 coordinate values and can model changing direction and linearly varying scale across three frames.
- 3.3. Training Method: Training combines focal classification loss, binary cross-entropy for center-ness, and Tube GIoU loss for Btube regression.Tube GIoU extends box GIoU to Btubes by comparing their volumes and smallest enclosing tube-shaped object.
- 3.2. Model Structure: A 3D CNN processes short video clips with a backbone, 3D FPN, and multi-scale heads to predict Btube regression, confidence, and center-ness maps.The 3D network extracts spatial-temporal features simultaneously, while the multi-scale heads handle targets at different scales.
- 3.2. Model Structure: Output-map points are assigned to a Btube when they lie inside its middle-frame box Bm, while points on other maps are excluded from predicting it.This assignment follows the multi-scale design and focuses regression on relative temporal and spatial positions associated with motion patterns.
- 3.4. Linking the Bounding-Tubes: Predicted Btubes are linked when their overlap matching score exceeds threshold β, with averaged overlap boxes and a direction-based coefficient reducing mismatches at track intersections.The final score is adjusted using the cosine of the angle between the moving-direction vectors of the track and candidate Btube.
4. Experiments
TubeTK is evaluated on MOT15, MOT16, and MOT17, including occlusion-focused analyses, ablations, and comparisons with online state-of-the-art models. The experiments examine its training setup, detection dependence, and tracking performance.
- TubeTK is evaluated on MOT15, MOT16, and MOT17, which contain many occlusions and provide public detection results for comparison.
- Ablation study: The ablation study examines clip length, NMS, and Tube GIoU loss, with shorter clips producing worse performance and original NMS removing highly occluded Btubes.Tube GIoU loss regresses Btube length more accurately than Tube IoU loss, yielding fewer false negatives and false positives.
- Benchmark protocol: TubeTK does not adopt external detection results in the reported online-model comparison, whereas RAN and CNNMTT use detections provided by POI.
- Benchmark results: 1.9 MOTA improvements are reported on MOT15, while MOT16 reaches 64.0 MOTA versus 54.5 for other state-of-the-art models using public detections.
- Benchmark results: On MOT16 with POI detections but without appearance features, TubeTK achieves 66.9 MOTA versus 66.1 for the comparison state-of-the-art model.
5. Discussion
The discussion attributes TubeTK’s occlusion robustness to Btubes encoding target motion trends and evaluates both occlusion performance and linking robustness. These analyses indicate continued tracking under severe occlusion and tolerance to substantial Btube noise.
- Overcoming the occlusion: TubeTK’s Btubes encode moving trends, supporting more robust performance when targets face severe occlusions.
- Overcoming the occlusion: TubeTK keeps tracking with much less false negatives or ID switches when targets are totally shielded by other targets.
- Overcoming the occlusion: When target visibility is low, TubeTK performs much better than other tracking-by-detection models.
- Robustness of Btubes for linking: With up to 25% center-position or scale jitter, linking on MOT17-02 still achieves MOTA > 86 and IDF1 > 79.The linking process has no learnable parameters and does not require rigorously accurate Btubes to finish tracking.
6. Conclusion
TubeTK is an end-to-end, one-step MOT model that uses Btubes to encode temporal-spatial position and local motion trails. It achieves state-of-the-art performance on mainstream benchmarks compared with online models, including models using private detections.
- TubeTK uses Btubes to encode targets’ temporal-spatial positions and local moving trails in an end-to-end, one-step training model.
- TubeTK achieves new state-of-the-art performance on mainstream benchmarks compared with other online models, even when those models use private detection results.The paper presents comprehensive analyses to validate TubeTK’s robustness.