Source-linked AI summary
Referring Multi-Object Tracking
Dongming Wu, Wencheng Han, Tiancai Wang, Xingping Dong, Xiangyu Zhang, Jianbing Shen
TL;DR
Existing referring-understanding tasks generally detect a single text-referred object, leaving multi-object and temporally varying video settings insufficiently covered. The paper introduces RMOT, builds the Refer-KITTI benchmark, and proposes TransRMOT, which achieves impressive detection performance and outperforms other counterparts on the new task.
Problem
Existing referring-understanding tasks tend to detect a single text-referred object and lack accurate evaluation for multiple referents with temporal status variation.
Method
The paper introduces language-guided RMOT, constructs Refer-KITTI from KITTI, and develops the Transformer-based TransRMOT architecture.
Results
TransRMOT achieves impressive detection performance on Refer-KITTI and outperforms other counterparts.
Takeaways & Limitations
RMOT provides a more flexible referring-understanding setting with arbitrary numbers of referent objects and temporal status variation.
Takeaways & Limitations
TransRMOT can miss fine-grained object features and suffer ID switches under long-term occlusion.
Abstract
from arXiv · showhide
Existing referring understanding tasks tend to involve the detection of a single text-referred object. In this paper, we propose a new and general referring understanding task, termed referring multi-object tracking (RMOT). Its core idea is to employ a language expression as a semantic cue to guide the prediction of multi-object tracking. To the best of our knowledge, it is the first work to achieve an arbitrary number of referent object predictions in videos. To push forward RMOT, we construct one benchmark with scalable expressions based on KITTI, named Refer-KITTI. Specifically, it provides 18 videos with 818 expressions, and each expression in a video is annotated with an average of 10.7 objects. Further, we develop a transformer-based architecture TransRMOT to tackle the new task in an online manner, which achieves impressive detection performance and outperforms other counterparts. The dataset and code will be available at https://github.com/wudongming97/RMOT.
1. Introduction
The paper introduces RMOT, a language-guided task for tracking an arbitrary number of semantically matched objects whose temporal status may vary. It also presents Refer-KITTI and TransRMOT to benchmark and address this setting.
- Existing referring benchmarks mainly target one object and do not accurately evaluate multiple referents with changing temporal status.
- RMOT grounds all semantically matched objects from a language expression, allowing the referent count to change across video frames.For example, one expression can correspond to one object at one frame and two objects later.
- TransRMOT is an end-to-end Transformer framework using cross-modal reasoning and cross-frame association for RMOT.Its architecture builds on DETR and integrates visual and linguistic features in an encoder-decoder design.
- The paper contributes RMOT, Refer-KITTI, and TransRMOT, reporting impressive performance on Refer-KITTI compared with handcrafted RMOT methods.
2. Related Work
Prior referring-understanding datasets span image grounding, video grounding, and video segmentation, but commonly remain limited to single-object expressions and overlook temporal variation. RMOT is positioned as a response to these limitations.
- Image-based datasets such as Flickr30k, ReferIt, and RefCOCO/+/g ground language expressions in visual regions but do not fit practical video scenes.
- Video referring datasets including A2D-Sentences, JHMDB-Sentences, Refer-DAVIS17, and Refer-Youtube-VOS still mainly describe one object and ignore temporal variants.
- Transformer-based referring methods provide strong cross-modal representation ability, but existing methods cannot process real-world multi-object scenarios.
3. Benchmark
Refer-KITTI is built from KITTI to support scalable, temporally varying referring expressions, with annotations designed to reduce labeling effort and evaluate both detection and association.
- Dataset Construction: Refer-KITTI reuses KITTI's diverse scenes and instance identities to construct an efficient annotation workflow for referring multi-object tracking.KITTI provides instance-level boxes and consistent identification numbers across frames, supporting tool-assisted labeling.
- Dataset Construction: The annotation process uses multiple annotators to create and verify discriminative expressions, then saves frame IDs, object IDs, box coordinates, and expressions for training and testing.The workflow lowers human cost compared with frame-by-frame annotation.
- Referent Objects: Each expression can refer to an arbitrary number of objects, averaging 10.7 objects and reaching a maximum of 105 referents.Most expressions describe 0–25 objects, demonstrating scalable referent cardinality.
- Temporal Variation: Referent objects vary substantially over time, with expressions spanning mostly 0–400 frames and some exceeding 600 frames.Many referents enter or leave visible scenes, creating a cross-frame association challenge.
- Evaluation Metrics: Refer-KITTI is evaluated with HOTA, which balances detection accuracy and association accuracy while treating predicted non-referent objects as false positives.Overall HOTA is averaged across sentence queries, with HOTA = DetA · AssA.
4. Method
TransRMOT processes a video stream and language query through cross-modal encoding, query-based decoding, and a referent head to predict tracked objects. Its training combines track and detection losses for flexible per-frame predictions and cross-frame association.
- Cross-modal Encoder: TransRMOT fuses visual and linguistic features before deformable encoder layers, reducing the cost of dense self-attention while promoting cross-modal interaction.Visual features are projected and flattened, linguistic features are projected to the same dimension, and vision-conditioned language features are added to the original visual features.
- Decoder: The decoder concatenates carried-over track queries with randomly initialized detection queries to represent continuing and newly appearing objects.Decoder embeddings from the previous frame are filtered and transformed into track queries, while detect queries handle new-born objects.
- Referent Head: A referent head predicts object validity, bounding boxes, and referring scores from the decoder representations.Its class branch distinguishes true from empty objects, its box branch predicts visible-instance locations, and its referring branch scores correspondence with the language description.
- Instance Matching and Loss: The model predicts a flexible-size set of objects per frame and trains it with separate track and detection losses.Tracking objects are matched one-to-one with ground truth, whereas new-born detections use bipartite matching and set prediction.
- Instance Matching and Loss: Track loss supervises visibility, box geometry, and referring probability using classification, box, and focal-loss terms.The predicted object includes a visibility scalar, normalized box coordinates, and a referring probability; box loss combines L1 and generalized IoU losses.
5. Experiments
Experiments evaluate TransRMOT on Refer-KITTI through implementation details, quantitative comparisons, qualitative examples, ablations, and generalization tests. The results show strong tracking and referring performance, component importance, benchmark stability, and generalization to reduced data, new expressions, and another dataset.
- Quantitative Results: TransRMOT outperforms CNN-based and Transformer-based competitors across all reported metrics on Refer-KITTI.The competitors are constructed by adding cross-modal learning to existing multi-object tracking models.
- Quantitative Results: Refer-KITTI remains stable across random seeds, with performance variance below ΔHOTA 0.87.All models were trained three times with different seeds.
- Qualitative Results: TransRMOT detects and tracks referent objects under multiple objects, temporal status changes, and varying object numbers.Qualitative comparisons also show that it detects all visible objects while highlighting those matching the expression.
- Ablation Study: Removing cross-modal fusion reduces HOTA from 34.29 to 17.01, DetA from 28.25 to 19.05, and AssA from 46.25 to 15.26.The ablation removes expressions as input and predicts all visible objects instead.
- Generalization Analysis: Using 50% of expressions achieves stable HOTA, while 75% of videos produces performance similar to the full dataset.A model pretrained on Refer-KITTI also produces promising referent results on BDD100K.
- Generalization Analysis: TransRMOT correctly recognizes referents for the unseen expression “the left persons in black.”The expression combines language knowledge represented by existing expressions in Refer-KITTI.
6. Conclusion
The paper introduces RMOT to overcome single-object referring limitations by supporting multiple referents and temporal status variation. It contributes Refer-KITTI and the end-to-end Transformer-based TransRMOT, which achieves state-of-the-art performance on the benchmark.
- Conclusion: RMOT supports flexible multi-object referring and temporal status variation for more general real-world video understanding.Each expression may refer to multiple objects whose temporal presence changes across frames.
- Conclusion: Refer-KITTI provides flexible referent counts and high temporal dynamics with low labeling costs.The benchmark is designed to promote research on RMOT.
- Conclusion: TransRMOT is an end-to-end Transformer-based framework that predicts referent objects frame by frame and achieves state-of-the-art performance.The framework is fully end-to-end optimized during training.
7. Supplementary Material
Supplementary material describes competitor modifications, the cross-modal architecture used for Transformer-based baselines, failure cases, and additional qualitative results. It highlights fine-grained recognition errors and identity switches under long-term occlusion as remaining challenges.
- Competitor Details: CNN-based competitors use cross-modal modifications of tracking-by-detection MOT models, while Transformer-based competitors receive an early-fusion module before encoder layers.The supplementary material names FairMOT, DeepSORT, ByteTrack, CStrack, TransTrack, and TrackFormer as modified competitors.
- Limitations: TransRMOT can fail to capture fine-grained object features such as human gender, reducing detection performance.The authors suggest exploring detection-then-fusion to focus on fine-grained object-region features.
- Limitations: Long-term occlusion can cause identity switches that degrade tracking performance.The authors suggest memory mechanisms for retaining object representations during long-term association.
- More Qualitative Results: Additional qualitative results show compelling performance with multiple objects, object entrance and exit, moving objects, and occlusion.These examples extend the reported qualitative evaluation beyond the main figures.