Source-linked AI summary
ReMOTS: Self-Supervised Refining Multi-Object Tracking and Segmentation
Fan Yang, Xin Chang, Chenyu Dang, Ziqiang Zheng, Sakriani Sakti, Satoshi Nakamura, Yang Wu
TL;DR
MOTS refinement is challenged by bounding-box ambiguity and the difficulty of adapting appearance features and selecting thresholds for target videos. ReMOTS retrains appearance features using target-video observations and short-term tracklets, then refines data association; it placed 1st in CVPR 2020 MOTS Challenge 1.
Problem
Bounding-box-based tracking faces ambiguity when different objects occupy the same box, motivating the use of instance masks for MOTS.
Source: §1 · p. 1Method
ReMOTS retrains an appearance encoder with target-video observations and short-term tracklets, then merges tracklets using appearance features and statistically selected thresholds.
Source: §2 · p. 1, §2.2 · p. 2, §2.3 · p. 2Results
ReMOTS placed 1st in CVPR 2020 MOTS Challenge 1.
Source: §1 · p. 1Takeaways & Limitations
The authors report that ReMOTS’s analysis of tracklet statistics for threshold selection could be useful for other multiple object tracking works.
Source: §4 · p. 4ReMOTS uses video-level statistics and does not consider temporal local statistics that might support fine-grained adaptation.
Source: §3 · p. 4
Abstract
from arXiv · showhide
We aim to improve the performance of Multiple Object Tracking and Segmentation (MOTS) by refinement. However, it remains challenging for refining MOTS results, which could be attributed to that appearance features are not adapted to target videos and it is also difficult to find proper thresholds to discriminate them. To tackle this issue, we propose a self-supervised refining MOTS (i.e., ReMOTS) framework. ReMOTS mainly takes four steps to refine MOTS results from the data association perspective. (1) Training the appearance encoder using predicted masks. (2) Associating observations across adjacent frames to form short-term tracklets. (3) Training the appearance encoder using short-term tracklets as reliable pseudo labels. (4) Merging short-term tracklets to long-term tracklets utilizing adopted appearance features and thresholds that are automatically obtained from statistical information. Using ReMOTS, we reached the $1^{st}$ place on CVPR 2020 MOTS Challenge 1, with an sMOTSA score of $69.9$.
1. Introduction
Bounding-box-based MOT struggles to distinguish objects that occupy the same box, motivating interest in MOTS and its object-instance masks. The paper reports that its solution won first place in CVPR 2020 MOTS Challenge 1.
- Different objects can occupy the same bounding box, increasing ambiguity in distinguishing them with bounding-box-based MOT.Source: §1 · p. 1
- MOTS researchers seek to use object-instance masks, and the first MOTS challenge was organized to explore solutions.Source: §1 · p. 1
- The authors state that their solution won first place on Challenge 1; the passage is dated May 30, 2020, and introduces the solution in the paper.Source: §1 · p. 1
2. Method Details
ReMOTS is an offline, four-step refinement pipeline: it trains appearance features on target-video data, links observations into short-term tracklets, retrains with those tracklets, and merges them into long-term tracks.
2.1. Generate Object-Instance Masks
- Modified NMS fuses object masks using IoM, then retains the highest-confidence mask in each overlapping area.IoM is introduced because heavily overlapping masks can still have low IoU.Source: §2.1 · pp. 1–2
2.2. Encoder Training with Intra-frame Data
- Target-video intra-frame observations adapt a SeResNeXt50 appearance encoder trained with triplet loss.Training triplets combine ground-truth-tracklet samples and target-video samples; augmented anchors provide positives, while exclusive same-frame predicted masks provide negatives.Source: §2.2 · p. 2
2.3. Short-term Tracker
- A two-frame tracker combines warped-mask IoU and appearance cosine distance, using intra-frame similarity statistics to set an appearance threshold.Linear assignment links adjacent-frame masks into short-term tracklets; misdetections and occlusions can limit their duration.Source: §2.3 · pp. 2–3
2.4. Inter-short-tracklet Encoder Retraining
- Short-term tracklets serve as pseudo-labels for encoder retraining, with sampled tracklets constrained to overlap temporally within the same video.Training samples select two identities in one frame and another frame containing one selected identity.Source: §2.4 · p. 3
2.5. Merging Short-term Tracklets
- Hierarchical clustering merges short-term tracklets using appearance distances while excluding pairs with temporal overlap or gaps beyond 15 frames.The clustering cut is derived from the separation between intra-frame and inter-short-tracklet cosine-similarity distributions after retraining, rather than a heuristic value.Source: §2.5 · p. 3
2.6. Experimental Setup
- The appearance encoder is not a contribution of ReMOTS, and the framework can use other appearance models.The authors otherwise make little change to the default settings of the borrowed encoder, apart from forming new training samples.Source: §2.6 · p. 4
3. Results
ReMOTS achieves state-of-the-art performance in the challenge, mainly outperforming the other top methods on IDF1. Sequence results vary, and fine-grained adaptation using local temporal statistics is not considered.
3. Results
- ReMOTS mainly outperforms the other two top methods on IDF1, leading to state-of-the-art performance in the challenge.Source: §3 · p. 4
- Performance varies considerably across test sequences, a pattern the authors attribute potentially to video diversity.Source: §3 · p. 4
- Fine-grained adaptation using temporal local statistical information is not considered.The authors note that this information might be useful for finer-grained adaptation.Source: §3 · p. 4
4. Conclusion
ReMOTS wins CVPR 2020 MOTS Challenge 1. Its contributions include intra-frame and inter-short-tracklet encoder training and statistical threshold selection; it is not usable in real-time scenarios.
4. Conclusion
- ReMOTS wins CVPR 2020 MOTS Challenge 1.Source: §4 · p. 4
- Intra-frame training and inter-short-tracklet training are introduced to learn appearance features for data association.Source: §4 · p. 4
- The method quantitatively analyzes tracklet statistics to select thresholds, an approach the authors say could be useful for other multiple object tracking works.Source: §4 · p. 4
- ReMOTS cannot be used in real-time scenarios, though it may offer insights for designing online MOTS methods with feature adaptation.Source: §4 · p. 4