MOTS: Multi-Object Tracking and Segmentation

• Bounding-box tracking is saturating, motivating a move toward pixel-level representations.Source: §1 · p. 1
• Pixel-wise masks describe partially occluded objects more precisely than boxes and avoid overlap ambiguities during evaluation.Masks provide a well-defined ground truth and produce non-overlapping tracking results.Source: §1 · p. 1, Figure 1 · p. 1
• MOTS extends multi-object tracking to instance segmentation tracking, combining temporal and mask cues.Source: §1 · pp. 1–2
• The paper contributes two temporally consistent datasets, the sMOTSA metric, and TrackR-CNN for joint detection, tracking, and segmentation.Source: §1 · p. 2
• Training on MOTS annotations enables joint segmentation and tracking and improves performance over training only for instance segmentation or bounding-box tracking.Source: §1 · p. 2

Multi-Object Tracking Datasets

• MOT datasets track an initially unknown number of targets as bounding boxes through entrances, exits, occlusions, and appearance changes.Source: §2 · p. 2
• MOTS focuses on predefined classes in crowded scenes while adding the challenge of discovering and tracking objects as they appear and disappear.Source: §2 · p. 2

Video Object Segmentation

• Video object segmentation datasets include single-object and multi-object benchmarks such as DAVIS, YouTube-VOS, Segtrackv2, FBMS, and YouTube-Objects.Source: §2 · p. 2

Mask-Based Tracking

• Earlier mask-tracking methods use superpixels, conditional random fields, or given detections, whereas the proposed baseline operates directly at pixel level.Source: §2 · p. 2
• The proposed baseline learns association features jointly with the detector instead of applying association only as post-processing to detections.Source: §2 · p. 2

Semi-Automatic Annotation

• Existing semi-automatic annotation methods require input per object or operate on isolated frames rather than exploiting track-level information.Source: §2 · p. 3

• 65,213 segmentation masks were added to two MOT datasets because frame-by-frame pixel annotation is extremely time-consuming.The resulting dataset size is described as viable for training and evaluating modern learning-based techniques.Source: §3 · p. 3

Semi-Automatic Annotation Procedure

• The annotation pipeline predicts masks from bounding boxes with a convolutional refinement network, followed by manual polygon correction and iterative refinement.Source: §3 · p. 3
• The refinement network uses an image crop with context and a box-encoding channel to predict the segmentation mask for each box.Source: §3 · p. 3
• Two manually annotated masks per object initialize training, after which an object-specific network is fine-tuned and applied across that object's bounding boxes.The two selected frames are chosen for diversity.Source: §3 · p. 3

Dataset Composition

• KITTI MOTS contains pedestrians and cars, while MOTSChallenge contains pedestrians, as summarized in the dataset statistics table.Source: Table 1 · p. 4

MOTSChallenge

• MOTSChallenge annotates 4 of 7 MOTChallenge training sequences and targets pedestrians in crowded scenes with many occlusions.Source: §3 · p. 4

Mask-based evaluation

• MOTS evaluation adapts CLEAR MOT metrics to pixel masks while requiring non-overlapping ground-truth and predicted masks.This unique-pixel constraint simplifies correspondences because at most one predicted mask can overlap a ground-truth mask.Source: §4 · p. 4

Formal setup

• A video is represented by ground-truth masks and hypotheses, each assigned a frame and track identity.Ground-truth masks form M, while predicted masks form H; both are non-empty pixel masks associated with time frames and track IDs.Source: §4 · p. 4

Correspondences

• True positives are matched hypotheses, false positives are unmatched hypotheses, and false negatives are uncovered ground-truth masks.These sets support the adapted tracking and segmentation metrics.Source: §4 · p. 4

Mask-based metrics

• MOTSA is a mask-IoU version of the box-based MOTA metric, while MOTSP provides mask-based tracking and segmentation precision.The measures extend established CLEAR MOT quantities to mask-based evaluation.Source: §4 · p. 4

Soft accuracy

• sMOTSA uses a soft true-positive count rather than counting only masks with IoU above 0.5, measuring segmentation, detection, and tracking jointly.The soft count accumulates mask overlap quality.Source: §4 · p. 5

Architecture

• TrackR-CNN extends Mask R-CNN with an association head and two 3D convolutional layers for temporal detection association.The network outputs mask-based detections and association features for downstream tracking.Source: §5 · p. 5

Association head

• The association head predicts an association vector for each proposal, training same-instance vectors to be close and different-instance vectors to be distant.The vectors represent object identity and use Euclidean distance for comparison.Source: §5 · p. 5

Association learning

• Batch-hard triplet loss adapts hard-positive and hard-negative sampling to video detections with a margin α.Each detection supplies a mask, association vector, time frame, and ground-truth track identity.Source: §5 · p. 5

Mask propagation

• Mask propagation experiments use mask IoU and optical-flow warping as an alternative cue for associating masks across adjacent frames.The score compares a warped previous mask with a current detection mask.Source: §5 · pp. 5–6

Tracking

• Tracking links confident current detections to recent tracks using association distances, Hungarian matching, and thresholds for history, distance, and confidence.Only detections within the configured temporal and distance limits are matched.Source: §5 · p. 6

Output constraints

• When resulting tracks contain overlapping masks, higher-confidence detections take precedence so the final MOTS output remains non-overlapping.Confidence comes from the network’s classification head.Source: §5 · p. 6

Main Results

• TrackR-CNN achieves higher sMOTSA and MOTSA than Mask R-CNN + maskprop on KITTI MOTS, while MOTSP remains similar.The comparison indicates benefits from the association head and 3D convolution layers for identifying objects in video.Source: §6 · p. 6

Box-Based Baselines

• Bounding-box tracking followed by mask generation performs worse than TrackR-CNN, especially for pedestrians, and often causes confusion, missed masks, and identity switches.The box-based baseline uses original, non-tight annotations and adds masks only as post-processing.Source: §6 · pp. 6–7, Figure 4 · p. 7

Training on MOTS Data

• Training on temporally consistent instance-segmentation data improves over training on image-only instance-segmentation data or bounding-box tracking data.The experiments identify joint training on MOTS annotations as the source of these advantages.Source: §6 · p. 7

Comparison with Existing Trackers

• TrackR-CNN outperforms CIWT when both generate masks from the same TrackR-CNN detections.CIWT uses stereo-based 3D information, whereas TrackR-CNN jointly addresses tracking and mask generation.Source: §6 · p. 7

Temporal Component

• Two 3D convolutions significantly improve pedestrian sMOTSA and MOTSA over no temporal component, while car performance remains comparable.Convolutional LSTM does not yield significant gains in the reported comparison.Source: Table 3 · p. 7, §6 · p. 8

Association Mechanisms

• The association head outperforms mask-propagation cues, and removing association loss further degrades MOTSA scores.Bounding-box IoU performs almost the same as mask IoU, while nearest-neighbor matching by box-center distance significantly degrades performance.Source: §6 · p. 8, Table 4 · p. 8

MOTSChallenge

• TrackR-CNN outperforms four MOT17 methods on MOTSChallenge despite their use of strong public detections, although accurate pedestrian segmentation remains difficult even with tight ground-truth boxes.TrackR-CNN was trained and tuned in a leaving-one-out fashion because MOTSChallenge contains four video sequences.Source: §6 · p. 8, Table 5 · p. 8

Conclusion

• The paper introduces two semi-automatically annotated MOTS datasets, MOTSA and sMOTSA metrics, and TrackR-CNN as a baseline.These resources address the absence of a benchmark and dataset for training and evaluating temporally consistent mask-based tracking.Source: §7 · p. 8
• Training on MOTS data enables TrackR-CNN to outperform comparable methods trained only with bounding-box tracks and single-image instance-segmentation masks.The datasets make joint training possible and open opportunities for future research.Source: §7 · p. 8

Association loss

• TrackR-CNN assigns each detection an association vector and ground-truth track identity for training the association head.Only detections sufficiently overlapping a ground-truth object are considered.Source: A. Losses for the Association Head · p. 11
• Batch-hard triplet loss selects the hardest positive and hardest negative for each detection anchor, enforcing a margin α between their distances.The hard positive shares the anchor’s identity, while the hard negative has a different identity.Source: A. Losses for the Association Head · p. 11

Alternative losses

• Batch-all triplet and contrastive loss provide alternative association-loss formulations evaluated against the batch-hard formulation.Batch-all considers all detection pairs.Source: A. Losses for the Association Head · p. 11

Results

• Batch-hard triplet loss performs better than Batch All Triplet and Contrastive on the KITTI MOTS validation set.Contrastive loss is especially weak for pedestrians.Source: A. Losses for the Association Head · p. 11, Table 6 · p. 11

Annotation considerations

• The annotation process corrected original bounding boxes when imprecise annotations produced wrong segmentation results.KITTI tracking boxes can be amodal and extend to the ground when only a person’s upper body is visible.Source: B. Details of the Annotation Procedure · p. 11

Experimental setup

• The experiments compare masks derived from original or adjusted KITTI boxes using filling, ellipses, or a fine-tuned Mask R-CNN.Instance identities are retained from the corresponding bounding boxes.Source: C. Ground Truth Experiments · pp. 11–12

Findings

• Rectangles and ellipses cannot accurately localize objects for mask-based matching, even with perfect track hypotheses.The limitation concerns mask quality rather than track identity assignment.Source: C. Ground Truth Experiments · p. 12
• Amodal bounding boxes amplify localization errors because they often include large regions where the object is not visible.The experiments therefore support pixel-wise evaluation for MOT tasks.Source: C. Ground Truth Experiments · p. 12
• The relatively low pedestrian scores indicate a limit to generating masks post hoc with the KITTI-fine-tuned Mask R-CNN.This limitation is reported for the mask-generation approach evaluated in the ground-truth experiments.Source: C. Ground Truth Experiments · p. 12

Association visualization

• PCA visualization groups detections with similar appearance using TrackR-CNN association vectors and association-loss minimization.The visualization uses detections from sequence 18 of KITTI MOTS.Source: Figure 5 · p. 13

MOTSChallenge

• Complex MOTSChallenge scenes can work well but still exhibit missing detections and ID switches during difficult occlusions.The failures are highlighted in the qualitative examples.Source: E. Qualitative results · p. 12, Figure 6 · p. 14

KITTI MOTS

• In simpler KITTI MOTS scenes, the model can continue a track with the same ID after a missing detection.This behavior is shown in a qualitative example.Source: Figure 7 · p. 15
• A rare KITTI MOTS failure occurs when pylons are confused with pedestrians.Most detections nevertheless correspond to real instances of the target class.Source: Figure 8 · p. 16
• Object distinction works well in less crowded scenes, although erroneous detections can still occur.The erroneous detections are highlighted in the qualitative results.Source: Figure 9 · p. 17