PaperlayerView exact revision on arXiv

Source-linked AI summary

MOTS: Multi-Object Tracking and Segmentation

Paul Voigtlaender, Michael Krause, Aljosa Osep, Jonathon Luiten, Berin Balachandar Gnana Sekar, Andreas Geiger, Bastian Leibe

arXiv:1902.03604v2cs.CV

TL;DR

Multi-object tracking lacked temporally consistent pixel-level annotations and evaluation for segmentation. The paper creates two MOTS datasets, extends tracking metrics, and introduces TrackR-CNN for joint detection, tracking, and segmentation. Training on the new data outperforms comparable methods trained only with bounding-box tracks and single-image masks.

  • Problem

    Existing tracking datasets provide bounding boxes rather than pixel masks, while segmentation datasets generally lack video annotations and object identities across frames.

  • Method

    The paper augments two MOT datasets with semi-automatic pixel masks, introduces MOTSA and sMOTSA metrics, and proposes TrackR-CNN with temporal convolutions and an association head.

  • Results

    Training on MOTS data enables joint segmentation and tracking and outperforms comparable methods trained with bounding-box tracks and single-image instance segmentation masks.

  • Takeaways & Limitations

    The datasets, metrics, and baseline provide resources for developing multi-object tracking approaches beyond 2D bounding boxes.

  • Takeaways & Limitations

    The original KITTI tracking benchmark does not count identity switches when the target was lost by the tracker.

Abstract

from arXiv · show

This paper extends the popular task of multi-object tracking to multi-object tracking and segmentation (MOTS). Towards this goal, we create dense pixel-level annotations for two existing tracking datasets using a semi-automatic annotation procedure. Our new annotations comprise 65,213 pixel masks for 977 distinct objects (cars and pedestrians) in 10,870 video frames. For evaluation, we extend existing multi-object tracking metrics to this new task. Moreover, we propose a new baseline method which jointly addresses detection, tracking, and segmentation with a single convolutional network. We demonstrate the value of our datasets by achieving improvements in performance when training on MOTS annotations. We believe that our datasets, metrics and baseline will become a valuable resource towards developing multi-object tracking approaches that go beyond 2D bounding boxes. We make our annotations, code, and models available at https://www.vision.rwth-aachen.de/page/mots.

1. Introduction

MOTS extends multi-object tracking from bounding boxes to pixel-level instance segmentation because boxes become ambiguous during occlusion and tracking performance is saturating. The paper introduces datasets, metrics, and TrackR-CNN to jointly address detection, segmentation, and tracking.

  • Bounding-box tracking is saturating, motivating a move toward pixel-level representations.
  • Pixel-wise masks describe partially occluded objects more precisely than boxes and avoid overlap ambiguities during evaluation.Masks provide a well-defined ground truth and produce non-overlapping tracking results.
  • MOTS extends multi-object tracking to instance segmentation tracking, combining temporal and mask cues.
  • The paper contributes two temporally consistent datasets, the sMOTSA metric, and TrackR-CNN for joint detection, tracking, and segmentation.
  • Training on MOTS annotations enables joint segmentation and tracking and improves performance over training only for instance segmentation or bounding-box tracking.

2. Related Work

Related work spans bounding-box MOT, video object segmentation, video instance segmentation, mask-based tracking, and semi-automatic annotation. MOTS differs by requiring discovery, tracking, and pixel-level segmentation of multiple interacting objects.

  • Multi-Object Tracking Datasets: MOT datasets track an initially unknown number of targets as bounding boxes through entrances, exits, occlusions, and appearance changes.
  • Multi-Object Tracking Datasets: MOTS focuses on predefined classes in crowded scenes while adding the challenge of discovering and tracking objects as they appear and disappear.
  • Video Object Segmentation: Video object segmentation datasets include single-object and multi-object benchmarks such as DAVIS, YouTube-VOS, Segtrackv2, FBMS, and YouTube-Objects.
  • Mask-Based Tracking: Earlier mask-tracking methods use superpixels, conditional random fields, or given detections, whereas the proposed baseline operates directly at pixel level.
  • Mask-Based Tracking: The proposed baseline learns association features jointly with the detector instead of applying association only as post-processing to detections.
  • Semi-Automatic Annotation: Existing semi-automatic annotation methods require input per object or operate on isolated frames rather than exploiting track-level information.

3. Datasets

The paper creates MOTS datasets by augmenting existing MOT bounding boxes with temporally consistent pixel masks through semi-automatic, track-specific refinement. It introduces annotations for KITTI MOTS and MOTSChallenge, including challenging pedestrian occlusions.

  • 65,213 segmentation masks were added to two MOT datasets because frame-by-frame pixel annotation is extremely time-consuming.The resulting dataset size is described as viable for training and evaluating modern learning-based techniques.
  • Semi-Automatic Annotation Procedure: The annotation pipeline predicts masks from bounding boxes with a convolutional refinement network, followed by manual polygon correction and iterative refinement.
  • Semi-Automatic Annotation Procedure: The refinement network uses an image crop with context and a box-encoding channel to predict the segmentation mask for each box.
  • Semi-Automatic Annotation Procedure: Two manually annotated masks per object initialize training, after which an object-specific network is fine-tuned and applied across that object's bounding boxes.The two selected frames are chosen for diversity.
  • Dataset Composition: KITTI MOTS contains pedestrians and cars, while MOTSChallenge contains pedestrians, as summarized in the dataset statistics table.
  • MOTSChallenge: MOTSChallenge annotates 4 of 7 MOTChallenge training sequences and targets pedestrians in crowded scenes with many occlusions.

4. Evaluation Measures

The paper adapts CLEAR MOT evaluation to MOTS by matching non-overlapping masks and defining mask-based accuracy and precision measures, including a soft accuracy metric.

  • Mask-based evaluation: MOTS evaluation adapts CLEAR MOT metrics to pixel masks while requiring non-overlapping ground-truth and predicted masks.This unique-pixel constraint simplifies correspondences because at most one predicted mask can overlap a ground-truth mask.
  • Formal setup: A video is represented by ground-truth masks and hypotheses, each assigned a frame and track identity.Ground-truth masks form M, while predicted masks form H; both are non-empty pixel masks associated with time frames and track IDs.
  • Correspondences: True positives are matched hypotheses, false positives are unmatched hypotheses, and false negatives are uncovered ground-truth masks.These sets support the adapted tracking and segmentation metrics.
  • Mask-based metrics: MOTSA is a mask-IoU version of the box-based MOTA metric, while MOTSP provides mask-based tracking and segmentation precision.The measures extend established CLEAR MOT quantities to mask-based evaluation.
  • Soft accuracy: sMOTSA uses a soft true-positive count rather than counting only masks with IoU above 0.5, measuring segmentation, detection, and tracking jointly.The soft count accumulates mask overlap quality.

5. Method

TrackR-CNN jointly performs detection, tracking, and segmentation by extending Mask R-CNN with temporal convolutions and learned association features, supplemented by mask propagation and tracking rules.

  • Architecture: TrackR-CNN extends Mask R-CNN with an association head and two 3D convolutional layers for temporal detection association.The network outputs mask-based detections and association features for downstream tracking.
  • Association head: The association head predicts an association vector for each proposal, training same-instance vectors to be close and different-instance vectors to be distant.The vectors represent object identity and use Euclidean distance for comparison.
  • Association learning: Batch-hard triplet loss adapts hard-positive and hard-negative sampling to video detections with a margin α.Each detection supplies a mask, association vector, time frame, and ground-truth track identity.
  • Mask propagation: Mask propagation experiments use mask IoU and optical-flow warping as an alternative cue for associating masks across adjacent frames.The score compares a warped previous mask with a current detection mask.
  • Tracking: Tracking links confident current detections to recent tracks using association distances, Hungarian matching, and thresholds for history, distance, and confidence.Only detections within the configured temporal and distance limits are matched.
  • Output constraints: When resulting tracks contain overlapping masks, higher-confidence detections take precedence so the final MOTS output remains non-overlapping.Confidence comes from the network’s classification head.

6. Experiments

Experiments show that TrackR-CNN benefits from jointly trained mask and temporal cues, outperforming box-based or post-hoc mask baselines on KITTI MOTS and MOTSChallenge.

  • Main Results: TrackR-CNN achieves higher sMOTSA and MOTSA than Mask R-CNN + maskprop on KITTI MOTS, while MOTSP remains similar.The comparison indicates benefits from the association head and 3D convolution layers for identifying objects in video.
  • Box-Based Baselines: Bounding-box tracking followed by mask generation performs worse than TrackR-CNN, especially for pedestrians, and often causes confusion, missed masks, and identity switches.The box-based baseline uses original, non-tight annotations and adds masks only as post-processing.
  • Training on MOTS Data: Training on temporally consistent instance-segmentation data improves over training on image-only instance-segmentation data or bounding-box tracking data.The experiments identify joint training on MOTS annotations as the source of these advantages.
  • Comparison with Existing Trackers: TrackR-CNN outperforms CIWT when both generate masks from the same TrackR-CNN detections.CIWT uses stereo-based 3D information, whereas TrackR-CNN jointly addresses tracking and mask generation.
  • Temporal Component: Two 3D convolutions significantly improve pedestrian sMOTSA and MOTSA over no temporal component, while car performance remains comparable.Convolutional LSTM does not yield significant gains in the reported comparison.
  • Association Mechanisms: The association head outperforms mask-propagation cues, and removing association loss further degrades MOTSA scores.Bounding-box IoU performs almost the same as mask IoU, while nearest-neighbor matching by box-center distance significantly degrades performance.
  • MOTSChallenge: TrackR-CNN outperforms four MOT17 methods on MOTSChallenge despite their use of strong public detections, although accurate pedestrian segmentation remains difficult even with tight ground-truth boxes.TrackR-CNN was trained and tuned in a leaving-one-out fashion because MOTSChallenge contains four video sequences.

7. Conclusion

The paper introduces datasets, metrics, and a baseline for MOTS, enabling joint training and evaluation with temporally consistent masks. Training on this data outperforms comparable methods trained only on bounding-box tracks and single-image masks.

  • Conclusion: The paper introduces two semi-automatically annotated MOTS datasets, MOTSA and sMOTSA metrics, and TrackR-CNN as a baseline.These resources address the absence of a benchmark and dataset for training and evaluating temporally consistent mask-based tracking.
  • Conclusion: Training on MOTS data enables TrackR-CNN to outperform comparable methods trained only with bounding-box tracks and single-image instance-segmentation masks.The datasets make joint training possible and open opportunities for future research.

A. Losses for the Association Head

The association head learns track identities from detection embeddings using batch-hard triplet loss, which outperforms batch-all triplet and contrastive alternatives on KITTI MOTS.

  • Association loss: TrackR-CNN assigns each detection an association vector and ground-truth track identity for training the association head.Only detections sufficiently overlapping a ground-truth object are considered.
  • Association loss: Batch-hard triplet loss selects the hardest positive and hardest negative for each detection anchor, enforcing a margin α between their distances.The hard positive shares the anchor’s identity, while the hard negative has a different identity.
  • Alternative losses: Batch-all triplet and contrastive loss provide alternative association-loss formulations evaluated against the batch-hard formulation.Batch-all considers all detection pairs.
  • Results: Batch-hard triplet loss performs better than Batch All Triplet and Contrastive on the KITTI MOTS validation set.Contrastive loss is especially weak for pedestrians.
  • Annotation considerations: The annotation process corrected original bounding boxes when imprecise annotations produced wrong segmentation results.KITTI tracking boxes can be amodal and extend to the ground when only a person’s upper body is visible.

C. Ground Truth Experiments

Ground-truth experiments show that box-derived rectangles and ellipses remain inadequate for mask-based matching, even when tracking is perfect, especially with amodal boxes.

  • Experimental setup: The experiments compare masks derived from original or adjusted KITTI boxes using filling, ellipses, or a fine-tuned Mask R-CNN.Instance identities are retained from the corresponding bounding boxes.
  • Findings: Rectangles and ellipses cannot accurately localize objects for mask-based matching, even with perfect track hypotheses.The limitation concerns mask quality rather than track identity assignment.
  • Findings: Amodal bounding boxes amplify localization errors because they often include large regions where the object is not visible.The experiments therefore support pixel-wise evaluation for MOT tasks.
  • Findings: The relatively low pedestrian scores indicate a limit to generating masks post hoc with the KITTI-fine-tuned Mask R-CNN.This limitation is reported for the mask-generation approach evaluated in the ground-truth experiments.

E. Qualitative results

Qualitative results show that TrackR-CNN can maintain identities and distinguish objects in simpler scenes, while difficult occlusions and rare confusions still cause failures.

  • Association visualization: PCA visualization groups detections with similar appearance using TrackR-CNN association vectors and association-loss minimization.The visualization uses detections from sequence 18 of KITTI MOTS.
  • MOTSChallenge: Complex MOTSChallenge scenes can work well but still exhibit missing detections and ID switches during difficult occlusions.The failures are highlighted in the qualitative examples.
  • KITTI MOTS: In simpler KITTI MOTS scenes, the model can continue a track with the same ID after a missing detection.This behavior is shown in a qualitative example.
  • KITTI MOTS: A rare KITTI MOTS failure occurs when pylons are confused with pedestrians.Most detections nevertheless correspond to real instances of the target class.
  • KITTI MOTS: Object distinction works well in less crowded scenes, although erroneous detections can still occur.The erroneous detections are highlighted in the qualitative results.
Loading 1902.03604v2…