PaperlayerView exact revision on arXiv

Source-linked AI summary

MOTSynth: How Can Synthetic Data Help Pedestrian Detection and Tracking?

Matteo Fabbri, Guillem Braso, Gianluca Maugeri, Orcun Cetintas, Riccardo Gasparini, Aljosa Osep, Simone Calderara, Laura Leal-Taixe, Rita Cucchiara

arXiv:2108.09518v1cs.CV

TL;DR

Crowded-scene pedestrian detection and tracking need substantial training data, but recording public environments raises privacy concerns and annotation is costly. The paper presents MOTSynth, a large and diverse synthetic dataset, and reports strong synthetic-only results across multiple tasks, with fine-tuning still needed for extremely crowded MOT20.

  • Problem

    Deep-learning pedestrian detection and tracking are data-hungry, while collecting and labeling crowded public-scene footage is difficult, expensive, and privacy-sensitive.

    Source: §1 · p. 1

  • Method

    The authors create MOTSynth, a diverse synthetic dataset with annotations for pedestrian detection, tracking, and segmentation, using varied virtual-world scenarios.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    Synthetic-trained models are on-par with state-of-the-art on MOT17 and MOT20, but extremely crowded MOT20 still requires fine-tuning.

    Source: §1 · p. 2

  • Takeaways & Limitations

    MOTSynth can substitute for real-world data in several high-level tasks and scenarios, including pedestrian detection, re-identification, tracking, and segmentation.

    Source: §1 · p. 2, §5 · p. 8

    Models trained solely on MOTSynth are not yet on-par with state-of-the-art on MOT20; fine-tuning on MOT20 is needed to surpass Tracktorv2.

    Source: §4.6 · p. 8

Abstract

from arXiv · show

Deep learning-based methods for video pedestrian detection and tracking require large volumes of training data to achieve good performance. However, data acquisition in crowded public environments raises data privacy concerns -- we are not allowed to simply record and store data without the explicit consent of all participants. Furthermore, the annotation of such data for computer vision applications usually requires a substantial amount of manual effort, especially in the video domain. Labeling instances of pedestrians in highly crowded scenarios can be challenging even for human annotators and may introduce errors in the training data. In this paper, we study how we can advance different aspects of multi-person tracking using solely synthetic data. To this end, we generate MOTSynth, a large, highly diverse synthetic dataset for object detection and tracking using a rendering game engine. Our experiments show that MOTSynth can be used as a replacement for real data on tasks such as pedestrian detection, re-identification, segmentation, and tracking.

1. Introduction

The paper asks whether synthetic data can advance pedestrian detection and tracking amid costly, data-hungry training and privacy constraints on recording public scenes. It introduces MOTSynth and reports that synthetic-only training can match state-of-the-art results in several settings, with fine-tuning still needed for extremely crowded MOT20.

  • Deep-learning methods are data-hungry, while acquiring public-scene imagery raises privacy concerns and collecting and labeling data are difficult and expensive.Source: §1 · p. 1
  • MOTSynth is presented as a diverse synthetic dataset intended to replace recorded data for pedestrian detection, tracking, and segmentation.Source: §1 · p. 2
  • Synthetic-only models are on-par with state-of-the-art on MOT17 and MOT20, although extremely crowded MOT20 still requires fine-tuning.Source: §1 · p. 2
  • The dataset contains more than 1.3 million densely annotated frames and 40 million pedestrian instances.Source: §1 · p. 2

2. Related Work

Related work spans multi-object tracking benchmarks, re-identification datasets, and synthetic data for computer-vision tasks. The paper situates MOTSynth against these resources as an effort to replace recorded training data for pedestrian detection, re-identification, and tracking.

  • KITTI offers bounding-box and segmentation labels but uses single-city, clear-weather car-mounted footage, whereas BDD100k covers over 100K videos with geographic, environmental, and weather diversity.Source: §2 · p. 2
  • MOTChallenge [21] provides consistently labeled crowded tracking sequences, with MOT17 [61] capturing severe occlusions and scale variation and MOT20 [22] targeting extremely dense scenes.Source: §2 · p. 2
  • Learned ReID features support several state-of-the-art trackers [8], while DukeMTMC was taken offline over privacy concerns and Market1501 and CUHK03 are commonly used ReID datasets.Source: §2 · p. 2
  • Synthetic worlds have supported low- and higher-level vision tasks, but apart from optical flow, prior attempts had not fully replaced manually labeled data.Source: §2 · pp. 2–3

3. MOTSynth Dataset

MOTSynth is a synthetic dataset for pedestrian detection, tracking, and segmentation, generated from manually configured virtual-world scenarios and varied renderings. Its scale, scenario diversity, and annotations distinguish it from prior datasets, including JTA.

3.1. Dataset Generation

  • The team manually configured camera viewpoints, pedestrian behaviors, and routes across 130 km2 of the GTA-V world, then varied attributes across 9,519 identities and 384 screenplays.Source: §3.1 · p. 3
  • Weather and daytime were randomized, producing 768 sequences, each with precise 3D visibility annotations, temporally consistent 2D boxes, segmentation masks, and depth maps.Source: §3.1 · p. 3

3.2. Statistical Analysis

  • MOTSynth sequences average 29.5 people per frame, reach 125 people, and comprise over 1.3M densely annotated frames.Source: §3.2 · p. 3
  • The authors report that MOTSynth exceeds prior datasets in size, instances, and labels, and Table 1 summarizes frames, instances, and label availability across datasets.Source: §3.2 · p. 4, Table 1 · p. 4
  • Compared with JTA [27], MOTSynth is larger, focuses on crowded pedestrian scenarios, and adds instance segmentation and scene-depth information.Source: §3.2 · p. 4
  • MOTSynth contains 40M bounding boxes with tracking and segmentation labels, one to three orders of magnitude more than the manually labeled MOTChallenge datasets.Source: §3.2 · p. 4

4. Experimental Evaluation

The evaluation tests MOTSynth-trained models for pedestrian detection, re-identification, tracking, and tracking with segmentation on MOTChallenge benchmarks, with real-data comparisons and fine-tuning experiments. Synthetic training is competitive across these tasks, while MOT20 results still require fine-tuning to reach state-of-the-art performance.

4.2. People Detection

  • On MOT17, MOTSynth–4 improved AP over COCO by +2.49 for YOLOv3, +3.48 for CenterNet, +2.3 for Faster R-CNN, and +1.87 for Mask R-CNN.The authors report consistent synthetic-to-real gains across detectors; Table 2 compares COCO and four MOTSynth subsets on MOT17 and MOT20.Source: §4.2 · p. 4, Table 2 · p. 5
  • The reported detector results show improvements of +10.97 AP for YOLOv3, +12.31 AP for CenterNet, +10.23 AP for Faster R-CNN, and +10.3 AP for Mask R-CNN.On the smallest MOTSynth–1 subset, 1/10 sampling reached 76.8 AP, matching the COCO model’s 76.69 AP; denser, less diverse sampling yielded 70 AP.Source: §4.2 · p. 5

4.3. Person Re-Identification

  • MOTSynth-trained ReID models outperformed real-data models on MOT17: the first split gained +6.9 mAP over Market1501 and +2.5 over combined Market1501/CUHK03.Using the first two MOTSynth splits, the reported mAP gains were +8.6 and +4.2, and Rank-1 gains were +3.5 and +1, respectively.Source: §4.3 · p. 6, Table 5 · p. 6

4.5. Multi-object Tracking and Segmentation

  • In tracking with segmentation on MOTS20, MOTSynth improved sMOTSA, MOTSA, and IDF1 by 0.5 for Lift T and MPNTrack, while CenterTrack and Tracktor dipped by 0.1.A Mask R-CNN Tracktor trained on MOTSynth alone outperformed Tracktor v2++ by +4 IDF1, +0.5 sMOTSA, and +0.7 MOTSA.Source: Table 7 · p. 7, §4.5 · p. 8

4.6. Benchmark Results

  • Fine-tuned CenterTrack reached a new MOT17 state of the art with 65.1 MOTA and 57.9 IDF1.Tracktor trained only on MOTSynth also outperformed COCO-trained, MOT17-fine-tuned Tracktor by +3.4 MOTA and +4.6 IDF1.Source: §4.6 · p. 8, Table 8 · p. 8
  • On MOT20, MOTSynth-trained trackers were not yet state-of-the-art without fine-tuning; after fine-tuning, they surpassed Tracktorv2 by +3.9 MOTA and +0.1 IDF1.The authors suggest the gap reflects MOTSynth's lack of sequences as extremely crowded as MOT20's.Source: §4.6 · p. 8, Table 8 · p. 8
  • On MOTS20, synthetic-only Tracktor† Mask R-CNN exceeded TrackR-CNN by +14.45 sMOTSA, +15.04 MOTSA, +3.47 MOTSP, and +21.47 IDF1.The TrackR-CNN baseline was trained on COCO, Mapillary Vistas, and MOTS20 training data.Source: §4.6 · p. 8, Table 9 · p. 8

5. Conclusion

MOTSynth is a massive synthetic dataset for pedestrian detection and tracking in urban scenarios. Experiments show synthetic data can substitute for real data across several high-level tasks, with state-of-the-art results on MOT17 using synthetic-only training.

  • Training recent methods solely on synthetic data yielded state-of-the-art results on MOTChallenge MOT17.Source: §5 · p. 8
  • MOTSynth experimentally replaced real data for pedestrian detection, re-identification tracking, and segmentation in high-level in-the-wild scenarios.Source: §5 · p. 8
  • The paper presents MOTSynth as a massive synthetic dataset for pedestrian detection and tracking in urban scenarios.Source: §5 · p. 8

Supplementary Material

The supplementary material expands the dataset comparison, visualizations, statistics, data-volume and diversity trade-offs, implementation details, and MOT20 sequence results.

  • The supplement extends the main-paper dataset comparison and adds dataset visualizations and statistics.Source: A. Overview · p. 13
  • It reports additional experiments on trade-offs between data volume and diversity.Source: A. Overview · p. 13
  • It provides implementation details for the experiments and MOT20 benchmark results for each sequence.Source: A. Overview · p. 13

B. Dataset Comparison

Compared with real-world urban-surveillance datasets, MOTSynth has about an order of magnitude more clips, annotated frames, and instances. It also offers 3D pose annotations, instance segmentation labels, and depth maps.

C. Dataset Visualizatons and Statistics

MOTSynth varies across weather, lighting, viewpoints, and pedestrian density, and its statistics indicate small pedestrian boxes, crowded frames, varied camera distances, and substantial occlusion.

D. Data Volume and Diversity

The experiments find that data volume and visual diversity are both important for competitive detection, and that sparse sampling can preserve performance when it increases diversity.

D. Data Volume and Diversity

  • YOLOv3 requires 104k images to perform favourably with respect to COCO, and higher sampling rates benefit both AP and MODA.For CenterNet, sampling rate does not affect AP, while higher data volume appears beneficial for MODA.Source: D. Data Volume and Diversity · p. 13
  • CenterNet surpasses COCO training with 17k images, and the 17k-image split 4 outperforms the 52k-image split 3.The comparison is attributed to split 4's greater visual diversity.Source: D. Data Volume and Diversity · p. 13
  • With 9k images, Faster R-CNN achieves higher AP than real-data training, while results appear to saturate on larger splits.Split 2 at 1/10 sampling (24k frames) and splits 3 and 4 at 1/60 (9k and 17k frames) obtain similar performance for Faster R-CNN and YOLOv3.Source: D. Data Volume and Diversity · p. 13
  • Visual diversity and data volume are equally important, with the best performance when both are maximized.The analysis states that diverse, smaller MOTSynth subsets can bridge the synthetic-to-real gap.Source: D. Data Volume and Diversity · p. 13

E.1. Object Detection Experiments

The appendix describes detection, re-identification, and tracking experiment configurations, alongside dataset overviews and examples of MOTSynth's visual variety.

E.1. Object Detection Experiments

  • Object-detection experiments use YOLOv3, CenterNet, Faster R-CNN, and Mask R-CNN, with distinct training and inference settings.The specified configurations include YOLOv3 at 200,000 iterations, CenterNet at 100,000 iterations, and Faster R-CNN at 35,000 iterations; Faster and Mask R-CNN use backbone freezing to avoid overfitting.Source: §E.1 · p. 13
  • MOTSynth examples depict variation in weather, lighting, viewpoints, and the number of people; the dataset overview compares annotation and data-type availability.The overview lists clips, annotated frames, instances, 3D data, occlusion information, pose, instance segmentation, depth, and data type.Source: Figure 3 · p. 14, Table 10 · p. 14

E.2. Person Re-Identification Experiments

  • The ReID experiments freeze CNN layers for five epochs of fully connected-layer pretraining, then train the full model for 55 more epochs.Images are resized to 128x56 with random cropping and flipping.Source: §E.2 · p. 14

E.3. Multi-Object Tracking Experiments

  • Tracking experiments fine-tune CenterTrack for 30 epochs on MOT17 and 70 on MOT20; Tracktor uses a Faster R-CNN regression head trained on static images.The Mask R-CNN adaptation uses bounding-box regression for tracking and its mask head for segmentation; additional statistics cover pedestrians with at least one visible joint.Source: §E.3 · pp. 14–15, Figure 4 · p. 15

F. Detailed Benchmark Results

Synthetic-only Tracktor and CenterTrack are competitive across most MOT20 sequences, with a notable exception where near-bird’s-eye viewpoints are rare in MOTSynth.

F. Detailed Benchmark Results

  • On MOT20-04, synthetic-only Tracktor and CenterTrack do not match Tracktorv2, MPNTrack, or LPC, likely because near-bird’s-eye viewpoints are rare in MOTSynth.The comparison concerns extremely crowded scenes.Source: F. Detailed Benchmark Results · p. 15
  • The sampling-density comparison indicates that sparse sampling can bridge the synthetic-to-real gap when training images remain diverse.Table 11 describes the effect of density in sampled data and relates sparser sampling to increased diversity.Source: Table 11 · p. 16
  • Outside MOT20-04, synthetic-only Tracktor and CenterTrack outperform Tracktorv2 by a significant margin and are on par with the state of the art.The per-sequence MOT20 benchmark results are shown in Table 12; fine-tuning on MOT20 further improves performance.Source: F. Detailed Benchmark Results · p. 15, Table 12 · p. 16
  • The detailed benchmark tables report results for MOT17 and MOT20.Table 13 covers MOT17 and Table 14 covers MOT20.Source: Table 13 · p. 16, Table 14 · p. 16
Loading 2108.09518v1…