PaperlayerView exact revision on arXiv

Source-linked AI summary

Improving Object Detector Training on Synthetic Data by Starting With a Strong Baseline Methodology

Frank A. Ruis, Alma M. Liezenga, Friso G. Heslinga, Luca Ballan, Thijs A. Eker, Richard J. M. den Hollander, Martin C. van Leeuwen, Judith Dijk, Wyke Huizinga

arXiv:2405.19822v1cs.CVcs.AIcs.ET

TL;DR

Synthetic-data object detection offers an alternative when real-world training data is costly or inaccessible, but bridging the reality gap remains difficult. The paper combines pretrained Transformer backbones with established augmentation practices for synthetic training. It reports improved results on VisDrone and RarePlanes and near-perfect performance on its in-house vehicle dataset, while noting limits in benchmark comparability and task scope.

  • Problem

    Synthetic-data object detection faces a reality gap, while established object-detection training practices have often received less attention than domain-transfer methods.

    Source: §1 · p. 1

  • Method

    The paper combines pretrained Transformer backbones with object-detection training practices, including strong augmentation, for synthetic-data training.

    Source: §1 · p. 2, §3.1 · p. 3, §3.3 · p. 5

  • Results

    The authors report improved synthetic-trained detection on VisDrone and RarePlanes and near-perfect real-set performance on the in-house vehicle dataset.

    Source: §6 · pp. 8–9, §5.3 · p. 8

  • Takeaways & Limitations

    Transformer shape bias and carefully selected augmentation can be useful for synthetic-trained detection when object shape is salient.

    Source: §6 · pp. 8–9

    The study focuses on object-detection tasks where shape is salient, and the lack of stronger synthetic-to-real benchmarks limited fair comparisons.

    Source: §6.2 · p. 9

Abstract

from arXiv · show

Collecting and annotating real-world data for the development of object detection models is a time-consuming and expensive process. In the military domain in particular, data collection can also be dangerous or infeasible. Training models on synthetic data may provide a solution for cases where access to real-world training data is restricted. However, bridging the reality gap between synthetic and real data remains a challenge. Existing methods usually build on top of baseline Convolutional Neural Network (CNN) models that have been shown to perform well when trained on real data, but have limited ability to perform well when trained on synthetic data. For example, some architectures allow for fine-tuning with the expectation of large quantities of training data and are prone to overfitting on synthetic data. Related work usually ignores various best practices from object detection on real data, e.g. by training on synthetic data from a single environment with relatively little variation. In this paper we propose a methodology for improving the performance of a pre-trained object detector when training on synthetic data. Our approach focuses on extracting the salient information from synthetic data without forgetting useful features learned from pre-training on real images. Based on the state of the art, we incorporate data augmentation methods and a Transformer backbone. Besides reaching relatively strong performance without any specialized synthetic data transfer methods, we show that our methods improve the state of the art on synthetic data trained object detection for the RarePlanes and DGTA-VisDrone datasets, and reach near-perfect performance on an in-house vehicle detection dataset.

1. INTRODUCTION

Synthetic data can scale object-detection training, but models still face a gap between synthetic and real imagery. This paper proposes combining a Transformer backbone with established training practices to improve synthetic-data training.

1. INTRODUCTION

  • The reality gap remains a central obstacle despite synthetic data’s scalability, cost-effectiveness, and environmental control.The paper also notes that data diversity and established training practices such as augmentation and task-specific optimization matter for generalization.Source: §1 · p. 1
  • CNN texture bias may hinder synthetic-to-real generalization because synthetic imagery can struggle to reproduce real-world textures, whereas Transformer shape bias aligns with synthetic data’s geometric strengths.The paper characterizes CNNs as emphasizing texture-related features and Transformers as emphasizing shape-based features.Source: §1 · p. 2
  • The paper aims to establish a strong synthetic-data baseline by combining Transformer shape bias with traditional object-detection practices such as strong augmentation.Source: §1 · p. 2

2. RELATED WORK

The related work covers object-detection architectures, synthetic-data challenges, and augmentation practices. It identifies a gap between established real-data training practices and how synthetic-data detection is commonly approached.

2. RELATED WORK

  • The literature review surveys object detection, synthetic data, and augmentation as approaches relevant to improving model performance.Source: §2 · p. 2

2.1 Object detection

  • CNNs are common detection backbones, while Transformers use attention and are associated with shape bias and early long-range connections.The review also describes two-stage and single-stage detectors and multi-scale feature methods.Source: §2.1 · p. 2

2.2 Synthetic data

  • Synthetic data includes generated, modified, or combined imagery, but building varied datasets and bridging the reality gap remain challenges.Prior approaches include varying viewpoints, scale, clutter, occlusions, and domain randomization.Source: §2.2 · p. 2

2.3 Data augmentation

  • Augmentation can increase training-data size and diversity, and RandAugment reduces the search space while selecting transformations against validation accuracy.The review notes that Transformer backbones may rely more on augmentation or regularization than CNNs.Source: §2.3 · p. 3

3. METHODOLOGY

The methodology uses pretrained Transformer backbones, selected synthetic-to-real dataset pairs, and augmentation methods. The dataset descriptions and augmentation discussion establish the training settings and their variation across tasks.

3. METHODOLOGY

  • The methodology covers backbone architecture, augmentation, and dataset quality and selection for the experiments.Source: §3 · p. 3

3.1 Backbone Architecture

  • Swin Transformer is used as a compute-conscious Transformer backbone, with model size chosen to approximate the convolutional baseline’s parameter count.Its window-based attention is presented as a trade-off against the computational cost of full-attention Transformers.Source: §3.1 · p. 3

3.2 Data

  • The authors identify dataset variety and label quality as concerns, including VisDrone labels for objects obscured by foliage.They evaluate DGTA-VisDrone, RarePlanes, and an in-house vehicle dataset in synthetic-to-real settings.Source: §3.2 · pp. 3–4

3.2.1 VisDrone

  • DGTA-VisDrone supplies 50,000 synthetic drone-view images but covers only 8 of 10 VisDrone classes; prior synthetic training reached 10.2% mAP@50.The real benchmark spans 14 cities, varied conditions and viewpoints, and 10,209 images.Source: §3.2.1 · p. 4, Figure 1 · p. 4

3.2.2 RarePlanes

  • For civilian-aircraft size classification (small, medium, or large), prior studies reported 75% mAP and 93% mAP@50 on real data, versus 35.9% mAP and 59.1% mAP@50 with synthetic-only training.RarePlanes contains 50,000 synthetic satellite images captured at 0.3m resolution.Source: §3.2.2 · p. 4

3.2.3 Vehicle Detection

  • The in-house vehicle dataset contains 1,600 synthetic and 333 real-world images across four vehicle types, with simulation axes tuned for synthetic-training mAP.Its construction uses one 3D model per vehicle type composited onto real photographs.Source: §3.2.3 · p. 5, Figure 3 · p. 5

3.3 Augmentations

  • The default pipeline uses MixUp, Mosaic, and large-scale jittering, while RandAugment selects among photometric distortions.Augmentations improved the Transformer baseline in one experiment but were not universally beneficial, so the set should be adjusted to the task.Source: §3.3 · p. 5, Figure 4 · p. 6

4. EXPERIMENTS

The experiments compare synthetic-trained detectors against benchmarks across VisDrone, RarePlanes, and vehicle detection. Model sizes, augmentation choices, and evaluation metrics differ by dataset and comparison.

4. EXPERIMENTS

  • The experiments use three synthetic/real dataset pairs and compare with published or study-trained baselines, selecting Swin counterparts of roughly equal or lower parameter count.Reported metrics depend on what comparative work makes available and what represents each experiment.Source: §4 · p. 6

4.1 VisDrone

  • VisDrone compares ResNeXt-101 and YOLOv5-X benchmarks with Swin-S, an unaugmented Swin-S, and a larger DINO with Swin-L, reporting mAP@50.Swin-S has 66.1M parameters; the authors omit mAP because the original publication did not provide it.Source: §4.1 · p. 6

4.2 RarePlanes

  • RarePlanes evaluates civilian-aircraft size classification against the original ResNet-50 benchmark and reports mAP.Because imagery has consistent resolution and lacks cloud cover or occlusion, the authors use RandAugment alone in their best run; large-scale jittering and MixUp harmed accuracy.Source: §4.2 · p. 6

4.3 Vehicle Detection

  • Vehicle detection compares a parameter-matched ResNet-101 with Transformer and larger ResNeXt-101 64x4d/Cascade R-CNN models, reporting both mAP and mAP@50.The authors emphasize mAP@50 because mAP can penalize detections that omit vehicle subparts they do not consider crucial.Source: §4.3 · p. 7

5. RESULTS

Across the three experiments, the Transformer approach achieved near-perfect performance on the in-house vehicle dataset, improved VisDrone benchmark results with augmentation, and yielded only slight gains on RarePlanes.

5.1 VisDrone

  • Transformer architecture plus augmentation significantly outperformed VisDrone benchmarks; the low-augmentation Transformer confirms augmentation remains crucial for high detection performance.A high-capacity DINO with a Swin-L backbone also reached impressive performance on real VisDrone data despite suboptimal synthetic training data.Source: Table 1 · p. 7, §5.1 · p. 7

5.2 RarePlanes

  • The Transformer produced only a slight mAP improvement over baseline; size-based airplane classes limited the advantage of shape bias despite distinctive airplane outlines.Tiled test images left some planes partly visible, making size judgments and bounding-box corners ambiguous; neither dataset included occlusion or cloud cover.Source: §5.2 · pp. 7–8, Table 2 · p. 8

5.3 Vehicles Detection

  • Near-perfect real-dataset detection was reached by the synthetic-trained Transformer, which also outperformed ResNeXt-101 64x4d at almost half its parameter count.The authors suggest the shape-focused vehicle imagery, which lacked depth and shadow, may have favored the Transformer.Source: Table 3 · p. 8, §5.3 · p. 8

6. DISCUSSION

The discussion finds that Transformer backbones and augmentation can improve synthetic-data object detection, but their value depends on the dataset and task. It also identifies limits in the available datasets and benchmarks and outlines directions for further study.

6. DISCUSSION

  • VisDrone performance improved with augmentations and a Transformer backbone individually and combined, while VisDrone-effective augmentations hurt RarePlanes performance.The discussion says backbone choice and augmentation need to be aligned with the task.Source: §6 · p. 8
  • Near-perfect real-dataset performance was reached using only synthetic training data, and the Transformer far outperformed a convolutional backbone with nearly twice as many parameters.The vehicle dataset used crude 3D models limited in quantity and quality; the authors expect shape bias to have strongly affected this result.Source: §6 · p. 8
  • Transformers are a potent synthetic-data backbone for shape-focused detection, but less suited to tasks with ambiguous shapes or a strong focus on texture.Augmentations can also help shape-focused detection, but should be selected for the dataset.Source: §6 · pp. 8–9

6.1 Limitations

  • The available synthetic-real dataset pairs are limited in availability, size, and quality, and the lack of adapted benchmark models makes fair comparisons difficult.RarePlanes has size-based class distinctions, no occlusion cases, and ambiguous test labels from image tiling; VisDrone's synthetic counterpart has label noise and lacks proper class mapping.Source: §6.1 · p. 9

6.2 Future work

  • Future work includes higher-quality synthetic-real dataset pairs and stronger benchmarks, alongside comparisons of Transformer, convolutional, and hybrid architectures across tasks.The authors also propose investigating how datasets built from a single 3D object differ from those representing an entire 3D world, and whether full-attention Transformers improve dense prediction.Source: §6.2 · p. 9

Related papers

Loading 2405.19822v1…