PaperlayerView exact revision on arXiv

Source-linked AI summary

G-MAD: A Game-Based Data Generation Framework for Multi-View RGB-T Aerial Object Detection

Yechan Kim, JongHyun Park, Dongho Yoon, Namhoon Jung, Moongu Jeon

arXiv:2607.19942v2cs.CVcs.AI

TL;DR

Aerial RGB-T dataset construction is constrained by limited viewpoint control, imperfect modality alignment, and costly annotation. G-MAD uses Arma3 to generate structured, synchronized multi-view RGB-T data with engine-grounded annotations, and releases AMOD for controlled benchmarking and transfer studies.

  • Problem

    Real-world aerial RGB-T datasets are costly to construct and often lack control over viewpoints, environmental factors, and modality alignment.

  • Method

    G-MAD transforms Arma3 into an end-to-end pipeline for structured scenario specification, synchronized multi-view RGB-T capture, and automatic geometric annotation.

  • Results

    Engine-grounded transformations produce geometrically consistent HBB and OBB annotations suitable as ground-truth labels for aerial object detection.

  • Takeaways & Limitations

    AMOD supports controlled benchmarking of viewpoint robustness and RGB-T learning, with preliminary evidence of transfer to real-world settings.

  • Takeaways & Limitations

    RGB images are not collected under nighttime conditions because Arma3’s visible nighttime rendering is usually unusable.

Abstract

from arXiv · show

This work introduces G-MAD, an open-source framework that uses Arma3 to generate synchronized multi-view RGB-T data for aerial object detection. G-MAD addresses key limitations of real-world aerial dataset construction, including limited viewpoint control, imperfect RGB-T alignment and high annotation cost. The framework supports structured scenario specification, controllable multi-view camera placement, simultaneous visible/thermal capture, and automatic bounding box annotation using engine-level geometric metadata. These capabilities enable controlled studies of viewpoint variation, multi-modal fusion, and synthetic-to-real transfer in aerial object detection. Besides, using G-MAD, we construct and release AMOD, a new large-scale multi-view aerial RGB-T object detection benchmark. The source code and the dataset are available at https://unique-chan.github.io/G-MAD-Project.

1 Introduction

G-MAD addresses the limited viewpoint control, RGB-T alignment, and annotation efficiency of aerial dataset construction by turning Arma3 into a structured, controllable generation pipeline. It supports synchronized multi-view RGB-T capture, engine-native annotations, and scenario-based data generation for object detection.

  • Real-world aerial RGB-T dataset construction is costly and difficult because it requires synchronized acquisition, controlled flights, and labor-intensive annotation.
  • Existing game-based pipelines remain insufficient for RGB-T aerial detection, while Arma3 provides scriptable cameras, outdoor environments, controllable entities, and paired RGB-T generation.
  • G-MAD transforms Arma3 into an open-source pipeline supporting scenario specification, synchronized multi-view RGB-T capture, and automatic annotation from engine-level geometry.
  • The framework samples scenarios from high-level constraints including asset categories, object counts, weather, time, maps, sensor settings, and viewpoint strategies.
  • G-MAD decouples scenario-level variables from viewpoint and modality variables, allowing multiple consistent observations from the same underlying layout.
  • The framework offers configurable air-to-air and air-to-land camera motion modes for tailoring viewpoint distributions to downstream tasks.
  • Synchronized RGB-T observations share identical annotations, supporting cross-modal learning, while multi-view supervision supports viewpoint-invariant representation learning.
  • G-MAD extends Arma3-based generation with constraint-aware multi-object generation, GUI-independent execution, HBB and OBB annotation, multi-map backgrounds, configurable resolution, and cache cleanup.

3 Application of G-MAD

G-MAD supports multi-view learning, cross-modal RGB-T learning and fusion, and pretraining for real-world object detection through synchronized and controllable synthetic observations.

  • G-MAD enables multi-view learning, cross-modal RGB-T learning and fusion, and pretraining for real-world object detection.

4 Benchmarking test of G-MAD

The benchmarking study constructs AMOD with synchronized RGB-T multi-view observations and OBB annotations, then evaluates multi-view learning, cross-angular detection, and transfer to real-world RGB and thermal benchmarks.

  • AMOD provides synchronized RGB-T multi-view aerial observations with oriented bounding-box annotations across 12 military-object categories.
  • The benchmark uses Oriented R-CNN with a Swin-S backbone as its baseline detector.
  • Real-world transfer: AMOD pretraining raises mean AP50 from 63.14 to 69.83 on DIOR-R, a +6.69 improvement over ImageNet initialization.
  • Real-world transfer: AMOD pretraining raises mean AP50 from 74.57 to 77.08 on HIT-UAV, a +2.51 improvement over ImageNet initialization.
  • Unseen real-world inference: Without fine-tuning, an AMOD-trained detector identifies plausible military targets in previously unseen real-world RGB satellite imagery.

5 Conclusions

The paper presents G-MAD as a scalable, configurable framework for synchronized multi-view RGB-T aerial data generation with automatic annotation, and releases AMOD for benchmarking. Its scope is primarily military because Arma3 supplies military-focused assets and terrains, while user-defined asset integration broadens possible use.

  • G-MAD generates synthetic RGB-T aerial detection data with synchronized multi-view capture and automatic annotation.
  • The authors construct and release AMOD as an RGB-T aerial multi-view object-detection dataset for controlled benchmarking and transfer studies.

§A.2 Transforming to world coordinates

The annotation procedure transforms object-space corners into world coordinates, projects them into screen coordinates, and derives horizontal or oriented 2D bounding boxes.

  • Each model-space corner point is transformed into the global Arma3 world coordinate frame using modelToWorldVisual.
  • World-space points are projected onto the rendered image plane with Arma3’s worldToScreen function.
  • Safe-zone compensation removes screen offsets and scales normalized coordinates into final pixel positions.
  • The HBB uses minimum and maximum projected x- and y-coordinates, while the OBB uses a convex hull followed by rotating calipers.

§A.5 Summary

AMOD is a synthetic benchmark built with Arma3 for synchronized multi-view aerial detection, emphasizing viewpoint variation and consistent supervision. Its annotations and capture setup support geometric analysis across visible imagery and multiple observation angles.

  • Annotation: The annotation pipeline derives HBB and OBB labels from projected 3D bounding-box geometry using engine functions, convex hulls, and rotating calipers.This produces geometrically consistent labels for detection training and evaluation.
  • Platform choice: Arma3 supplies rich 3D models and terrain assets, avoiding the heavy custom modeling required by general-purpose engines.The benchmark adopts Arma3 because its existing assets reduce the cost of large-scale construction.
  • AMOD benchmark: AMOD provides synchronized multi-view aerial images captured from six distinct observation angles for systematic viewpoint analysis.The benchmark focuses on simultaneous viewpoints rather than primarily increasing dataset volume.
  • AMOD benchmark: AMOD assigns consistent object identities across views within each region of interest, supporting cross-view analysis.The dataset is designed for observations of the same area from multiple viewpoints.
  • Related benchmarks: Unlike several prior multi-view datasets, AMOD targets category-level aerial detection rather than geo-localization, retrieval, or single-category thermal detection.The comparison positions AMOD as complementary to existing multi-view research directions.

§B.2 General setup

G-MAD converts user-specified constraints into sampled Arma3 scenarios, synchronized six-view RGB-T captures, and automatically exported annotations. Its setup controls assets, placement, sensing conditions, maps, and camera configuration.

  • Scenario specification: G-MAD samples asset classes, object counts, arrangements, camera targets, and time while retaining specified weather and sensor settings.The framework accepts constraints covering assets, objects, environment, maps, cameras, modalities, image size, and scenario count.
  • Object placement: Each scene places 8–14 objects from 513 assets across 1–8 sampled classes, with unique rotations, valid terrain placement, and no overlap.Objects are restricted to compatible land or sea locations.
  • Recording setup: The recording setup captures six simultaneous views while varying camera observation angles around the same target region.This preserves shared scene content across viewpoints for multi-view analysis.
  • Collection scope: The benchmark excludes nighttime RGB-T pairs because Arma3 visible images are usually rendered entirely black at night.The stated collection scope therefore focuses on usable daytime RGB-T imagery.
  • Annotation: G-MAD obtains each object’s 3D bounding box, projects its corner points, and derives both HBB and OBB annotations.HBBs use coordinate extrema; OBBs use a convex hull followed by rotating calipers.

§B.3 Statistics, challenges, and applications

AMOD combines multi-view observations, varied model appearance, and geographically diverse Arma3 maps. Its design supports cross-view research while exposing recognition challenges from occlusion and intra-class diversity.

  • Dataset statistics: The benchmark is partitioned into 47,808 training images, 11,988 validation images, and 14,124 test images.These correspond to 64.68%, 16.21%, and 19.11% of the dataset, respectively.
  • Dataset characteristics: AMOD provides simultaneous multi-view observations of each region of interest, enabling research on cross-view consistency.Complex terrain can cause some objects to be occluded from particular viewpoints.
  • Dataset characteristics: AMOD contains low inter-class and high intra-class variation across its object categories.Visually similar vehicle classes complicate fine-grained recognition, while RADAR instances vary substantially in structure.
  • Annotation and visibility: Fully occluded objects are automatically filtered when another scene element intersects their direct line of sight.This operational rule handles visibility during annotation.
  • Dataset characteristics: AMOD uses diverse official Arma3 maps to increase geographical background variation and support model generalization.The maps provide varied scene contexts beyond a single environment.

§B.4 Detailed experimental setup and more experimental results for our AMOD benchmark dataset

Experiments examine AMOD performance across classes, data quantity, angular diversity, and synthetic-to-real transfer. Results indicate that dataset quantity and angular diversity both matter, while synthetic imagery retains a visual domain gap from real data.

  • Experimental setup: Oriented R-CNN with a Swin-S backbone is used as the default AMOD detector under MMRotate.The model is initialized from ImageNet-pretrained weights.
  • Data quantity and diversity: Data quantity or scene diversity is the dominant factor for overall performance gain, with lower quantity reducing test mAP and increasing angle-wise AP variance.The reported trend links smaller datasets to weaker and less stable performance across viewing angles.
  • Angular diversity: When data quantity is controlled, greater angular diversity produces higher mAP and lower performance variance across look angles.This isolates viewpoint diversity as an additional contributor to generalization.
  • Synthetic-to-real transfer: Synthetic AMOD images differ from real-world imagery in color tone, texture, and background complexity, creating a domain gap.The paper treats this discrepancy as relevant to synthetic-to-real transfer.
  • Synthetic-to-real transfer: The transfer pipeline translates synthetic images with UNIT, pretrains a detector on the translated samples, and then finetunes it on labeled real-world data.This sequence is intended to adapt detector features and predictions to the target distribution.

§C.3 Stochastic Boundary Smoothing (SBS): a thermal-specific augmentation trick for AMOD-pretraining

SBS addresses the ill-posed one-to-many nature of thermal translation by stochastically smoothing object boundaries during detector pretraining. It augments deterministic translated images to reduce overly crisp synthetic edges and expand thermal variability.

  • Motivation: Thermal translation is inherently ill-posed because identical optical inputs can correspond to diverse thermal outcomes, limiting deterministic UNIT representations.The framework therefore treats thermal variability as a data-augmentation problem rather than modifying the translator.
  • Method: SBS perturbs pseudo-thermal translated images during pretraining, allowing deterministic translators to represent greater thermal variability.The perturbation is applied as augmentation instead of changing the translation model itself.
  • Design rationale: SBS selectively smooths object contours to emulate thermal sensor edge attenuation, mitigating overly crisp synthetic boundaries and narrowing the domain gap.The design reflects thermal sensors’ suppression of inner textures, preservation of boundary gradients, and possible boundary blur from weak contrast or sensor smoothing.
  • Implementation: The image transform stochastically blends the translated image with a Gaussian-blurred version over an aggregated contour mask.The image X has size w×h, z is Bernoulli-sampled with probability p, and G_σ is a two-dimensional Gaussian blur operator.
  • Implementation: The contour mask is formed by a pixel-wise logical OR over instance contour bands with randomly sampled thicknesses between τ_min and τ_max.Each instance mask M_i contributes a boundary band generated by the contour-band operator Γ.
  • Experimental use: SBS is applied during AMOD pretraining with p=0.5, σ=11, τ_min=2, and τ_max=5, and the resulting HIT-UAV evaluation uses this pretraining configuration.The reported passage states that SBS consistently provides additional performance gains; the comparison is provided in Table S-2.

§C.4 Detailed experimental setup for real-world generalization

The real-world generalization setup pretrains a detector on translated AMOD data, transfers only its backbone, and then fine-tunes on downstream benchmarks with dataset-specific training protocols. The adaptation pipeline uses a proxy target-style domain during translation, which may differ from the final evaluation benchmark.

  • Pretraining setup: AMOD is translated into the corresponding target-domain style before detector pretraining, which lasts 5 epochs.This setup follows the training configuration in the referenced section except for the number of training epochs.
  • Thermal adaptation: UNIT-generated thermal-style images can be further processed with SBS to blur object boundaries and emulate attenuation observed in real airborne thermal sensors.The example contrasts visible input, UNIT translation, SBS output, and a real HIT-UAV thermal image.
  • Domain adaptation: The adaptation pipeline uses X_T as a proxy target-style domain for translation, and this proxy may differ from the downstream benchmark used for fine-tuning and evaluation.The distinction separates the style-transfer domain used during pretraining from the final real-world evaluation domain.
  • Fine-tuning setup: Only backbone weights are transferred after pretraining, while the detection head is re-initialized for the target dataset’s number of classes.Fine-tuning is conducted for 25 epochs on datasets including DIOR-R, DroneVehicle, and HIT-UAV.
Loading 2607.19942v2…