PaperlayerView exact revision on arXiv

Source-linked AI summary

MultiFly: A Real-World Multimodal Aerial Dataset with Annotation-Efficient Label Transfer and Cross-Modal Semantic Consistency

Markus Gross, Andreas Greiner, Taehyoung Kim, Sivasubiramaniam Subbiah, Tomaž Cotič, Sai Bharadwaj Matha, Conrad Christoph, Oussema Dhaouadi, Simon Zieher, Surya Vijaya Kumar, Gordon Elger, Henri Meeß, Olaf Wysocki, Paul Spannaus, Daniel Cremers

arXiv:2610.10359v1cs.ROcs.CV

TL;DR

Public real-world UAV datasets lack synchronized frame-wise semantic annotations across RGB, thermal, LiDAR, and radar, despite differences in UAV scene statistics and viewing geometries. MultiFly combines these four modalities and transfers labels from 115 manually annotated RGB images through shared geometry. The transfer achieves 89.93% average accuracy, with 90.94% average semantic consistency across modality pairs, and the paper establishes segmentation baselines for all four modalities.

  • Problem

    Public real-world low-altitude UAV datasets lack synchronized frame-wise semantics across RGB, thermal, LiDAR, and radar, while UAV scene statistics and viewing geometries differ from conventional ground-based data.

    Source: §I · p. 1, A. Multimodal Semantic Aerial Datasets · p. 2

  • Method

    MultiFly uses shared 3D scene geometry to transfer labels from 115 manually annotated RGB images to synchronized RGB, thermal, LiDAR, and radar data.

    Source: §I · pp. 1–2

  • Results

    89.93% average accuracy across the four modalities accompanies 90.94% average semantic consistency across all six modality pairs.

    Source: §I · p. 2, A. Modal-Wise Label Transfer Evaluation · p. 6

  • Takeaways & Limitations

    MultiFly provides initial semantic-segmentation baselines for all four modalities on a synchronized multimodal UAV dataset.

    Source: §VIII · p. 8

Abstract

from arXiv · show

We introduce MultiFly, a real-world, low-altitude UAV dataset for semantic perception across RGB, thermal, LiDAR, and radar modalities. MultiFly provides 17,272 synchronized samples from four suburban scenes with frame-wise annotations for 15 semantic classes, together with calibration and GNSS-RTK/IMU measurements. To avoid costly and inconsistent modality-specific annotation, we propagate labels from only 115 manually annotated RGB images through shared geometric representations to all four modalities. This approach generates semantic labels for 17,157 additional RGB images, 17,272 thermal images, 840M LiDAR points, and 3.4M radar points. Transferred annotations achieve 89.93% average agreement with held-out manual annotations, and 90.94% average semantic consistency across all six modality pairs. We further establish semantic segmentation benchmarks for all four modalities, revealing distinct architectural behavior for dense LiDAR and sparse radar data. Taken together, MultiFly provides a scalable foundation for multimodal aerial perception and, to the best of our knowledge, the first public real-world low-altitude aerial benchmark that combines consistent frame-wise semantic annotations for RGB, thermal, LiDAR, and radar. Data at https://github.com/markus-42/multifly.

I. INTRODUCTION

UAV semantic-perception datasets remain limited despite the importance of semantic scene understanding, and existing resources lack synchronized frame-wise labels across RGB, thermal, LiDAR, and radar. MultiFly combines a calibrated four-modal dataset with geometry-driven label transfer from sparse RGB annotations and segmentation benchmarks for all four modalities.

I. INTRODUCTION

  • UAV scenes differ from conventional ground-based data in scene statistics, object scales, and viewing geometries, including nadir and oblique perspectives.Large-scale multimodal datasets have enabled systematic sensor alignment in autonomous driving, while comparable UAV resources remain limited.Source: §I · p. 1, A. Multimodal Semantic Aerial Datasets · p. 2

A. Multimodal Semantic Aerial Datasets

  • Existing aerial datasets generally cover one or two modalities, and none provides synchronized frame-wise semantics across all four modalities in a public real-world UAV dataset, to the authors’ knowledge.Modalities also differ in geometry and sampling, making consistent annotation difficult and costly.Source: §I · p. 1, A. Multimodal Semantic Aerial Datasets · p. 2

I. INTRODUCTION

  • MultiFly uses 3D scene geometry as a common semantic interface, lifting sparse RGB annotations and transferring them to synchronized RGB, thermal, LiDAR, and radar measurements.The common taxonomy and geometric reference support dense multimodal annotations from 115 manually labeled RGB images.Source: §I · pp. 1–2

A. Multimodal Semantic Aerial Datasets

  • 17,272 synchronized samples across four suburban scenes cover 15 semantic classes, with sensor calibration and GNSS-RTK/IMU measurements.The dataset spans 56,700 m2.Source: §I · p. 2

I. INTRODUCTION

  • 89.93% average agreement with held-out manual annotations and 90.94% average consistency across six modality pairs validate transfer from 115 manually annotated RGB images.Labels extend to 17,157 additional RGB frames, 17,272 thermal frames, 8.4×108 LiDAR points, and 3.4×106 radar points.Source: §I · p. 2
  • The paper establishes semantic-segmentation benchmarks for RGB, thermal, LiDAR, and radar, and analyzes architectural influences for dense LiDAR versus sparse radar.Source: §I · p. 2

B. Cross-Modal Semantic Consistency

Prior geometry-driven annotation methods demonstrate cross-modal label transfer but have limited modality coverage and require manual verification and correction. MultiFly extends this approach to four modalities and pairs it with a platform integrating multiple sensors and navigation measurements.

B. Cross-Modal Semantic Consistency

  • Earlier geometry-driven approaches transfer labels between images and 3D geometry, but remain limited in modality coverage and require manual verification and correction.Examples include KITTI-360, WildScenes, UAVScenes, and SegFly.Source: B. Cross-Modal Semantic Consistency · p. 2
  • MultiFly propagates sparse RGB annotations to RGB, thermal, LiDAR, and radar through a unified geometry-driven pipeline, without requiring manual correction.This provides a shared frame-wise semantic reference for multimodal UAV perception and cross-modal evaluation.Source: B. Cross-Modal Semantic Consistency · p. 2

A. Design

  • The MultiFly platform integrates RGB and thermal cameras, spinning LiDAR, two forward-facing radars, and GNSS-RTK-aided inertial navigation.Sensors are calibrated to the LiDAR frame and synchronized to a common time base; the rig is pitched downward to observe the flight direction and ground below.Source: A. Design · p. 2, Table II · p. 3, Figure 2 · p. 3

B. Time Synchronization

MultiFly synchronizes sensor measurements to a common GNSS-disciplined time base, then calibrates sensor relationships around the LiDAR reference frame. Camera intrinsics are calibrated independently, while LiDAR and radar use factory intrinsics.

B. Time Synchronization

  • Sensor-specific triggering and phase compensation align acquisition times across modalities to the LiDAR’s 10 Hz scan cycle.The INS is the PTP grandmaster; radar–LiDAR deviations are within ±3 ms, and timestamped GigE Vision commands control the thermal camera.Source: B. Time Synchronization · p. 3

C. Sensor Calibration and Registration

  • The calibration tree is rooted at LiDAR, with RGB–LiDAR, radar–LiDAR, and RGB–thermal extrinsics estimated using landmarks, corner reflectors, and checkerboard corners, respectively.Composing RGB–thermal and RGB–LiDAR transforms places the thermal camera in the LiDAR frame.Source: C. Sensor Calibration and Registration · pp. 3–4
  • RGB and thermal camera intrinsics are calibrated independently using Zhang’s planar method, while LiDAR and radar use factory intrinsics.The thermal calibration target is a steel plate with plotter-cut vinyl squares whose emissivity contrast makes corners detectable in the thermal band.Source: C. Sensor Calibration and Registration · p. 4

D. Registration.

Registration uses the metric RGB reconstruction and fixed RGB–thermal extrinsic to map RGB imagery onto each thermal image grid. This shared geometry supports both registration and semantic label transfer.

D. Registration.

  • The RGB reconstruction provides scene-wide geometry and image–point correspondences, avoiding a separate geometric representation for registration.Because the same representation supports registration and semantic transfer, the two stages remain consistent.Source: D. Registration · p. 4
  • For each synchronized RGB–thermal pair, reconstructed landmarks visible in both cameras determine the shared image support used to warp RGB imagery into the thermal image grid.The process resamples into a common undistorted pinhole domain, maps into thermal distortion space, and masks pixels outside valid overlap.Source: D. Registration · p. 4
  • Registration results exhibit precise cross-modal alignment and high geometric fidelity.Source: D. Registration · p. 4

IV. ANNOTATION-EFFICIENT LABEL TRANSFER

  • The annotation framework aims to generate dense frame-wise labels for RGB, thermal, LiDAR, and radar from sparse manually annotated RGB views.It adapts and integrates established geometry-driven transfer operations into a unified aerial framework.Source: §IV · p. 4, A. RGB and Thermal Semantic Label… · p. 4

1) 2D-to-3D:

MultiFly transfers sparse RGB annotations through a shared 3D semantic point cloud to other image modalities and sensor measurements. Shared reconstruction and RGB–thermal calibration support geometrically consistent labels across those modalities.

1) 2D-to-3D:

  • Sparse, spatially stratified RGB annotations are lifted through image–point correspondences into a reconstructed point cloud, then completed using neighborhoods.This produces the RGB-derived semantic point cloud used for subsequent rendering.Source: 1) 2D-to-3D · p. 4, B. LiDAR and Radar Semantic Label… · p. 4

2) 3D-to-2D:

  • The semantic point cloud is rendered into remaining RGB and thermal images using visibility-aware Z-buffering and camera poses.The rendering produces dense pseudo-label maps for both modalities.Source: 2) 3D-to-2D · p. 4

B. LiDAR and Radar Semantic Label Transfer

  • Using the same metric reconstruction for registration and rendering, with fixed RGB–thermal extrinsics, makes the resulting pseudo-labels geometrically consistent across both modalities.Spatially stratified RGB views also transfer labels to LiDAR and radar without additional modality-specific manual annotation.Source: 2) 3D-to-2D · p. 4, B. LiDAR and Radar Semantic Label… · p. 4

1) LiDAR Label Transfer:

LiDAR and radar scans are aggregated in a common scene frame while retaining scan associations for frame-wise annotations. Both use visibility-aware RGB label transfer with a 0.75 majority threshold, while radar receives additional outlier filtering.

1) LiDAR Label Transfer:

  • LiDAR scans are aggregated per scene and altitude, rendered into annotated RGB views, and assigned the majority observed class when its relative frequency exceeds 0.75.Labels are mapped back to original scans; points without a class are treated as ignore labels during supervised training.Source: 1) LiDAR Label Transfer · p. 5

2) Radar Label Transfer:

  • Radar scans are aggregated in a common scene frame, then filtered with RANSAC plane fitting and DBSCAN before the same visibility-aware RGB label transfer and 0.75 majority rule.Points removed during filtering are marked invalid; labels are mapped back to scans, with invalid and unlabeled points available as training ignore labels.Source: 2) Radar Label Transfer · p. 5

V. MULTIFLY DATASET

MultiFly was collected across four suburban scenes at two nominal flight altitudes using a synchronized UAV sensor platform. Only 115 RGB images were manually annotated for label propagation.

A. Data Acquisition and Source Annotation

  • Four suburban scenes near Ingolstadt were recorded in summer under sunny and cloudy conditions, at nominal altitudes of 30 m and 50 m.The UAV followed automated double-grid trajectories at 3 m s−1 with nominal RGB side overlap of 40%, and its sensor stack was pitched downward by 45°.Source: A. Data Acquisition and Source… · p. 5
  • Synchronized 10 Hz perception streams were subsampled by retaining every second frame at 30 m and every third at 50 m, while the 100 Hz INS stream was preserved.LiDAR scans were restricted offline to the forward 90° sector aligned with the RGB camera.Source: A. Data Acquisition and Source… · p. 5
  • The only manual annotations used for data generation were 115 RGB images labeled with 15 semantic classes; thermal images and 3D points were not manually annotated.Table IV is identified as the modal-wise label-transfer evaluation.Source: A. Data Acquisition and Source… · p. 5, Table IV · p. 6

B. Dataset Statistics

MultiFly provides synchronized, frame-wise semantic annotations and sensor metadata across four modalities. Evaluation reports 89.93% average label-transfer accuracy across modalities, with individual results for image and point-cloud data.

B. Dataset Statistics

  • Each of 17,272 synchronized samples includes RGB, thermal, LiDAR, and two radar-sensor measurements, with a six-degree-of-freedom pose.The dataset provides frame-wise pixel labels for images, point labels for LiDAR and radar, calibration and synchronization metadata, and the complete navigation stream.Source: B. Dataset Statistics · p. 6

A. Modal-Wise Label Transfer Evaluation

  • RGB and thermal transfer accuracy averaged 92.12% and 90.31%, respectively, on par with SegFly at 91.27% and 87.65%.The evaluation used held-out manually annotated frames from scene #4.Source: A. Modal-Wise Label Transfer Evaluation · p. 6
  • LiDAR and radar label-transfer accuracy was 91.45% and 85.84%, respectively.The rendered 3D annotations were compared with the same held-out RGB labels used for evaluation.Source: A. Modal-Wise Label Transfer Evaluation · p. 6
  • 89.93% average accuracy across all four modalities measures transferred labels against held-out manual annotations.RGB and thermal results were evaluated on 40 held-out frames; LiDAR and radar labels were rendered into the same held-out RGB labels, so their agreement reflects semantic quality and geometric consistency.Source: A. Modal-Wise Label Transfer Evaluation · p. 6

B. Cross-Modal Semantic Consistency Evaluation

MultiFly achieves high semantic agreement across modality pairs, using calibrated image and point correspondences to evaluate consistency. The reported agreement also serves as a proxy for geometric fidelity, while static reconstruction may affect dynamic objects.

B. Cross-Modal Semantic Consistency Evaluation

  • 90.94% average cross-modal semantic consistency across all six modality pairs and four scenes; LiDAR–Radar agreement ranges from 96% at r = 0.1 m to 93% at r = 2.0 m.Unconstrained nearest-neighbor association (r = ∞) yields 86% agreement.Source: Table V · p. 6, B. Cross-Modal Semantic Consistency… · p. 6
  • RGB–Thermal labels are compared after geometric registration, image-to-point labels after calibrated projection, and LiDAR–Radar labels through nearest-neighbor association within 1 m.The 1 m radius yields correspondences for 71% of radar points; only valid semantic classes are evaluated.Source: B. Cross-Modal Semantic Consistency… · p. 6
  • High cross-modal agreement also serves as a proxy for geometric fidelity because evaluation relies on calibrated intrinsics, extrinsics, and RGB–thermal registration.Source: B. Cross-Modal Semantic Consistency… · p. 7
  • Static 3D reconstruction for label propagation and point-cloud aggregation could affect dynamic objects.The authors identify 4D reconstruction as a promising future direction.Source: B. Cross-Modal Semantic Consistency… · p. 7

C. Annotation Efficiency

MultiFly transfers labels from a small set of manually annotated RGB frames to the remaining RGB frames and all thermal, LiDAR, and radar data. Its benchmarks use held-out-scene evaluation alongside cross-modal image adaptation and modality-specific 3D segmentation.

C. Annotation Efficiency

  • 115 RGB frames (0.67% of the dataset) were manually annotated, enabling automatic labels for 17,157 more RGB frames, 17,272 thermal frames, 8.4 × 10^8 LiDAR points, and 3.4 × 10^6 radar points.The 115 source frames took 29 minutes per frame on average; no thermal frame or 3D point was manually annotated for data generation.Source: C. Annotation Efficiency · p. 7
  • The resulting labels show high within-modality semantic quality and strong cross-modal consistency.Source: C. Annotation Efficiency · p. 7

A. Experimental Setup

  • The benchmark evaluates cross-modal RGB-to-thermal adaptation and independent LiDAR and radar segmentation, with dataset-level semantic consistency assessed separately.Source: A. Experimental Setup · p. 7
  • Scenes 1–3 are used for training and scene 4 exclusively for final evaluation to assess scene-level generalization.Models use their official configurations.Source: A. Experimental Setup · p. 7

B. RGB and Thermal Semantic Segmentation

Firefly performs best for both RGB and thermal segmentation, while the strongest model differs between LiDAR and radar. These benchmarks provide initial segmentation baselines for all four modalities.

B. RGB and Thermal Semantic Segmentation

  • Firefly achieves the strongest RGB and thermal results in both mean intersection over union (mIoU) and mean accuracy (mAcc).The benchmark follows a three-stage training protocol: RGB training, RGB-to-thermal adaptation, and thermal fine-tuning.Source: B. RGB and Thermal Semantic Segmentation · p. 7

C. LiDAR and Radar Semantic Segmentation

VIII. CONCLUSION

  • MultiFly provides initial semantic segmentation baselines for RGB, thermal, LiDAR, and radar using labels propagated from 115 manually annotated RGB images.Source: §VIII · p. 8
Loading 2610.10359v1…