Source-linked AI summary
AGT-CV: An Aerial-Ground Team Cross-View Dataset for Heterogeneous Robot Teams in Unstructured Environments
Siwei Cai, Knut Peterson, Quan Tran, Christian Ricks, Dhanush Parthasarathy, Amir Kaidarov, Neil Deshpande, Sukaina Najm, David Han, Lifeng Zhou
TL;DR
Real-world collaborative perception datasets for heterogeneous air-ground teams in unstructured terrain remain limited. AGT-CV provides a multimodal dataset pairing ground and aerial robots, with annotations and synchronized control data. Adaptation on AGT-CV improves segmentation on both views, with the largest gains on UAV data.
Problem
Real-world datasets with overlapping multimodal observations from heterogeneous air-ground platforms in unstructured terrain remain scarce.
Source: §1 · p. 2Method
AGT-CV pairs a sensor-equipped Husky UGV and EVO II UAV to collect complementary, synchronized observations in diverse off-road environments, with annotations and teleoperation logs.
Source: §1 · p. 3Results
Adaptation on AGT-CV improves segmentation performance on both UGV and UAV views, with the largest gains on UAV data; backbone fine-tuning further improves UAV and overall performance.
Source: Discussion · p. 13Takeaways & Limitations
AGT-CV supports domain adaptation and challenging evaluation for collaborative air-ground perception in unstructured outdoor environments.
Source: Discussion · p. 13
Abstract
from arXiv · showhide
Heterogeneous air-ground robot teams combine complementary sensing modalities, mobility characteristics, and spatial viewpoints that can significantly enhance perception in complex outdoor environments. However, progress in multi-robot collaborative perception has been constrained by the lack of real-world datasets featuring overlapping multi-modal observations from platforms operating in unstructured terrain. We present the \textbf{AGT-CV} (\textbf{A}erial-\textbf{G}round \textbf{T}eam \textbf{C}ross-\textbf{V}iew) dataset, a real-world multi-robot collaborative perception dataset collected using a Clearpath Husky UGV and an Autel EVO~II UAV across diverse unstructured environments, including forest trails, rocky paths, muddy terrain, snow piles, and grass-covered fields. The ground platform provides 3D LiDAR, stereo camera, IMU, and GPS data, while the aerial platform contributes RGB imagery, thermal/infrared observations, and GPS from a complementary overhead viewpoint, allowing for rich cross-modal and cross-view perception. The dataset is collected in 4 unique environments, with over 13,000 synchronized frames across approximately 29 minutes of operation, and includes both SAM~3-based zero-shot segmentation and almost 8,000 manually labeled images. A unique aspect of the dataset is its early-spring collection period, during which sparse tree canopies allow the aerial robot to partially observe the ground robot and terrain through the trees, allowing for occlusion-aware collaborative perception. Unlike prior multi-robot datasets that primarily focus on SLAM or simulated cooperative driving, AGT-CV is specifically designed to support research on cross-view perception, air-ground viewpoint fusion, terrain-aware perception, and collaborative scene understanding in real off-road environments.
1 Introduction
AGT-CV addresses the shortage of real-world collaborative perception data for heterogeneous air-ground teams in unstructured environments. It combines complementary aerial and ground sensing with annotations and synchronized control data to support cross-view perception and related off-road tasks.
1 Introduction
- Existing datasets often emphasize SLAM, simulation, on-road driving, or single ground platforms, leaving real-world heterogeneous perception across diverse off-road conditions underrepresented.The introduction identifies a lack of datasets combining overlapping multimodal air-ground observations, varied off-road terrain, and thermal imagery.Source: §1 · p. 2
- AGT-CV pairs a Clearpath Husky UGV carrying stereo, LiDAR, IMU, and GPS with an Autel EVO II UAV carrying RGB, thermal, IMU, and GPS sensors.The platforms provide complementary ground-level and overhead views of scene structure, terrain, and robot-environment interactions.Source: §1 · p. 3
- Early-spring collection leaves sparse canopies that let the UAV partially observe the ground scene and UGV, supporting study of collaborative perception under partial visibility.The collection includes occlusions and environmental complexity, with aerial sensing able to complement ground-level blind spots and vice versa.Source: §1 · p. 3
- AGT-CV includes SAM 3-based zero-shot segmentation and manually labeled images for perception research.The introduction specifies 8,000 manually labeled images and cites SAM 3-based zero-shot segmentation [3].Source: §1 · p. 3
- The dataset is designed for cross-view semantic prediction, UAV–UGV-context path planning, and collaborative scene understanding beyond bird’s-eye-view assumptions.Synchronized teleoperation commands also support learning from demonstration and visuomotor policy learning for off-road navigation.Source: §1 · pp. 3–4
2 Related Work
Prior work covers off-road perception, heterogeneous sensing, air-ground cooperation, and aerial vision, but leaves gaps in dense cross-view perception for cluttered off-road settings. AGT-CV combines these areas with real-world, multimodal air-ground data and annotations.
2.1 Off-Road Robotic Perception Datasets
- RUGD and RELLIS-3D support off-road perception and scene understanding, but focus on single ground platforms rather than heterogeneous collaboration.RELLIS-3D includes RGB, LiDAR, and semantic annotations.Source: §2.1 · p. 4
2.2 Heterogeneous and Air-Ground Multi-Robot Datasets
- CoPeD and the ITU heterogeneous robot team dataset explore collaborative or air-ground sensing, but offer less focus on dense off-road scene understanding under vegetation occlusion and mixed terrain.The cited datasets provide realistic collaborative perception or 3D sensing useful for mapping and localization.Source: §2.2 · p. 4
2.3 Air-Ground Cooperation and Cross-View Robotics
- UAV assistance for UGV localization illustrates complementary viewpoints, while much prior air-ground work focuses on specific algorithms rather than broadly usable collaborative-perception datasets.Cristofalo et al. demonstrate UAV support for UGV localization in GPS-denied environments.Source: §2.3 · pp. 4–5
2.4 Aerial Vision Benchmarks
- VisDrone and LRDD advance aerial vision, but lack synchronized air-ground observations, ground trajectories, and terrain-centric multimodal sensing for joint UAV–UGV perception.Source: §2.4 · p. 5
2.5 Position of This Work
- AGT-CV combines aerial and ground views, off-road terrain variation, thermal imagery, GPS-grounded observations, control logs, and manual and zero-shot annotations.The dataset is positioned for cross-view semantics, terrain-aware perception, air-ground path planning, perception-conditioned control, and collaborative autonomy.Source: §2.5 · p. 5
3 Dataset
AGT-CV pairs a sensor-rich ground robot with an RGB–thermal aerial platform to collect synchronized observations across varied off-road environments. Its processing includes GPS-referenced trajectory refinement, aerial RGB–thermal alignment, and human-corrected segmentation annotations.
3.1 Robots and Equipment
- The Clearpath Husky carries LiDAR, stereo camera with built-in IMU, and GPS; the Autel EVO II Dual 640T V3 provides aerial RGB, thermal, and GPS observations.The Husky is teleoperated and its control commands are logged with sensor streams.Source: §3.1 · pp. 5–6, Figure 2 · p. 6
- GPS time timestamps sensor streams, and interpolated UGV odometry and LiDAR frames provide temporal matches to UAV video timestamps.The maximum synchronization offset is maintained within 200 ms, although the approach does not aim for centimeter-level relative localization.Source: §3.1 · pp. 6–7
3.2 Dataset content
- Four locations yielded 13,326 Husky and Drone images and metadata, including 8k manually annotated images across varied terrain.The environments include forest paths, rocky riverbank, and muddy field, capturing occlusion, partial viewpoint overlap, uneven terrain, and traversability ambiguity.Source: Table 2 · p. 8, §3.2 · p. 8
UGV Trajectory Refinement and Ground Truth Generation
- The dataset combines LiDAR odometry with GPS measurements to refine the UGV trajectory into a globally consistent ground reference.KISS-ICP estimates local motion; GPS alignment provides a heading offset that corrects long-term drift while retaining local LiDAR consistency.Source: UGV Trajectory Refinement and Ground… · p. 9
Cross-Modal Aerial Alignment
- A coarse-to-fine gradient-domain matching pipeline aligns UAV thermal imagery to RGB using a similarity transform with scale and translation.The aligned streams support thermal overlays and help identify the UGV thermal signature under vegetation or partial canopy occlusion.Source: Cross-Modal Aerial Alignment · p. 9, Figure 3 · p. 10
Foundation-Model-Assisted Segmentation Annotation
- The annotation pipeline generates SAM3 mask proposals for human semantic-class assignment and local error correction instead of drawing segmentations from scratch.The class set is adapted from RUGD and RELLIS, and later annotation phases use an updated SAM3 model trained on refined labels.Source: Foundation-Model-Assisted Segmentation… · pp. 10–11
4 Dataset Benchmarking
The benchmark fine-tunes SAM3 on paired UAV and UGV images to evaluate class-conditioned terrain segmentation across viewpoints. Adaptation improves both views, with the largest gains on UAV imagery, where zero-shot performance is weaker.
4 Dataset Benchmarking
- The benchmark fine-tunes SAM3 jointly on UAV and UGV images to test whether combined data improve aerial terrain segmentation while preserving ground-view performance.The Field subset contains approximately 8,000 manually annotated images, split 85/15 into training and validation sets with both platforms represented.Source: §4 · p. 11
- SAM3 selects semantic categories from a fixed class list rather than receiving free-form text prompts during training or evaluation.Source: §4 · p. 12
- Training uses head-only decoder adaptation followed by vision-encoder fine-tuning from the best first-stage checkpoint.The decoder is trained for 40 epochs with the encoder frozen, followed by 20 epochs with the encoder unfrozen.Source: §4 · p. 12
Benchmark Results
- Class-wise mask IoU is accumulated over the held-out validation set, and viewpoint-specific and pooled mIoU are reported.The pooled score aggregates per-class intersections and unions across UAV and UGV images before computing IoU.Source: Benchmark Results · p. 12
Discussion
- AGT-CV fine-tuning improves segmentation on both views, with the largest gains on UAV data, where zero-shot SAM3 is weaker.The discussion attributes aerial difficulty to viewpoint shift, partial canopy occlusion, and weaker local perspective structure; Table 3 reports class-conditioned mask IoU for zero-shot and fine-tuned models.Source: Discussion · p. 13, Table 3 · p. 13
5 Conclusion
AGT-CV provides paired aerial and ground sensing for UAV-UGV teams in diverse off-road environments. Its annotated camera views support terrain identification across viewpoints and potential studies from multi-view SLAM to terrain-optimized path planning.
5 Conclusion
- AGT-CV combines diverse off-road environments and multimodal UAV-UGV sensing, with over 8,000 paired-view frames bearing dense semantic segmentation annotations.The environments include forest trails, rocky paths, muddy trenches, grass-covered fields, standing water, and snow piles; sensors include 3D LiDAR, stereo camera, GPS, and RGB+IR imagery.Source: §5 · p. 13