PaperlayerView exact revision on arXiv

Source-linked AI summary

CARLA-Air: Fly Drones Inside a CARLA World -- A Unified Infrastructure for Air-Ground Embodied Intelligence

Tianle Zeng, Yanci Wen, Hong Zhang

arXiv:2603.28032v2cs.ROcs.AIcs.CVcs.HC

TL;DR

Existing simulation platforms do not provide a unified, physically coherent environment for aerial and ground agents. CARLA-Air integrates CARLA and AirSim in one Unreal Engine process, achieving stable joint operation and validating five representative workflows.

  • Problem

    No widely adopted open-source platform jointly models aerial and ground agents within one physically coherent environment, although simulation is needed for costly, safety-critical, hard-to-scale deployment settings.

  • Method

    CARLA-Air composes CARLA’s ground simulation subsystems with AirSim’s aerial flight actor in one process, using shared physics and rendering pipelines while preserving both upstream Python APIs.

  • Results

    Stable operation at ≈20 FPS under joint workloads, zero crashes across 357 reset cycles in a 3-hour run, and five representative workflows validate the platform’s core capabilities.

  • Takeaways & Limitations

    CARLA-Air provides a practical shared infrastructure for air-ground embodied-intelligence workflows, including cooperative operation, synchronized sensing, navigation, and reinforcement-learning environments.

  • Takeaways & Limitations

    The current release is validated only for single- and dual-drone operations over moderate-density urban traffic scenes.

Abstract

from arXiv · show

The convergence of low-altitude economies, embodied intelligence, and air-ground cooperative systems creates growing demand for simulation infrastructure capable of jointly modeling aerial and ground agents within a single physically coherent environment. Existing open-source platforms remain domain-segregated: driving simulators lack aerial dynamics, while multirotor simulators lack realistic ground scenes. Bridge-based co-simulation introduces synchronization overhead and cannot guarantee strict spatial-temporal consistency. We present CARLA-Air, an open-source infrastructure that unifies high-fidelity urban driving and physics-accurate multirotor flight within a single Unreal Engine process. The platform preserves both CARLA and AirSim native Python APIs and ROS 2 interfaces, enabling zero-modification code reuse. Within a shared physics tick and rendering pipeline, CARLA-Air delivers photorealistic environments with rule-compliant traffic, socially-aware pedestrians, and aerodynamically consistent UAV dynamics, synchronously capturing up to 18 sensor modalities across all platforms at each tick. The platform supports representative air-ground embodied intelligence workloads spanning cooperation, embodied navigation and vision-language action, multi-modal perception and dataset construction, and reinforcement-learning-based policy training. An extensible asset pipeline allows integration of custom robot platforms into the shared world. By inheriting AirSim's aerial capabilities -- whose upstream development has been archived -- CARLA-Air ensures this widely adopted flight stack continues to evolve within a modern infrastructure. Released with prebuilt binaries and full source: https://github.com/louiszengCN/CarlaAir

1 Introduction

CARLA-Air addresses the lack of a unified, physically coherent open-source environment for aerial and ground agents by integrating CARLA and AirSim in one Unreal Engine process. Its shared simulation infrastructure preserves upstream interfaces, synchronizes sensing and dynamics, and supports diverse air-ground embodied-intelligence workloads.

  • Motivation: Existing open-source simulators remain domain-segregated: CARLA lacks aerial agents, while AirSim lacks realistic ground scenes.This leaves air-ground workloads without a shared simulation foundation.
  • Motivation: Bridge-based co-simulation introduces synchronization complexity and communication overhead, while separate processes cannot guarantee strict spatial-temporal consistency across sensor streams.These limitations matter for perception, learning, and evaluation workloads.
  • Architecture: CARLA-Air integrates CARLA and AirSim within a single Unreal Engine process through a composition-based design with shared physics and rendering pipelines.AirSim’s aerial flight actor is composed as a regular world entity alongside CARLA’s ground subsystems.
  • Architecture: Both CARLA and AirSim native Python APIs and ROS 2 interfaces are preserved, allowing existing research codebases to run without modification.This maintains compatibility while extending the shared simulation environment.
  • Capabilities: The platform provides photorealistic, physically coherent scenes with rule-compliant traffic, socially-aware pedestrians, aerodynamically consistent multirotor dynamics, and up to 18 synchronized sensor modalities per tick.The shared world combines urban and natural environments with synchronized aerial-ground sensing.
  • Workloads: CARLA-Air supports air-ground cooperation, embodied navigation and vision-language action, multi-modal perception and dataset construction, and reinforcement-learning-based policy training.Custom robot platforms, UAV configurations, vehicles, and maps can be imported through an extensible asset pipeline.

2 Related Work

Related platforms provide strong capabilities in driving, aerial robotics, co-simulation, or embodied AI, but generally optimize for separate domains or different design objectives. CARLA-Air is positioned as a unified alternative combining simulation fidelity with broad aerial-ground agent coverage without inter-process bridging.

  • Driving simulators: Driving simulators such as CARLA emphasize photorealistic urban environments, traffic agents, and ground-vehicle perception.LGSVL, SUMO, and MetaDrive provide complementary driving-oriented capabilities.
  • Aerial simulators: Aerial simulators such as AirSim provide physics-accurate multirotor flight, aerial sensing, and UAV-oriented control interfaces.Other systems target fast reinforcement learning, photogrammetry-based perception, or ROS-integrated multirotor control.
  • Joint simulation: Prior joint approaches connect heterogeneous simulators through bridges, RPC layers, or message-passing middleware across independent processes.These systems demonstrate functional connectivity but do not provide the same unified integration properties.
  • Embodied AI platforms: Embodied AI platforms prioritize scalable policy training for locomotion, manipulation, indoor navigation, or tabletop tasks rather than realistic urban air-ground infrastructure.Isaac Lab, Isaac Gym, Habitat, SAPIEN, and RoboSuite exemplify these objectives.
  • Platform positioning: CARLA-Air is designed at the intersection of high simulation fidelity and broad agent-domain coverage.Figure 3 positions it in the high-fidelity, multi-domain quadrant without inter-process bridging.

3 System Architecture

CARLA-Air integrates CARLA and AirSim in one Unreal Engine process while preserving native interfaces and coordinating ground and aerial simulation through shared execution components. Its architecture resolves the single-GameMode constraint, reconciles coordinate frames, and supports custom assets in the joint world.

  • Runtime integration: CARLA-Air integrates CARLA and AirSim within a single Unreal Engine process through a minimal bridging layer while preserving native APIs, physics engines, and rendering pipelines.Two independent RPC servers allow each simulator’s native Python client to connect without modification.
  • Runtime integration: All world actors use a shared rendering pipeline, while the unified process provides shared physics execution and strict spatial-temporal consistency across sensor viewpoints.The runtime architecture retains independent RPC servers while consolidating simulation and rendering within one engine process.
  • GameMode conflict resolution: CARLAAirGameMode inherits CARLA’s game mode and composes AirSim’s flight actor during BEGINPLAY, avoiding competition for UE4’s single game mode slot.The design acquires ground subsystems through inheritance and adds aerial functionality as a regular world entity.
  • Coordinate system mapping: CARLA-Air reconciles CARLA’s centimeter-scale UE4 frame with AirSim’s meter-scale NED frame using a scale conversion, Z-axis reversal, and no X-Y permutation.The position transform uses a shared initialization origin; the orientation transform negates qz to account for frame handedness.
  • Asset import pipeline: An extensible asset pipeline imports custom robots, UAVs, vehicles, and environments into the shared world without modifying core CARLA-Air code.Imported assets participate in the same physics tick and rendering pass and remain visible across sensor modalities.

4 Performance Evaluation

CARLA-Air is evaluated under representative joint air-ground workloads using controlled warm-up, measurement, resource, stability, and latency procedures. The results show sustained integrated performance, bounded VRAM use, stable repeated lifecycle operation, and synchronized aerial-ground timing.

  • Evaluation Protocol: The evaluation covers frame-rate/resource scaling, memory stability, and communication latency under representative joint air-ground workloads.Profiles use 200 warm-up ticks and 2,000 measurement ticks; latency uses 500 warm-up and 5,000 measurement calls.
  • Integration Design: CARLAAirGameMode resolves UE4’s single-game-mode constraint by inheriting CARLA’s ground functionality and composing the aerial flight actor as a world actor.The aerial actor is added during BEGINPLAY after ground initialization, avoiding competition for the game-mode slot.
  • Frame Rate and Resource Scaling: 19.8 ± 1.1 FPS is sustained by the moderate joint configuration, while integration overhead is 8.6 FPS relative to the 28.4 FPS standalone ground baseline.The overhead is attributed to 2.1 FPS of ground co-hosting and 6.5 FPS from the aerial physics engine; traffic surveillance remains at 20.1 FPS despite doubled vehicle count.
  • Memory Stability: All steady-state profiles remain within 3,878 MiB of VRAM, leaving approximately 12,506 MiB, or 76% of the 16 GB device budget, for co-located workloads.Town10HD at idle uses approximately 3,702 MiB, while map loading produces a transient peak of approximately 5,000 MiB that resolves within 30 seconds.
  • Memory Stability: Across 357 spawn-and-destroy cycles over three hours, VRAM drift is approximately 10 MiB and the regression gives R2 = 0.11, with zero API errors and zero simulation crashes.The reported drift is attributed to residual render-target caching rather than lifecycle leakage.
  • Latency and Tick-Rate Reconciliation: Lightweight state queries take 280–490 µs, while aerial physics advances at approximately 1,000 Hz and rendering at approximately 20 Hz, aligning sensor callbacks at rendering tick boundaries.Each ground-aerial sensor pair reflects the drone’s integrated state over approximately 50 aerial physics steps.

5 Representative Applications

CARLA-Air is validated across five workflows spanning air-ground cooperation, embodied navigation, multi-modal data collection, cross-view perception, and reinforcement-learning policy training. These workflows use shared world state and synchronized sensing to demonstrate practical air-ground applications.

  • Overview: Five workflows exercise CARLA-Air across four research directions: air-ground cooperation, embodied navigation and vision-language action, multi-modal perception and dataset construction, and reinforcement-learning policy training.All workflows use the platform’s shared simulation infrastructure.
  • Shared architecture: A shared dual-client architecture lets both API clients run in one Python process against the same world state without inter-process communication.The architecture connects the CARLA and AirSim clients to independent RPC servers inside the unified engine process.
  • W1: Air-ground cooperative precision landing: In cooperative precision landing, a drone tracks a moving vehicle through approach, descent, and touchdown phases using synchronous cross-domain control.The drone descends from approximately 12 m to touchdown over approximately 20 s, while horizontal error decreases from approximately 6 m to within ±0.5 m.
  • W2: Embodied navigation and VLN/VLA: CARLA-Air supports VLN/VLA dataset generation by pairing aerial bird’s-eye and ground-level visual observations with route planning and shared scene conditions.The platform provides RGB, depth, and semantic-segmentation cameras for both aerial and ground agents.
  • W3: Synchronized multi-modal dataset collection: Synchronized multi-modal collection produces 1 000 fully synchronized 12-stream records at approximately 17 Hz, with a maximum cross-stream alignment deviation of one tick.Records share a common tick index and require no timestamp interpolation.
  • W4: Air-ground cross-view perception: Cross-view perception provides spatially and temporally co-registered aerial-depth and ground-segmentation streams, producing 500 pairs at approximately 18 Hz with zero RPC errors.All 14 weather presets pass the reported evaluation.
  • W5: Reinforcement-learning policy training: A representative reinforcement-learning scenario trains a drone to maintain an aerial observation position relative to a moving ground vehicle under varying traffic conditions.The observation includes drone, vehicle, and traffic state; actions are 3D velocity commands, and reward encodes lateral tracking error and altitude maintenance.

6 Limitations and Future Work

CARLA-Air’s current validation covers moderate-density urban traffic and single- or dual-drone operations, while broader scale and reset capabilities remain under development. Future work targets physics-state synchronization, ROS 2 integration, and GPU-parallel multi-environment execution.

  • Scope boundaries: The current release is validated for single- and dual-drone operations over moderate-density urban traffic scenes.The stated validation scope covers the workflows presented in Section 5.
  • Actor density: High-density scenes with large simultaneous actor populations remain an active engineering target.Joint simulation performance has been characterized at moderate traffic loads.
  • Environment resets: Map switching requires a full process restart because of independent actor lifecycle management in the simulator backends.Staged in-session resets are planned for a future release.
  • Multi-drone scale: Configurations beyond two drones are functional but have not yet been formally validated across a wide range of scenarios.Expanded multi-drone characterization is planned after inter-agent behavior is fully profiled.
  • Maintenance: CARLA-Air maintains and extends AirSim’s aerial subsystem within its own regular update cycle after AirSim’s upstream development was archived.The project manages bug fixes, compatibility updates, and feature extensions for the aerial stack.
  • Future work: Future work includes physics-state synchronization, a ROS 2 bridge, and GPU-parallel multi-environment execution for higher reinforcement-learning throughput.These targets are described as near-term and longer-term directions.

7 Conclusion

CARLA-Air addresses fragmented air-ground simulation by integrating CARLA and AirSim in one Unreal Engine process with shared physics and rendering. It provides coherent multi-agent sensing, validated performance, and workflows for embodied-intelligence research.

  • CARLA-Air integrates CARLA and AirSim within one Unreal Engine process, preserving native Python APIs over shared physics and rendering pipelines.
  • Its composition-based design resolves Unreal Engine’s single-GameMode constraint while providing a shared tick and consistent sensing across viewpoints.
  • The platform combines photorealistic urban traffic, socially-aware pedestrians, aerodynamically consistent multirotor dynamics, and up to 18 synchronous sensor modalities.
  • CARLA-Air supports air-ground cooperation, embodied navigation, cross-view perception and datasets, and reinforcement-learning-based policy training.
  • ≈20 FPS joint workloads, zero crashes across 357 reset cycles during a 3-hour run, and five representative workflows validate stable operation.

A.1 System Configuration

CARLA-Air supports custom assets within its shared simulation world and is distributed with specified hardware, software, networking, and binary/source packages.

  • Custom mobile robots and vehicles can be imported through the extensible asset pipeline and operate alongside built-in CARLA traffic and AirSim agents.
  • Experiments use Ubuntu 20.04/22.04, an NVIDIA RTX A4000, an AMD Ryzen 7 5800X, and 32 GB DDR4-3200 memory.
  • The software stack comprises CARLA 0.9.16, AirSim 1.8.1, Unreal Engine 4.26, and Python 3.8+.
  • Both RPC servers bind to localhost by default, while remote connections require explicit IP configuration.
  • The release includes a roughly 19 GB prebuilt binary package, a roughly 651 MB MIT-licensed source distribution, and a one-command launcher.

A.1.1 API Compatibility Summary

CARLA-Air reports broad compatibility with CARLA and AirSim interfaces and publishes ROS 2 topics across both simulation backends.

  • All 89 automated CARLA API tests pass without modification, while AirSim flight-control and sensor APIs were verified through manual and scripted testing.
  • A total of 63 ROS 2 topics are published across the CARLA-Air simulation backends.

A.2 Upstream Source Modifications

CARLA-Air minimizes upstream CARLA changes by placing integration code in the aerial simulation plugin and modifying only a small number of upstream files.

  • The integration touches two header files and one source file, totaling approximately 35 lines of upstream changes.
  • The additive aerial simulation plugin contains the CARLAAirGameMode class, comprising approximately 1,405 lines of C++.

A.3 Custom Asset Import

CARLA-Air provides an Unreal Engine asset pipeline for importing custom robots, vehicles, UAVs, and environment objects into the shared simulation.

  • Imported custom assets become spawnable actor classes accessible through CARLA’s standard world.spawn_actor() API.The pipeline uses Unreal Engine’s content framework to register imported assets.
  • Registered assets share the common physics tick, rendering pass, and sensor visibility of built-in actors.This preserves consistency within the joint simulation environment.
Loading 2603.28032v2…