PaperlayerView exact revision on arXiv

Source-linked AI summary

AeroManip-VLA: Scalable Vision-Language-Action Learning for Aerial Manipulation with RL-Generated Demonstrations

Rui Huang, Yanlin Mu, Lidong Li, Yucong Wang, Zichen Yan, Lin Zhao

arXiv:2609.36915v1cs.ROcs.AI

TL;DR

Aerial VLA research needs diverse training data and systematic evaluation, but physical-platform demonstration collection and testing are costly, safety-critical, and difficult to repeat. AeroManip-VLA uses parallel simulation to generate and analyze demonstrations and evaluate policies; pretrained VLA models outperform visuomotor imitation baselines on mean success, though aerial picking remains challenging.

  • Problem

    Aerial VLA policies must handle coupled flight and manipulation, changing observations, and physical interactions, while real-platform data collection and evaluation are costly and difficult to repeat.

    Source: §1 · p. 2

  • Method

    AeroManip-VLA combines GPU-parallel simulation, payload-aware control, expert rules and RL policies for automated demonstrations, trajectory annotation, and policy evaluation.

    Source: §1 · pp. 2–3

  • Results

    Pretrained VLA models achieve higher mean success rates than visuomotor imitation baselines, although reliable aerial picking remains challenging.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The benchmark provides more than 80K demonstrations across basic skills and long-horizon navigation-and-manipulation tasks for aerial policy evaluation.

    Source: §5 · p. 9

    Evaluation is limited to simulation, and sim-to-real transfer remains unvalidated.

    Source: §5 · p. 9

Abstract

from arXiv · show

Aerial manipulators extend robotic manipulation into 3D workspaces that are difficult for ground-based robots to access, creating new opportunities for general-purpose manipulation. However, extending Vision-Language-Action (VLA) models to aerial robots introduces distinct challenges due to the tight coupling between manipulation and flight, continuously changing observations, and safety-critical physical interactions. These challenges demand diverse training data and systematic policy evaluation, yet collecting demonstrations and evaluating policies directly on physical aerial platforms are costly, difficult to scale, and hard to repeat under controlled conditions. We present AeroManip-VLA, a scalable benchmark for aerial VLA data generation and policy evaluation. AeroManip-VLA provides a GPU-accelerated simulation framework with low-level payload-aware flight and manipulation control in massively parallel environments. Building on this framework, we combine reusable reinforcement learning policies with expert task rules to automatically generate demonstrations without human teleoperation across diverse objects, environments, and randomized initial conditions. The generated data include basic skills such as grasping and placing, as well as long-horizon tasks that require both navigation and manipulation. We further introduce automated event labeling and trajectory categorization to filter demonstrations. These mechanisms enable fine-grained analysis of task progress, behavioral outcomes, and safety-related failures. Finally, we evaluate a range of imitation learning and VLA baselines across different task settings, revealing their performance characteristics and failure modes. Together, AeroManip-VLA enables scalable aerial manipulation data generation, structured trajectory analysis, and systematic VLA evaluation in simulation prior to real-world deployment.

1 INTRODUCTION

Aerial manipulation combines 3D flight with changing observations and physical interactions, while collecting demonstrations and evaluating policies on real platforms is costly and hard to repeat. AeroManip-VLA combines parallel simulation, automated demonstrations, trajectory analysis, and baseline evaluation to support scalable aerial VLA research.

1 INTRODUCTION

  • Aerial VLA training and evaluation must account for coordinated 3D flight, changing viewpoints, and physical interactions that affect flight dynamics.These conditions broaden the operating conditions and failure modes aerial VLA models must handle.Source: §1 · p. 2
  • Real-platform demonstration collection is costly and difficult to scale, while policy evaluation is slow, safety-critical, and hard to repeat under controlled conditions.The paper motivates simulation as a scalable setting for diverse data generation and controlled evaluation before deployment.Source: §1 · p. 2
  • AIR-VLA's reliance on manual teleoperation makes expanding demonstrations to new objects, scenes, initial poses, and task variations labor-intensive.This limitation motivates automated generation as scenarios and environments change.Source: §1 · p. 2
  • AeroManip-VLA combines GPU-accelerated massively parallel simulation and payload-aware low-level control with automated demonstrations across varied objects, environments, and initial poses.The demonstrations combine expert rules and RL policies without human teleoperation, and include basic skills and long-horizon navigation-plus-manipulation tasks.Source: §1 · pp. 2–3
  • The benchmark claims to be the first aerial VLA benchmark combining massively parallel simulation, automated demonstration generation, and trajectory-level analysis.Its low-level control is payload-aware and supports data generation, policy training, and evaluation across diverse aerial manipulation tasks.Source: §1 · pp. 2–3
  • Automated event labeling and trajectory categorization analyze task progress, behavioral outcomes, and safety-related failures beyond aggregate success rates.The paper also evaluates imitation-learning and VLA baselines across task settings to characterize performance and failure modes.Source: §1 · pp. 2–3

2 RELATED WORK

Prior robot-learning benchmarks largely target ground-based manipulation, leaving aerial flight–manipulation coupling and safety constraints less explored. Aerial benchmarks and automated demonstration methods have emerged, but scalable generation and analysis for diverse, long-horizon aerial tasks remain underexplored.

2 RELATED WORK

  • Major manipulation benchmarks target ground robots, leaving coupled flight–manipulation dynamics and aerial safety constraints underexplored.Examples include LIBERO [Liu et al., 2023], BEHAVIOR-1K, and GPU-parallel ManiSkill3 [Tao et al., 2024].Source: §2 · p. 3
  • AIR-VLA provides teleoperated aerial demonstrations, while AM-Bench adds diverse embodiments, disturbances, and baselines; scalable aerial demonstration generation and analysis remain underexplored.Existing aerial datasets still rely heavily on manually collected trajectories, while synthetic generation has focused primarily on navigation.Source: §2 · p. 3
  • Prior demonstration-generation methods mostly rely on human source demonstrations and target fixed-base or ground manipulation, unlike AeroManip-VLA's automated aerial approach.AeroManip-VLA combines reusable closed-loop RL policies with expert task rules and composes skills into long-horizon navigation-and-manipulation demonstrations.Source: §2 · p. 3

3 METHODOLOGY

AeroManip-VLA combines GPU-parallel aerial simulation and payload-aware control with automated expert demonstrations, learned skill policies, and event-based trajectory analysis. Its benchmark spans basic manipulation and long-horizon tasks that couple navigation with contact-rich interaction.

3.1 BENCHMARK DESIGN

  • The benchmark covers household and kitchen Pick-and-Place, outdoor package delivery, and continuous rearrangement that combines navigation, manipulation, and appliance-door operation.Five parameterized skills—Pick, Place, Open, Close, and Nav—serve as building blocks for long-horizon tasks.Source: §3.1 · p. 4, §3.1.1 · p. 4

3.1 SIMULATION, ROBOT SYSTEM, AND OBSERVATION/ACTION SPACES

  • AeroManip-VLA uses GPU-parallel simulation with lightweight quadrotors configured with either a 1-DoF gripper or a 3-DoF arm, and task policies receive RGB-D views, proprioception, and optionally language.The platform simulates physical contact rather than attaching or teleporting objects.Source: §3.1.2 · p. 4, §3.1.3 · p. 4, §3.1.4 · pp. 4–5

3.2 PAYLOAD-AWARE CONTROL FOR AERIAL MANIPULATION

  • A shared cascaded flight controller tracks velocity and yaw-rate commands, compensates for payload changes, and supports batched GPU execution across parallel environments.The low-level controller is shared by all task-level policies; payload compensation is enabled for selected object classes.Source: §3.2 · p. 5

3.3.1 PRIVILEGED HYBRID EXPERT

  • The privileged hybrid expert combines simulator state, geometric planning, learned components, rule-based guards, and scripted feedback to generate successful demonstrations and initialization resources.It is a demonstration generator, not an evaluated policy or a policy trained with reinforcement learning.Source: §3.3.1 · p. 6

3.3.2 SKILL POLICY LEARNING

  • Behavior cloning imitates continuous expert actions as a standalone skill policy, while PPO optimizes skill policies with task rewards and can start from scratch or from BC initialization.Learned skills are composed with privileged geometric planning and scripted control for task-level rollouts.Source: §3.3.2 · p. 6, §3.3 · p. 6

3.4 EVENT ANNOTATION FOR AERIAL MANIPULATION

  • Event annotations combine task interactions with aerial constraints such as tilt, altitude, contact, payload loss, stability, and clearance to assess task completion alongside physical constraints.Skill outcomes remain distinct from full-trajectory outcomes, and time-limit truncations are separated from physical failures.Source: §3.4 · p. 6
  • Transition checks require downstream skill segments to begin from eligible handoff states, supporting traceable segmentation, demonstration selection, and failure analysis.The checks exclude starts already close to task completion.Source: §3.4 · p. 6
  • BC is substantially more interaction-efficient than PPO in Figure 2, but depends on expert demonstrations.The caption reports one training seed per method.Source: Figure 2 · p. 7

4 EXPERIMENTS

The experiments compare data-generation strategies and downstream visuomotor and VLA policies. Pretrained VLAs lead the tested policies in mean success, while PPO produces more diverse and faster successful trajectories at higher training cost.

4.1 EVALUATION OF DATA GENERATION STRATEGIES

  • On Pick, BC approaches expert success with 67–81k demonstration transitions, while PPO requires millions of interactions and remains below expert on held-out evaluation.The results support using the hybrid expert for the released dataset and BC for low-cost policy distillation.Source: §4.1 · p. 7
  • PPO covers more successful approach behaviors than expert and BC, with higher spatial coverage and velocity-direction entropy on both tasks.Coverage is 205 vs. 53 (Expert) and 77 (BC) on TidyHouse, and 283 vs. 46 and 71 on PrepareGroceries; entropy is 3.0 vs. 1.8 and 2.0 bits, and 3.7 vs. 1.7 and 1.7 bits, respectively.Source: §4.1 · p. 7, Figure 3 · p. 8
  • PPO’s successful episodes are faster than expert and BC episodes on both tasks, primarily because its approach phase is shorter.Mean success time falls from 15.8 s to 3.8 s on TidyHouse and from 14.7–14.6 s to 4.0 s on PrepareGroceries; these statistics are conditioned on success.Source: §4.1 · p. 8, Figure 4 · p. 9

4.2 DOWNSTREAM POLICY BENCHMARKING

  • 44.6% mean success for π0.5 leads π0 (38.9%), DP (33.9%), and ACT (29.5%) across the evaluated baselines.The pretrained VLA models outperform the visuomotor imitation baselines, but task completion remains incomplete.Source: §4.2 · pp. 8–9
  • Success varies substantially by skill and object: Place is consistently easier than Pick, making reliable aerial object acquisition the primary bottleneck.The reported variation across objects reflects differences in geometry, graspability, and manipulation difficulty.Source: §4.2 · p. 9

5 CONCLUSION AND LIMITATIONS

AeroManip-VLA combines GPU-parallel simulation, payload-aware control, automated demonstrations, and event-based trajectory annotation. In simulation, PPO yields more diverse and faster successful trajectories at higher training cost, while pretrained VLAs achieve higher mean success than visuomotor imitation baselines; sim-to-real transfer remains unvalidated.

5 CONCLUSION AND LIMITATIONS

  • The benchmark integrates payload-aware flight control, automated demonstration generation, and event-based trajectory annotation in a GPU-parallel framework.It provides more than 80K demonstrations spanning basic skills and long-horizon navigation-and-manipulation tasks.Source: §5 · p. 9
  • BC efficiently reproduces expert behavior, while PPO finds more diverse and faster successful trajectories at higher training cost.Source: §5 · p. 9
  • Evaluation is limited to simulation, and sim-to-real transfer remains unvalidated.The authors identify validation on physical aerial manipulators as future work.Source: §5 · p. 9

A.1 TASK EXAMPLES AND QUALITATIVE ROLLOUTS

The examples span short-horizon object interaction and longer tasks that combine manipulation with aerial transport.

A.1 TASK EXAMPLES AND QUALITATIVE ROLLOUTS

  • PICK and PLACE are short-horizon examples focused on local object interaction.Source: §A.1 · p. 13
  • PICK–NAVIGATE–PLACE adds aerial transport between spatially separated source and destination regions.Source: §A.1 · p. 13
  • OPEN–PICK–NAVIGATE–PLACE adds refrigerator opening before retrieval, transport, and placement; the camera illustration shows forward and gripper-mounted views.The central third-person view is not supplied to the evaluated policy.Source: §A.1 · p. 13, Figure 6 · p. 14

A.2 OBSERVATION SPACE

The policy observation combines three onboard RGB-D views with proprioception, and language-conditioned policies also receive a task instruction.

A.2 OBSERVATION SPACE

  • The visual input uses one forward-facing FPV camera for global scene perception and navigation, plus two gripper-mounted cameras for local manipulation views.The RGB-D streams are recorded at 128 × 128 resolution.Source: §A.2 · p. 14
  • The 19-dimensional proprioceptive state contains gripper states, UAV position and orientation, and linear and angular velocities.Source: §A.2 · p. 14
  • Language-conditioned policies additionally receive a natural-language task instruction, while the third-person camera view is illustrative and not a policy input.Source: §A.2 · p. 14

A.3 GPU-PARALLEL DATA COLLECTION AND INFERENCE

AeroManip-VLA combines parallel simulation, automated RL demonstration generation, and batched VLA inference. It supports large-scale aerial manipulation research on both data-center and consumer NVIDIA GPUs.

A.3 GPU-PARALLEL DATA COLLECTION AND INFERENCE

  • RL policies automatically generate diverse aerial manipulation trajectories across parallel simulation environments without manual teleoperation.The same framework supports batched policy inference for efficient, large-scale VLA evaluation.Source: §A.3 · p. 14
  • Automated data generation and parallel inference provide a scalable foundation for aerial manipulation research.The passage identifies NVIDIA H200, A100, and GeForce RTX 5090 as supported GPUs.Source: §A.3 · p. 14
  • The simulator runs on NVIDIA H200, A100, and GeForce RTX 5090 GPUs, spanning data-center and consumer hardware.Source: §A.3 · p. 14

A.4 RAY TRACING AND VISUAL FIDELITY

AeroManip-VLA supports optional ray-traced rendering for visually richer aerial manipulation scenes. Figure 8 compares simulator-rendered views with ray tracing disabled and enabled.

A.4 RAY TRACING AND VISUAL FIDELITY

  • Optional ray tracing provides visually richer observations for aerial manipulation demonstration collection and VLA evaluation.It complements the non-ray-traced rendering option.Source: §A.4 · p. 14
  • GPU-parallel environments support automated RL demonstration generation and batched VLA inference alongside the rendering options.Source: Figure 7 · p. 15
  • Figure 8 contrasts the same scene rendered with ray tracing disabled and enabled.The comparison illustrates differences in illumination, shadows, and material appearance.Source: Figure 8 · p. 15

A.5.1 SIMULATION THROUGHPUT AND SCALABILITY

This section reports simulation scalability, payload-aware control and policy failure findings, and describes the experimental pipeline and aerial platforms. It also details trajectory outcome checks and labeling, including completion and safety criteria and separate skill-level outcomes and parent-rollout diagnostics.

A.5.1 SIMULATION THROUGHPUT AND SCALABILITY

  • At the highest-throughput tested configurations, AeroManip-VLA reaches 1707 SPS with 1024 environments versus AIR-VLA’s 317 SPS with 64, while using 12.4 GB versus 30.5 GB.At N = 64, AeroManip-VLA has approximately 1.6× the throughput and uses 86% less GPU memory; AIR-VLA is faster at N ≤16, and throughput is comparable at N = 32.Source: §A.5.1 · pp. 15–16, Figure 9 · p. 16

A.5.2 PAYLOAD-AWARE FLIGHT CONTROL

  • Payload-aware control raises success from 41% to 92% and reduces median grasp-to-lift-off time from 3.9 s to 2.3 s, mean vertical tracking error from 0.73 to 0.07, and RMS tilt rate from 15.5 to 7.9 deg/s.The controller adds object mass to feedforward and raises attitude gains after three grasped control steps, then restores nominal feedforward when the object is supported. At the nominal controller’s 1.4 Hz tilt-rate peak, payload-aware control reduces median PSD from 94 to 6.2 (deg/s)2/Hz, while its spectrum is higher over 2–5 Hz.Source: §A.5.2 · pp. 16, 18, Figure 12 · p. 18, Figure 13 · p. 19

A.5.3 FAILURE ANALYSIS

  • Replacing expert grasps with PPO grasps raises shared-transport failures to 16.7% in TIDYHOUSE and 30.3% in PREPAREGROCERIES, and reduces expert-place success to 79.4% and 72.9%, respectively.PPO-place success is 79.2% versus 69.9% in TIDYHOUSE, but 90.6% versus 93.0% in PREPAREGROCERIES, depending on grasp source; the findings show that expert-state-only skill evaluation can overestimate end-to-end performance.Source: §A.5.3 · p. 19, Figure 14 · p. 20
  • PPO pick failures concentrate by object class, while fast approaches, missed gripper closure, moving releases, and rotated handoffs expose distinct failure mechanisms.PPO fails on every PREPAREGROCERIES bowl episode and 48% of potted-meat episodes; its median contact speed is 0.152 m/s in TIDYHOUSE and 0.072 m/s in PREPAREGROCERIES, and it never closes the gripper in 11.4% of PREPAREGROCERIES episodes. PPO releases at 0.20–0.25 m/s; cracker and sugar boxes topple in 57% and 43% of TIDYHOUSE episodes, while potted-meat and tomato cans are dropped in 43% and 13% of PREPAREGROCERIES episodes. PPO handoff rotation reaches medians of 6.4° and 5.1°; grasp-loss rates reach 24% and 48% above 16° in TIDYHOUSE and PREPAREGROCERIES, respectively.Source: §A.5.3 · pp. 20–22

A.6 EXPERIMENTAL IMPLEMENTATION

  • AeroManip-VLA combines simulation, automated expert and RL data generation, short- and long-horizon tasks, and trajectory validation for IL, RL, and VLA use.Recorded trajectories include synchronized multi-view RGB-D observations, robot states, actions, task annotations, and provenance metadata, and are checked for physical execution, collisions, task completion, and temporal consistency.Source: §A.6 · p. 22, Figure 19 · p. 23

A.6.1 ROBOT PLATFORMS

  • AeroManip-VLA supports downward-facing gripper and forward-reaching arm platforms that share an X500 airframe, flight interface, and simulation setup.The arm platform extends reach to vertical handles and uses stronger finger drive and rotors, arm-weight feedforward torque, and a contact mode that disables shoulder torque and arm feedforward while grasping a handle.Source: §A.6.1 · pp. 22, 24, Figure 20 · p. 23

A.7 TRAJECTORY ANNOTATION AND OUTCOME CATEGORIZATION

  • Trajectory outcomes use simulator-derived progress indicators, failure flags, and task-specific completion checks; event times are independent observations, and time-limit truncations indicate incomplete execution rather than physical failures.Source: §A.7 · pp. 24–25
  • Outcome checks cover goal attainment, release, arm retraction, platform stability, airframe contact, and geometric clearance, with explicit thresholds for retraction, stability, release, and clearance.Retraction requires joint deviation below 0.05 rad; stability requires linear speed below 0.05 m/s and angular speed below 0.2 rad/s; release requires the fingers not both exceed 0.1 N; clearance requires at least 0.03 m.Source: §A.7 · pp. 25–26
  • Export configuration determines whether failed and truncated trajectories are retained alongside successful demonstrations for task-specific selection and analysis.Source: §A.7 · p. 26
  • Segmented trajectories store skill labels separately from parent-rollout diagnostics, so completed skills remain valid despite later-stage failure while concurrent physical failures remain failures.Place segments must also start from an eligible held state outside the destination support region.Source: §A.7 · p. 27
Loading 2609.36915v1…