Source-linked AI summary
Aerial Manipulation in the Wild with Onboard Perception, Policy Learning, and Whole-Body Control
Yuanzhu Zhan, Yufei Jiang, Zemu Zhang, Junyi Geng
TL;DR
Outdoor aerial manipulation requires state estimation, stable flight, and precise manipulation under less controlled conditions. This paper combines a demonstration-trained Diffusion Policy, onboard LiDAR-inertial state estimation, and whole-body MPC, then tests outdoor transfer from indoor demonstrations. The system completes all three reported outdoor peg-in-hole trials without human intervention after policy execution begins.
Problem
Outdoor aerial manipulation must manage reliable state estimation, stable flight, and precise manipulation without relying on laboratory motion-capture infrastructure, while transfer from demonstrations across environments remains an open question.
Source: §I · p. 1Method
The framework combines a Diffusion Policy trained from demonstrations, onboard LiDAR-inertial state estimation, and whole-body MPC that coordinates the aerial platform and manipulator.
Source: §I · p. 1, C. Whole-Body Model Predictive Control · p. 2Results
All three outdoor peg-in-hole trials succeeded without human intervention after policy execution began, using a policy trained entirely from indoor demonstrations without retraining.
Source: 2) Outdoor Experiments · pp. 3–4Takeaways & Limitations
The experiments demonstrate outdoor deployment of indoor-trained aerial manipulation without external motion capture, outdoor demonstrations, or policy retraining.
Source: §IV · p. 4
Abstract
from arXiv · showhide
Aerial manipulation in outdoor environments remains challenging due to the simultaneous requirements of reliable state estimation, stable aerial motion, and precise manipulation under external disturbances. In this work, we present a real-world outdoor aerial manipulation framework that integrates imitation learning, onboard LiDAR-inertial state estimation, and whole-body model predictive control. A Diffusion Policy is trained from manipulation demonstrations to generate desired end-effector motions from onboard observations. These learned commands are executed by a whole-body MPC that jointly coordinates the aerial platform and manipulator to realize the desired end-effector trajectory. To eliminate reliance on external motion-capture infrastructure, the platform employs onboard LiDAR-inertial odometry for state estimation during outdoor operation. We validate the complete framework on a physical aerial manipulator and demonstrate successful execution of outdoor manipulation tasks. The experimental results show that demonstration-driven manipulation policies can be effectively integrated with onboard state estimation and model-based whole-body control to enable aerial manipulation beyond controlled indoor environments.
I. INTRODUCTION
Outdoor aerial manipulation must combine stable flight, precise end-effector control, and state estimation without laboratory motion capture. The paper presents an integrated framework and tests whether an indoor-demonstration policy transfers outdoors without retraining.
I. INTRODUCTION
- Aerial manipulation adds coupled control demands because arm motion affects the floating base while the vehicle must maintain stable flight and control the end effector.Outdoor operation also brings less controlled conditions and removes access to laboratory motion-capture infrastructure.Source: §I · p. 1
- The study tests whether a visuomotor policy trained solely from indoor demonstrations can work outdoors without retraining.Prior learning-based aerial manipulation systems had largely been evaluated indoors, while outdoor task-specific pipelines did not establish cross-environment transfer.Source: §I · p. 1
- The framework combines Diffusion Policy, onboard LiDAR-inertial state estimation, and whole-body MPC for outdoor aerial manipulation.The paper also reports real-world outdoor validation without external motion capture, outdoor demonstrations, or policy retraining.Source: §I · p. 1
- A whole-body MPC coordinates aerial-base and manipulator motion to execute learned end-effector trajectories.This targets the coupled challenge of keeping the vehicle stable while accurately controlling the end effector.Source: §I · p. 1
II. METHODOLOGY
The system pairs a fully actuated hexarotor and serial arm with onboard sensing and coordinated control. Its policy is trained offline from handheld demonstrations using an end-effector-centric interface.
A. System Overview
- The platform comprises a fully actuated hexarotor, a 4-DoF serial manipulator, and an end-effector with an OAK-1W camera.During deployment, the policy generates end-effector commands, LiDAR–inertial odometry supplies floating-base feedback, and the controller coordinates the base and arm.Source: A. System Overview · p. 1
- At deployment, onboard LiDAR–inertial odometry provides floating-base state feedback while a whole-body controller coordinates the aerial base and manipulator.The learned policy supplies the end-effector commands.Source: A. System Overview · p. 1
- Policy demonstrations are collected offline with a handheld UMI device using an end-effector-centric interface intended to facilitate transfer to the aerial manipulator.The shared interface links handheld demonstrations to the aerial system's end effector.Source: A. System Overview · p. 1
B. Diffusion Policy for End-Effector Action Generation
The Diffusion Policy learns to denoise action trajectories conditioned on observation history, then generates future end-effector actions at inference. Its end-effector-centric observations and outputs separate learned behavior from platform-specific control.
B. Diffusion Policy for End-Effector Action Generation
- During training, the conditional denoising network learns to recover demonstrated action sequences after noise is progressively added; at inference it iteratively denoises a noisy sequence conditioned on observations.The denoising procedure follows DDIM.Source: B. Diffusion Policy for End-Effector… · pp. 1–2
- The observation window contains egocentric RGB images, relative end-effector poses, and gripper widths, while the policy predicts future positions, orientations, and gripper-width commands.The observation horizon is L; the planning horizon is N.Source: B. Diffusion Policy for End-Effector… · p. 2
- The end-effector-centric policy avoids directly commanding the base, joints, or actuators, leaving platform-specific kinematics and dynamics to the downstream whole-body controller.Relative pose inputs also reduce dependence on precise global localization.Source: B. Diffusion Policy for End-Effector… · p. 2
- Human demonstrations collected with a handheld UMI device are converted into observation–action sequences and used to train a conditional UNet-based Diffusion Policy.The handheld camera–gripper configuration closely matches that of the aerial end effector.Source: B. Diffusion Policy for End-Effector… · p. 2
C. Whole-Body Model Predictive Control
The whole-body MPC coordinates the aerial platform and manipulator to track policy-generated end-effector trajectories. Its formulation combines whole-body dynamics with tracking and regularization costs, using forward kinematics to obtain end-effector pose.
C. Whole-Body Model Predictive Control
- The whole-body MPC jointly coordinates the aerial platform and manipulator to execute learned end-effector trajectories.It tracks the end-effector trajectory generated by the policy.Source: C. Whole-Body Model Predictive Control · p. 2
- The state includes aerial-base position, velocity, attitude, angular velocity, and arm configuration; controls are force, torque, and desired joint configuration.Source: C. Whole-Body Model Predictive Control · p. 2
- The controller uses a discretized lightweight whole-body dynamics model, with a small-angle approximation for base attitude dynamics.The nonlinear optimal-control problem is solved with SQP-RTI, applying only the first optimized input at each receding-horizon step.Source: C. Whole-Body Model Predictive Control · pp. 2–3
- The stage cost tracks policy-generated end-effector position and orientation while regularizing base motion, attitude, arm posture, control effort, and command smoothness.The terminal cost retains tracking and state-regularization terms but excludes control-effort and input-smoothness penalties.Source: C. Whole-Body Model Predictive Control · p. 3
- Forward kinematics maps aerial-base state and manipulator configuration to the world-frame end-effector pose used for tracking.Source: C. Whole-Body Model Predictive Control · p. 3
III. EXPERIMENTS
The experiments assess whether onboard LiDAR-inertial odometry provides sufficiently reliable outdoor state feedback and whether an indoor-trained policy transfers to an unseen outdoor environment without retraining.
III. EXPERIMENTS
- The evaluation asks whether onboard LIO provides sufficiently reliable state feedback for outdoor aerial manipulation.Source: §III · p. 3
- The evaluation also tests whether a policy trained exclusively from indoor demonstrations transfers to an unseen outdoor environment without retraining.Source: §III · p. 3
- Together, the experiments examine state-estimation reliability and indoor-to-outdoor policy transfer.Source: §III · p. 3
A. Experimental Setup
The setup evaluates a policy trained on indoor peg-in-hole demonstrations, used unchanged indoors and outdoors with onboard FAST-LIO2 state estimation. Trials and controller metrics are specified for both environments.
A. Experimental Setup
- The policy is trained on 80 indoor demonstrations and used unchanged outdoors, without outdoor demonstrations, fine-tuning, or environment-specific adaptation.The demonstrations vary initial gripper position and target-hole location; outdoor state estimation uses FAST-LIO2 without motion-capture infrastructure.Source: A. Experimental Setup · p. 3
- The peg-in-hole task inserts a 20 mm stick into a 50 mm hole positioned approximately 130 cm above ground.Source: A. Experimental Setup · p. 3
- Table I covers end-effector tracking and base-attitude performance in indoor and outdoor experiments.Source: Table I · p. 3
- Trials comprise five indoor runs and three runs in an unseen outdoor grass field; success requires insertion without human intervention after policy execution begins.Source: A. Experimental Setup · p. 3
- Controller performance is assessed using normalized end-effector position tracking error and maximum base tilt.Maximum base tilt measures the largest roll-pitch attitude excursion during task execution.Source: A. Experimental Setup · p. 3
1) Indoor Experiments:
The learned policy completed all five indoor peg-in-hole trials despite varied starting gripper configurations and target locations. The whole-body MPC tracked the policy trajectory with a normalized end-effector error of 0.102 while maintaining small vehicle tilt.
1) Indoor Experiments:
- All five indoor peg-in-hole trials succeeded despite variations in initial gripper configuration and target-hole location.Source: 1) Indoor Experiments · p. 3
- 0.102 was the normalized end-effector tracking error achieved by the whole-body MPC indoors.Source: 1) Indoor Experiments · p. 3
- The indoor experiments establish baseline performance before outdoor evaluation, with the controller tracking policy-generated trajectories while maintaining stable vehicle motion.Source: 1) Indoor Experiments · p. 3
2) Outdoor Experiments:
The same policy trained indoors generalized to an unseen outdoor environment, completing peg-in-hole trials from a different starting configuration. Onboard LiDAR–inertial odometry supported outdoor execution while tracking performance remained similar to indoor experiments.
2) Outdoor Experiments:
- Outdoor deployment used onboard LIO for floating-base state estimation while keeping the learned Diffusion Policy and low-level controller unchanged.The evaluation tested the same learned policy in an unseen outdoor environment.Source: 2) Outdoor Experiments · p. 3
- Three outdoor peg-in-hole trials succeeded, including from an initial configuration different from the demonstrations.The policy generated the approach and final insertion motion.Source: 2) Outdoor Experiments · p. 3, Figure 3 · p. 4
- Indoor and outdoor experiments showed comparable normalized end-effector tracking errors, with maximum base tilt below 4° in both environments.The outdoor tracking comparison used onboard LIO in place of motion capture.Source: 2) Outdoor Experiments · pp. 3–4, Figure 4 · p. 4
IV. CONCLUSION
The framework combines Diffusion Policy, onboard LiDAR–inertial state estimation, and whole-body MPC. Experiments showed that indoor demonstrations could support deployment in an unseen outdoor environment without motion capture or retraining.
IV. CONCLUSION
- The framework integrates Diffusion Policy, onboard LiDAR–inertial state estimation, and whole-body MPC for outdoor aerial manipulation.It was evaluated on a physical aerial manipulator.Source: §IV · p. 4
- A policy trained entirely from indoor demonstrations was deployed in an unseen outdoor environment without external motion capture, outdoor demonstrations, or policy retraining.Source: §IV · p. 4
- The authors identify evaluation on more diverse manipulation tasks and challenging outdoor conditions as future work.Source: §IV · p. 4