PaperlayerView exact revision on arXiv

Source-linked AI summary

Whole-Body Aerial Grasping and Lifting via Partial Visual Observations

Jiaye Jin, Rui Jin, Xinhang Xu, Haotian Jin, Ruiyang Liu, Yi Wang, Jiayan Zhao, Kun Cao, Lihua Xie

arXiv:2610.00404v1cs.RO

TL;DR

Aerial grasp-and-lift requires whole-body coordination under partial target observations, while early approach failures can restrict learning of later stages and changing visibility complicates control. The paper trains a recurrent teacher–student policy with a critical-state curriculum and dual-view visual feedback; its frozen student achieves high full-task success across tested simulated conditions.

  • Problem

    Aerial grasp-and-lift requires coordinating base, arm, and closure under changing viewpoints and occlusions, while early approach failures can prevent learning from reaching later task stages.

    Source: §I · p. 1

  • Method

    A privileged teacher trained with critical-state resets is distilled into a recurrent visual student using dual-view point clouds, proprioception, and closure supervision, with no explicit task-phase input.

    Source: §I · p. 2

  • Results

    Across 8,996 completed episodes, the frozen student achieves 99.97%, 97.14%, and 95.84% full-task success under nominal, physics/control-randomized, and additional camera-randomized conditions.

    Source: §I · p. 2

  • Takeaways & Limitations

    The results demonstrate high full-task success for a phase-free visual student under the tested simulated acquisition-and-payload conditions.

    Source: §I · p. 2

    Physical-platform validation remains outside this study; primary evaluation uses a simulated acquisition-and-payload model.

    Source: §I · p. 2

Abstract

from arXiv · show

Aerial grasp-and-lift tasks require whole-body coordination across approach, acquisition, and lifting under partial target observations. Early approach failures can limit exposure to later task stages during training, while changing visibility complicates alignment and closure timing during execution. We present a recurrent teacher-student framework that learns a single policy in simulation to jointly command flight, arm motion, and gripper closure without an explicit task-phase input. A privileged teacher learns through reinforcement learning with a critical-state curriculum that exposes acquisition and lifting states before connecting them to normal approach trajectories. Its behavior is distilled into a recurrent visual student that replaces privileged target states with dual-view point clouds and proprioception, integrating observation history for closed-loop control. A dedicated closure objective supervises closure timing from sustained model-defined readiness sequences. Training and primary evaluation use a simulated acquisition-and-payload model with condition-triggered latching, virtual attachment, and wrench-based payload loading for short-distance lifting. Across 8,996 completed simulation episodes under this model, the frozen student achieves full-task success rates of 99.97%, 97.14%, and 95.84% under nominal, physics/control-randomized, and additional camera-randomized conditions, respectively. The nominal latch-count-weighted mean of per-seed 90th-percentile alignment errors at acquisition is 8.12 mm.

I. INTRODUCTION

Aerial grasp-and-lift requires coordinated flight, arm motion, and closure despite changing observations and early training failures that can block access to later task stages. The paper presents a recurrent visual student distilled from a privileged teacher and reports high full-task success under a simulated acquisition-and-payload model.

I. INTRODUCTION

  • Changing viewpoints and occlusions can leave a single point cloud short of the information needed to assess alignment or time closure, motivating recurrent visual feedback.The aerial base, arm, and gripper must coordinate relative motion, closure timing, and stabilization after acquisition.Source: §I · p. 1
  • A recurrent visual student replaces privileged target states with dual-view point clouds and proprioception while jointly commanding flight, arm motion, and gripper closure without a task-phase input.A shared geometric encoder and recurrent state integrate target geometry and observation history; closure timing is supervised from sustained, model-defined readiness.Source: §I · p. 2
  • The frozen student achieves 99.97%, 97.14%, and 95.84% full-task success under nominal, physics/control-randomized, and additional camera-randomized conditions across 8,996 completed simulation episodes.The nominal latch-count-weighted mean of per-seed 90th-percentile alignment errors at acquisition is 8.12 mm.Source: §I · p. 2
  • A critical-state curriculum exposes near-acquisition, bridge, and post-acquisition behaviors before connecting them to normal approach starts.Early approach failures can prevent reinforcement learning from reaching successful closure or loaded flight.Source: §I · pp. 1–2

II. RELATED WORKS

Prior aerial manipulation work uses compliant grippers, optimized base–arm trajectories, or learned flight policies, while this paper directly commands flight, arm joints, and gripper from partial geometry. It also relates critical-state curriculum learning and recurrent visual transfer to manipulation methods, then formulates the task using an environment-defined simulated acquisition model.

A. Whole-Body Coordination for Aerial Manipulation

  • The actor directly commands flight, arm joints, and gripper from partial geometry, unlike cited approaches using compliant grippers, optimized base–arm trajectories, or whole-body model predictive control.Different interaction models and evaluation platforms preclude claims of superior physical grasping performance.Source: A. Whole-Body Coordination for Aerial… · p. 2

B. Learning Manipulation with Geometric Feedback

  • The curriculum exposes critical acquisition and lifting states before connecting behaviors to normal approach starts, while recurrent visual transfer replaces privileged target geometry with dual-view observations and state history.The recurrent state supports motion and closure decisions as visibility changes and coordinates alignment, gripper timing, and post-acquisition stabilization.Source: B. Learning Manipulation with… · p. 2

III. PROBLEM STATEMENT

  • The task uses a recurrent policy in a partially observable setting, with acquisition defined by an environment-triggered simulated latch and success requiring a sustained lift of both object and aerial base.The latch model retains alignment, relative-motion, closure-timing, and payload-loading couplings while avoiding costly or unstable thin-finger contact simulation in large-scale parallel reinforcement learning.Source: §III · pp. 2–3

IV. METHOD

The method trains a privileged recurrent teacher and transfers its shared flight–arm–gripper action policy to visual control. Teacher rewards guide approach, braking, acquisition, and lifting.

IV. METHOD

  • The teacher and student use the same flight–arm–gripper action interface throughout the task.The teacher is trained first from target geometry, then transferred to a student using dual-view camera observations.Source: §IV · p. 3

1) Observation and action spaces:

  • The teacher observes target points and proprioceptive and privileged state, then issues thrust, body-rate, arm-joint, and gripper commands.The teacher observation combines target points with body motion, altitude, arm and finger state, end-effector position, previous action, and privileged errors.Source: 1) Observation and action spaces · p. 3

2) Reward design:

  • The PPO teacher is trained with approach, acquisition, and lifting rewards, alongside visibility, motion-regularization, and progress-shaping terms.Additional costs penalize excessive radial approach speed, near-target motion, and overshoot.Source: 2) Reward design · p. 3
  • A smooth distance-dependent blend shifts alignment from base positioning toward end-effector positioning as the target nears.Near and far distances bound the transition; the resulting reference velocity provides braking guidance as alignment error decreases.Source: 2) Reward design · p. 3
  • Acquisition rewards readiness and closure dwell while penalizing premature closure; lifting rewards progress, height, uprightness, and holding.Acquisition also requires admissible geometry, posture, clearance, altitude, recent visibility, and limits on relative speed, body angular speed, and object motion.Source: 2) Reward design · p. 3

3) Policy representation and termination:

The policy combines an encoded point-cloud feature with state input and a recurrent network, without a task-phase label or acquisition flag. Episodes end at configured failures or the time limit.

3) Policy representation and termination:

  • A PointNet maps the point-cloud input to a 128-dimensional feature that is concatenated with state input.The available passage identifies the feature as order-invariant.Source: 3) Policy representation and termination · p. 3
  • The policy uses a 256–128 MLP, a 128-unit GRU, and PPO action and value heads; the actor receives no task-phase label or acquisition flag.The recurrent state resets each episode.Source: 3) Policy representation and termination · p. 4
  • Episodes end at the time limit or configured failures, including crashes and numerical failures, and may also end on enabled drop conditions.Controlled-approach success can end early curriculum episodes, but acquisition does not end full-task rollouts.Source: 3) Policy representation and termination · p. 4

4) Critical-state curriculum:

The critical-state curriculum samples from approach, near-acquisition, bridge, and acquired-object reset distributions. It progressively connects assisted acquisition and lifting starts to normal approach starts.

4) Critical-state curriculum:

  • Four reset distributions cover approach, near acquisition, bridge states linking approach to closure, and acquired-object lifting starts.Each reset uses consistent base, arm, gripper, and object states; acquired-object starts begin below the required lift height.Source: 4) Critical-state curriculum · p. 4
  • Training progresses from controlled approach to closure and lifting with assisted resets, then extends bridge starts toward normal approach conditions.The effective reset distribution is defined by stage-dependent sampling probabilities.Source: 4) Critical-state curriculum · p. 4
  • The final curriculum stage removes assisted resets and trains exclusively from the approach-start distribution ρ0.Source: 4) Critical-state curriculum · p. 4

1) Dual-view geometry and recurrent policy:

The visual student combines body- and wrist-camera geometry with proprioception and recurrent state. Teacher initialization and readiness-based labels supervise its policy and closure decisions.

1) Dual-view geometry and recurrent policy:

  • The student back-projects masked depth pixels from body- and wrist-mounted cameras and transforms the resulting points into the body frame.The simulator supplies target instance masks, so segmentation is outside the learned policy; each view has 64 point slots and validity masks.Source: 1) Dual-view geometry and recurrent… · p. 4
  • A shared PointNet encoder pools valid points from both views, and the resulting feature is fused with proprioception for a recurrent GRU policy.The student omits the teacher’s seven privileged components; its recurrent state has 128 dimensions.Source: 1) Dual-view geometry and recurrent… · pp. 4–5

2) Teacher initialization and closure supervision:

  • The student’s encoder, fusion MLP, GRU, and flight–arm head are initialized from compatible teacher parameters, while its gripper head is initialized separately.The gripper head uses the teacher’s gripper output with reversed sign and a threshold shift, then is trained with binary readiness labels.Source: 2) Teacher initialization and closure… · p. 5
  • Model-defined readiness requires all specified geometry, precision, relative-speed, body-rate, object-motion, posture, clearance, and altitude checks to pass.This readiness is distinct from the acquisition gate’s recent-dynamics window and closure dwell.Source: 2) Teacher initialization and closure… · p. 5
  • Closure labels are positive for an acquired object or readiness sustained over the next three steps, with the first five steps overridden to open.The training objective weights action and action-change matching, gripper classification, and target-error regression; execution uses current and past observations.Source: 2) Teacher initialization and closure… · p. 5

3) Behavioral cloning, DAgger, and consolidation:

Teacher rollouts provide the initial flight–arm labels, while closure labels use look-ahead readiness; learner-visited trajectories then add supervision before final consolidation.

3) Behavioral cloning, DAgger, and consolidation:

  • Offline behavioral cloning first trains the student on teacher rollouts.The archived student uses 30 behavioral-cloning epochs.Source: 3) Behavioral cloning, DAgger, and… · p. 5
  • DAgger-style aggregation alternates learner-visited trajectory collection with supervised updates, using teacher labels for flight–arm actions and look-ahead labels for closure.Teacher-action selection falls from 0.25, 0.15, and 0.10 in the first three rounds to 0.05 thereafter.Source: 3) Behavioral cloning, DAgger, and… · p. 5
  • Final consolidation trains on accumulated data, including camera-randomized trajectories, and evaluation uses the frozen student without teacher intervention.Consolidation uses 40 epochs at a learning rate of 5 × 10^-5.Source: 3) Behavioral cloning, DAgger, and… · p. 5

V. EXPERIMENTS

The experiments assess curriculum learning, visual transfer, whole-body coordination, and robustness using a configured acquisition-and-payload model, with separate checks of acquisition and loaded-flight failures.

V. EXPERIMENTS

  • The experimental program evaluates curriculum learning, privileged-to-visual transfer, whole-body coordination, and robustness, while separating pre-acquisition from post-acquisition lift-and-hold failures.Experiments use the acquisition-and-payload model unless specified otherwise.Source: §V · p. 5

A. Experimental Setup

  • Training and primary evaluation use Isaac Lab, with separate MuJoCo trials for sim-to-sim transfer to additional object configurations.Main evaluations use a 10-ms physics timestep and 50-Hz policy control with a nominal 100-g cylinder.Source: A. Experimental Setup · p. 5
  • Configured acquisition requires admissible geometry, motion, posture, a close command, and a three-step dwell; acquired objects follow the gripper frame under payload-reaction wrench loading.Filtered finger–cup collisions mean the setup does not validate frictional holding, force closure, or resistance to slip.Source: A. Experimental Setup · pp. 5–6
  • Table II compares full-task success across direct, simple two-stage, and selected full-curriculum schedules, with the first two averaged across three trained policies.Each policy is evaluated with three rollout seeds.Source: Table II · p. 6
  • Full-task success requires acquisition followed by a sustained lift-and-hold endpoint, with specified height, posture, position, and speed conditions.The completion conditions must persist for ten control steps (0.20 s).Source: A. Experimental Setup · p. 6
  • Main tests evaluate the same frozen student under nominal, physics/control-randomized, and additionally camera-perturbed conditions, pooling completed episodes across rollout seeds.Each main test uses 64 parallel environments and 2,000 control steps per rollout seed.Source: A. Experimental Setup · p. 6

B. Comparison of Curriculum Learning Strategies

The experiments compare training schedules and visual transfer, then examine acquisition alignment and coordinated execution under the configured interaction model.

B. Comparison of Curriculum Learning Strategies

  • Approach pretraining yields no full-task success in the tested two-stage runs, while the selected full-curriculum reference exceeds the baseline averages.Figure 4 plots full-task success against cumulative environment timesteps for nominal and physics/control-randomized conditions.Source: Figure 4 · p. 6, B. Comparison of Curriculum Learning… · p. 6

C. Visual Policy Learning and Teacher–Student Distillation

  • The final Student trails the Teacher by 0.03 percentage points under nominal conditions and 2.76 points under physics/control DR.This comparison evaluates the privileged Teacher, BC-only visual Student, and final Student; Table III reports full-task success under the configured model.Source: Table III · p. 6, C. Visual Policy Learning and… · p. 6
  • The BC-only checkpoint has no full-task successes after 30 offline imitation epochs, and the final Student's gains cannot be attributed to data aggregation alone.The final Student succeeds after learner-visited training, further optimization, and consolidation.Source: C. Visual Policy Learning and… · p. 6
  • Removing D405 leaves no successes under nominal or combined physics/control-plus-camera DR, while D405-only success is 12.40% and 9.86%, respectively.These single-view variants test the frozen policy's dependence on joint camera inputs and are not retrained baselines.Source: C. Visual Policy Learning and… · p. 6

D. Alignment at Acquisition and Coordinated Execution

  • The latch-count-weighted alignment-error p90 is 8.12 mm nominally, 8.53 mm under physics/control DR, and 8.36 mm with additional camera DR.These are weighted means of per-seed p90 values, not percentiles of pooled samples; the same updated runs recorded 3,056, 2,931, and 2,833 latch samples.Source: D. Alignment at Acquisition and… · pp. 6–7
  • Accepted-latch alignment changes little across profiles despite lower acquisition rates, and the acquisition gate means these statistics do not describe all approaches.Fewer episodes satisfy the acquisition conditions under randomization.Source: D. Alignment at Acquisition and… · p. 7
  • Nominally, alignment error and relative speed decrease before closure, followed by increasing finger positions and height gain; shaded execution phases are not policy inputs.Median acquisition time is 1.70 s and completion time is 2.32 s.Source: D. Alignment at Acquisition and… · p. 7, Figure 5 · p. 7
  • Geometric overlap at latch does not establish contact-force support, force closure, or stable physical retention because finger–cup collisions are filtered.The nominal diagnostic observed sampled points from both finger collision-mesh surfaces inside the cup at the first latch frame in 121 completed episodes.Source: D. Alignment at Acquisition and… · p. 7

E. Robustness and Payload-Load Sensitivity

The frozen student sustains high full-task success under randomized conditions, while heavier payloads expose a post-acquisition lift-and-hold limitation. The largest tested payload meeting the study’s confirmation criteria is 300 g.

Robustness

  • For the nominal payload, the final Student maintains pooled success above 95% under both physics/control domain randomization and additional camera perturbations, though the latter shows greater rollout variability.Per-seed success ranges were 97.09–97.21% under physics/control DR and 94.52–97.17% with additional camera perturbations.Source: E. Robustness and Payload-Load… · p. 7
  • Pre-acquisition crashes occur in 2.40% of physics/control-randomized and 3.51% of additionally camera-randomized trials, versus post-acquisition rates of 0.43% and 0.51%.Most failures occur before acquisition under the nominal payload; after acquisition, the policy usually completes lift-and-hold.Source: E. Robustness and Payload-Load… · p. 7

Payload-load sensitivity

  • The single-seed mass sweep gives 99.80–100% success from 25 to 200 g, then 91.03%, 52.88%, and 17.09% at 300, 400, and 500 g.The three-seed confirmations include the sweep seed and should not be pooled again.Source: Figure 6 · p. 7, E. Robustness and Payload-Load… · p. 7
  • In three-seed confirmation, 300 g achieves 90.50% success and 400 g achieves 51.75%; only 300 g meets both study-defined payload criteria.Those criteria require at least 90% pooled success and at least 85% success for every evaluation seed.Source: E. Robustness and Payload-Load… · pp. 7–8
  • The 300 g result is the largest tested passing confirmation point, not a continuous payload limit or evidence about untested masses.The evaluation uses an acquisition-and-payload model with condition-triggered latching and payload-wrench loading.Source: E. Robustness and Payload-Load… · p. 8
  • Both confirmed payloads achieve 100% acquisition without recorded crashes or drops, so heavier-load failures arise in post-acquisition lift-and-hold conditions.This separates acquisition from loaded stabilization under the attachment-and-wrench model.Source: E. Robustness and Payload-Load… · p. 8

F. Evaluation on Additional Objects

MuJoCo trials evaluate acquisition and lifting on three additional objects, with outcomes that distinguish stable-hold failures from unsafe surface contact. The conclusion confines the results to the tested interaction models and leaves physical grasp retention unvalidated.

F. Evaluation on Additional Objects

  • Across 30 trials per object, the screwdriver succeeds in all 30 native-contact trials, while sugar-box and Coke-can outcomes include distinct post-acquisition failures.Four sugar-box trials pass acquisition-and-lift but fail stable holding; both Coke-can failures involve robot–table contact.Source: F. Evaluation on Additional Objects · p. 8
  • All trials passing acquisition-and-lift complete the full task, but the different attachment mechanisms prevent attributing object differences to geometry alone.The sugar box and Coke can use virtual attachment; the screwdriver uses native contact.Source: F. Evaluation on Additional Objects · p. 8

VI. CONCLUSION AND FUTURE WORKS

  • The conclusion reports over 95% pooled full-task success under tested simulation perturbations, with additional objects evaluated in MuJoCo.It states that physical grasp retention is unvalidated.Source: §VI · p. 8
  • Future work targets camera calibration, online perception, sim-to-real transfer, and sustained transport under varied loads and dynamics.The conclusion bounds current results by the stated interaction models.Source: §VI · p. 8

Related papers

Loading 2610.00404v1…