PaperlayerView exact revision on arXiv

Source-linked AI summary

Curriculum-Based Adversarial Heterogeneous Agent Reinforcement Learning for Autonomous Quad-Copter Landing in Maritime Settings

Allan Minh-Tam Nguyen, Sree Showrya Kotala, Stefan Banioi-Crijman, Kurt Driessens, Rico Möckel

arXiv:2609.12758v1cs.LG

TL;DR

Conventional UAV landing can be unreliable amid strong maritime wind and ship-deck motion, motivating simulated mid-air capture by a robotic arm. The paper trains cooperative heterogeneous-agent policies with HAPPO, a curriculum, and an adversarial wind agent, then compares them with domain-randomized policies. HARL-AC shows better OOD performance than the compared baselines, with a cautious behavior trade-off.

  • Problem

    Conventional landing can risk equipment damage on small ships or in stronger winds, motivating an alternative for recovering UAVs in maritime settings.

    Source: §1 · p. 1

  • Method

    The paper trains cooperative UAV–arm policies with HAPPO, a curriculum, and a zero-sum wind adversary, and compares them with curriculum-based domain randomization.

    Source: §1 · p. 2

  • Results

    Up to 16% higher success rates and 14% lower crash rates were reported for HARL-AC than the DR and Baseline SS4 models in OOD tests.

    Source: §6 · p. 15

  • Takeaways & Limitations

    The adversarially trained policy develops more cautious landing behavior and generalizes better to unseen, higher sea states, with slightly longer average landing times.

    Source: §6 · p. 15

Abstract

from arXiv · show

Recovering unmanned aerial vehicles (UAVs) in maritime environments is challenging due to wind turbulence and ship-deck motion, making it a valuable test case for alternative control and learning approaches as conventional landing approaches often become unreliable. We study simulated mid-air capture of quadrotor UAVs by a ship-mounted robotic arm, learning robust cooperative control policies with Heterogeneous-Agent Proximal Policy Optimization (HAPPO) Reinforcement Learning. We train with HAPPO using a curriculum and an adversarial wind agent (HARL-AC) in NVIDIA Isaac Lab, and compare the obtained control policies against those generated through curriculum-based domain randomization and a benchmark trained on a single sea state. In-distribution evaluation on sea states $0/4/5$ shows comparable success for HARL-AC and domain randomization of up to $97.5\%$. On out-of-distribution sea states $7/8/10$, HARL-AC generalizes better, achieving up to $16\%$ higher median success rate at sea state 10, and substantially lower crash rates of up to $14\%$ compared to the domain randomization policy. Furthermore, we show that the adversarially trained policy shows more cautious behavior, slightly increasing timeouts by $<3\%$, but yields safer recovery behavior in severe, unseen conditions.

1 Introduction

Conventional UAV landing can become unreliable on small, moving ship decks in strong winds, motivating mid-air capture by a robotic arm. The paper compares curriculum-based domain randomization with adversarial heterogeneous-agent reinforcement learning for this cooperative task.

1 Introduction

  • Robotic-arm capture avoids a landing platform and can compensate for small boat movements, but requires coordinated control of the UAV and arm.Conventional landing can risk equipment damage on small ships or in stronger winds.Source: §1 · pp. 1–2
  • The study compares curriculum-based domain randomization with curriculum-based adversarial heterogeneous-agent reinforcement learning under stochastic wind, gusts, and ship-deck motion.The setting tests the two controllers in a maritime environment with disturbances from prevailing wind, gusts, and reactive deck movement.Source: §1 · p. 2
  • Mid-air capture is explored as an alternative to relying on strong, responsive UAV thrust, building on prior demonstrations of mechanical capture in controlled environments.The paper investigates autonomous control in less predictable maritime conditions; Huang et al. instead optimize a drone's response to aerodynamic effects.Source: §1 · p. 2
  • The paper adopts HAPPO for the heterogeneous UAV and robot-arm agents because independent policies avoid the value-collapse risk associated with shared-policy MAPPO in heterogeneous settings.HAPPO provides monotonic improvement guarantees and evolves independent policies.Source: §1 · p. 2
  • The HARL-AC approach uses a zero-sum wind adversary within a dynamic curriculum to expose the cooperative team to disturbances during capture.The adversary injects wind-gust and corresponding wave disturbances for sea states up to 5.Source: §1 · p. 2

2 Related work

Prior work motivates combining heterogeneous multi-agent learning, adversarial training, and curriculum progression, but the cited examples address different tasks and settings. This work applies that combination to cooperative heterogeneous agents facing an adversary.

2 Related work

  • HAPPO uses sequential per-agent updates without shared network weights, providing monotonic-improvement guarantees for individual agents in heterogeneous systems.By contrast, MAPPO's parameter sharing can flatten policies for structurally distinct agents and contribute to instability.Source: §2 · p. 3
  • HARL-A combines team-separated critics with HAPPO's sequential updates to address shared-critic credit-assignment failure in multi-agent adversarial learning [12].The cited work describes the shared critic's advantage estimate collapsing toward the average, degrading PPO updates.Source: §2 · p. 3
  • Curriculum progression and individual pre-training are presented as ways to scale adversarial intensity, alongside prior curriculum studies in drone swarm search and UAV-assisted target tracking [17].Bhati et al. [3] report that a medium-skilled pre-trained teammate balances team reward and individual learning better than a single pre-trained teammate.Source: §2 · p. 3
  • Prior curriculum-plus-adversarial studies include heterogeneous adversarial play using a student-teacher framework and adversarial-cooperative HARL for a throwing-and-catching task with two robot arms [9].The arm policies in the throwing-and-catching task are distinct despite the arms' homogeneous physical structures.Source: §2 · pp. 3–4
  • The paper develops robust cooperative control by combining curricula and adversarial multi-agent reinforcement learning for heterogeneous agents competing against an adversary.The application is a cooperative team of structurally distinct agents.Source: §2 · p. 4

3 Methods

The methods combine a staged curriculum with adversarial training for cooperative UAV–arm capture in a simulated maritime environment. HAPPO-based policies coordinate the cooperative agents against a wind adversary whose disturbances are coupled to turbulence and, in later stages, ocean motion.

3 Methods

  • The training flow combines adversarial training with curriculum components that introduce increasing complexity across simulation stages.The curriculum stages progressively add environmental and adversarial challenges.Source: Figure 1 · p. 4, §3 · p. 4

3.1 Multi-agent formulation

  • The three-agent task pairs a UR10 arm and Vertex One quadcopter as a cooperative team against a wind adversary, with capture defined by staying within 0.5 m of the end effector for 3 seconds below 3 m/s.The UAV, arm, and wind adversary have distinct observation and action spaces; HAPPO updates their policies sequentially and uses separate critics for the cooperative team and adversary.Source: §3.1 · p. 5

3.2 Adversarial model

  • The adversary's bounded wind action is smoothed and combined with stochastic Dryden turbulence, while later curriculum stages couple wind magnitude to sea state and platform motion.The sea-state process parameterizes a Pierson–Moskowitz wave spectrum and platform motion across five degrees of freedom; Table 1 describes stage difficulty, wind disturbance, workspace limits, and ocean effects.Source: Dryden turbulence filter · p. 6, Wind-Wave Coupling · p. 6

3.3 Reward structure

  • Team A receives rewards for cooperation, while Team B's reward combines a zero-sum component against Team A with adversary-specific shaping terms.The active reward terms are listed in Appendix D.Source: §3.3 · p. 7

3.4 Curriculum learning

  • A four-stage curriculum begins with drone pre-training while the arm is frozen upright, then introduces adversarial pressure from Stage 2.Stage progression uses an exponential moving average of success rate, with conservative advancement and rapid retreat to manage policy stability.Source: §3.4 · pp. 7–8

3.5 Bidirectional adversarial curriculum

  • A bidirectional curriculum adjusts Dryden turbulence intensity around a target adversary win rate of 30%, keeping disturbances within stage-specific bounds.Turbulence intensity changes multiplicatively according to whether the adversary's win rate falls below or rises above the target tolerance band.Source: §3.5 · p. 8

4 Training and Evaluation

The evaluation includes a stochastic-wind ablation, repeated training runs, and zero-shot tests at higher sea states. The OOD tests assess trained policies in unseen conditions without online adaptation.

4.1 Ablation Study: Stochastic Wind with Curriculum (Pure Dryden)

  • The pure-Dryden stochastic-wind baseline removes the adversarial wind agent and uses white noise filtered through a Dryden turbulence model, with σ set to 4 in Stage 2 and 7 in Stage 3.The wind energy is split into slow rolling thermals and fast micro-turbulence, with Gaussian noise driving both processes to test generalized disturbance rejection.Source: §4.1 · pp. 8–9

4.2 Training Instances

  • The ablation and HARL-AC approaches each use 10 random initializations, with each run lasting 250 million environment steps across 254 instances.Runs took about 6.5 hours, and performance was assessed by curriculum progression, success, crash, and timeout rates.Source: §4.2 · p. 9

4.3 Evaluation Environment and Out-of-Distribution Testing

  • For zero-shot OOD evaluation, trained policies were tested at sea states 7, 8, and 10, with 20 trials of 10,000 episodes each and no online adaptation.The higher sea states are represented by higher σ values to test spectral resilience.Source: §4.3 · p. 9

5 Results and Discussion

Across unseen higher sea states, HARL-AC is more robust than domain randomization (DR); this gain accompanies slightly higher timeout rates. In trained sea states, their success rates are comparable, while HARL-AC reaches the final curriculum stage sooner.

5.1 Training results

  • HARL-AC transitioned from stage 0 to 1, 1 to 2, and 2 to 3 at 37, 93, and 108 million steps, versus DR at 38, 94, and 114 million steps.The metrics were averaged over 10 Monte Carlo trials, with curriculum-stage transitions marked in Fig. 2.Source: §5.1 · p. 10, Figure 2 · p. 11
  • In stage 3, HARL-AC settled at 90.8% success, below DR's approximately 94.3%, after crash spikes coincided with success dips at 138 and 159 million steps.Success subsequently recovered; the passage attributes the challenge to the wind adversary learning ways to disrupt the cooperative team.Source: §5.1 · p. 10, Figure 2 · p. 11

5.2 Out-of-distribution evaluation

  • At sea states 0/4/5, HARL-AC versus DR success rates were 97.5% vs 95.4%, 96.3% vs 95.9%, and 94.0% vs 94.3%.HARL-AC results showed slightly greater variability; the comparison excludes a DR outlier noted in Fig. 3a.Source: §5.2 · p. 12, Figure 3 · p. 13
  • At sea states 7/8/10, HARL-AC median success was 83.2%, 79.3%, and 60%, versus DR's 82.6%, 65.4%, and 45.9%; crash rates differed by 14% at states 8 and 10.The evaluation also included Baseline SS4 across trained and out-of-distribution sea states.Source: §5.2 · pp. 12, 14, Figure 3 · p. 13
  • Timeout rates stayed under 3%, but HARL-AC had slightly higher medians as its cautious navigation sometimes exceeded time limits, trading marginal timeouts for fewer out-of-distribution crashes.The authors describe this as a few-seconds-longer landing behavior in unseen sea states.Source: §5.2 · p. 14
  • DR behaved more reactionarily in unseen wind changes, whereas HARL-AC prioritized stability over speed and showed greater resistance to unpredictable wind.The authors connect DR's overcommitment to higher relative crash rates and describe HARL-AC's cautious behavior as a crash-prevention trade-off.Source: §5.2 · pp. 14–15

6 Conclusion

HARL-AC is reported as more robust than domain randomization, particularly in out-of-distribution conditions. Its cautious landing policy trades slightly longer landing times for better generalization, while adversarial pressure supports continued learning of recovery strategies.

6 Conclusion

  • Adversarial pressure was associated with continued learning of more generalized recovery strategies despite marginally faster overall training.The authors hypothesize that broader search-space exposure during training contributes to robustness; they also note that stage-3 success-rate dips and returns correspond to rises and falls in crash rates.Source: §6 · p. 15
  • HARL-AC outperformed both domain randomization and the SS4 baseline in out-of-distribution tests, with differences of up to 16% in success rates and 14% in crash rate.The authors report that adversarially trained agents develop a more cautious landing policy and generalize better to unseen conditions and higher sea states.Source: §6 · p. 15
  • The cautious policy slightly increases average landing times, but the authors say this does not meaningfully inhibit practical deployability.The authors qualify possible physical-world robustness as anticipated if simulated benefits extend to real environments.Source: §6 · p. 15

A Adversarial wind pseudocode

The adversarial wind update takes a time step and curriculum stage and outputs an updated wind velocity. It disables wind in early stages, then smooths adversarial wind, adds Dryden turbulence, and returns the resulting wind.

A Adversarial wind pseudocode

B HAPPO training hyperparameters

The supplied passage identifies Table 3 as presenting HAPPO training hyperparameters.

B HAPPO training hyperparameters

  • Table 3 is identified as the HAPPO training-hyperparameter table.Source: Table 3 · p. 19

C Active reward terms

The supplied passage identifies Table 4 as presenting active reward terms.

C Active reward terms

  • Table 4 is identified as the table of active reward terms.Source: Table 4 · p. 20
Loading 2609.12758v1…