PaperlayerView exact revision on arXiv

Source-linked AI summary

Cross-Region Cooperation of UAV Swarms Towards Air-Ground ISAC via Hierarchical Multi-Agent Reinforcement Learning

Linghui Miao, Nhan Thanh Nguyen, Shijian Gao

arXiv:2610.31054v1eess.SYcs.SY

TL;DR

Regional UAV allocations may not match vehicle demand, and local observations may lack diversity for accurate sensing. The paper coordinates cross-region reallocation and adaptive handshaking with hierarchical multi-agent reinforcement learning; under the considered simulations, reallocation improves QoS under severe mismatches and cooperative sensing achieves a mean CRB close to perfect synchronization.

  • Problem

    Regional UAV allocations may not match vehicle demand, while local sensing observations may lack diversity.

    Source: §I · p. 2

  • Method

    A hierarchical multi-agent reinforcement learning approach coordinates slow-timescale regional reallocation with fast-timescale local control, including mobility, transmit power, and handshaking.

    Source: §I · p. 2

  • Results

    Under severe resource-demand mismatches, reallocation improves QoS by over 20 percentage points, and cooperative sensing achieves a mean CRB of 0.0237 m2, close to perfect synchronization.

    Source: §VII · p. 13

  • Takeaways & Limitations

    Under the considered simulation settings, cross-region reallocation improves communication service and adaptive handshaking supports cooperative sensing across regions.

    Source: §VII · p. 13

Abstract

from arXiv · show

The limited communication resources and sensing perspectives of a single uncrewed aerial vehicle (UAV) motivate UAV swarms for integrated sensing and communication (ISAC) in air-ground networks. However, in multi-region scenarios, regional UAV allocations may not match vehicle demand, while local observations may lack sufficient diversity for accurate sensing. To address these issues, this paper proposes a cross-region cooperation framework for UAV swarms. For communication, cross-region UAV reallocation reduces service disparities by matching resources to vehicle demand. For sensing, combining UAV observations from different regions increases observation diversity, while adaptive handshaking guided by marginal Cram\'er--Rao bound (CRB) utilities balances phase synchronization gains against handshaking energy consumption. Joint optimization of UAV positions and transmit power within each region then further improves communication service and sensing performance. These coupled decisions are coordinated by a hierarchical multi-agent reinforcement learning (H-MARL) approach through slow-timescale inter-region UAV reallocation and fast-timescale intra-region UAV control. Under the considered simulation settings, reallocation can improve quality of service (QoS) by over 20 percentage points compared with retaining the initial allocation under severe resource-demand mismatches. Cooperative sensing with adaptive handshaking incurs only about a 5.0% increase in mean CRB relative to perfect synchronization.

I. INTRODUCTION

The paper frames cross-region UAV cooperation as a way to address regional resource-demand mismatches and limited sensing diversity, coordinating regional reallocation with local control. Its system model combines vehicle service and cooperative target sensing across neighboring regions on two decision timescales.

I. INTRODUCTION

  • Cross-region cooperation addresses resource-demand mismatches and limited local sensing diversity, while balancing communication gains against source-region losses, sensing degradation, and transit energy.Adaptive handshaking uses regional marginal CRB utilities; H-MARL coordinates slow-timescale reallocation with fast-timescale mobility, power, and handshaking control.Source: §I · p. 2

A. Air-Ground Network

  • The system serves ground vehicles and cooperatively senses an aerial target in hotspot-centered Voronoi regions, with interactions limited to a region and its neighbors.Reallocation occurs every K fast-timescale slots; UAV mobility, transmit power, and handshaking preferences are adjusted each slot.Source: A. Air-Ground Network · p. 2

B. ISAC Implementation with UAV Swarms

  • Communication uses UAV swarm resources for regional vehicle service, while cross-region reallocation matches those resources to vehicle demand and reduces service disparities.Bandwidth reuse can cause intra-region and inter-region interference, while each UAV divides bandwidth and transmit power among served vehicles.Source: B. ISAC Implementation with UAV Swarms · p. 3
  • For cooperative sensing, UAVs combine observations across regions using mutually orthogonal waveforms for coherent processing and target localization.Inter-region handshaking reduces residual timing, frequency, and phase offsets; its frequency balances synchronization accuracy against energy consumption.Source: B. ISAC Implementation with UAV Swarms · p. 3

III. CHARACTERIZATION OF UTILITY FUNCTIONS FOR UAV-SWARM-ENABLED ISAC

  • The paper characterizes communication, sensing, and energy utilities for UAV-swarm-enabled ISAC.The utility framework covers propulsion, ISAC transmission, and inter-region handshaking energy.Source: §III · p. 3

A. Cross-Region Communication Service Utility

The communication utility models vehicle service under shared bandwidth and interference. The sensing utility models cross-region observations, residual synchronization errors, and handshaking choices that trade localization accuracy against energy use.

A. Cross-Region Communication Service Utility

  • Vehicles are associated with UAVs using link quality and bandwidth sharing, and are served only when their achievable rates meet the minimum requirement.UAVs reuse bandwidth across regions, producing interference, and allocate bandwidth and transmit power among served vehicles.Source: A. Cross-Region Communication Service… · pp. 3–4

B. Cross-Region Cooperative Sensing Utility

  • Coherent distributed sensing combines cross-region observations, while residual inter-region phase offsets are modeled relative to a reference region under synchronization assumptions.More frequent handshaking improves sensing accuracy but costs more energy; regional marginal CRB utility measures a region's sensing contribution by recomputing localization CRB without that region's UAVs.Source: B. Cross-Region Cooperative Sensing… · pp. 4–5
  • The residual phase-error variance depends on handshaking frequency, which sets the interval over which clock and oscillator drift accumulate between synchronization updates.The utility combines sensing utility with a weighted handshaking-energy cost.Source: B. Cross-Region Cooperative Sensing… · pp. 4–5

C. UAV-Swarm Energy Utility

The energy utility accounts for propulsion, unified ISAC transmission, and inter-region handshaking. Handshaking energy is explicitly tied to reducing synchronization uncertainty.

C. UAV-Swarm Energy Utility

  • The energy model includes propulsion, unified ISAC transmission, and inter-region handshaking costs.Handshaking is the energy component that explicitly accounts for reducing synchronization uncertainty.Source: C. UAV-Swarm Energy Utility · p. 5
  • Propulsion energy is calculated from steady-flight power over each slot, with vertical motion adjusting power for ascent or descent.The vertical-motion adjustment follows different expressions depending on the sign of vertical velocity.Source: C. UAV-Swarm Energy Utility · p. 5
  • Each UAV’s unified ISAC signal is counted once for transmission energy, while handshaking energy is modeled per participating UAV update.The handshaking model includes a per-update energy cost.Source: C. UAV-Swarm Energy Utility · p. 5

IV. PROBLEM FORMULATION AND TWO-TIMESCALE DECOMPOSITION

The section formulates cross-region cooperation as a joint optimization problem and presents coupled subproblems for slow-timescale UAV reallocation and fast-timescale UAV control.

IV. PROBLEM FORMULATION AND TWO-TIMESCALE DECOMPOSITION

  • The section formulates a joint optimization problem for cross-region cooperation.It then presents two coupled subproblems: slow-timescale UAV reallocation and fast-timescale UAV control.Source: §IV · p. 5
  • The optimization is decomposed into slow-timescale UAV reallocation and fast-timescale UAV control.These are presented as coupled subproblems.Source: §IV · p. 5
  • The formulation first addresses cross-region cooperation, then presents the two-timescale subproblems.Source: §IV · p. 5

A. Joint Optimization of Cross-Region UAV Cooperation

The section formulates joint cross-region UAV cooperation and slow-timescale reallocation. It combines dispatch decisions with intra-region control and accounts for communication, sensing, and energy considerations.

A. Joint Optimization of Cross-Region UAV Cooperation

  • The dispatch matrix and selected-UAV vector jointly determine which available UAVs move between neighboring regions in each reallocation round.A zero dispatch-matrix row indicates no dispatch; the resulting allocation supports subsequent mobility, power, and handshaking decisions.Source: A. Joint Optimization of Cross-Region… · p. 5
  • A neighboring UAV guarantees a positive CRB reduction over local-only sensing, with the bound shaped by its ranging information and observation direction.The result motivates adapting sensing geometry and handshaking frequency while accounting for communication requirements.Source: A. Joint Optimization of Cross-Region… · pp. 5–6
  • The joint problem separates round-level reallocation from slot-level velocity, transmit-power, and handshaking-preference updates.The slow problem prioritizes regional QoS and fairness subject to a sensing-loss tolerance; the fast problem jointly optimizes communication and sensing under the resulting allocation.Source: A. Joint Optimization of Cross-Region… · p. 6

B. Slow-Timescale Cross-Region Reallocation Problem

  • Reallocation candidates are screened by tolerated sensing loss, and dispatch choices account for communication losses and transit energy.The slow problem weights global QoS, regional fairness, and dispatch energy; it also restricts dispatch between neighboring regions and limits regional roles and UAV removals.Source: B. Slow-Timescale Cross-Region… · p. 6
  • Round-level QoS and fairness are averaged over the final W slots before the next reallocation decision.The averaging window is constrained by 1 ≤ W ≤ K.Source: B. Slow-Timescale Cross-Region… · p. 6

C. Fast-Timescale Intra-Region Control Problem

Under the allocation set by reallocation, each region controls UAV operation to improve local communication QoS while reducing global CRB and energy consumption.

C. Fast-Timescale Intra-Region Control Problem

V. CROSS-REGION COOPERATION OF UAV SWARMS VIA HIERARCHICAL MARL

The H-MARL framework coordinates cross-region UAV reallocation and local UAV control across two timescales. Stagewise training fixes the fast-timescale policy while the regional reallocation policy is learned, keeping its evaluation environment consistent.

A. Motivation for Hierarchical MARL

  • A centralized single-agent policy faces a joint action space that grows with the number of regions and UAVs, motivating hierarchical multi-agent decisions.Regional and UAV decisions interact through resource redistribution, communication interference, and cooperative sensing.Source: A. Motivation for Hierarchical MARL · p. 7

V. CROSS-REGION COOPERATION OF UAV SWARMS VIA HIERARCHICAL MARL

  • Regional and UAV agents are organized hierarchically to implement slow-timescale reallocation and fast-timescale local control.The framework uses graph-based IPPO for reallocation and MAPPO for intra-region control.Source: A. Motivation for Hierarchical MARL · p. 7, §V · p. 7

A. Motivation for Hierarchical MARL

  • Stagewise training pretrains MAPPO and holds it fixed during graph-based IPPO training, making the slow-timescale transition and reward mappings independent of policy-update index.This avoids changes in the reallocation policy's expected outcomes caused by simultaneous evolution of the control policy.Source: A. Motivation for Hierarchical MARL · p. 7

B. Slow-Timescale Inter-Region Reallocation

Slow-timescale reallocation is modeled as a regional-graph POMDP in which regions select feasible UAV dispatches or stay in place. Its reward is evaluated after the subsequent fast-timescale control segment against a paired no-dispatch reference.

B. Slow-Timescale Inter-Region Reallocation

  • Each region acts as an agent in a graph-based POMDP whose neighboring-region links represent feasible inter-region connections.The graph includes regional node inputs and edge features; agents observe their own region and one-hop neighbors.Source: B. Slow-Timescale Inter-Region… · p. 8
  • Feasibility masks exclude ineligible UAVs and conflicting destinations, enforcing self-stay when no dispatch is feasible.Accepted regional actions form the dispatch matrix and UAV-selection vector.Source: B. Slow-Timescale Inter-Region… · p. 8
  • Reallocation reward is evaluated after the associated fast-timescale control segment and compared with a paired rollout that keeps the initial allocation.The reference shares the policy rollout's exogenous randomness; the shared regional reward accounts for collective outcomes and dispatch energy.Source: B. Slow-Timescale Inter-Region… · p. 8

C. Fast-Timescale Intra-Region Control

At the fast timescale, each region models its UAVs as cooperative agents in a POMDP under centralized training with decentralized execution. Their continuous actions control motion and transmit power, while regional handshaking preferences are aggregated using marginal sensing utilities.

C. Fast-Timescale Intra-Region Control

  • Given the slow-timescale allocation, UAVs act as cooperative agents for local vehicle service and, when required, cross-region cooperative sensing.Each region's decision process uses centralized training with decentralized execution.Source: C. Fast-Timescale Intra-Region Control · p. 8
  • Each UAV's continuous action controls its velocity and transmit power and supplies a local handshaking preference for aggregation.Target- and handshaking-related observation entries are masked in regions not participating in cooperative sensing.Source: C. Fast-Timescale Intra-Region Control · p. 9
  • Handshaking preferences use previous-slot marginal sensing utilities because current sensing feedback is available only after execution.Regions with larger marginal sensing contributions receive greater weight, and sensing utilities are reevaluated after execution for the next slot.Source: C. Fast-Timescale Intra-Region Control · p. 9
  • The regional reward combines communication, sensing, and energy terms with soft penalties for boundary and inter-UAV safety-distance violations.The sensing term is active only when the region participates in cooperative sensing.Source: C. Fast-Timescale Intra-Region Control · p. 9

D. Hierarchical Policy Optimization

The hierarchical policies are trained in stages, with fast-timescale experience collected around feasible reallocations and the pretrained control policy fixed during regional-policy updates. Online execution alternates round-level dispatch with slot-level UAV actions and handshaking aggregation.

D. Hierarchical Policy Optimization

  • Each fast-timescale pretraining episode includes one feasible cross-region reallocation, with experience collected before and after it to help MAPPO adapt to allocation changes.Fig. 5 summarizes the two-timescale interaction and stagewise training.Source: D. Hierarchical Policy Optimization · p. 9
  • Fast-timescale MAPPO uses temporal-difference errors and generalized advantage estimation, while graph-based IPPO updates regional actors and local critics with the pretrained fast policy fixed.The regional PPO actor uses the joint action probability, and the local critic fits normalized GAE return targets using Huber loss.Source: D. Hierarchical Policy Optimization · p. 9
  • The stated computational bounds assume U ≥ M ≥ 1 and fixed network depths, attention-head counts, and map resolution.The cost accounting includes communication and CRB evaluations, candidate-removal evaluations, and energy accounting.Source: D. Hierarchical Policy Optimization · pp. 9–10
  • Online execution performs regional reallocation at round boundaries, then selects UAV actions and aggregates handshaking preferences at each slot.The algorithm updates the system state and communication, sensing, and energy feedback after execution.Source: D. Hierarchical Policy Optimization · p. 10

VI. SIMULATIONS

The simulations evaluate graph-based reallocation, intra-region UAV control, cooperative sensing with adaptive handshaking, and reward-weight effects. They use a five-region, ten-UAV air-ground network and compare the proposed methods with learning-based and heuristic baselines.

A. Simulation Setup

  • The evaluation uses ten UAVs in a 1000 m × 1000 m area partitioned into five Voronoi regions, with 200 vehicles initially distributed unevenly across them.Vehicles start around regional hotspots and subsequently follow random walks; training includes fast-timescale control and five slow-timescale reallocation stages.Source: A. Simulation Setup · p. 10
  • Slow-timescale baselines include IPPO, IA2C, demand-based reallocation, random decisions, and All-Stay, which retains the initial allocation.IPPO removes graph-based representation; IA2C changes the update method, while the heuristic baselines use demand or random selection.Source: A. Simulation Setup · p. 10
  • Fast-timescale baselines include MADDPG, Reward-based candidate-action selection, and random control.Random control samples velocity, transmit power, and handshaking preferences uniformly within their bounds.Source: A. Simulation Setup · p. 10

B. Graph-Based Representation Evaluation

Across the considered simulations, hierarchical control coordinates regional UAV reallocation with local UAV operation, while adaptive handshaking supports cooperative sensing. The reported outcomes include improved QoS and fairness, and cooperative-sensing CRB close to the perfect-synchronization reference.

B. Graph-Based Representation Evaluation

C. Regional UAV Reallocation Assessment

D. Intra-Region UAV Control Evaluation

E. Adaptive Handshaking and Cooperative Sensing

F. Effects of Reward Weights

  • At every tested weight setting, the proposed slow-timescale method achieves higher QoS and fairness than IPPO and IA2C; fast-timescale QoS exceeds both baselines at every tested setting.After QoS plateaus as communication weight increases, further increases yield little additional QoS and markedly increase CRB.Source: F. Effects of Reward Weights · p. 12, Table V · p. 12, Table IV · p. 12

VII. CONCLUSIONS

  • Over 20 percentage points of QoS improvement is reported for severe resource-demand mismatches; the reallocation policy reaches regional fairness 0.98, and adaptive-handshaking mean CRB is 0.0237 m2.The conclusion reports these outcomes under the considered simulation settings.Source: §VII · pp. 12–13

Related papers

Loading 2610.31054v1…