Source-linked AI summary
AERIS: Offline Policy Improvement for Multi-UAV Integrated Sensing and Communication
Ziyuan Wang, Yifan Sui, Wei Wei, Wenjie Xin, Zekai Zhang, Xiangwang Hou, Xiao-Ping, Zhang
TL;DR
Dynamic multi-UAV ISAC must balance communication, sensing, and flight safety under stochastic mobility without relying on repeated global optimization or risky online trial flights. AERIS uses fixed-log centralized training with decentralized execution, while STAR-CRDT performs support-aware local corrections and trust-gated distillation. STAR-CRDT ranks first across tested system scales and scene-scale settings, including unseen OSM road maps.
Problem
Multi-UAV ISAC requires offline improvement that evaluates coupled team effects centrally while executing from decentralized local histories, avoiding residual violations and unsupported actions.
Method
AERIS formulates fixed-log multi-UAV ISAC as CTDE offline MARL and uses STAR-CRDT to distill support-aware teacher corrections selected by centralized critic feedback.
Results
STAR-CRDT ranks first at every tested UAV scale and obtains the best return in all six scene-scale settings, including unseen OSM road-map scenes.
Takeaways & Limitations
AERIS consistently improves communication, sensing, safety, and return from fixed flight logs before deployment.
Abstract
from arXiv · showhide
Unmanned aerial vehicle (UAV)-enabled integrated sensing and communication (ISAC) is a promising 6G paradigm, but dynamic multi-UAV ISAC control must jointly balance communication quality, sensing reliability, and flight safety under stochastic mobility. Existing optimization methods often require repeated global non-convex solving, while online reinforcement learning (RL) depends on risky trial-and-error flights that may cause sensing loss or collision-risk events. This paper proposes AERIS, an offline policy improvement framework for multi-UAV ISAC. AERIS learns from fixed flight logs under centralized training and decentralized execution, so each UAV acts from local histories while training uses logged global information to assess team-level effects. We further design STAR-CRDT, an offline multi-agent RL algorithm that performs support-aware local action rectification and distills only trusted improvements into the decentralized actor. We prove an offline-support policy improvement guarantee. Experiments show that STAR-CRDT improves the main ISAC objective return by 29.3% over the strongest baseline. It further improves communication sum rate, sensing pass rate, and sensing margin by 3.4%, 4.8%, and 69.1%, while reducing collision-risk events by 54.2%. On unseen real-road maps built from OpenStreetMap data, STAR-CRDT still obtains the best return.
I. INTRODUCTION
Multi-UAV ISAC control must jointly manage communication, sensing, and safety under stochastic mobility, but repeated optimization and online exploration are costly or risky. AERIS addresses this with offline policy improvement using fixed flight logs, centralized evaluation, decentralized execution, and support-aware corrections.
- Stochastic mobility couples each UAV’s movement to channels, beam directions, interference, sensing margins, and inter-UAV separation.
- Repeated global non-convex optimization creates computation and coordination overhead for distributed, fast UAV response.
- Online trial rollouts can reduce sensing reliability, degrade service, or create collision-risk events before a controller becomes useful.
- AERIS learns offline from fixed flight logs while preserving local UAV execution and using global training information to evaluate team-level effects.
- STAR-CRDT uses centralized critic scoring, support-aware candidate search, and trust-gated distillation to improve decentralized actors without unsupported updates.
- Fixed-log baselines face a tradeoff: conservative methods retain violations, whereas aggressive rectification increases joint-action OOD shift and violation cost.
C. Motivation and Design Requirements
The design requirements are to avoid repeated exploratory flights, preserve decentralized execution, and improve within reliable offline support. The ISAC system jointly evaluates communication rate, sensing feasibility and robustness, and flight safety under stochastic entity motion.
- Motivation and Design Requirements: Offline policy improvement should stop exploratory flights after an initial behavior log is available and freeze the dataset for refinement.
- Motivation and Design Requirements: Each UAV should act from local history, while training may use logged global states and joint actions to assess team-level effects.
- Motivation and Design Requirements: Reliable offline improvement requires centralized-critic-selected local corrections filtered by proximity to logged data support.
- System Model: Beamforming and movement decisions affect communication quality, sensing reliability, and safety separation over time.
- System Model: The system models M UAVs serving K users and L sensing targets under bounded random-walk mobility, with road-constrained motion during OSM deployment.
- System Model: The objective combines communication sum rate with sensing utility, including target-threshold satisfaction and normalized beampattern margin.
B. Optimization Problem
The optimization asks distributed UAVs to coordinate trajectories and beams for communication, sensing, and safety under stochastic dynamics. Its main difficulties are non-convex long-horizon control, risky online improvement, and support-sensitive decentralized learning from fixed logs.
- Optimization Problem: Each UAV jointly chooses flight locations and beamforming actions so users receive high-rate service, targets remain detectable, and UAVs stay separated.
- Optimization Problem: Distributed execution uses local histories and limited local geometry, channel, and sensing estimates rather than instantaneous global-state exchange.
- Optimization Problem: Sensing-threshold satisfaction, transmit-power limits, and inter-UAV separation are enforced as constraints, with collision risk below d_min.
- Optimization Problem: Coupled SINR, beampattern, interference, associations, and stochastic mobility make repeated centralized trajectory-and-beamforming optimization impractical.
- Optimization Problem: Online policy improvement requires trial rollouts that can consume resources, weaken sensing, reduce service, or create collision-risk events before convergence.
- Optimization Problem: Fixed-log learning must convert global team-level evidence into decentralized updates without imitating residual violations or trusting unsupported joint actions.
IV. AERIS DESIGN
AERIS converts multi-UAV ISAC control into fixed-log CTDE policy improvement: UAVs execute from local histories while training evaluates joint effects with global logged information. STAR-CRDT is trained offline to improve the shared decentralized actor without additional exploratory flights.
- AERIS design: AERIS reformulates stochastic non-convex control as sequential local action selection while preserving distributed execution and centralized team-level evaluation.The actor uses local observations, actions, and return-conditioning signals; training can use global logged states to assess joint effects.
- Offline dataset: AERIS freezes conservative-controller flight logs, so policy refinement does not require additional exploratory flights.The fixed dataset stores transitions and centralized information for critic training while retaining local histories for decentralized actor training.
- AERIS design: The reward-and-penalty formulation preserves communication, sensing, and safety meanings while making the control problem suitable for fixed-log policy learning.Positive terms represent communication spectral efficiency, sensing-threshold passes, and sensing margin; negative terms represent sensing violations and collision-risk events.
- STAR-CRDT training: STAR-CRDT trains from fixed-length local-history windows using a shared actor, twin critics, value network, local correction steps, teacher distillation, and centralized regularization.Algorithm 1 updates critics and values, constructs local rectifications, builds teacher targets, and updates the actor across epochs.
B. STAR-CRDT Offline Learning
STAR-CRDT combines a local sequence actor with centralized critic evaluation to correct logged actions conservatively. It searches supported local alternatives, gates trusted improvements, and distills them into the decentralized actor.
- Local sequence actor and centralized critic: Each UAV uses a shared return-conditioned causal Transformer actor based on local histories, while centralized twin critics and a value network evaluate global joint effects during training.Parameter sharing supports a common local rule, while CTDE supplies global communication, sensing, and safety evaluation.
- Behavior-preserving backbone: The behavior-preserving backbone anchors the actor near logged support through plain and advantage-weighted behavior cloning before correction.This provides a feasible local behavior before critic-guided updates are introduced.
- Support-aware local rectification: STAR-CRDT searches for local corrections near actor and behavior actions, then accepts them only after centralized critic evaluation in joint-action contexts.The selected correction must improve the centralized critic relative to the anchor action.
- Trust-gated distillation: Trust-gated distillation transfers only conservative portions of critic-improving candidates into the actor when gains and support proximity justify the update.The actor loss distills the teacher, while the centralized Q-regularizer rewards predicted joint actions only when they improve over logged behavior.
- Deployment: At deployment, STAR-CRDT removes critic, value, candidate-search, and trust-gate modules, leaving each UAV to act autonomously from local history.The offline training machinery is not required for runtime control.
C. Policy Improvement Proof
The proof establishes policy improvement only on the behavior-log state distribution under local ranking, regularity, and bounded distillation-error assumptions. If expected trusted rectification gain exceeds the critic-error term, the updated actor improves the centralized-critic surrogate in expectation.
- Guarantee scope: The guarantee is restricted to the offline-support distribution induced by the behavior log, rather than claiming global optimality outside the dataset.The analysis uses dβ and explicitly limits the result to supported states and actions.
- Assumptions: The proof assumes local critic ranking, concavity, regularity, and bounded approximation error for supervised distillation.These are local conditions for the trust-gated correction segment, not global claims about the non-convex wireless objective.
- Local correction: The selected local correction has nonnegative centralized-critic gain over the anchor because it maximizes critic value within the supported candidate set.This is the basis of Lemma 1 and the joint-action evaluation used by STAR-CRDT.
- Trust gate: A trust-gated partial move toward the rectified action preserves a proportional critic gain while remaining conservative.Proposition 1 explains why the teacher can improve the anchor without applying the full correction.
- Theorem 1: If expected trusted rectification gain exceeds LQε, the post-update actor has larger centralized-critic surrogate value than the anchor in expectation over dβ.The result depends on the stated local ranking, regularity, and distillation-error conditions.
A. Evaluation Setup
The evaluation uses fixed flight logs and consistent offline training conditions to compare policy-improvement methods under random mobility, system-scale variation, and road-map transfer. Metrics jointly cover return, communication, sensing, and collision risk.
- Evaluation protocols: Evaluation covers random-mobility offline improvement, zero-shot system-scale transfer, and road-map deployment without road-specific adaptation.Random mobility isolates fixed-log improvement, while road-map deployment tests mismatch between training logs and application scenarios.
- Scenario: The default setting uses M = 3 UAVs serving K = 6 users and L = 3 sensing targets, with scale studies varying M from 3 to 6.Each UAV is assigned two users and one target under quota-based nearest-UAV association.
- Data and fairness: All offline methods receive the same trajectories, rewards, local histories, centralized-training states, and decentralized observations.This keeps the comparison on the same logged support without extra online interaction or road-map adaptation.
- Baselines: The comparison includes imitation, conservative offline updates, sequence modeling, and multi-agent actor-rectification baselines under consistent datasets, budgets, capacities, and evaluation episodes.CRDT is the sequence-model ablation that removes support-aware candidate search and trust-gated distillation from STAR-CRDT.
- Metrics: Evaluation reports return, communication sum rate, sensing pass rate, sensing margin, and collision-risk count, with lower collision-risk count preferred.Effective improvement must jointly improve return, communication, sensing, and safety.
B. Random-Mobility Offline Policy Improvement
Under random mobility, STAR-CRDT most successfully improves the frozen behavior-policy log while controlling unsupported action changes. It achieves the strongest combined communication, sensing, return, and safety performance among the evaluated offline methods.
- Offline improvement: STAR-CRDT reaches the highest and most stable offline return plateau after the dataset is frozen.Imitation and conservative updates improve only part of the objective, while online TD3 remains noisy because value learning depends on trial rollouts.
- Ablation: STAR-CRDT lowers ISAC violation cost while avoiding the large joint-action OOD shift caused by aggressive rectification.CRDT stays closer to the log but leaves higher violation cost, showing that critic regularization alone is insufficient.
- Ablation: Support-aware rectification and trust-gated distillation are required alongside centralized critic feedback for reliable offline improvement.CRDT is stronger than plain DT but still trails STAR-CRDT.
- Results: 29.3% higher return over TD3+BC is achieved by STAR-CRDT, the strongest non-STAR baseline, under random mobility.It also improves communication sum rate by 3.4%, sensing pass rate by 4.8%, and sensing margin by 69.1%, while reducing collision-risk count by 54.2%.
- Baseline behavior: BC and DT remain close to the log, conservative methods limit improvement, and aggressive OMAR rectification obtains poor return.The comparison supports a tradeoff between imitation, behavior regularization, and leaving joint-action support.
C. System-Scale Evaluation
STAR-CRDT transfers its shared local actor across larger UAV teams without additional offline retraining. It remains the strongest method as joint coordination, interference, and safety coupling become harder.
- Zero-shot scale transfer: STAR-CRDT ranks first at every tested fleet scale from M = 3 to 6 UAVs without additional offline retraining.The evaluation keeps two communication users and one sensing target per UAV while increasing coordination difficulty.
- Results: At M = 6, STAR-CRDT improves return by 39.3% over TD3+BC and 60.8% over CRR while maintaining the best sensing pass rate and sensing margin.Local observation and action semantics remain fixed as the joint-action space expands.
- Interpretation: The centralized critic evaluates team-level effects before distillation, supporting local actors as interference, coupled beams, and pairwise safety constraints grow.Conservative baselines retain residual violations, while aggressive rectification becomes less stable at larger scales.
D. Zero-Shot Road-Map Deployment
Frozen policies trained on random-mobility logs are evaluated on unseen OSM road-constrained urban scenes at multiple fleet sizes. STAR-CRDT remains strongest across diverse road structures and jointly balances communication, sensing, and safety.
- Deployment protocol: Road-map deployment uses three unseen 500 m×500 m Hong Kong OSM scenes spanning dense-grid, arterial-corridor, and sparse-local-road structures.Policies receive no road-scene rollout for training, early stopping, checkpoint selection, or hyperparameter tuning.
- Evaluation layout: Table II reports metrics by row for each road scene, marking best and second-best values while treating collision risk as lower-is-better.The table evaluates zero-shot deployment across scene and fleet-size settings.
- Representative rollout: The representative rollout shows coordinated motion around road-constrained users and targets with feasible altitude control and safety separation.Shared corridors require a joint ISAC response rather than communication-only motion.
- Results: STAR-CRDT obtains the best return in all six scene-scale settings, ranking best in 24 of 30 metric rows and top-two in 29.At M = 6, return improves by 36.7% in the dense grid and 132.1% in the sparse-road scene over the strongest baseline.
- Robustness: Across differing road structures, STAR-CRDT remains strongest through support-aware corrections rather than memorizing one mobility geometry.The reported transfer includes coverage recovery, sensing-margin preservation, and safety-aware separation.
- Framework: AERIS frames trajectory-and-beamforming control as fixed-log CTDE offline MARL, combining local execution with logged global assessment of team effects.STAR-CRDT distills support-aware teacher corrections selected by centralized critic feedback.