PaperlayerView exact revision on arXiv

Source-linked AI summary

Privacy-Preserving Federated Learning for UAV-Enabled Networks: Learning-Based Joint Scheduling and Resource Management

Helin Yang, Jun Zhao, Zehui Xiong, Kwok-Yan Lam, Sumei Sun, Liang Xiao

arXiv:2011.14197v1eess.SPcs.AI

TL;DR

UAV-enabled federated learning must protect device privacy while coping with limited UAV resources, dynamic channels, and heterogeneous device capabilities. The paper proposes asynchronous federated learning with device selection and an A3C-based joint optimization of device selection, UAV placement, and resources, reporting higher accuracy and faster execution than existing solutions.

  • Problem

    UAV federated learning must avoid transmitting raw device data while addressing limited UAV resources, dynamic channels, and heterogeneous device capabilities.

  • Method

    The paper develops a privacy-preserving AFL framework with device selection and an A3C-based joint device-selection, UAV-placement, and resource-management algorithm.

  • Results

    The proposed AFL framework and A3C-based algorithm outperform existing solutions in learning accuracy and federated execution time.

  • Takeaways & Limitations

    Asynchronous local training and selective participation support federated learning in multi-UAV networks without uploading raw private data.

Abstract

from arXiv · show

Unmanned aerial vehicles (UAVs) are capable of serving as flying base stations (BSs) for supporting data collection, artificial intelligence (AI) model training, and wireless communications. However, due to the privacy concerns of devices and limited computation or communication resource of UAVs, it is impractical to send raw data of devices to UAV servers for model training. Moreover, due to the dynamic channel condition and heterogeneous computing capacity of devices in UAV-enabled networks, the reliability and efficiency of data sharing require to be further improved. In this paper, we develop an asynchronous federated learning (AFL) framework for multi-UAV-enabled networks, which can provide asynchronous distributed computing by enabling model training locally without transmitting raw sensitive data to UAV servers. The device selection strategy is also introduced into the AFL framework to keep the low-quality devices from affecting the learning efficiency and accuracy. Moreover, we propose an asynchronous advantage actor-critic (A3C) based joint device selection, UAVs placement, and resource management algorithm to enhance the federated convergence speed and accuracy. Simulation results demonstrate that our proposed framework and algorithm achieve higher learning accuracy and faster federated execution time compared to other existing solutions.

I. INTRODUCTION

UAV-enabled networks support wireless communications, data collection, and AI training, but federated learning faces privacy, resource, latency, and dynamic-environment challenges. Existing work leaves a need for joint scheduling and resource management that accounts for learning accuracy and execution time.

  • UAVs provide flexible aerial BSs for wireless communications, data collection, AI model training, and coverage enhancement.
  • Large-scale UAV federated learning is constrained by device privacy concerns, limited UAV computation and communication resources, transmission latency, and dynamic channel conditions.
  • Federated learning keeps raw data at devices while exchanging local model parameters for UAV-server aggregation, preserving privacy and reducing traffic congestion.
  • Device selection can address limited wireless spectrum by allowing only a subset of devices to participate in aggregation, particularly those with stronger communication and computation capabilities.
  • Prior studies generally did not jointly consider FL learning accuracy, execution time, dynamic scheduling, and UAV resource management under device and UAV resource constraints.
  • The paper identifies a gap in applying reinforcement learning to UAV-enabled FL while jointly designing scheduling and resource management for convergence or accuracy.

B. Contributions and Organization

The paper develops privacy-preserving asynchronous federated learning for multi-UAV networks and an A3C-based joint optimization algorithm. The framework trains locally and asynchronously, while the algorithm coordinates device selection, UAV placement, and resource management.

  • Results: Simulations report improved model accuracy, convergence speed, and federated execution efficiency compared with baseline solutions under different settings.
  • Contributions: The AFL framework enables devices to train locally and asynchronously upload model parameters for UAV-server aggregation without uploading raw private data.
  • Contributions: The framework selects mobile devices with high communication and computation capabilities instead of waiting for all associated devices to complete local updates.
  • Contributions: The proposed A3C-based algorithm jointly manages device selection, UAV placement, and resource management under dynamic environments and large-scale continuous action spaces.
  • System model: The multi-UAV network uses local datasets and model-parameter exchange: devices train locally, upload parameters, and UAV servers aggregate and broadcast global parameters.
  • System model: The system models communication and computation resources, including CPU capability, learning time, learning accuracy, transmission rate, latency, and time-varying capabilities.

A. FL Model

The FL model defines local and global learning objectives for devices associated with UAV servers, while the communication model captures probabilistic air-to-ground links, interference, and uplink and downlink rates.

  • A. FL Model: Each device trains a local model on its private dataset, while each UAV server forms a global model from selected devices’ local model parameters.
  • A. FL Model: The device loss function evaluates prediction error on local samples, and the UAV-server global loss aggregates distributed losses from selected devices.
  • A. FL Model: The global objective searches for UAV-server model parameters that minimize the average global loss without directly accessing selected devices’ raw datasets.
  • B. Communication Model: The channel model uses probabilistic LoS and NLoS path loss based on UAV-device distance, elevation angle, propagation environment, carrier frequency, and additional losses.
  • B. Communication Model: Uplink communication uses OFDMA with M orthogonal subchannels reused across cells, producing inter-cell interference and device-specific SINR and data rates.
  • B. Communication Model: Downlink UAV servers broadcast global parameters over assigned channels, with overlapping cells also experiencing inter-cell interference.

C. FL Model Update Latency Analysis

The section decomposes one-round FL latency into local computation, model upload, aggregation, and broadcast components. These costs depend on device and UAV computation capabilities, model sizes, selected devices, and communication resources.

  • Computation latency: Device and UAV computation capabilities are represented by CPU cycles per second and determine local computation and global aggregation latency.The model distinguishes device capability f_k from UAV-server capability f_n, including their minimum and maximum ranges.
  • Computation latency: Local model computation latency depends on the training workload and device CPU capability, while UAV aggregation latency depends on the corresponding server capability.The passage set defines separate latency expressions for device k and UAV server n.
  • Communication latency: Broadcast latency is determined for UAV servers sending global model parameters to associated devices.L_n denotes the number of bits required for each broadcast.
  • Communication latency: Upload latency is modeled for each device sending its local model parameters to its associated UAV server.L_k denotes the number of bits uploaded by device k.
  • Latency components: One-round FL scheduling time combines local model update, uplink upload, global aggregation, and downlink broadcast latencies.The total cost is defined for scheduling each device’s FL model in one round.

D. Problem Formulation

The paper formulates FL scheduling in multi-UAV networks as minimizing a weighted combination of execution time and learning-accuracy loss. The resulting joint design selects devices, positions UAVs, and allocates communication resources, then addresses the problem with asynchronous A3C-AFL.

  • D. Problem Formulation: Heterogeneous device computation capacities and time-varying communication channels complicate efficient FL implementation in UAV-enabled networks.These factors affect both computation and global-model communication.
  • D. Problem Formulation: The optimization jointly selects devices, places UAVs, allocates uplink subchannels, and manages UAV transmit power.The objective is to improve communication rates and reduce FL execution time and learning-accuracy loss.
  • D. Problem Formulation: The objective minimizes a weighted sum of one-round FL execution time and learning-accuracy loss, with λ balancing the two costs.The loss is measured at the end of each time slot.
  • D. Problem Formulation: Constraints limit device association, available subchannels, UAV transmit power, and device computation capability ranges.Each device can associate with only one UAV server, while downlink broadcasting uses a given band and device transmit power is assumed fixed.
  • III. ASYNCHRONOUS FEDERATED REINFORCEMENT LEARNING SOLUTION: The optimization is non-convex and NP-hard, while mobility and heterogeneous capacities make synchronized communication difficult.Synchronous aggregation also requires each UAV server to wait for all selected devices, increasing global learning delay.
  • III. ASYNCHRONOUS FEDERATED REINFORCEMENT LEARNING SOLUTION: A3C-AFL addresses these challenges with asynchronous advantage actor-critic learning for joint device selection, UAV placement, and resource management.The method is designed for continuous action components such as UAV horizontal locations and transmission power.
  • III. ASYNCHRONOUS FEDERATED REINFORCEMENT LEARNING SOLUTION: The framework has three phases: device selection, UAV placement and resource management; local training; and global aggregation.Selected devices train on local datasets, upload local parameters, and receive updated global parameters after aggregation.
  • III. ASYNCHRONOUS FEDERATED REINFORCEMENT LEARNING SOLUTION: Each UAV server selects higher-capability devices, applies A3C-based placement and resource management, and aggregates uploaded models by averaging.The update process iterates between local and global model updates.

A. Modeling of Reinforcement Learning Environment

The paper models joint UAV placement, device selection, subchannel allocation, and power allocation as a multi-agent Markov decision process. Its reward balances FL execution time against learning-accuracy loss.

  • MDP formulation: The combinatorial optimization problem is formulated as a Markov decision process with state space, action space, transition probability, and reward.Multiple UAV servers act as agents and iteratively update policies through environmental interaction.
  • State space: Each UAV server’s state includes its horizontal location, prior device-selection and subchannel states, selected-device locations, and remaining payload needs.The joint network state collects the states of all UAV servers.
  • Action space: Each agent action specifies UAV horizontal position, device-selection indicators, subchannel allocation indicators, and transmit-power allocation.After each slot, UAVs update positions, device associations, subchannels, and power resources.
  • Transitions and policy: The transition probability describes a UAV server entering a new state after executing an action in the current state.The policy maps perceived states to action probabilities π(a|s) = P(a|s).
  • Reward design: The reward evaluates policy quality by capturing federated execution time and learning-accuracy loss.The network objective is to minimize both quantities, equivalent to maximizing the corresponding cumulative reward.
  • Reward design: The MDP seeks actions that maximize cumulative discounted reward, where γ is the discount factor.This objective represents minimizing total cumulative execution time and learning-accuracy loss.

B. Multi-Agent A3C Algorithm

The multi-agent A3C design uses asynchronous UAV-server agents, actor–critic value estimates, multi-step rewards, and advantage-based policy updates. It avoids the memory and computation demands associated with experience replay.

  • A3C motivation: Large state and action spaces motivate asynchronous multiple agents that interact with the environment in parallel using different exploration policies.Each UAV server acts as one agent and optimizes its policy from value-function estimates.
  • A3C motivation: Experience replay requires sufficient memory and computation resources and uses data generated through an old policy, motivating A3C instead.A3C is presented as an asynchronous advantage actor-critic alternative to these DRL approaches.
  • A3C motivation: Unlike classical actor–critic learning with one agent, A3C supports multiple learning agents operating asynchronously.This design matches the multi-UAV setting in which UAV servers learn policies concurrently.
  • Actor–critic architecture: The actor selects actions under a parameterized policy, while the critic estimates the state value and expected return.The state-action value function instead evaluates expected return after executing an action in a state.
  • A3C updates: A3C uses multi-step rewards to update both the actor policy parameters and critic value-function parameters.The state value is the expected return under a policy, whereas the state-action value conditions that return on an action.
  • Advantage estimation: The advantage function replaces Qπ(s,a) in the policy gradient to address high variance in the critic network.Because Qπ(s,a) cannot be determined directly in A3C, U_t−V^π(s_t;θ_c) estimates the advantage.
  • Network losses: The actor loss combines an advantage-weighted policy term with entropy regularization that favors exploration during training.The hyperparameter ϑ controls the entropy regularization strength.
  • Optimization: Thread-specific actor and critic gradients are accumulated and applied with non-centered RMSProp to minimize the two network losses.Shared gradient mode is described as more robust than separated gradients.

C. Training and Execution of A3C-AFL

A3C-AFL combines asynchronous A3C training for joint network decisions with asynchronous federated local training and aggregation. UAV agents learn device selection, placement, and resource policies, while selected devices train locally and UAVs update global models asynchronously.

  • The A3C Procedure: The A3C training loop synchronizes thread-specific parameters, accumulates actor and critic gradients, and asynchronously updates global network parameters.Training repeats across learning-agent threads until the shared iteration limit is reached, after which the trained model is used for network control.
  • The A3C Procedure: A3C agents at UAV servers jointly learn device selection, UAV placement, subchannel allocation, and power allocation through environment interaction.Each agent observes network states, selects actions using an actor policy, receives rewards, and updates actor–critic parameters asynchronously.
  • Asynchronous Implementation of A3C-AFL: The trained A3C model is loaded to perform device selection, UAV placement, and resource management during federated model updating.Multiple UAV agents explore policies asynchronously, while local model training is also executed asynchronously across participating devices.
  • The AFL Procedure: AFL proceeds through local training and global aggregation at each global communication round.UAVs broadcast global parameters, selected devices update local models using their loss gradients, upload them, and UAVs aggregate the updates.
  • Asynchronous Implementation of A3C-AFL: The asynchronous implementation lets UAV servers aggregate collected local parameters and broadcast updated global models without requiring every selected device to finish simultaneously.Only a number of selected-device responses may be needed to advance, and some device updates may be excluded from aggregation.

D. Complexity and Convergence Analysis

The paper analyzes computational complexity and convergence for the A3C-AFL framework. It establishes A3C policy convergence under stated assumptions and gives federated convergence results under convexity, smoothness, and suitable iteration choices.

  • Computational Complexity: With E episodes and T time steps per episode, the overall A3C training complexity is obtained by scaling per-iteration complexity across the training process.The analysis first derives each network’s iteration complexity and then aggregates it over episodes and time steps.
  • Computational Complexity: The A3C algorithm uses actor and critic fully connected DNNs, with layer complexity determined by products of adjacent-layer neuron counts.The actor and critic networks are analyzed separately, and total training complexity depends on their layer structures and episode length.
  • A3C Convergence: The A3C policy converges to a policy π∗ that guarantees Qπ∗(st, at) > Qπ(st, at) for policies in Q, assuming a finite action space.The theorem requires conditions including bounded reward variance, differentiable policies, i.i.d. state-action-reward sequences, and bounded second moments.
  • FL Convergence: For η-convex and σ-smooth F(w), Theorem 2 bounds the divergence between federated and globally optimal loss.The analysis defines this divergence as |F(w) − F(w∗)| and states that suitable global and local iteration numbers yield global optimality.
  • Simulation Setup: The evaluation compares A3C-AFL with device selection against random-selection AFL, synchronous FL with device selection, and related resource-management variants.Simulations are conducted in Python and use MNIST with a CNN local model.

3) SFL with device selection:

The simulations examine deployment, convergence, and accuracy for federated approaches with and without device selection. Device selection and asynchronous aggregation jointly improve the reported convergence and accuracy behavior under the tested UAV-network setup.

  • Training Convergence: A large actor learning rate βa = 0.001 causes oscillations, whereas βa = 0.00001 produces slow convergence; βa around 0.0001 provides fast convergence and low system cost.The system cost includes model aggregation time and learning accuracy loss.
  • Training Convergence: Around 500 training episodes, system cost gradually converges, and the proposed A3C-based algorithm achieves lower system cost than the gradient-based benchmark.The two methods have comparable convergence speed, with fluctuations attributed to dynamic-environment characteristics and policy exploration.
  • UAV and Device Distribution: UAVs adapt their locations to associated-device counts and distributions to improve channel gain and minimize communication delay.Selected devices are those with high communication and computation capabilities, while low-quality devices do not participate in that aggregation slot.
  • Accuracy Comparison: Both AFL and SFL with device selection reach 90.0% accuracy in about 25 global rounds and show similar convergence speed and accuracy.AFL advances after a number of selected-device responses, whereas SFL waits for all selected devices, increasing aggregation completion time.
  • Accuracy Comparison: With device selection, the proposed AFL outperforms the compared approaches in both wall-clock convergence speed and accuracy.Without device selection, low-quality devices can reduce aggregate accuracy and increase aggregation time; synchronous FL also waits for slower selected devices.

D. Performance Comparison Versus Number of Devices

As the number of devices changes, A3C-AFL and related device-selection methods provide stronger accuracy and completion-time performance than approaches without device selection. The proposed A3C algorithm jointly improves device selection, UAV placement, and resource management relative to the gradient-based benchmark.

  • A3C-AFL and A3C-SFL outperform AFL without device selection in accuracy, with the advantage gap widening as more devices participate.Low-quality model parameters compromise aggregation accuracy, whereas A3C selects devices with high-quality model parameters.
  • AFL without device selection remains near 78.0% accuracy with fluctuations because low-quality devices reduce aggregation accuracy.
  • A3C-AFL achieves higher accuracy than gradient-AFL as the number of devices increases.
  • Aggregation completion time increases with device count for A3C-SFL and AFL without device selection, but decreases slightly for A3C-AFL and gradient-AFL.
  • A3C improves decision making for device selection, UAV placement, and resource management compared with the gradient-based benchmark.Gradient-AFL requires more decision-making time, while SFL waits for all selected devices to finish local updates before aggregation.
  • The proposed AFL framework with its A3C-based algorithm outperforms existing solutions in learning accuracy and federated execution time under different settings.

APPENDIX A PROOF OF THEOREM 1

The appendix analyzes convergence of coordinated multi-agent A3C using alternating critic evaluation and actor policy enhancement. Under the stated assumptions, the critic and actor updates converge to characterized limiting points.

  • A3C searches for a joint policy among UAV-server agents to maximize their cumulative return.
  • Two-time-scale stochastic approximation analyzes critic convergence under a fixed joint policy before analyzing actor policy-parameter convergence.
  • The critic network approximates the action-value function Q(s, a; θc) using linear functions and coordinated agent updates.
  • The critic parameter converges to a final point, and the Q value converges to Q* with probability 1.
  • The actor network converges to a point in a set of asymptotically stable policy parameters under the stated assumptions.
  • When the long-term averaged return satisfies ∇θn c J(θ) = 0, the coordinated A3C policy reaches the supported stationary solution condition.
Loading 2011.14197v1…