PaperlayerView exact revision on arXiv

Source-linked AI summary

Adaptive Beam Hopping and Power Control for Dual-Layer Over-the-Air Online Federated Learning in LEO Satellite Networks

Zhendong Li, Shaojie Wang, Zhou Su, Zihao Zhang, Haixia Peng, Nan Cheng, Ying Wang, Wen Chen

arXiv:2609.03202v1cs.ITeess.SPmath.IT

TL;DR

Online FL over LEO satellites must balance data freshness and aggregation accuracy under limited beams, power, and changing network conditions. The paper models two-hop OTA aggregation and jointly learns beam hopping and power control with PPO. Simulations report that PPO outperforms the listed baselines across learning and data-utilization outcomes.

  • Problem

    OTA-enabled online FL in LEO networks must coordinate learning-oriented beam scheduling and aggregation under changing coverage, data freshness, and satellite resource constraints.

    Source: §I · p. 2

  • Method

    The paper formulates dual-layer OTA aggregation with beam hopping and power control as an MDP and develops a PPO-based framework with an MSE-aware reward.

    Source: §I · p. 2, 3) Reward · p. 7

  • Results

    PPO consistently outperforms SAC, DDPG, TD3, and a channel-gain-based greedy policy in training reward, final training loss and test accuracy, and long-term data utilization.

    Source: §V · p. 12

  • Takeaways & Limitations

    The simulations support PPO-based adaptive scheduling for the evaluated OTA-enabled online-FL tasks, with gains reported across learning outcomes and long-term data utilization.

    Source: §V · p. 12

Abstract

from arXiv · show

This paper investigates over-the-air (OTA) computation enabled online federated learning (FL) in low-Earth orbit (LEO) satellite networks. Specifically, we consider a dual-layer OTA aggregation architecture, where ground devices upload analog model updates to serving satellites via uplink OTA aggregation, and satellites forward the aggregated signals to a data processing center through the second round OTA aggregation. Then, we formulate a long-term data-utilization maximization problem in which devices continuously collect new data and untrained samples gradually lose freshness. The problem is subject to the satellite beam budget, transmit-power limit, and global mean squared error (MSE) constraint that governs end-to-end aggregation distortion. This yields a coupled mixed-integer nonlinear programming (MINLP) problem, involving tightly coupled discrete beam-hopping decisions and continuous power control. Due to the combinatorial action space and nonconvex constraints, the problem is NP-hard and computationally intractable. Furthermore, the time-varying satellite topology and dynamic data generation render it a sequential decision-making problem, necessitating adaptive online scheduling. To address these issues, we cast the problem as a Markov decision process and develop a proximal policy optimization (PPO)-based deep reinforcement learning framework that jointly optimizes adaptive beam hopping and power control, using an MSE-aware reward to balance data utilization and aggregation accuracy. Numerical simulation results verify that the proposed algorithm consistently outperforms other benchmark schemes, achieving superior long-term data utilization and faster FL convergence while satisfying the MSE requirement.

I. INTRODUCTION

The paper targets learning-centric OTA aggregation and beam scheduling for online FL in LEO networks, where data freshness, aggregation distortion, and satellite resource limits interact. It proposes a dual-layer OTA architecture and formulates joint beam-hopping and power control as a sequential optimization problem.

I. INTRODUCTION

  • Existing satellite-FL work leaves physical-layer aggregation and satellite-specific beam management underexplored, while OTA-FL research largely assumes terrestrial networks.The paper identifies communication-efficient aggregation and agile beam allocation under changing coverage as coupled bottlenecks.Source: §I · pp. 1–2
  • The proposed framework sends device updates to serving satellites by uplink OTA aggregation, then forwards satellite aggregates to a data processing center over a second OTA link.It models data freshness with a time-discounted data amount and constrains global MSE.Source: §I · p. 2
  • The system jointly optimizes adaptive beam hopping and power control under beam budgets, power limits, and aggregation-error thresholds.The resulting mixed-integer nonlinear program is treated as an NP-hard sequential decision problem in a Markov decision process.Source: §I · pp. 2–3
  • A PPO-based deep reinforcement learning algorithm learns policies for the hybrid continuous-discrete action space through interaction with the OTA online-FL environment.The framework addresses time-varying topology and dynamic data generation.Source: §I · p. 3

II. SYSTEM MODEL AND PROBLEM FORMULATION

  • The network comprises ground devices, moving LEO satellites, and a logically central DPC, with satellite coverage and channels treated as quasi-static within each communication round.The DPC is described as physically realized by globally interconnected ground gateways.Source: §II · p. 3
  • Each round selects satellite-served cells, trains participating devices locally, aggregates their updates at satellites, and sends intermediate models to the data processing center for a second OTA aggregation.The DPC maintains and broadcasts the global model; satellites cannot select the same cell simultaneously.Source: §II · p. 3

A. Channel Model

The channel model specifies satellite-to-device and satellite-to-DPC links using directional antenna gains and a line-of-sight propagation assumption.

A. Channel Model

  • The model focuses on the dominant line-of-sight component and neglects shadowing and multipath under clear-sky conditions.It also assumes Doppler compensation and negligible rain attenuation and atmospheric absorption.Source: A. Channel Model · p. 4
  • Device and satellite-beam antenna gains parameterize the device-to-beam channel, with the satellite receive pattern following a directional model.The satellite antenna pattern uses an aperture-based expression.Source: A. Channel Model · p. 4
  • Device and DPC antennas use a simplified directional model, while the satellite points its boresight toward the DPC during intermediate-model uploads.The satellite transmit and receive antenna gains are assumed identical.Source: A. Channel Model · p. 4

B. OTA Computation

The passage identifies each satellite’s participating devices and introduces a normalized signal vector for each device.

B. OTA Computation

1) Device-to-LEO Uplink:

The uplink model describes device-to-satellite transmission and satellite forwarding to the DPC, then evaluates distortion in the recovered global model.

1) Device-to-LEO Uplink:

  • Device signals are received at the serving satellite with channel gains and noise, and each device’s transmit power is constrained.The device power coefficient is b_k(t), and the satellite receiver noise is modeled as additive white Gaussian noise.Source: 1) Device-to-LEO Uplink · p. 5

2) LEO-to-DPC Uplink:

  • Satellite forwarding preserves the weighted OTA sum by transmitting a unit-variance normalization of its received sum, with power controlled by b_n(t).The DPC receives the forwarded signal with additive white Gaussian noise and a satellite power constraint.Source: 2) LEO-to-DPC Uplink · p. 5
  • The end-to-end MSE measures distortion between the recovered global model and the desired model, including aggregation bias and noise contributions.The bias term arises from mismatch between channel- and power-induced analog weights and the desired digital aggregation weights; noise is added on the uplink and at the DPC.Source: 2) LEO-to-DPC Uplink · p. 5

C. Optimization Problem Formulation

The formulation maximizes long-term use of fresh training data while satisfying beam, power, and aggregation-MSE constraints. Dynamic data and satellite conditions make the coupled optimization an NP-hard MINLP.

C. Optimization Problem Formulation

  • A device’s available data accounts for new samples and cached data from earlier rounds, while a finite buffer preserves freshness and bounds workload.Data from unselected rounds may be retained, but the buffer has a maximum capacity.Source: C. Optimization Problem Formulation · p. 5
  • Under the stated IID-data assumption, maximizing participating data reduces stochastic-gradient variance and contributes to minimizing global loss.The paper assumes local datasets are IID across the network.Source: C. Optimization Problem Formulation · p. 5
  • The objective is to maximize total training data exploited over time, under beam, transmit-power, and global aggregation-MSE constraints.Beam activation must respect budgets, visibility, and interference spacing; device power depends on beam activation.Source: C. Optimization Problem Formulation · p. 6
  • Coupled binary beam decisions and continuous power controls, together with nonconvex constraints, make the problem NP-hard and static algorithms computationally intractable.Time-varying topology and data generation turn it into a sequential decision problem.Source: C. Optimization Problem Formulation · p. 6

III. PROXIMAL POLICY OPTIMIZATION-BASED DRL FRAMEWORK

The framework casts adaptive scheduling as an MDP: each round’s observation informs an action and reward, and the next state depends on the current state and action. The state combines data availability with satellite-link conditions.

A. MDP Formulation

  • The MDP transition depends on the current state and scheduling action, not historical states, and the scheduler receives a reward based on data use and aggregation distortion.The next state includes updated data queues, new channel realizations, and the newly aggregated global model.Source: A. MDP Formulation · p. 6

1) State:

  • The state encodes available device data, satellite-to-device channel gains, and satellite-to-DPC channel gains.Vectorizing the channel information gives the DRL observation a fixed dimension.Source: 1) State · p. 6
  • These state components capture factors that influence data utilization and OTA aggregation quality.The passage identifies data distribution and instantaneous channel conditions as the state’s key information.Source: 1) State · p. 6

2) Action:

The framework jointly selects beam activations and device and satellite transmit-power coefficients, using PPO to schedule OTA-enabled online FL. Its reward favors on-time data utilization while penalizing aggregation distortion above the MSE threshold.

2) Action:

  • PPO jointly controls binary beam activation and device and satellite transmit-power coefficients through a policy over discrete and continuous actions.The actor maps state observations to actions, which are made feasible before being applied to the FL environment.Source: 2) Action · pp. 6–7, B. PPO for OTA-Enabled Online FL · p. 7

3) Reward:

  • The per-round reward combines normalized utilized data with a capped penalty when instantaneous aggregation MSE exceeds its threshold.No penalty applies at or below the threshold; above it, the penalty is bounded.Source: 3) Reward · p. 7
  • The reward aligns cumulative discounted return with the long-term data-utilization objective while softly discouraging actions that violate the global MSE constraint.The normalized reward is bounded, which the authors say benefits PPO training stability and sample efficiency.Source: 3) Reward · p. 7

B. PPO for OTA-Enabled Online FL

  • PPO uses a hybrid actor for beam scores and Gaussian power coefficients, with a critic and clipped policy updates for stable learning.A feasibility mapping converts beam scores into binary activations; entropy regularization encourages exploration of beam and power configurations.Source: B. PPO for OTA-Enabled Online FL · pp. 7–8

A. Experimental Setup and Benchmark Algorithms

The evaluation uses two FL workloads with different datasets, models, and training horizons, and compares PPO against three DRL algorithms and a channel-gain greedy policy.

A. Experimental Setup and Benchmark Algorithms

  • The experiments use MNIST digit recognition with an MLP for 60 rounds and CIFAR-10 classification with a CNN for 180 rounds.The respective tasks use 2 and 3 local epochs; device data are assigned by unique MNIST shards.Source: A. Experimental Setup and Benchmark… · p. 8
  • PPO trains for 1,500 episodes, with γ = 0.95, λ_GAE = 0.95, clip ratio ϵ = 0.2, and an update interval of 256 transitions.The actor learning rates differ by task, while the critic learning rate is fixed at 2×10^-3.Source: A. Experimental Setup and Benchmark… · p. 9
  • The benchmark set comprises DDPG, TD3, SAC, and a channel-gain-based greedy heuristic.The DRL baselines share PPO’s state, action, and reward definitions and are trained in the same environment.Source: A. Experimental Setup and Benchmark… · p. 9
  • The greedy policy fixes device and satellite powers at their maxima and selects the V cells with strongest effective channel gains each epoch.It is described as a lower bound for learning-based methods and drives aggregation MSE well below the prescribed threshold.Source: A. Experimental Setup and Benchmark… · p. 9

B. DRL Training Performance

PPO attains the highest steady-state training reward on both FL tasks and ultimately achieves the highest data amount in the MNIST+MLP task.

B. DRL Training Performance

  • PPO attains the highest steady-state reward in MNIST+MLP, while the greedy baseline remains near 0.7 and trails all DRL schedulers.PPO’s reward rises smoothly with relatively small fluctuations; the greedy policy does not learn over time.Source: B. DRL Training Performance · pp. 9–10, Figure 3 · p. 10
  • PPO ultimately attains the highest training reward among schemes in CIFAR-10+CNN, despite noisier reward curves in this more complex task.TD3 and SAC can briefly reach high rewards, but their steady-state rewards remain below PPO’s.Source: Figure 4 · p. 10, B. DRL Training Performance · p. 10
  • PPO’s data amount keeps increasing before stabilizing at the highest level, while SAC and TD3 settle at intermediate levels and DDPG remains lower.Early data-amount drops occur as initial policies violate the MSE threshold and become more conservative; agents later learn to satisfy the requirement while increasing utilization.Source: Figure 5 · p. 10, B. DRL Training Performance · p. 10

C. FL Performance on MNIST and CIFAR-10

On MNIST, PPO reduces loss fastest and reaches above 84% final accuracy; on CIFAR-10, it maintains the best loss and accuracy trajectories among the compared schedulers.

C. FL Performance on MNIST and CIFAR-10

  • PPO reduces MNIST loss fastest, reaches around 70% accuracy by roughly round 25, and converges above 84% after 60 rounds.SAC and TD3 form the next-best tier, while DDPG and the greedy baseline converge more slowly.Source: C. FL Performance on MNIST and CIFAR-10 · p. 11
  • On CIFAR-10+CNN, PPO attains the lowest loss and highest accuracy trajectory, with faster initial accuracy growth than the baselines.The gap over the greedy policy is more pronounced than on MNIST; the paper notes that blindly minimizing aggregation MSE is insufficient for the more complex workload.Source: C. FL Performance on MNIST and CIFAR-10 · p. 11

D. Impact of BH and MSE Threshold on Data Utilization

PPO consistently outperforms the benchmark schedulers in data utilization across the tested beam budgets and MSE thresholds. Greater beam budgets and less stringent MSE thresholds increase the average data amount per round.

D. Impact of BH and MSE Threshold on Data Utilization

  • Across beam budgets V = 3 to V = 7, PPO consistently achieves the highest data utilization; SAC, TD3, and DDPG also benefit from larger V, but their gains are comparatively smaller.More beams enable each satellite to activate more cells and aggregate more device updates per communication round.Source: D. Impact of BH and MSE Threshold on… · p. 12
  • Relaxing the global MSE threshold ρ from -7 dB to -3 dB monotonically increases average data amount for all algorithms, with PPO highest at every threshold.A looser distortion requirement allows aggregation of noisier updates.Source: D. Impact of BH and MSE Threshold on… · p. 12, Figure 11 · p. 12

V. CONCLUSION

  • The paper formulates long-term training-data utilization under satellite beam, power, and global MSE constraints, then uses an MDP and PPO to learn beam-hopping and power-control strategies.The framework addresses settings where devices continuously collect data and untrained samples lose freshness.Source: §V · p. 12
  • Across MNIST+MLP and CIFAR-10+CNN, PPO consistently outperforms SAC, DDPG, TD3, and a channel-gain-based greedy policy in training reward, final training loss and test accuracy, and long-term data utilization.The conclusion reports this comparison across both tasks.Source: §V · p. 12
  • The study assumes IID local data and perfect scheduler CSI; delayed or imperfect CSI would degrade channel-inversion alignment and OTA aggregation accuracy.Future work includes model heterogeneity and performance guarantees under delayed and imperfect CSI.Source: §V · pp. 12–13
Loading 2609.03202v1…