Source-linked AI summary
Multi-Agent Reinforcement Learning Based Resource Allocation for UAV Networks
Jingjing Cui, Yuanwei Liu, Arumugam Nallanathan
TL;DR
The paper tackles dynamic resource allocation in multi-UAV communication networks, where long-term reward optimization must handle environmental uncertainty and limited information exchange. It models the problem as a stochastic game and develops an independent Q-learning-based MARL framework in which UAVs select users, power levels, and subchannels from local observations. Simulations show improved performance with suitable exploration and exploitation parameters and an acceptable tradeoff between performance and information-exchange overhead.
Problem
Dynamic multi-UAV resource allocation must maximize long-term rewards despite time-dependent environmental uncertainty and limited information exchange.
Method
The paper formulates resource allocation as a stochastic game and uses independent Q-learning-based MARL agents that select users, power levels, and subchannels.
Results
Simulations show that suitable exploitation and exploration parameters enhance the proposed MARL algorithm, which achieves an acceptable performance tradeoff with reduced information exchange.
Takeaways & Limitations
The framework balances performance gains against information-exchange overheads for multi-UAV resource allocation.
Abstract
from arXiv · showhide
Unmanned aerial vehicles (UAVs) are capable of serving as aerial base stations (BSs) for providing both cost-effective and on-demand wireless communications. This article investigates dynamic resource allocation of multiple UAVs enabled communication networks with the goal of maximizing long-term rewards. More particularly, each UAV communicates with a ground user by automatically selecting its communicating users, power levels and subchannels without any information exchange among UAVs. To model the uncertainty of environments, we formulate the long-term resource allocation problem as a stochastic game for maximizing the expected rewards, where each UAV becomes a learning agent and each resource allocation solution corresponds to an action taken by the UAVs. Afterwards, we develop a multi-agent reinforcement learning (MARL) framework that each agent discovers its best strategy according to its local observations using learning. More specifically, we propose an agent-independent method, for which all agents conduct a decision algorithm independently but share a common structure based on Q-learning. Finally, simulation results reveal that: 1) appropriate parameters for exploitation and exploration are capable of enhancing the performance of the proposed MARL based resource allocation algorithm; 2) the proposed MARL algorithm provides acceptable performance compared to the case with complete information exchanges among UAVs. By doing so, it strikes a good tradeoff between performance gains and information exchange overheads.
I. INTRODUCTION
The paper addresses dynamic resource allocation in multi-UAV communication networks, where rapidly changing environments and limited information make centralized approaches difficult. It proposes MARL to jointly select users, power levels, and subchannels while balancing performance and information-exchange overhead.
- UAV-enabled networks support rapidly deployable wireless services and can enhance network capacity and coverage.
- Resource allocation across transmit power, serving users, and subchannels is essential for improving energy efficiency and coverage.
- Existing work largely emphasizes deployment and trajectory design, while multi-UAV resource allocation from a MARL perspective remains insufficiently investigated.
- Complete network information is difficult to obtain in dynamic environments, and centralized methods become challenging as network size increases.
- The paper formulates dynamic multi-UAV resource allocation as a stochastic game and develops an independent-learning MARL algorithm based on Q-learning.
- Simulation results indicate that suitable exploration and exploitation parameters improve performance, while the MARL framework trades information-exchange overhead against performance gains.
C. Organization
The article proceeds from the downlink multi-UAV system model and resource-allocation formulation to stochastic-game modeling, Q-learning-based MARL, simulations, and conclusions.
- Organization: Section II presents the system model for downlink multi-UAV networks.
- Organization: Section III formulates the resource-allocation problem and presents the stochastic-game framework.
- Organization: Section IV designs the Q-learning-based MARL resource-allocation algorithm, followed by simulations in Section V and conclusions in Section VI.
II. SYSTEM MODEL
The system models multiple autonomous UAVs providing direct connectivity to randomly distributed ground users over potentially overlapping subchannels. It studies dynamic joint selection of users, power levels, and subchannels under local channel knowledge and UAV-to-ground channel models.
- The network contains multiple single-antenna UAVs and users, with UAVs flying over a region and providing direct aerial connectivity.
- The total bandwidth is divided into K orthogonal subchannels, which different UAVs may simultaneously occupy.
- UAV trajectories are predetermined by autonomous, pre-programmed flight plans, while resource allocation adapts dynamically.
- Each UAV selects its communicating user, transmit power level, and subchannel without a central controller or global wireless-environment knowledge.
- The model considers probabilistic and LoS UAV-to-ground channels whose path loss depends on propagation conditions, distance, altitude, and elevation angle.
B. Signal Model
The signal model represents each UAV's transmission through user, subchannel, and discrete-power decisions, while accounting for interference and QoS-constrained throughput-minus-power rewards. An online learning formulation targets long-term energy-efficiency performance under changing decisions and partial information.
- UAVs may share subchannels, creating interference that enters each user link's SINR.
- The online learning design addresses unknown network-stay duration and targets long-term energy-efficiency performance.
- A feasible decision must keep served users' SINRs at or above the predefined QoS threshold.
- When the QoS constraint is satisfied, the immediate reward equals selected-user throughput minus power-consumption cost; otherwise, it is zero.
- The long-term objective uses discounted rewards, with δ near 0 emphasizing near-term gains and δ near 1 weighting future rewards more heavily.
B. Stochastic Game Formulation
The resource-allocation problem is formulated as a non-cooperative stochastic game in which UAVs act using local observations while other agents' actions and state transitions are unknown. Each UAV learns a strategy intended to reach a Nash-equilibrium response over repeated interactions.
- The joint user, subchannel, and power-selection problem is modeled as a stochastic game under uncertain environments.
- Each UAV observes its own state and SINR but does not know other UAVs' actions, states, rewards, or transition probabilities.
- The stochastic game comprises state sets, players, joint and individual action sets, transition probabilities, and individual reward functions.
- UAV rewards depend on the joint strategy, so all players may not be able to maximize their expected rewards simultaneously.
- A Nash equilibrium requires each UAV's strategy to be a best response to the others, such that unilateral deviation provides no benefit.
- Repeated interaction with the imperfect-information environment treats each UAV as a learning agent seeking an equilibrium strategy.
IV. PROPOSED MULTI-AGENT REINFORCEMENT-LEARNING ALGORITHM
The paper proposes a MARL framework for multi-UAV resource allocation and uses Q-learning to maximize expected long-term rewards. The framework is illustrated as an agent-independent architecture for the network.
- The proposed framework applies multi-agent reinforcement learning to resource allocation in multi-UAV networks.
- Its resource-allocation algorithm is based on Q-learning and targets the expected long-term reward.
A. MARL Framework for Multi-UAV Networks
The paper models each UAV as an independent learner in a stochastic multi-agent environment, using local observations without observing other UAVs’ rewards or actions. The framework uses individual MDPs and Q-learning-based action-value reasoning to seek reward-maximizing strategies.
- The proposed agent-independent method lets all agents make decisions independently while sharing a common Q-learning structure.
- Each UAV’s MDP comprises finite state and action sets, transition dynamics, and an expected-reward function.Rewards depend on the current state and action under the Markov property.
- The Q-function gives expected reward from a state after taking an action and following a policy.State-value and action-value functions provide the basis for evaluating strategies.
- Bellman optimality equations select actions that maximize the current state’s Q-function.The optimal state-value formulation maximizes over the action space rather than the strategy space.
- In the multi-agent setting, each Q-function depends on joint actions and the joint policy, complicating optimal joint-strategy learning.The proposed independent-learner formulation addresses this complexity by removing inter-agent information exchange.
- Each UAV operates as an independent learner without observing other UAVs’ rewards or actions.The agents interact with the environment as if no other UAVs existed.
B. Q-Learning based Resource Allocation for Multi-UAV Networks
The resource-allocation algorithm has each UAV independently learn action values through Q-learning while balancing exploration and exploitation. It uses ε-greedy action selection, decreasing learning rates, and iterative updates of rewards and Q-values.
- The independent-learners MARL algorithm applies Q-learning separately to each UAV for resource allocation.Each UAV runs a standard Q-learning procedure to learn its optimal Q-values and strategy.
- Q-values evaluate current actions using the current state and successor-state values.The update rule recursively learns action quality from observed transitions and rewards.
- ε-greedy selection chooses a random action with probability ε and the highest-Q action with probability 1 − ε.The exploration parameter satisfies ε ∈ (0, 1).
- The learning rate decreases according to its prescribed iteration-dependent rule to support Q-learning convergence.The rate parameters satisfy cα > 0 and ϕα ∈ (1/2, 1].
- The algorithm initializes Q-values to zero and uses equal-probability actions when the initial state is unknown.This procedure is called zero-initialized Q-learning.
- At each iteration, the algorithm measures SINR, updates the instantaneous reward and action value, then advances the state and time slot.
C. Analysis of the proposed MARL algorithm
The analysis establishes convergence properties for the proposed independent multi-agent Q-learning procedure. Under the stated assumptions, individual UAV learning converges to optimal action values for individual strategies.
- Each UAV’s Q-learning procedure converges to the Q-value associated with its individual optimal strategy.The proposition relies on the non-cooperative structure and convergence of Q-learning.
- The Q-learning algorithm converges with probability one to the optimal Q* under the theorem’s assumptions.The assumptions include finite state and action spaces and bounded reward variance.
V. SIMULATION RESULTS
Simulations evaluate the MARL resource-allocation algorithm under multiple UAV-network configurations and compare it with information-sharing and random-selection baselines. The results identify ε = 0.5 as effective in the considered setups and show a tradeoff between reward improvement and reduced information exchange.
- Learning dynamics: Average reward per time slot decreases with algorithm iterations as the Q-learning update rate becomes slower.The learning rate αt decreases with iteration count.
- Exploration and exploitation: ε = 0.5 is a good choice in the considered setup for average-reward performance.The comparison evaluates ε = {0, 0.2, 0.5, 0.9}.
- Network configurations: Under different network configurations, average-reward curves show similar trends and attain the optimal average reward at ε = 0.5.This behavior is reported for the LoS and probabilistic channel configurations.
- Baseline comparisons: The matching-based user-selection algorithm achieves higher average reward than the proposed MARL algorithm when it exchanges UAV information.The matching procedure gives each UAV access to other UAVs’ actions before its decision.
- Baseline comparisons: The proposed MARL algorithm achieves higher average reward than random user selection while avoiding information exchange among UAVs.The reported result is framed as a tradeoff between reducing exchange overhead and improving reward.
- Speed effects: For fixed speed, average reward increases monotonically with algorithm iterations, while higher speeds initially improve learning faster but later reduce reward.At high speeds, shorter serving time after the UAV leaves the disk limits the eventual reward.
VI. CONCLUSIONS
The paper formulates multi-UAV dynamic resource allocation as a stochastic game and solves it with independently learning Q-based agents. Simulations indicate a tradeoff between performance gains and information-exchange overhead, while motivating extensions involving cooperation and UAV mobility optimization.
- The study targets real-time resource allocation in multi-UAV downlink networks to maximize long-term rewards.
- A stochastic-game formulation models uncertainty and assigns each UAV an expected-reward-maximizing resource-allocation strategy.
- An independently learning MARL algorithm has UAVs make decisions separately using a shared Q-learning structure.
- The proposed MARL resource allocation achieves a tradeoff between performance gains and information-exchange overhead.
- Future directions include joint learning with cooperation under partial information exchange and optimizing UAV deployment and trajectories for energy efficiency.
APPENDIX A: PROOF OF PROPOSITION 1
The appendix proves Proposition 1 by expressing a UAV’s state value through expected next-step rewards and state values, then establishing the associated Q-learning convergence conditions. The proof uses stochastic-approximation results and contraction properties of the optimal Q-function.
- The proof expands one UAV’s state value into expected quantities over next states and actions.
- The appendix substitutes intermediate expressions into the proposition’s equation to complete the proof.
- The convergence argument assumes a finite state space and bounded variance conditions involving the past information and a weighted maximum norm.
- Q-learning is treated as a stochastic process whose convergence is established using stochastic-approximation theory.
- A contraction mapping for the optimal Q-function supports the convergence proof under the infinity norm.