PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Agent Deep Reinforcement Learning for Dynamic Power Allocation in Wireless Networks

Yasar Sinan Nasir, Dongning Guo

arXiv:1808.00490v3eess.SPcs.ITstat.ML

TL;DR

Wireless power-control methods often require computationally difficult optimization and full, current cross-cell CSI, limiting scalability. This paper develops a distributed, model-free deep-reinforcement-learning scheme in which transmitters adapt power from local neighbor information. The approach achieves comparable or better performance than centralized benchmarks in the reported simulations, including with delayed and limited CSI.

  • Problem

    Power allocation is generally nonconvex and NP-hard, while existing centralized methods require full cross-link CSI and can face scalability and delay burdens.

  • Method

    Each transmitter uses a local state and deep Q-network to select power actions, with information exchanged among nearby neighbors under delayed and incomplete CSI.

  • Results

    The proposed distributed algorithm achieves comparable or better performance than centralized WMMSE and FP benchmarks in simulations, including convergence to delayed-CSI central allocation in one reported setting.

  • Takeaways & Limitations

    The results support using model-free deep reinforcement learning for real-time dynamic power allocation when full, instantaneous cross-cell CSI is impractical.

  • Takeaways & Limitations

    Centralized training remains a scalability limitation, although smaller-network DQNs can be applied to larger networks and reused to accelerate training.

Abstract

from arXiv · show

This work demonstrates the potential of deep reinforcement learning techniques for transmit power control in wireless networks. Existing techniques typically find near-optimal power allocations by solving a challenging optimization problem. Most of these algorithms are not scalable to large networks in real-world scenarios because of their computational complexity and instantaneous cross-cell channel state information (CSI) requirement. In this paper, a distributively executed dynamic power allocation scheme is developed based on model-free deep reinforcement learning. Each transmitter collects CSI and quality of service (QoS) information from several neighbors and adapts its own transmit power accordingly. The objective is to maximize a weighted sum-rate utility function, which can be particularized to achieve maximum sum-rate or proportionally fair scheduling. Both random variations and delays in the CSI are inherently addressed using deep Q-learning. For a typical network architecture, the proposed algorithm is shown to achieve near-optimal power allocation in real time based on delayed CSI measurements available to the agents. The proposed scheme is especially suitable for practical scenarios where the system model is inaccurate and CSI delay is non-negligible.

I. INTRODUCTION

The paper targets scalable inter-cell interference management by learning distributed transmit-power decisions under practical information constraints. It uses deep reinforcement learning to address dynamic channels, delayed or incomplete CSI, and large-network scalability while optimizing weighted sum-rate objectives.

  • Motivation: Dense wireless networks make power control difficult because increasing one transmitter’s power can improve its rate while degrading interfered links.The weighted sum-rate objective includes maximum sum-rate and proportionally fair scheduling as special cases.
  • Limitations of Existing Methods: Existing WMMSE and fractional-programming benchmarks are centralized and generally require full, up-to-date cross-cell CSI.These requirements create communication and scalability burdens in practical networks.
  • Limitations of Existing Methods: Supervised learning can reach 90% or higher of WMMSE sum-rate, but it requires full CSI and large datasets generated by computationally expensive optimization.Its scalability and adaptability also depend on the underlying system model and network size.
  • Contribution: The proposed algorithm is distributively executed by transmitters to compute power allocations in real time under time-varying channel conditions.The paper uses deep Q-learning and evaluates robustness and scalability through simulations with Jakes fading.
  • Contribution: The scheme uses model-free deep reinforcement learning to handle delayed information exchange and incomplete cross-link CSI without requiring optimization-generated training data.Its computational complexity is claimed not to depend on network size under a bounded link-density condition.

III. DYNAMIC POWER CONTROL

The dynamic power-control problem maximizes a weighted sum-rate in every time slot despite nonconvexity, delayed CSI, and restricted local information exchange. The paper therefore limits coordination to nearby transmitters and uses local measurements to support scalable decisions.

  • III. DYNAMIC POWER CONTROL: The per-slot weighted sum-rate power-allocation problem is generally nonconvex and NP-hard, subject to 0 ≤ p_i ≤ Pmax.The optimal allocation can change at the beginning of every time slot.
  • III. DYNAMIC POWER CONTROL: Setting all weights to one yields sum-rate maximization, while history-dependent weights implement proportional fairness.The proportional-fair formulation maximizes the sum of log-average spectral efficiencies.
  • Information Constraints: Centralized methods must transfer direct and interfering-channel CSI to a controller, where cross-link CSI can be delayed or outdated and iterative convergence limits responsiveness.Backhaul communication adds another burden before centralized allocation can be returned.
  • Information Constraints: To improve scalability, each transmitter exchanges information only with nearby transmitters defined through SNR-based neighborhood sets.These sets distinguish interferers from receivers whose links are interfered with.
  • Local Information: Each transmitter combines direct-channel feedback, interference-plus-noise measurements, received interferer powers, spectral efficiency, and delayed neighbor information.The uncertainty in these measurements is attributed to one-slot exchange latency rather than measurement inaccuracy.

A. Overview of Deep Q-Learning

Deep Q-learning replaces impractical tabular Q-learning for large state-action spaces by approximating the Q-function with a neural network. In the multi-agent setting, each transmitter acts from a local state, but changing policies make the shared environment non-stationary.

  • Q-Learning: Q-learning agents observe states, choose actions, receive rewards, and learn policies through trial-and-error interaction with the environment.The policy seeks to maximize expected future cumulative discounted reward.
  • Q-Learning: The ϵ-greedy policy usually selects the highest-valued action but explores randomly with probability ϵ.This exploration is intended to avoid becoming stuck at non-optimal policies.
  • Deep Q-Learning: Classical Q-learning becomes impractical when many states are rarely visited and its lookup table is too large to store.These issues arise in the large state and action spaces of the power-control problem.
  • Deep Q-Learning: A deep Q-network estimates the Q-function with finite-dimensional parameters instead of a lookup table over the full state-action space.The network is trained from interaction experiences, including experience replay and stochastic-gradient updates.

B. Proposed Multi-Agent Deep Reinforcement Learning Algorithm

The proposed scheme distributes DQN execution across transmitters while centralizing training, addressing multi-agent non-stationarity and scalability constraints. Agents act from local states, shared parameters, and replayed experiences.

  • Each transmitter acts as an agent in a multi-agent deep reinforcement learning system whose joint actions affect a common environment.
  • The method executes identical DQNs distributively at transmitters but trains one shared DQN centrally to ease implementation and improve stability.Centralized training uses experiences gathered from all agents and shares the trained parameters.
  • Agents select synchronized actions from local states without observing neighbors’ current actions, while past experiences help estimate future neighbor impacts.
  • The fully connected DQN has an input layer, three hidden layers, and an output layer whose ports estimate Q-values for available actions.
  • Training samples mini-batches from a shared FIFO experience-replay memory, updates parameters by minimizing the DQN loss, and broadcasts new parameters periodically.Broadcast parameters become available after transmission delay, and training may stop after convergence.

C. States

Each agent builds a fixed-length local state by prioritizing the strongest relevant interferers and interfered neighbors. Virtual noise agents pad missing entries to preserve the DQN input size.

  • The state design limits communication to c interferers and interfered neighbors, controlling the local information and DQN complexity.
  • Interferers are sorted by their current received power at agent i’s receiver so the strongest sources receive priority.
  • When fewer than c neighbors exist, virtual noise agents provide inconsequential placeholder inputs that fill the fixed-length state.
  • Interfered neighbors are prioritized according to the share of agent i’s interference at their receivers, emphasizing the most significantly affected neighbors.
  • The state is organized into three main feature groups based on systematic design principles refined through preliminary simulations.

1) Local Information:

The local-information feature group combines agent i’s recent transmit behavior, objective contribution, direct-channel measurements, and interference-plus-noise observations. Recent measurements help track channel changes.

  • The local feature group begins with agent i’s transmit power during the previous time slot.
  • It includes two measurements of agent i’s recent potential contribution to the network objective, using 1/w_i rather than w_i because w_i can become large.
  • The final four local inputs contain two recent direct downlink-channel measurements and the total interference-plus-noise power at receiver i.
  • Seven input ports are reserved for this local feature group, and additional past measurements can help track slower channel variations.The design weighs improved environmental tracking against the increased amount of state information.
  • The feature group also represents interference from neighbors to receiver i and their contributions to the network objective.

2) Interfering Neighbors:

The interfering-neighbor features encode each selected interferer’s effect on receiver i and retain recent history to track changing interference conditions. Feedback from interfered neighbors captures agent i’s impact elsewhere.

  • Three state inputs are reserved for each selected interferer, including its interference at receiver i and two terms representing its objective significance.
  • The agent retains interferer history from the previous neighbor set to track changes in its receiver’s interference condition.
  • This interfering-neighbor feature group reserves 6c state elements in total.
  • Feedback from interfered neighbors lets agent i gauge its interference to nearby receivers and their contributions to the objective.
  • When a link becomes inactive, the agent preserves information from its last active time to support deciding when to reactivate.

D. Actions

The algorithm uses discrete transmit-power actions shared across agents and trains rewards to balance each link’s weighted sum-rate contribution against interference imposed on neighbors.

  • Action space: The action set uses absolute power levels rather than discrete increments from the previous transmit power.This design increases DQN output ports compared with incremental actions but improves robustness as channel variation increases.
  • Action space: Each agent selects one of |A| shared discrete power levels between 0 and Pmax to update its transmit-power strategy.The action space is identical for all agents, and the DQN has |A| output ports.
  • Reward function: Reward computation uses neighbor-link spectral efficiencies and interference measurements delivered to a network trainer.For relevant interfered links, the trainer evaluates spectral efficiency without the agent’s interference and computes interference-related terms from measured interference-plus-noise power.
  • Reward function: The price charged to a transmitter represents the externality its interference causes to another link.Peak-power transmission maximizes both direct contribution and penalty, while silence earns zero reward.
  • Reward function: The reward is designed to optimize the network weighted sum-rate objective while accounting for each transmitter’s interference to affected neighbors.The reward combines the agent’s direct contribution with a penalty for interference to interfered links.

V. SIMULATION RESULTS

Simulations evaluate the distributed DQN across network configurations, channel conditions, training settings, and benchmark allocations. The experiments include delayed and idealized centralized references, with a matched DQN reported to outperform ideal centralized algorithms in a 19-link case.

  • Simulation setup: The simulations use 19–100 homogeneously deployed links, with transmitter spacing R from 100 to 1000 meters and receiver exclusion radius r from 10 to R−1 meters.Receivers are placed randomly within cells, and multi-link-per-cell scenarios are considered separately.
  • Simulation setup: The default time slot is 20 ms with maximum Doppler frequency fd = 10 Hz, full-buffer traffic, and SINR capped at 30 dB for spectral-efficiency calculation.The threshold η is set to 5 in the relevant constraints.
  • Algorithm configuration: The DQN uses three hidden layers of 200, 100, and 40 neurons, with 36,150 total parameters and a moderate discount factor γ = 0.5.The moderate γ reflects the smaller correlation between actions and future rewards under fading; parameters are periodically distributed through the backhaul.
  • Evaluation protocol: Each simulation averages at least 10 random initializations, with 40,000-slot training and 5,000-slot testing phases.Testing removes the trainer and terminates ε-greedy exploration.
  • Benchmarks: The five benchmarks are ideal WMMSE, ideal FP, delayed central FP, random allocation, and full-power allocation.The first two use instantaneous full CSI; delayed central FP uses one-slot-delayed full CSI.
  • Sum-rate maximization: For n = 19 links, a matched DQN trained for the specific initialization outperforms centralized WMMSE and FP algorithms evaluated with ideal full CSI and no delay.An unmatched DQN reuses a model trained under another network initialization with the same R and r parameters.

1) Robustness:

The proposed DQN remains effective across changing receiver geometry, Doppler conditions, and network sizes, while retaining fast convergence and execution. Its robustness includes delayed or incomplete CSI settings and scalability to larger networks.

  • Robustness: 25,000 time slots: the DQN converges while matching the central power allocation under delayed full CSI and using distributed execution with incomplete CSI.Training from scratch also learns to compensate for CSI delays and specialize to the network initialization scenario.
  • Robustness: 15,000 time slots at reduced fd versus more than 35,000 time slots when fd →∞ and ρ = 0, while sum-rate performance remains unchanged as fd decreases.The convergence-time change is associated with the variation of states visited during training.
  • Robustness: 25,000 time slots: convergence remains about this level as the receiver-free inner region increases, despite greater need for interference mitigation.Random and full-power allocations show much lower sum-rate performance than the central algorithms in that setting.
  • Scalability: 50 links: the DQN still converges in 25,000 time slots with high sum-rate performance, while at 100 links matched-DQN outperformance drops with fixed neighbor input c = 5.Increasing c can improve performance at higher computational complexity, and an unmatched DQN trained for 19 links still performs well as links increase.
  • Scalability: Less than 0.5 ms per agent action and less than T = 20 ms per batch remain unchanged from 19 to 100 links because the DQN architecture is fixed.By comparison, FP convergence increases from about 15 ms to 35 ms, while WMMSE convergence remains proportional to n.

3) Extendability to Multi-Link per Cell Scenarios and Different Channel Models:

The proposed distributed deep-Q-learning scheme extends to multi-link cells and different channel models while retaining convergence and strong performance. It also supports proportionally fair scheduling and adapts to varied deployment parameters, although training remains globally centralized and inaccurate CSI is not addressed.

  • Multi-link cells and channel models: With up to 4 links per cell, training converges in about 25,000 time slots, including under the UMi street-canyon channel model.The 4-links-per-cell case contains 76 links, and the alternative channel model does not affect convergence rate.
  • Multi-link cells and channel models: Average sum-rate decreases as the number of links per cell increases, but the algorithm still outperforms centralized algorithms for both channel models.The comparison covers scenarios with 2 links, 4 links, and a random 1–4 links per cell.
  • Proportionally fair scheduling: For proportionally fair scheduling, the training stage converges to a desirable scheduling in about 30,000 time slots.The link weights are adjusted to enforce fairness, with performance measured by convergence to the proportional-fair objective.
  • Proportionally fair scheduling: After link-weight reinitialization, the trained DQN reaches optimal scheduling distributively as fast as centralized algorithms.A DQN trained from scratch outperforms centralized algorithms in most tested initializations, while the unmatched DQN achieves similarly high performance.
  • Practical scalability: The approach uses limited local CSI and realistic constraints, while usually outperforming generic WMMSE and FP methods that require full CSI.The proposed algorithm returns new suboptimal power allocations more quickly than the two centralized benchmarks.
  • Practical scalability: The paper uses global training; local or completely distributed training remains an open direction, and distributed training may face environment non-stationarity.An extension to inaccurate CSI measurements is also left for future work.
Loading 1808.00490v3…