PaperlayerView exact revision on arXiv

Source-linked AI summary

Power Allocation in Multi-User Cellular Networks: Deep Reinforcement Learning Approaches

Fan Meng, Peng Chen, Lenan Wu, Julian Cheng

arXiv:1901.07159v1cs.IT

TL;DR

The paper addresses challenging management problems with DRL-based designs and compares multiple DRL algorithms. Its simulations report that the approaches outperform state-of-the-art model-based methods, while DDPG performs best among the compared methods.

  • Problem

    Management problems are described as crucial and challenging.

  • Method

    The paper develops DRL algorithms with defined state and action designs, including DDPG.

  • Results

    The proposed approaches outperform state-of-the-art model-based methods, while DDPG performs best among the compared methods.

  • Takeaways & Limitations

    The approaches show good generalization ability, and DDPG offers stronger performance than the other compared methods.

  • Takeaways & Limitations

    The common batch operation cannot be used under distributed execution in a real scenario.

Abstract

from arXiv · show

The model-based power allocation algorithm has been investigated for decades, but it requires the mathematical models to be analytically tractable and it usually has high computational complexity. Recently, the data-driven model-free machine learning enabled approaches are being rapidly developed to obtain near-optimal performance with affordable computational complexity, and deep reinforcement learning (DRL) is regarded as of great potential for future intelligent networks. In this paper, the DRL approaches are considered for power control in multi-user wireless communication cellular networks. Considering the cross-cell cooperation, the off-line/on-line centralized training and the distributed execution, we present a mathematical analysis for the DRL-based top-level design. The concrete DRL design is further developed based on this foundation, and policy-based REINFORCE, value-based deep Q learning (DQL), actor-critic deep deterministic policy gradient (DDPG) algorithms are proposed. Simulation results show that the proposed data-driven approaches outperform the state-of-art model-based methods on sum-rate performance, with good generalization power and faster processing speed. Furthermore, the proposed DDPG outperforms the REINFORCE and DQL in terms of both sum-rate performance and robustness, and can be incorporated into existing resource allocation schemes due to its generality.

I. INTRODUCTION

Power allocation in dense multi-user cellular networks is difficult because interference, imperfect models, NP-hard optimization, and high computational complexity limit conventional methods. The paper develops DRL-based approaches with centralized training and distributed execution to address static and dynamic power allocation.

  • Motivation: Dense cellular deployments intensify interference-management and power-allocation challenges as more terminals and access points share wireless resources.The paper frames these challenges in multi-user and multi-cell communication environments.
  • Limitations of existing methods: Model-based methods assume analytically tractable models, yet hardware and channel imperfections and realistic scenarios can make such designs inaccurate or impractical.Examples include low-resolution A/D, nonlinear amplifiers, user distribution, and high computational complexity.
  • DRL background: Classic RL suffers from dimensionality and generalization issues, while value-based and policy-based DRL methods have discrete-action, quantization, exploration, convergence, or gradient-variance limitations.Actor-critic methods combine an actor that generates policies with a critic that assesses them.
  • Problem formulation: Dynamic transmit-power selection for maximizing downlink sum-rate under maximal power constraints is NP-hard in shared-frequency multi-user systems.Inter-cell interference further complicates the power-allocation problem.
  • Results: The proposed DQL outperforms standard DQL on sum-rate, while the proposed DRL approaches outperform state-of-the-art model-based methods.The study evaluates sum-rate performance, generalization ability, and computation complexity.

II. SYSTEM MODEL

The paper models cross-cell dynamic power allocation in an IMAC cellular network and formulates sum-rate maximization under transmitter power constraints. The resulting optimization is non-convex and NP-hard, motivating data-driven DRL methods.

  • The network contains N cells, each with a base station serving K users over shared frequency bands.
  • Channel gains include small-scale Rayleigh fading and large-scale fading that accounts for geometric attenuation and shadow fading.
  • The downlink SINR and rate account for intra-cell interference, inter-cell interference, and additional noise power.
  • The objective is to choose transmitter powers under maximum-power constraints to maximize the network sum-rate.
  • The formulated problem is non-convex and NP-hard, while model-based methods face difficult-to-quantify optimality gaps and high computational complexity.

III. DEEP REINFORCEMENT LEARNING

The paper recasts cellular power allocation in an MDP framework and analyzes when cumulative-reward optimization reduces to single-step reward maximization. This analysis motivates tailored DRL designs, including setting γ = 0 for the considered scenario.

  • The MDP framework represents states, actions, transition probabilities, and immediate rewards for agent-environment interaction.
  • The value functions quantify finite-horizon and γ-discounted cumulative rewards, while Q functions evaluate state-action pairs.
  • Under action-independent transitions and instantaneous action-dependent rewards, the optimal cumulative-reward policy is equivalent to combining single-step rewards.
  • Because the channel follows a first-order Markov process, the paper states that DRL optimization is equivalent to the corresponding single-step formulation.
  • The paper reports that increasing γ negatively affects DQN sum-rate performance and suggests γ = 0 or T = 1 in this scenario.
  • The resulting design guidance sets the Q function equal to the reward function for the subsequent DRL algorithms.

B. Centralized Training & Distributed Execution

A centralized agent can coordinate network-wide power allocation but becomes difficult to train and deploy as the number of cells grows. The paper therefore uses centralized training with distributed execution and shared policies.

  • The centralized framework trains one center agent that uses network information and broadcasts allocated powers to transmitters.
  • The centralized DNN suffers space explosion because its input-output cardinalities scale with cell number and the state-action space grows exponentially.
  • Full CSI delivery and allocation broadcasting create latency pressure when the network is large or low-latency service is required.
  • Distributed execution treats each transmitter link as an agent operating synchronously with partial channel information and producing its own power.
  • To reduce multi-agent complexity, agents share one policy learned from data across links, yielding centralized training and distributed execution.

C. On-line Training

The paper combines off-line pretraining with selective on-line adaptation to balance training cost and responsiveness to changing wireless environments. An environment-tracking mechanism triggers updates when the learned model no longer accurately reflects current conditions.

  • Off-line pretraining and transfer learning reduce on-line training stress, while initial deployment still requires on-line adaptation to real-network imperfections and unknown issues.
  • Continuous on-line training wastes resources, requires real-environment exploration, can reduce communication performance, and demands expensive computing hardware.
  • Training is unnecessary under negligible environmental fluctuation but cannot respond promptly to sudden changes.
  • The proposed environment-tracking mechanism dynamically controls agent training to balance performance and efficiency.
  • A changing environment alters the reward function, requiring corresponding adjustment of the policy or Q function.
  • The normalized critic-loss index is monitored over an observation window, and DNN training starts when it reaches the threshold lmax.

IV. DRL ALGORITHM DESIGN

The section designs DRL state and reward representations for scalable cellular power control. It selects informative features, normalizes channel information, and localizes rewards to reduce learning difficulty while preserving sum-rate relevance.

  • State: Logarithmic normalization is preferred because channel amplitudes can vary by orders of magnitude.Channel amplitudes are normalized by the downlink quantity d_l_n,k.
  • State: Sorting selects the first y elements from x in decreasing order, retaining the strongest or most relevant environment components.The selected component indices are denoted by i, reducing input dimension.
  • State: Irrelevant or weakly correlated inputs consume computation and can degrade performance, while auxiliary information can improve DNN sum-rate performance.
  • State: The DRL state uses partially observed features f1 or f2, with input dimensions 2Ic and 3Ic.
  • Reward: The localized reward function is proposed because irrelevant remote-cell rates enlarge reward variance and make large-network training difficult.The sum of local rewards is proportional to sum-rate when the cell number N is sufficiently large.

3) Action:

The action design distinguishes discrete power control for REINFORCE and DQL from continuous power control for DDPG. DDPG avoids discretization error but uses a more complex continuous-action setup.

  • Action: Downlink power is a non-negative continuous scalar bounded by maximum power Pmax.
  • Action: REINFORCE and DQL require finite action spaces, so emitting power is quantized into |A| levels.Discretization of a continuous variable results in quantization error.
  • Action: DDPG directly outputs a deterministic continuous action through a scaled sigmoid function bounded by [0, Pmax].The scaled sigmoid eliminates quantization error.
  • Action: For multi-action tasks, DDPG output dimension is NA, whereas REINFORCE and DQL action-space size grows exponentially, making such tasks impractical.
  • Training: Experience Replay is optional here because temporal data correlation is not strong in the investigated problem.
  • REINFORCE: REINFORCE learns a stochastic policy by Monte-Carlo policy gradients, but is susceptible to reward scaling during practical training.Whitening rewards before gradient computation alleviates this dependency.

C. Value-based: DQL

DQL is a value-based off-policy DRL method that estimates action values with a DQN and selects actions using value maximization with dynamic ε-greedy exploration.

  • Architecture: DQL uses the same topology as REINFORCE while replacing direct stochastic-policy generation with DQN-based value estimation.
  • Value estimation: DQL estimates action values with a DQN Q(s, a; θq), then selects the action maximizing the Q value.
  • Exploration: Dynamic ε-greedy exploration controls the exploration probability during training.The exploration probability changes from an initial value ε1 toward a final value εNe over Ne episodes.

D. Actor-Critic: DDPG

DDPG combines a deterministic actor with a critic for continuous power-control actions. Its semi-model-free critic incorporates prior knowledge while retaining neural-network flexibility for training.

  • Design: DDPG is an actor-critic, model-free algorithm based on deterministic policy gradients for continuous action spaces.
  • Actor and critic: The actor maps state s to a deterministic action, while the critic evaluates an action-state pair and transfers learning gradients.
  • Actor and critic: Actor and critic cooperate: the actor maximizes critic evaluation, while the critic seeks accurate assessment through differentiable optimization.
  • Action: The exploration action adds noise to the actor output, with actions bounded by [0, Pmax].
  • Critic design: A semi-model-free critic combines prior knowledge with neural-network flexibility because purely model-based actors cannot accommodate unknown online-training issues.
  • Simulation configuration: The evaluation uses N = 25 cells, |Dn| = 18 adjacent cells, Ic = 16 retained interferers, and |A| = 10 power levels.
  • Simulation configuration: Training repeats each algorithm design 50 times, with testing over 500 generated scenarios.

B. DRL Algorithm Comparison

The experiments compare REINFORCE, DQL, and DDPG using variance, average sum-rate, and top-20% average sum-rate. DDPG achieves the strongest combination of sum-rate performance and robustness, while experience replay mainly improves stability-related measures.

  • Experience replay: Experience replay reduces sum-rate variance and improves average sum-rate, but has negligible influence on the average best sum-rate.
  • Robustness: REINFORCE remains difficult to stabilize even with experience replay and normalization, whereas DQL is much more stable.
  • Robustness: The sum-rate variance of DDPG is the lowest, reaching one or more orders of magnitude below REINFORCE.
  • Algorithm comparison: DDPG achieves the highest average best sum-rate and outperforms REINFORCE and DQL in both sum-rate performance and robustness.Its sum-rate variance is also substantially lower than REINFORCE’s.
  • Algorithm comparison: REINFORCE and DQL obtain nearly identical best-result performance, with REINFORCE slightly better but less stable.

2) Feature Engineering:

The experiments examine feature design, action-space quantization, and generalization across changing cellular conditions. DDPG consistently provides the strongest sum-rate performance, while richer features improve results at additional training and input costs.

  • Feature Engineering:: Adding the additional feature improves sum-rate performance, especially the average best sum-rate of DDPG.The improvement requires enlarged input dimensions and more training time.
  • Feature Engineering:: A simplified feature state remains meaningful for online training when data and computational resources are restricted.
  • Quantization Error: REINFORCE and DQL best results rise slightly as |A| increases from 3 to 10, but larger output dimensions do not improve performance.
  • Quantization Error: 1.54 bps to 1.19 bps: REINFORCE’s average best sum-rate declines sharply as |A| increases from 14 to 40.The authors associate large action spaces with practical training difficulty, especially for REINFORCE.
  • Quantization Error: DDPG avoids action-space discretization by nature and outperforms both DQL and REINFORCE under quantization-error comparisons.
  • Generalization: The data-driven algorithms outperform random power, maximum power, FP, and WMMSE across the reported cell-range experiments.WMMSE performs slightly better than FP, while random and maximum power are lowest.
  • Generalization: Across varying cell ranges and user densities, the learned data-driven models retain good generalization, with DDPG achieving the highest sum-rate.Average sum-rate decreases as interference strengthens at smaller cell ranges and as users become denser.

3) Doppler frequency:

The paper evaluates robustness to Doppler variation and execution cost, then summarizes the broader DRL design. Data-driven models remain robust across the tested Doppler range, while distributed execution supports low computational cost and DDPG performs best overall.

  • 3) Doppler frequency:: Fast fading can degrade the proposed data-driven models because current power allocation uses information from the last instant.The model-driven algorithms are not influenced by f_d by nature.
  • 3) Doppler frequency:: Average sum-rates of the data-driven algorithms drop slowly as Doppler frequency increases from 4 Hz to 18 Hz, indicating robustness to f_d.
  • Computation complexity: GPU execution can take longer than CPU execution because small-scale DNNs and distributed execution do not fully utilize the GPU.
  • Computation complexity: On CPU, the three DRLs are about 15.5 times faster than FP and 61.0 times faster than WMMSE.Their execution times are nearly identical because they use similar DNN models.
  • Computation complexity: Distributed execution keeps time expense constant as the total user count NK increases, at the cost of additional calculation devices.
  • Conclusions & Discussions: The proposed design analyzes inter-cell cooperation, centralized offline/online training, and distributed execution before introducing concrete DRL algorithms.
  • Conclusions & Discussions: DDPG eliminates quantization error and outperforms DQL and REINFORCE in the simulations, consistent with the theoretical comparison.
  • Conclusions & Discussions: The proposed DDPG can address discrete or continuous state/action spaces and joint optimization problems involving multiple variables.The paper names user scheduling, channel management, and power allocation as example applications.
Loading 1901.07159v1…