Source-linked AI summary
Resource Management in Wireless Networks via Multi-Agent Deep Reinforcement Learning
Navid Naderializadeh, Jaroslaw Sydir, Meryem Simsek, Hosein Nikopour
TL;DR
The paper addresses distributed wireless resource management under interference, delayed observations, and changing network sizes. It uses MARL agents for joint user scheduling and power control with scalable observation and action spaces, and reports better rate tradeoffs than decentralized baselines, competitive performance with ITLinQ, and robustness to deployment mismatches. The authors conclude that learning-based methods should be complemented by expert optimization- and information-theoretic policies.
Problem
Wireless resource management becomes computationally complex and can produce unfair allocations, while prior learning-based methods often assume ideal messaging and lack scalability across deployment variations.
Method
The paper trains distributed MARL agents to jointly select users and control transmit power using delayed local and neighboring observations with fixed-size, scalable neural-network inputs and actions.
Results
The agents outperform decentralized baselines on the tradeoff between sum-rate and 5th percentile rate, achieve competitive or sometimes better performance than ITLinQ, and retain gains across train/test mismatches.
Takeaways & Limitations
The results support distributed MARL as a practical resource-management approach for varied wireless deployments with delayed feedback and changing network configurations.
Takeaways & Limitations
The authors state that learning-based resource management should be augmented with expert optimization- and information-theoretic policies rather than treated as an absolute replacement.
Abstract
from arXiv · showhide
We propose a mechanism for distributed resource management and interference mitigation in wireless networks using multi-agent deep reinforcement learning (RL). We equip each transmitter in the network with a deep RL agent that receives delayed observations from its associated users, while also exchanging observations with its neighboring agents, and decides on which user to serve and what transmit power to use at each scheduling interval. Our proposed framework enables agents to make decisions simultaneously and in a distributed manner, unaware of the concurrent decisions of other agents. Moreover, our design of the agents' observation and action spaces is scalable, in the sense that an agent trained on a scenario with a specific number of transmitters and users can be applied to scenarios with different numbers of transmitters and/or users. Simulation results demonstrate the superiority of our proposed approach compared to decentralized baselines in terms of the tradeoff between average and $5^{th}$ percentile user rates, while achieving performance close to, and even in certain cases outperforming, that of a centralized information-theoretic baseline. We also show that our trained agents are robust and maintain their performance gains when experiencing mismatches between train and test deployments.
I. INTRODUCTION
Ultra-dense wireless networks make radio resource management increasingly important because limited resources must support more simultaneous transmissions and mitigate interference. The paper proposes scalable, distributed MARL for joint scheduling and power control, targeting fairness and robustness under practical deployment conditions.
- Motivation: Ultra-dense networks increase simultaneous transmissions, making interference mitigation and effective radio resource management critical.The underlying resource-management problem is generally non-convex and computationally complex, especially as network size grows.
- Results: The proposed agents outperform two decentralized scheduling baselines on the tradeoff between sum-rate and 5th percentile rate, while achieving competitive performance against centralized ITLinQ.The agents also maintain performance gains across varied network configurations and train/test mismatches.
- Approach: The proposed MARL agent jointly optimizes user selection and transmit-power decisions across multiple transmitters and users.Agents make distributed scheduling decisions using observations from local and neighboring users.
- Scalability and robustness: MARL remains effective with undersampled and delayed observations caused by measurement feedback, communication, and processing delays.This addresses practical conditions that prior work often treats as ideal message passing.
- Scalability and robustness: The agent design supports fixed-size neural networks across wireless networks with different numbers of transmitters and users.The framework is intended to handle varying network sizes and densities without changing the DNN structure.
- Objective: The configurable reward balances average and 5th percentile user rates to represent cell-center and cell-edge experiences.This responds to the fairness limitations of objectives focused only on sum-throughput.
II. SYSTEM MODEL AND PROBLEM FORMULATION
The paper models distributed downlink scheduling and power control across APs serving fixed local UE pools. It optimizes the tradeoff between network sum-rate and 5th percentile user rate under interference and a challenging non-linear, combinatorial formulation.
- Wireless model: The achievable UE rate is modeled from the received signal, interference, and additive white Gaussian noise under Shannon capacity.Channel gains include constant long-term and time-varying short-term components.
- Network model: Each AP serves a fixed local pool of associated UEs, with every UE assigned to exactly one AP.User association may be refreshed periodically, for example due to mobility.
- Decision variables: At each scheduling interval, every AP selects one local UE and a transmit power between 0 and P_max.These are the paper’s two joint resource-management decisions.
- Objectives: The objectives are aggregate network sum-rate and 5th percentile rate, which respectively represent average and worst-case user performance.The 5th percentile rate is the threshold achieved by at least 95% of UEs.
- Optimization problem: The optimization seeks a Pareto front of joint scheduling and power-control sequences balancing the two naturally conflicting metrics.The problem is challenging because of its combinatorial and non-linear structure.
A. Distributed Scheduling, Feedback and Backhaul Delays
The distributed design relies on periodic UE feedback and delayed exchange of measurements among neighboring APs. These delays represent practical communication and measurement constraints in the scheduling process.
- Feedback: Each UE reports M status indicators to its associated AP every Δ_FB scheduling intervals.The reports arrive after an additional δ_FB scheduling-interval delay.
- Backhaul exchange: APs exchange the received measurements with neighboring APs through a backhaul interface with an additional δ_BH scheduling-interval delay.Figure 2 depicts the timing of UE-to-AP reporting and AP-to-AP exchange.
III. PROPOSED DEEP MARL FRAMEWORK
The proposed framework equips each AP with a deep MARL agent using local and neighboring observations to make distributed scheduling and power-control decisions. Its fixed-size observation design accommodates varying network sizes while incorporating delayed feedback and practical information exchange.
- Framework: Each AP uses a deep MARL agent to approximately solve the resource-management optimization in a distributed manner.The framework includes a centralized-training, distributed-execution paradigm.
- Caveats: Deep RL may converge only to a local optimum, so global optimality is not guaranteed in general.The framework also incurs additional complexity from using feedback.
- Observations and training: Agents observe local UE states and exchanged observations describing neighboring APs’ associated UEs.A centralized reward based on all agents’ actions trains a shared policy.
- Practical constraints: The framework explicitly handles measurement and communication delays instead of assuming instantaneous information access.Feedback undersampling and latency are treated as practical aspects of wireless resource management.
- Observation construction: Agents use weight and SINR measurements from a bounded number k of local UEs, with remote observations from a bounded number n of nearby APs.The nearest remote APs are selected because they are likely to be the strongest interferers.
- Scalability: The observation-vector dimension does not depend on the total numbers of APs or UEs, making the algorithm scalable across environment sizes.When fewer than k UEs are present, default values provide a zero-padding-like representation.
2) Actions:
Each agent jointly selects a target user and quantized transmit power, with a silent action included. The fixed action-space dimension supports deployment across networks of different sizes, while continuous power control remains future work.
- Each action jointly specifies a transmit power level and target UE for scheduling.Positive transmit powers are quantized into p levels.
- Agents restrict target-user selection to the top-k UEs represented in their local observations.This bounds the action space when an AP has many associated UEs.
- 1 + pk actions are available, including one silent action; invalid target selections are mapped to being off.
- Quantized powers support discrete-action deep RL, while continuous power control is left for future work.
- The action-space dimension is independent of network size, enabling deployment in environments with different numbers of APs or UEs.
- Across T scheduling intervals, inference complexity scales with network and action parameters including N, K, n, k, and p.
3) Rewards:
The agents receive a centralized weighted sum-rate reward that balances network sum-rate and 5th-percentile rate through λ_rew. Observation and reward normalization uses empirical percentiles and reward statistics, with penalties for jointly silent or invalid actions.
- Each agent receives a centralized weighted sum-rate reward based on all agents’ scheduling actions.The reward uses the most recently reported weight and the achieved rate of each selected UE.
- λ_rew ∈ [0, 1] controls the tradeoff between R_sum and R_5%, favoring cell-center or cell-edge users at opposite endpoints.λ_rew = 0 yields sum-rate optimization, while λ_rew = 1 approximately sums scheduled users’ PF ratios.
- If all agents remain silent, the agent whose top user has the highest PF ratio receives the negative of that ratio and the others receive zero.
- An AP selecting an invalid UE receives zero reward regardless of the other agents’ weighted sum-rate reward.
- Weight and SINR observations are normalized using empirical distributions and percentile levels before neural-network input.Figure 4 illustrates observations in blue and corresponding percentiles in orange for Q = 10 levels.
- Rewards are standardized using their dataset-estimated mean and standard deviation, producing approximately zero-mean, unit-variance outputs.
C. Training and Validation Procedure
Training uses randomized episodic wireless environments with varying AP and UE deployments and channel realizations. At each interval, agents independently choose user-scheduling and power-control actions within the simulated channel setting.
- Each episode randomly selects AP and UE locations and channel realizations subject to minimum-distance constraints.Deployment density is controlled by fixing the area and varying the numbers of APs and UEs.
- An episode contains T scheduling intervals, with agents choosing user-scheduling and power-control actions at each interval.
- The experimental section specifies the channel model, comparison baselines, deep MARL agents, simulations, and learned decision-making analysis.Simulation parameters are listed in Table I.
- The channel model combines dual-slope path loss, log-normal shadowing, and time-varying short-term Rayleigh fading.The path-loss exponents are 2 and 4 before and after a 100 m break point; shadowing has 7 dB standard deviation, and node velocity is 1 m/s.
B. User Association
Users are associated with the AP providing the highest average received power, an approach equivalent here to max-SINR association. The proposed scheduler is compared with full reuse, TDM, and centralized ITLinQ baselines.
- Max-RSRP association assigns each UE to the AP from which it receives the highest average power over time.Ties are broken arbitrarily.
- The association policy is equivalent to max-SINR association under the stated interference setting.
- The authors caution that this association policy may not be optimal in general.
- Full reuse schedules each AP’s highest-PF-ratio local UE at full transmit power.
- TDM serves users round-robin while only the AP associated with the scheduled UE transmits.
- Centralized ITLinQ selects users by PF scheduling, orders AP–UE pairs by PF ratio, and activates pairs satisfying its interference condition.The condition compares interference-to-noise ratios with the signal-to-noise ratio of each candidate link.
D. Deep MARL Agents
The paper evaluates DQN and A2C agents for distributed wireless resource management, using validation environments and a reward that balances sum-rate with 5th percentile rate. DQN ultimately achieves better coverage and score, while A2C achieves higher sum-rate.
- Agent types: DQN uses concurrent experience sampling so agents’ joint actions and correlated experiences are learned together.
- Agent types: A2C is a synchronous, policy-based actor-critic agent, whereas DQN is a value-based model-free method.
- Training and evaluation: The configurable reward balances average sum-rate against 5th percentile rate, representing cell-center and cell-edge user experiences.The score uses R_score = K + 3 × R_5%.
- Training and evaluation: DQN achieves better coverage and score than A2C and surpasses centralized ITLinQ after 12 epochs.A2C achieves a better sum-rate, while DQN converges faster because of its experience buffer.
- Observation design: Interference from the closest remote AP causes the largest SINR reduction, while farther interference terms contribute progressively less.The result supports omitting farther APs from the observation space.
F. Final Test Performance with Similar Train and Test Configurations
Final tests compare models on matched and mismatched network configurations using sum-rate and 5th percentile rate. The agents generally retain robust performance, although some cross-density deployments expose degradation as the number of agents increases.
- Matched configurations: On matched configurations, deep RL methods significantly outperform TDM on both sum-rate and 5th percentile rate.They also provide considerable 5th percentile rate gains over full reuse.
- Matched configurations: DQN’s 5th percentile rate is on par with ITLinQ for 32–40 UEs and exceeds it for 16–24 UEs.
- Matched configurations: A2C consistently performs well in sum-rate, approaching ITLinQ as the number of users increases.
- Matched configurations: With 40 UEs, both agents outperform TDM and full reuse in 5th percentile rate but underperform centralized ITLinQ as the number of APs increases.A2C outperforms ITLinQ in sum-rate for networks with 8 APs.
- Mismatched configurations: Cross-tested models are generally robust, but the DQN model trained with 4 APs loses 5th percentile performance at higher AP counts.Training with few agents may not scale well as the number of agents increases.
- Mismatched configurations: Changing observation mapping from 10 to 100 percentile levels produces results within 3% of using 20 levels.
H. Interpreting Agent’s Decisions
The interpretation analysis shows that DQN decisions depend on PF ratios, SINR, and neighboring agents’ priorities. The agent usually favors no transmission or the highest-priority user, while accounting more strongly for nearby interferers.
- Activity decisions: The agent’s activity decision follows an approximate threshold on the top UE’s PF ratio, combining weight and SINR.Higher weights or SINRs increase the chance that the AP transmits.
- User scheduling: When transmitting, scheduling among the top three UEs depends on their relative PF ratios, with the second and third favored when close to the top UE.The third UE has a substantially smaller scheduling region than the second.
- Neighbor interaction: Remote observations create a nonlinear decision boundary, and the agent gives greater consideration to the closest remote AP’s top-UE PF ratio.This reflects stronger interference between nearby APs.
- User filtering: With top-3 observations, the agent selects no transmission or the top UE most often, so observing more than three users is unlikely to improve performance in this formulation.The PF ratio combines short-term rate ability with long-term scheduling demand.
- User filtering: Using all associated UEs in arbitrary order produces sum-rate and 5th percentile rate within 6% of the sorted top-3 model.The resulting behavior resembles, but is not identical to, PF-based scheduling.
B. Multiple Power Levels
Increasing the number of discrete transmit power levels provides little additional benefit beyond binary power control. The discussion also identifies training complexity, delayed observations, and simulation-to-reality transfer as remaining boundaries.
- Multiple power levels: Increasing power levels from p + 1 = 2 to p + 1 = 4 modestly increases sum-rate, while p + 1 = 4 and p + 1 = 6 perform nearly identically.The result confirms that binary power control is already optimal in many scenarios of interest.
- Training limitations: The training procedure explores only a limited subset of environment states because it uses individual randomized snapshots with fixed environment parameters.
- Future training: Future training could vary deployment density or use curriculum learning, but measuring environment complexity and generating graded scenarios remain difficult.
- Temporal dynamics: Delayed observations are handled successfully, but recurrent or attention architectures could learn temporal dynamics from observation sequences.Using all UEs in unsorted order may be necessary, creating a variable-user-count challenge.
- Deployment boundary: The framework’s real-network applicability still requires verifying that agents trained in simulation perform as expected in deployment.