PaperlayerView exact revision on arXiv

Source-linked AI summary

Distributed Constrained Resource Management in 6G Networks: A Scalable Hybrid Model-Learning Framework

Thang X. Vu, Cuong Le, Symeon Chatzinotas, Bjorn Ottersten

arXiv:2610.11739v1cs.ITmath.IT

TL;DR

Dynamic, distributed radio resource management must handle stochastic users, interference, and QoS constraints, while centralized optimization and conventional MARL face information, scalability, or constraint-satisfaction challenges. The paper combines model-based QoS allocation with OFDM-informed decomposed cooperative MARL, reporting up to 40% throughput improvement and full constraint satisfaction. Its results are simulation-based, and the authors identify real-data fine-tuning and propagation-model mismatch as deployment concerns.

  • Problem

    Stochastic user demands, interference, and hard QoS constraints challenge distributed RRM, while conventional MARL struggles with constraint satisfaction and scalability.

    Source: §I · pp. 1–2

  • Method

    A model-based greedy phase allocates minimum QoS-satisfying resources before cooperative MARL allocates the remainder, with OFDM structure decomposing the learning problem.

    Source: §I · p. 2

  • Results

    40% system sum-rate improvement and 100% constraint satisfaction are reported while using the least number of active subchannels.

    Source: §I · p. 2

  • Takeaways & Limitations

    The combined domain-aware model-learning design is presented as a scalable and effective approach to distributed learning in wireless systems.

    Source: §VI · p. 12

    Evaluation is simulation-based; deployment calls for fine-tuning on measurement data, and propagation-model mismatch may arise in field tests.

    Source: §VI · p. 12

Abstract

from arXiv · show

Radio Resource Management (RRM) is a fundamental challenge in 6G wireless networks, particularly under dynamic user demands, inter-cell interference, and heterogeneous QoS constraints. Centralized optimization solutions are often infeasible in practice due to the lack of system-wide state information, dynamic conditions, and signaling delays, making distributed learning-based approaches attractive. However, conventional multi-agent reinforcement learning (MARL) struggles with scalability and constraint satisfaction in such highly dynamic environments. We introduce a scalable two-phase hybrid learning framework to address the RRM challenges where orthogonal-frequency division multiplexing (OFDM) domain knowledge is explicitly incorporated into the MARL pipeline. In our proposed two-phase learning framework, the first phase allocates a minimum resource to satisfy users' QoS requirements based on channel statistics, avoiding the inefficiencies of training policies under hard QoS constraints. Subsequently, a multiagent system is employed in the second phase to optimally allocate the remaining resources for system throughput maximization. By exploiting the structural interference information, we propose a scalable MARL algorithm which decomposes the original learning problem into smaller subproblems that can be handled independently, thereby avoiding exponential growth of the action space without compromising performance. Extensive simulations in realistic scenarios with 50MHz bandwidth and different numerologies show that our method significantly outperforms existing optimization and learning baselines, offering up to 40% throughput improvement, and 100% constraint satisfaction with minimal resource usage.

I. INTRODUCTION

Dynamic RRM requires distributed decisions under local observations, but stochastic users and strict QoS constraints make scalable MARL difficult. The paper combines OFDM domain knowledge with a two-phase learning framework and reports improved throughput, constraint satisfaction, and resource use.

I. INTRODUCTION

  • Centralized RRM lacks system-wide channel and user information and incurs signaling overhead and latency, while local distributed decisions cannot guarantee an exact global optimum.The cited 3GPP specifications do not allow a base station to estimate channels to other base stations’ users.Source: §I · p. 1
  • Dynamic user arrivals, departures, locations, and QoS demands compound inter-cell interference and make strict QoS satisfaction alongside system-performance maximization difficult.Standard MARL struggles to enforce hard constraints as user demands and problem sizes evolve.Source: §I · p. 2
  • The framework first uses model-based greedy allocation to satisfy QoS constraints, then cooperative MARL allocates remaining resources to maximize system sum-rate.The paper also decomposes MARL using OFDM interference structure to reduce training sample complexity.Source: §I · p. 2
  • 40% higher system sum-rate and 100% constraint satisfaction are reported while using the least number of active subchannels.The framework is evaluated in realistic scenarios with 50MHz bandwidth and various numerologies.Source: §I · p. 2

II. RELATED WORKS

  • Prior RRM learning studies face action-space growth, often restricting spectrum management or evaluating only toy configurations with few subchannels.Earlier work also restricts configurations or assumes limited actions to avoid action-space explosion.Source: §II · p. 3

III. SYSTEM MODEL AND PROBLEM FORMULATIONS

The system model considers dynamic users served by multiple base stations over shared OFDM bandwidth, with subchannel assignment and power allocation managing inter-cell interference. The formulation seeks system sum-rate subject to QoS constraints, while decentralized optimization relies on approximate interference information.

III. SYSTEM MODEL AND PROBLEM FORMULATIONS

  • Multiple base stations serve a user set that varies over time, with active users associated to non-overlapping base-station subsets.Each base station may serve up to Kmax users; new arrivals use idle slots and are denied service when all slots are occupied.Source: §III · p. 3
  • OFDM divides total bandwidth into subchannels, with subchannel assignment and power allocation used to manage inter-cell interference.The number of subchannels and subcarrier spacings vary with numerology.Source: §III · p. 3

A. Problem Formulation and Optimization-based Solution

  • The objective is to maximize system sum-rate subject to users’ QoS requirements.Source: A. Problem Formulation and… · p. 3
  • Centralized optimization is impractical because global CSI is unavailable and signaling overhead and latency are excessive.3GPP standardization allows each base station to estimate CSI only to its connected users, precluding optimal solution of the global problem.Source: A. Problem Formulation and… · p. 4
  • Decentralized base-station optimization approximates aggregated interference using interference-channel statistics and assumes uniform power allocation across base stations.The approximation ignores fast fading on interference channels and assumes all other base stations’ subchannels are active.Source: A. Problem Formulation and… · p. 4
  • The decentralized optimization approach can violate QoS constraints through interference-approximation errors and lacks cooperative inter-base-station interference mitigation.Without coordination, one base station’s selfish allocation may negatively affect other base stations’ strategies.Source: A. Problem Formulation and… · p. 4

B. Standard Constrained Dec-POMDP Formulation

The constrained RRM problem is formulated as a Dec-POMDP with each base station acting as an agent. Conventional MARL then faces a combinatorial joint action space and difficult constraint handling.

B. Standard Constrained Dec-POMDP Formulation

  • Each base station is an agent responsible for assigning its own subchannels, with the problem represented as a Dec-POMDP.The formulation defines joint state, action, transition, shared reward, and observation spaces, and a discount factor.Source: B. Standard Constrained Dec-POMDP… · p. 4
  • Each agent’s policy maps its action-observation history to a distribution over that base station’s channel assignments.The joint policy is optimized for expected discounted reward while satisfying QoS constraints.Source: B. Standard Constrained Dec-POMDP… · p. 4
  • A three-base-station, 20 MHz, 120 kHz-numerology example has about 4 billion joint actions, while Lagrangian constraint handling increases training complexity and can impede convergence.Unpredictable arrivals and heterogeneous QoS add environmental variability, alongside MARL’s nonstationarity from concurrent policy updates.Source: B. Standard Constrained Dec-POMDP… · pp. 4–5

IV. PROPOSED FRAMEWORK

The framework separates QoS satisfaction from throughput optimization. It uses a greedy first phase and decomposes the remaining-resource MARL task using OFDM interference structure.

A. Methodological Overview

  • The framework assigns minimum resources to satisfy QoS first, then uses cooperative MARL to allocate remaining resources for system throughput.Each base station observes local CSIs and long-term network information.Source: A. Methodological Overview · p. 5, Figure 1 · p. 5
  • Phase I uses a greedy procedure to satisfy user QoS requirements, preserving resources and reducing the constraint-handling burden on later learning.The procedure is described as an expert handling the difficult constraint-satisfaction task.Source: A. Methodological Overview · p. 5
  • Phase II exploits interference limited to transmissions on the same OFDM subchannel to split MARL into independent subchannel-wise QMIX problems.With QoS already satisfied, these learning problems focus on maximizing throughput.Source: A. Methodological Overview · p. 5

B. Phase I: Greedy Assignment for QoS Satisfaction

Each base station independently assigns subchannels using local channel information and interference statistics to approximate rates and meet user QoS targets.

B. Phase I: Greedy Assignment for QoS Satisfaction

  • The greedy assignment uses local channel gains, user rate requirements, and pathloss to produce a partially filled channel-assignment vector.Unassigned subchannels remain available for later allocation.Source: B. Phase I: Greedy Assignment for QoS… · pp. 5–6
  • Each base station iteratively assigns the available subchannel with the highest approximated SINR to users still below their QoS targets.Users are removed from consideration once their target is met; the process ends when all targets are met or no subchannels remain.Source: B. Phase I: Greedy Assignment for QoS… · pp. 5–6
  • ϵ ≃3% was numerically found to trade off QoS satisfaction and system sum rate, but its selection is not mathematically analyzed.Because the cumulative rate is approximate, constructive fading on interference channels can leave the true rate below the QoS target.Source: B. Phase I: Greedy Assignment for QoS… · p. 6

C. Phase II: Scalable MARL-Based Sum-rate Maximization

The section proceeds from decomposing the problem into tractable subproblems to integrating the greedy assignment output into phase-II MARL training.

C. Phase II: Scalable MARL-Based Sum-rate Maximization

1) Problem decomposition:

The framework decomposes resource allocation by OFDM subchannel, using coordinated agent groups to reduce learning complexity while preserving the original problem’s optimal actions. Its environment models bounded serving slots and uses local observations, shared rewards, and invalid-action masking to support distributed allocation.

1) Problem decomposition:

  • The problem splits into F subproblems, each coordinating N agents on one subchannel while decoupling agents across subchannels and reducing their action and state spaces.Each group shares a reward and coordinates to mitigate inter-cell interference; the resulting actions are jointly optimal for the original problem.Source: 1) Problem decomposition · p. 6, 2) Details of the learning environment · p. 7

2) Details of the learning environment:

  • Each BS has Kmax serving slots; arriving users take idle slots or are denied service when all slots are occupied, and departing users free their slots.This models a dynamic user set as resource allocation over a fixed set of slots.Source: 2) Details of the learning environment · p. 6
  • Invalid-action masking preserves first-phase assignments and restricts other agents to deactivating a subchannel or assigning it to an occupied slot.This converts the value-based constraint into a feasible-action-set constraint.Source: 2) Details of the learning environment · p. 7
  • Agents observe local channel gains and interference plus long-term network information, while each subchannel group receives a shared sum-rate reward.The observation design supports distributed deployment at each BS.Source: 2) Details of the learning environment · p. 7
  • Each BS uses F agents with action spaces of size Kmax + 1, and all agents can share one policy conditioned on BS and subchannel indices.Action 0 lets agents deactivate subchannels for interference mitigation.Source: 2) Details of the learning environment · p. 7

3) QMIX algorithm for training agents:

The framework uses QMIX for centralized training and decentralized execution: local agent values are combined into a global value function. Monotonicity supports independent local greedy choices, while shared recurrent agent networks and a state-conditioned mixing network implement the factorization.

3) QMIX algorithm for training agents:

  • QMIX factorizes a joint action-value function into local utilities and uses a global critic to capture joint dependencies during training.At execution, agents select actions using their local utility functions.Source: 3) QMIX algorithm for training agents · pp. 7–8
  • QMIX’s monotonicity condition ensures that increasing a local utility does not decrease global value, allowing local greedy actions to recover the joint greedy action.The mixing network weights are constrained to be non-negative to preserve monotonicity.Source: 3) QMIX algorithm for training agents · p. 8
  • Agent networks are DRQNs with parameter sharing, and a state-conditioned mixing network combines their values into the joint action-value function.A hypernetwork generates non-negative mixing weights and biases.Source: 3) QMIX algorithm for training agents · p. 8
  • The framework trains end-to-end with experience replay by minimizing temporal-difference loss against a one-step bootstrapped target.A slowly updated target network supplies the target value.Source: 3) QMIX algorithm for training agents · p. 8

V. SIMULATION RESULTS

The section evaluates the proposed algorithm through simulation, first describing the simulation settings and then analyzing the results.

V. SIMULATION RESULTS

  • The section evaluates the proposed algorithm’s performance through simulation.The section introduces the evaluation before presenting settings and results.Source: §V · p. 8
  • The section first describes the simulation settings.The settings are presented before the results analysis.Source: §V · p. 8
  • The section then analyzes the simulation results in detail.The analysis follows the description of the settings.Source: §V · p. 8

A. Setup

The evaluation uses a simulated downlink OFDM network with two resource scales, time-correlated fading, and randomized user and QoS conditions. GREQ is compared with decentralized optimization, constrained MARL, and heuristic allocation baselines.

A. Setup

  • 4 BSs occupy a 2000 × 2000 m area, each with 30 m height and 46 dBm maximum transmit power.The channel model uses free-space path loss and first-order Gauss–Markov Rayleigh fading.Source: A. Setup · p. 8
  • The small-scale setting supports Kmax = 5 users per BS over 20 MHz, producing 27 subchannels per BS.The large-scale setting supports Kmax = 20 over 50 MHz, producing 416 subchannels per BS.Source: A. Setup · p. 8
  • GREQ is compared against decentralized optimization (DOPT), MAPPO-Lagrange, round-robin allocation (RRS), and random allocation (RAND).MAPPO-Lagrange applies Lagrangian relaxation to QoS constraints; RRS and RAND allocate subchannels independently at each BS.Source: A. Setup · pp. 8–9
  • Episodes regenerate active users, locations, and QoS requirements; user counts are Poisson with mean λ = 0.5Kmax and QoS requirements range from 1 to 3 Mbps.The setup evaluates performance over 1800 seconds, with pedestrian mobility at 1 m/s.Source: A. Setup · pp. 8–9

B. Results on Small-scale Setting

In small-scale testing, GREQ leads in throughput across user-arrival scenarios while keeping QoS violations near zero and activating the fewest subchannels. Its training and allocation snapshots show how deterministic QoS handling and selective activation support these outcomes.

B. Results on Small-scale Setting

  • During training, GREQ rapidly improves episode reward with zero QoS violations, whereas MAPPO-Lagrange earns lower rewards and retains violations despite declining reward as its constraint focus increases.The first-phase greedy procedure offloads QoS satisfaction, leaving the second phase smaller unconstrained subproblems.Source: B. Results on Small-scale Setting · p. 9
  • The dynamic small-scale test varies arrivals from 1 to 5 users per minute, models user stays with a one-minute mean, and includes random-walk mobility.Locations are updated every second with probability 0.5 at a constant speed of 1 m/s; each test runs for 1,800 seconds.Source: B. Results on Small-scale Setting · pp. 9–10
  • Across all arrival scenarios, GREQ achieves the highest throughput, near-zero QoS violations, and the fewest activated subchannels; RRS, RAND, and DOPT have about 30% lower throughput despite using all resources.Fig. 5 compares throughput, QoS violations, and activated-subchannel share across arrival rates of 1, 3, and 5 users per minute per BS.Source: Figure 5 · p. 10, B. Results on Small-scale Setting · pp. 10–11
  • GREQ selectively deactivates subchannels to mitigate inter-BS interference, activates more as user arrivals rise, and uses less total power when fewer subchannels are active.Fig. 6 shows sample assignments at a single time step; transmit power per subchannel is fixed and uniformly allocated.Source: Figure 6 · p. 10, B. Results on Small-scale Setting · p. 11

C. Results on Large-scale Setting

In the large-scale setting, GREQ achieves high throughput with near-zero QoS violations while using fewer subchannels. The conclusions describe its two-phase OFDM-aware design and note simulation-to-field and large-network considerations.

C. Results on Large-scale Setting

  • The large-scale tests model up to 20 users per BS over 416 subchannels and extend average user activity to three minutes to represent higher load.The bandwidth is 50 MHz, and the user arrival process and mobility match the small-scale setting.Source: C. Results on Large-scale Setting · p. 11
  • GREQ has the highest throughput in most large-scale scenarios, nearly zero QoS violations, and uses 20–70% of available subchannels; RRS and DOPT activate all.The comparison covers RRS, RAND, DOPT, and GREQ; MAPPO-Lagrange is excluded because of its prohibitive action-space size.Source: C. Results on Large-scale Setting · p. 11, Figure 8 · p. 12

VI. CONCLUSIONS AND DISCUSSIONS

  • GREQ greedily assigns minimum resources for QoS, then allocates the remainder for throughput; OFDM orthogonality enables parallel subproblems that avoid action-space explosion.The conclusions describe this as a hybrid model-learning approach for distributed resource management.Source: §VI · p. 12
  • Results are simulation-only; field deployment calls for measurement-campaign fine-tuning because real propagation may produce model mismatch.Source: §VI · p. 12
  • For large networks, the paper proposes clustering BSs and applying the method independently, while each BS estimates both intra-cluster and other-cluster interference statistics.Source: §VI · p. 13
Loading 2610.11739v1…