Dynamic SDN-based Radio Access Network Slicing with Deep Reinforcement Learning for URLLC and eMBB Services

I. INTRODUCTION

• The proposed two-level allocation gives the SDN controller a large-timescale role and gNodeBs a short-timescale role, with adjacent-gNodeB borrowing when local resources are insufficient.The design responds to centralized-allocation signaling overhead and to lower-level allocation schemes that may wait for an update or solicit the controller when resources run short.Source: §I · p. 2
• The optimization maximizes total achievable eMBB and URLLC data rate subject to URLLC latency and eMBB minimum-rate requirements.The authors formulate the global resource-block allocation as a nonlinear binary program and study its NP-hardness.Source: §I · p. 2
• The proposed single-agent multi-agent reinforcement-learning framework is presented as dynamic, adapting allocation policies to network and service conditions under SDN supervision.The stated factors include user density, eMBB and URLLC requirements, and wireless-channel transmission conditions.Source: §I · p. 2
• The paper models the allocation levels as Markov decision processes, using a single-agent approach for SDN allocation and multi-agent deep Q-learning for gNodeB allocation.The SDN controller partitions resource blocks among gNodeBs; each gNodeB then allocates its pre-allocated resources to end-users and can share resources with neighboring gNodeBs.Source: §I · p. 2
• The authors report extensive simulations to evaluate the mechanism against benchmark algorithms and describe the proposed framework as superior.The introduction does not provide quantitative comparison results.Source: §I · p. 2

III. SYSTEM MODEL

• The model includes URLLC and eMBB slices, a shared resource-block pool, and users each served by one gNodeB and belonging to one slice.It assumes downlink OFDMA, with transmissions scheduled orthogonally.Source: §III · p. 3
• The SDN controller assigns resource blocks to gNodeBs at a large timescale; each gNodeB schedules its pre-allocated blocks to users at a short timescale.A gNodeB may borrow an unallocated block from another gNodeB only after using its pre-allocated blocks.Source: §III · pp. 3–4
• The model supports multiple 5G NR numerologies when an appropriate trained allocation model is available for each selected subcarrier spacing.The gNodeB selects the numerology based on channel-state information; the authors assume near-perfect channel-state knowledge, justified for slowly varying channels and low user mobility.Source: §III · pp. 3–4
• The model evaluates user rates using bandwidth, transmit power, channel gain, and additive white Gaussian noise, with equal bandwidth and downlink power across resource blocks.The rate definition applies to URLLC and eMBB users.Source: §III · p. 4
• Packet arrivals are modeled as Poisson with independent exponential inter-arrival times, yielding an M/M/1 queue for average-delay calculations.The authors note that bursty traffic may violate this assumption and make the mathematical analysis more complicated.Source: §III · p. 4

A. Problem Formulation

• The objective maximizes the total sum data rate of URLLC and eMBB users while meeting their QoS requirements.The question is how to allocate resource blocks at both the controller and gNodeB levels.Source: A. Problem Formulation · p. 4
• Each user’s allocation is capped at Kmax, and each resource block can be assigned to only one user at a time.The constraints also allow borrowing only after a gNodeB has used all its pre-allocated blocks.Source: A. Problem Formulation · p. 5
• eMBB users must meet a minimum data-rate threshold Rmin, while URLLC users must remain below a maximum delay threshold Dmax.These constraints encode the two slices’ stated QoS requirements.Source: A. Problem Formulation · p. 5
• The per-user resource cap and borrowing rule represent limited resources and constrain how remaining blocks can be allocated.A gNodeB can borrow from others only once its own SDN-allocated resources are in use.Source: A. Problem Formulation · p. 5
• The allocation problem is a nonlinear binary program because its assignment variables are binary and its delay expression is nonlinear.The authors describe solving it generally as very challenging.Source: A. Problem Formulation · p. 5
• Because optimization variables are binary and user delay is nonlinear, the RB allocation problem is a nonlinear binary program, challenging to solve generally; the authors say they study its NP-hardness.Source: A. Problem Formulation · p. 5

B. NP-hardness

• The proof reduces the NP-hard 0-1 knapsack problem to a restricted instance of the allocation problem.The restriction has one gNodeB, only eMBB users, and a known pre-allocation of resource blocks.Source: B. NP-hardness · p. 5
• The restricted allocation problem therefore corresponds to selecting eMBB users to maximize total data rate under resource, exclusivity, and minimum-rate constraints.The allocation formulation also caps the number of blocks and assigns each block to at most one user.Source: B. NP-hardness · p. 5
• Each eMBB user becomes an item, its achievable data rate is the item’s profit, its allocated resource blocks are its weight, and the gNodeB’s pre-allocated blocks are capacity.The capacity bounds the total blocks allocated to eMBB users.Source: B. NP-hardness · p. 5
• Because the reduction is polynomial-time and 0-1 knapsack is NP-hard, the authors conclude that the restricted problem, and thus the general problem, is NP-hard.The theorem states that P is NP-hard.Source: B. NP-hardness · p. 5

V. SINGLE-AGENT MULTI-AGENT REINFORCEMENT LEARNING BASED RAN RESOURCE SLICING

• SAMA-RL applies reinforcement learning to the two-level resource-block allocation problem, which the authors describe as challenging to solve optimally.The framework covers the controller and gNodeB allocation levels.Source: §V · p. 5
• The SDN controller uses EXP3 to allocate resource blocks to gNodeBs, while each gNodeB uses an independent DQL agent to schedule blocks and share unexploited resources when needed.The controller’s state and action spaces are described as small enough to learn in reasonable time; the gNodeB level’s larger state space motivates distributed deep RL.Source: §V · pp. 5–6
• Figure 2 depicts single-agent and multi-agent interaction with the MDP environment.The framework models each allocation level as an MDP.Source: §V · pp. 5–6, Figure 2 · p. 6

A. MDP formulation of the SDN allocation level

• The SDN allocation-level MDP is defined by state, action, and reward spaces.The central agent’s state includes previously allocated resource blocks for each gNodeB and the sets of gNodeBs and resource blocks.Source: A. MDP formulation of the SDN… · p. 6, 1) The state space · p. 6

2) The action space:

• The central agent selects RB subsets for gNodeBs, with each RB assigned to only one gNodeB per allocation.The action is represented as a vector whose elements indicate whether an RB is assigned to a gNodeB.Source: 2) The action space · p. 6

3) The reward function:

• The central agent’s reward is the network’s total sum-rate, evaluated after gNodeBs allocate their assigned RBs to end-users.The controller waits for the second-level allocation results, so the two allocation levels are performed jointly.Source: 3) The reward function · p. 6

B. MDP formulation of the gNodeB allocation level

• The gNodeB-level allocation problem is modeled as an MDP with state, action, and reward components for each agent.Source: B. MDP formulation of the gNodeB… · p. 6

1) The state space:

• Each agent’s state includes its associated users, pre-allocated RBs, channel gains, and eMBB rate and URLLC delay thresholds.The controller also informs agents about RBs allocated to the other agents.Source: 1) The state space · p. 7

2) The action space:

• Each gNodeB assigns its pre-allocated RBs to associated users and may request additional RBs; agents communicate actions to coordinate sharing.The action representation encodes RB-to-user assignments, and the action space applies allocation constraints.Source: 2) The action space · p. 7

3) The reward function:

• An agent’s reward relates to total sum-rate subject to URLLC delay and eMBB minimum-rate requirements, while infeasible allocations receive -1.Feasibility includes checking whether borrowed RBs are already used by their owner or selected by another agent.Source: 3) The reward function · p. 7

• EXP3 models the SDN controller as a multi-armed bandit player, with each controller action treated as an arm.Each action has a weight that reflects its performance for the controller.Source: C. Single-agent EXP3 algorithm · p. 7
• At each round, EXP3 samples an action according to probabilities derived from its weights, then uses the received reward to update them exponentially.The exploration parameter controls the tendency to choose actions uniformly at random.Source: C. Single-agent EXP3 algorithm · p. 8
• EXP3 increases the probability of higher-performing actions and reduces the probability of lower-performing ones.Source: C. Single-agent EXP3 algorithm · p. 8
• The controller’s reward is the total data rate reported by gNodeBs after they allocate RBs to end-users.The reward is scaled and estimated to compensate for a potentially low probability of observing it before action weights are updated.Source: C. Single-agent EXP3 algorithm · p. 8

D. Multi-agent deep Q-Learning algorithm

• Each gNodeB agent uses deep Q-learning, which approximates the Q-function with a deep neural network, to select actions from its observed state.The paper’s multi-agent approach has an offline training phase and an online implementation phase.Source: D. Multi-agent deep Q-Learning algorithm · p. 8

1) The training phase of DQL:

• Each agent trains a DDQN using replayed state transitions, actions, and rewards, with a main network estimating Q-values and a target network supplying target values.The DDQN separates action selection from evaluation to mitigate overestimation of action Q-values.Source: 1) The training phase of DQL · pp. 8–9
• During training, agents use an ϵ-greedy policy to explore or select the highest-Q-value action, and ϵ decays as learning proceeds.Source: 1) The training phase of DQL · p. 9
• Training is performed offline because DDQN learning is computationally heavy, and datasets can cover different network topologies and channel conditions.Source: 1) The training phase of DQL · p. 9

2) The implementation phase of DQL:

• In online implementation, each agent selects the action with the highest Q-value for its observed state, then agents form a joint action that yields an RB allocation.The implementation phase uses the agents’ trained DDQNs.Source: 2) The implementation phase of DQL · pp. 9–10

VI. SIMULATION RESULTS

• The simulation-results section investigates the proposed two-level RB allocation mechanism through several simulated scenarios.Source: §VI · p. 10

A. Experiment scenarios and setup

• The simulated SDN-enabled RAN places gNodeBs and uniformly distributed end-users across a 1 km^2 area, with each user associated with one gNodeB.The DDQN setup uses two 256-neuron hidden layers, ReLU, Adam at a 0.001 learning rate, and target-network updates every 1000 steps; users have low mobility.Source: A. Experiment scenarios and setup · p. 10

B. DDQN training results

• Agents’ cumulative average rewards improve as training episodes increase.The reward trajectories are reported for agents per episode.Source: Figure 3 · p. 10, B. DDQN training results · p. 10
• Average agent loss decreases with training and converges to a minimum after episode 2500, which the authors say demonstrates accurate Q-value approximation.The loss is plotted against episodes.Source: B. DDQN training results · pp. 10–11, Figure 4 · p. 11
• The reported reward and loss convergence assumes gNodeBs have perfect channel-state information; imperfect information may increase DDQN convergence time.After offline training, the learned policy can be applied rapidly to obtain a resource-allocation solution.Source: B. DDQN training results · p. 11

C. SAMA-RL performance evaluation

• The benchmarks are adapted to the system model, and comparisons are averaged over 1000 trials.3-SRA uses hierarchical reservation and DRL adjustment; 1-SRA uses a greedy single-stage heuristic that prioritizes URLLC users.Source: C. SAMA-RL performance evaluation · p. 11
• SAMA-RL achieves the best objective-function performance as the number of end-users rises and for all tested minimum data-rate thresholds.The performance gap over both benchmarks grows as the minimum data-rate threshold increases.Source: Figure 5 · p. 11, C. SAMA-RL performance evaluation · pp. 11–12, Figure 6 · p. 12
• For all three approaches, the objective value decreases as user count rises, while SAMA-RL’s per-user RB allocation better supports total sum data rates than 3-SRA’s slice-level allocation.The paper attributes the decline to increased competition for RBs; 1-SRA’s highest-SNR-only allocation may also poorly estimate the RBs needed for QoS.Source: C. SAMA-RL performance evaluation · p. 12
• SAMA-RL serves more users meeting the minimum data-rate threshold and more users meeting the maximum-delay threshold than both benchmarks across the tested thresholds.3-SRA may wait for a resource-reservation update when slice resources are insufficient; SAMA-RL can request RBs from another gNodeB.Source: Figure 7 · p. 12, C. SAMA-RL performance evaluation · p. 12, Figure 8 · p. 12

• Simulations report that the two-level mechanism outperformed a benchmark while ensuring URLLC and eMBB data-rate and delay requirements.Source: §VII · p. 13
• The SDN level allocates RBs to gNodeBs at a large time scale, while gNodeBs allocate to users at a short time scale and can borrow RBs from other gNodeBs.Borrowing is available when pre-allocated RBs are insufficient.Source: §VII · pp. 12–13
• The authors formulate a data-rate maximization problem subject to URLLC ultra-low-latency and eMBB minimum-data-rate requirements, and prove it NP-hard.Source: §VII · p. 13
• They model SDN allocation as a single MDP solved with EXP3 and gNodeB allocation as a multi-agent MDP solved with DQL.The DQL approach uses double DQN and replay memory.Source: §VII · p. 13