PaperlayerView exact revision on arXiv

Source-linked AI summary

Agentic Quantum Deep Reinforcement Learning for RAN Slicing

Tingnan Bao, Medhat Elsayed, Pedro Enrique Iturria-Rivera, Yigit Ozcan, Majid Bavand, Melike Erol-Kantarci

arXiv:2608.30206v1eess.SP

TL;DR

The paper addresses how to maximize eMBB throughput in downlink URLLC/eMBB RAN slicing while satisfying URLLC delay-reliability and resource constraints. It proposes Agentic-QDRL, combining slow PMAR slice control with fast VQC-based PRB scheduling and safety mechanisms; simulations report higher feasible throughput and maintained URLLC reliability versus baselines.

  • Problem

    The central problem is maximizing queue-aware long-term eMBB throughput while satisfying URLLC delay-violation, PRB exclusivity, and slice-budget constraints.

  • Method

    Agentic-QDRL couples a slow PMAR budget controller with a fast compact VQC-based QDRL scheduler, using feasibility projection and safety fallback for constrained scheduling.

  • Results

    About 107 Mbps versus 60 Mbps at λe = 8000, with both learned methods keeping URLLC violation rates below ε across all loads.

  • Takeaways & Limitations

    Agentic-QDRL improves the practical URLLC delay-reliability–eMBB throughput tradeoff while reducing eMBB queue buildup and maintaining URLLC reliability.

Abstract

from arXiv · show

Radio access network (RAN) slicing enables ultra-reliable low-latency communications (URLLC) and enhanced mobile broadband (eMBB) services to share radio resources, but their requirements create a challenging reliability--throughput tradeoff. URLLC requires low-latency and reliable packet delivery, whereas eMBB targets high sustained throughput. This paper considers downlink URLLC/eMBB RAN slicing and formulates it as a queue-aware long-term eMBB throughput maximization problem subject to URLLC delay-violation, physical resource block (PRB) exclusivity, and slice-budget constraints. To solve this problem, we propose agentic quantum deep reinforcement learning (Agentic-QDRL), a two-time-scale framework that combines agentic slice-level resource control with quantum-enhanced PRB scheduling. At the slow time scale, a perceive--memory--act--reflect (PMAR) controller adapts the resource shares of URLLC and eMBB slices. At the fast time scale, a compact variational quantum circuit (VQC)-based QDRL scheduler performs PRB allocation under the current slice configuration. A feasibility projection and a safety fallback mechanism is further introduced to satisfy scheduling constraints and reduce URLLC deadline violations. Simulation results under different eMBB traffic loads show that Agentic-QDRL improves eMBB throughput, reduces eMBB queue buildup, and maintains URLLC delay reliability compared with classical DRL and heuristic baselines.

I. INTRODUCTION

The paper frames URLLC/eMBB RAN slicing as a constrained, queue-aware resource-allocation problem and proposes Agentic-QDRL to coordinate slice budgets with PRB scheduling. The framework combines slow-timescale agentic control, fast quantum-enhanced scheduling, and safety mechanisms, with simulations reporting improved eMBB performance while preserving URLLC reliability.

  • Motivation: URLLC/eMBB coexistence creates a reliability–throughput tradeoff because URLLC requires low-latency reliable delivery, while eMBB requires sustained throughput.
  • Problem formulation: The paper formulates queue-aware long-term eMBB throughput maximization subject to URLLC delay-violation, PRB exclusivity, and slice-budget constraints.
  • Proposed framework: Agentic-QDRL uses two timescales: a PMAR controller adapts URLLC/eMBB resource shares, while a compact VQC-based QDRL scheduler allocates PRBs under current budgets.
  • Safety mechanisms: Feasibility projection and safety fallback enforce PRB exclusivity and slice-budget constraints while reducing URLLC deadline violations.
  • Evaluation: Simulations across eMBB traffic loads and ablations report improved eMBB throughput while maintaining URLLC delay reliability against classical DRL and heuristic baselines.

A. Channel and Service Model

The channel and service model combines path loss, shadowing, Rician small-scale fading, fixed per-PRB transmit power, and AWGN to characterize downlink service. Achievable PRB rates are aggregated into scheduled rates and potential service bits.

  • Channel model: The large-scale channel gain uses a log-distance path-loss model with distance, a path-loss exponent, and a shadowing term.
  • Channel model: Small-scale fading follows a Rician model combining a deterministic line-of-sight component with a random non-line-of-sight component.
  • Signal model: The received signal model uses equal transmit power across PRBs, unit-power transmitted symbols, and additive white Gaussian noise.
  • Service model: The model derives an achievable rate for each PRB and aggregates scheduled rates into potential service bits for each user.

B. Traffic and Queuing Model

The model represents downlink traffic with UE-level queues, separate URLLC and eMBB arrival processes, and service metrics that capture backlog-limited delivery, delay reliability, and sustained throughput.

  • Each UE has a queue storing data awaiting downlink transmission, with served bits limited by the available backlog.The queue length is measured at the beginning of each scheduling interval.
  • Service removes backlog before newly admitted traffic enters the queue, so new arrivals begin with zero waiting time in the next state.The queue evolution uses a post-service arrival convention.
  • URLLC arrivals are modeled as bursty packet generation using a Bernoulli process, parameterized by packet size and arrival probability.
  • eMBB arrivals use a Poisson bit-arrival model to approximate persistent downlink traffic, with λe denoting mean traffic volume per UE per interval.
  • URLLC reliability is deadline-based: a packet succeeds only when delivered before its latency deadline, tracked through HoL delay and delay-violation indicators.Residual backlog keeps the HoL state active and increases its delay by ∆t; otherwise, the HoL state resets to zero.
  • eMBB performance is represented by an instantaneous throughput utility and long-term expected throughput, while URLLC performance uses long-term expected delay-violation rate.

D. Problem Formulation

The problem jointly optimizes slice-level PRB budgets and UE scheduling to maximize long-term eMBB throughput under URLLC reliability and allocation constraints. The proposed framework separates slow budget adaptation from fast feasible scheduling in a feedback loop.

  • D. Problem Formulation: The optimization chooses slice-level PRB budgets and scheduling-interval PRB-to-UE assignments to maximize long-term eMBB throughput while satisfying URLLC delay reliability.
  • D. Problem Formulation: Constraint C1 enforces the maximum URLLC delay-violation rate, while C2 enforces OFDMA orthogonality so each PRB serves at most one UE per interval.
  • D. Problem Formulation: Constraints C3 and C4 enforce total and per-slice PRB-budget feasibility together with binary PRB-to-UE scheduling decisions.
  • D. Problem Formulation: The problem is NP-hard because it contains 0-1 knapsack as a deterministic special case.
  • D. Problem Formulation: The knapsack reduction selects eMBB transmission options with values au and PRB costs cu under the eMBB budget Be.
  • D. Problem Formulation: Agentic-QDRL addresses the resulting scalability challenge while enforcing hard PRB-assignment and slice-budget constraints.
  • III. PROPOSED AGENTIC-QDRL FRAMEWORK: The framework uses two time scales: slow slice-resource adaptation and fast PRB scheduling, connected through service and queue feedback.
  • A. Agentic PMAR Budget Control: The PMAR controller regulates the resource envelope rather than individual PRBs, updating URLLC/eMBB budgets after aggregating interval-level scheduling outcomes.

3) Act:

The slow controller updates slice-level PRB budgets using reliability, fallback, nominal-violation, and eMBB-backlog feedback. It enforces a load- and safety-aware minimum for URLLC resources while allocating residual resources to eMBB.

  • The controller extracts reliability, fallback, and backlog feedback from the current slicing interval and episodic memory to update slice-level PRB budgets.
  • The composite budget error increases the URLLC budget after reliability violations, frequent fallback, or nominal violations hidden by safety correction.Its control gains weight fallback usage, nominal-realized violation gaps, and eMBB backlog pressure.
  • Persistent eMBB backlog above Qth decreases the URLLC budget through a backlog-pressure regularizer.
  • An adaptive lower projection bound prevents the URLLC slice from falling below a load- and safety-aware minimum.The bound incorporates offered URLLC traffic and recent fallback or nominal-realized delay-violation feedback.
  • The eMBB budget receives the residual resources after the URLLC budget update.

4) Reflect:

The fast scheduler maps compact RAN observations and slice budgets to UE priority scores using a VQC-based actor. Its quantum circuit encodes compressed features, applies trainable rotations and entanglement, and supports NISQ-oriented compact scheduling.

  • The scheduler maps current RAN observations and slice budgets to UE priority scores at each scheduling interval.A classical encoder compresses the observation to match the available number of qubits.
  • The VQC uses bounded angle encoding, trainable single-qubit rotations, ring CNOT entanglement, and repeated feature re-uploading.The circuit is described as a compact quantum analog of a neural network suitable for NISQ devices.
  • The compressed feature vector is encoded into qubit rotation angles before variational processing.The encoding initializes an n-qubit register in |0⟩⊗n and applies Rx rotations.
  • The VQC architecture used by the QDRL scheduler is illustrated in Fig. 3.

4) Measurement and Classical Head:

The VQC’s quantum measurements are converted by a classical output head into UE priority scores, with Gaussian exploration applied during training.

  • Pauli-Z expectation values form the VQC measurement vector used for classical score generation.
  • A classical output head maps the measurement vector to UE priority scores under the current observation and slice budgets.The score z_u(t) represents the scheduler’s preference for UE u.
  • Gaussian exploration is added in score space during training, and the exploratory scores feed the feasibility projection.

C. Feasibility Projection and Safety Fallback

Agentic-QDRL post-processes continuous UE scores into feasible binary PRB schedules and invokes a deterministic safety fallback when URLLC delays approach unsafe levels.

  • The safety fallback replaces the nominal schedule when URLLC head-of-line delays approach the latency deadline.The fallback is a deterministic safety layer around the learned scheduler.
  • The feasibility projection converts continuous UE preferences into a binary nominal allocation satisfying PRB exclusivity and slice-budget constraints.It partitions PRBs into disjoint URLLC and eMBB budget sets and selects at most one UE per PRB.
  • Within each slice, PRBs are assigned using the largest score-rate product, with backlogged URLLC users prioritized when present.Unused URLLC PRBs are released to eMBB when URLLC queues are empty.
  • The fallback uses earliest-deadline-first scheduling for URLLC and proportional-fair scheduling for eMBB.The eMBB rule uses moving-average served throughput in its scheduling decision.
  • The learned policy handles normal scheduling, while fallback explicitly protects URLLC when the operating point becomes unsafe.

D. Learning Algorithm

The learning procedure couples slow PMAR budget adaptation with fast VQC-based score scheduling, using feasibility projection, fallback execution, and actor–critic updates.

  • Two-time-scale learning: The two-time-scale procedure fixes budgets and safety thresholds within each slicing interval while QDRL generates feasible PRB schedules and PMAR updates them afterward.The scheduler collects KPIs and replay transitions for the subsequent PMAR update.
  • Safety-aware learning: Fallback-triggered transitions evaluate throughput and delay counterfactually under the nominal schedule while penalizing unsafe actions in the learning reward.Executed fallback schedules protect operation, whereas nominal schedules provide the counterfactual learning signal.
  • Actor–critic updates: The critic minimizes squared temporal-difference error using a slowly updated target critic, while the actor follows score-space Gaussian policy-gradient ascent.The critic uses discounting and target-network smoothing; the actor parameters include the encoder, VQC, and output head.
  • Quantum actor: The hybrid actor produces score vectors through a VQC and classical output head, with VQC derivatives computed by the parameter-shift rule for NISQ-compatible training.The parameter-shift rule estimates each circuit derivative using positively and negatively shifted gate parameters.
  • Algorithm flow: Algorithm 1 alternates PMAR budget updates with per-interval observation, VQC scoring, exploratory actions, feasibility projection, fallback decisions, and replay updates.The PMAR state remains fixed throughout each slicing interval.

IV. SIMULATION RESULTS

Simulations evaluate Agentic-QDRL in a single-cell OFDMA slicing scenario across eMBB loads and against classical, heuristic, and ablation baselines. The results show faster convergence, lower eMBB backlog, and stronger SLA-feasible throughput while preserving URLLC reliability.

  • Setup: The simulations use a single-cell downlink OFDMA RAN with 52 PRBs, 32 URLLC users, and 15 eMBB users across four eMBB loads.The tested loads are λe ∈ {500, 2000, 4000, 8000} bits per scheduling interval; the quantum actor uses a 6-qubit, 2-layer VQC.
  • Convergence: Agentic-QDRL converges within the first few tens of slicing windows, whereas Agentic-DRL requires several hundred windows to reach a comparable level.Agentic-QDRL also achieves stronger steady-state eMBB throughput, especially at the higher load.
  • Agentic control: PMAR adaptation keeps URLLC delay-violation rates below ε for λe = 2000 and λe = 4000, while fixed-budget QDRL remains close to or above the SLA target.Higher eMBB load increases contention, but PMAR reallocates sufficient URLLC resources using memory-averaged violations.
  • Safety fallback: Safety fallback keeps URLLC violations below ε at λe = 4000 and λe = 8000, whereas removing it raises violations into the 10^-1 range.The fallback replaces unsafe nominal allocations with deadline-aware URLLC and PF-based eMBB scheduling.
  • Queue behavior: Agentic-QDRL maintains substantially lower eMBB backlog than Agentic-DRL at λe = 4000 and λe = 8000, with the gap widening at λe = 8000.The MLP baseline accumulates backlog faster under the heavier load.
  • Cross-algorithm comparison: At λe = 8000, Agentic-QDRL reaches about 107 Mbps feasible throughput versus 60 Mbps for Agentic-DRL, while the heuristic reaches about 0.28 violation rate and violates ε.Both learned methods remain below ε across all tested loads, making Agentic-QDRL the strongest feasible method under heavy load.

V. CONCLUSION

The paper formulates URLLC/eMBB slicing as constrained long-term throughput maximization and proposes Agentic-QDRL to balance throughput with delay reliability. Simulations and ablations show higher feasible throughput, smaller backlog, and reliable URLLC operation across tested loads.

  • Framework: Agentic-QDRL couples a slow PMAR budget controller with a fast QDRL scheduler, enforcing constraints through feasibility projection and EDF/PF safety fallback.The framework targets downlink OFDMA RAN slicing with URLLC delay-reliability and eMBB throughput requirements.
  • Results: Agentic-QDRL converges to higher eMBB throughput, maintains substantially smaller eMBB backlog, and keeps steady-state URLLC violations below the SLA target across tested loads.Under heavy load, it provides the best feasible throughput while the heuristic baseline becomes infeasible.
  • Ablation conclusion: Ablations show that both agentic budget adaptation and safety fallback are essential for maintaining URLLC delay reliability.The conclusion identifies both slow- and fast-time-scale protections as necessary components of the design.
Loading 2608.30206v1…