Energy Saving in 5G and Beyond Networks: A Quantum Reinforcement Learning Approach

I. INTRODUCTION

• 5G traffic growth and network densification increase energy demand and expected MNO operational expenditure.The RAN accounts for about 60–80% of MNOs’ total energy.Source: §I · p. 1
• Dynamic UEs and large, rapidly changing state spaces challenge conventional learning, while DNN-based DRL can converge too slowly for sub-millisecond decisions.The authors propose QRL, using quantum principles for faster exploration and convergence; they state the framework can run on classical computers using libraries such as Cirq.Source: §I · p. 1
• The proposed framework uses three antennas spaced 120° apart and dynamically optimizes BS transmit power as UEs move.Parameterized quantum circuits are used to explore high-dimensional state spaces for per-antenna power allocation.Source: §I · p. 1
• The solution jointly manages throughput and energy efficiency in dynamic 5G settings while seeking to ensure QoS.The introduction contrasts this joint objective with existing approaches, [10].Source: §I · pp. 1–2

II. SYSTEM MODEL AND PROBLEM FORMATION

• The paper proceeds from a system overview to the problem formulation used by its framework.Source: §II · p. 2

A. Network Scenario

• Three antennas cover distinct sectors, and moving UEs may change association; a full antenna redirects new users even to a lower-rate antenna until capacity is available.Antennas can be deactivated when lightly loaded, with users reassigned to a neighboring antenna whose transmission power may rise within allowable limits.Source: A. Network Scenario · p. 2
• The model assumes enough resources are allocated to serve all UEs, placing packet loss and latency outside the study’s scope.Each antenna has finite RU capacity, and multiple RUs may be assigned to a UE to meet its required data rate.Source: A. Network Scenario · p. 2
• UEs move over time and can change antenna regions, while each antenna is modeled as off, active, or in wake-up mode.The BS position is fixed within each time slot.Source: A. Network Scenario · p. 2
• Per-UE resource allocation is based on traffic demand and data rate per RU, with SINR determined by received signal relative to interference and noise.The model relates required RUs to each UE’s traffic demand and its rate per RU.Source: A. Network Scenario · p. 2
• Antenna energy accounting includes operational, UE-dependent transmission, and switching power, and the system model gives the QRL agent network state, antenna actions, and a throughput-and-energy reward with a minimum-rate penalty.The switching cost includes handover between antennas of the same BS; an inactive antenna is assumed to consume no energy.Source: A. Network Scenario · pp. 2–3, Figure 1 · p. 3

B. Problem Formulation

• The objective minimizes total antenna energy over time while maintaining throughput and accounting for the cost of switching antennas.The formulation seeks to avoid frequent on/off changes because switching incurs a cost.Source: B. Problem Formulation · p. 3
• At each time slot, the system chooses antenna activations, UE-to-antenna assignments, and transmission power levels.The objective determines these variables across antennas, UEs, and time.Source: B. Problem Formulation · p. 3
• The optimization is NP-hard and constrains RU capacity, single-antenna association per UE per slot, and per-RU transmission power.The power is bounded by the antenna’s minimum and maximum transmission-power limits.Source: B. Problem Formulation · p. 3

III. QRL APPROACH FOR 5G AND BEYOND ENERGY SAVING

• The optimization is modeled as a Markov Decision Process to capture system dynamics and uncertainty.Source: §III · p. 3
• The MDP is represented by state space, action space, and immediate reward.Source: §III · p. 3
• Antennas choose actions such as transmission power and coordination based on current SINR and data rate to maximize long-term average reward.Source: §III · p. 3

A. State

• Across T time slots, the system state is represented as S = {s1, . . . , sT}.Source: A. State · p. 3
• Each BS observation includes UE SINR, inverse data rate, and antenna connectivity.Source: A. State · p. 3
• The global state is defined from the set of observations at time t.Source: A. State · pp. 3–4

B. Action

• The action sequence over time is A = {a1, . . . , aT}.Source: B. Action · p. 4
• At each step, each UE selects an antenna and its transmission power.Source: B. Action · p. 4
• The action also includes the UE’s SINR, data rate, and noise, with available transmission power bounded by its stated range.Source: B. Action · p. 4

C. Reward

• The reward guides agents toward energy efficiency while preserving throughput for all UEs.A soft penalty applies when achieved throughput falls below the per-UE target of 5 Mbps.Source: C. Reward · p. 4

D. Quantum Optimization Aspect of MDP Problem

• A qubit state is a linear combination of the basis states |0⟩ and |1⟩.The complex amplitudes determine observation probabilities and satisfy a normalization condition.Source: D. Quantum Optimization Aspect of MDP… · p. 4
• Unitary operators transform superposition components simultaneously, a property described as quantum parallelism.Hadamard and phase gates are identified as gates that generate and manipulate superpositions.Source: D. Quantum Optimization Aspect of MDP… · p. 4
• A networked system uses multiple qubits; representing 2^u states requires u qubits and corresponding multi-qubit gates.Source: D. Quantum Optimization Aspect of MDP… · p. 4

E. Quantum State and Action Representation

• QRL uses quantum computation for action selection and reinforcement, with collapse supporting probabilistic selection and amplitude amplification strengthening preferred actions.Source: E. Quantum State and Action… · p. 4
• For each node and time, an orthogonal quantum state represents a classical RL state–action pair.The passage enumerates discrete quantum state and action sets for each node.Source: E. Quantum State and Action… · p. 4
• Encoding Nst eigenstates and Nac eigenactions requires u qubits for the state register and v qubits for the action register.Source: E. Quantum State and Action… · p. 4

• A unitary transformation maps each classical state into orthogonal quantum eigenstates in QRL.Source: F. QRL-Based State Value Updating · p. 4
• With u qubits, quantum parallelism evaluates 2u computational basis states simultaneously.The passage describes each basis state as part of the encoded state space.Source: F. QRL-Based State Value Updating · p. 4
• QRL updates the encoded state values in parallel using the TD(0) rule, rather than sequentially.A u-qubit register represents exponentially many states in one operation.Source: F. QRL-Based State Value Updating · p. 4

A. Parameter Setting

• UE mobility updates UE–BS distance at each time step, and antennas use four power levels: 0%, 25%, 50%, or 100%.The simulation uses 20 MHz bandwidth at a 2 GHz carrier frequency and 75 dBm per antenna.Source: A. Parameter Setting · p. 5
• The DRL baseline uses two hidden layers of 256 and 128 neurons, tanh activation, Adam, and a replay buffer of 10,000 transitions.Its learning rate is 0.001, batch size 32, and target network updates every 5,000 steps.Source: A. Parameter Setting · p. 5

IV. PERFORMANCE EVALUATION

• The evaluation compares QRL with three baselines using transmit power and throughput.Throughput is computed from antenna effective transmit power and the bits delivered per second to UEs.Source: A. Parameter Setting · p. 5

A. Parameter Setting

• Fig. 2 presents reward progression against training steps for the models.Source: Figure 2 · p. 5

B. Simulation Results

• QRL reaches an average reward of approximately 65 within 6,000 steps, while DRL and Q-Learning do not achieve comparable performance even after 100,000 steps.The passage reports this comparison across repeated runs.Source: B. Simulation Results · p. 5
• During training, QRL consistently achieves the lowest power and converges much faster than DRL and Q-Learning.The power-consumption figure tracks the models during training.Source: Figure 3 · p. 5, B. Simulation Results · p. 5
• DRL and QRL have similar average BS power levels, but DRL converges more poorly than QRL.Source: Figure 4 · p. 5
• Average throughput remains stable during energy optimization, with a minimum transmit power of 25% needed to maintain QoS at low UE density.The passage states that the energy strategies do not degrade performance.Source: Figure 5 · p. 5, B. Simulation Results · p. 6
• QRL has fewer throughput fluctuations during training and the highest, most consistent rewards as UE count increases from 0 to 100.Q-Learning fluctuates most, while DRL is smoother but less consistent than QRL.Source: Figure 6 · p. 6

V. CONCLUSION

• The reported simulator uses a simple network model with a few UEs and one BS; extension to multiple BSs and denser networks is future work.Source: §V · p. 6