PaperlayerView exact revision on arXiv

Source-linked AI summary

Intelligent Power Control for Spectrum Sharing in Cognitive Radios: A Deep Reinforcement Learning Approach

Xingjian Li, Jun Fang, Wen Cheng, Huiping Duan, Zhi Chen, Hongbin Li

arXiv:1712.07365v3cs.IT

TL;DR

The paper addresses secondary-user power control for spectrum sharing when the primary user follows an unknown policy and the users do not cooperate. It uses sensor-based RSS observations with deep reinforcement learning, enabling the secondary user to reach successful joint transmission from any initial state within a few steps in the reported experiments.

  • Problem

    The problem is to let a secondary user share spectrum with a non-cooperative primary user without knowing the primary user’s transmit power or power-control strategy.

  • Method

    The method uses spatial sensor RSS measurements and a deep reinforcement learning action-value policy to adjust the secondary user’s transmit power.

  • Results

    After 103 training iterations, the learned policy reached a goal state from any initial state with probability one in an average of 1.5 transition steps for one reported setting.

  • Takeaways & Limitations

    The experiments show that the learned method can achieve successful transmission for both users and remain robust to random variation in state observations.

Abstract

from arXiv · show

We consider the problem of spectrum sharing in a cognitive radio system consisting of a primary user and a secondary user. The primary user and the secondary user work in a non-cooperative manner. Specifically, the primary user is assumed to update its transmit power based on a pre-defined power control policy. The secondary user does not have any knowledge about the primary user's transmit power, or its power control strategy. The objective of this paper is to develop a learning-based power control method for the secondary user in order to share the common spectrum with the primary user. To assist the secondary user, a set of sensor nodes are spatially deployed to collect the received signal strength information at different locations in the wireless environment. We develop a deep reinforcement learning-based method, which the secondary user can use to intelligently adjust its transmit power such that after a few rounds of interaction with the primary user, both users can transmit their own data successfully with required qualities of service. Our experimental results show that the secondary user can interact with the primary user efficiently to reach a goal state (defined as a state in which both users can successfully transmit their data) from any initial states within a few number of steps.

I. INTRODUCTION

The paper motivates cognitive-radio spectrum sharing to improve spectrum efficiency and develops deep reinforcement learning for secondary-user power control under non-cooperative, partially unknown primary-user behavior.

  • Motivation: Spectrum bands can be severely underutilized, motivating secondary users to access licensed spectrum without causing harmful interference.The cited FCC example reports utilization rates as low as 15% for some bands.
  • Prior work: Existing spectrum-sharing studies often assume the primary user need not adapt because secondary transmissions are transparent to it.Transparency is pursued through spectrum sensing or strict secondary transmit-power control.
  • Prior work: Most existing dynamic power-control methods formulate the problem as optimization, including distributed constrained power control that targets receivers’ SINR requirements.The DCPC algorithm iteratively adjusts transmitter powers using SINR measurements and required thresholds.
  • Approach: Deep reinforcement learning can learn action-value policies from high-dimensional, noisy observations where conventional Q-learning becomes impractical because noise creates infinitely many states.This property motivates its use for wireless state measurements affected by random variation and measurement errors.
  • Contribution: The paper develops a deep reinforcement learning policy that lets a secondary user adjust power without knowing the primary user’s transmit power or control strategy.Spatially deployed sensors provide received signal strength information to assist learning.
  • Organization: The paper is organized around the system model, the deep reinforcement learning power-control algorithm, experiments, and conclusions.The sections cover symbols, problem formulation, algorithm development, experimental results, and concluding remarks.

II. SYSTEM MODEL

The system models two non-cooperative users sharing spectrum, with the secondary user learning power adjustments from spatial RSS measurements to satisfy both users’ SINR-based QoS requirements.

  • System setting: The primary and secondary users transmit over a common spectrum, while the secondary user seeks sharing without harmful interference to the primary user.The primary user follows its own power-control policy and is unaware of the secondary user.
  • State observation: Spatially deployed sensors measure RSS at each time frame, and those measurements reveal state information related to both users’ transmit powers.Each sensor reports once per frame, enabling low-rate feedback to the secondary user.
  • QoS model: Both users’ QoS requirements are expressed as minimum SINR thresholds for successful reception.The model requires SINR_i ≥ η_i for i = 1, 2.
  • Primary-user control: The primary user is modeled with two adaptive power-control strategies, including a classical discretized update and a conservative stepwise policy.The stepwise policy changes power by one level based on measured and predicted SINR conditions, or keeps the current level otherwise.
  • Observation model: The wireless environment includes shadowing and estimation-error variation in sensor observations, with path gains determined by transmitter-to-sensor distances under free-space propagation.The RSS model uses zero-mean Gaussian variation and Friis-law path gains.
  • Learning objective: The secondary user chooses transmit power from a finite set and learns adjustments based on collected RSS so both users can meet their QoS requirements after a few rounds.The model assumes at least one pair of transmit powers satisfies both SINR requirements.

III. A DEEP REINFORCEMENT LEARNING APPROACH FOR POWER CONTROL

The secondary user models spectrum sharing as an MDP and learns a power-control policy with a DQN from state transitions, rewards, and sensor observations. The learned policy selects transmit powers that can preserve goal states and can operate with asynchronous user updates.

  • MDP formulation: The secondary user chooses transmit power from a predefined action set according to its current state.The state reflects received signal-strength observations, and the action is the next secondary-user transmit power.
  • MDP formulation: A goal state is reached when both users satisfy their SINR requirements, maximizing discounted cumulative reward until that state.The goal condition is SINR_i(k) ≥ η_i for i = 1, 2.
  • Q-learning and DQN: Q-learning evaluates actions through an action-value function, then constructs the policy by selecting the highest-valued action in each state.The Bellman update iteratively refines the action-value function.
  • Q-learning and DQN: Because random RSS measurements produce a continuous state space, the method replaces the Q-table with a deep neural network approximating Q(s, a; θ).Training uses exploratory or value-maximizing actions, replayed transitions, minibatches, and an updated target network.
  • Deployment: After training, the secondary user selects the action with the largest estimated Q-value and needs only sensor feedback to determine its next transmit power.During training, QoS satisfaction feedback is also required.
  • Convergence and timing: Once a goal state is reached, the learned policy keeps the system in that goal state until transmission ends, including under asynchronous power updates.The convergence discussion covers both primary-user power-control policies, while asynchronous operation removes the need for strict frame synchronization.

IV. EXPERIMENTAL RESULTS

The experiments evaluate a DQN power-control algorithm under predefined transmit-power, noise, and channel settings. The reported results show rapid training and robustness when sensor count and measurement-noise variance vary.

  • Experimental setup: The experiments use transmit powers from P1 = P2 = {0.05, 0.1, . . ., 0.4}, noise powers N1 = N2 = 0.01W, and unit channel gains.The minimum SINR requirements and other setup parameters are specified in the experimental model.
  • Experimental setup: The DNN uses three fully connected hidden layers with 256, 256, and 512 neurons, together with ReLU and tanh activations.The Adam algorithm updates the weights, with minibatch size 256 and replay memory capacity ND = 400.

0. Specifically, at iteration k, we let

The experiments evaluate DQN-based secondary-user power control using success rate and transition steps across training settings, primary-user policies, sensor counts, and observation noise. The method reaches goal states efficiently, remains robust under moderate variation, and approaches DCPC performance with fewer transition steps.

  • Success rate and average transition steps measure whether the trained network reaches a goal state within 20 time frames.A goal state is one in which both users can successfully transmit their data.
  • After 103 training iterations, the learned policy reaches goal states from any initial state with probability one and 1.5 average transition steps.
  • Increasing observation variance or reducing sensor count increases loss, but success rate and transition steps remain similar in the first policy setting.The reported result is presented as evidence of robustness to these observation conditions.
  • Under the second, more conservative primary-user policy, about 1.5×10^4 training iterations yield success rate one and 2.5 average transition steps.The primary user changes transmit power by only one level per step, requiring more transitions to reach the goal state.
  • Large observation variance and too few sensors cause considerable performance loss because states become less distinguishable for the agent.Using more sensors helps neutralize random observation variations.
  • Compared with DCPC, the DQN method reaches a goal state in only a few transition steps, while DCPC takes tens, and converges near the DCPC solution.Unlike DCPC's cooperative global coordination, DQN learns interaction with the primary user in real time after training.

V. CONCLUSIONS

The paper develops a deep reinforcement learning method for non-cooperative spectrum sharing between primary and secondary users. Experiments show robust goal-state reachability from any initial state within only a few steps.

  • The proposed deep reinforcement learning method lets the secondary user learn transmit-power adjustments while the primary user follows its own predefined policy.
  • Experiments show robustness to random state-observation variation and goal-state reachability from any initial state within only a few steps.
Loading 1712.07365v3…