Source-linked AI summary
Distributive Dynamic Spectrum Access through Deep Reinforcement Learning: A Reservoir Computing Based Approach
Hao-Hsuan Chang, Hao Song, Yang Yi, Jianzhong Zhang, Haibo He, Lingjia Liu
TL;DR
The paper addresses distributed spectrum access with multiple primary and secondary users, imperfect sensing, and no centralized controller. It combines deep reinforcement learning with reservoir computing so users can learn access strategies from sensing histories without known system statistics. Experiments report fewer collisions, better performance than the myopic strategy, and faster convergence than Q-learning when channel counts are large.
Problem
Distributed secondary users must avoid harmful interference with primary users and coordinate access with other secondary users despite imperfect sensing and unknown system statistics.
Method
The paper combines DRL with reservoir computing, using an echo state network as the DQN Q-network to learn distributed spectrum-access strategies from sensing outcomes and rewards.
Results
The proposed strategy reduces transmission collisions, achieves higher successful transmission rates than the myopic strategy, and converges faster than Q-learning when the number of channels is large.
Takeaways & Limitations
The results support learning-based distributed spectrum access when channel statistics are unknown and sensing is imperfect.
Abstract
from arXiv · showhide
Dynamic spectrum access (DSA) is regarded as an effective and efficient technology to share radio spectrum among different networks. As a secondary user (SU), a DSA device will face two critical problems: avoiding causing harmful interference to primary users (PUs), and conducting effective interference coordination with other secondary users. These two problems become even more challenging for a distributed DSA network where there is no centralized controllers for SUs. In this paper, we investigate communication strategies of a distributive DSA network under the presence of spectrum sensing errors. To be specific, we apply the powerful machine learning tool, deep reinforcement learning (DRL), for SUs to learn "appropriate" spectrum access strategies in a distributed fashion assuming NO knowledge of the underlying system statistics. Furthermore, a special type of recurrent neural network (RNN), called the reservoir computing (RC), is utilized to realize DRL by taking advantage of the underlying temporal correlation of the DSA network. Using the introduced machine learning-based strategy, SUs could make spectrum access decisions distributedly relying only on their own current and past spectrum sensing outcomes. Through extensive experiments, our results suggest that the RC-based spectrum access strategy can help the SU to significantly reduce the chances of collision with PUs and other SUs. We also show that our scheme outperforms the myopic method which assumes the knowledge of system statistics, and converges faster than the Q-learning method when the number of channels is large.
I. INTRODUCTION
The paper motivates distributed dynamic spectrum access as a response to scarce, underused spectrum and imperfect sensing. It proposes combining DRL with reservoir computing so secondary users can learn access strategies without centralized control or known system statistics.
- Motivation: Spectrum scarcity and underutilization motivate dynamic spectrum access for expanding wireless traffic.Global mobile data traffic was projected to increase sevenfold between 2016 and 2021, while licensed bands were often underused.
- Problem setting: Secondary users must protect primary users from harmful interference while coordinating access with other secondary users.These challenges are intensified when multiple users share channels without centralized control.
- Problem setting: Imperfect sensing can cause false alarms or missed detections, leading secondary users to make inappropriate channel-access decisions.Listen-before-talk depends completely on current sensing outcomes, which are unreliable in practical wireless environments.
- Proposed approach: The proposed distributed strategy combines DRL and reservoir computing to learn access policies from sensing outcomes without known system statistics.The method is designed for multiple primary and secondary users, imperfect sensing, and no centralized controller.
- Reported findings: Extensive evaluations report fewer collisions with primary and secondary users, superiority over the myopic strategy, and faster convergence than Q-learning.The comparison also reports substantial improvement over DRL combined with a multilayer perceptron by exploiting temporal sensing correlations.
II. DEEP REINFORCEMENT LEARNING AND RESERVOIR COMPUTING
The paper frames spectrum access as reinforcement learning and uses reservoir computing to capture temporal structure while simplifying recurrent-network training. Each secondary user learns a policy from observed sensing states, actions, and rewards.
- Deep reinforcement learning: The paper uses an echo state network as the Q-network of its deep Q-network approach.DRL provides Q-value approximation for large state and action spaces, while the reservoir supplies recurrent temporal processing.
- Reinforcement learning: In this formulation, each secondary user is an agent that maps observed sensing states to access actions through a learned policy.The agent learns by interacting with the environment and maximizing cumulative reward.
- Reinforcement learning: Reinforcement learning learns behavior through exploration and exploitation rather than supervised ground-truth action labels.The policy maps observed states to actions while accumulating environmental knowledge.
- Q-learning: Q-learning estimates state-action values and updates the policy iteratively toward an optimal policy.Its Q-value represents expected cumulative reward under a policy, with updates using learning and discount rates and an ϵ-greedy policy.
B. Deep Reinforcement Learning
The section explains why tabular Q-learning becomes unsuitable as the number of channels grows. Deep reinforcement learning addresses this scaling problem by approximating Q-values with a neural network.
- Deep reinforcement learning: Q-learning performs poorly in large-scale models because increasing state counts make the Q-table difficult or impossible to train.The method remains effective for small-scale models but requires iterative updates across a rapidly growing table.
- Deep reinforcement learning: In the spectrum-access problem, the state space grows exponentially with the number of channels.Each channel can be occupied by a primary user in either an active or inactive state.
- Deep reinforcement learning: Deep reinforcement learning approximates Q-values with a deep neural network instead of updating every Q-table entry.This approach is intended to address the computational difficulty created by large state spaces.
C. Reservoir Computing
RNNs capture temporal behavior through feedback and memory, while reservoir computing simplifies their training by fixing recurrent and input weights. The resulting reservoir provides high-dimensional dynamics with only output weights trained.
- RNNs model temporal behavior using feedback connections and memory of previous neuron activations.
- Reservoir computing simplifies RNN training by fixing the reservoir and input weights while training only output weights.
- The fixed recurrent connections create high-dimensional dynamics that support the reservoir’s representation of temporal inputs.
III. SYSTEM MODEL
The system consists of primary and secondary users sharing orthogonal channels, with PUs protected from harmful interference. Channel propagation, interference, and received quality are modeled using spatial and wireless-channel assumptions.
- The network contains N PUs, L SUs, and N orthogonal channels, with each PU assigned a unique channel.
- All channels are shared by SUs, which select channels while protecting the corresponding PU from harmful interference.
- Interference occurs when a PU or another SU uses the same channel as the receiving SU.
- The model uses WINNER II path loss and a Rician channel model under a strong line-of-sight assumption.
- SU received SINR depends on desired-signal power, PU and SU interference, channel gains, bandwidth, and noise spectral density.
IV. DQN-BASED SPECTRUM ACCESS STRATEGY
The proposed strategy formulates distributed spectrum access as reinforcement learning using sensing observations, channel actions, and transmission rewards. Each SU independently trains a DQN-based policy from local sensing and feedback without knowing channel-transition or sensing-error probabilities.
- The learning cycle stores state, action, reward, and next-state sequences before updating the DQN and periodically replacing its target network.
- Each SU senses all N channels, observes potentially erroneous channel states, and uses those observations as its DQN input.
- Each SU chooses at most one channel or remains idle after sensing, with actions representing channel selection or no access.
- Transmission rewards are based on received SINR and distinguish interference conditions involving PUs, other SUs, or no access.
- The distributed strategy gives each SU an independent DQN and shares no sensing or access decisions between SUs.
- SUs learn access strategies from received SINR feedback without knowing channel-transition or sensing-error probabilities.
V. PERFORMANCE EVALUATION
The evaluation uses simulated DSA networks with stochastic PU activity, sensing errors, and myopic and Q-learning references to assess the proposed distributed learning method.
- Experiments randomly place SUs and PUs in a 150m×150m square and use WINNER II and Rician models for propagation and channel calculations.
- PU dynamics are modeled as independent two-state Markov chains with randomly selected transition parameters reflecting low licensed-band utilization.
- The myopic baseline selects the channel with maximum immediate reward using channel transition probabilities and sensing-error probabilities.
- DQN approximates the state-to-Q-value mapping, avoiding frequent Q-table updates and offering faster convergence than Q-learning for large state spaces.
A. Single SU
The single-SU experiment evaluates convergence in a 22-channel network and finds that DQN+RC converges faster than Q-learning while achieving comparable converged performance.
- The single-SU experiment uses 22 channels, a PU-collision penalty of −2, 64 RC neurons, and a learning rate of 0.01.
- The evaluation tracks average success rate, PU-collision rate, and reward over training time using 4000 training sequences per iteration.
- DQN+RC converges much faster than Q-learning when the number of channels is large, although their converged performance is similar.
- DQN+RC and Q-learning outperform the myopic method despite the myopic method knowing channel transition and sensing-error probabilities.
B. Multiple SUs
The multiple-SU experiment examines learning under mutual interference in a six-channel, two-SU network. Learning-based methods eventually eliminate SU collisions and achieve higher reward than the myopic baseline.
- The experiment uses six channels and two SUs, with performance measured by average success rate, PU- and SU-collision rates, and reward over training time.
- The myopic method initially performs better because it knows the system dynamics, but it cannot learn the access strategies of other SUs.
- After convergence, all learning-based methods achieve zero SU collisions and higher average reward than the myopic method.
- DQN+RC performs better than DQN+MLP1 and DQN+MLP2, while DQN+RC and Q-learning have similar performance in the six-channel setting.
- DQN+MLP2 performs slightly worse than DQN+MLP1, as its deeper structure requires more training data and time to converge.
C. Comparison Between DQN+RC and DQN+MLP1
This comparison tests whether recurrent processing helps exploit temporal correlation in spectrum sensing. DQN+RC learns periodic PU activity, whereas DQN+MLP1 does not.
- The comparison uses temporally correlated sensing outcomes and equalizes MLP1 and RC at 64 neurons with a learning rate of 0.01.
- DQN+MLP1 always accesses the channel, producing a success rate of 1/3 and a collision rate of 2/3.
- DQN+RC learns to access the channel only when the PU is Inactive by learning temporal correlation in the sensing outcomes.
- The experiment shows that RC learns underlying temporal correlation, whereas MLP learns only a one-to-one mapping from state to action.
VI. CONCLUSIONS
The paper studies distributed spectrum access under imperfect sensing without centralized controllers, introducing a DQN-plus-RC strategy for secondary users. Experiments report improved transmission outcomes and faster convergence than comparison methods.
- The study targets spectrum access in a distributive DSA network with imperfect spectrum sensing and no centralized controllers.
- The experiments evaluate average success number, collision number with primary users, and reward over training time for one channel and one secondary user.
- The proposed strategy combines DQN-based deep reinforcement learning with reservoir computing to exploit temporal correlation in the DSA network.
- The strategy enables secondary users to make distributed access decisions using their sensing outcomes and learned information.
- The learning method achieves higher successful transmission and lower collision rates than the myopic strategy, while the DQN approach converges faster and performs better than Q-learning.
- The DQN+RC scheme outperforms the DQN+MLP scheme when spectrum-sensing outcomes exhibit temporal correlation.