PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Reinforcement Learning based Modulation and Coding Scheme Selection in Cognitive Heterogeneous Networks

Lin Zhang, Junjie Tan, Ying-Chang Liang, Gang Feng, Dusit Niyato

arXiv:1811.02868v1cs.ITmath.IT

TL;DR

Imperfect sensing can let secondary transmitters interfere with the primary receiver, while time causality leaves the interference unknown when the primary transmitter begins transmission. A DRL agent at the primary receiver learns secondary interference patterns to infer future interference and select MCSs, with a switching cost to balance rate and overhead. The proposed algorithm is 30% ∼100% higher than benchmark algorithms.

Abstract

from arXiv · show

We consider a cognitive heterogeneous network (HetNet), in which multiple pairs of secondary users adopt sensing-based approaches to coexist with a pair of primary users on a certain spectrum band. Due to imperfect spectrum sensing, secondary transmitters (STs) may cause interference to the primary receiver (PR) and make it difficult for the PR to select a proper modulation and/or coding scheme (MCS). To deal with this issue, we exploit deep reinforcement learning (DRL) and propose an intelligent MCS selection algorithm for the primary transmission. To reduce the system overhead caused by MCS switchings, we further introduce a switching cost factor in the proposed algorithm. Simulation results show that the primary transmission rate of the proposed algorithm without the switching cost factor is 90 percent to 100 percent of the optimal MCS selection scheme, which assumes that the interference from the STs is perfectly known at the PR as prior information, and is 30 percent to 100 percent higher than those of the benchmark algorithms. Meanwhile, the proposed algorithm with the switching cost factor can achieve a better balance between the primary transmission rate and system overheads than both the optimal algorithm and benchmark algorithms.

I. INTRODUCTION

The introduction presents cognitive HetNets as a way to improve capacity through shared spectrum, while noting that coexistence and imperfect sensing can expose primary links to interference. It motivates learning that interference pattern and places DRL among approaches for large decision spaces.

I. INTRODUCTION

  • Cognitive HetNets share spectrum between high-priority primary users and lower-priority secondary users to address coexistence without dedicating separate bands.HetNets are presented as a way to improve capacity as wireless link efficiency approaches a fundamental limit.Source: §I · p. 1

A. Motivations

  • Imperfect sensing can let secondary transmitters interfere with primary uplink reception, where the primary transmitter cannot know future interference when transmission begins.The cited scenario has a weak primary signal at secondary transmitters and a non-negligible channel from secondary transmitters to the primary receiver.Source: A. Motivations · pp. 1–2
  • The paper proposes that the primary receiver learn secondary interference patterns from historical observations to infer future interference and adapt primary transmission.The stated aim is to enhance primary transmission rate and network capacity.Source: A. Motivations · p. 2

B. Related work

  • Prior RL work addresses wireless decision-making, but its effectiveness diminishes as state-action spaces grow, motivating DRL for larger spaces.The introduction also surveys DRL applications including scheduling, channel selection, anti-jamming, handover, access, and power allocation.Source: B. Related work · p. 2

C. Contributions of the paper

The paper proposes DRL-based MCS selection for the primary receiver and a switching cost to balance primary transmission rate against system overhead. The section then outlines the paper and describes the considered network and channel model.

C. Contributions of the paper

  • The PR-side DRL agent learns secondary interference patterns to infer future interference and adapt MCS, while a switching cost adjusts the rate–overhead balance.The paper describes these as its principal algorithmic contributions.Source: C. Contributions of the paper · p. 2
  • Without the switching cost factor, the proposed algorithm achieves transmission rates 30% ∼100% higher than benchmark algorithms; with the factor, it better balances transmission rate and system overheads than optimal and benchmark algorithms.Source: C. Contributions of the paper · pp. 2–3

D. Organization of the Paper

  • The paper proceeds from the system model and optimal MCS policy to the DRL algorithm, simulations, and conclusion.The stated sequence is Sections II through VI.Source: D. Organization of the Paper · p. 3

II. SYSTEM MODEL

  • The considered overlay network has a primary uplink to the macro BS and K secondary-user pairs that sense before accessing the same spectrum band.Figure 1 depicts the PU, macro BS, and K secondary pairs.Source: §II · p. 3, Figure 1 · p. 3

A. Channel model

  • Each channel combines large-scale path loss with small-scale block Rayleigh fading; path loss is distance-dependent and fading is modeled across frames.The small-scale fading follows Jake’s model, with successive-frame correlation ρ and random variation δ.Source: A. Channel model · p. 3

B. Coexistence model

Primary and secondary transmissions use synchronized frames but different initial phases: the primary selects an MCS while secondary users sense the spectrum. Imperfect sensing can cause interference during later primary transmission, which the signal model treats as a burst-interference setting.

B. Coexistence model

  • Each PU frame begins with training, channel and SNR estimation, and MCS feedback, followed by uplink data transmission using the selected MCS.The MCS-selection interval τp is small relative to frame duration T and is neglected in system design.Source: B. Coexistence model · p. 3
  • Secondary users sense during the initial τ of each synchronized frame, then transmit if the channel is declared idle; missed detection can interfere with the PU during T − τ.The miss-detection/interference probability αk denotes the chance that ST-k accesses the channel and interferes with the primary transmission.Source: B. Coexistence model · pp. 3–4

C. Signal model of the PU transmission

  • The PU’s initial frame interval is interference-free, while active secondary transmitters may interfere during the later interval; a bit-interleaver is assumed to address burst interference.The signal model then considers the average SINR of each bit at the BS.Source: C. Signal model of the PU transmission · p. 4

III. OPTIMAL MCS SELECTION POLICY

At a given average SINR, MCS selection balances reliability against per-frame rate, and the policy seeks the rate-maximizing choice. The BS cannot compute that optimum at selection time because future secondary interference is unknown.

A. Basic principle

  • Low-order MCSs reduce symbol and packet error rates but also provide lower transmission rates, creating a reliability-rate tradeoff.The PU uses fixed transmit power, and MCS selection is performed at the BS, which can measure and analyze SINR including ST interference.Source: A. Basic principle · p. 4

B. The optimal modulation policy

  • The model considers M MCS levels, each characterized by its symbol error rate f_m(γ̄) and average transmission efficiency r_m in bits per symbol.The packet error rate is approximated from the symbol error rate over N transmitted symbols.Source: B. The optimal modulation policy · p. 4
  • The policy selects the MCS that maximizes transmission rate in each frame.The rate calculation depends on average SINR, which the BS cannot determine at selection time because it depends on later secondary interference.Source: B. The optimal modulation policy · p. 4
  • The BS’s straightforward alternative selects an MCS from training-signal SNR γ0 while ignoring ST interference, making it suboptimal for transmission rate.The ideal optimization requires the average SINR, but its secondary-interference component is unknown during MCS selection.Source: B. The optimal modulation policy · p. 4

IV. INTELLIGENT DRL-BASED MCS SELECTION

The BS cannot know future secondary interference when choosing an MCS, but can learn its pattern from historical measurements. The section motivates DRL as a way to learn that pattern and develop a rate-maximizing policy, then introduces the reinforcement-learning framework underlying the approach.

A. Basic principle

  • The proposed DRL-based algorithm learns hidden ST-interference patterns from historical measurements so the BS can infer future interference and select an MCS to maximize transmission rate.The interference pattern depends mainly on ST transmit powers and channel gains, which are unknown to the BS, although interference can be measured after each data-transmission phase.Source: A. Basic principle · p. 5
  • The section frames the MCS problem as optimal decision-making and identifies DRL as a way to learn a hidden pattern through trial and error toward an optimal policy.The proposed agent uses this approach to maximize primary transmission rate.Source: §IV · p. 4, A. Basic principle · p. 5

B. General RL framework

  • The RL framework comprises action and state spaces, immediate rewards, transition probabilities, a Q-function, and a policy mapping states to actions.The Q-function represents expected cumulative discounted future reward for a state-action pair.Source: B. General RL framework · p. 5
  • The policy targets long-term reward, which depends on both immediate and future rewards rather than only the immediate payoff.The section expresses long-term reward recursively, with η discounting future rewards and (s′, a′) denoting the next state-action pair.Source: B. General RL framework · p. 5
  • Q-learning uses a Q-table and ε-greedy action selection, but large state-action spaces make the table costly and can leave many pairs unexplored.ε-greedy selection exploits the highest-valued action with probability 1−ε and explores randomly with probability ε.Source: B. General RL framework · pp. 5–6

C. General DRL framework

The DRL framework replaces a tabular Q-function with a deep Q-learning network that estimates long-term action rewards. It updates network parameters from recorded state-action-reward transitions by minimizing prediction error.

C. General DRL framework

  • A deep Q-learning network approximates Q(s, a), outputting the estimated long-term reward for each available action given environmental state s.Optimizing the Q-function is recast as optimizing the network weights θ.Source: C. General DRL framework · p. 6
  • The agent records experiences as state, action, reward, and next state, then defines a prediction error for each experience.The target output is denoted yTar.Source: C. General DRL framework · p. 6
  • The agent updates network weights by gradient descent to minimize prediction error after obtaining a new experience.The network learns iteratively from historical states, actions, and rewards while interacting with the environment.Source: C. General DRL framework · p. 6

D. Intelligent DRL-based MCS selection algorithm

The BS’s DRL agent chooses an available MCS at the start of each frame. Its immediate reward is the number of data bits successfully transmitted, with zero reward for a failed transmission.

1) Action space:

  • At the beginning of each frame, the agent’s action space includes all available MCS levels for the PU uplink.The selected action is denoted a(t) and belongs to action space A.Source: 1) Action space · p. 6

2) Immediate reward function:

  • The immediate reward is designed to be proportional to the data bits successfully transmitted from the PU to the BS.The reward function uses transmitted data bits when transmission succeeds and zero otherwise.Source: 2) Immediate reward function · p. 6
  • A successful transmission earns r_mN bits of reward, while a failed transmission earns zero.Here the reward corresponds to the selected MCS’s transmission efficiency over N symbols.Source: 2) Immediate reward function · p. 6

3) State space:

The state combines current channel information with recent observations and outcomes so the DRL agent can infer interference patterns and select an MCS. The algorithm stores experience and trains a DQN using replay and a target network; convergence is assessed through simulation.

3) State space:

  • The state includes current BS SNR and the previous Φ frames’ SNR, SINR, actions, and rewards, supporting channel-aware MCS selection and inference of future interference.The BS cannot observe current-frame interference at frame start, so the agent uses historical SINR and actions/rewards to learn interference patterns and the mapping to MCS choices.Source: 3) State space · p. 6
  • At each frame, the BS measures PU SNR, forms the state and experience, selects an action with an ε-greedy module, then measures SINR and reward during transmission.The DRL agent stores experiences and randomly samples them to train the DQN.Source: 3) State space · p. 7
  • Training uses experience replay and a quasi-static target network, with the target network’s weights refreshed every L frames.The agent samples mini-batches from local memory to update the trained DQN.Source: 3) State space · p. 7
  • A general DRL convergence proof remains an open problem, while convergence of this proposed algorithm is validated through simulation.Source: 3) State space · p. 7

E. Intelligent DRL-based MCS selection algorithm with switching costs

Frequent MCS changes can increase signaling, reconfiguration, spectrum, and energy costs, including when switching has little effect on long-term reward. The method penalizes switching in the immediate reward to balance transmission rate against overhead.

E. Intelligent DRL-based MCS selection algorithm with switching costs

  • Some switches may offer only a slightly larger Q-value than retaining the previous MCS, despite little impact on long-term reward.The paper identifies such switches as avoidable to reduce switching overhead.Source: E. Intelligent DRL-based MCS selection… · p. 8
  • The switching cost factor c is added to the immediate reward, and adjusting c yields different balances between transmission rate and MCS-switching overhead.The cost represents the overall impact of switching on system overhead, expressed relative to transmitted data bits.Source: E. Intelligent DRL-based MCS selection… · p. 8
  • Switching costs are applied to successful and failed transmissions when the selected MCS differs from the previous frame’s MCS.The reward subtracts c for a successful switch and assigns −c for a failed switch; unchanged MCS outcomes receive no switching penalty.Source: E. Intelligent DRL-based MCS selection… · p. 8

V. SIMULATION RESULTS

The simulations compare the proposed algorithm with an SINR-informed theoretical upper bound and two benchmark algorithms. The settings describe an always-active primary transmission and an uncoded four-MCS system, with DQN and frame parameters specified.

V. SIMULATION RESULTS

  • The proposed method is evaluated against an optimal MCS selector using beginning-of-frame average SINR as a theoretical upper bound, plus SNR-based and UCB benchmarks.The upper-bound assumption is impractical because the BS cannot know average SINR at frame start; the SNR benchmark substitutes measured SNR for average SINR.Source: §V · p. 8

A. Assumptions and settings in the simulation

B. Performance comparison in quasi-static and dynamic interference scenarios

DRL-based MCS selection approaches the optimal transmission rate under quasi-static interference and remains near it under dynamic interference. Increasing the history window has little effect in quasi-static conditions but reduces rate in dynamic conditions.

B. Performance comparison in quasi-static and dynamic interference scenarios

  • Around 3 kbits/frame, DRL approaches the optimal rate and is around 50% above UCB and 100% above SNR-based selection under quasi-static interference.The quasi-static scenario uses two secondary-user pairs and slowly changing interference.Source: B. Performance comparison in… · p. 9
  • Around 2.6 kbits/frame, DRL reaches around 90% of the optimal rate, 30% above UCB, and 100% above SNR-based selection under dynamic interference.The comparison averages transmission rates over the previous 200 frames and 20 trials.Source: B. Performance comparison in… · p. 9, Figure 5 · p. 9

C. Performance of the proposed algorithm with different Φ

  • In quasi-static interference, increasing Φ from 1 to 10 leaves transmission rate almost unchanged because older frames provide nearly the same pattern information as the last frame.The authors conclude that multiple previous frames are unnecessary in each state for this setting.Source: C. Performance of the proposed… · pp. 9–10
  • In dynamic interference, transmission rate decreases as Φ rises from 1 to 10; multiple previous frames can confuse the agent rather than add useful pattern information.The authors say the agent consequently needs more frames to extract useful information and infer future interference.Source: C. Performance of the proposed… · p. 10

D. Balance between transmission rate and system overheads

The switching-cost version of DRL-based MCS selection trades transmission rate against switching overhead. Across the reported settings, it achieves a better balance than the optimal and benchmark algorithms.

D. Balance between transmission rate and system overheads

  • In quasi-static interference, DRL can exceed SNR-based selection in rate while switching less for costs 0.5–6, and exceed UCB in rate at nearly identical switching rates for costs 4–6.The authors conclude that this gives DRL a better rate–overhead balance than the optimal, UCB, and SNR-based algorithms in this setting.Source: D. Balance between transmission rate… · p. 11
  • In dynamic interference, DRL matches UCB at a tight switching-rate constraint for costs 3.5–6, and exceeds UCB’s rate above a 0.03 switching rate for costs 0–3.5.The authors conclude that DRL achieves a better rate–overhead balance than the optimal and benchmark algorithms in this setting.Source: D. Balance between transmission rate… · p. 11, Figure 9 · p. 11

VI. CONCLUSIONS

  • Without switching cost, DRL achieves 90%–100% of the optimal MCS-selection rate and 30%–100% higher rates than benchmark algorithms.With switching cost, it achieves a better rate–overhead balance than the optimal and benchmark algorithms.Source: §VI · p. 11
Loading 1811.02868v1…