Source-linked AI summary
Anti-jamming Communications Using Spectrum Waterfall: A Deep Reinforcement Learning Approach
Xin Liu, Yuhua Xu, Luliang Jia, Qihui Wu, Alagan Anpalagan
TL;DR
The paper targets anti-jamming communication in dynamic, unknown environments without requiring estimation of jamming patterns or parameters. It represents the environment through raw spectrum information, uses a deep anti-jamming Q-network and reinforcement learning, and reports improved anti-jamming performance across scenarios.
Problem
Existing anti-jamming approaches require estimating jamming strategies, patterns, or parameters, while raw spectrum information has an effectively infinite state space for traditional Q-learning.
Method
The approach defines raw spectrum information as the state, processes it with a deep anti-jamming Q-network, and proposes a deep anti-jamming reinforcement learning algorithm.
Results
Simulation results show improved anti-jamming performance across various scenarios, with normalized throughput close to one under comb jamming after convergence.
Takeaways & Limitations
The proposed model-free approach relies on locally observed information and does not require advance estimation of jamming patterns and parameters.
Abstract
from arXiv · showhide
This letter investigates the problem of anti-jamming communications in dynamic and unknown environment through on-line learning. Different from existing studies which need to know (estimate) the jamming patterns and parameters, we use the spectrum waterfall, i.e., the raw spectrum environment, directly. Firstly, to cope with the challenge of infinite state of raw spectrum information, a deep anti-jamming Q-network is constructed. Then, a deep anti-jamming reinforcement learning algorithm is proposed to obtain the optimal anti-jamming strategies. Finally, simulation results validate the the proposed approach. The proposed approach is relying only on the local observed information and does not need to estimate the jamming patterns and parameters, which implies that it can be widely used various anti-jamming scenarios.
I. INTRODUCTION
The paper addresses anti-jamming communication in dynamic, unknown environments where existing methods require estimating jamming patterns and parameters. It proposes using raw spectrum information with deep reinforcement learning to learn anti-jamming strategies.
- Motivation: Existing anti-jamming methods, including game-theoretic approaches, require estimating jamming patterns and parameters from observations.The paper motivates a method that does not require prior knowledge of jammer strategies.
- Motivation: Spectrum waterfall provides raw environmental information without estimating jamming patterns and parameters, but Q-learning cannot directly handle its nearly infinite state space.The paper identifies raw spectrum processing and state-space size as central design challenges.
- Proposed approach: The proposed approach defines raw spectrum information as the environment state to preserve detailed jammer features.This state representation is intended to avoid losing jammer information as much as possible.
- Proposed approach: A deep anti-jamming Q-network processes raw spectrum information, followed by a deep anti-jamming reinforcement learning algorithm for strategy selection.The approach is presented as a smart anti-jamming communication scheme based on deep reinforcement learning.
- Relation to prior work: Unlike related work that uses extracted SINR and primary-user-occupancy features and requires matched channel-slot structures, this approach uses raw spectrum information without that requirement.The paper states that removing the shared channel-slot requirement makes the approach more general.
II. SYSTEM MODEL AND PROBLEM FORMULATION
The system models anti-jamming as a dynamic decision problem using recent raw spectrum observations as the state, without estimating specific jamming models. Actions receive rewards based on transmission rate, switching cost, and SINR success.
- Spectrum-state representation: The receiver continuously senses the communication bands and stores power measurements across frequency and time.The spectrum vector records power at N frequency sampling points for each time t.
- Spectrum-state representation: The state St is a two-dimensional time-frequency matrix containing recent spectrum information, with history length M chosen for the environment’s time variation.Although M approaching infinity retains all history, larger M increases decision-optimization difficulty.
- Spectrum-state representation: Spectrum-waterfall patterns can reveal a jammer’s next frequency range and intensity, supporting corresponding anti-jamming decisions.The paper illustrates this reasoning using swept jamming thermal charts.
- Decision formulation: The agent interacts with the unknown, dynamic environment through observations St, actions at, and rewards rt rather than estimation-based jamming strategies.Actions may combine frequency, power, coding, spread-spectrum, and other anti-jamming decisions.
- Decision formulation: The reward combines selected-action bit rate with a switching cost and assigns zero when received SINR falls below the action-dependent threshold.The switching indicator is one when the action changes and zero otherwise.
- Decision formulation: The agent seeks actions maximizing cumulative discounted future reward through an optimal action-value function and Bellman relation.The discount factor is γ, and the policy is a probability distribution over actions conditioned on state.
III. ANTI-JAMMING COMMUNICATION SCHEME
The proposed DAQN uses a convolutional network to process spectrum-waterfall states and estimate action values, while experience replay stabilizes Q-learning updates. DARLA selects exploratory or greedy actions, stores transitions, and updates network weights from sampled minibatches.
- DAQN architecture: Traditional Q-learning cannot readily handle the almost-infinite state space of raw spectrum states, motivating the deep anti-jamming Q-network.DAQN contains decision and update networks for interactive decision-making with raw spectrum input.
- DAQN architecture: The DAQN decision network inputs an M × N thermal-chart state, applies two convolutional and two fully connected layers, and outputs K estimated action values.The decision layer then selects the corresponding action from the estimated Q function.
- Network updating: Experience replay samples transitions uniformly from a stored experience pool to reduce correlation in sequential observations.The sampled transitions are used to construct target values and train the Q-network.
- Network updating: At iteration i, the update compares the network output Q(S, a; θi) with a target yi computed using the previous network parameters and a greedy next-action value.The gradient is then used to update θi by gradient descent.
- DARLA procedure: DARLA initializes an empty replay pool and random network weights, chooses random actions with probability ϵ otherwise greedy actions, and updates from stored minibatches.The exploration probability is decreased but bounded below by 0.1.
- DARLA procedure: Although the architecture depicts separate decision and updating networks, implementation uses one CNN because target computation and weight updating occur in different stages.The paper presents the complete procedure as Algorithm 1, DARLA.
IV. NUMERICAL RESULTS AND DISCUSSIONS
Simulations evaluate DARLA across multiple jamming patterns using normalized throughput, environmental states, and action probabilities. The results show improved anti-jamming behavior, including near-complete avoidance for comb jamming and pattern-adaptive actions.
- Simulation Setting: The simulations use a 20MHz band with 100kHz sensing resolution, 200 × 200 spectrum-state matrices, and four 4MHz-bandwidth jamming patterns.The patterns include sweep, comb, random, and intelligent jamming.
- Throughput Results: DARLA significantly improves normalized average throughput across the evaluated jamming patterns, with comb-jamming throughput approaching one after convergence.The paper interprets this as the jamming being almost completely avoided in the comb-jamming case.
- Environmental States: Environmental states encode time-frequency information that reflects prior user and jammer actions during both initial and converged learning stages.The states are shown for sweep, comb, and random jamming patterns.
- Environmental States: Under sweep jamming, randomized initial actions become frequency changes made before the jammer arrives after convergence.The converged rectangular-block pattern follows the sweep trajectory, unlike the initially random distribution.
- Action Probabilities: For intelligent jamming, the learned probabilities of user actions become nearly identical, matching the strategy motivated by the jammer’s reliance on those probabilities.Figure 7 reports action-selection probabilities throughout learning.
V. CONCLUSION
The letter addresses anti-jamming in unknown, dynamic environments by directly using waterfall spectrum information. It constructs a deep Q-network and reinforcement-learning algorithm, with simulations validating the approach across various scenarios.
- Conclusion: The study targets anti-jamming communication in unknown and dynamic environments using waterfall spectrum information directly.The conclusion frames the problem as learning from spectrum observations rather than specifying a jamming model.
- Conclusion: A deep anti-jamming Q-network handles the infinite state space of waterfall spectrum information, and a deep reinforcement-learning algorithm learns strategies through repeated actions and sensing.The user learns by continuously trying actions and sensing the spectrum environment.
- Conclusion: Simulation results across various scenarios validate the proposed anti-jamming communication approach.
- Conclusion: Future work concerns multi-user deep anti-jamming reinforcement-learning algorithms.