Deep Reinforcement Learning for Cyber Security

Thanh Thi Nguyen; Vijay Janapa Reddi

Source-linked AI summary

Deep Reinforcement Learning for Cyber Security

Thanh Thi Nguyen, Vijay Janapa Reddi

arXiv:1906.05799v4Jun 13, 2019cs.CRcs.AIcs.LGstat.ML

TL;DR

Cyber attacks on increasingly complex, dynamic, and high-dimensional Internet-connected systems require responsive, adaptive, and scalable defenses. This paper surveys deep reinforcement learning approaches for cyber security, covering cyber-physical systems, intrusion detection, and multiagent defense simulations, while identifying future research directions.

Problem
Increasingly complex and dynamic cyber attacks create a need for responsive, adaptive, and scalable security mechanisms for Internet-connected systems.
Method
The paper surveys cyber-security approaches that combine deep learning’s representation learning and function approximation with reinforcement learning’s sequential decision-making.
Results
The survey covers DRL-based security methods for cyber-physical systems, autonomous intrusion detection, and multiagent game-theoretic simulations for cyber-defense strategies.
Takeaways & Limitations
The review provides foundations and future research directions for applying DRL to increasingly complex cyber-security problems.
Takeaways & Limitations
Progress is constrained by limited realistic CPS and intrusion-training environments and by challenges in large-scale multiagent DRL, including non-stationarity and partial observability.

Abstract

from arXiv · show

The scale of Internet-connected systems has increased considerably, and these systems are being exposed to cyber attacks more than ever. The complexity and dynamics of cyber attacks require protecting mechanisms to be responsive, adaptive, and scalable. Machine learning, or more specifically deep reinforcement learning (DRL), methods have been proposed widely to address these issues. By incorporating deep learning into traditional RL, DRL is highly capable of solving complex, dynamic, and especially high-dimensional cyber defense problems. This paper presents a survey of DRL approaches developed for cyber security. We touch on different vital aspects, including DRL-based security methods for cyber-physical systems, autonomous intrusion detection techniques, and multiagent DRL-based game theory simulations for defense strategies against cyber attacks. Extensive discussions and future research directions on DRL-based cyber security are also given. We expect that this comprehensive review provides the foundations for and facilitates future studies on exploring the potential of emerging DRL to cope with increasingly complex cyber security problems.

I. INTRODUCTION

IoT expansion has increased exposure to cyber attacks, motivating adaptive defenses. The paper surveys how DRL combines deep learning with reinforcement learning for cyber security and organizes applications across major defense settings.

I. INTRODUCTION: IoT adoption across critical sectors and advances in connected technologies have increased vulnerability to cyber attacks and the need for cyber security technologies.The cited passage describes applications spanning telecommunications, transportation, manufacturing, healthcare, finance, government, and entertainment.
I. INTRODUCTION: Machine learning supports both cyber attack and defense, including intrusion detection, malware detection, cyber-physical attack protection, and data privacy protection.
II. DEEP REINFORCEMENT LEARNING PRELIMINARY: DRL combines reinforcement learning’s sequential decision-making with deep learning’s function approximation and representation learning for complex, high-dimensional problems.RL learns through interaction with an environment using states, actions, and rewards, while deep learning supports compact representations of high-dimensional data.
I. INTRODUCTION: The survey focuses on DRL for defending against cyber attacks and groups applications into CPS security, autonomous intrusion detection, and game-theoretic cyber defense.It distinguishes defense applications from DRL for optimizing IoT communications and networking capabilities.
II. DEEP REINFORCEMENT LEARNING PRELIMINARY: Q-learning maximizes discounted cumulative reward, but lookup tables become inefficient for large or continuous state and action spaces.The discount factor γ controls the importance of future rewards; deep learning addresses scalability through function approximation and representation learning.
II. DEEP REINFORCEMENT LEARNING PRELIMINARY: DQN stabilizes deep Q-learning with experience replay and a target network, while actor-critic methods combine value estimation with direct policy learning.DQN randomly reuses stored experience tuples, whereas actor-critic methods address limitations associated with value-based and policy-gradient approaches.

III. DRL IN CYBER SECURITY: A SURVEY

Traditional reinforcement learning has been applied across cyber security, but its limitations constrain complex, large-scale problems. This survey examines DRL-powered cyber-security solutions and identifies areas not yet covered by existing applications.

Traditional RL applications span data privacy and critical infrastructure protection, but drawbacks restrict their use for complex, large-scale cyber-security problems.
DRL solutions address sophisticated attacks including falsified data injection, deception, distributed denial-of-service, intrusion, jamming, spoofing, malware, and adversarial networking attacks.
The survey covers DRL applications ranging from cyber-physical-system defense to autonomous intrusion detection and game-theory-based solutions.
The survey is limited to existing DRL applications in cyber security, while other topics are reserved for discussion and future research.

A. DRL-based Security Methods for Cyber-Physical Systems

DRL-based CPS security methods model cyber attacks and defenses within dynamic physical systems, including game-based defense and automated defect detection. Surveyed applications also address autonomous vehicles exposed to manipulated sensor information.

CPS applications span manufacturing, health monitoring, smart grids, and transportation, where computer-controlled physical systems communicate through shared networks.
DRL models CPS cyber-state dynamics by incorporating physical states, control inputs, disturbances, attacks, and defenses into a cyber-physical model.
CPS defense is modeled as a two-player zero-sum game, with an actor-critic DRL defender learning timely and accurate strategies against unknown cyber attacks.
Double DQN and A3C automatically find falsified inputs for CPS models, outperforming simulated annealing and cross entropy through fewer simulation runs.
Autonomous vehicles face cyber-physical attacks that manipulate sensor data, while DRL methods select suitable velocities or learn adversarial interactions to improve robust control.

B. DRL-based Intrusion Detection Systems

Intrusion detection systems analyze host or network audit data using signature-based or anomaly-based methods, but growing data volumes and complex temporal intrusions challenge conventional approaches. DRL architectures support adaptive analysis of these security environments.

Intrusion detection systems analyze audit data to identify and report abnormal or malicious activity on host computers or network equipment.
Manual intrusion detection becomes difficult as network size increases because audit-data volume and intrusion complexity grow rapidly.
Host-based IDSs inspect host logs or settings, whereas network-based IDSs collect and examine transmitted packets over network segments.
An adversarial DRL architecture uses an LSTM, fully connected layer, and regression to approximate Q functions from past player interactions for robust autonomous-vehicle control.
Signature detection compares activity with known attack patterns, while anomaly detection flags deviations from observed normal behavior.

1) Host-based IDS:

Host-based intrusion-detection research addresses temporal and high-dimensional intrusion behaviors using reinforcement-learning and kernel-based temporal-difference methods.

Adaptive intrusion detection has limited effectiveness when complex intrusions consist of temporal sequences rather than isolated labeled or unlabeled data.
Kernel least-squares TD enhances generalization in high-dimensional nonlinear feature spaces and predicts anomaly probabilities for multi-stage attacks.

2) Network-based IDS:

Network-based IDS research addresses known and unknown intrusions, DDoS, and coordinated defense using reinforcement-learning and game-theoretic approaches. Reported methods improve adaptability, but multiagent convergence remains difficult.

Network-based IDS: Combining anomaly- and signature-based detection aims to identify both known and unknown attacks.Anomaly detection can generate false alarms, whereas signature detection misses new attack types.
Network-based IDS: Coordinated multiagent SARSA throttling uses task decomposition, hierarchical communication, and team rewards to improve scalability against DDoS floods.The approach involves defensive nodes across different locations and was evaluated with a network emulator.
Network-based IDS: A multiagent reinforcement-learning system lets agents learn policies from local databases and information exchanged with other agents for quick responses to complex attacks.Agents use decisions and events received from other agents as part of their learning information.
Network-based IDS: A fuzzy Q-learning, game-theoretic IDS uses coordinated sink nodes and a base station in separate detection and defense phases against application-layer DDoS attacks.The sink node detects anomalies and checks malicious information against a threshold before the base station selects defense actions.
Network-based IDS: Adversarial reinforcement learning addresses class imbalance in network intrusion datasets through oversampling, while DDQN can match or exceed traditional machine-learning methods in some cases.The adversarial environment treats labelled-dataset samples as reinforcement-learning states.
Network-based IDS: Reinforcement-learning IDSs can adapt to environmental changes, but multiagent systems do not guarantee an optimal solution because convergence is difficult.The limitation concerns the difficulty of obtaining convergence in multiagent systems.

C. DRL-based Game Theory for Cyber Security

Traditional cyber security mechanisms often lag behind dynamic attacks because cyberspace contains interacting components whose decisions affect one another. These interactions substantially enlarge the security decision space.

C. DRL-based Game Theory for Cyber Security: Firewalls, antivirus software, and intrusion detection are characterized as passive, unilateral, and lagging behind dynamic attacks.The paper contrasts these methods with the need to account for interactions among cyber components.
C. DRL-based Game Theory for Cyber Security: Security policies applied to one cyber component can influence decisions taken by other components.Accounting for these interactions increases the size of the decision space.

1) Jamming attacks:

Jamming-defense studies use reinforcement learning, DQN, and multiagent methods to adapt transmission and offloading decisions in hostile wireless environments. Game-theoretic policies improve anti-jamming performance, while state-space limitations remain a constraint for some approaches.

Jamming attacks: Jamming is treated as a special case of denial-of-service that reduces or eliminates a network’s ability to perform its expected function.The survey describes jamming mitigation as an active application area for machine learning and reinforcement learning.
Jamming attacks: DQN handles high-dimensional secure-offloading states by selecting parameters such as offloading rate, time, channel, and transmit power.The method targets edge offloading under jamming and other attacks; hotbooting initializes CNN weights using prior experience.
Jamming attacks: Multiagent reinforcement learning enables wideband cognitive radios to learn sub-band selection policies for avoiding jamming.Hardware-in-the-loop simulation showed detection of jamming patterns and successful learning of an optimal sub-band policy.
Jamming attacks: In cognitive radio networks, secondary users must avoid primary-user disruptions and contend with smart jammers that adapt to transmission strategies.RL methods model secondary users as intelligent agents that select transmission powers against jamming.
Jamming attacks: Simulation results show improved anti-jamming SINR, while a Stackelberg-game strategy minimizes jammer damage in the worst-case scenario.The reported optimal strategy is obtained using Q-learning and WoLF-PHC.
Jamming attacks: DQN-based frequency-spatial anti-jamming games model many jammers while combining frequency hopping and user mobility.Secondary users must protect primary-user communications while defeating smart jammers.
Jamming attacks: Using spectrum-waterfall data with temporal features avoids requiring prior knowledge of jamming patterns.This state representation differs from approaches based on SINR and primary-user occupancy.

2) Spoofing attacks:

Spoofing attacks use forged identities to gain illegitimate network access, potentially enabling man-in-the-middle or denial-of-service attacks. Q-learning and Dyna-Q model receiver–spoofer interactions as a zero-sum authentication game.

Spoofing attacks: Spoofing attackers claim another node’s identity, such as its MAC address, to obtain illegitimate network access.The resulting penetration may lead to man-in-the-middle or denial-of-service attacks.
Spoofing attacks: Q-learning and Dyna-Q address spoofing detection by modelling legitimate receivers and spoofers as players in a zero-sum authentication game.Receiver and spoofer utility is computed from Bayesian risk, representing expected payoff in spoofing detection.

3) Malware attacks:

The surveyed approaches apply learning and adversarial game models to malware detection, adaptive network defense, and cloud offloading. These methods address changing attacks, incomplete information, and operational trade-offs such as congestion and detection delay.

Cloud-based malware detection: Cloud offloading can improve malware detection speed and accuracy, but excessive offloading may congest the radio network and increase detection delay.Mobile devices have limited computational power, battery life, and bandwidth, whereas cloud servers provide stronger resources and updated malware databases.
SDN defense: Software-defined networking supports adaptive reconfiguration, while reinforcement learning has been broadly demonstrated for SDN control.Traditional direct client-server communication can make reconfiguration time-consuming and inefficient.
Cloud-based malware detection: DQN selects dynamic mobile task-offloading rates using network status and malware detection reports to avoid congestion and detection delay.Detection runs on a more powerful cloud server, while the agent formulates states, rewards, and sequential actions from network conditions and server utility.
SDN defense: Adversarial RL builds autonomous SDN defense against attackers that compromise important network nodes and may falsify the defender’s training process.The attacker can target backbone or target-subnet nodes and propagate through the network.
Adversarial environments: Defenders must dynamically reconfigure strategies because they may lack the attacker’s type, target, frequency, and location.Repeated defender-attacker strategy changes can model defenses against attack classes including buffer over-read and code reuse attacks.
Adversarial environments: Adaptive RL addresses non-persistent attackers with diminishing exploration, whereas robust RL addresses persistent attackers with constant exploration.The model assumes awareness of attacker classes and access to utilities jointly contributed by defense and attack activities.
Adversarial environments: Stochastic Markov-game simulations model cyber security as two-agent interaction under incomplete information and partial observability.The attacker seeks a valuable asset without knowing network topology, while the defender knows the internal network but not attack types or intruder positions.

IV. DISCUSSIONS AND FUTURE RESEARCH DIRECTIONS

The survey reviews DRL applications in cyber security and identifies cyber-physical-system security as an emerging area. It also highlights realistic simulation and sim-to-real transfer as major implementation challenges.

Survey scope: The paper presents a comprehensive survey of DRL methods and cyber security applications, including game-theoretic models with multiple DRL agents.The review summarizes notable examples and discusses the adversarial nature of cyber systems.
CPS security: DRL-based CPS security targets large-scale, complex systems such as environmental monitoring, smart grids, transportation, and cyber manufacturing.The surveyed approaches include TRPO, LSTM-Q-learning, double DQN, and A3C.
Challenges: Realistic CPS simulations remain scarce, and integrating Matlab/Simulink models with OpenAI Gym can impose substantial computational overhead.The survey encourages CPS models embedded directly in DRL-enabled environments.
Challenges: Transferring policies trained in simulations to real-world environments remains a common challenge for DRL-based CPS security.The passage contrasts simulation’s safety and low cost for training with the difficulty of real-world deployment.

B. Challenges and future work on applying DRL for IDS

DRL for intrusion detection is promising but constrained by limited live training environments, data inefficiency, unrealistic simulators, and adversarial manipulation. Future work emphasizes richer environments, model-based learning, continuous actions, and adversarial DRL.

IDS challenges: DRL intrusion-detection research remains limited despite extensive traditional-RL work, partly because integrating deep learning with RL is relatively recent.DRL combines representation learning and function approximation with sequential decision-making.
IDS challenges: Training IDS agents requires simulated or real interactive intrusion environments, but real environments are costly and simulations may be unrealistic.Existing studies often rely on game settings or labelled intrusion datasets, including supervised adaptations that lack a live environment.
Model-based DRL: Most cyber-defense DRL algorithms are model-free and sample inefficient, requiring large quantities of training data that are difficult to obtain in practice.Researchers commonly use simulators, which may not fully characterize real IoT cyberspace complexity and dynamics.
Model-based DRL: Model-based or hybrid DRL is proposed for limited-data settings through scalable data collection, learned proxy models, and planning methods such as Monte-Carlo tree search.The survey also discusses model-free policies augmented with planning capabilities.
Future directions: Continuous-action methods are needed because discretizing mobile offloading rates can miss performance effects from small rate changes.The survey identifies policy-gradient and actor-critic algorithms as relevant approaches.
Adversarial DRL: Attackers can bypass detection, poison training data, or manipulate RL states, policies, and rewards to induce sub-optimal actions.The survey characterizes these attacks as difficult to prevent, detect, and counter, while adversarial RL remains sparsely studied.

E. Human-machine teaming with human-on-the-loop models

The survey argues that AI-assisted cyber defense needs human-machine teaming because autonomous systems scale response but cannot provide creative responses to new threats. It presents human-on-the-loop operation as a future direction alongside larger-scale multiagent defense.

Human-machine teaming: AI-enabled defenses can process and deploy responses rapidly, but they cannot alone issue creative responses when new threats appear.The survey therefore identifies a need to combine human intellect with machines.
Human-machine teaming: Human-on-the-loop models let agents act autonomously while humans monitor and intervene only when necessary.This is presented as a future alternative to human-in-the-loop systems that pause for human responses.
Multiagent DRL: Multiagent DRL is proposed for increasingly sophisticated and large-scale attacks, but current game-theoretic models involve few agents with limited communication, cooperation, and coordination.The survey identifies non-stationarity, partial observability, and efficient multiagent training as additional challenges.
Emerging threats: Recently emerged attacks remain largely unaddressed by RL-based cyber-defense research.Examples include bit-and-piece DDoS attacks, cloud-origin attacks, and attacks using powerful quantum computers to crack encryption.