PaperlayerView exact revision on arXiv

Source-linked AI summary

Applications of Deep Reinforcement Learning in Communications and Networking: A Survey

Nguyen Cong Luong, Dinh Thai Hoang, Shimin Gong, Dusit Niyato, Ping Wang, Ying-Chang Liang, Dong In Kim

arXiv:1810.07862v1cs.NIcs.LG

TL;DR

The paper addresses the lack of a survey focused specifically on deep reinforcement learning applications in communications and networking. It provides a tutorial and comprehensive review of methods across networking issues, then identifies challenges, open issues, and future research directions.

  • Problem

    Existing surveys cover deep reinforcement learning in other fields or deep learning in networking, leaving applications of deep reinforcement learning in communications and networking insufficiently surveyed.

  • Method

    The paper combines tutorials on reinforcement learning, deep learning, and deep reinforcement learning with a literature review, analysis, and comparison of applications across communications and networking.

  • Results

    The survey covers deep reinforcement learning applications for dynamic network access, data rate control, caching, offloading, security, connectivity, traffic routing, resource sharing, and data collection.

  • Takeaways & Limitations

    Deep reinforcement learning supports complex network optimization, local policy learning under uncertain environments, and autonomous decision-making with reduced information exchange.

  • Takeaways & Limitations

    Training and evaluating deep reinforcement learning frameworks remains constrained by limited real-world wireless data and reliance on simulated datasets that may overlook hidden patterns.

Abstract

from arXiv · show

This paper presents a comprehensive literature review on applications of deep reinforcement learning in communications and networking. Modern networks, e.g., Internet of Things (IoT) and Unmanned Aerial Vehicle (UAV) networks, become more decentralized and autonomous. In such networks, network entities need to make decisions locally to maximize the network performance under uncertainty of network environment. Reinforcement learning has been efficiently used to enable the network entities to obtain the optimal policy including, e.g., decisions or actions, given their states when the state and action spaces are small. However, in complex and large-scale networks, the state and action spaces are usually large, and the reinforcement learning may not be able to find the optimal policy in reasonable time. Therefore, deep reinforcement learning, a combination of reinforcement learning with deep learning, has been developed to overcome the shortcomings. In this survey, we first give a tutorial of deep reinforcement learning from fundamental concepts to advanced models. Then, we review deep reinforcement learning approaches proposed to address emerging issues in communications and networking. The issues include dynamic network access, data rate control, wireless caching, data offloading, network security, and connectivity preservation which are all important to next generation networks such as 5G and beyond. Furthermore, we present applications of deep reinforcement learning for traffic routing, resource sharing, and data collection. Finally, we highlight important challenges, open issues, and future research directions of applying deep reinforcement learning.

I. INTRODUCTION

The introduction motivates deep reinforcement learning for decentralized, autonomous, and large-scale networks, then outlines its applications and advantages in communications and networking.

  • Motivation: Reinforcement learning can require excessive exploration to reach good policies, making it unsuitable for large-scale networks with complex state and action spaces.Deep learning is introduced to address this limitation through deep reinforcement learning.
  • Motivation: Modern IoT, heterogeneous, and UAV networks require local autonomous decisions such as spectrum access, rate selection, power control, and base-station association.These decisions must be made in decentralized and dynamic environments.
  • Benefits of DRL: DRL can solve sophisticated non-convex network optimizations without complete and accurate network information.Examples include joint user association, computation, and transmission scheduling.
  • Benefits of DRL: DRL enables network entities to learn policies such as channel selection, handover, caching, and offloading without knowing channel or mobility models.The approach builds knowledge from observations of the communication and networking environment.
  • Benefits of DRL: DRL supports autonomous local decisions with minimal information exchange, reducing communication overheads while improving network security and robustness.It also improves learning speed for problems with large state and action spaces.
  • Survey scope: The survey covers DRL foundations, advanced models, and applications including network access, rate control, caching, offloading, security, connectivity, routing, and data collection.It also discusses challenges, open issues, and future research directions.

1) Partially Observable Markov Decision Process:

The section introduces partially observable and multi-agent decision settings, then reviews Q-learning and SARSA as methods for learning policies through interaction with an environment.

  • Partially Observable Markov Decision Process: A POMDP models decisions when an agent observes only part of the system state.Its tuple extends the MDP with observations and observation probabilities.
  • Markov Games: A Markov game models probabilistic transitions among multiple agents with individual action spaces and payoff functions.Agents act simultaneously, receive rewards and new observations, and seek long-term rewards.
  • Q-Learning: Q-learning updates action values iteratively from rewards and successor states to derive an optimal policy.Under the stated step-size assumption, Q-learning converges to optimum action-values with probability one.
  • Q-Learning: The Q-learning algorithm interacts with the environment, stores state-action updates, and outputs the action maximizing the learned Q-value.It terminates after convergence or a specified iteration count.
  • SARSA: SARSA provides an online, on-policy alternative that updates from actions actually taken and can select actions without waiting for convergence.Q-learning is described as offline and off-policy by comparison.

3) Q-Learning for Markov Games:

The section extends Q-learning to general-sum Markov games by modeling each agent’s action values and assuming Nash-equilibrium behavior, but requires restrictive assumptions for convergence guarantees.

  • Q-Learning for Markov Games: For agent i, the Markov-game Q-function conditions on its action and the actions of all other agents.The notation a−i denotes the actions of every agent except i.
  • Q-Learning for Markov Games: The Nash Q-function uses immediate reward plus discounted future value under joint equilibrium strategies.The future term represents total discounted reward over an infinite horizon.
  • Q-Learning for Markov Games: The proposed multi-agent algorithm updates Q-values by assuming Nash-equilibrium behavior over current Q-values.Each agent observes the state, its reward, other agents’ actions, and their rewards during learning.
  • Q-Learning for Markov Games: Agents must estimate other agents’ Q-values because the information needed to calculate the Nash equilibrium is not directly available.These estimates are initialized and updated from observed rewards and previous actions.
  • Q-Learning for Markov Games: The multi-agent Q-learning algorithm is guaranteed to converge only under highly restrictive assumptions about the state games during learning.The passage states the guarantee without specifying those assumptions in full.

C. Deep Learning

The section presents deep learning as automatic feature and abstraction learning through neural networks, then contrasts feedforward and recurrent architectures, including CNNs and LSTMs.

  • Deep Learning: Deep learning seeks important data features and high-level abstractions without manually specifying the data structure.Its models commonly use neural networks with multiple hidden layers.
  • Deep Learning: An artificial neural network uses interconnected input, hidden, and output layers to support tasks such as classification, prediction, and decision-making.Neurons combine weighted inputs with activation functions to produce outputs.
  • Architectures: Feedforward networks pass information in one direction, while recurrent networks use directed cycles and retain information over time.CNNs are a prominent feedforward model, whereas RNNs are designed for sequential data.
  • Architectures: Standard RNNs can struggle with long-term temporal dependencies because their loss gradients decay exponentially over time.LSTMs are used to model temporal sequences and long-range dependencies more accurately than conventional RNNs.

D. Deep Q-Learning

Deep Q-learning replaces tabular value estimates with a deep network for large state and action spaces, while replay, target networks, and advanced architectures address instability and estimation errors.

  • Deep Q-Learning: DQL uses a DQN to approximate Q∗(s, a) when complicated systems make tabular state and action spaces too large for Q-learning.The DQN is a deep neural network replacing the Q-table.
  • Deep Q-Learning: Experience replay and a fixed target Q-network reduce correlations and shifting-target instability during nonlinear Q-value learning.Replay trains on randomly sampled transitions, while the target network is updated more slowly.
  • Advanced Deep Q-Learning Models: Double Q-learning uses separate value functions to select and evaluate actions, addressing the positive bias behind Q-learning’s action-value overestimation.DDQL replaces the second network with target-network weights for evaluation and has been applied to spectrum access and resource allocation.
  • Advanced Deep Q-Learning Models: 41 out of 49 Atari games showed better performance for prioritized-experience-replay DQL than uniformly replayed DQL.The approach prioritizes transitions but is appropriate only when important experiences can be identified.
  • Advanced Deep Q-Learning Models: 50 out of 57 learned Atari games showed dueling DQN outperforming DDQN, but benefits were clearest for MDPs with large action spaces.For small state spaces, dueling DQL performed worse than double DQL in the cited simulations.

4) Asynchronous Multi-step Deep Q-Learning:

Asynchronous multi-step DQL uses multiple agents to train a neural network in parallel, reducing training time while stabilizing policy and value updates. Its performance was evaluated against several reinforcement-learning methods and state-of-the-art Atari algorithms.

  • Asynchronous multi-step learning: Multiple agents train the DNN in parallel to address experience replay’s memory, computation, and off-policy requirements.This approach is introduced to overcome limitations that restrict DQL applications.
  • Asynchronous multi-step learning: Asynchronous updates stabilize policy and value updates across 1-step Q-learning, 1-step SARSA, and n-step Q-learning.The effects of training speed and quality were analyzed for these reinforcement-learning methods.
  • Performance: Asynchronous DQL outperforms current state-of-the-art Atari algorithms while training for half the time on a single multi-core CPU instead of a GPU.The result connects the method’s performance with reduced training time and CPU-based execution.

5) Distributional Deep Q-learning:

Distributional DQL models return distributions rather than expected values, addressing stochastic environments with multimodal future rewards. The surveyed extensions improve Atari performance in several settings, but their effectiveness depends on model choices and requires validation beyond Atari.

  • Distributional DQL: Distributional reinforcement learning updates the Q-value function from its return distribution rather than its expectation.This addresses cases where expected values may not identify the optimal outcome in stochastic environments with multimodal rewards.
  • Performance: Distributional deep Q-learning outperforms conventional DQL on 45 of 57 Atari 2600 games.Its performance is substantially better when the distribution function Z is well defined, but can be worse otherwise.
  • Noisy Nets: Adding Gaussian noise layers significantly improves conventional, dueling, and asynchronous DQL across a wide range of Atari games.The survey notes that the impact of noise remains debated and requires further investigation.
  • Rainbow DQL: Rainbow DQL integrates seven DQL improvements, including asynchronous multi-step, distributional, and double Q-learning components.The combined method uses a multi-step distributional loss and selects the bootstrap action greedily through the Q-network.
  • DQL extensions: Noisy Net, prioritized replay, dueling architectures, and related Rainbow components modify exploration, transition prioritization, and value estimation.Rainbow variants use KL loss for transition prioritization, while Noisy Net replaces linear layers with noise-aware layers.
  • Rainbow DQL: Rainbow DQL achieves impressive Atari 2600 results, although further experiments in different domains are needed to confirm its efficiency.The survey reports the comparison in Table II and identifies cross-domain validation as an open need.

F. Deep Q-Learning for Extensions of MDPs

Extensions of DQL address continuous actions, partial observability, online learning, and multiagent games. The reviewed methods adapt actor-critic policies, recurrent state estimation, SARSA updates, and self-play to these settings.

  • Continuous actions: DQL handles high-dimensional states but is limited to discrete, low-dimensional actions, motivating actor-critic methods for continuous action spaces.A continuous action space makes exhaustive maximization of the Q-value impractical.
  • Continuous actions: DDPG uses a deterministic actor to map states to actions and a critic that estimates Q(s, a) with Bellman-based learning.Target actor and critic networks are slowly updated using a factor τ much smaller than 1.
  • Partial observability: DRQN replaces DQN’s first post-convolutional fully connected layer with an LSTM to integrate observation history in partially observable environments.DRQN matches DQN on standard MDPs and outperforms it in partially observable domains.
  • Online learning: Deep SARSA selects actions online with ε-greedy learning, observes rewards and next states, and updates Q-values from subsequent state-action values.This provides an online alternative to Q-learning, which obtains the optimal policy only after Q-values converge.
  • Markov games: Multiagent DRL for sequential prisoner’s dilemma games uses offline and online phases to learn mutual cooperation policies.Representative policies are trained first and then used to synthesize policies spanning cooperation degrees.
  • Markov games: NFSP combines fictitious self-play with neural-network function approximation to learn approximate Nash equilibria in imperfect-information games.Agents maintain separate memories for reinforcement learning and supervised classification, then act from average and best-response strategies.
  • Scope: The surveyed DQL extensions target different problems and network scenarios rather than a single universal setting.The section introduces extensions spanning continuous control, partial observability, online learning, and multiagent interaction.

III. NETWORK ACCESS AND RATE CONTROL

The survey reviews DQL applications for decentralized network access and adaptive decisions under incomplete, dynamic, or uncertain network information. Covered problems include spectrum access, user association, channel allocation, packet forwarding, and energy-aware control.

  • Dynamic spectrum access: DQL enables distributed solutions for non-convex joint user association and spectrum access without complete network information.
  • Dynamic spectrum access: In IoT channel access, DQL maps observed channel conditions and buffer states to channel-selection and forwarding actions.
  • Dynamic spectrum access: DQL jointly selects forwarding channels, packet counts, and modulation modes while using stacked autoencoders to reduce Q-learning computation and storage.The proposed scheme converges and improves utility over random action selection, although utility decreases as packet arrival rates increase.
  • Dynamic spectrum access: DQL-based energy-aware channel selection outperformed a myopic policy in total rate, with battery prediction error close to zero.The reward combines total rate with prediction error, and the DQN uses two LSTM-based layers.
  • Dynamic spectrum access: A proposed DQN for joint dynamic channel access and interference management requires synchronization between small base stations and WLANs.
  • Dynamic spectrum access: 24.4% traffic improvement was achieved over fixed channel allocation for newly arriving users in a multibeam satellite system.The advantage became insignificant when increasing numbers of current users left few or no available channels.

2) Joint User Association and Spectrum Access:

Joint user association and spectrum access are modeled as difficult, non-convex decisions in heterogeneous and UAV-supported networks. DQL provides distributed decision-making, with extensions that incorporate content caching and request prediction.

  • Joint User Association and Spectrum Access: DQL addresses non-convex joint user association and spectrum access through distributed solutions that do not require complete and accurate network information.
  • Joint User Association and Spectrum Access: A UAV-enabled LTE network extends the joint problem to user association, spectrum access, and content caching.The UAVs act as cache-enabled LTE base stations with access to licensed and unlicensed bands.
  • Joint User Association and Spectrum Access: Liquid State Machine prediction estimates users’ content-request distributions before each UAV applies learning-based resource allocation.

B. Adaptive Rate Control

The survey covers adaptive bitrate control in DASH and multi-objective rate control in unpredictable communication environments, alongside caching methods for high-dimensional content-placement decisions. These approaches use DQL to optimize QoE, multiple transmission objectives, and cache management.

  • Adaptive Rate Control: DASH clients choose video representations as MDP actions, with rewards combining visual quality, stability, rebuffering, and buffer state.
  • Adaptive Rate Control: 25% average QoE improvement over a bitrate-control scheme was reported for A3C-based DQL using a mobile dataset.The approach also uses buffer capacity to handle throughput fluctuations.
  • Adaptive Rate Control: A CNN-RNN video-quality prediction network reduces the raw-segment state space before its output is supplied to DQL.
  • Adaptive Rate Control: DQL supports multiple conflicting rate-control objectives, including BER, throughput, power efficiency, and spectral efficiency, in unpredictable space communications.
  • Adaptive Rate Control: DQL caching agents use high-dimensional request-history features to decide whether to store requested content and which cached content to replace.
  • Adaptive Rate Control: DDPG with Wolpertinger architecture combines actor, K-NN, and critic components to reduce the effective caching action-space challenge.
  • Adaptive Rate Control: Delayed Experience Injection tracks incomplete transitions when cache rewards and next states are not immediately available at runtime.The method was evaluated with a Yahoo! cloud-serving benchmark and customized workloads.

2) Joint Caching and Transmission Control:

Joint caching and transmission control couples content placement with interference management, QoE, energy efficiency, networking, and computation. The reviewed DQL frameworks target these coupled objectives across wireless, heterogeneous, vehicular, and smart-city networks.

  • Joint Caching and Transmission Control: DQL frameworks address joint caching and interference alignment in MIMO systems with limited backhaul capacity and transmitter-side caches.
  • Joint Caching and Transmission Control: DQL jointly optimizes cache allocation and transmission rate to maximize IoT-device QoE in content-centric wireless networks.PER improves sample efficiency, while DDQN separates value functions to stabilize learning and reduce overestimation.
  • Joint Caching and Transmission Control: Heterogeneous networks require unified handling of mutual interference, differentiated QoS, and resource allocation across multiple communication technologies.
  • Joint Caching and Transmission Control: DQL frameworks jointly consider networking, in-network caching, and computation for energy-efficient resource allocation in green wireless networks.
  • Joint Caching and Transmission Control: Dueling-DQN enhancements decompose state-action values into state value and relative action advantage, improving performance and training speed.
  • Joint Caching and Transmission Control: Multitime-scale DQN jointly designs communication, caching, and computing in VANETs under mobility, storage, computation, and service-deadline constraints.
  • Joint Caching and Transmission Control: The VANET framework was generalized to smart-city applications requiring dynamic orchestration of networking, caching, and computation.

B. Data and Computation Offloading

The survey reviews DQL for data and computation offloading across cellular, MEC, fog, and mobile-cloudlet settings. These approaches formulate dynamic allocation and offloading decisions as sequential control problems under resource, delay, energy, and mobility constraints.

  • IoT devices can offload computation to MEC servers to reduce processing delay and battery energy for computation-intensive applications.
  • Offloading decisions span WLAN access, single or shared MEC servers, multiple MEC servers, and mobile cloudlets.
  • In ultra-dense networks, DQL jointly learns user association, computational-resource allocation, transmission control, and offloading policies from channel, energy, and queue states.
  • For cloud-based malware detection, DQL improves detection speed and accuracy while increasing mobile-user battery life.
  • DQL models fog-container migration as a multidimensional MDP whose states include delay, power consumption, and migration cost.
  • The survey observes that caching DQL is typically centralized, whereas offloading spans diverse network architectures and resource-allocation settings.

A. Network Security

The survey covers DQL applications to jamming and cyber-physical attacks in wireless, UAV, vehicular, and IoT systems. These approaches learn policies from observations to select channels, mobility, power, relay, control, or detection actions under attack.

  • 1) Jamming Attack: DQL learns anti-jamming policies for channel hopping, mobility, transmit-power control, and UAV relaying.
  • 1) Jamming Attack: RCNN-based DQL preprocesses noisy SINR observations into extracted features and converges in fixed and dynamic jamming scenarios where Q-learning fails in the dynamic case.
  • 1) Jamming Attack: 200 time slots yields convergence to the optimal UAV relay policy, 83.3% less than Q-learning, while BER decreases 46.6% versus hill climbing.
  • 1) Jamming Attack: The UAV-security DQL improves utility by up to 13% and safe rate by 7% compared with WoLF-PHC.
  • 2) Cyber-Physical Attack: For cyber-physical attacks, DQL helps autonomous vehicles select measurement weights and find equilibrium strategies in a zero-sum attacker–vehicle game.

B. Connectivity Preservation

DRL is applied to preserve connectivity in robot and UAV networks by learning velocity or control policies from network states. The reviewed work extends from two-robot connectivity to multi-robot systems, while also addressing safety, handover, and self-healing concerns.

  • B. Connectivity Preservation: A DQN enables a base station to keep a follower UAV within a leader’s communication range, outperforming the APF method.The model uses relative position and velocity as states, follower velocity as actions, and rewards connectivity.
  • B. Connectivity Preservation: Actor-critic-based DQN control keeps multiple followers aligned with dynamically changing leader trajectories by maintaining positive algebraic connectivity.More followers increase the action space and make convergence slower than in the two-robot case.
  • B. Connectivity Preservation: A minimum-distance penalty is added to prevent leaders and followers from colliding when connectivity-preservation policies bring them too close.Earlier schemes did not explicitly constrain the distance between leaders and followers.
  • B. Connectivity Preservation: DQL-based self-healing adjusts transmit diversity and antenna azimuth after network faults, achieving throughput close to oracle-based control with less fault-message overhead.The MBS selects corrective actions from observed alarms and receives rewards based on fault counts.
  • B. Connectivity Preservation: Cyber-physical security remains relatively underexplored compared with anonymous robot and intelligent transportation systems.The section observes that CNNs are commonly used for DQL-based network security.

A. Traffic Engineering and Routing

DRL is reviewed as a model-free approach for traffic engineering, routing, resource sharing, and scheduling in dynamic wireless and computing environments. The surveyed methods use DQL or policy-gradient techniques to optimize paths, coverage, capacity, and resource objectives.

  • A. Traffic Engineering and Routing: DQL learns routing paths for source-destination pairs from bandwidth requests and rewards based on mean network delay.The agent interacts with the network environment at a controller rather than relying solely on a fixed model.
  • A. Traffic Engineering and Routing: TE-aware exploration and actor-critic-based prioritized experience replay significantly reduce end-to-end delay and improve network utility over DDPG and NUM baselines.The approach incorporates shortest-path and NUM solutions during exploration.
  • A. Traffic Engineering and Routing: Actor-critic DRL models UAV navigation and multi-UAV path planning under obstacle, energy-efficiency, latency, interference, and association considerations.Multi-agent DRL casts interactions among UAVs as a dynamic game.
  • B. Resource Sharing and Scheduling: Policy-gradient scheduling achieves better network coverage and capacity than optimization-based algorithms operating with incomplete network information.The policy is learned directly from trajectories using a REINFORCE variant.
  • B. Resource Sharing and Scheduling: DeepRM learns cluster-resource scheduling from experience, matching or exceeding heuristics while adapting to conditions and converging quickly without prior system knowledge.Its objectives include reducing average job slowdown or completion time.

C. Power Control and Data Collection

The survey covers DRL for sensing, data collection, social-network information extraction, and related wireless control problems. These applications use learned policies to manage sensing locations, vehicle movement, incentives, and dynamic network decisions.

  • C. Power Control and Data Collection: DRL supports spectrum sensing, power control, and actor movement or sensing policies in cognitive-radio and wireless sensor networks.The reviewed systems operate with distributed sensing or shared sensor information.
  • C. Power Control and Data Collection: DRQN uses LSTM layers to capture temporal state-transition patterns, while transfer learning reduces training data for correlated mobile-crowdsensing tasks.The approach selects sensing cells and is evaluated with real sensor-network datasets.
  • C. Power Control and Data Collection: A DQL controller schedules unmanned-vehicle data collection across priority-weighted subregions under limited energy supply.The state includes sample priorities, charging-point location, and vehicle movement traces.
  • C. Power Control and Data Collection: DRL applications span diverse wireless and networked systems involving dynamic modeling and multi-agent interactions with large state-transition and action spaces.The survey identifies DQL and policy-gradient methods as common choices for discrete and continuous control, respectively.
  • C. Power Control and Data Collection: Users may need frequent local-state reporting, while dense deployments make temporary base-station selection from similar RSSIs challenging.This creates a practical limitation for DRL access-policy approaches.

4) Training and Performance Evaluation of DRL Framework:

The survey identifies practical constraints around training data, distributed implementation, information collection, and rapidly changing wireless conditions. It concludes by organizing DRL applications, challenges, open issues, and future research directions across communications and networking.

  • 4) Training and Performance Evaluation of DRL Framework:: Most wireless DRL studies rely on simulated datasets because large real-world training and evaluation data are difficult to access.Simulation based on simplified stochastic models can reduce confidence in practical deployment.
  • 4) Training and Performance Evaluation of DRL Framework:: Distributed DRL must separate resource-intensive information collection, sharing, and DNN training from individual devices while controlling information-exchange overhead.The survey identifies supporting network infrastructure as an open design issue.
  • 4) Training and Performance Evaluation of DRL Framework:: DRL systems must balance information quality against gathering costs such as delay, preprocessing, energy consumption, and reduced learning speed.The open issue is avoiding substantial resource costs for only marginal learning-performance gains.
  • 4) Training and Performance Evaluation of DRL Framework:: Massive-MIMO channel estimation remains challenging in large-scale heterogeneous cellular networks, motivating DRL for time-varying wireless channels.The survey frames channel estimation as a prerequisite for realizing massive-MIMO communications.
  • 4) Training and Performance Evaluation of DRL Framework:: The paper surveys DRL fundamentals, techniques, applications, comparisons, challenges, open issues, and future directions in communications and networking.Covered issues include access, rate control, caching, offloading, security, connectivity, routing, resource sharing, and data collection.
Loading 1810.07862v1…