PaperlayerView exact revision on arXiv

Source-linked AI summary

Single and Multi-Agent Deep Reinforcement Learning for AI-Enabled Wireless Networks: A Tutorial

Amal Feriani, Ekram Hossain

arXiv:2011.03615v1cs.LGcs.NI

TL;DR

Wireless networks need scalable and decentralized learning for multi-agent 6G systems, while practical deployment also faces model-learning and convergence challenges. This tutorial synthesizes DRL, MBRL, and cooperative MARL methods and reviews their wireless applications. It concludes by identifying heterogeneity, mobility, robustness, limited data, and theoretical guarantees as important research directions.

  • Problem

    Future 6G technologies involve multiple interacting agents, but single-agent RL does not adequately model scalable, self-organizing systems and multi-agent learning loses single-agent convergence guarantees under non-stationarity.

  • Method

    The tutorial presents mathematical frameworks for RL and MARL, explains MBRL planning and cooperative MARL, and reviews decentralized wireless applications and research directions.

  • Results

    The paper synthesizes how MBRL and cooperative MARL can support practical, self-organizing, reliable, and scalable wireless systems while documenting theoretical progress and unresolved challenges.

  • Takeaways & Limitations

    Decentralized MARL is presented as a promising framework for future wireless networks, with communication-efficient cooperation and practical handling of heterogeneity and mobility remaining important directions.

  • Takeaways & Limitations

    Practical algorithm design must address heterogeneous and mobile wireless systems, while convergence, robustness to approximation errors, and limited-data behavior remain unresolved under current theoretical guarantees.

Abstract

from arXiv · show

Deep Reinforcement Learning (DRL) has recently witnessed significant advances that have led to multiple successes in solving sequential decision-making problems in various domains, particularly in wireless communications. The future sixth-generation (6G) networks are expected to provide scalable, low-latency, ultra-reliable services empowered by the application of data-driven Artificial Intelligence (AI). The key enabling technologies of future 6G networks, such as intelligent meta-surfaces, aerial networks, and AI at the edge, involve more than one agent which motivates the importance of multi-agent learning techniques. Furthermore, cooperation is central to establishing self-organizing, self-sustaining, and decentralized networks. In this context, this tutorial focuses on the role of DRL with an emphasis on deep Multi-Agent Reinforcement Learning (MARL) for AI-enabled 6G networks. The first part of this paper will present a clear overview of the mathematical frameworks for single-agent RL and MARL. The main idea of this work is to motivate the application of RL beyond the model-free perspective which was extensively adopted in recent years. Thus, we provide a selective description of RL algorithms such as Model-Based RL (MBRL) and cooperative MARL and we highlight their potential applications in 6G wireless networks. Finally, we overview the state-of-the-art of MARL in fields such as Mobile Edge Computing (MEC), Unmanned Aerial Vehicles (UAV) networks, and cell-free massive MIMO, and identify promising future research directions. We expect this tutorial to stimulate more research endeavors to build scalable and decentralized systems based on MARL.

I. INTRODUCTION

Future 6G networks require AI-driven, scalable, reliable, and decentralized control, motivating DRL and especially MARL for systems involving many interacting agents. This tutorial covers single-agent RL, MBRL, cooperative MARL, and decentralized wireless applications.

  • Motivation: 6G networks target scalable, reliable, secure services for massive connectivity, high data rates, and converged communications and computing.AI-driven radio access and network control are identified as enabling technologies for scalability, context-awareness, energy efficiency, capacity, and connectivity.
  • Deep Reinforcement Learning: DRL uses DNNs to approximate policies or utility functions, selecting actions from predicted probabilities or expected rewards.The agent updates DNN parameters from environmental rewards, while DNN generalization supports high-dimensional and continuous or combinatorial state spaces.
  • Model-Based Reinforcement Learning: MBRL learns environment dynamics and plans with the learned model, offering greater sample efficiency, robustness to changing dynamics or rewards, and better exploration.The tutorial also notes that accurate world-model learning is challenging, which helps explain the continued preference for model-free methods in many applications.
  • Multi-Agent Reinforcement Learning: MARL extends single-agent RL to agents learning policies through interactions with the environment and one another, creating non-stationarity, scalability, and coordination challenges.In multi-agent environments, other agents’ concurrent policy updates can violate the Markov assumption and invalidate single-agent convergence guarantees.
  • Cooperative MARL: Without coordination, agents can independently select actions that form a non-optimal joint action even when each action belongs to an optimal joint strategy.This motivates decentralized approaches and learned cooperative policies for multi-agent problems.
  • Tutorial Scope: The tutorial emphasizes decentralized MARL for self-organizing wireless systems and distinguishes it from federated learning and distributed RL.Its reviewed applications include MEC, UAV networks, cell-free MIMO, spectrum sharing, THz communications, and IRS deployment.

B. Existing Surveys and Tutorials

This section positions the tutorial against prior DRL and 6G surveys, emphasizing its combined treatment of MBRL, cooperative MARL, applications, and open challenges.

  • Scope relative to existing surveys: The tutorial differs from prior surveys by covering both MBRL and cooperative MARL for future 6G networks.It presents an applied perspective focused on wireless communication problems.
  • Single-agent and model-based learning: It presents single-agent DRL frameworks and introduces MBRL fundamentals and potential applications in 6G networks.MBRL is characterized by learning an environment model and using it for policy computation; its appeal includes sample efficiency and adaptation capabilities.
  • MARL frameworks: The paper summarizes MARL frameworks involving emergent communication, decentralized learning cooperation, and networked heterogeneous agents with limited shared information.These frameworks address communication protocols, collaborative behavior, and cooperation among agents with restricted information exchange.
  • 6G wireless applications: It reviews MARL applications in MEC, aerial networks, cell-free massive MIMO, spectrum management, THz communications, and IRS-aided systems.The reviewed areas include distributed beamforming, spectrum sharing, and distributed IRS deployment and control.
  • Open challenges and organization: The tutorial identifies open research directions and challenges for efficient, scalable, and decentralized RL algorithms.The paper organizes these topics across mathematical preliminaries, single-agent RL, MBRL, MARL, wireless applications, and future directions.

A. Single-Agent Reinforcement Learning

Single-agent RL models sequential decision-making through environment interaction, while MBRL learns environment dynamics; MDPs handle full observability and POMDPs handle partial observations.

  • Markov Decision Process: An MDP models a fully observable sequential decision problem with state and action spaces, transition probabilities, rewards, and discounting.The agent interacts with the environment to optimize expected return.
  • Return formulations: Finite-horizon returns describe episodic tasks with an ending horizon H, and can be represented as infinite-horizon problems using an absorbing zero-reward state.The absorbing state transitions continuously to itself.
  • Policies and value functions: A policy maps states to actions, while deterministic and stochastic policies respectively choose an action or output an action distribution.Value and Q-functions measure expected accumulated rewards under a policy.
  • Model-free and model-based RL: Model-free RL avoids requiring known environment dynamics, whereas MBRL estimates transition and reward functions to improve a policy using an approximate model.Dynamic Programming methods require known dynamics and rewards.
  • Wireless MDP example: Downlink power allocation can be formulated as an MDP where base stations select power levels to maximize total network throughput.The state includes users’ channel quality and localization relative to base stations.
  • Partially Observable Markov Decision Process: POMDPs extend MDPs for noisy or limited observations by adding observation spaces and observation probabilities to the formulation.History-based methods use observation or action-observation histories to learn policies or Q-functions.

B. Multi-Agent Reinforcement Learning

MARL extends reinforcement learning to multiple interacting agents whose joint actions influence system dynamics and rewards. The tutorial presents Markov Games, Dec-POMDPs, and Networked Markov Games as frameworks for modeling increasingly cooperative, heterogeneous, and decentralized settings.

  • Multi-agent formulation: MARL models sequential decision-making by multiple agents whose joint actions influence system dynamics and individual rewards.Agents must account for the policies of other agents when maximizing long-term rewards.
  • Markov Games: Markov Games extend MDPs by representing joint action spaces, transition probabilities, agent-specific rewards, discounting, and joint policies.Each agent’s optimal policy depends on the policies of its opponents, creating non-stationarity during learning.
  • Solution concepts: Fully cooperative Markov Games use a shared reward and are also called Multi-agent MDPs, whereas mixed settings address agents with differing objectives.The tutorial focuses subsequent discussion on cooperative extensions of Markov Games.
  • Dec-POMDPs: Dec-POMDPs model cooperative multi-agent decision-making under uncertainty when agents lack full information about states and other agents’ actions.They extend POMDPs to cooperative multi-agent settings and are used for problems such as task offloading and edge caching.
  • Networked Markov Games: Networked Markov Games support heterogeneous agents with distinct rewards while reducing coordination costs through neighbor-to-neighbor communication.They also preserve privacy because agents need not share their reward functions.

III. SINGLE AGENT MODEL-FREE RL ALGORITHMS

Single-agent model-free RL methods can be organized by how they learn values or policies, how they generate data, and whether they bootstrap from estimated future values. The tutorial focuses on online policy-based and value-based methods using deep neural networks as function approximators.

  • Method classes: Model-free RL methods divide into value-based methods that learn value functions and policy-based methods that directly learn parameterized policies.Value-based policies act greedily with respect to approximate values.
  • Learning principles: Monte Carlo methods use complete episodic experience, whereas Dynamic Programming solves Bellman optimality equations; Temporal Difference combines both principles.These frameworks differ in how they use experience and bootstrapping.
  • Data-generation policies: On-policy methods use the same policy for data collection and learning, while off-policy methods separate behavior and target policies.Off-policy learning permits a more exploratory behavior policy.
  • Bootstrapping: Bootstrapping updates estimated state values using estimates of subsequent states, unlike Monte Carlo methods that rely on complete episodic returns.Dynamic Programming and Temporal Difference methods bootstrap.
  • Function representation: Tabular methods store values for visited states, whereas neural function approximators generalize across high-dimensional state data.Deep neural networks can represent value functions or policies.
  • Training regime: Online RL collects interactions from a real environment or simulator, while batch or offline RL learns from a fixed interaction dataset without further interaction.The distinction concerns the source of training data.

B. Policy-Based Algorithms

Policy-based algorithms directly optimize a parameterized policy, making them applicable to discrete and continuous action spaces. The section covers policy gradients, actor-critic methods, and trust-region updates, while emphasizing data inefficiency and update-stability challenges.

  • Policy gradients: Policy-based methods maximize expected long-term reward by directly optimizing the parameters of a policy approximator πθ(a|s).Policy-gradient methods use gradient ascent based on sampled trajectories and estimated action values.
  • Policy gradients: Policy-gradient methods are on-policy, so samples collected for one update cannot be reused for the next policy update.This sample inefficiency is a major drawback of policy-gradient methods.
  • REINFORCE: REINFORCE is defined for episodic problems and can have high variance because action likelihoods are weighted by estimated returns.Small return variations can therefore produce substantially different policy updates.
  • Actor-critic methods: A3C parallelizes actor-critic learning with workers that collect local experiences and update a shared global network.The original architecture shares layers between the actor and critic and aggregates worker gradients globally.
  • Trust-region methods: TRPO constrains policy updates within a trust region using a KL-divergence bound, improving control over changes to the policy distribution.The trust-region radius is δ, and conjugate gradients avoid explicitly inverting the Fisher information matrix.
  • Applications: Policy-based algorithms apply to both discrete and continuous action spaces, including TRPO-based routing optimization.This flexibility is a stated advantage over methods tied to discrete action representations.

C. Value-Based Algorithms

Value-based algorithms estimate value functions and derive policies greedily, spanning classical dynamic programming, Monte Carlo, and temporal-difference methods. Deep Q-learning improves practical learning with neural networks and replay, but nonlinear approximation introduces instability and bias concerns.

  • Value-based methods: Value-based methods estimate state or state-action values and derive the policy implicitly or greedily from those estimates.The value function is central to action selection.
  • Dynamic Programming: Dynamic Programming computes optimal values through Bellman optimality equations, while value iteration repeatedly applies the Bellman operator.The Bellman operator is a contraction, giving a unique fixed point for the optimal value function.
  • Temporal Difference: Dynamic Programming requires known transition dynamics, whereas Temporal Difference methods combine experience-based learning with bootstrapping.This avoids the explicit known-dynamics requirement of Dynamic Programming.
  • TD algorithms: SARSA is an on-policy Temporal Difference method that removes the next-action maximization operator because the next action is observed.Q-learning instead uses an off-policy behavior policy and minimizes Bellman error with function-approximated values.
  • Deep Q-learning: DQN parameterizes the Q-function with a neural network and uses experience replay to reduce sample correlation and improve gradient estimation.A periodically updated target network makes learning targets more stationary.
  • Deep Q-learning: Q-learning with nonlinear function approximators lacks general convergence guarantees and can suffer maximization bias from the max operator.Double learning, prioritized replay, and dueling networks address distinct stability or representation issues.
  • Wireless applications: DQN is widely used for wireless problems with discrete action spaces, while continuous actions require careful discretization.An example applies DQN to dynamic multichannel access, and cited work includes multi-agent scenarios.

D. Deterministic Policy Gradient (DPG) Algorithms

DPG extends Q-learning to continuous action spaces by replacing maximization over actions with a differentiable policy. DDPG applies this approach with neural-network actor and critic models to wireless communication problems.

  • D. Deterministic Policy Gradient (DPG) Algorithms: DPG extends Q-learning to continuous action spaces by evaluating the critic at the policy-selected action instead of maximizing over all actions.The replacement is Qφ(s′, μθ(s′)) for maxa′ Qφ(s′, a′).
  • D. Deterministic Policy Gradient (DPG) Algorithms: DPG requires continuous actions because its policy-gradient term differentiates the critic with respect to the action.Consequently, DPG is unsuitable for discrete action spaces and is off-policy when states come from a replay buffer.
  • D. Deterministic Policy Gradient (DPG) Algorithms: DDPG uses neural networks for both the deterministic policy μθ and critic Qφ.DDPG has been widely applied to wireless communication problems.
  • D. Deterministic Policy Gradient (DPG) Algorithms: TD3 addresses overestimation in DDPG, while SAC adds policy entropy to the expected return to improve robustness and stability.These are presented as variants of DDPG-related methods.
  • D. Deterministic Policy Gradient (DPG) Algorithms: DDPG has been applied to energy-harvesting communications and multi-agent tasks including computation offloading, edge caching, and UAV management.The cited multi-agent applications are summarized in Table II.

E. Theoretical Analysis and Challenges of RL

The tutorial reviews theoretical difficulties in deep RL, including instability, biased or impractical estimation, convergence analysis, and interpretability. It also summarizes convergence and sample-complexity results for policy-gradient methods.

  • Theoretical analysis: Bellman-error minimization lacks the contraction property that guarantees stability for tabular Q-learning with function approximators.The tutorial frames this as a central source of instability in value-based methods.
  • Instability of off-policy TD learning with function approximators: Function approximation, bootstrapping, and off-policy training form the deadly triad that can cause instability and divergence in value-based RL.Removing one element sacrifices generalization power, data efficiency, or exploration.
  • Convergence of PG with neural function approximators: Unbiased policy-gradient estimation is difficult because sampling and approximate Q-functions affect the gradients, while discounted infinite-horizon Monte Carlo estimates can be biased.Compatible function approximation and random-horizon policy-gradient methods are discussed as responses.
  • Convergence of PG with neural function approximators: O(1/T) convergence rates are reported for neural-network actor-critic methods, with compatibility emphasized for convergence and global optimality.The cited results concern natural actor-critic methods and related policy-gradient algorithms.
  • Convergence of PG with neural function approximators: O(ϵ^-2 log(1/ϵ)) sample complexity is reported for minibatch and natural actor-critic methods to reach specified stationary or globally optimal points.The guarantees are stated for an ϵ-accurate stationary point or globally optimal point, respectively.
  • On eXplainable RL (XRL): Explainable RL remains challenging because decisions depend on complex deep networks and RL is unsupervised, while real-world deployment requires understandable explanations.The tutorial distinguishes the model or decision being explained from the generated explanation.

IV. SINGLE AGENT MODEL-BASED RL ALGORITHMS

Model-based RL learns an approximate environment model and plans within it, either by generating simulated experience for policy learning or by selecting actions at decision time. The tutorial describes model learning, planning methods, advantages, and practical limitations.

  • IV. SINGLE AGENT MODEL-BASED RL ALGORITHMS: MBRL learns an approximate model of environment dynamics and performs planning in that model, combining a learned world model with a planning algorithm.The model may approximate transition and reward functions.
  • IV. SINGLE AGENT MODEL-BASED RL ALGORITHMS: State-based dynamics models can parameterize the transition function with a neural network and train it by maximum likelihood on interaction data.Learning accurate models becomes difficult in high-dimensional state spaces, making state representation important.
  • IV. SINGLE AGENT MODEL-BASED RL ALGORITHMS: Background planning augments real experience with simulated experience from the learned model, after which standard RL methods estimate value functions and optimize the policy.This approach supports discrete and continuous action spaces.
  • IV. SINGLE AGENT MODEL-BASED RL ALGORITHMS: Decision-time planning selects an action sequence that maximizes expected trajectory reward without requiring a policy or value function for action selection.Discrete actions use heuristic search, Monte Carlo rollouts, or Monte Carlo Tree Search.
  • IV. SINGLE AGENT MODEL-BASED RL ALGORITHMS: Continuous-action planning can optimize a sampled action sequence using trajectory returns and gradients when the learned model is differentiable.Random shooting is sensitive to initial actions and has poor convergence guarantees.
  • IV. SINGLE AGENT MODEL-BASED RL ALGORITHMS: Compared with model-free RL, MBRL is more data efficient, robust to changes in dynamics and rewards, and supportive of richer exploration, although model-free methods may deploy more efficiently.The comparison depends on the specific MBRL method and setting.
  • IV. SINGLE AGENT MODEL-BASED RL ALGORITHMS: MBRL models face insufficient experience, approximation error, error propagation, planner exploitation, and reduced reliability over longer rollouts.The tutorial recommends continuous replanning and uncertainty estimation to mitigate these issues.

B. Applications of MBRL

The tutorial presents MBRL as relevant to wireless networks because simulation-trained policies face a reality gap and learned models can support transfer across tasks and environments. It also connects MBRL to multi-agent modeling and planning choices.

  • B. Applications of MBRL: MBRL can help address the simulation-to-real-world gap in 6G wireless systems by using learned models as a bridge between simulation and reality.The reality gap arises when a policy learned in simulation performs worse in the real world.
  • B. Applications of MBRL: A learned model from a source task can be reused to learn new tasks faster and, with meta-learning, generalize to new environments and world changes.The tutorial gives aerial and drone networks as potential application domains.
  • B. Applications of MBRL: MBRL planning can use simulated experience for policy optimization or select current-state action sequences through rollouts, tree search, or trajectory optimization.The appropriate decision-time method depends on whether the action space is discrete or continuous.
  • B. Applications of MBRL: Opponent modeling learns representations of other agents’ behavior to support cooperation, coordination, and compensation for partial observability.Modeling other agents can also help decentralize the problem by enabling agents to infer hidden behavior.
  • B. Applications of MBRL: Cooperative Prioritized Sweeping models cooperative multi-agent environments as factored MMDPs represented by Dynamic Bayesian Networks and coordination graphs.The cited approach extends prioritized sweeping to the multi-agent setting.
  • B. Applications of MBRL: Multi-step generative models learn future trajectory segments from past segments while capturing interactions between agents in continuous cooperative and competitive control.This differs from learning only from one-step samples.

V. COOPERATIVE MULTI-AGENT REINFORCEMENT LEARNING

Cooperative MARL addresses coordination among agents while confronting non-stationarity, scalability, partial observability, privacy, and communication constraints. The tutorial organizes solutions around centralized, decentralized, communication-based, and learning-based training schemes.

  • MARL challenges: Non-stationarity invalidates single-agent RL convergence guarantees because rewards and transitions depend on concurrently changing policies.
  • MARL challenges: The joint action space grows exponentially with the number of agents, creating scalability and theoretical-analysis challenges.
  • MARL challenges: Partial observations make it difficult for individual learners to account for other agents’ policies, especially alongside non-stationarity.
  • MARL challenges: Information sharing introduces privacy and security vulnerabilities, motivating fully decentralized algorithms that keep agent information private.
  • Training schemes: CTDE uses centralized information during training but executes decentralized policies from local observations, addressing partial observability and non-stationarity while preserving decentralization.
  • Coordination methods: Coordination methods either learn communication protocols and neighbor exchange or learn cooperative behavior without information sharing.
  • Coordination methods: Communication-efficient MARL uses pruning, attention, and gating to reduce messages under limited-bandwidth constraints.
  • Decentralized MARL: Networked-agent methods enable cooperation through communication structures without central intervention and can accommodate heterogeneous reward functions.

VI. APPLICATIONS

MARL applications in MEC target decentralized task offloading, coordinated channel access, and cooperative edge caching. These studies use independent learning or CTDE-based cooperation to address low-latency and reliability requirements, while scalability remains an open concern.

  • MEC motivation: MEC supports computationally intensive applications, making efficient algorithms important for low-latency and high-reliability services.
  • Task offloading: Independent DDPG/DQN agents learn local task-offloading policies using buffer length, SINR, CSI, power allocation, and offloading decisions.
  • Task offloading: MADDPG jointly optimizes multi-channel access and task offloading in MEC through CTDE under partial channel observability.
  • Edge caching: MADDPG-like methods formulate cooperative edge caching as Dec-POMDPs with rewards based on average delay reduction or weighted local and neighbor hit rates.
  • Future directions: Future MEC research should examine the reliability-latency-scalability trade-off and networked-agent schemes for massive URLLC services.

B. MARL for UAV-Assisted Wireless Communications

UAV-network MARL applications emphasize decentralized cooperation for communication, spectrum sharing, and mission planning. The reviewed problems include heterogeneous local observations, distributed learning, and mixed cooperation–competition objectives.

  • UAV coordination: Decentralized MARL is preferred for cooperative UAV management because it can reduce communication cost and improve energy efficiency.
  • Link discovery and selection: UAV link discovery and selection uses local perceived-channel sets, establishing links when two UAVs transmit messages on the same channel.
  • Spectrum sharing: Spectrum sharing divides UAVs into relaying and sensing clusters and applies distributed tabular Q-learning to local cluster decisions.
  • Target assignment and path planning: Multi-UAV target assignment and path planning combines cooperation and competition by adding collision penalties while minimizing flight distance.
  • Cell-free beamforming: Cell-free networks use coordinated access points and have applied DRL to beamforming, while computational complexity and scalability remain concerns.
  • THz communications: Directional beamforming and beam selection are promising MARL applications for mitigating THz attenuation, path loss, and interference.

E. Intelligent Reflecting Surfaces (IRS)-Aided Wireless Com-

The tutorial reviews MARL and related DRL frameworks for wireless systems and identifies future directions spanning IRS deployment, heterogeneous network topologies, constrained learning, theory, privacy, and robustness. For distributed multiple-IRS systems, a decentralized algorithm had not been identified at the time of writing.

  • IRS applications: Multiple IRS research has addressed joint beamforming and phase-shift optimization, including coordinated systems using alternating fractional programming.
  • IRS applications: For distributed multiple IRS, secure beamforming has used alternating optimization, successive convex approximation, and manifold optimization, but no decentralized algorithm was identified.
  • Conclusions: The tutorial reviews model-free and model-based single-agent RL alongside cooperative MARL and surveys their wireless-communication applications.
  • Future research directions: Networked MARL is motivated by heterogeneous reward functions and mobility in real-world MEC-IoT and sensing systems.
  • Future research directions: Constrained RL is motivated because reward shaping may not ensure constraint-satisfying exploration or policies that respect system constraints.
  • Future research directions: Convergence speed, robustness to approximation errors, and other theoretical properties remain active research questions despite extensive experimental work.
  • Future research directions: Privacy remains insufficiently explored in wireless communication, while learned environment dynamics and reward functions may be inferred by malicious agents.
  • Future research directions: Adversarial vulnerability and robustness to simulation differences are central concerns for deploying DRL in critical real-world applications.
Loading 2011.03615v1…