PaperlayerView exact revision on arXiv

Source-linked AI summary

Applications of Multi-Agent Reinforcement Learning in Future Internet: A Comprehensive Survey

Tianxu Li, Kun Zhu, Nguyen Cong Luong, Dusit Niyato, Qihui Wu, Yang Zhang, Bing Chen

arXiv:2110.13484v3cs.AIcs.LGcs.MA

TL;DR

Future Internet’s heterogeneous and decentralized networks require local decisions under dynamic conditions, while single-agent learning does not model interactions among entities and can face non-stationarity. This paper tutorials MARL and surveys its applications across key networking problems, reporting that surveyed works commonly use CTDE and cooperation, with examples including a 20% throughput improvement over BGP routing.

  • Problem

    Future-Internet entities must make local decisions with partial observations, while single-agent learning does not model cooperation or competition among entities and may encounter non-stationarity.

  • Method

    The paper introduces single-agent RL and MARL, explains multi-agent challenges and learning schemes, and reviews MARL applications across future-Internet networking issues.

  • Results

    Surveyed works commonly adopt CTDE and cooperation; one reviewed routing algorithm improves overall throughput by 20% compared with BGP routing.

  • Takeaways & Limitations

    MARL applications span network access, power control, offloading, caching, routing, UAV trajectory design, and network security, with challenges and future directions identified.

Abstract

from arXiv · show

Future Internet involves several emerging technologies such as 5G and beyond 5G networks, vehicular networks, unmanned aerial vehicle (UAV) networks, and Internet of Things (IoTs). Moreover, future Internet becomes heterogeneous and decentralized with a large number of involved network entities. Each entity may need to make its local decision to improve the network performance under dynamic and uncertain network environments. Standard learning algorithms such as single-agent Reinforcement Learning (RL) or Deep Reinforcement Learning (DRL) have been recently used to enable each network entity as an agent to learn an optimal decision-making policy adaptively through interacting with the unknown environments. However, such an algorithm fails to model the cooperations or competitions among network entities, and simply treats other entities as a part of the environment that may result in the non-stationarity issue. Multi-agent Reinforcement Learning (MARL) allows each network entity to learn its optimal policy by observing not only the environments, but also other entities' policies. As a result, MARL can significantly improve the learning efficiency of the network entities, and it has been recently used to solve various issues in the emerging networks. In this paper, we thus review the applications of MARL in the emerging networks. In particular, we provide a tutorial of MARL and a comprehensive survey of applications of MARL in next generation Internet. In particular, we first introduce single-agent RL and MARL. Then, we review a number of applications of MARL to solve emerging issues in future Internet. The issues consist of network access, transmit power control, computation offloading, content caching, packet routing, trajectory design for UAV-aided networks, and network security issues.

I. INTRODUCTION

Future Internet networks are heterogeneous, decentralized, and dynamic, creating local decision-making challenges that single-agent learning does not adequately model. This survey introduces MARL and reviews its applications, learning schemes, challenges, and research directions across major future-Internet issues.

  • Heterogeneous devices, dense users, frequent communication requests, and ubiquitous interference make future-Internet network management challenging.
  • Local observations and partial network-state information make independent learning struggle with other entities’ policies and non-stationarity.Network entities may need to select channels using local CSI without observing the instantaneous global state.
  • MARL models cooperation and competition among network entities, supporting stable, collaborative, decentralized, or non-cooperative decision-making.Its stated applications include channel selection, transmit-power selection, offloading, caching, and Nash-equilibrium learning.
  • The survey reviews MARL applications in network access, transmit power control, computation offloading, content caching, packet routing, UAV trajectory design, and network security.
  • Network access and transmit power control receive more attention than other issues, while cellular systems, CTDE, and cooperation are common in surveyed works.
  • The paper provides tutorials on single-agent RL and MARL, analyzes algorithms and challenges, reviews applications, and outlines open problems and future directions.

II. SINGLE-AGENT REINFORCEMENT LEARNING

Single-agent reinforcement learning models sequential decisions with MDPs or partially observed environments and learns policies from interaction without requiring known dynamics. Classical value-based methods estimate action values, while deep variants address large state and action spaces but introduce stability and convergence challenges.

  • Foundations: MDPs represent sequential decisions using states, actions, rewards, transitions, and a discount factor, while POMDPs model settings with only partial environmental observation.RL learns policies through interaction without requiring prior knowledge of transition and reward dynamics.
  • Value-based Algorithms: Value-based algorithms learn an optimal state-action value function and derive the policy by selecting the action with the highest estimated value.Q-learning updates Q-values using temporal-difference learning and can reuse experiences from different policies through off-policy learning.
  • Value-based Algorithms: Tabular Q-learning becomes inefficient for complicated problems with large state and action spaces because maintaining the Q-table is difficult and convergence can be slow.DQN replaces the table with a deep neural network and uses experience replay, while a target network helps stabilize learning.
  • Value-based Algorithms: DQN variants address instability and estimation issues through mechanisms such as target networks, double Q-networks, prioritized replay, and dueling architectures.DDQN decouples action selection from action evaluation to reduce overestimation bias caused by the max operator.

D. Policy-based Algorithms

Policy-based reinforcement learning directly optimizes parameterized policies, commonly through policy gradients and actor-critic estimates. The section also describes parallel actor training, synchronous gradient aggregation, learning-rate constraints, and deterministic or stochastic policies.

  • Policy-based Algorithms: Policy-based algorithms directly optimize a parameterized policy over the policy space rather than deriving it indirectly from an estimated action-value function.The policy parameters are learned by maximizing expected long-term reward, commonly using gradient ascent.
  • Policy-based Algorithms: REINFORCE estimates the Q-function from episode returns but is on-policy, has poor sample efficiency, and can exhibit high variance from random trajectories.Actor-critic methods reduce variance by using a critic to estimate expected return, although estimation can introduce bias.
  • Policy-based Algorithms: A3C trains multiple actors in parallel with a shared global actor-critic network, whereas A2C synchronously aggregates and averages actors’ gradients to update that network.Asynchronous updates can use outdated policies, motivating the synchronous mechanism.
  • Policy-based Algorithms: TRPO constrains policy differences to address the difficulty of selecting a learning rate that avoids poor policies.Both excessively low and high learning rates can degrade performance.
  • Policy-based Algorithms: Stochastic policies sample actions from probability distributions to encourage exploration, whereas deterministic policies output executable actions directly.The section notes that actor-critic frameworks and variants have been used in decentralized future-Internet systems.

III. MULTI-AGENT REINFORCEMENT LEARNING

This section introduces MARL as a framework for sequential decision-making among interacting agents, extending single-agent models to cooperative, competitive, and mixed settings. It formalizes agent policies, rewards, value functions, and Nash equilibrium in Markov Games.

  • Markov Games: MARL models communication networks as multi-agent environments in which entities interact with both the environment and one another.Markov Games or Stochastic Games extend MDPs to capture these interactions.
  • Markov Games: Each agent selects an action from its action space based on the system state, receives a reward, and seeks a policy maximizing long-term reward.The next state is determined by the joint actions of all agents.
  • Markov Games: An agent’s value function depends on the joint policy, so its performance is determined by both its own policy and the policies of other agents.This dependence motivates equilibrium-based solutions in multi-agent settings.
  • Markov Games: Nash equilibrium requires each agent’s policy to be the best response to the policies of all other agents.A Nash equilibrium exists in finite-space infinite-horizon discounted Markov Games but may not be unique.
  • MARL Settings: Future Internet applications include cooperative users maximizing cellular sum rate and competing users seeking limited edge-computing resources for lower task delays.Markov Games represent fully cooperative, fully competitive, and mixed relationships among agents.

B. Dec-POMDP

Dec-POMDPs model cooperative multi-agent decision-making when agents cannot observe the global state, while MARL learning schemes address partial observability, non-stationarity, scalability, and communication constraints.

  • Dec-POMDP: Distributed networking agents often lack timely global state information, motivating partially observed Markov Games modeled as Dec-POMDPs.Each agent observes only part of the system state before making local decisions.
  • Dec-POMDP: In a Dec-POMDP, each agent learns a local policy from its partial observation to maximize cooperative long-term reward.The model retains Markov Game reward and transition functions while restricting each agent’s observation.
  • Challenges: Concurrent policy learning makes rewards and state transitions depend on other agents’ actions, violating the single-agent stationarity assumption.This creates convergence challenges for single-agent RL in multi-agent environments.
  • Challenges: Accounting for other agents’ joint actions causes the joint action space to grow exponentially with the number of agents, creating scalability and convergence-analysis challenges.Deep neural network approximators are used to process the resulting combinatorial information.
  • Learning Schemes: CTDE trains agent-specific policies from collective experiences and executes them using local observations, reducing reliance on instantaneous global state.Other schemes exchange neighboring information, use centralized global information, or prohibit communication under costly or unreliable links.

D. CTDE-based MARL Algorithms

This section surveys CTDE-based MARL algorithms for future Internet problems, emphasizing centralized critics, counterfactual credit assignment, and mean-field approximations for scalability.

  • MADDPG: MADDPG uses a decentralized actor and centralized critic for each agent and applies to cooperative, competitive, and mixed settings.Each critic accesses other agents’ actions and policies to evaluate the joint policy’s effect on the agent’s expected reward.
  • COMA: COMA uses local actors and a single centralized critic to address credit assignment through a counterfactual baseline.The baseline helps estimate each agent’s individual contribution to the total reward, and a single forward pass computes counterfactual advantages.
  • COMA: COMA is limited to cooperative scenarios because it uses a single centralized critic, whose global-state assumption is difficult to satisfy in large-scale systems.The method’s critic calculates advantages under knowledge of the global state.
  • Mean-Field Q-Learning: MADDPG and COMA can suffer from poor scalability because the joint action space grows exponentially with the number of agents.MF-Q addresses this issue by approximating neighboring agents’ actions with a mean-field value.
  • Mean-Field Q-Learning: MF-Q reduces the action-space growth by replacing interactions with neighboring agents by interaction with a mean-field agent.This approximation supports large-scale multi-agent problems.

4) Multi-Actor Attention-Critic (MAAC):

MAAC uses attention to represent how other agents contribute to an agent’s value function, while value-decomposition methods address cooperative credit assignment and decentralized execution.

  • Multi-Actor Attention-Critic (MAAC): MAAC extends SAC to multi-agent environments by giving each agent a critic that shares a central attention mechanism with other critics.The attention mechanism centralizes the calculation of other agents’ contributions to address non-stationarity.
  • Multi-Actor Attention-Critic (MAAC): MAAC represents an agent’s Q-function using local encodings of its observation and action together with the contribution of other agents.The local networks encode the agent’s own information, while the centralized attention mechanism computes the other-agent contribution.
  • Multi-Actor Attention-Critic (MAAC): Attention weights selectively scale other agents’ embeddings before converting them into the current agent’s contribution.This allows different agents to receive different weights according to their represented contribution.
  • Multi-Actor Attention-Critic (MAAC): MAAC uses each agent’s own embedding and the aggregated contribution to make action decisions, addressing scalability and improving learning efficiency.The method converts other-agent embeddings into a contribution through attention rather than directly using the full joint representation.
  • Value Decomposition: Value-decomposition methods target cooperative credit assignment by decomposing global value functions into agent-level values using local observations.VDN back-propagates the global TD-error to sub-value functions, while QMIX uses a monotonic mixing network with the global state during training.

3) Weighted QMIX (WQMIX):

WQMIX extends QMIX with weighted joint-action representation to address interactions that violate QMIX’s monotonicity constraint. The survey also compares MARL algorithms by settings, information sharing, and reward structures.

  • 3) Weighted QMIX (WQMIX):: WQMIX assigns different weights to joint actions to overcome QMIX’s limitations in interacting-agent tasks.The weight function maps global states and joint actions to values in (0,1].
  • 3) Weighted QMIX (WQMIX):: Evaluating joint actions through an approximated Q-function improves learning efficiency compared with directly searching the joint-action Q-function.The approximator evaluates joint actions to maximize the monotonic total Q-function.
  • MARL algorithm comparison: MADDPG is the only surveyed algorithm applicable to environments containing both cooperative and competitive agents.Its local critic and local actor accommodate varied reward structures.
  • MARL algorithm comparison: COMA and VDN are restricted to cooperative settings because they use parameter sharing.Most surveyed algorithms also address the credit-assignment issue.
  • Section summary: MARL design is shaped by non-stationarity, scalability, and partial observability, motivating different learning schemes and algorithms.The survey next examines applications to challenging future-Internet problems.

A. Dynamic Spectrum Access

The survey reviews MARL for dynamic spectrum access across D2D, UAV, cognitive-radio, vehicular, and joint-access settings. Approaches range from independent and fully decentralized learning to CTDE and neighbor-based critics.

  • A. Dynamic Spectrum Access: MARL is applied to dynamic spectrum access in D2D, UAV, cognitive-radio, and vehicular networks, including joint spectrum access and user association.The reviewed problems span multiple emerging-network scenarios.
  • Spectrum access for D2D communications: D2D users independently learn spectrum-selection policies with tabular Q-learning under non-cooperative operation while respecting cellular-user SINR constraints.Rewards equal achieved throughput when cellular QoS requirements are satisfied; otherwise, a penalty is imposed.
  • Cooperative spectrum access: CTDE-based MAAC uses global information during training, while NAAC reduces critic complexity by using neighboring agents’ states and actions.NAAC approximates all-agent information from neighboring-agent information at the base station.
  • Spectrum access for UAVs: Fully decentralized MARL suits UAV spectrum allocation when unreliable, dynamic links make global information exchange unsuitable.In the UAV relay model, agents independently choose whether to relay for the primary user or transmit their own packets.
  • Spectrum access for UAVs: The UAV tabular Q-learning approach converges to a Nash equilibrium in sum-rate for both 2-UAV and 6-UAV scenarios.The 6-UAV scenario requires more episodes because its larger state space slows tabular Q-learning.
  • Spectrum access for cognitive radios: Cooperative Q-learning and cooperative SARSA formulate cognitive-radio allocation as a Markov game with common energy-efficiency rewards under spectrum, QoS, interference, and power constraints.The common reward is zero when the joint action violates the stated constraints.
  • Spectrum access for vehicular networks: Dedicated DQNs outperform shared DQNs and random selection in vehicular spectrum allocation on sum capacity and successful transmission probability.Dedicated networks reduce signaling overhead caused by parameter sharing.
  • Spectrum access for vehicular networks: GNN-based decentralized state compression avoids global CSI in high-mobility environments but may scale poorly as neighboring traffic increases.The survey identifies GATs as a possible way to weight neighboring-node embeddings by their features.

2) Joint user association and spectrum access:

The surveyed MARL methods address joint association, spectrum access, and transmit-power control across heterogeneous cellular, UAV, vehicular, and IoT networks. They combine decentralized decisions with shared, local, or processed interference information.

  • 2) Joint user association and spectrum access:: Joint user association and spectrum access in HetNets is non-convex and requires complete network information, motivating MARL solutions.Each user must select both a serving base station and spectrum resource.
  • Joint user association and spectrum access: Decentralized D3QN achieves higher system capacity, faster convergence, and better scalability than DQN, Q-learning, and a genetic algorithm for HetNet association and spectrum allocation.The approach exchanges associated users’ state information among base stations through backhaul links.
  • Joint user association and spectrum access: In multi-UAV downlink networks, independent Q-learning uses each UAV’s partial SINR observation to select a user, sub-channel, and power level.Rewards combine throughput and power-consumption cost when the SINR threshold is exceeded.
  • 1) Transmit power control:: Dense cellular deployments increase interference and can reduce system capacity and QoS, motivating transmit-power control in HetNets.The surveyed HetNet model contains multiple APs sharing a spectrum band to serve users.
  • Power control for cellular systems: TRPO-based partially decentralized power control uses local CSI and previous power allocations under a shared network spectral-efficiency reward.A shared policy reduces training parameters and accelerates convergence.
  • Power control for cellular systems: Interference preprocessing and fixed-size interferer selection reduce state dimensionality and help power-control algorithms scale across user densities.The methods retain dominant interference signals or interferers based on received power.
  • Power control for cellular systems: The HetNet power-control problem maximizes weighted sum rate under power constraints and is non-convex and NP-hard.Transmitters use local channel and interference information together with neighbor feedback.
  • Power control for IoTs: Mean-field MARL provides decentralized power allocation for large energy-harvesting IoT networks where exchanging node state information consumes substantial energy.The approach targets energy-constrained sensor nodes.

2) Joint user association and power control:

Joint user association and power control are addressed with decentralized multi-agent policies that account for neighboring entities and centralized rewards. MAPPO improves throughput while reducing handover frequency in a two-tier heterogeneous network.

  • 2) Joint user association and power control:: APs independently select users and transmit powers using local observations while neighboring APs are fixed through distance-based selection for scalability.The fixed neighbor count keeps each AP’s observation dimension unchanged as AP density varies.
  • 2) Joint user association and power control:: A weighted-sum-rate centralized reward emphasizes each UE’s local transmit rate rather than only the average reward, addressing fairness in resource allocation.
  • 2) Joint user association and power control:: A2C achieves a better sum rate than DDQN, whereas DDQN converges faster because replay-buffer sampling improves sample efficiency.
  • 2) Joint user association and power control:: MAPPO uses CTDE and a counterfactual baseline to train decentralized UE policies and address non-stationarity and credit assignment.The counterfactual baseline marginalizes the current UE’s action while keeping other UEs’ actions fixed.
  • 2) Joint user association and power control:: MAPPO outperforms IL-based PPO, MADDPG, and A3C in individual and total throughput while significantly reducing handover frequency through reward penalization.

V. OFFLOADING AND CACHING

MARL is applied to decentralized computation offloading and content caching in dynamic MEC networks, where limited device resources, latency demands, and multiple servers create joint optimization challenges. Reviewed methods include independent learning, CTDE, MADDPG, SAC combinations, and multi-agent Q-learning.

  • A. Computation Offloading: MEC offloading can reduce execution delay for computation-intensive tasks, but mobile devices must choose whether and where to offload under limited computation, battery, and memory.
  • B. Content Caching: Content caching reduces nearby users’ access latency and core-network congestion, but limited capacity and multiple servers and users require broader coordination.
  • A. Computation Offloading: A2C-based non-cooperative offloading reduces execution delay, task drop rate, and mobile CPU usage compared with local-only, offloading-only, and DQN policies.
  • A. Computation Offloading: MADDPG for partition-enabled offloading outperforms DDPG in average energy consumption, total utility, and average task delay.The setting jointly optimizes task partitioning, transmit power, and communication and computational resource allocation.
  • A. Computation Offloading: Combining SAC with MADDPG improves task latency, task drop rate, and energy consumption relative to DDPG, MADDPG, and SAC.The combination uses SAC’s maximum-entropy term to enlarge exploration and enhance stability for mixed discrete-continuous actions.
  • A. Computation Offloading: COMA learns task-migration policies across multiple MEC servers using a centralized critic and counterfactual baseline to address credit assignment.The global reward is based on the change in average estimated completion time across tasks.

B. Content Caching

Cooperative edge caching lets neighboring edge servers share cached content, addressing limited storage and user mobility in MEC networks. MARL methods adapt caching to time-varying demand and coordinate cache decisions across servers or users.

  • 1) Cooperative edge caching:: Cooperative edge caching allows users to request content from local or neighboring edge servers instead of relying only on the remote CDN.
  • 1) Cooperative edge caching:: MAA2C uses neighboring BS states and policy fingerprints to adapt caching to time-varying, heterogeneous content similarity across areas.A real-world video-watching dataset shows high neighboring-area similarity, while also revealing temporal and cross-edge heterogeneity.
  • 1) Cooperative edge caching:: 21% lower latency and 26% lower cost are achieved by MAA2C versus state-of-the-art caching methods including DRL and joint action learners.
  • 1) Cooperative edge caching:: Collaborative MAMAB coordinates SBS caching through a graph whose vertices are SBSs connected when they cover common users, with rewards assigned across relevant edges.
  • 2) Cooperative coded caching:: Cooperative coded caching divides content into segments so mobile users can retrieve portions across SBSs when mobility prevents complete download from one station.
  • 2) Cooperative coded caching:: A shared global state of prior-slot actions and observations supports decentralized coded-caching decisions while reducing interaction-data overhead.
  • 2) Cooperative coded caching:: MARL-based DDPG learns STTL caching policies for encoded packets to minimize network loads under heterogeneous request processes.

VI. PACKET ROUTING

MARL supports adaptive packet routing across edge networks, inter-domain systems, MANETs, UAANETs, and wireless sensor networks. Approaches use local, neighbor, or global information to address congestion, dynamic topology, and large state spaces.

  • VI. PACKET ROUTING: MARL addresses routing’s need for adaptive performance-based decisions, unlike conventional rule-based protocols that cannot change routing rules according to network conditions.
  • VI. PACKET ROUTING: A2C-based inter-domain routing improves overall throughput by 20% over BGP, with performance increasing as the information-sharing scope expands.
  • VI. PACKET ROUTING: MAPOKTR uses attention to identify relevant information, such as packet destination, within huge global network states for edge-node routing.Its reported throughput is better than state-of-the-art MADRL algorithms, although deploying the global-information collection thread is challenging in realistic networks.
  • VI. PACKET ROUTING: Fully decentralized deep Q-routing avoids the challenging collection of instantaneous global network information by selecting adjacent routers as next hops.
  • VI. PACKET ROUTING: DEEPCQ+ combines MADRL with CQ+ and PPO, using a shared policy and fixed top-neighbor preprocessing to scale across network sizes and dynamic topologies.
  • VI. PACKET ROUTING: Routing methods use local or global rewards and neighbor information for sensor, UAV, and other network entities across WSNs, UAANETs, MANETs, and edge networks.

VII. TRAJECTORY DESIGN FOR UAV-AIDED NETWORKS

MARL is applied to UAV trajectory design alongside access control, power allocation, sensing, transmission, and task selection in dynamic UAV networks. Reviewed methods use centralized or independent learning to coordinate UAV actions and improve throughput, fairness, sensing freshness, or related objectives, while facing computing and energy constraints.

  • UAV-aided cellular networks: UAV trajectory design improves ground-user QoS and is commonly coupled with power allocation or other network-resource decisions in a non-convex optimization problem.The reviewed network objectives include throughput and fair throughput.
  • UAV-aided cellular networks: AG-PAMADDPG jointly handles UAV movement and ground-user frequency selection in a mixed cooperative-competitive game with throughput and fairness objectives.Its continuous UAV actions and discrete ground-user actions are transformed into a compatible hybrid-action policy.
  • UAV-aided cellular networks: 142% and 56% sum-rate gains over circular and 2D trajectories, respectively, are achieved by multi-agent MDQN with 3D trajectory design.MDQN also converges faster than conventional DQN in the multi-agent scenario.
  • Multi-UAV sensing networks: Multi-UAV sensing methods optimize sub-channel allocation, trajectory design, sensing, transmission, and task selection using DQN, DDPG, and CA2C-based MARL.The objectives include successful transmission, successful sensing, AoI minimization, and hybrid discrete-continuous control.
  • Challenges: Centralized training can provide UAV location information for coordinated decisions, but limited UAV computing capacity and energy consumption constrain complex MARL trajectory policies.These constraints are identified as important design considerations for energy-efficient trajectory management.

VIII. NETWORK SECURITY

MARL is reviewed for defending future Internet networks against eavesdropping and jamming attacks. The approaches coordinate UAV jammers, users, UAV clusters, and dense IoT edge servers through multi-agent policies that account for other agents or approximate their joint actions.

  • Eavesdropping attack: MADRL enables friendly UAV jammers to cooperate on jamming policies that prevent ground eavesdroppers from wiretapping UAV-to-ground communications.The model includes one UAV transmitter, multiple UAV jammers, ground users, and ground eavesdroppers.
  • Eavesdropping attack: Attention-based CAA-MADDPG focuses UAV observations on relevant entities, with transmitters emphasizing ground users and jammers emphasizing eavesdroppers and other jammers.The attention mechanism is introduced to improve learning efficiency.
  • Jamming attack: Cooperative multi-agent Q-learning models users’ sequential channel selection as a Markov game to jointly address co-channel interference and anti-jamming.Users learn decentralized policies to maximize long-term reward in the presence of one jammer.
  • Jamming attack: UAV clusters can switch between collaborative and independent anti-jamming policies according to sensed co-channel interference, using joint Q-tables when interference exceeds a threshold.The joint Q-value depends on all users’ channels, the jamming channel, and joint actions.
  • Jamming attack: Mean-field DeepMellow reduces computational complexity and improves scalability in ultra-dense IoT anti-jamming by approximating other edge servers’ joint actions.Its simulations report higher average reward than mean-field Q-learning and individual Q-learning.

C. DDoS Attack

The survey describes MARL for collaborative DDoS defense using hosts, routers, and servers to control traffic in distributed attack environments. Centralized critics support training, while decentralized actors reduce the need for centralized action execution.

  • DDoS attack: DDoS attacks exhaust target resources through large traffic volumes generated by compromised nodes, potentially paralyzing systems and denying legitimate requests.The attack is distributed, dynamic, and large-scale.
  • DDoS attack: Collaborative DDoS-defense MARL assigns hosts, routers, and servers roles in controlling packet forwarding and evaluates router or server capability using aggregate traffic.Hosts may act as attackers or legitimate users, while routers regulate forwarded packets.
  • DDoS attack: Central DDPG information exchange can incur huge communication overhead and increase decision-making latency.This is identified as a limitation of centralized coordination.
  • DDoS attack: ComDDPG uses multiple decentralized actors for execution and one central critic for offline centralized training with global traffic and resource information.The critic uses router and server traffic, residual resources, and server queue lengths.
  • Summary: Most reviewed security works use CTDE to learn decentralized defense policies because centralized policies may have high computational complexity for network-wide action decisions.The survey covers eavesdropping, jamming, and DDoS defense across wireless, IoT, and Internet networks.

4) Privacy and security of the use of MARL:

The survey identifies privacy, adversarial robustness, communication overhead, state-processing cost, and reward-design difficulties as challenges for deploying MARL in future Internet networks. It also points to state-information extraction and additional application domains as continuing research directions.

  • Privacy and security: MARL coordination requires sharing observations that may expose task profiles, battery levels, CPU utilization, and locations to transmission attacks.DNN-based MARL is also vulnerable to adversarial input manipulation that can trigger harmful actions.
  • Multi-agent state information extraction: GNNs and attention mechanisms can extract relevant multi-agent state information, but frequent collection of other agents’ information creates communication overhead.Efficient state-information extraction for network communication remains open.
  • Reward design: Reward design remains open because future networks contain entities with different objectives, including throughput maximization and energy-efficiency maximization.Reviewed works commonly use either global rewards or individual local rewards.
  • Future directions: MARL spectrum management is proposed as a direction for reliable NB-IoT communications facing massive-device and cross-network co-channel interference.The setting includes interference from existing networks such as 5G.
  • Future directions: MARL can test blockchain incentive vulnerabilities by allowing rational users to interact with unknown incentives and other users while learning Nash equilibrium.This can reveal vulnerabilities exploitable by attackers seeking higher profits.
  • Conclusions: The survey concludes by synthesizing MARL applications across seven future-Internet issues and outlining challenges, open issues, and future directions.The covered issues include network access, power control, offloading, caching, routing, UAV trajectory design, and security.
Loading 2110.13484v3…