Source-linked AI summary
Multi-Agent Reinforcement Learning for Resources Allocation Optimization: A Survey
Mohamad A. Hady, Siyi Hu, Mahardhika Pratama, Jimmy Cao, Ryszard Kowalczyk
TL;DR
Resource Allocation Optimization requires effective distribution of finite resources in dynamic, decentralized settings, but existing surveys do not comprehensively cover its intersection with MARL. This survey synthesizes MARL foundations, algorithms, applications, benchmarks, challenges, and future directions, concluding that MARL has substantial potential for complex RAO environments while identifying limitations in classical approaches.
Problem
Existing surveys do not provide a comprehensive review of MARL-driven RAO across different industries, despite growing interest in this intersection.
Method
The survey reviews MARL foundations, algorithms, classifications, applications, benchmarks, and challenges for RAO across diverse domains.
Results
The survey finds that MARL has demonstrated considerable potential for decentralized and dynamic resource allocation in complex, uncertain environments.
Takeaways & Limitations
The survey provides a structured reference for researchers and practitioners studying MARL-based resource allocation across telecommunications, energy, computing, transportation, and manufacturing.
Takeaways & Limitations
Classical RAO approaches are limited by assumptions of static environments, poor scalability, centralized control, and full system visibility.
Abstract
from arXiv · showhide
Multi-Agent Reinforcement Learning (MARL) has become a powerful framework for numerous real-world applications, modeling distributed decision-making and learning from interactions with complex environments. Resource Allocation Optimization (RAO) benefits significantly from MARL's ability to tackle dynamic and decentralized contexts. MARL-based approaches are increasingly applied to RAO challenges across sectors playing pivotal roles to Industry 4.0 developments. This survey provides a comprehensive review of recent MARL algorithms for RAO, encompassing core concepts, classifications, and a structured taxonomy. By outlining the current research landscape and identifying primary challenges and future directions, this survey aims to support researchers and practitioners in leveraging MARL's potential to advance resource allocation solutions.
1 Introduction
MARL supports distributed decision-making in dynamic, decentralized environments, while RAO focuses on distributing resources among tasks or agents to improve efficiency and related objectives. These capabilities make MARL relevant to resource allocation across multiple application domains.
- MARL enables agents to interact and learn in shared settings under non-stationary and evolving conditions.
- RAO distributes resources such as time, energy, network bandwidth, and computational power to improve efficiency and effectiveness.
- MARL is suited to RAO because it enables decentralized, adaptive decision-making in telecommunications, energy management, cloud computing, and transportation.
- In cloud computing, MARL can optimize resource scheduling and load balancing to improve system performance and reduce costs.
PRELIMINARY AND FUNDAMENTAL SOLUTION AND APPLICATION
This survey addresses the lack of a comprehensive review focused specifically on MARL-driven RAO across industries. It organizes foundational concepts, applications, benchmarks, challenges, and research opportunities into a structured overview.
- The survey fills a gap left by related reviews that do not extensively examine MARL-driven RAO across different industries.
- It maps MARL algorithms and frameworks used in RAO into a consolidated research resource.
- The review synthesizes advancements, trends, challenges, and opportunities unique to MARL applications in RAO.
- It catalogs recent MARL training frameworks and application areas.
- The survey lists real-world benchmarks and testbeds for developing and evaluating RL and MARL algorithms in RAO.
- Its coverage spans telecommunications, energy systems, distributed computing, transportation, and manufacturing, alongside dynamic environments, scalability, communication, and coordination challenges.
2 Resource Allocation Optimization
RAO distributes finite resources among tasks or activities under operational constraints, using objectives such as efficiency, cost reduction, throughput, or fairness. Its formulation combines resource limits with application-specific objective functions and allocation preferences.
- RAO distributes finite resources among tasks or activities to improve efficiency, productivity, or fairness.
- Resource allocation must account for available resources, task demands, and constraints such as time, budget, or capacity.
- Distributed resources improve scalability, fault tolerance, and flexibility but require coordination across interconnected systems.
- Centralized manufacturing resources are managed within one facility to minimize downtime and improve throughput, while distributed cloud resources require network load balancing.
- Problem Formulation: The allocation model constrains total resource use by Σ_i x_i ≤ N and bounds each task allocation with l_i ≤ x_i ≤ u_i.
- 2.1.2 Objective of RAO: RAO objectives may maximize benefits or minimize costs, with maximization of f equivalent to minimization of −f.
- 2.1.2 Objective of RAO: Objective functions can be separable, convex, minimax or maximin, or fairness-oriented depending on application requirements.
2.2 Resources Properties
Resource properties determine how allocation strategies should model divisibility, availability over time, and location. The section also situates classical RAO methods and their limitations in large-scale and decentralized settings.
- Resource classification: Resources are classified by divisibility, duration, and location, with each property influencing management approaches and system design.
- Divisibility: Discrete resources are allocated as whole units and modeled with integer variables, whereas continuous resources support variable allocations.
- Availability: Renewable resources require time-dependent constraints, while non-renewable resources are governed by cumulative upper bounds.
- Location: Distributed resources improve scalability, fault tolerance, and flexibility but require coordination across interconnected locations.
- Classical approaches: Classical RAO methods include linear programming, heuristics, and game theory, but large-scale and decentralized settings expose computational and coordination limitations.LP handles linear objectives and constraints; heuristic methods trade optimality guarantees for computational efficiency, while game theory models competing or cooperative agents.
2.4 Limitation and Challenge in Classical RAO Approach
Classical RAO approaches struggle with modern systems that are dynamic, large-scale, decentralized, partially observable, and heterogeneous. The section motivates MARL because it can learn across distributed agents and adapt policies to changing conditions.
- Core limitations: Classical methods cannot fully address the scalability, adaptability, and decentralization demands of increasingly complex, interconnected, and dynamic RAO systems.
- Dynamic environments: Rapidly changing demand, resource availability, uncertainty, and strict time constraints make real-time allocation difficult for methods based on static assumptions or iterative computation.
- Decentralization and observability: Decentralized systems require agents to make decisions from local information, limiting centralized approaches that depend on global knowledge and control.
- Why MARL: MARL distributes learning across agents, supports decentralized decisions under incomplete information, and can develop specialized policies for heterogeneous agent roles.
- Survey scope: The survey addresses these challenges by reviewing MARL solutions for rapidly changing conditions and decentralized resource allocation.
3 MARL Foundations
Reinforcement learning provides the foundation for adaptive sequential decision-making, while MARL extends these principles to multiple interacting agents. This supports decentralized learning for scalable and real-time resource allocation.
- RL foundations: RL provides essential principles for adaptive decision-making and sequential optimization that extend to multiagent environments.
- MARL extension: MARL leverages multiple interacting agents to address scalability and enable real-time adaptability in resource allocation.
3.1 Reinforcement Learning as Optimization
Reinforcement learning frames resource allocation as sequential decision-making under uncertainty, with agents learning policies that maximize cumulative rewards. The section introduces value-based, policy-based, actor-critic, and trust-region methods for improving learning stability and scalability.
- RL foundations: RL agents learn policies through environment interaction, adapting actions to changing resource demands while maximizing long-term cumulative rewards.This supports resource allocation in dynamic and uncertain settings.
- RL foundations: An MDP represents states, actions, transition probabilities, rewards, and discounting, enabling decisions based on observed states and obtained rewards.The discount factor γ controls the relative importance of future rewards.
- Learning objectives: Value functions estimate expected returns for states or state-action pairs, while Bellman recursions relate current values to future rewards under a policy.Qπ(s, a) evaluates taking action a in state s before following policy π.
- Deep RL: DQN uses neural-network Q-value approximation, experience replay, and a target network to improve training stability in complex environments.Double DQN and Dueling DQN further address overestimation bias and separate state values from action advantages.
- Policy optimization: Policy-gradient methods directly optimize expected return, while actor-critic methods combine policy updates with value estimation and advantage-based variance reduction.TRPO and PPO constrain policy updates to prevent large deviations from the current policy.
3.2 Multi-Agent Reinforcement Learning
MARL extends reinforcement learning to multiple interacting agents in shared environments, supporting decentralized, adaptive resource allocation. Its formal models and training paradigms expose trade-offs among coordination, scalability, observability, and communication.
- MARL for RAO: MARL addresses multi-agent resource allocation by enabling decentralized decisions, real-time adaptation, and coordination across large-scale dynamic systems.These properties are relevant to telecommunications, smart grids, cloud computing, transportation, and other RAO settings.
- MARL for RAO: MARL can balance local and system-wide objectives across heterogeneous resources and support decisions under uncertainty through stochastic games and Dec-POMDPs.Examples include jointly managing bandwidth, energy, and memory under fluctuating network loads or energy supplies.
- Formal multi-agent models: A stochastic game models N agents with state transitions driven by joint actions, individual action spaces, and agent-specific rewards in cooperative, competitive, or mixed environments.Fully cooperative settings use a shared reward, whereas competitive settings can involve conflicting objectives.
- Formal multi-agent models: Dec-POMDPs model decentralized decisions with partial observations, joint actions, state transitions, global rewards, observation spaces, and discounting.This formalism reflects agents operating with localized information in cooperative resource allocation tasks.
- Training and execution paradigms: CTCE centralizes control using global information, whereas DTDE lets agents act independently from local observations, creating a coordination-versus-scalability trade-off.CTCE can face exponential joint-action growth and communication demands, while DTDE may produce non-stationary environments because agents adapt to one another.
- Training and execution paradigms: MARL training paradigms differ in how they distribute information and decision-making, with the choice depending on resource scale, distribution, and interdependence.Figure 4 contrasts centralized and decentralized arrangements, while the surrounding discussion identifies their practical trade-offs.
4 RAO Leveraging MARL
This section frames MARL as a scalable and adaptive approach for modern resource allocation, then organizes algorithmic applications by domain and the challenges addressed in each application.
- MARL applications: MARL supports resource allocation where traditional methods face limitations in scalability, dynamic adaptation, and decentralized decision-making.The section presents MARL as a collaborative approach involving multiple agents in shared environments.
- Survey organization: The survey reviews available MARL algorithms across applications and follows each application with the primary challenges addressed in that field.This organization links algorithmic methods to domain-specific resource allocation problems.
4.1 MARL for RAO in Different Application Fields
MARL is applied to resource allocation across telecommunications, IoT, vehicular and computer networks, energy systems, distributed computing, edge computing, and fog computing. The reviewed applications emphasize decentralized coordination, adaptability, scalability, and resource management under dynamic conditions.
- MARL-based resource allocation is reviewed across telecommunications, IoT, vehicular networks, energy systems, distributed computing, MEC, and fog computing.
- Vehicular Networks: Vehicular-network studies use adaptive MARL to support distributed decision-making, partial observability, and responses to time-varying network conditions.Reported algorithms include MAD3QN and MADDPG for changing network conditions and user demands.
- Telecommunication and Computer Networks: Wireless and computer-network applications use MADQRL, MAPPO, and related frameworks for decentralized resource management, packet routing, virtual-network allocation, and joint handover and power control.
- IoT and Edge Computing: IoT and edge-computing studies apply MAPPO, MAD3QN, MAAC, Com-DDPG, and MATD3 to resource allocation and task offloading under scale, heterogeneity, and strict QoS requirements.
- Energy Systems: Energy applications use MARL for microgrid operation, residential energy management, renewable-energy scheduling, voltage control, and reactive-power regulation.
4.2 MARL as a Solution for Modern RAO Challenges
MARL addresses modern RAO challenges through centralized, decentralized, and hybrid learning paradigms. These approaches trade global coordination and adaptability against scalability, information limitations, computational cost, and decision quality.
- MARL extends RL to multiple interacting agents and supports centralized, decentralized, and hybrid resource-allocation paradigms.
- Adaptability and Centralization: Centralized training and execution can adapt quickly and optimize globally, but processing all agents’ information creates computational, memory, reliability, and privacy constraints.In one evaluated setting, SAC outperformed DQN and DDPG on time, energy consumption, utility, execution rate, and aging.
- Partial Observability: Partial observability limits agents’ knowledge of system-wide conditions and makes coordination difficult when local actions affect other agents.
- Decentralized Execution: CTDE trains agents with centralized environmental information while allowing decentralized execution from local observations.An attention mechanism added to a centralized critic was used to support customized energy demands during decentralized execution.
- Scalability: CTCE is suited to small-scale tasks requiring precise synchronization, whereas DTDE reduces centralized overhead and scales better but may reduce decision quality when information or coordination is insufficient.
- Scalability and Heterogeneity: Large-scale RAO becomes difficult because the joint action space grows exponentially with the number of agents and resources.Heterogeneous resources also differ in capacity, cost, efficiency, and task compatibility.
5 Available RAO Simulators for Benchmarking
The survey identifies publicly available Gymnasium-based environments for benchmarking MARL in RAO-related domains. These simulators represent spacecraft operations, power networks, urban traffic, and industrial waste processing.
- RAO benchmarking environments are commonly developed with the OpenAI Gymnasium library so they can integrate with reinforcement-learning algorithms.
- Spacecraft Operations: The BSK-RL spacecraft environment supports single- or multi-satellite missions with downlinking, imaging or scanning, charging, and desaturating actions.
- Power Networks: The power-grid benchmark models energy distribution across four zones and 33 bus networks, including voltage control and photovoltaic sources that complicate voltage stability.
- Urban Traffic: CityFlow simulates traffic-signal control for large-scale urban networks, supporting dynamic traffic conditions, multi-agent coordination, and configurable synthetic or real-world data.
- Tested Algorithms: The survey reports algorithms tested in the listed benchmarks and simulators in a separate summary table.
- Industrial Waste Processing: ContainerGym models continuously arriving waste material stored in multiple containers and allocated to processing units, with stochastic container flow rates and extensibility to MARL.
6 Future Directions and Potential Challenges
Future MARL research for RAO centers on scaling to larger and more dynamic systems, improving coordination, adaptability, security, and resilience. The survey also identifies partial observability, convergence, safety, communication overhead, and exploration as continuing challenges.
- Future Directions: Scalability and efficiency are primary future priorities as IoT, edge-computing, and smart-city systems involve more agents and larger environments.Hierarchical MARL, mean-field approximations, and decentralized learning are identified as possible approaches to computational and communication burdens.
- Dynamic Systems: Future applications include 6G networks, energy-efficient management, and autonomous transportation requiring real-time adaptive allocation and decentralized learning.
- Coordination: Improved coordination and communication mechanisms, including graph-based models and communication-free methods, are proposed for resource sharing and fairness.
- Adaptability: Future systems must improve adaptability and generalization across non-stationary environments and heterogeneous agents with diverse objectives.
- Emerging Technologies: Federated learning and quantum computing are identified as potential integrations for secure, efficient, decentralized, and resilient resource management.
- Potential Challenges: Key unresolved challenges include partial observability, convergence, scalability, safety constraints, communication overhead, and multi-agent exploration.
7 Conclusion
The survey reviews MARL methodologies and applications for RAO, emphasizing its potential for decentralized and dynamic resource allocation. It identifies persistent challenges and priorities for future research and deployment.
- The survey reviews core MARL methodologies, applications across diverse domains, and the strengths and limitations of existing approaches for RAO.
- Key remaining challenges include non-stationarity, limited scalability, coordination complexity, and insufficiently generalizable algorithms.
- Future research should develop improved training paradigms, adaptive communication mechanisms, and hybrid approaches integrating MARL with classical optimization.
- The field should prioritize deployment in emerging domains, standardized RAO benchmarks, and task-relevant evaluation metrics.