Source-linked AI summary
RODE: Learning Roles to Decompose Multi-Agent Tasks
Tonghan Wang, Tarun Gupta, Anuj Mahajan, Bei Peng, Shimon Whiteson, Chongjie Zhang
TL;DR
Role-based multi-agent learning lacks an efficient way to discover useful roles without prior task decomposition. RODE clusters actions by their effects to create restricted role action spaces and trains a bi-level role selector and role policies. It performs best on 10 of 14 StarCraft II maps and transfers to tasks with three times as many agents.
Problem
Efficiently discovering roles for task decomposition remains unclear, while learning roles from scratch still requires exploration in a large joint space.
Method
RODE learns effect-based action representations, clusters actions into restricted role action spaces, and trains a lower-resolution role selector with policies conditioned on action representations.
Results
RODE has the best performance on 10 of 14 StarCraft II maps, including all 9 hard and super hard maps, and transfers to tasks with three times as many agents.
Takeaways & Limitations
RODE provides a scalable and transferable role-based framework by decomposing cooperative learning into factored action spaces and hierarchical sub-problems.
Takeaways & Limitations
Transfer of learned policies to larger tasks remains a promising direction for future work.
Abstract
from arXiv · showhide
Role-based learning holds the promise of achieving scalable multi-agent learning by decomposing complex tasks using roles. However, it is largely unclear how to efficiently discover such a set of roles. To solve this problem, we propose to first decompose joint action spaces into restricted role action spaces by clustering actions according to their effects on the environment and other agents. Learning a role selector based on action effects makes role discovery much easier because it forms a bi-level learning hierarchy -- the role selector searches in a smaller role space and at a lower temporal resolution, while role policies learn in significantly reduced primitive action-observation spaces. We further integrate information about action effects into the role policies to boost learning efficiency and policy generalization. By virtue of these advances, our method (1) outperforms the current state-of-the-art MARL algorithms on 10 of the 14 scenarios that comprise the challenging StarCraft II micromanagement benchmark and (2) achieves rapid transfer to new environments with three times the number of agents. Demonstrative videos are available at https://sites.google.com/view/rode-marl .
1 INTRODUCTION
RODE addresses the difficulty of discovering roles for scalable multi-agent learning by first factoring joint action spaces according to action effects. Its hierarchy then coordinates role selection and restricted role policies, achieving strong benchmark performance and transfer to larger-agent tasks.
- CTDE methods avoid execution-time global information but still search exponentially growing joint action-observation spaces during training.
- Predefined roles require prior knowledge and may prevent role-based methods from transferring across environments.
- Automatically discovering roles from scratch remains difficult because it requires exploration in the same large joint space as other CTDE methods.
- RODE clusters actions by learned effects on the environment and other agents, then uses those clusters to define role action spaces and role observations.
- The bi-level hierarchy assigns roles in a smaller space at lower temporal resolution while role policies learn in reduced primitive action-observation spaces.
- RODE achieves the best performance on 10 of 14 StarCraft II maps, including all 9 hard and super hard maps, and transfers to tasks with three times as many agents.
2 RODE LEARNING FRAMEWORK
RODE learns effect-based action representations, clusters primitive actions into restricted role action spaces, and coordinates role selection with role policies in a bi-level hierarchy. The framework uses shared representations and centralized training mechanisms to reduce policy search while preserving decentralized execution.
- RODE models cooperative tasks as Dec-POMDPs with local observations, joint actions, shared rewards, and local action-observation histories.
- Roles associate agents with sub-tasks and shared policies, providing dynamic grouping that reduces target-policy search as the number of agents grows.
- Determining role action spaces: Action representations encode effects through information sufficient to predict next observations and rewards given current observations and other agents’ actions.
- Determining role action spaces: After initialization and predictive-model training, k-means clusters action representations into role action spaces, with outliers added to every cluster.
- Learning the role selector and role policies: The role selector assigns roles every c timesteps, after which agents choose primitive actions only from the corresponding restricted action spaces.
- Learning the role selector and role policies: The role selector represents each role by the average representation of its available actions, while role policies use action representations to estimate primitive-action values.
- Learning the role selector and role policies: A QMIX-style mixing network trains concurrent role assignments with global rewards, addressing coordination problems such as overkill.
3 RELATED WORK
Prior work addresses scalability through action-space representations, structured exploration, and curriculum learning, while multi-agent action-space organization remains challenging. RODE instead uses action representations to factor multi-agent tasks.
- Hierarchical MARL: Hierarchical reinforcement learning has been studied for sparse rewards and transfer learning through temporal task decomposition.
- Large Discrete Action Spaces: Large discrete action spaces have been addressed with binary action codes, continuous representations, predefined embeddings, and learned representations.These approaches aim to reduce action-space size or expose underlying structure for more efficient training.
- Large Discrete Action Spaces: Action representations have supported exploration and policy transfer across different tasks.
- Large Discrete Action Spaces: Curriculum-learning methods scale training by exposing agents to action spaces of increasing sizes.
- Large Discrete Action Spaces: RODE differs by using action representations to factor multi-agent tasks rather than only addressing large action spaces.
4 EXPERIMENTS
RODE is evaluated on action representations, SMAC performance, component ablations, and transfer to larger-agent environments. The experiments show that effect-based restricted role action spaces drive performance gains and support transfer without retraining.
- 4.1 ACTION REPRESENTATIONS: The predictive model learns action representations that reflect action effects across scenarios and remain robust across random seeds.On corridor, it identifies three action-effect clusters; on 3s5z vs 3s6z, it separates attacks on heterogeneous enemy types.
- 4.2 PERFORMANCE AND ABLATION STUDY: RODE outperforms VDN, QMIX, QPLEX, HSD, and ROMA by at least 1/32 on 10 of 14 SMAC scenarios after 2M training steps.It has the best performance on all 9 hard and super hard maps, while typically requiring more samples on 5 easy maps.
- 4.2 PERFORMANCE AND ABLATION STUDY: RODE’s strongest performance is mainly attributable to restricted role action spaces, while action representations improve learning and hierarchical structure contributes little by itself.Full or random restricted action spaces fail to outperform QMIX, whereas conventional Q-networks with restricted spaces remain close to full RODE.
- 4.3 POLICY TRANSFER: A policy trained on corridor transfers to unseen maps without further policy training and still wins 50% of games with three times the number of agents.The transfer setting also increases the number of actions as new enemy units are introduced.
- 4.3 POLICY TRANSFER: The authors identify iteratively training transferred policies on larger tasks as a promising direction for scaling MARL to large-scale problems.This statement is presented as a future direction beyond the demonstrated transfer experiment.
5 CONCLUSION
RODE addresses the long-standing difficulty of discovering roles by first decomposing joint action spaces according to action effects, then learning efficiently over the resulting factored spaces.
- 5 CONCLUSION: RODE first decomposes joint action spaces according to action effects instead of learning roles from scratch.The method uses this decomposition to make role discovery easier.
- 5 CONCLUSION: The framework uses hierarchical learning over factored action spaces to achieve efficient learning.The conclusion links this design to the method’s scalability and transferability.
- 5 CONCLUSION: RODE is intended to provide scalability and transferability for flexible, general-purpose multi-agent systems.This is the paper’s stated broader consequence of the approach.
A ARCHITECTURE, HYPERPARAMETERS, AND INFRASTRUCTURE
The implementation uses lightweight role policies and a compact role selector, with RMSprop optimization, epsilon-greedy exploration, k-means role-action clustering, and author-provided baseline implementations.
- ARCHITECTURE, HYPERPARAMETERS, AND INFRASTRUCTURE: Each role policy is a linear network without hidden layers or activation functions, while the role selector is a two-layer network with a 64-dimensional hidden layer.Agents share a trajectory encoder composed of a fully connected layer and a GRU with a 64-dimensional hidden state.
- ARCHITECTURE, HYPERPARAMETERS, AND INFRASTRUCTURE: RODE uses RMSprop with learning rate 5×10−4 and epsilon-greedy exploration annealed from 1.0 to 0.05 over 50K time steps.On three hard-exploration maps, epsilon annealing is extended to 500K time steps for RODE, baselines, and ablations.
- ARCHITECTURE, HYPERPARAMETERS, AND INFRASTRUCTURE: The role action spaces are determined with k-means clustering, using k=3 for homogeneous-enemy maps, k=5 for heterogeneous-enemy maps, and k=2 when one enemy is present.The passage notes that more advanced clustering could avoid treating k as a hyperparameter.
- ARCHITECTURE, HYPERPARAMETERS, AND INFRASTRUCTURE: Baseline algorithms use code provided by their authors with hyperparameters fine-tuned on the SMAC benchmark.
B CASE STUDY: ROLE DYNAMICS
The corridor case study shows how effect-based clustering creates interpretable roles whose selection supports exploration and coordinated combat, while restricted action spaces reduce learning complexity across maps.
- CASE STUDY: ROLE DYNAMICS: On corridor, the action encoder forms three role action spaces for moving eastward and northward, attacking, and moving westward and southward.These clusters correspond to actions with similar effects on the environment.
- CASE STUDY: ROLE DYNAMICS: Agents initially select Role 2 to move toward the edges, then alternate Role 0 and Role 1 to retract and attack enemies.This sequence supports the cooperative strategy described for corridor.
- CASE STUDY: ROLE DYNAMICS: Role assignments are visualized with colored ally-agent circles, while circle size represents remaining unit health.The figure’s first row contains game snapshots, and the second row contains corresponding role assignments.
- CASE STUDY: ROLE DYNAMICS: On 3s5z vs 3s6z, clustering decomposes the task into attacking Zealots and attacking Stalkers, reducing the complexity of learning the restricted sub-problems.
C EXPERIMENTAL DETAILS
The experimental-details section supplements the earlier results and discusses the settings and outcomes of rapid policy transfer.
- EXPERIMENTAL DETAILS: This section provides experimental results supplementary to those presented in Section 4.2.
- EXPERIMENTAL DETAILS: It discusses the details of the rapid policy transfer experimental settings.
- EXPERIMENTAL DETAILS: It analyzes the results of the rapid policy transfer experiments.
C.1 BENCHMARKING ON STARCRAFT II MICROMANAGEMENT TASKS
RODE outperforms baselines across the SMAC benchmark, with especially strong performance on super-hard maps, while using more samples than baselines on easy maps.
- Evaluation protocol: The benchmark reports median win rates with 25–75% percentiles from eight random seeds.These statistics are used for the map-by-map comparisons.
- Super-hard maps: RODE outperforms all baselines by a large margin on the most exploration-intensive super-hard maps.The strongest differences occur on 3s5z vs 3s6z, corridor, and 6h vs 8z.
- Easy maps: On easy maps, RODE tends to use more samples to achieve performance similar to baselines.The authors hypothesize that easy maps do not require substantial exploration, making RODE’s continued exploration disadvantageous.
- Overall benchmark: RODE establishes a new state of the art by outperforming all baselines on 10 of 14 SMAC scenarios.Its superiority is particularly apparent across all hard and super-hard maps.
C.2 POLICY TRANSFER
RODE transfers learned policies to similar tasks by matching new actions with old actions according to their effects and adding them to corresponding role action spaces.
- Transfer mechanism: RODE assumes new actions have similar functionalities to some existing actions and clusters them using an action encoder.The encoder identifies old actions with effects similar to each new action.
- Transfer mechanism: For each new action, RODE represents it using related original-task actions and assigns it to their corresponding role action spaces.The number of roles remains unchanged during transfer.
- Outcome: The transfer procedure enables the original role selector and role policies to be rapidly transferred to a similar task.Transfer depends on matching new and old actions by functionality.
- Experimental setting: RODE is trained on corridor for 5 million timesteps before transfer to similar maps with increased numbers of agents and enemies.Each additional enemy introduces a new attack action.
D.1 ROLE INTERVAL
The role interval controls how often roles are reassigned and action spaces change, and it substantially affects RODE’s performance across map difficulties.
- Definition: The role interval c is the number of timesteps between consecutive role selections.The role selector assigns roles to each agent every c timesteps.
- Influence: Changing the role interval significantly influences performance across easy, hard, and super-hard environments.The study evaluates two easy maps, one hard map, and one super-hard map.
- Selected settings: Role intervals of 5 or 7 typically produce satisfactory results, and the paper uses 5 on most maps.The reported experiments keep other components unchanged while varying the interval.
D.2 RECURRENT ROLE SELECTOR AND FULLY-CONNECTED ROLE SELECTOR
A recurrent role selector generally performs better than a fully connected selector based only on local observations, but its advantage depends on the role interval and environment.
- Selector comparison: RODE compares a recurrent role selector with a fully connected selector conditioned on local observations.The comparison examines the influence of recurrent processing in role selection.
- Selector comparison: The recurrent role selector performs better than the fully connected local-observation selector.The paper reports this as the expected comparison outcome.
- Role-interval interaction: On 10m vs 11m, a role interval of 7 gives recurrent RODE performance similar to the fully connected selector, whereas other intervals underperform.This indicates an interaction between role interval, exploration, optimization, and learning stability.
- Related work: Role-based learning is motivated by decomposing tasks into sub-tasks and specializing agents with the same role.Prior role-based systems commonly predefined roles using prior knowledge, which can hurt generalization.